Wednesday, 18 March 2020

IP calculations in BASH with bitwise operations

This simple BASH script uses a network CIDR and two reserved IPs to calculate the pool range for a DHCP configuration. The two reserved IPs can be any valid IP within the CIDR. The IP range with the largest number of available IP, not using one of the static IPs, is selected for use as the pool. The highest and lowest IPs of the pool are returned.

The script uses ipcalc for getting the network values from the network CIDR. It is able to get values for Network, Netmask, Address, Broadcast, HostMin, HostMax and the number of host addresses in the network.

Unit conversion from decimal to binary is done using bc. BASH is able to perform bitwise operations for AND, OR and XOR. The BASH operation of shift is also used in this script.

As always you can download this code along with all the others from this blog using GitLab.com Linux Code Snippets


#!/bin/bash

# Given the network CIDR and two static IPs, calculate the lower and upper IPs for the DHCP pool.
# This script uses ipcalc that may need to be installed.


# https://stackoverflow.com/questions/40667382/how-to-perform-bitwise-operations-on-hexadecimal-numbers-in-bash
# https://unix.stackexchange.com/questions/223338/convert-a-value-into-a-binary-number-in-a-shell-script
# echo "obase=2; 34" | bc

# https://unix.stackexchange.com/questions/65280/binary-to-hexadecimal-and-decimal-in-a-shell-script
# echo $((2#101010101))
# printf "%x\n" "$((2#101010101))"
# echo $((0xf8 ^ 0x1f)) XOR
# echo $((0xf8 & 0x1f)) AND
# echo $((0xf8 | 0x1f)) OR
# echo $((0xf8 >> 3)) shift right 3 bits
# echo $((0xf8 << 4)) shift left 3 bits

function AddOne {
   IPCI=
   [ "$1" = "NS_ADDR" ] && IPCI=$(( NS_ADDR_INT + 1 ))
   [ "$1" = "GW_ADDR" ] && IPCI=$(( GW_ADDR_INT + 1 ))
   IPCB=$(echo "obase=2; $IPCI" | bc)
   Bin2Addr $(( 2#$B_ADDRESS_MASK | 2#$IPCB ))
}

function SubOne {
   IPCI=
   [ "$1" = "NS_ADDR" ] && IPCI=$(( NS_ADDR_INT - 1 ))
   [ "$1" = "GW_ADDR" ] && IPCI=$(( GW_ADDR_INT - 1 ))
   IPCB=$(echo "obase=2; $IPCI" | bc)
   Bin2Addr $(( 2#$B_ADDRESS_MASK | 2#$IPCB ))
}

function Bin2Addr {
   GA=$(echo "obase=2; $1" | bc)
   D_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   C_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   B_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   A_BLOCK=$(( 2#$GA & 2#11111111 ))
   echo "built IP = '${A_BLOCK}.${B_BLOCK}.${C_BLOCK}.${D_BLOCK}'"
}

function NetCalcs {
   echo
   echo --------------------------------------------------
   # set -x
   echo "NAMESERVER_IP=$NAMESERVER_IP, GATEWAY_IP=$GATEWAY_IP"
   NAMESERVER_NET=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Network: | awk '{print $2}' )
   #echo "NAMESERVER_NET=$NAMESERVER_NET"
   NETWORK_MASK=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Netmask: | awk '{print $2}' )
   #echo "NETWORK_MASK=$NETWORK_MASK"
   B_ADDRESS_MASK=$(ipcalc -n ${NETWORK_CIDR} | grep ^Address: | sed -e 's/\.//g' | awk '{print $(NF-1)$NF}' )
   #echo "Binary ADDRESS_MASK=$B_ADDRESS_MASK"
   NETWORK_BROADCAST=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Broadcast: | awk '{print $2}' )
   #echo "NETWORK_BROADCAST=$NETWORK_BROADCAST"
   NET_MIN_IP=$(ipcalc -nb ${NETWORK_CIDR} | grep ^HostMin: | awk '{print $2}' )
   #echo "NET_MIN_IP=$NET_MIN_IP"
   NET_MAX_IP=$(ipcalc -nb ${NETWORK_CIDR} | grep ^HostMax: | awk '{print $2}' )
   #echo "NET_MAX_IP=$NET_MAX_IP"
   NET_MIN_BIN=$(ipcalc -n ${NETWORK_CIDR} | grep ^HostMin: | sed -e 's/\.//g' | awk '{print $NF}')
   #echo "NET_MIN_BIN=$NET_MIN_BIN"
   NET_MAX_BIN=$(ipcalc -n ${NETWORK_CIDR} | grep ^HostMax: | sed -e 's/\.//g' | awk '{print $NF}')
   #echo "NET_MAX_BIN=$NET_MAX_BIN"
   NS_ADDR_BIN=$(ipcalc -n ${NAMESERVER_IP}/${NETWORK_CIDR##*/} | grep ^Address: | sed -e 's/\.//g' | awk '{print $NF}')
   NS_ADDR_INT=$(( 2#$NS_ADDR_BIN ))
   #echo "NS_ADDR_BIN=$NS_ADDR_BIN"
   GW_ADDR_BIN=$(ipcalc -n ${GATEWAY_IP}/${NETWORK_CIDR##*/} | grep ^Address: | sed -e 's/\.//g' | awk '{print $NF}')
   GW_ADDR_INT=$(( 2#$GW_ADDR_BIN ))
   #echo "GW_ADDR_BIN=$GW_ADDR_BIN"

   read LOWEST HIGHEST <<< $( printf "%d " $(printf "%d\n" $(( 2#$GW_ADDR_BIN )) $(( 2#$NS_ADDR_BIN ))|sort -n))
   LOWCOUNT=$(( LOWEST - $NET_MIN_BIN ))
   #echo "LOWCOUNT=$LOWCOUNT"

   # Mid IP pool size
   MIDCOUNT=$(( HIGHEST - LOWEST ))
   #echo "MIDCOUNT=$MIDCOUNT"

   # Upper IP pool size
   HICOUNT=$(( 2#$NET_MAX_BIN - HIGHEST ))
   #echo "HICOUNT=$HICOUNT"


   read JUNK LOWNET JUNK HIGHNET JUNK <<< $( echo $( (echo "$GW_ADDR_INT GW_ADDR"; echo "$NS_ADDR_INT NS_ADDR"; ) | sort -n) )
   read JUNK POOL JUNK <<< $( (echo "$LOWCOUNT LOWCOUNT"; echo "$MIDCOUNT MIDCOUNT"; echo "$HICOUNT HICOUNT";) | sort -nr)


   #echo "NAMESER_IP=$NAMESERVER_IP GATEWAY_IP=$GATEWAY_IP LOWNET=$LOWNET HIGHNET=$HIGHNET POOL=$POOL"

   case $POOL in
      LOWCOUNT)
         echo "From $NET_MIN_IP to $(SubOne $LOWNET)";;
      MIDCOUNT)
         echo "From $(AddOne $LOWNET) to $(SubOne $HIGHNET)";;
      HICOUNT)
         echo "From $(AddOne $HIGHNET) to $NET_MAX_IP";;
   esac

}


clear
NETWORK_CIDR="172.16.0.0/20"
echo "NETWORK_CIDR=$NETWORK_CIDR"

NAMESERVER_IP="172.16.1.2"
GATEWAY_IP="172.16.1.1"
NetCalcs


NAMESERVER_IP="172.16.14.200"
GATEWAY_IP="172.16.15.9"
NetCalcs


NAMESERVER_IP="172.16.1.22"
GATEWAY_IP="172.16.1.22"
NetCalcs


NETWORK_CIDR="192.168.1.0/24"
echo "NETWORK_CIDR=$NETWORK_CIDR"

NAMESERVER_IP="192.168.1.2"
GATEWAY_IP="192.168.1.1"
NetCalcs


NAMESERVER_IP="192.168.1.120"
GATEWAY_IP="192.168.1.121"
NetCalcs


NAMESERVER_IP="192.168.1.2"
GATEWAY_IP="192.168.1.1"
NetCalcs

Sunday, 16 February 2020

Using Jq to work with JSON

While starting to work with Packer & Terraform, our deployment server needed some information about the AMIs that are being built. This sounds like a job for jq


#!/bin/bash

TMP=$(mktemp /dev/shm/JQ_XXXXXXXXXXXXXXX)
echo $TMP

echo Create a new JSON data set.
sleep 1
jq -Rn '{ "firewall": { amiId: "ami-1234", base: "amazon-linux-2", built: "'$(date +%F_%T)'" }}' > $TMP
cat $TMP | jq
sleep 5

echo Append to a JSON data set.
sleep 1
cat $TMP | jq '.["webserver"].amiId="ami-5678"' > ${TMP}.swap && mv ${TMP}.swap $TMP
cat $TMP | jq '.["webserver"].base="centos-7"' > ${TMP}.swap && mv ${TMP}.swap $TMP
cat $TMP | jq
sleep 5

echo Get all values for an entry by name
sleep 1
cat $TMP | jq '.["firewall"]'
sleep 5

echo Get the amiId by name
sleep 1
cat $TMP | jq '.["firewall"].amiId'
sleep 5

echo Set the amiId by name
sleep 1
cat $TMP | jq '.["firewall"].amiId="AMI-9876"' # > ${TMP}.swap && mv swap $TMP
sleep 5

echo Select a data set by exact match
sleep 1
cat $TMP | jq 'with_entries(select(.value.base=="centos-7"))'
sleep 5

echo Select a data set by partial match
sleep 1
cat $TMP | jq 'with_entries(select(.value.base | startswith("amazon")))'
sleep 5

rm -f $TMP


Create a new JSON data set.









Append to a JSON data set.












Get all values for an entry by name.







Get the amiId by name




Set the amiId by name













Select a data set by exact match







Select a data set by partial match


























Tuesday, 11 February 2020

The One Line Python Directory Web Server

While testing some adjustments to iptables I needed a simple way to test web requests. This is a handy way to serve a directory and a simple way to see if the firewall is working.

On the server-side create directory content that will be hosted from a one-line Python script.
mkdir /tmp/fakeserver

cd /tmp/fakeserver

git clone https://github.com/torvalds/linux.git

sudo python -m SimpleHTTPServer 80


On the client-side use wget to spider and download the entire content of the server.
Replace <TEST SERVER> with the servers IP or resolvable name.
mkdir /tmp/fakeserver

cd /tmp/fakeserver

wget --mirror --convert-links --adjust-extension \
--page-requisites --no-parent http://<TEST SERVER>/