IP calculations in BASH with bitwise operations

This simple BASH script uses a network CIDR and two reserved IPs to calculate the pool range for a DHCP configuration. The two reserved IPs can be any valid IP within the CIDR. The IP range with the largest number of available IP, not using one of the static IPs, is selected for use as the pool. The highest and lowest IPs of the pool are returned.

The script uses ipcalc for getting the network values from the network CIDR. It is able to get values for Network, Netmask, Address, Broadcast, HostMin, HostMax and the number of host addresses in the network.

Unit conversion from decimal to binary is done using bc. BASH is able to perform bitwise operations for AND, OR and XOR. The BASH operation of shift is also used in this script.

As always you can download this code along with all the others from this blog using GitLab.com Linux Code Snippets

#!/bin/bash

# Given the network CIDR and two static IPs, calculate the lower and upper IPs for the DHCP pool.
# This script uses ipcalc that may need to be installed.


# https://stackoverflow.com/questions/40667382/how-to-perform-bitwise-operations-on-hexadecimal-numbers-in-bash
# https://unix.stackexchange.com/questions/223338/convert-a-value-into-a-binary-number-in-a-shell-script
# echo "obase=2; 34" | bc

# https://unix.stackexchange.com/questions/65280/binary-to-hexadecimal-and-decimal-in-a-shell-script
# echo $((2#101010101))
# printf "%x\n" "$((2#101010101))"
# echo $((0xf8 ^ 0x1f)) XOR
# echo $((0xf8 & 0x1f)) AND
# echo $((0xf8 | 0x1f)) OR
# echo $((0xf8 >> 3)) shift right 3 bits
# echo $((0xf8 << 4)) shift left 3 bits

function AddOne {
   IPCI=
   [ "$1" = "NS_ADDR" ] && IPCI=$(( NS_ADDR_INT + 1 ))
   [ "$1" = "GW_ADDR" ] && IPCI=$(( GW_ADDR_INT + 1 ))
   IPCB=$(echo "obase=2; $IPCI" | bc)
   Bin2Addr $(( 2#$B_ADDRESS_MASK | 2#$IPCB ))
}

function SubOne {
   IPCI=
   [ "$1" = "NS_ADDR" ] && IPCI=$(( NS_ADDR_INT - 1 ))
   [ "$1" = "GW_ADDR" ] && IPCI=$(( GW_ADDR_INT - 1 ))
   IPCB=$(echo "obase=2; $IPCI" | bc)
   Bin2Addr $(( 2#$B_ADDRESS_MASK | 2#$IPCB ))
}

function Bin2Addr {
   GA=$(echo "obase=2; $1" | bc)
   D_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   C_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   B_BLOCK=$(( 2#$GA & 2#11111111 ))
   GA=$( echo "obase=2; $(( 2#$GA >> 8 ))" | bc)
   A_BLOCK=$(( 2#$GA & 2#11111111 ))
   echo "built IP = '${A_BLOCK}.${B_BLOCK}.${C_BLOCK}.${D_BLOCK}'"
}

function NetCalcs {
   echo
   echo --------------------------------------------------
   # set -x
   echo "NAMESERVER_IP=$NAMESERVER_IP, GATEWAY_IP=$GATEWAY_IP"
   NAMESERVER_NET=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Network: | awk '{print $2}' )
   #echo "NAMESERVER_NET=$NAMESERVER_NET"
   NETWORK_MASK=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Netmask: | awk '{print $2}' )
   #echo "NETWORK_MASK=$NETWORK_MASK"
   B_ADDRESS_MASK=$(ipcalc -n ${NETWORK_CIDR} | grep ^Address: | sed -e 's/\.//g' | awk '{print $(NF-1)$NF}' )
   #echo "Binary ADDRESS_MASK=$B_ADDRESS_MASK"
   NETWORK_BROADCAST=$(ipcalc -nb ${NETWORK_CIDR} | grep ^Broadcast: | awk '{print $2}' )
   #echo "NETWORK_BROADCAST=$NETWORK_BROADCAST"
   NET_MIN_IP=$(ipcalc -nb ${NETWORK_CIDR} | grep ^HostMin: | awk '{print $2}' )
   #echo "NET_MIN_IP=$NET_MIN_IP"
   NET_MAX_IP=$(ipcalc -nb ${NETWORK_CIDR} | grep ^HostMax: | awk '{print $2}' )
   #echo "NET_MAX_IP=$NET_MAX_IP"
   NET_MIN_BIN=$(ipcalc -n ${NETWORK_CIDR} | grep ^HostMin: | sed -e 's/\.//g' | awk '{print $NF}')
   #echo "NET_MIN_BIN=$NET_MIN_BIN"
   NET_MAX_BIN=$(ipcalc -n ${NETWORK_CIDR} | grep ^HostMax: | sed -e 's/\.//g' | awk '{print $NF}')
   #echo "NET_MAX_BIN=$NET_MAX_BIN"
   NS_ADDR_BIN=$(ipcalc -n ${NAMESERVER_IP}/${NETWORK_CIDR##*/} | grep ^Address: | sed -e 's/\.//g' | awk '{print $NF}')
   NS_ADDR_INT=$(( 2#$NS_ADDR_BIN ))
   #echo "NS_ADDR_BIN=$NS_ADDR_BIN"
   GW_ADDR_BIN=$(ipcalc -n ${GATEWAY_IP}/${NETWORK_CIDR##*/} | grep ^Address: | sed -e 's/\.//g' | awk '{print $NF}')
   GW_ADDR_INT=$(( 2#$GW_ADDR_BIN ))
   #echo "GW_ADDR_BIN=$GW_ADDR_BIN"

   read LOWEST HIGHEST <<< $( printf "%d " $(printf "%d\n" $(( 2#$GW_ADDR_BIN )) $(( 2#$NS_ADDR_BIN ))|sort -n))
   LOWCOUNT=$(( LOWEST - $NET_MIN_BIN ))
   #echo "LOWCOUNT=$LOWCOUNT"

   # Mid IP pool size
   MIDCOUNT=$(( HIGHEST - LOWEST ))
   #echo "MIDCOUNT=$MIDCOUNT"

   # Upper IP pool size
   HICOUNT=$(( 2#$NET_MAX_BIN - HIGHEST ))
   #echo "HICOUNT=$HICOUNT"


   read JUNK LOWNET JUNK HIGHNET JUNK <<< $( echo $( (echo "$GW_ADDR_INT GW_ADDR"; echo "$NS_ADDR_INT NS_ADDR"; ) | sort -n) )
   read JUNK POOL JUNK <<< $( (echo "$LOWCOUNT LOWCOUNT"; echo "$MIDCOUNT MIDCOUNT"; echo "$HICOUNT HICOUNT";) | sort -nr)


   #echo "NAMESER_IP=$NAMESERVER_IP GATEWAY_IP=$GATEWAY_IP LOWNET=$LOWNET HIGHNET=$HIGHNET POOL=$POOL"

   case $POOL in
      LOWCOUNT)
         echo "From $NET_MIN_IP to $(SubOne $LOWNET)";;
      MIDCOUNT)
         echo "From $(AddOne $LOWNET) to $(SubOne $HIGHNET)";;
      HICOUNT)
         echo "From $(AddOne $HIGHNET) to $NET_MAX_IP";;
   esac

}


clear
NETWORK_CIDR="172.16.0.0/20"
echo "NETWORK_CIDR=$NETWORK_CIDR"

NAMESERVER_IP="172.16.1.2"
GATEWAY_IP="172.16.1.1"
NetCalcs


NAMESERVER_IP="172.16.14.200"
GATEWAY_IP="172.16.15.9"
NetCalcs


NAMESERVER_IP="172.16.1.22"
GATEWAY_IP="172.16.1.22"
NetCalcs


NETWORK_CIDR="192.168.1.0/24"
echo "NETWORK_CIDR=$NETWORK_CIDR"

NAMESERVER_IP="192.168.1.2"
GATEWAY_IP="192.168.1.1"
NetCalcs


NAMESERVER_IP="192.168.1.120"
GATEWAY_IP="192.168.1.121"
NetCalcs


NAMESERVER_IP="192.168.1.2"
GATEWAY_IP="192.168.1.1"
NetCalcs

The One Line Python Directory Web Server

While testing some adjustments to iptables I needed a simple way to test web requests. This is a handy way to serve a directory and a simple way to see if the firewall is working.

On the server-side create directory content that will be hosted from a one-line Python script.

mkdir /tmp/fakeserver

cd /tmp/fakeserver

git clone https://github.com/torvalds/linux.git

sudo python -m SimpleHTTPServer 80

On the client-side use wget to spider and download the entire content of the server.
Replace <TEST SERVER> with the servers IP or resolvable name.

mkdir /tmp/fakeserver

cd /tmp/fakeserver

wget --mirror --convert-links --adjust-extension \
--page-requisites --no-parent http://<TEST SERVER>/

nmon on your headless Proxmox server

Proxmox is great and after it is up and running you can control it from the web page or ssh in, so what do you do with the monitor connected to it. Here is a one-line command to get nmon to show you what the server is doing.

I set my script up in cron so it always comes up, also nmon will not refresh the entire screen so cron can also kill the old nmon and start a new one.

I called this headless because you don't need to login to have this work. The pgrep command finds the getty that has the login prompt.

( sleep 2; 
export NMON=cmVdn; 
export TERM=xterm-color; 
export NCURSES_NO_UTF8_ACS=1; 
nmon > /proc/$(pgrep -f agetty)/fd/1 ) &

And this is what it looks like.

Ooooh that shell, Can't you shell that shell

Staying up late to work on a fun project. Building a custom PXE boot image that will be used to pre-load our servers and get them ready for Puppet management. It has been about eight years since I had built a custom boot image. At that time busybox was my goto for small, powerful environments that needed to run from very small RAM mounted file systems.

BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

When most distros offer boot images that do little more than boot, or fail to boot; for my systems I used busybox to create a very compact environment that would allow you to perform diagnostics and fix any problems.

As great as busybox was at the time, I found it lacked a lot of compatibilities with even old POSIX systems and scripts. Arguments that you rely on for find, grep and even ls, just were not there. Things have changed; somewhat. Busybox has grown up and is full of new abilities.

Screenshot of "make menuconf" providing the ability
to choose the features to build into the binary.

The ancestry of a process

When it comes to viewing the heritage of a process, the go-to command is pstree. By itself, pstree will show the descendant processes. Using the -s argument it will show ancestor processes. A good trick to remember is slap, pstree -slap <PID>.

I personally like the format of ps and being the kind of person that likes to come up with one line tools, I created this little snippet.

{ TPID=$$; while [[ $TPID -ne 1 ]]; do ps -f $TPID | tail -n1; TPID=$(ps h -o ppid -p $TPID); done; ps -f $TPID|tac; } | tac

Saving it to use later we get this.

#!/bin/bash

{ 
        TPID=${1:-$$}; 
        while [[ $TPID -ne 1 ]]; do 
                ps -f $TPID | tail -n1; 
                TPID=$(ps h -o ppid -p $TPID); 
        done; 
        ps -f $TPID | tac; 
} | tac

This is what the two commands output.

Sed: Append file to end of line

Sed can let you replace part of a line with new content from another file. This is how I do this in three steps. You can see the code run a codingground

By using a file to insert the replacement text the content can contain special characters that would normally require a lot of extra work to escape. 

#!/bin/bash

# Replace the right portion of a line with the content of a file.
# 1) Replace the right portion of a line with a marker word
# 2) Append after the marker with the content from a file
# 3) Remove the marker and new line to pull the next line up.

cat > outfile <<!EOF
hello there tom
        How are you today?
    Where is the dog show?
!EOF

cat outfile

cat > testfile <<!EOF
flowers doing?
!EOF


sed -i outfile -e 's/you.*/REPLACE_MARKER/g'
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/r testfile"
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/{:a;N;s/REPLACE_MARKER\n//}"
printf "\n\n\n"
cat outfile

BASH date picker

Sometimes I need users to pick a date from within a bash shell but I need the format to be correct. This little script lets users pick a date and it will output in whatever format the script needs.

Users can use the arrow keys to move the date on the calendar and press enter to select the date.

#!/bin/bash

DONE=0
DATE=$(date +%F)
YEAR="$(echo $DATE | cut -b1-4)"
MONTH="$(echo $DATE | cut -b6-7)"
DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
while [ $DONE -eq 0 ]; do
 clear
 cal -h ${MONTH} ${YEAR} | GREP_COLOR='47;30' grep "$DAY" -wC6 --color=always
 read -rsn1 KEY
 [[ $KEY == $'\x1b' ]] && { 
  read -rsn1 -t 0.1 DC
  read -rsn1 -t 0.1 KP
  case $KP in
   "A") DATE=$(date +%F -d "$DATE 7 day ago");;
   "B") DATE=$(date +%F -d "$DATE 7 day");;
   "C") DATE=$(date +%F -d "$DATE 1 day");;
   "D") DATE=$(date +%F -d "$DATE 1 day ago");;
  esac
 } || { 
  DONE=1
 }
 
 YEAR="$(echo $DATE | cut -b1-4)"
 MONTH="$(echo $DATE | cut -b6-7)"
 DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
done
echo $DATE

# To change the output format use the date command like these examples do
# date +%d/%m/%Y -d "$DATE"
# date -d "$DATE"

3.141592653589793238462643383279

Happy Pi day everyone. March 14th is Pi day because 3.14.

I created a BASH script to test the calculation in this video. It calculates Pi without pre-known values but by adding and subtracting fractions of one over odd numbers. 1/1 - 1/3 + 1/5 - 1/7 + 1/9 - 1/11 + 1/13 ... It gets more accurate the longer you let it run but only for a short time as fractional math in BASH is limited.

#!/bin/bash

D=1
P=0
i=1
while true; do
 O="-"
 (( i++ % 2 )) && O="+"
 P=$(echo "scale=66; ${P} ${O} 1/${D}" | bc)
 printf "%0.20f               \r" $(echo "scale=21; ${P} * 4" | bc)
 ((D+=2))
done

Tunneling UDP

We use Graylog at work, a very nice tool that collects logs from all our systems and helps us analyze them. We ran into a problem with a special cluster of servers that are isolated and could not get to our Graylog server. We only needed to connect them to Graylog for a short time so this is not a long term solution.

These systems have never been able to connect to Graylog so they were configured to send all logs to 127.0.0.1, and that is great because we can setup a listening tunnel on localhost to forward the connection and we can do all this without making changes to the target systems.

The Graylog server has SSH access to the target systems so we use that to create a reverse tunnel. SSH works over TCP but Graylog uses UDP so we use NC (netcat) to convert the UDP to TCP for tunneling and then convert it back again on the server side.

The first command forks a local listener to send TDP packets to our Graylog on UDP.
The second command opens a reverse tunnel and starts listening on UDP to send packets back via the tunnel over TCP.

nc -l 127.0.0.1 12203 | nc -u 127.0.0.1 12201 &
ssh target2 -R 12201:localhost:12203 "nc -kluw 0 127.0.0.1 12201|nc 127.0.0.1 12201"

Some important points.

• Each connection needs its own port, Graylog is using 12201, so target1 would listen on 12202 and target2 is using 12203
• Netcat wants to disconnect when the logger on target finishes a message to that nc uses 'k' to keep the connection alive and is uses 'w 0' to have no wait time out.

Update

After a day or so running netcat likes to die so this little change will automatically relaunch nc instantly by running this on the target side.

while nc -kluw 0 127.0.0.1 12201; do true; done | while cat - | nc 127.0.0.1 12201; do true; done

Stop eating my pipe!

We love to use while loops in our scripts and they are a great way to read a file one line at a time to get a job done.

For example:

cat myFile.txt | while read LINE; do
   echo $LINE
   sleep 1
done

Now comes along SSH and lets say that your file contains a list hostnames you want to get uptime from:

cat myHostsFile.txt | while read LINE; do
   echo -n "Host = $LINE "
   ssh $LINE "updtime"
done

How disappointed you are when your loops stops after the first host. This is because every child process inherits it's first three file descriptors from it's parent, so SSH takes everything from STDIN for itself.

Sometimes you may want that but this time you don't. What can you do?

The simple solution here is to disassociate SSH from STDIN, and you do this using a simple re-director.

cat myHostsFile.txt | while read LINE; do
   echo -n "Host = $LINE "
   ssh 0>/dev/zero $LINE "updtime"
done

Something to keep in mind is that STDIN supplies a data stream, it does not take it. So for this reason we attache to /dev/zero not /dev/null.

Some tail or some head

Often times I find that head and tail don't put out enough. Some times I just want more. I don't want to create aliases for them because there are times that I just need it to work normally and I am too lazy to absolute path to remove the alias.

This is how I am able to me some tail and some head with more.

In my home I have a bin directory that is just for my bin scripts, but you could put this in /usr/local/bin/ if you like.

I wanted one script that does both, so I use basename and the built in BASH test replacement operation to strip the command down to either head or tail. I created the script as somehead and then made a symlink that points to it called sometail.

$(basename ${0//some/}) -n $(( $(tput lines) - 3 ))

Notice this uses tput to get the number of lines in the terminal and then subtracts three, just so it does not overflow the screen and you get an idea of where the output started.

Remember to

chmod 755 somehead
ln -s somehead sometail

Alpha sequence

When doing a quick little shell script the seq command is very handy to generate a list of numbers but all too often I find that I need a sequence of letters and there is no alphabetical version of seq. There are hundreds of examples on the Internet of how to do this in almost every script language you can think of, this is my version in bash.

This version has the added feature of being able to run up a sequence of upper or lower case letter. The script only accepts one argument of the last letter, but could easily be expanded to do all sorts of extra tricks.

#!/bin/bash

OUT=""
TARGET=$(echo ${1:0:1} | grep -i "[a-z]")
UPPER=$(echo $TARGET | tr "[a-z]" "[A-Z]")
START=97 # a
[ "$TARGET" = "$UPPER" ] && {
 START=65 # A
}
while [ "$OUT" != "$1" ]; do
 OUT=$(awk 'BEGIN{printf "%c",'${START}'}')
 echo -n "$OUT "
 let "START=$START+1"
done

Examples:

$ alphaseq J
A B C D E F G H I J

$ alphaseq m
a b c d e f g h i j k l m

Running a tight BASH script

A few things I like to do with important bash scripts so that they live on as useful tools for many years is:
1) Send error output to the standard system logger.
2) Clean up any child processes that were started
3) Clean up any temporary files


You don't always need all of these but they are good to have handy.

Notice how trap is used to call a function so that the exit of the script can do a few extra things.

#!/bin/bash

# Log any errors to the standard system logs
exec 2> >(logger -s -t $(basename $0))

# Clean up when the program exits
function CleanExit {
 # stop any long running commands
 for k in $(jobs -p); do { kill -p $k; }

 # remove any temporary files created
 # rm -f $TEMPFILE
 exit
}
trap "CleanExit" EXIT

# Your code goes here

Get values and catch exit code in one line of BASH

Often it is helpful if a variable is only used when a command is successful.

FILE_SIZE=$(stat --format="%s" /bin/bash) && { 
 echo "Size is $FILE_SIZE"; 
} || { 
 echo "File size is unknown."; 
}

This becomes tricky if you used a pipe to parse the result because the pipe will fork a new shell completely isolated from the original command. Using PIPESTATUS it is possible to check the exit code of the previous command to the left and then it is easy to test the result.

HOST_IP=$(host $SOME_HOST_NAME | awk '{print $NF}'; [ ${PIPESTATUS[0]} -eq 0 ] ) && {
 echo "Host IP is $HOST_IP"
} || {
 echo "no such host"
}

Bash: Embed functions in commands.

Sometimes you want to do a complex action with your shell script but you don't want the mess of creating another shell file to call externally.

You can export the function and use it much like a call back.

#!/bin/bash

function Stats {    
    FILE="$@"
    STAT=$(stat --format="%s" "$FILE")
    echo "File $FILE is $STAT bytes"
}
export -f Stats
find . -type f -exec bash -c 'Stats "{}"' \;

Bash me with a pipe

I love to use the pipe. It is a simple tool that brings so much power. So this is how you can take things to the next level.

The snippet below shows how to redirect STDOUT to the STDIN of one script and the STDERR to the STDIN of another.

You cannot use a pipe for both because it will only redirect the file descriptor connected at fd1, normally STDOUT unless you change it. When you use a pipe everything to the right becomes a different program, so STDERR needs a different solution.

The solution is to redirect fd2 before you use the pipe on fd1.

$ crontab -l
# This cron entry sends STDERR to MyErrorLogger.sh and STDOUT to MyLogger.sh
* *  *  *  * ./MyScript.sh 2> >(MyErrorLogger.sh) | MyLogger.sh



$ cat MyLogger.sh 
#!/bin/bash
# This takes STDIN and sends it out to a file that has the date in its name
cat - >> mylog_$(date +%F).log 



$ cat MyErrorLogger.sh 
#!/bin/bash
# This takes STDIN and sends it out to a file that has the date in its name
cat - >> myerrors_$(date +%F).log



$ cat MyScript.sh
#!/bin/bash
# This sends sample text to both STDOUT and STDERR
echo "Hello world!" | tee /dev/fd/2

Better shell history for power users.

One of the very first things I set up on any new OS install is my history. I like to keep a lot of it and I like to know when I ran the command. This is very easy to do with these two environment variables set in your profile. You can set them locally or globally.

[royce@royce-desktop ~]$ cat /etc/profile.d/history.sh 
export HISTSIZE=100000
export HISTTIMEFORMAT="[%F %T] "
[royce@royce-desktop ~]$ history |tail -n 2
74519  [2015-01-22 10:52:40] cat /etc/profile.d/history.sh 
74520  [2015-01-22 10:52:41] history |tail -n 2