Using Jq to work with JSON

While starting to work with Packer & Terraform, our deployment server needed some information about the AMIs that are being built. This sounds like a job for jq

#!/bin/bash

TMP=$(mktemp /dev/shm/JQ_XXXXXXXXXXXXXXX)
echo $TMP

echo Create a new JSON data set.
sleep 1
jq -Rn '{ "firewall": { amiId: "ami-1234", base: "amazon-linux-2", built: "'$(date +%F_%T)'" }}' > $TMP
cat $TMP | jq
sleep 5

echo Append to a JSON data set.
sleep 1
cat $TMP | jq '.["webserver"].amiId="ami-5678"' > ${TMP}.swap && mv ${TMP}.swap $TMP
cat $TMP | jq '.["webserver"].base="centos-7"' > ${TMP}.swap && mv ${TMP}.swap $TMP
cat $TMP | jq
sleep 5

echo Get all values for an entry by name
sleep 1
cat $TMP | jq '.["firewall"]'
sleep 5

echo Get the amiId by name
sleep 1
cat $TMP | jq '.["firewall"].amiId'
sleep 5

echo Set the amiId by name
sleep 1
cat $TMP | jq '.["firewall"].amiId="AMI-9876"' # > ${TMP}.swap && mv swap $TMP
sleep 5

echo Select a data set by exact match
sleep 1
cat $TMP | jq 'with_entries(select(.value.base=="centos-7"))'
sleep 5

echo Select a data set by partial match
sleep 1
cat $TMP | jq 'with_entries(select(.value.base | startswith("amazon")))'
sleep 5

rm -f $TMP

Create a new JSON data set.

Append to a JSON data set.

Get all values for an entry by name.

Get the amiId by name

Set the amiId by name

Select a data set by exact match

Select a data set by partial match

Ooooh that shell, Can't you shell that shell

Staying up late to work on a fun project. Building a custom PXE boot image that will be used to pre-load our servers and get them ready for Puppet management. It has been about eight years since I had built a custom boot image. At that time busybox was my goto for small, powerful environments that needed to run from very small RAM mounted file systems.

BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

When most distros offer boot images that do little more than boot, or fail to boot; for my systems I used busybox to create a very compact environment that would allow you to perform diagnostics and fix any problems.

As great as busybox was at the time, I found it lacked a lot of compatibilities with even old POSIX systems and scripts. Arguments that you rely on for find, grep and even ls, just were not there. Things have changed; somewhat. Busybox has grown up and is full of new abilities.

Screenshot of "make menuconf" providing the ability
to choose the features to build into the binary.

The ancestry of a process

When it comes to viewing the heritage of a process, the go-to command is pstree. By itself, pstree will show the descendant processes. Using the -s argument it will show ancestor processes. A good trick to remember is slap, pstree -slap <PID>.

I personally like the format of ps and being the kind of person that likes to come up with one line tools, I created this little snippet.

{ TPID=$$; while [[ $TPID -ne 1 ]]; do ps -f $TPID | tail -n1; TPID=$(ps h -o ppid -p $TPID); done; ps -f $TPID|tac; } | tac

Saving it to use later we get this.

#!/bin/bash

{ 
        TPID=${1:-$$}; 
        while [[ $TPID -ne 1 ]]; do 
                ps -f $TPID | tail -n1; 
                TPID=$(ps h -o ppid -p $TPID); 
        done; 
        ps -f $TPID | tac; 
} | tac

This is what the two commands output.

BASH date picker

Sometimes I need users to pick a date from within a bash shell but I need the format to be correct. This little script lets users pick a date and it will output in whatever format the script needs.

Users can use the arrow keys to move the date on the calendar and press enter to select the date.

#!/bin/bash

DONE=0
DATE=$(date +%F)
YEAR="$(echo $DATE | cut -b1-4)"
MONTH="$(echo $DATE | cut -b6-7)"
DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
while [ $DONE -eq 0 ]; do
 clear
 cal -h ${MONTH} ${YEAR} | GREP_COLOR='47;30' grep "$DAY" -wC6 --color=always
 read -rsn1 KEY
 [[ $KEY == $'\x1b' ]] && { 
  read -rsn1 -t 0.1 DC
  read -rsn1 -t 0.1 KP
  case $KP in
   "A") DATE=$(date +%F -d "$DATE 7 day ago");;
   "B") DATE=$(date +%F -d "$DATE 7 day");;
   "C") DATE=$(date +%F -d "$DATE 1 day");;
   "D") DATE=$(date +%F -d "$DATE 1 day ago");;
  esac
 } || { 
  DONE=1
 }
 
 YEAR="$(echo $DATE | cut -b1-4)"
 MONTH="$(echo $DATE | cut -b6-7)"
 DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
done
echo $DATE

# To change the output format use the date command like these examples do
# date +%d/%m/%Y -d "$DATE"
# date -d "$DATE"

Are you mocking me up?

There are a lot of mockup / script test sites, one that I like and try to make use of is JSFiddle. It allows you to test snippets of code. You cat enter HTML, CSS & JavaScript. It gives you the option to include and dozens of JavaScript libraries including AngularJS, jQuery and many others.

See the screenshot below the code snippets.


Paset this into the HTML box

<canvas id="myCanvas" width="578" height="200"></canvas>

Paste this into the JavaScript box, select jQuery 3.3.1 then click run

$(function() {
  var c = document.getElementById("myCanvas");
  var ctx = c.getContext("2d");

  function DrawBox(x, y, w, h) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "black";
    ctx.moveTo(x, y);
    ctx.lineTo(x + w, y);
    ctx.lineTo(x + w, y + h);
    ctx.lineTo(x, y + h);
    ctx.lineTo(x, y);
    ctx.stroke();
  }

  function DrawLine(x1, y1, x2, y2) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "red";
    ctx.moveTo(x1, y1);
    ctx.lineTo(x2, y2);
    ctx.stroke();
  }

  DrawBox(10, 10, 40, 20);
  DrawBox(70, 30, 40, 20);
  DrawLine(50, 20, 70, 40);
  DrawBox(70, 70, 40, 20);
  DrawLine(30, 30, 90, 70);

});

wget script hacking the Cisco DPC3825

My ISP at home has some of the best prices on high speed Internet around. They can do this because they only use the most budget of all the junky equipment small amounts of money can buy. Ever since I first subscribed my home router (CATV modem) will progressively get slower and slow with decreasing Wi-Fi range until I go through the steps to login to the web interface and click "Reset" or pull the power and plug it back in. After that it is good for a few days. The problem with that is it dies at the worst time. I have complained for many months but the only thing that happens is they replace the junk modem with another junk modem.


The best way to solve this is to create a wget script that cron will run around 4:00 AM when no one is using the Internet. The script will reboot the modem and have it ready for me to use problem free all day.

wget is a very powerful command line web client. Reading the source file of the page you want to script you can automate almost any action.

#!/bin/bash

wget \
  --save-cookies=cookies \
  --post-data="username_login=cusadmin&password_login=yourpassword" \
  http://192.168.0.1/goform/Docsis_system -O /dev/null 2>&1

wget \
  --load-cookies=cookies \
  --post-data="devicerestart=1" \
  http://192.168.0.1/goform/Devicerestart -O /dev/null 2>&1

CLI Tools: screen

One of my favourite CLI tools is screen. Basically it lets you detach from your shell session and it keeps running. You can reattach to it later from anywhere, start your shell from a local TTY then reattach via SSH. You can even share your session and have others join in so you can collaborate on shell.

The sample code for today is how to make an init script to run your program as a detached daemon using screen. The cool thing about this is it lets you connect to it at any time to see what it is doing. This is not recommend for normal use but it is a great way to debug that new program before you commit to making it a proper daemon.

#!/bin/bash

cd /etc/init.d

start() {
 echo "Starting Service: "
 APP_CMD="/usr/local/bin/MyProgram.pl"
 screen -dmS pmsgd $APP_CMD
 echo
}

stop() {
 echo "Cannot stop screen programs this way. Open the screen sessions."
 echo
}

restart() {
 stop
 start
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  restart)
 restart
 ;;
  *)
 printf "Usage: %s {start|stop|restart}\n" "$0"
 exit 1
esac

exit 0

Find zen for your users with Zenity

"That is a great program you made for us but we don't like that command line thing. Can you make it a GUI?" To which you reply, "Why yes I can. Just give me a few minutes."

Shell scripts are great. You have all the power from thousands of Linux commands that you can connect together to make the most impressive scripts since the ENIAC. No matter how great your scripts are there will be users that just don't like it because of the command line.

You have three options:  create a web server interface,  re-write it as a binary with graphical objects or use pre-existing dialog tools to add GUI features to your program. Zenity can do that for you.

There are a few dialog tools like zenity, kdialog, xdialog and may more. They are not very hard to use and can provide a lot of helpful iteration with things like information boxes, input boxes, selections and radio buttons.

zenity --entry --text "Please enter your name"