Running shell commands from awk

There are a few ways to run shell commands from inside an awk command. One is system(...) but it is better for handing values off to a program to do something, not the best way to get some data back from the shell command.

The best way to get data back from a command is to define the command as a variable then execute it piping the output to a new variable using the built in getline function.

The input of this example is a log file were the first field is epoch time and we need to see the time in human readable time format.

tail /var/log/my.log | awk ' {
   DC="date -d@"$1; 
   DC | getline T; 
   printf "%s\t", T; 
   for(i=2;i<NF;i++) {
     printf $i"\t"
   }; 
   printf "\n";
}'

In this example DC becomes the date command that is given the epoch value in variable $1.  T is the variable for the time as a string that we display using printf. Then a loop prints the rest of the data from each line of the log from the second filed to the NF, end number of fields.

Truncate a file with sed

Log files grow over time but you don't want to fill your disk. Often a log file is just for simple debug and does not need to be rotated or kept in /var/log/, sometimes a simple debug log just goes well in /dev/shm/. In this case it is important to keep it short and trim off the top of the file from time to time.

I don't know why but this problem seems very difficult for a lot of people and they end up writing long complex multi line blocks of code to truncate a file and just keep the end of it. It is super simple with sed.

# only keep the last 100 lines of the log file
sed -i /dev/shm/my_debug.log -e :a -e '$q;N;100,$D;ba'

Alpha sequence

When doing a quick little shell script the seq command is very handy to generate a list of numbers but all too often I find that I need a sequence of letters and there is no alphabetical version of seq. There are hundreds of examples on the Internet of how to do this in almost every script language you can think of, this is my version in bash.

This version has the added feature of being able to run up a sequence of upper or lower case letter. The script only accepts one argument of the last letter, but could easily be expanded to do all sorts of extra tricks.

#!/bin/bash

OUT=""
TARGET=$(echo ${1:0:1} | grep -i "[a-z]")
UPPER=$(echo $TARGET | tr "[a-z]" "[A-Z]")
START=97 # a
[ "$TARGET" = "$UPPER" ] && {
 START=65 # A
}
while [ "$OUT" != "$1" ]; do
 OUT=$(awk 'BEGIN{printf "%c",'${START}'}')
 echo -n "$OUT "
 let "START=$START+1"
done

Examples:

$ alphaseq J
A B C D E F G H I J

$ alphaseq m
a b c d e f g h i j k l m

wget script hacking the Cisco DPC3825

My ISP at home has some of the best prices on high speed Internet around. They can do this because they only use the most budget of all the junky equipment small amounts of money can buy. Ever since I first subscribed my home router (CATV modem) will progressively get slower and slow with decreasing Wi-Fi range until I go through the steps to login to the web interface and click "Reset" or pull the power and plug it back in. After that it is good for a few days. The problem with that is it dies at the worst time. I have complained for many months but the only thing that happens is they replace the junk modem with another junk modem.


The best way to solve this is to create a wget script that cron will run around 4:00 AM when no one is using the Internet. The script will reboot the modem and have it ready for me to use problem free all day.

wget is a very powerful command line web client. Reading the source file of the page you want to script you can automate almost any action.

#!/bin/bash

wget \
  --save-cookies=cookies \
  --post-data="username_login=cusadmin&password_login=yourpassword" \
  http://192.168.0.1/goform/Docsis_system -O /dev/null 2>&1

wget \
  --load-cookies=cookies \
  --post-data="devicerestart=1" \
  http://192.168.0.1/goform/Devicerestart -O /dev/null 2>&1

svn log to RPM change history, the Ruby way

This is the Ruby version of the AWK script to convert SVN logs to a change history report suitable for use in an RPM spec file.

I tried to keep the code very similar to the AWK version but there are many ways this code could be optimized to reduce the number of lines.

#!/usr/bin/ruby

require 'date'

nextRow = 0
lastDate = ""
lastRev = ""
newDate = ""
newRev = ""
docs = Array.new

$stdin.each_line do |l|
 if l.include?("-"*72) 
  nextRow = 0
 else
   if nextRow > 0
    if l.chomp.length == 0
     nextRow += 1
    else
     docs.push(l)
    end
   else
    $Sl = l.split
    newRev = $Sl[0]
    newDate = $Sl[4]
    nextRow += 1
    if lastDate != ""
     if lastDate != newDate and docs.length > 0
      printf "* %s Revision %s\n", Date.parse(lastDate).strftime("%a %b %d %Y"), lastRev
      docs.each { |dl|
       puts "- #{dl}"
      }
      puts
      docs.clear
     end
    end
    lastDate = newDate
    lastRev = newRev
   end
 end
end

if docs.length > 0
 printf "* %s Revision %s\n", Date.parse(lastDate).strftime("%a %b %d %Y"), lastRev
 docs.each { |dl|
  puts "- #{dl}"
 }
end

Super Simple Linked List In C

When writing code in a high level scripting language like Perl, Ruby or Python it is easy to take for granted what a nice feature it is to have automatic types like arrays or hashs. Variable that you can just dump data in to and pull it back out any way you like. These features were not always there, someone had to write the code to make it to this.

A linked list in C is a way to store values, as many as you like for as much memory as you have. Memory for a variable is created on the fly as needed and stored in a location that knows information about it's neighbour.  A good linked list will have features to let you search, insert, delete from any point, move backwards and forwards in the list and most importantly a good list would protect your code from accidental memory bugs. My example today has non of those features.

This is the bare minimum of a simple linked list. It is a single linked list so you can only move forward. A double linked list lets you move backwards also.

#include<stdio.h>
#include<stdlib.h>

struct DataNode {
 void *Load;
 struct DataNode *Next;
};

struct DataNode *TheFirst = NULL;
struct DataNode *Current = NULL;
struct DataNode *Last = NULL;

void* NewList(void *Load) {
 struct DataNode *Pointer = (struct DataNode*)malloc(sizeof(struct DataNode));
 if(NULL == Pointer) {
  return NULL;
 }
 Pointer->Load = Load;
 Pointer->Next = NULL;

 TheFirst = Last = Current = Pointer;
 return(Pointer);
}

void* Add(void *Load) {
 if(TheFirst == NULL) {
  return (NewList(Load));
 }

 struct DataNode *Pointer = (struct DataNode*)malloc(sizeof(struct DataNode));
 Pointer->Load = Load;
 Pointer->Next = NULL;

 Last->Next = Pointer;
 Current = Last = Pointer;
 return(Pointer);
}

void* First() {
 Current = TheFirst;
 if(Current == NULL) return(NULL);
 return(Current->Load);
}

void* GetNext() {
 struct DataNode *Pointer = Current;

 // Advance to next
 if(Pointer->Next) {
  Current = Pointer->Next;
  return(Current->Load); 
 }
 return(NULL);
}

int DeleteFirst() {
 int Return = 0;
 if(TheFirst) {
  if(TheFirst->Next) {
   Current = TheFirst->Next;
   Return = 1;
  } else {
   Current = NULL;
  }
 }
 free(TheFirst);
 TheFirst = Current;
 return(Return);
}

int main(void) {
 char* ToDoList;
 Add("Pet the dog");
 Add("Feed the fish");
 Add("Let the cat out");
 Add("Do the dishes");
 Add("Gas up the car");
 Add("Watch some TV");
 ToDoList = First();
 while(ToDoList != NULL) {
  printf("%s\n", ToDoList);
  ToDoList = GetNext();
 }
 while(DeleteFirst()) {}
 return 0;
}

Instant hash in Perl, just do the splits

Today I needed a Perl script that could create a lot of hash cells fast and the names of the cells will not be know until run time. I did not want to use a for loop, I looked for a better way to do this in less code. When I found the solution I was reminded of some Perl documentation that I read saying that most people think the @ symbol means an array but it does not, it means many. The way this works it when you reference the hash with the @ symbol you can put many values in to many cells in one line of code.

The first split turns the Fields string in to the cell name identifiers, the second split turns the Values string in to the data, putting many data in to many cells named in a hash.

#!/usr/bin/perl

my $Fields = "name:address:phone";
my $Values = "Bob:123 Hill Rd:505-050-4321";
my %Hash;
@Hash{split(/:/, $Fields)} = split(/:/, $Values);

for my $key (keys %Hash) {
 print "$key = '$Hash{$key}'\n";
}