Running shell commands from awk

There are a few ways to run shell commands from inside an awk command. One is system(...) but it is better for handing values off to a program to do something, not the best way to get some data back from the shell command.

The best way to get data back from a command is to define the command as a variable then execute it piping the output to a new variable using the built in getline function.

The input of this example is a log file were the first field is epoch time and we need to see the time in human readable time format.

tail /var/log/my.log | awk ' {
   DC="date -d@"$1; 
   DC | getline T; 
   printf "%s\t", T; 
   for(i=2;i<NF;i++) {
     printf $i"\t"
   }; 
   printf "\n";
}'

In this example DC becomes the date command that is given the epoch value in variable $1.  T is the variable for the time as a string that we display using printf. Then a loop prints the rest of the data from each line of the log from the second filed to the NF, end number of fields.

Unit convertion wih awk

I needed a simple unit converter to change number to kilo, mega and giga to I went looking and found some large complex code and decided I could make it so must smaller.

This will convert your number to a human readable value.

Try this with it.
for i in $(seq 1 1 64); do echo -n "$(echo "2^$i"|bc) =  "; echo "2^$i"|bc|awk -f convert.awk; done

{
 S=$1;
 Us=" kMGTPEZY";
 U="";
 i=2;
 while(S>1024){
  S/=1024;
  U=substr(Us,i,1);
  i++;  
 };
 printf "%0.0f%sb\n", S, U
}

Extracting named columns of data with awk

Some times when a vendor updates a diagnostic tool they change the order of the fields so it is not a good idea to extract columns of data by index. This simple awk script is able to get data by the name of the column rather then the index.

The samples contain the same data but the columns are in different orders. The results from both files should be the same.

samp0.txt

id;width;rank;age;height
1;900;4;20;500
2;1900;11;32;200
3;70;8;43;50

samp1.txt

id;age;height;width;rank
1;20;500;900;4
2;32;200;1900;11
3;43;50;70;8

GetField.awk

BEGIN { 
 FS=";"
 C=0
} {  
 if((C > 0) && ($C!="")) {
  print $C
 }
 
 if(C==0) {
  for(i=1; i<=20; i++) {
   if($i == ARGV[2]) {
    C=i
   }
  }
 }
}

Run the commands

]$ cat samp0.txt | awk -f GetField.awk - age 2>/dev/null
20
32
43
]$ cat samp1.txt | awk -f GetField.awk - age 2>/dev/null
20
32
43

The Power of AWK: svn log to RPM change history.

For many years I used awk and gawk to just do simple column parsing because it is so good at stripping out the white space. Recently I have begun to use awk to do more complex task. One of these is reading the subversion log and converting it to RPM change log history.

I am using this awk as part of my build system to automatically generate rpm.spec files as one step of my build system for my many projects.
The goal is to turn this

------------------------------------------------------------------------
r4 | royce | 2015-01-21 13:03:49 -0700 (Wed, 21 Jan 2015) | 1 line

New SVN project
------------------------------------------------------------------------
r3 | royce | 2015-01-21 10:13:03 -0700 (Wed, 21 Jan 2015) | 1 line


------------------------------------------------------------------------
r2 | royce | 2015-01-21 10:12:25 -0700 (Wed, 21 Jan 2015) | 1 line


------------------------------------------------------------------------
r1 | royce | 2015-01-21 10:02:31 -0700 (Wed, 21 Jan 2015) | 1 line

First build
------------------------------------------------------------------------

In to this

* Wed Jan 21 2015 Revision r4
- First build
- New SVN project

Below is the awk code.
Save it as the file SvnLog2SpecChangeLog.awk.
Then from your project directory run it as svn log | awk -f SvnLog2SpecChangeLog.awk

BEGIN {
   DI=1;
   LASTDATE="";
}

{
   # print "working on "$0
   if($1 == "------------------------------------------------------------------------") {
      nextrow=0
      # print "split row"
   } else {
      # print "text row"
      if(nextrow > 0) {
         if ($1 == "") {
            # print "Blank row"
            nextrow++;
         } else {
            DOCS[DI++]=$0;
         }
      }
      if(nextrow==0) {
         # print "Revison row also contains date"
         NEWREV=$1;
         NEWDATE=$5;
         nextrow++;
         if(LASTDATE != "") {
            # print "LASTDATE="LASTDATE;
            if((LASTDATE!=NEWDATE) && (DI > 1)) {
               DATECMD="date +\"%a %b %d %Y\" -d"LASTDATE
               DATECMD | getline DATEOUT;               
               printf("* %s Revision %s\n", DATEOUT, LASTREV);
               for (DK in DOCS) {
                  printf("- %s\n", DOCS[DK]);
               }
               printf "\n";
               delete DOCS;
               DI=1
               LASTDATE=NEWDATE;
               LASTREV=NEWREV;
            }
         } else {
            LASTDATE=NEWDATE;
            LASTREV=NEWREV;
         }
      }
   }
}

END {
   DATECMD="date +\"%a %b %d %Y\" -d"LASTDATE
   DATECMD | getline DATEOUT;               
   printf("* %s Revision %s\n", DATEOUT, LASTREV);
   for (DK in DOCS) {
      printf("- %s\n", DOCS[DK]);
   }
   printf "\n";
}