Running shell commands from awk

There are a few ways to run shell commands from inside an awk command. One is system(...) but it is better for handing values off to a program to do something, not the best way to get some data back from the shell command.

The best way to get data back from a command is to define the command as a variable then execute it piping the output to a new variable using the built in getline function.

The input of this example is a log file were the first field is epoch time and we need to see the time in human readable time format.

tail /var/log/my.log | awk ' {
   DC="date -d@"$1; 
   DC | getline T; 
   printf "%s\t", T; 
   for(i=2;i<NF;i++) {
     printf $i"\t"
   }; 
   printf "\n";
}'

In this example DC becomes the date command that is given the epoch value in variable $1.  T is the variable for the time as a string that we display using printf. Then a loop prints the rest of the data from each line of the log from the second filed to the NF, end number of fields.

Truncate a file with sed

Log files grow over time but you don't want to fill your disk. Often a log file is just for simple debug and does not need to be rotated or kept in /var/log/, sometimes a simple debug log just goes well in /dev/shm/. In this case it is important to keep it short and trim off the top of the file from time to time.

I don't know why but this problem seems very difficult for a lot of people and they end up writing long complex multi line blocks of code to truncate a file and just keep the end of it. It is super simple with sed.

# only keep the last 100 lines of the log file
sed -i /dev/shm/my_debug.log -e :a -e '$q;N;100,$D;ba'