Parsing a moving target.

A problem was presented to me where the raw data was expected to have columns added and moved around at some point but a report was needed that could deal with this. I chose to use Perl Named Capture Containers to solve this problem.

#!/usr/bin/perl -w

# This is an example of using named capture containers to parse data from lines 
# when the columns of data could move around, new columns are added and even if 
# the column is removed.

# Matching a pattern with multiple parts cannot deal with columns that move or are missing.
# Each part must be matched on it's own line.
# Every time a match is made it is put into the hash for use later.

# sample3.data as input
#######################
# 01:00:00 httpd=on sshd=on crond=on ntpd=on winbind=off cups=off
# 01:01:00 [567] httpd=on sshd=on crond=on ntpd=on winbind=off cups=off
# 01:02:00 [567] crond=on ntpd=on winbind=off httpd=off cups=off sshd=on
# 01:03:00 [PID:567] ntpd=on winbind=off cups=off sshd=on named=on httpd=on crond=on 
# 01:04:00 [PID:567] named=on httpd=on crond=on

# results from output
#######################
# Time  httpd ntpd sshd 
# 01:00:00  on  on  on 
# 01:01:00  on  on  on 
# 01:02:00  off  on 
# 01:03:00  on  on  on 
# 01:04:00  on 


my %DataSet; 
# The data set is dynamically gathered so to change
# the report just add or remove column names here.
my @ReportFields = ( "httpd", "ntpd", "sshd" );

sub Pack {
 my $Time = shift;
 my $FieldName = shift;
 my $DataValue = shift;
 $DataSet{$Time}->{$FieldName} = $DataValue;
}

while (<>) {
 $Line = $_;
 $Line =~ s/\n//;
 $Line =~ m/^(?<time>\d\d:\d\d:\d\d).*/;
 my $NewTime = $+{time};

 foreach my $R ( sort @ReportFields) {
  if ($Line =~ m/.* \Q$R\E=(?<value>\w*) .*/) { Pack($NewTime, $R, $+{value}) }; 
 }
}

printf "Time\t\t";
foreach my $F ( sort @ReportFields ) {
 printf $F."\t";
}
printf "\n";

foreach my $Time (sort keys %DataSet) {
 printf $Time."\t";
 foreach my $F ( sort @ReportFields ) {
  printf " ".$DataSet{$Time}->{$F}."\t" if (defined $DataSet{$Time}->{$F});
 }
 printf "\n";
}

Stop eating my pipe!

We love to use while loops in our scripts and they are a great way to read a file one line at a time to get a job done.

For example:

cat myFile.txt | while read LINE; do
   echo $LINE
   sleep 1
done

Now comes along SSH and lets say that your file contains a list hostnames you want to get uptime from:

cat myHostsFile.txt | while read LINE; do
   echo -n "Host = $LINE "
   ssh $LINE "updtime"
done

How disappointed you are when your loops stops after the first host. This is because every child process inherits it's first three file descriptors from it's parent, so SSH takes everything from STDIN for itself.

Sometimes you may want that but this time you don't. What can you do?

The simple solution here is to disassociate SSH from STDIN, and you do this using a simple re-director.

cat myHostsFile.txt | while read LINE; do
   echo -n "Host = $LINE "
   ssh 0>/dev/zero $LINE "updtime"
done

Something to keep in mind is that STDIN supplies a data stream, it does not take it. So for this reason we attache to /dev/zero not /dev/null.

LibreOffice BASIC Easy GetCell

Improving on GetCell from an earlier post I have added a Column2Index function to allow you to pass the names of the column and row rather then the index.

If the column is a number then it will use the values as index values, otherwise it will convert both the column and the row.

These two calls are the same
GetCell("Sheet1", "D", 3) <- using the names of the columns and rows
GetCell("Sheet1", 3, 2) <- using the index of the columns and rows


For cell that contain numbers, ether fixed or formula generated you can simply read from them or write to them like so.

If GetCell("Sheet1", "D", 3).Value > 100 Then
   GetCell("Sheet1", "D", 3).Value = 0
End If

For cells that contain text you use GetCell("Sheet1", "D", 3).String the same way you would any normal string variable.

Function Column2Index(ColNameX As String)
 ReturnInt = 0
 ColNameU = Ucase(ColNameX)
 While ColNameU <> ""
  LastChar = Left(ColNameU, 1)
  ColNameU = Right(ColNameU, Len(ColNameU) - 1)
  ReturnInt = ReturnInt * 26
  ReturnInt = ReturnInt + Asc(LastChar) - 64
 Wend
 Column2Index = ReturnInt - 1
End Function

' Now accepts column and row as they are named or their index value
' These two calls are the same
' GetCell("Sheet1", "D", 3) <- using the names of the columns and rows
' GetCell("Sheet1", 3, 2) <- using the index of the columns and rows


Function GetCell(SheetName As String, Column, Row) As com.sun.star.table.XCell
 ColumnNumber = 0
 RowNumber = 0
 If ISNUMERIC(Column) Then  
  ColumnNumber = Column
  RowNumber = Row
 Else
  ColumnNumber = Column2Index(Column)
  RowNumber = Row - 1 
 End If
 AllSheets = ThisComponent.Sheets()
 FindSheet = AllSheets.GetByName(SheetName)
 TheCell = FindSheet.GetCellByPosition(ColumnNumber,RowNumber)
 GetCell = TheCell
End Function

Simple Process Locking In Perl

My co-work Rick asked for a simple process lock control in Perl that would work over a server load balanced pool with a common NFS share. The easiest way is a loop that tries to create a directory, if the command fails then it waits and tries again, if the command succeeds then it has the lock and can do it's job. When done it removes the directory and another process gets to continue.

We started to build in an extra step so that if the script dies during the "meat and potatoes" section anther process can check if the PID from the previous process is still running and if not remove it. We did not get that far, I will let you figure the rest out.

The first usleep command is just to prevent process hammering, you should set it to match your expected time for a process to run. The second usleep is to simulate process congestion and is only for testing, you should remove it from your project.

#!/usr/bin/perl -w

use Time::HiRes qw(usleep nanosleep);

my $COUNTER=0;
my $dir = "/mnt/share/templock.dir";

while (! mkdir($dir) )
{
 $COUNTER++; 
 usleep(100);

 printf ("$$ waiting: $COUNTER\n");
 if ($COUNTER > 1000)
 {
  #open(my $fh, "<", $dir/PID");
  #my $row = <$fh>;
  
  die("squak\n");
 }
}

open(my $fh, ">", "$dir/PID");
printf $fh "$$\n";
close($fh);

#meat and potatoes
usleep(10000);


unlink("$dir/PID");
rmdir($dir);

Some tail or some head

Often times I find that head and tail don't put out enough. Some times I just want more. I don't want to create aliases for them because there are times that I just need it to work normally and I am too lazy to absolute path to remove the alias.

This is how I am able to me some tail and some head with more.

In my home I have a bin directory that is just for my bin scripts, but you could put this in /usr/local/bin/ if you like.

I wanted one script that does both, so I use basename and the built in BASH test replacement operation to strip the command down to either head or tail. I created the script as somehead and then made a symlink that points to it called sometail.

$(basename ${0//some/}) -n $(( $(tput lines) - 3 ))

Notice this uses tput to get the number of lines in the terminal and then subtracts three, just so it does not overflow the screen and you get an idea of where the output started.

Remember to

chmod 755 somehead
ln -s somehead sometail

Running shell commands from awk

There are a few ways to run shell commands from inside an awk command. One is system(...) but it is better for handing values off to a program to do something, not the best way to get some data back from the shell command.

The best way to get data back from a command is to define the command as a variable then execute it piping the output to a new variable using the built in getline function.

The input of this example is a log file were the first field is epoch time and we need to see the time in human readable time format.

tail /var/log/my.log | awk ' {
   DC="date -d@"$1; 
   DC | getline T; 
   printf "%s\t", T; 
   for(i=2;i<NF;i++) {
     printf $i"\t"
   }; 
   printf "\n";
}'

In this example DC becomes the date command that is given the epoch value in variable $1.  T is the variable for the time as a string that we display using printf. Then a loop prints the rest of the data from each line of the log from the second filed to the NF, end number of fields.

Truncate a file with sed

Log files grow over time but you don't want to fill your disk. Often a log file is just for simple debug and does not need to be rotated or kept in /var/log/, sometimes a simple debug log just goes well in /dev/shm/. In this case it is important to keep it short and trim off the top of the file from time to time.

I don't know why but this problem seems very difficult for a lot of people and they end up writing long complex multi line blocks of code to truncate a file and just keep the end of it. It is super simple with sed.

# only keep the last 100 lines of the log file
sed -i /dev/shm/my_debug.log -e :a -e '$q;N;100,$D;ba'