Sed: Append file to end of line

Sed can let you replace part of a line with new content from another file. This is how I do this in three steps. You can see the code run a codingground

By using a file to insert the replacement text the content can contain special characters that would normally require a lot of extra work to escape. 

#!/bin/bash

# Replace the right portion of a line with the content of a file.
# 1) Replace the right portion of a line with a marker word
# 2) Append after the marker with the content from a file
# 3) Remove the marker and new line to pull the next line up.

cat > outfile <<!EOF
hello there tom
        How are you today?
    Where is the dog show?
!EOF

cat outfile

cat > testfile <<!EOF
flowers doing?
!EOF


sed -i outfile -e 's/you.*/REPLACE_MARKER/g'
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/r testfile"
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/{:a;N;s/REPLACE_MARKER\n//}"
printf "\n\n\n"
cat outfile

Ultra minimal docker Node.JS example

This is how to get a minimal docker image with just node and run a simple service.
Docker will automatically download the image for alpine-node if you do not already have it.
Use the run command to create and start the new container.
Use -p to publish the container port 3000 on the host at port 3000 on the IP address of the host.
Use -it to connect interactively to the TTY so you can enter and run commands

docker run -p 3000:3000 -it mhart/alpine-node

Your command line is now inside the docker container.
Edit the code of the service.
Copy and paste the sample code below and save the file.

vi service.js

Run node on the service.

node service.js

Open your web browser to the IP address of your host system using :3000 to set the request port.
You should see the hello message.
Use Ctrl+c to exit node & Ctrl+d to exit the container.

You now have a functional container that is set up to run this simple script.
Use the ls --all command to see the container you built.

docker container ls --all

Use the start command to start the container up again.
Use the container ID you found from the ls command.

docker container start <CONTAINER ID>

Check that the container is running and publishing port 3000.

docker container ls

Use the attach command to re-attach to the container.

docker container attach

Run the node command again.

node service.js

var http = require('http');

var server = http.createServer( function(req, res) {
 if (req.url === '/') {
  res.write("Hello World!");
  res.write("I have been expecting you.");
  res.end();
 }

 if (req.url === '/snarky') {
  res.write("If you code it, it will crash!");
  res.end(); 
 }

});

// Port 3000 on IPv4
server.listen(3000, "0.0.0.0");

BASH date picker

Sometimes I need users to pick a date from within a bash shell but I need the format to be correct. This little script lets users pick a date and it will output in whatever format the script needs.

Users can use the arrow keys to move the date on the calendar and press enter to select the date.

#!/bin/bash

DONE=0
DATE=$(date +%F)
YEAR="$(echo $DATE | cut -b1-4)"
MONTH="$(echo $DATE | cut -b6-7)"
DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
while [ $DONE -eq 0 ]; do
 clear
 cal -h ${MONTH} ${YEAR} | GREP_COLOR='47;30' grep "$DAY" -wC6 --color=always
 read -rsn1 KEY
 [[ $KEY == $'\x1b' ]] && { 
  read -rsn1 -t 0.1 DC
  read -rsn1 -t 0.1 KP
  case $KP in
   "A") DATE=$(date +%F -d "$DATE 7 day ago");;
   "B") DATE=$(date +%F -d "$DATE 7 day");;
   "C") DATE=$(date +%F -d "$DATE 1 day");;
   "D") DATE=$(date +%F -d "$DATE 1 day ago");;
  esac
 } || { 
  DONE=1
 }
 
 YEAR="$(echo $DATE | cut -b1-4)"
 MONTH="$(echo $DATE | cut -b6-7)"
 DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
done
echo $DATE

# To change the output format use the date command like these examples do
# date +%d/%m/%Y -d "$DATE"
# date -d "$DATE"

Inserting ID with case

I use sqlite3 for a lot of small projects. It has many features that are very impressive for such a small DB engine.

One thing I like to do is set the value of an index field using MAX(id)+1 from the table itself but that does not work on a blank table. To make this work with a blank table a CASE statement can be used.

CREATE TABLE microServices (
   id INT NOT NULL, 
   packageName CHAR(45) NULL, 
   serviceName CHAR(45) NULL, 
   deployTargets CHAR(45) NULL, 
PRIMARY KEY (id));

INSERT INTO microServices (
   id, 
   packageName, 
   serviceName) 
   SELECT CASE 
      WHEN MAX(id) > 0 
         THEN MAX(id)+1 
      ELSE 1 
   END, 
   'trucks', 
   'trucksd' 
FROM microServices;

Are you mocking me up?

There are a lot of mockup / script test sites, one that I like and try to make use of is JSFiddle. It allows you to test snippets of code. You cat enter HTML, CSS & JavaScript. It gives you the option to include and dozens of JavaScript libraries including AngularJS, jQuery and many others.

See the screenshot below the code snippets.


Paset this into the HTML box

<canvas id="myCanvas" width="578" height="200"></canvas>

Paste this into the JavaScript box, select jQuery 3.3.1 then click run

$(function() {
  var c = document.getElementById("myCanvas");
  var ctx = c.getContext("2d");

  function DrawBox(x, y, w, h) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "black";
    ctx.moveTo(x, y);
    ctx.lineTo(x + w, y);
    ctx.lineTo(x + w, y + h);
    ctx.lineTo(x, y + h);
    ctx.lineTo(x, y);
    ctx.stroke();
  }

  function DrawLine(x1, y1, x2, y2) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "red";
    ctx.moveTo(x1, y1);
    ctx.lineTo(x2, y2);
    ctx.stroke();
  }

  DrawBox(10, 10, 40, 20);
  DrawBox(70, 30, 40, 20);
  DrawLine(50, 20, 70, 40);
  DrawBox(70, 70, 40, 20);
  DrawLine(30, 30, 90, 70);

});

3.141592653589793238462643383279

Happy Pi day everyone. March 14th is Pi day because 3.14.

I created a BASH script to test the calculation in this video. It calculates Pi without pre-known values but by adding and subtracting fractions of one over odd numbers. 1/1 - 1/3 + 1/5 - 1/7 + 1/9 - 1/11 + 1/13 ... It gets more accurate the longer you let it run but only for a short time as fractional math in BASH is limited.

#!/bin/bash

D=1
P=0
i=1
while true; do
 O="-"
 (( i++ % 2 )) && O="+"
 P=$(echo "scale=66; ${P} ${O} 1/${D}" | bc)
 printf "%0.20f               \r" $(echo "scale=21; ${P} * 4" | bc)
 ((D+=2))
done

Tunneling UDP

We use Graylog at work, a very nice tool that collects logs from all our systems and helps us analyze them. We ran into a problem with a special cluster of servers that are isolated and could not get to our Graylog server. We only needed to connect them to Graylog for a short time so this is not a long term solution.

These systems have never been able to connect to Graylog so they were configured to send all logs to 127.0.0.1, and that is great because we can setup a listening tunnel on localhost to forward the connection and we can do all this without making changes to the target systems.

The Graylog server has SSH access to the target systems so we use that to create a reverse tunnel. SSH works over TCP but Graylog uses UDP so we use NC (netcat) to convert the UDP to TCP for tunneling and then convert it back again on the server side.

The first command forks a local listener to send TDP packets to our Graylog on UDP.
The second command opens a reverse tunnel and starts listening on UDP to send packets back via the tunnel over TCP.

nc -l 127.0.0.1 12203 | nc -u 127.0.0.1 12201 &
ssh target2 -R 12201:localhost:12203 "nc -kluw 0 127.0.0.1 12201|nc 127.0.0.1 12201"

Some important points.

• Each connection needs its own port, Graylog is using 12201, so target1 would listen on 12202 and target2 is using 12203
• Netcat wants to disconnect when the logger on target finishes a message to that nc uses 'k' to keep the connection alive and is uses 'w 0' to have no wait time out.

Update

After a day or so running netcat likes to die so this little change will automatically relaunch nc instantly by running this on the target side.

while nc -kluw 0 127.0.0.1 12201; do true; done | while cat - | nc 127.0.0.1 12201; do true; done