Expandable Disk Server

In my SOHO I have tried many types of storage solutions. Currently the system is a Qnap, four bay system. It's turn key, user friendly approach does make it easy to setup and use, but it's highly limited with not having a fifth drive bay. Four bays allows for two disks in RAID-1 or one big disk in RAID-10, not having a fifth bay prevents upgrading to larger disks, without fronting the cash to buy an entire new system and disks all at once.

It's time for a storage solution that can grow over time, not only to replace smaller disk but to allow for the system to start with just one disk and add to the cluster over the next few month.

Looking at using FreeNAS to manage a NAS server built from a Linux box. This will not be a dedicated box, so I will be building a Proxmox and hosting the FreeNAS as a VM inside the Proxmox. The FreeNAS needs to grow as new disks are added so it will be using the disks as an NFS share from Proxmox.

The plan is to use Wester Digital 12TB disks, as these disks are almost $700 each, only one disk will be added to the system every two weeks over the next two months.

Building the disk array will use Linux software RAID-1, LVM, and XFS.

RAID-1
This level of RAID is for redundancy, all disks in the RAID are kept identical. When one disk fails the file system continues to work.

LVM
LVM visualises the block layer of block devices. This is a powerful tool for manipulating block devices on a running system. LVM does have builtin support for mirroring and RAID, including RAID-1 and RAID-10, but happens at the logical volume layer and using these would prevent the migration to larger disks.

XFS
This file system allows for growing live without the need to format or reboot the system.

To test this plan, and demonstrate how this will work, VMware running Fedora 27 was used. The images used a 8GB SCSI for the root file system. The disks in the cluster are SATA and are marked a through f.

In this example the commands used create Physical Volumes with names that match the RAID-1 device, while the images show the Physical Volumes with PV names.

Step 1 Create a new RAID-1
mdadm --create /dev/md0 --run --level=1 --raid-devices=2 /dev/sda missing

Step 2 Create a new physical volume
pvcreate /dev/md0

Step 3 Create a new volume group
vgcreate vg0 /dev/md0

Step 4 Create a new logical volume
lvcreate -l 100%VG -n lv0 vg0

Step 5 Create the XFS
mkfs.xfs /dev/vg0/lv0
mkdir /xfs0
mount /dev/vg0/lv0 /xfs0/

Step 6 Add another disk to the RAID
mdadm --add /dev/md0 /dev/sdb

Step 7 Create a new RAID-1
mdadm --create /dev/md1 --run --level=1 --raid-devices=2 /dev/sdc missing

Step 8 Create a new physical volume
pvcreate /dev/md1

Step 9 Grow the existing volume group
vgextend vg0 /dev/md1

Step 10 Grow the existing logical volume
lvextend -l 100%VG vg0/lv0

Step 11 Grow the existing XFS live
xfs_growfs /xfs0/

Add the last disk the same way as Step 6.



Expand the disk cluster by adding new PVs and migrating

Step 12 Create the new RAID and PV
mdadm --create /dev/md2 --run --level=1 --raid-devices=2 /dev/sd/ missing
pvcreate /dev/md2

Step 13 Extend the volume group to include the new PV
vgextend vg0 /dev/md2

Step 14 Move all blocks from the old PV to the new PV.
pvmove /dev/md0 /dev/md2

Step 15 Remove the first PV.
vgreduce vg0 /dev/md0

Step 16 Grow the logial volume again
lvextend -l 100%VG vg0/lv0

Step 17 Re-grow the existing XFS live
xfs_growfs /xfs0/

Now on GitLab.com

You can now download all the code from GitLab.com

git clone git@gitlab.com:SiliconTao-open-source/linux-code-snippets.git

Host your projects on GitLab.com

Sed: Append file to end of line

Sed can let you replace part of a line with new content from another file. This is how I do this in three steps. You can see the code run a codingground

By using a file to insert the replacement text the content can contain special characters that would normally require a lot of extra work to escape. 

#!/bin/bash

# Replace the right portion of a line with the content of a file.
# 1) Replace the right portion of a line with a marker word
# 2) Append after the marker with the content from a file
# 3) Remove the marker and new line to pull the next line up.

cat > outfile <<!EOF
hello there tom
        How are you today?
    Where is the dog show?
!EOF

cat outfile

cat > testfile <<!EOF
flowers doing?
!EOF


sed -i outfile -e 's/you.*/REPLACE_MARKER/g'
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/r testfile"
printf "\n\n\n"
cat outfile

sed -i outfile -e "/REPLACE_MARKER/{:a;N;s/REPLACE_MARKER\n//}"
printf "\n\n\n"
cat outfile

Ultra minimal docker Node.JS example

This is how to get a minimal docker image with just node and run a simple service.
Docker will automatically download the image for alpine-node if you do not already have it.
Use the run command to create and start the new container.
Use -p to publish the container port 3000 on the host at port 3000 on the IP address of the host.
Use -it to connect interactively to the TTY so you can enter and run commands

docker run -p 3000:3000 -it mhart/alpine-node

Your command line is now inside the docker container.
Edit the code of the service.
Copy and paste the sample code below and save the file.

vi service.js

Run node on the service.

node service.js

Open your web browser to the IP address of your host system using :3000 to set the request port.
You should see the hello message.
Use Ctrl+c to exit node & Ctrl+d to exit the container.

You now have a functional container that is set up to run this simple script.
Use the ls --all command to see the container you built.

docker container ls --all

Use the start command to start the container up again.
Use the container ID you found from the ls command.

docker container start <CONTAINER ID>

Check that the container is running and publishing port 3000.

docker container ls

Use the attach command to re-attach to the container.

docker container attach

Run the node command again.

node service.js

var http = require('http');

var server = http.createServer( function(req, res) {
 if (req.url === '/') {
  res.write("Hello World!");
  res.write("I have been expecting you.");
  res.end();
 }

 if (req.url === '/snarky') {
  res.write("If you code it, it will crash!");
  res.end(); 
 }

});

// Port 3000 on IPv4
server.listen(3000, "0.0.0.0");

BASH date picker

Sometimes I need users to pick a date from within a bash shell but I need the format to be correct. This little script lets users pick a date and it will output in whatever format the script needs.

Users can use the arrow keys to move the date on the calendar and press enter to select the date.

#!/bin/bash

DONE=0
DATE=$(date +%F)
YEAR="$(echo $DATE | cut -b1-4)"
MONTH="$(echo $DATE | cut -b6-7)"
DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
while [ $DONE -eq 0 ]; do
 clear
 cal -h ${MONTH} ${YEAR} | GREP_COLOR='47;30' grep "$DAY" -wC6 --color=always
 read -rsn1 KEY
 [[ $KEY == $'\x1b' ]] && { 
  read -rsn1 -t 0.1 DC
  read -rsn1 -t 0.1 KP
  case $KP in
   "A") DATE=$(date +%F -d "$DATE 7 day ago");;
   "B") DATE=$(date +%F -d "$DATE 7 day");;
   "C") DATE=$(date +%F -d "$DATE 1 day");;
   "D") DATE=$(date +%F -d "$DATE 1 day ago");;
  esac
 } || { 
  DONE=1
 }
 
 YEAR="$(echo $DATE | cut -b1-4)"
 MONTH="$(echo $DATE | cut -b6-7)"
 DAY="$(echo $DATE | cut -b9-10 | sed -e 's/^0/ /')"
done
echo $DATE

# To change the output format use the date command like these examples do
# date +%d/%m/%Y -d "$DATE"
# date -d "$DATE"

Inserting ID with case

I use sqlite3 for a lot of small projects. It has many features that are very impressive for such a small DB engine.

One thing I like to do is set the value of an index field using MAX(id)+1 from the table itself but that does not work on a blank table. To make this work with a blank table a CASE statement can be used.

CREATE TABLE microServices (
   id INT NOT NULL, 
   packageName CHAR(45) NULL, 
   serviceName CHAR(45) NULL, 
   deployTargets CHAR(45) NULL, 
PRIMARY KEY (id));

INSERT INTO microServices (
   id, 
   packageName, 
   serviceName) 
   SELECT CASE 
      WHEN MAX(id) > 0 
         THEN MAX(id)+1 
      ELSE 1 
   END, 
   'trucks', 
   'trucksd' 
FROM microServices;

Are you mocking me up?

There are a lot of mockup / script test sites, one that I like and try to make use of is JSFiddle. It allows you to test snippets of code. You cat enter HTML, CSS & JavaScript. It gives you the option to include and dozens of JavaScript libraries including AngularJS, jQuery and many others.

See the screenshot below the code snippets.


Paset this into the HTML box

<canvas id="myCanvas" width="578" height="200"></canvas>

Paste this into the JavaScript box, select jQuery 3.3.1 then click run

$(function() {
  var c = document.getElementById("myCanvas");
  var ctx = c.getContext("2d");

  function DrawBox(x, y, w, h) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "black";
    ctx.moveTo(x, y);
    ctx.lineTo(x + w, y);
    ctx.lineTo(x + w, y + h);
    ctx.lineTo(x, y + h);
    ctx.lineTo(x, y);
    ctx.stroke();
  }

  function DrawLine(x1, y1, x2, y2) {
    ctx.beginPath();
    ctx.lineWidth = "2";
    ctx.strokeStyle = "red";
    ctx.moveTo(x1, y1);
    ctx.lineTo(x2, y2);
    ctx.stroke();
  }

  DrawBox(10, 10, 40, 20);
  DrawBox(70, 30, 40, 20);
  DrawLine(50, 20, 70, 40);
  DrawBox(70, 70, 40, 20);
  DrawLine(30, 30, 90, 70);

});