Assign and return the old value

Finally, after 20 years, my old idea of "assign and return the old value" has been implemented in C++14 in the exchange function incarnation!

Nice!

Hot plug/unplug SATA drives on Linux

From time to time I need these commands.

Before hot-unplugging a disk drive sdX make sure that no file systems from the disk are mounted, no swap active, no LVM volumes active (N - partition number, VG - volume group, LV - logical volume, PV - physical volume):

lsblk /dev/sdX                 # see what's there
swapon -s | grep sdX           # check for active swap
swapoff /dev/sdXN              # deactivate swap if found
mount | grep sdX               # check for mounted FS
umount /dev/sdXN               # umount them if found
pvs | grep sdX                 # check for PVs
pvdisplay -m /dev/sdXN         # find LVs if a PV found
umount /dev/mapper/VG-LV       # umount LVs if found
lvchange -an /dev/mapper/VG-LV # deactivate LVs if any 

If it is the last PV of a volume group, the VG itself should be deactivated:

vgchange -an VG

Then prepare the device for unplugging (it makes the kernel send power down command to the drive and unregister it):

echo 1 > /sys/block/sdX/device/delete

And then unplug it physically.

After hot-plugging a disk drive on a host adapter X:

echo - - - > /sys/class/scsi_host/hostX/scan 

Then you can mount file systems, activate swap, activate LVs and mount them if needed.

Hanoi tower algorithm to move from any to any state (1993)

In the year 1993 I have developed an algorithm to solve a generalized Hanoi tower problem: move from any state to any state.

The algorithm is very simple and uses a nice state representation: a ternary number. The right digit represents the number of rod where the largest disk resides, then the number of the next disk and so on to the number of rod of the smallest disk. 64 bit unsigned integer can hold state of up to 40 disks.

For the original Hanoi tower problem we have this encoding: source=0000...0, destination=2222...2.

It was also possible to use strings (with the largest disk at the left), but it was harder to allocate memory in this case, so I chose the integer representation.

It is easy to note that for moving a large disk it is required to move all smaller disks to an intermediate rod. So the algorithm first reduces the source and destination states if some large disks are on their target positions, then recursively moves the smaller disks to an intermediate rod, then moves the large disk, then recursively moves the smaller disks to their final destinations.

Here is the algorithm encoded in C (written in 1993, adapted now for ANSI C):

#include <stdio.h>
 

void Move(int f,int t)
{
        printf("%c%c  ",f+'0',t+'0');
}

void Hanoi3(unsigned from,unsigned to,int n)
{
        unsigned via,f,t,v;

        int i;

        if(n==0)
                return; /* nothing to do */
 

        /* reduce equal positions of largest disks */

        do
        {
                f=from%3;
                t=to%3;
                from/=3;
                to/=3;
                n--;
        }
        while(f==t && n>=0);
 

        if(n<0)
                return; /* nothing to do */

        /* now f is the source position of the largest disk,

           and t is its target position */

        v=3-f-t; /* intermediate rod */
        via=0; /* intermediate state */
        for(i=0; i<n; i++)
                via=via*3+v;

        Hanoi3(from,via,n);
        Move(f,t);
        Hanoi3(via,to,n);
}

Recovering after loss of a physical volume with LVM2 on Linux

Recently I have lost a hard drive which contained a physical volume of LVM2.

It contained an insignificant file system with scratch data and a part of a RAID1 logical volume.

To recover, it was necessary to repair the RAID1 LV and remove other partial LV. I was surprised that the network lacked information on how to repair a RAID1 LV.

So here are the commands:

lvconvert --repair /dev/mapper/VG-LVraid1
lvremove /dev/mapper/VG-LVpartial
vgreduce --removemissing VG

A few glitches with Fedora 24 installations

I have noticed multiple times that when a process seems to hang, strace sometimes can kick it forward and cause it to proceed and either terminate as it should or fail with a signal.

There was a strange hang within systemd (PID 1), it apparently created a forked copy of itself which hung, and when I tried to strace the child process it failed with a signal. Then the parent (PID 1) ran pause() but any tries to send it SIGTERM or SIGUSR1 caused nothing, the pause() system call did not interrupt. I think that all signals were blocked. The systemd (PID 1) did not process parentless zombies either. Reboot was required.

One time when restarting mysqld it took forever to terminate, but finished in a second when I ran strace on it.

Another time, df hung due to unresponsive nfs. After I had unmounted the nfs filesystem using "umount -l -f" the df still lingered for minutes until I ran strace on it.

I suspect there is a bug in the kernel. And a bug in systemd.

kernel-4.9.9-100.fc24.x86_64
systemd-229-18.fc24.x86_64

Graceful squid restart

Squid is a well-known open source http proxy server.

Sometimes it is necessary to change its configuration, for example to update ACL lists. Fortunately it has the command "squid -k reconfigure". Unfortunately, during reconfiguration squid refuses new connections. If the configuration is complex and ACLs are large it can take several seconds.

Some people recommend setting up multiple squid servers with a load balancer to solve the problem, but I believe it's an overkill for small installations.

So here is my approach.

To avoid service disruption, start another copy of squid on the same machine with identical configuration but with different TCP ports and without persistent storage:

a_conf=/etc/squid/squid.conf
b_conf=/etc/squid/squid-b.conf
b_pid=/var/run/squid-b.pid

sed -e 's/^http_port \([0-9]\+\)/http_port 1\1/' \
    -e '/^cache_dir/d' < $a_conf > $b_conf
echo "pid_filename $b_pid" >> $b_conf

squid -f $b_conf

Then redirect new connections to the new port (i.e. 13128 by default) using iptables NAT.

Reconfigure the first copy of squid using "squid -k reconf", then remove NAT redirection and shutdown the second copy of squid using "squid -f $b_conf -k shutdown".

Port scan reporting

Internet worms scan the internet infecting new hosts, creating botnets, abusing services.

For a long time I have had implemented port scan detection and blocking script for our local users. The perl script analyzed netflow information and when a certain level was exceeded it informed the user by e-mail and blocked the port.

It had to handle a few special cases, for example smtp servers (MTA) easily exceeded threshold on ports 25 and 113, so the script probed the port 25 on the suspected host before taking an action. A few IP addresses were added to a white list.

Recently an idea to inform other providers came to me. Remote scanners were detected anyway, I had just to add an action. Whois service gives an abuse e-mail address, so composing a letter template was the only really creative task.

To avoid sending the letter too often I have added dynamic firewall rules with a timeout to block the scan traffic. So the warnings are sent in 1, 2, 4, 8 ... 32 days if the activity does not stop.

The script works and while Chinese internet providers largely ignore the warnings, I've received many replies which indicate that a real problem was noticed and fixed due to my messages.

It would be nice though if all ISP implemented local scan detection themselves.

Internet scanning activity

Recently I have analyzed scanning activity from the Internet on our ASBR routers.

Here are the most popular TCP ports being scanned (>1% of all scan attempts):

Port	Service	Percent	Purpose
22	ssh	12.6877	Shell access
23	telnet	9.4989	Shell access
1433	SQL server	7.13683	Data compromise, exploits
21320	SpyBot proxy	5.80395	Spam
3389	Remote Desktop	5.71959	Desktop access
3128	Squid Proxy	3.88055	Spam
8080	potential HTML proxy	3.82993	Spam
3306	MySQL	3.74557	Data compromise
445	SMB	1.56909	File compromise
103	?	1.48473
8888	potential HTML proxy?	1.45099	Spam
110	POP3	1.36663	E-Mail compromise, spam
20	FTP-data	1.33288	FTP exploit?
8000	DVR control port	1.14729	DVR access
3398	?	1.14729
79	finger	1.13042	User identity leak
789	?	1.09668
3397	?	1.0798
119	nntp	1.06293	NNTP exploit
3396	?	1.04606
21	FTP	1.04606	File compromise
465	smtps	1.02919	Spam

Some ports and their purpose are unknown to me.

P.S. I have found this useful resource on port scanning statistics.

P.P.S. Top scanning providers: chinanet.cn.net (21%), cnnic.cn (10%), chinaunicom.cn (8%), chinamobile.com (3%).

GSM modem woes

I've had a task to send SMS messages over a bunch of GSM USB modems.

Gnokii proved to work well with Nokia phones only, so I have written a custom perl script to send AT commands directly.

It took some time to program encoding the SMS in packet mode, but the main difficulty was unreliability of the modems and/or USB subsystem of linux.

Sometimes /dev/ttyUSBx devices stopped responding. Unplugging the modems by hand seemed unworthy thing to do, so I searched the internet for solutions.

At first I have found usbreset.c and fortunately it could recover the /dev/ttyUSBx devices, but it required /dev/bus/usb/... path to work on, so I searched again and found a method of mapping ttyUSB to /dev/bus/usb here.

Then I have added the following function to my perl script and called it during modem initialization if the modem did not reply to the first AT command.

sub usbreset($) {
        my ($device)=@_;
        use File::Slurp;
        require 'sys/ioctl.ph';
        my ($rdev)=(stat($device))[6];
        return if !$rdev;
        my ($major,$minor)=($rdev>>8,$rdev&255);
        my $path="/sys/dev/char/$major:$minor/../../../..";
        my $bus=read_file("$path/busnum");
        my $dev=read_file("$path/devnum");
        my $ctl=sprintf('/dev/bus/usb/%03d/%03d',$bus,$dev);
        open my $fd,'>',$ctl || return;
        # USBDEVFS_RESET from linux/usbdevice_fs.h
        my $res=ioctl($fd,_IO(ord('U'),20),0);
        return if !defined $res;
        return 1;
}

I'll have to monitor the modems for some time to find out if there are other problems with them.

Feel free to use this piece of code if you like (public domain).

P.S. using a quality USB hub solved the problem completely.

VLAN calculation algorithm design

I've had a task to automate calculation and setting up VLANs on switches of our regional network.

The task was naturally split into two phases:

1. calculate VLAN numbers on the switches and trunks;
2. synchronize switch configurations to the calculated result.

The first one was not quite obvious, a new algorithm needed to be designed. The second was done by another programmer.

I have designed and implemented the algorithm. Key points:

• We have list of devices, trunks, client port VLANs on each device as input data. VLAN sets for each trunk is the output. The network has multiple loops for redundancy.
• VLAN should be included on a trunk if it is two-way on the trunk, this means that we can spread VLANs from their endpoints on the graph and then calculate VLANs on trunks as intersection of sets of one direction and the other.
• We spread VLAN set from each device as a bit vector. Each trunk has two associated  VLAN sets, one for each direction.
• If we use STP protocol, then VLANs should be unified across each group of connected cycles (which have a common trunk).
• If we use ERPS, then the VLANs should be unified starting from most outer half-loop to the base loop.
• We can prune VLAN propagation if the trunk already contains all VLANs from the propagation set (an exception was necessary for multiply connected non-switch devices), thus runtime is reduced significantly.

I used perl and Bit::Vector module. The first version did not prune VLAN propagation and it took 1-2 minutes to complete on our network topology (more than 1600 devices), and after implementing the pruning it takes just 4 seconds (including database queries).

The following article was invaluable for understanding multi-ring ERPS topologies:

D. Lee, K. Lee, S. Yoo and J. K. K. Rhee, "Efficient Ethernet Ring Mesh Network Design," in Journal of Lightwave Technology, vol. 29, no. 18, pp. 2677-2683, Sept.15, 2011.

Reflections on the ancient ruins

When travelling in Greece I have visited Delphi and Athens. Watching the ruins in Delphi I wondered why all the buildings were destroyed and how.

And suddenly I have noticed a bronze peg in one of the stones. What is that?
Other stones had pits in them, but no pegs. Probably the stones of columns were connected with bronze pegs. They could be poured in via a small opening with molten bronze.

I have noticed only a few such pegs, but many pits. Also I have noticed that many of the pegs have holes from a wooden core. Bronze was not cheap at that time, armour and weapons were made of it. And the wooden inserts could be used to save bronze and reduce costs.

Next I thought: if the bronze was costly, it was logical for looters to try and get it.

So probably it was looters who finished the column destruction to get the bronze pegs. (Disclaimer: I'm not an archaeologist)

upgrade-routeros script

I have developed "upgrade-routeros" perl script for safe and client-friendly graceful upgrade (or reboot) of RouterOS on ASBR and BRAS MikroTik routers.

It solves some problems automatically, which required manual work before.

• BGP route updates don't propagate instantly. If we just upgrade the software by rebooting an ASBR, the traffic would be blackholed or looped for several minutes. To solve this problem, we need to disable BGP peers and wait for route propagation. Then we may safely reboot the router as the traffic is not directed to it any more. The new script waits for 5 mintes to let the network converge.
• PPPoE sessions should be terminated gracefully. Otherwise cheap CPEs may hang or stop reconnecting automatically. To solve this problem, the script gracefully disconnects PPPoE users before rebooting the BRAS. To avoid new PPPoE sessions it sets max-sessions=1. To avoid disturbing the users too much it waits for the sessions to be at least 2 hours old. To avoid hammering the radius server the sessions are disconnected one per second. PPPoE servers are disabled after disconnecting all users (but one) to allow one more reboot for firmware upgrade.

This allows quite graceful software upgrade, provided that the network has at least N+1 redundancy in ASBR and BRAS.

After upgrade the script checks if the upgrade was successful, upgrades routerboard firmware if needed, then re-enables BGP peers and/or PPPoE servers.

The script uses MikroTik::API perl module.