I recently found hard drives from an old RAID array I stopped using over a decade ago. I wanted to recover the data from these disks, and that turned out to be more challenging than expected. This post outlines the steps, and hopefully helps someone else in future.

This was a four 750GB disk RAID-5 array using Intel Storage Matrix “fake-raid” (now called Intel Rapid Storage Technology). This is a RAID solution that uses a mix of software and hardware. I no longer have this Intel hardware, and in fact I no longer have a machine that would accept four drives. Luckily mdadm seems to have a pure software implementation of Intel Storage Matrix, so I hatched a plan. I would:

1. Create disk images for each of the four drives,
2. Mount the images locally as block devices,
3. Use mdadm to construct an array,
4. Copy the data into my backups.

1. Create disk images

I have a USB SATA adapter, and connected one drive at a time to my PC. This computer has a single local 12 TB drive, which I would store the disk images to. I start to create the disk images using:

$ sudo dd if=/dev/sdc of=1.raw

This worked great for the first disk, but the 2nd disk fail around the 600GB point. It seems this drive has developed bad blocks, but I kept my fingers crossed that this was still recoverable since this was RAID-5 after all. I switched up to using ddrescue.

$ sudo ddrescue /dev/sdc 2.raw 2.log --try-again --force --verbose

This worked great, and was able to create a full 750GB image, slowly retiring the failed blocks, recovering as much as possible. After about a week of copying I had four disk images, 1.raw, 2.raw, 3.raw, 4.raw, with only the 2nd disk having problems.

I now, chmod -w *.raw to remove write permissions to the images, helping to prevent a future step accidently altered the images.

2. Mounting the images

To mount the images I use losetup (roughly following instructions here), specifically:

$ sudo losetup -r /dev/loop31 1.raw
$ sudo losetup -r /dev/loop32 2.raw
$ sudo losetup -r /dev/loop33 3.raw
$ sudo losetup -r /dev/loop34 4.raw

Later I would use sudo losetup -d /dev/loop3[1234] to unmount these images. I then decided to inspect these drives, to see what partitions were on them:

$ sudo fdisk -l /dev/loop31

Disk /dev/loop31: 698.65 GiB, 750156374016 bytes, 1465149168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd204616a

Device        Boot Start        End    Sectors Size Id Type
/dev/loop31p1          1 4294967295 4294967295   2T ee GPT

$ sudo fdisk -l /dev/loop32

Disk /dev/loop32: 698.65 GiB, 750156374016 bytes, 1465149168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

$ sudo fdisk -l /dev/loop33

Disk /dev/loop33: 698.65 GiB, 750156374016 bytes, 1465149168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x899c1289

Device        Boot      Start        End    Sectors   Size Id Type
/dev/loop33p1        33488921 4294836216 4261347296     2T ee GPT
/dev/loop33p2        35651584   35651584          0     0B  0 Empty
/dev/loop33p3               0    1377535    1377536 672.6M 12 Compaq diagnostics
/dev/loop33p4      3071040408 3104693987   33653580    16G 64 Novell Netware 286

Disk 1 had a single partition, disk 2 and 4 had no partitions, and the 3rd disk had four! Those partitions looked a little weird, and I wondered for a minute if I mixed up my drives, or reformatted them at some point. I tried to mount them to no success, so I just assumed the RAID added something that looked like a real partition table. So I moved onto the next step.

3. Use mdadm to construct an array.

This is where it got difficult, due to limitations of mounting local disks, and the Intel Storage Matrix support.

I started by asking mdadm to examine the images (telling it to use imsm):

$ sudo mdadm --examine -e imsm /dev/loop31
mdadm: /dev/loop31 is not attached to Intel(R) RAID controller.
mdadm: Failed to retrieve serial for /dev/loop31
mdadm: Failed to load all information sections on /dev/loop31

Well that’s not a great start. If I understand the error /dev/loop31 is not attached to Intel(R) RAID controller it implies I need to connect my drive (or in this case loopback disk image) via a real RAID controller. Well that defeats my whole plan. After some googling, I found this stackoverflow post pointing out there is a IMSM_NO_PLATFORM=1 environment various I could set. The messaging is not attached to Intel(R) RAID controller was really a warning, and had no actual bearing on the problem.

$ sudo IMSM_NO_PLATFORM=1 mdadm --examine -e imsm \
  /dev/loop31 /dev/loop32 /dev/loop33 /dev/loop34
mdadm: no recogniseable superblock on /dev/loop34
mdadm: Cannot assemble mbr metadata on /dev/loop33
mdadm: no recogniseable superblock on /dev/loop32
mdadm: Cannot assemble mbr metadata on /dev/loop31

A new set of errors, but they did not look promising. More head scratching, and I hit a bit of a dead end. I now wondered if the drives were corrupt, making the superblocks unreadable. I decided to start to read the source code for mdadm to try and understand the superblock format, and see what was wrong.

It indicated the superblock (the data structure containing information about the array) was two sectors from the end of the disk, starting with the string Intel Raid ISM Cfg Sig. .

Guessing that a sector is 512 bytes long, I did the following:

$ tail -c 1024 3.raw  | hd

00000000  49 6e 74 65 6c 20 52 61  69 64 20 49 53 4d 20 43  |Intel Raid ISM C|
00000010  66 67 20 53 69 67 2e 20  31 2e 33 2e 30 30 00 00  |fg Sig. 1.3.00..|
00000020  cc c0 3d de 48 02 00 00  40 d5 11 d4 09 ae 19 00  |..=.H...@.......|
00000030  f8 11 00 00 10 00 00 a0  04 01 02 00 00 00 00 00  |................|
00000040  40 d5 11 d4 00 00 00 00  00 00 00 00 00 00 00 00  |@...............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000d0  00 00 00 00 00 00 00 00  53 31 33 55 4a 31 4b 51  |........S13UJ1KQ|
000000e0  34 30 33 33 33 37 00 00  f0 66 54 57 00 00 01 00  |403337...fTW....|
000000f0  3a 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |:...............|
00000100  00 00 00 00 00 00 00 00  53 31 33 55 4a 44 57 51  |........S13UJDWQ|
00000110  33 34 36 34 35 37 00 00  f0 66 54 57 00 00 02 00  |346457...fTW....|
00000120  3a 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |:...............|
00000130  00 00 00 00 00 00 00 00  53 31 33 55 4a 44 57 51  |........S13UJDWQ|
00000140  33 34 36 36 36 38 00 00  f0 66 54 57 00 00 03 00  |346668...fTW....|
00000150  3a 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |:...............|
00000160  00 00 00 00 00 00 00 00  53 31 33 55 4a 31 4b 51  |........S13UJ1KQ|
00000170  34 30 33 33 32 34 3a 30  00 66 54 57 ff ff ff ff  |403324:0.fTW....|
00000180  02 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  52 41 49 44 00 00 00 00  |........RAID....|
000001a0  00 00 00 00 00 00 00 00  00 f8 fc 05 01 00 00 00  |................|
000001b0  8c 10 00 00 00 00 00 00  00 00 01 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001e0  00 00 00 00 00 00 00 00  a6 a8 ae 00 00 00 00 00  |................|
000001f0  00 02 00 ff 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

Boom, the super block was there, started with a valid header, and even had other fields that looked correct (e.g S13UJ1KQ being a serial number of the drive).

Ok, so now I’m confused about what is wrong, and I wondered if this was a bug in mdadm. Going back I remember the first error I got contained Failed to retrieve serial, and I noticed the serial numbers were in the super block (e.g S13UJ1KQ). It then occurred to me, that once I imaged the hard drives, the images don’t contain the serial numbers!

Inspecting the code some more, it would fail with that error if it was unable to read the drive’s serial number. The loopback device doesn’t support serial numbers, so this started to make sense. I did however found a undocumented environment variable IMSM_DEVNAME_AS_SERIAL, which would instead of reading the serial number from the hardware, just use the name of the device as the serial (e.g /dev/loop31). This feature seems explicitly designed to help testing the mdadm codebase.

$ sudo IMSM_DEVNAME_AS_SERIAL=1 IMSM_NO_PLATFORM=1 mdadm --examine -e imsm /dev/loop31 /dev/loop32 /dev/loop33 /dev/loop34
…
/dev/loop31:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.3.00
    Orig Family : d411d540
         Family : d411d540
     Generation : 0019ae09
     Attributes : All supported
           UUID : ff44bc31:56060902:afb34379:b0faf183
       Checksum : de3dc0cc correct
    MPB Sectors : 2
          Disks : 4
   RAID Devices : 1

[RAID]:
           UUID : 676c222f:760eaf46:97bd30b8:989d2470
     RAID Level : 5
        Members : 4
          Slots : [UUU_]
    Failed disk : 3
      This Slot : ?
    Sector Size : 512
     Array Size : 4395431936 (2095.91 GiB 2250.46 GB)
   Per Dev Size : 1465144328 (698.64 GiB 750.15 GB)
  Sector Offset : 0
    Num Stripes : 11446438
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : degraded
    Dirty State : clean
     RWH Policy : off

  Disk00 Serial : S13UJ1KQ403337
          State : active
             Id : 00010000
    Usable Size : 1465138766 (698.63 GiB 750.15 GB)

  Disk01 Serial : S13UJDWQ346457
          State : active
             Id : 00020000
    Usable Size : 1465138766 (698.63 GiB 750.15 GB)

  Disk02 Serial : S13UJDWQ346668
          State : active
             Id : 00030000
    Usable Size : 1465138766 (698.63 GiB 750.15 GB)

  Disk03 Serial : S13UJ1KQ403324:0
          State : active
             Id : ffffffff
    Usable Size : 1465138526 (698.63 GiB 750.15 GB)

Ok, slowly making progress! Now it lists all the superblock information, and I was happy to see Checksum : de3dc0cc correct, etc. However, it listed Failed disk : 3, and This Slot : ?. It made me think without the valid serial numbers, it didn’t know which drive was which, and thus couldn’t assemble the array.

This made me ponder that if I was ever going to create a RAID array implementation, I would not make it depend on information from the hardware. How do folks re-image disks? What is wrong with some GUID in the superblock to identify the disk? Ok digression aside.

To move forward, I needed to trick mdadm to think that serial /dev/loop31 was actually the real hardware. I went back to my drives, and visibility inspected them to check the serial numbers.

  Disk00 Serial : S13UJ1KQ403337   1.raw
  Disk01 Serial : S13UJDWQ346457   2.raw
  Disk02 Serial : S13UJDWQ346668   4.raw
  Disk03 Serial : S13UJ1KQ403324   3.raw

At this point, I realised I had accidentally swapped drives 3 and 4. Quickly renaming them got them into the correct order.

Since I had already looked over the mdadm source code, it seemed a simple clean codebase, so I decided to change it to accept serial numbers. After a little while I did the hackiest thing possible:

diff --git a/super-intel.c b/super-intel.c
index da376251..d466d911 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -3994,6 +3994,20 @@ static int nvme_get_serial(int fd, void *buf, size_t buf_len)
        if (!name)
                return 1;

+       if (strcmp(name, "loop31") == 0) {
+               strcpy((char *)buf, "S13UJ1KQ403337");
+               return 0;
+       } else if (strcmp(name, "loop32") == 0) {
+               strcpy((char *)buf, "S13UJDWQ346457");
+               return 0;
+       } else if (strcmp(name, "loop33") == 0) {
+               strcpy((char *)buf, "S13UJDWQ346668");
+               return 0;
+       } else if (strcmp(name, "loop34") == 0) {
+               strcpy((char *)buf, "S13UJ1KQ403324");
+               return 0;
+       }
+
        if (strncmp(name, "nvme", 4) != 0)
                return 1;

The nvme_get_serial function now had hard coded serial numbers when reading loop3[1234]. This obviously isn’t a generalised solution, but worked for me. Go open source!.

$ make mdadm
…

$ sudo IMSM_NO_PLATFORM=1 ./mdadm --examine -e imsm /dev/loop31 /dev/loop32 /dev/loop33 /dev/loop34

Examine looked good, so the moment of truth, let’s assemble.

$ sudo IMSM_NO_PLATFORM=1 ./mdadm --assemble --readonly -e imsm /dev/md0 /dev/loop31 /dev/loop32 /dev/loop33 /dev/loop34
mdadm: Container /dev/md0 has been assembled with 3 drives

Ok mixed success, it says 3 drives, but I would expect 4… But let’s keep going

$ sudo ./mdadm --assemble --scan
mdadm: Started /dev/md/RAID_0 with 3 devices

W00t! It Started without errors!

I now have a /dev/md0, /dev/md127 and /dev/md127p1 devices.

$ sudo mount -o ro /dev/md127p1 /mnt/raid5

$ ls /mnt/raid5
… lots of old files...

YAY. Finished!

Ok, I’m not sure why it says three drives not four.

$ sudo ./mdadm --detail /dev/md127
/dev/md127:
         Container : /dev/md0, member 0
        Raid Level : raid5
        Array Size : 2197715968 (2.05 TiB 2.25 TB)
     Used Dev Size : 732572032 (698.64 GiB 750.15 GB)
      Raid Devices : 4
     Total Devices : 3

             State : clean, degraded
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0

            Layout : left-asymmetric
        Chunk Size : 64K

Consistency Policy : resync


              UUID : 676c222f:760eaf46:97bd30b8:989d2470
    Number   Major   Minor   RaidDevice State
       2       7       31        0      active sync   /dev/loop31
       1       7       32        1      active sync   /dev/loop32
       0       7       33        2      active sync   /dev/loop33
       -       0        0        3      removed

This does seem to imply a drive is missing. Maybe it doesn’t matter, as it mounted successfully, and I can copy all my data off the array.

Conclusion

This did not seem the easiest task, and there were a few road bumps along the way. Hopefully the hacks in here will help someone else out in a similar situation.

To finally clean up, you can run this:

$ sudo umount /mnt/raid5
$ sudo mdadm --stop /dev/md127
$ sudo mdadm --stop /dev/md0
$ sudo losetup -d /dev/loop3[1234]

I got a nice warning email from Let’s Encrypt that my cert was going to expire soon, and hadn’t been renewed. I found in /var/log/letsencrypt/letsencrypt.log the following error:

Renewal configuration file /etc/letsencrypt/renewal/mydomain.bramp.net.conf (cert: mydomain.bramp.net) produced an unexpected error: 'Namespace' object has no attribute 'dns_cloudflare_credentials'. Skipping.

I manually ran certbot in dry-run mode and it worked fine:

$ sudo certbot renew --dry-run

So this error only occurs when certbot is running as a cron job. Looking at /etc/cron.d/certbot I see the user runs as root, so I tried the certbot renew --dry-run again, but this time as the root user:

$ sudo su
root@:~$ sudo certbot renew --dry-run

and bam, the same error. This error somehow related to the certbot-dns-cloudflare plugin, which proves the ownership of the domain with a DNS01 challenge via Cloudflare’s DNS. I use this form of challenge, because the domain in question is internal and not available on the Internet.

I had forgotten how I installed the plugin, but searching Google, it seems to be via pip3. Clearly something was different between my root and normal user w/ sudo environments. So I did the following

$ sudo pip3 list | grep certbot
certbot (0.23.0)
certbot-apache (0.23.0)
certbot-dns-cloudflare (0.24.0)

$ sudo su
root@:~$ pip3 list | grep certbot
certbot (0.23.0)
certbot-apache (0.23.0)

Aha, no certbot-dns-cloudflare when running as root. Clearly I hadn’t installed this correctly. Running pip3 install certbot-dns-cloudflare as root fixed the problem, and voila, certbot correctly fetches new certs via a regular cron.

pt-kill is a neat little application that can stop long running MySQL queries. It was formally know as mk-kill before Percona took over the project. Here is the init.d script I use (as one doesn’t seem provided by the project):

#!/bin/sh
#
# pt-kill	This shell script takes care of starting and stopping
#               the pt-kill services.
#
# chkconfig: - 60 20
# description: pt-kill stops long running MySQL queries
#
# probe: true

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0

# See how we were called.
case "$1" in
  start)
    echo -n $"Starting pt-kill: "
 
    pt-kill \
      --pid /var/run/pt-kill.pid \
      --daemonize \
      --interval 5 \
      --busy-time 60 \
      --wait-after-kill 15  \
      --ignore-info '(?i-smx:^insert|^update|^delete|^load)' \
      --match-info '(?i-xsm:select)' \
      --ignore-user '(?i-xsm:root)' \
      --log /var/log/mysql-kill.log \
      --print \
      --execute-command '(echo "Subject: pt-kill query found on `hostname`"; tail -1 /var/log/mysql-kill.log)|/usr/sbin/sendmail -t you@example.com' \
      --kill-query
 
    RETVAL=$?
    echo
    [ $RETVAL -ne 0 ] && exit $RETVAL
 
  ;;
  stop)
        # Stop daemons.
       	echo -n $"Shutting down pt-kill: "
        killproc pt-kill
        echo
	;;
  restart)
	$0 stop
        $0 start
        ;;
  *)
    	echo $"Usage: pt-kill {start|stop}"
        RETVAL=3
        ;;
esac
 
exit $RETVAL

Usage:

Create the script as /etc/init.d/pt-kill, and change the pt-kill command in the middle of the script to suit your needs. Then run ‘chkconfig –level 345 pt-kill on’ to ensure this script starts up at boot. Alternatively test the script with ‘/etc/init.d/pt-kill start’ or ‘/etc/init.d/pt-kill stop’.

Thanks to MySQL Diary as they provided their default pt-kill command line. Perhaps in future someone could create a more generic startup script.

In a previous post I obtained the Linksys E4200v2 firmware, now I plan to break it apart and see what I can find.

I start off by simplying using “file” on the firmware:

$ file FW_E4200_2.0.36.126507.SSA 

FW_E4200_2.0.36.126507.SSA: u-boot legacy uImage, Linux-2.6.35.8, Linux/ARM, OS Kernel Image (Not compressed), 2677476 bytes, Thu Dec 22 19:40:21 2011, Load Address: 0x00008000, Entry Point: 0x00008000, Header CRC: 0x6ADD9801, Data CRC: 0xB010442D

Well this is a great start. We know we are dealing with Linux, and that this is a normal uImage. I then move on to use a neat little tool called binwalk. By using libmagic, binwalk tries to find interesting sections of the file.

$ /usr/local/bin/binwalk FW_E4200_2.0.36.126507.SSA 

DECIMAL   	HEX       	DESCRIPTION
-------------------------------------------------------------------------------------------------------
0         	0x0       	uImage header, header size: 64 bytes, header CRC: 0x6ADD9801, created: Thu Dec 22 19:40:21 2011, image size: 2677476 bytes, Data Address: 0x8000, Entry Point: 0x8000, data CRC: 0xB010442D, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: Linux-2.6.35.8
1124      	0x464     	LZMA compressed data, properties: 0x87, dictionary size: 250216448 bytes, uncompressed size: 14786800 bytes
16636     	0x40FC    	gzip compressed data, from Unix, last modified: Thu Dec 22 19:40:18 2011, max compression
2752512   	0x2A0000  	JFFS2 filesystem data little endian, JFFS node length: 49
..A whole lot of JFFS2 sections..
20974612  	0x1400C14 	JFFS2 filesystem data little endian, JFFS node length: 51
20974664  	0x1400C48 	JFFS2 filesystem data little endian, JFFS node length: 193

We find a small LZMA section, and large gzip section, and lots of JFFS2 sections. JFFS2 is a popular embedded file system, so we can guess the bulk of the file system is here. Next we can extract each section using dd:

dd bs=1 skip=1124  count=15512   if=FW_E4200_2.0.36.126507.SSA of=image-1.lzma
dd bs=1 skip=16636 count=2735876 if=FW_E4200_2.0.36.126507.SSA of=image-2.gz
dd bs=1 skip=2752512 if=FW_E4200_2.0.36.126507.SSA of=image-3.jffs2

Notice we are using a block size of 1 (so we can count in bytes), and we skip the offset into the file. Then we manually work out the sizes for the lzma and gzip sections. They can be no larger than their start until the next section. If they don’t fill that full space, then not to worry as these tools will normally ignore trailing data.

As I’m interested to see what’s in the JFFS filesystem, we should mount it. You can’t mount JFFS like a normal loopback device, you have to create a fake flash device. The following set of command can solve that:

sudo modprobe mtdram total_size=32768 erase_size=256
sudo modprobe mtdblock
sudo modprobe mtdchar
# sudo mknod /dev/mtdblock0 b 31 0
dd if=image-3.jffs2 of=/dev/mtdblock0
mount -t jffs2 /dev/mtdblock0 /mnt/disk

The mknod line is only needed if you don’t already have a /dev/mtdblock0. Also a /mnt/disk needs to be created ahead of time so the mounting works. Anyway once that was done, I cd /mnt/disk and found that it does appear to contain most of the file system. There are all the HTML pages, and binaries (for example busybox).

Now we should go back to image-1.lzma and image-2.gz. Well straight away trying to decompress image-1

$ lzma -dc image-1.lzma > image-1
lzma: Decoder error

results in a error. So we can assume that was a incorrectly detected by binwalk. Lets now try and decompress image-2.gz:

$ gzip -dc image-2.gz > image-2
gzip: image-2.gz: decompression OK, trailing garbage ignored

So that does indeed produce a large image-2 file, so we can ignore the trailing garbage warning. A quick “file” on image-2 doesn’t reveal anything useful, so I run binwalk on it. This turns up a set of false positives. So I take a different approach. I run:

$ strings image-2

This produces a whole host of valid looking strings. From the contents of the strings it makes me think it’s the actual kernel. A line like this:

Linux version 2.6.35.8 (root@ubuntu) (gcc version 4.2.0 20070413 (prerelease) (CodeSourcery Sourcery G++ Lite 2007q1-21)) #1 Thu Dec 22 16:40:10 PST 2011

helps me come to that conclusion.

I haven’t finished poking around image-2.gz, but I suspect the interesting parts are mostly in the JFFS2 filesystem. Hopefully this will lead to me getting ssh access to the router, and eventually being able to customise the firmware.

I recently got a Linksys E4200v2 wireless router. It’s pretty cool, supports IPv6, 2.4Ghz and 5Ghz wifi networks, VPN, etc. The one downside is that the firmware for the router is not available from Linksys’s website, and the GPL code has not been made available, yet… However, the router has been able to pull updates itself from the Internet.

So I wanted to acquire the firmware to see if I could do something fun with the router. I set about to figure out how the router does this. My plan was to set my laptop up between Internet interface on the router, and the cable modem. Since my laptop doesn’t have two network cards, I plugged into a switch and used Ethercap to ARP poison to redirect traffic via the laptop.

Then using Wireshark I could see the traffic coming out of the router. All I needed to do now was to hit the “check for updates button”.

Firstly I saw two DNS requests, one for the AAAA (IPv6) record for update.linksys.com, then a A request for update.linksys.com. Clearly the updates are coming from there. Secondly I saw a HTTPS connection form to that domain. That makes this a little more complex, as I am unable to see the encrypted traffic, and thus see what is being transferred.

So, I grabbed a simple DNS server, and set up a simple SSL server following these instructions.

Now with DNS spoofing, and a fake SSL server, I could intercept encrypted traffic from the router, as long as it does not validate the SSL certificate. Luckily it didn’t check the validity, and thus I was able to capture the request: (BTW Not checking the cert completely defeats the point of using SSL… bad Linksys!).

POST /cds/update HTTP/1.1
Host: update.linksys.com
Accept: */*
Content-Type: text/xml
Content-Length: 573

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header/>
  <SOAP-ENV:Body>
    <ns:GetFirmwareFromDeviceRequest xmlns:ns="http://cisco.com/schemas">
      <ns:LanguageCode>en</ns:LanguageCode>
      <ns:CountryCode>us</ns:CountryCode>
      <ns:MacAddress>12:34:56:78:90:ab</ns:MacAddress>
      <ns:ModelNo>E4200</ns:ModelNo>
      <ns:HardwareVersion>2</ns:HardwareVersion>
      <ns:CurrentFirmwareVersion>2.0.36.126507</ns:CurrentFirmwareVersion>
    </ns:GetFirmwareFromDeviceRequest>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

and the response:

HTTP/1.1 200 OK
Content-Language: en-US
Content-Type: text/xml
SOAPAction: ""

<soapenv:Envelope
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header/>
  <soapenv:Body>
    <ns:GetFirmwareFromDeviceResponse xmlns:ns="http://cisco.com/schemas">
      <ns:FirmwareList xmlns:ns="http://cisco.com/schemas">
        <ns:Firmware xmlns:ns="http://cisco.com/schemas">
          <ns:Version xmlns:ns="http://cisco.com/schemas">2.0.36.126507</ns:Version>
          <ns:Revision xmlns:ns="http://cisco.com/schemas">D</ns:Revision>
          <ns:ReleaseDate xmlns:ns="http://cisco.com/schemas">2012-01-06T16:48:08Z</ns:ReleaseDate>
          <ns:DownloadURI xmlns:ns="http://cisco.com/schemas">http://download.linksys.com/updates/to0037258865.pdx/FW_E4200_2.0.36.126507.SSA</ns:DownloadURI>
          <ns:DateFormat xmlns:ns="http://cisco.com/schemas">yyyy-MM-dd';T';HH:mm:ss';Z';</ns:DateFormat>
          <ns:Checksum xmlns:ns="http://cisco.com/schemas">1958710861</ns:Checksum>
        </ns:Firmware>
      </ns:FirmwareList>
    </ns:GetFirmwareFromDeviceResponse>
  </soapenv:Body>
</soapenv:Envelope>

(I slightly modified portions of the request and response to hide the identify of my router.).

I might write a script to make fake requests, but until then you can easily create a request with curl:

curl -d @request.raw https://update.linksys.com/cds/update

Then you just extract the DownloadURI and

curl http://download.linksys.com/updates/to0037258865.pdx/FW_E4200_2.0.36.126507.SSA

Voila I now have the firmware. Now I need to figure out what to do with it.

Update: As requested I fetched the more recent version of the file:
2.0.37.131047 – http://download.linksys.com/updates/to0040829450.pdx/FW_E4200_2.0.37.131047.SSA

I just tried to compile the Debian PHP packages, so I could make some minor tweaks to the source, to fix a bug I’m hopefully going to document shortly. This turned out to be very problematic, mainly due to the testing phase!

To build the Debian packages you typically do the following:

mkdir php
cd php
apt-get source php5
cd php5-*
debuild -us -uc -j4

While testing, the debian/setup_mysql.sh script is called, to create a temporary MySQL database. This however failed to execute correctly because I had some custom options in my ~/.my.cnf. Thus it failed like so:

# start our own mysql server for the tests
/bin/sh debian/setup-mysql.sh 1029 /home/bramp/vendor/php/php5-5.3.8.0/mysql_db
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
make: *** [test-results.txt] Error 1
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1348:
dpkg-buildpackage -rfakeroot -D -us -uc -j8 failed

After removing the my.cnf things were ok. The below patch fixes it in a more general way:

-- debian/setup-mysql.sh.org	2011-11-21 21:57:07.244481175 -0500
+++ debian/setup-mysql.sh	2011-11-21 21:40:39.384455880 -0500
@@ -16,7 +16,7 @@
 
 socket=$datadir/mysql.sock
 # Commands:
-mysqladmin="mysqladmin -u root -P $port -h localhost --socket=$socket"
+mysqladmin="mysqladmin --no-defaults -u root -P $port -h localhost --socket=$socket"
 mysqld="/usr/sbin/mysqld --no-defaults --bind-address=localhost --port=$port --socket=$socket --datadir=$datadir"
 
 # Main code #

The next problem I encountered is that all the automated PHP unit tests were failing, and they would eventually get stuck and use all my RAM and swap space (at least 16GiB of it) :( I’m not sure what made the machine run out of RAM, but the tests were failing because the version of PHP that was running the tests was incorrectly loading extensions from my system. The quick fix for this was to disable any extensions I had installed to my system. I just

sudo mv /etc/php5/conf.d /etc/php5/conf.d.tmp

to do that.

This made me think that in future I should perhaps find a cleaner environment to build these packages. In fact, it makes me wonder if the builds are just broken if my environment clearly impacts the successful run of tests.

One trick I found while building again and again, is that you can pass “-nc” to debuild so that it does not clean, and thus reuses the existing build materials for a faster build. Either way, I now have my own version of PHP built and installed! Next time I might just ignore the .deb files and build direct from source.

I really like mtr (the traceroute tool), however, it always bugged me that it launches a GUI app instead of using the curses interface. You can easily pass the “-t” or “–curses” flag to default to the curses interface, but I always forget.

So today I set about writing a patch for mtr to read a environment varable to force the curses interface. While I was reading the source I noticed there was already a simple undocumented way!

export MTR_OPTIONS=-t

Voila the curses interface is now used by default. This is certainly going into my ~/.bashrc

Does your Linux 64bit version of Flash now make anonying beeping/distortion noises while playing videos? Well it turns out a recent “improvement” to glibc has caused some programs to now crash or do weird things. This is to do with an improvement of memcpy, which makes its use more strict, causing those applications that incorrectly used it to now crash.

On Debian, there is a simple a fix that allows you to use the original memcpy. You can load the application using an additional .so file:

LD_PRELOAD=/usr/lib/libc/memcpy-preload.so /path/to/your/program

As I use Google Chrome when using Flash I had to do the following:

LD_PRELOAD=/usr/lib/libc/memcpy-preload.so /usr/bin/google-chrome

but if you want to fix chrome on a system level, you can edit the Chrome Wrapper used to launch it.

sudo nano /opt/google/chrome/google-chrome

add the following line

export LD_PRELOAD="/usr/lib/libc/memcpy-preload.so"

for example

export LD_LIBRARY_PATH
export LD_PRELOAD="/usr/lib/libc/memcpy-preload.so"

export CHROME_VERSION_EXTRA="beta"

Out of fun I wrote a simple Python script to pull the version number out of Intel’s ucode firmware, for example, the firmware used by my wifi driver. I needed this so I could see what version I was running versus a new version I had downloaded from Intel’s Linux Wireless site.

So here is the code if anyone finds it interesting:

and example of using it is:

$ ./ucode.py /lib/firmware/*.ucode
iwlwifi-1000-3.ucode    : ver 128.50.3.1
iwlwifi-3945-1.ucode    : ver 15.28.1.6
iwlwifi-3945-2.ucode    : ver 15.32.2.9
iwlwifi-4965-1.ucode    : ver 228.57.1.21
iwlwifi-4965-2.ucode    : ver 228.61.2.24
iwlwifi-5000-1.ucode    : ver 5.4.1.16
iwlwifi-5000-2.ucode    : ver 8.24.2.12
iwlwifi-5150-2.ucode    : ver 8.24.2.2
iwlwifi-6000-4.ucode    : ver 9.221.4.1
iwlwifi-6000g2a-5.ucode : ver 17.168.5.1 (6000g2a fw v17.168.5.1 build 33993)
iwlwifi-6000g2b-5.ucode : ver 17.168.5.1 (6000g2b fw v17.168.5.1 build 33993)
iwlwifi-6050-4.ucode    : ver 9.201.4.1
iwlwifi-6050-5.ucode    : ver 41.28.5.1 (6050 fw v41.28.5.1 build 33926)

I’ve recently been running network benchmarks and I wanted to highlight a quick gotcha that caught me out. My benchmark script would run ifconfig before and after the experiment and record the number of packets sent and received. The same would be recorded on another machine which was generating the packets.

I started to notice a problem that more packets were being received than being sent. It was only a small number, but nevertheless an seemingly impossible situation to have occurred. I tracked the problem down to the fact that the network card driver did not always report accurate and up to date stats. In fact, after checking the source, the e1000, ixgb, ixgbe and most likely the other Intel drivers all refresh their stats every 2 seconds. The other NIC I was using, an embedded smsc9500, always updated its stats in realtime.

Delaying the update of stats is obviously a performance optimisation, which no doubt avoids costly locking, or problems such as cache line bouncing. I did some searching and other people have noticed similar delays in other networking devices, in some cases as much as 30 seconds on cisco routers.

From now on I will always pause my scripts for a few seconds to allow the ifconfig stats to catch up before I record my data. An alternative would be to hack the drivers to either stay up to date, or offer a API that forces a update.

A while ago I wrote a python script that does a similar job to GNU’s watch command. You use it like so:

./persec.py [--interval=<n>] <command>

so for example

./persec.py ifconfig

Now in a similar way to watch, it executes the command every second, and highlights the differences between each execution. However, in addition to this it finds any numbers that have changed and works out the rate at which they are changing. So for example, ifconfig would typically output this:

usb0      Link encap:Ethernet  HWaddr 02:04:4b:00:d3:cf
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1017422291 errors:0 dropped:0 overruns:0 frame:0
          TX packets:549382406 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1910704266 (1.9 GB)  TX bytes:1834667124 (1.8 GB)

but now outputs something like:

usb0      Link encap:Ethernet  HWaddr 02:04:4b:00:d3:cf
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:<b>2001/s</b> errors:0 dropped:0 overruns:0 frame:0
          TX packets:<b>2002/s</b> errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:<b>168120/s</b> (1.9 GB)  TX bytes:<b>217144/s</b> (1.8 GB)

Notice the per second (/s) values for RX/TX packets and RX/TX bytes. I have found this quite useful many times in the past, on commands such as:

./persec.py cat /proc/interrupts
./persec.py df
./persec.py ls -l somefile

Download version 1.1 or View on Github

I just purchased the Humble Indie Bundle, 5 games which run on Windows, Linux and Mac. They kindly offered md5 hashes, however they weren’t in a convenient format, so I created a md5sums file.

187287db6226ef6a306a4cd0cfa8dd45 *Aquaria111.2008.12.12.exe
8b24ddeeb553e028bbd501102f891cc2 *aquaria-lnx-humble-bundle.mojo.run
336d89eb431e0b00535a2b50146c579d *WorldOfGooSetup.1.30.exe
f5afa40893d0fbcc37885191404f6d8c *WorldOfGooSetup.1.41.tar.gz
b373132f6e78665f1076752b038a8218 *gish153-1.tar.gz
94e82d53e2104914f19ec97f7cf5da42 *gish1531.zip
41ea2e41fe42c40b5ad017ae1afb45c7 *lugaru-full-linux-x86-1.0c.bin
d40d4076ed3ffa2a9bc9913202c55a06 *lugaru-windows.zip
56060bb550e0cd7e90bc7bc60d7c4c23 *penumbra_overture_1.1.exe
539567b590cf53e65a8e8f1b2bc728b3 *penumbra_overture_1.1.sh

and for those that prefer sha1:

ec8fd66bb44c6f67eab40f73e57838dcb5e2f4dd *Aquaria111.2008.12.12.exe
48267967b3a57fb406fa6545261f8b758edb8fb5 *aquaria-lnx-humble-bundle.mojo.run
70fad514746d6fc482fc3d681a7e9d498374bdad *WorldOfGooSetup.1.30.exe
4f7202a4ac17dd1665a1ab7f90153e5b813e16f1 *WorldOfGooSetup.1.41.tar.gz
27b862939b6a01c29b1b146ed1307c0027217855 *gish153-1.tar.gz
7a1341822d4d4e0010cc1e8cce68da6bb02ea904 *gish1531.zip
2349bdea3d595c3aaedca0810229d12f96323874 *lugaru-full-linux-x86-1.0c.bin
73f09414e0cabd371802eea1b9c75c76522c5934 *lugaru-windows.zip
557c6988eda16c6269d09a35031fd1754e042c02 *penumbra_overture_1.1.exe
e115f7cfcf9710d7aa5b2a5a9186b3736bb55cf2 *penumbra_overture_1.1.sh

Note, these were correct and validated on the 8th of May, however, if any of the files have been updated the hashes might no longer much. Feel free to contact me to tell me if this happens.

I came into my office today to find that my keyboard would no longer type capital letters. The shift, caps lock, and even num lock keys seemed to be broken. After a quick WTF it was pointed out to me that VMWare occasionally does this to Linux machines. Thanks to Matt who showed me a quick solution. In a terminal just type:

setxkbmap

And everything should be back to normal. After using VMWare on Linux for a couple of years I’ve never encountered this problem!

Very occasionally I come across a Linux application that does not respect the http_proxy variable. This stops the application from working while I’m at university as they forbid traffic on port 80 unless it goes via their webcache. So today I came up with a hack of a solution that involves iptables:

# IP address and port number of the webcache
WEBCACHE=194.80.32.10:8080

# Flush any previous rules
iptables -t nat --flush

# Delete and recreate the chain
iptables -t nat -X HTTPFORCE
iptables -t nat -N HTTPFORCE

# Don't touch local traffic (localhost and internal network)
iptables -t nat -A HTTPFORCE -o lo -j RETURN
iptables -t nat -A HTTPFORCE --dst 127.0.0.1/8 -j RETURN
iptables -t nat -A HTTPFORCE --dst 10.0.0.0/8 -j RETURN
# Add any other local networks here.

# Now we have two options. Please uncomment out one of them
# 1) Redirect packets on port 80 to the webcache
#    This may not work unless the webcache is generous with its input
#iptables -t nat -A HTTPFORCE -p tcp --dport 80 -j DNAT --to $WEBCACHE

# 2) Redirect packets on port 80 to localhost port 1234
#    On port 1234 you need to run a local web proxy, which forwards 
#    requests to the real webcache
#iptables -t nat -A HTTPFORCE -p tcp --dport 80 -j REDIRECT --to-port 1234

# Capture all outgoing TCP syns
iptables -t nat -A OUTPUT -p tcp --syn -j HTTPFORCE

All outgoing TCP packets on port 80 which are not destined for a local network are captured and changed in one of two ways. The first option just manipulates the IP/TCP header, and the second redirects the traffic to a proxy running on localhost. The reason for the two options was that my university’s webcache seemed unable to deal with HTTP requests without the full URL in the GET line, even though the request contains a valid Host header. I think this is a misconfiguration of their squid proxy, so instead you can redirect to a local proxy which forwards the request on to the webcache.

This all seems a hassle but sometimes it is needed when a application does not respect the http_proxy environment. On a good note all libcurl applications should respect it by default.

Today I had the need to install some FreeBSD and Ubuntu packages inside my home directory because I did not have root permissions to install them.

It was quite simple to install the packages on FreeBSD

pkg_add -rRP /home/bramp nano

Where nano is the name of the package I wanted.

I should point out you don’t need to be root to do this!, but the installed packages only work for you, and in future could cause you hassle if the base OS is updated.

On Ubuntu I however failed :( I tried the “–root” option and did some Googleing but I didn’t find a solution. Unitl then I’ll just install from source.