Windows on bramp.net

UTF-8 Directory Listing

Thu, 23 Sep 2010 00:00:00 +0000

I had a need to create a directory listing with all the UTF-8 characters intact. This seems quite a chore on Windows, as doing anything via the shell seems to mangle the characters and show ???? instead of the real characters. For example, both the built in dir and Cygwin ls or find seemed affected. This turns out to be a limitation in the windows shell.

To solve this problem I wrote a bit of python to read the file names in full UTF-8 and output the results directly to a file (and not via a pipe, which would again be via the shell). The resulting very simple script is as follows:

import os
import codecs

log = codecs.open('listing', mode='w', encoding="utf-8")

for root, dirs, files in os.walk(u'.'):
	log.write(root + u"\n")

	for file in sorted(files):
		log.write(os.path.join(root, file) + u"\n")

log.close()

Converting MBR to GPT without deleting your partitions

Mon, 30 Aug 2010 00:00:00 +0000

Today I tired to convert my Windows 2TB RAID disk from a master boot record (MBR) layout to a GUID partition table (GPT) one. The reason I wanted to swap is that GPT supports partitions larger than 2TB. Normally it is easy to convert a MBR disk to a GPT one by using the Disk Management GUI. However, Microsoft do not allow you to convert you disk if you have any partitions on the disk.

After some searching I found that you could convert the disk non-destructively on Linux, but due to various reasons, my RAID does not work inside my Debian Linux :( I had also read a document outlining how to the conversion works, and it seems such a simple process I was surprised that Microsoft didn’t support it.

Some more searching and I found a Linux/Windows tool that will convert MBR to GPT quite easily. Gptgen is a simple command line tool that seemed to work like a charm. I will quickly outline the steps I took to use it “safely”.

Firstly I identified with disk I wanted to alter. This is done by looking at which number the disk is in Disk Management. I then quickly tested the tool without writing the changes

gptgen.exe \\.\\physicaldrive1

This outputted quite a few lines, including the following:

Write primary.img to LBA address 0.
Write secondary.img to LBA address 4395431903.

When you don’t write the changes to disk, gptgen creates two binary files “primary.img”, and “secondary.img”, which contain what it would have written to disk. The console output from gptgen tells me it would have written primary to LBA address 0, and secondary.img to LBA address 4395431903. So I figured it was a good idea to make a backup of those sections first. To do this I use the Windows version of the classic tool dd

dd if=\\.\\physicaldrive1 of=primary-backup.img bs=512 count=34 
dd if=\\.\\physicaldrive1 of=secondary-backup.img bs=512 count=33 skip=4395431903

The count and skip values might have to change. The two count values relate to the file sizes of primary.img and secondary.img. Find their file sizes and divide them by 512. In my case, primary.img was exactly 17,408 bytes, so 17408 / 512 = 34. Do the same for secondary.img. The skip number is the LBA address shown by gptgen just a minute ago.

Ok, if you have run dd ok, you should have backups of the two sections it is going to overwrite. Now you can tell gptgen to actually make the changes:

gptgen.exe -w \\.\\physicaldrive1

That should be it, BUT if you need to (for whatever reason) restore the backups issue these commands:

dd if=primary-backup.img   of=\\.\\physicaldrive1 bs=512 count=34 
dd if=secondary-backup.img of=\\.\\physicaldrive1 bs=512 count=33 seek=4395431903

(notice how the “if” and “of” arguments have swapped, and the word “skip” has changed to “seek”.)

Humble Indie Bundle md5sums

Fri, 07 May 2010 00:00:00 +0000

I just purchased the Humble Indie Bundle, 5 games which run on Windows, Linux and Mac. They kindly offered md5 hashes, however they weren’t in a convenient format, so I created a md5sums file.

187287db6226ef6a306a4cd0cfa8dd45 *Aquaria111.2008.12.12.exe
8b24ddeeb553e028bbd501102f891cc2 *aquaria-lnx-humble-bundle.mojo.run
336d89eb431e0b00535a2b50146c579d *WorldOfGooSetup.1.30.exe
f5afa40893d0fbcc37885191404f6d8c *WorldOfGooSetup.1.41.tar.gz
b373132f6e78665f1076752b038a8218 *gish153-1.tar.gz
94e82d53e2104914f19ec97f7cf5da42 *gish1531.zip
41ea2e41fe42c40b5ad017ae1afb45c7 *lugaru-full-linux-x86-1.0c.bin
d40d4076ed3ffa2a9bc9913202c55a06 *lugaru-windows.zip
56060bb550e0cd7e90bc7bc60d7c4c23 *penumbra_overture_1.1.exe
539567b590cf53e65a8e8f1b2bc728b3 *penumbra_overture_1.1.sh

and for those that prefer sha1:

ec8fd66bb44c6f67eab40f73e57838dcb5e2f4dd *Aquaria111.2008.12.12.exe
48267967b3a57fb406fa6545261f8b758edb8fb5 *aquaria-lnx-humble-bundle.mojo.run
70fad514746d6fc482fc3d681a7e9d498374bdad *WorldOfGooSetup.1.30.exe
4f7202a4ac17dd1665a1ab7f90153e5b813e16f1 *WorldOfGooSetup.1.41.tar.gz
27b862939b6a01c29b1b146ed1307c0027217855 *gish153-1.tar.gz
7a1341822d4d4e0010cc1e8cce68da6bb02ea904 *gish1531.zip
2349bdea3d595c3aaedca0810229d12f96323874 *lugaru-full-linux-x86-1.0c.bin
73f09414e0cabd371802eea1b9c75c76522c5934 *lugaru-windows.zip
557c6988eda16c6269d09a35031fd1754e042c02 *penumbra_overture_1.1.exe
e115f7cfcf9710d7aa5b2a5a9186b3736bb55cf2 *penumbra_overture_1.1.sh

Note, these were correct and validated on the 8th of May, however, if any of the files have been updated the hashes might no longer much. Feel free to contact me to tell me if this happens.

Threadnetperf v1.0

Mon, 01 Jun 2009 00:00:00 +0000

Today I am proud to announce that I’m releasing a tool I’ve used privately for well over a year now. It is a multi-threaded network testing tool which gives the user great control over how many connections, threads and processes the tool uses, and which CPU cores the threads/processes are allowed to execute on.

Currently the tool runs on Linux, FreeBSD, and Windows, and has performance as good as (if not better) than other similar tools, such as netperf, iperf, but with the added flexibility. Additionally, Threadnetperf has been completely written from scratch in C, using modern programming techniques and APIs (such as ePoll on Linux) which ensures the program runs as far as possible with minimal overheads.

You can download the source from GitHub, which contains instructions on how to build it.

So that you get an extra idea of it’s features here is the README.txt

****** Threadnetperf ******
A multi-threaded network benchmark tool
 by  Andrew Brampton (2007-2009)
 and Mathew Faulkner (2007-2009)

Threadnetperf is a highly customisable high performance network benchmarking
tool. The key difference from previous tools is that the user is able to control
how many threads threadnetperf uses, as well as which cores these threads are
pinned to. This tool was useful in our research for measuring the effect of
sending or receiving from one core, and having the OS network stack run on
another core. Additionally the tool can be configured to use an unlimited number
of connections, threads or processes, which scale considerably well due to the
use of the epoll API.

Threadnetperf has previously worked on Windows, Linux, and FreeBSD, however, we
do not test it very rigorously so one of these platforms may be broken. All
patches are welcome.

To build on a *nix style system just use the simple "Makefile" by executing the
command "make" or "gmake". To build on Windows just use the provided
threadnetperf.2008.sln file.

All the commands of Threadnetperf are documented within the application, but a
quick overview is here:

Usage: threadnetperf [options] tests
Usage: threadnetperf -D [options]
       -c level,interval   Confidence level, must be 95 or 99
       -D         Use daemon mode (wait for incoming tests)
       -d time    Set duration to run the test for
       -e         Eat the data (i.e. dirty it)
       -H host    Set the remote host(and port) to connect to
       -h         Display this help
       -i min,max Set the minimum and maximum iterations
       -m [t,p]   What programming model to use, [thread or process]
       -n         Disable Nagle's algorithm (e.g no delay)
       -p port    Set the port number for the first server thread to use
       -s size    Set the send/recv size
       -T         Timestamp packets, and measure latency (only on *nix)
       -t         Use TCP
       -r         Packets per second rate (default: ~0)
       -u         Use UDP
       -v         Verbose
       -V         Display version only

       tests      Combination of cores and clients
       tests      Core numbers are masks, for example 1 is core 0, 3 is core 0
and core 1
               N{c-s}   N connections
                        c client cores mask
                        s server cores mask

You can run tests locally, or across two machines. Here are some examples:

    threadnetperf -n -s 10000 1{1-1}

Will run a local TCP test, with Nagle's algorithm disabled, a send size of
10,000, and 1 TCP connection between cores 0 and 0.

    threadnetperf 10{1-1} 10{2-2} 10{4-4}

Will again run a local TCP test, however this time 10 connection from core 0 to
core 0, 10 connections from core 1 to core 1, and 10 connections from core 2 to
core 2

To run across a network just start the threadnetperf daemon on one of the
machines like so:

     threadnetperf -D

and on the other machine execute:

     threadnetperf -H serverIP 1{1-2}

which will conduct a TCP test from the client to the server, with one thread
executing on client's core 0, and the server's core 1.

There is no need to give the threadnetperf daemon any options, as the options
configured on the server will be passed over the network. This make running many
tests very easy as the daemon will continue to wait for new tests until it is
told otherwise.

If you find this tool useful, or you have any suggestions for improvements then
please contact us.