Python on bramp.net

Intel ucode firmware version parser

Mon, 24 Jan 2011 00:00:00 +0000

Out of fun I wrote a simple Python script to pull the version number out of Intel’s ucode firmware, for example, the firmware used by my wifi driver. I needed this so I could see what version I was running versus a new version I had downloaded from Intel’s Linux Wireless site.

So here is the code if anyone finds it interesting:

and example of using it is:

$ ./ucode.py /lib/firmware/*.ucode
iwlwifi-1000-3.ucode    : ver 128.50.3.1
iwlwifi-3945-1.ucode    : ver 15.28.1.6
iwlwifi-3945-2.ucode    : ver 15.32.2.9
iwlwifi-4965-1.ucode    : ver 228.57.1.21
iwlwifi-4965-2.ucode    : ver 228.61.2.24
iwlwifi-5000-1.ucode    : ver 5.4.1.16
iwlwifi-5000-2.ucode    : ver 8.24.2.12
iwlwifi-5150-2.ucode    : ver 8.24.2.2
iwlwifi-6000-4.ucode    : ver 9.221.4.1
iwlwifi-6000g2a-5.ucode : ver 17.168.5.1 (6000g2a fw v17.168.5.1 build 33993)
iwlwifi-6000g2b-5.ucode : ver 17.168.5.1 (6000g2b fw v17.168.5.1 build 33993)
iwlwifi-6050-4.ucode    : ver 9.201.4.1
iwlwifi-6050-5.ucode    : ver 41.28.5.1 (6050 fw v41.28.5.1 build 33926)

UTF-8 Directory Listing

Thu, 23 Sep 2010 00:00:00 +0000

I had a need to create a directory listing with all the UTF-8 characters intact. This seems quite a chore on Windows, as doing anything via the shell seems to mangle the characters and show ???? instead of the real characters. For example, both the built in dir and Cygwin ls or find seemed affected. This turns out to be a limitation in the windows shell.

To solve this problem I wrote a bit of python to read the file names in full UTF-8 and output the results directly to a file (and not via a pipe, which would again be via the shell). The resulting very simple script is as follows:

import os
import codecs

log = codecs.open('listing', mode='w', encoding="utf-8")

for root, dirs, files in os.walk(u'.'):
	log.write(root + u"\n")

	for file in sorted(files):
		log.write(os.path.join(root, file) + u"\n")

log.close()

Persec python script

Tue, 31 Aug 2010 00:00:00 +0000

A while ago I wrote a python script that does a similar job to GNU’s watch command. You use it like so:

./persec.py [--interval=&lt;n&gt;] &lt;command&gt;

so for example

./persec.py ifconfig

Now in a similar way to watch, it executes the command every second, and highlights the differences between each execution. However, in addition to this it finds any numbers that have changed and works out the rate at which they are changing. So for example, ifconfig would typically output this:

usb0      Link encap:Ethernet  HWaddr 02:04:4b:00:d3:cf
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1017422291 errors:0 dropped:0 overruns:0 frame:0
          TX packets:549382406 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1910704266 (1.9 GB)  TX bytes:1834667124 (1.8 GB)

but now outputs something like:

usb0      Link encap:Ethernet  HWaddr 02:04:4b:00:d3:cf
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2001/s errors:0 dropped:0 overruns:0 frame:0
          TX packets:2002/s errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:168120/s (1.9 GB)  TX bytes:217144/s (1.8 GB)

Notice the per second (/s) values for RX/TX packets and RX/TX bytes. I have found this quite useful many times in the past, on commands such as:

./persec.py cat /proc/interrupts
./persec.py df
./persec.py ls -l somefile

Download version 1.1 or View on Github

Python close_fds issue

Tue, 11 May 2010 00:00:00 +0000

So I spent the better part of my evening trying to track down a bug, which turns out to be a “feature” of python.

I had just installed the GitPlugin for trac but I started to experience problems. When browsing the source inside trac it was taking over 30seconds to load the page and sometimes it would fail completely. A lot of searching didn’t help much, so I attempted to debug the problem myself. The first thing I noticed was Apache was taking 100% of the processor for a good 30seconds. I attached strace to it and saw something like this:

[pid 22682] close(43029)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43030)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43031)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43032)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43033)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43034)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43035)                = -1 EBADF (Bad file descriptor)
[pid 22682] close(43036)                = -1 EBADF (Bad file descriptor)

This obviously didn’t look good! After some tinkering I found the problem went away when I ran trac standalone, instead of using mod_python or fcgi. This turned out to be a bit of a red herring because I spent my time trying to figure out what was different between a standalone executable and one being run inside Apache.

After playing around with environment variables, I gave up and attempted to printf debug the trac git plugin. I found that the actual call to git was taking on the order of seconds, whereas calling it myself from the command took milliseconds. The line of code (in PyGIT.py) looked a bit like this:

p = Popen(self.__build_git_cmd(git_cmd, *cmd_args), stdin=None, stdout=PIPE,
			                                        stderr=PIPE, close_fds=True)

Now, when I removed the close_fds argument the problems went away! After some more digging I found this bug report which describes the behaviour of close_fds. Python will spin in a tight loop calling close for all possible valid fd number just incase it was previously used. WTF! You can see the python code here:

def _close_fds(self, but):
    os.closerange(3, but)
    os.closerange(but + 1, MAXFD)

So the simple fix to this was to remove the close_fds, so that Python doesn’t stupidly spin calling close(). I suspect the reason I only noticed this when running inside Apache, is because Apache must have a larger MAXFD. Hopefully in the future Python will change this behaviour and find a more sensible way to close all file descriptors, especially when I read this bug report which advises changing close_fds default to true.

Follow HTTP Stream (with decompression)

Sun, 10 Jan 2010 00:00:00 +0000

I was using Wireshark to capture an exchange of HTTP packets, however, some of the HTTP responses were using “content-encoding: gzip”, which meant I couldn’t view them decompressed in the “Follow TCP Stream”. Wireshark does decompress them in Packet Details view, but it is hard to follow the full stream like this.

The solution was to write some Python which made use of the dpkt library. My code naively reassembles the TCP flow and then assumes traffic on port 80 is HTTP. Therefore there is much room for improvement, but here is the code anyway.

#!/usr/bin/env python
# Turns a pcap file with http gzip compressed data into plain text, making it
# easier to follow.

import dpkt

def tcp_flags(flags):
	ret = ''
	if flags & dpkt.tcp.TH_FIN:
		ret = ret + 'F'
	if flags & dpkt.tcp.TH_SYN:
		ret = ret + 'S'
	if flags & dpkt.tcp.TH_RST:
		ret = ret + 'R'
	if flags & dpkt.tcp.TH_PUSH:
		ret = ret + 'P'
	if flags & dpkt.tcp.TH_ACK:
		ret = ret + 'A'
	if flags & dpkt.tcp.TH_URG:
		ret = ret + 'U'
	if flags & dpkt.tcp.TH_ECE:
		ret = ret + 'E'
	if flags & dpkt.tcp.TH_CWR:
		ret = ret + 'C'

	return ret

def parse_http_stream(stream):
	while len(stream) &gt; 0:
		if stream[:4] == 'HTTP':
			http = dpkt.http.Response(stream)
			print http.status
		else:
			http = dpkt.http.Request(stream)
			print http.method, http.uri
		stream = stream[len(http):]

def parse_pcap_file(filename):
	# Open the pcap file
	f = open('market.pcap', 'rb')
	pcap = dpkt.pcap.Reader(f)
	
	# I need to reassmble the TCP flows before decoding the HTTP
	conn = dict() # Connections with current buffer
	for ts, buf in pcap:
		eth = dpkt.ethernet.Ethernet(buf)
		if eth.type != dpkt.ethernet.ETH_TYPE_IP:
			continue
	
		ip = eth.data
		if ip.p != dpkt.ip.IP_PROTO_TCP:
			continue
	
		tcp = ip.data
	
		tupl = (ip.src, ip.dst, tcp.sport, tcp.dport)
		#print tupl, tcp_flags(tcp.flags)
	
		# Ensure these are in order! TODO change to a defaultdict
		if tupl in conn:
			conn[ tupl ] = conn[ tupl ] + tcp.data
		else:
			conn[ tupl ] = tcp.data
	
		# TODO Check if it is a FIN, if so end the connection
	
		# Try and parse what we have
		try:
			stream = conn[ tupl ]
			if stream[:4] == 'HTTP':
				http = dpkt.http.Response(stream)
				#print http.status
			else:
				http = dpkt.http.Request(stream)
				#print http.method, http.uri
	
			print http
			print

			# If we reached this part an exception hasn't been thrown
			stream = stream[len(http):]
			if len(stream) == 0:
				del conn[ tupl ]
			else:
				conn[ tupl ] = stream
		except dpkt.UnpackError:
			pass

	f.close()

if __name__ == '__main__':
	import sys
	if len(sys.argv) &lt;= 1:
		print "%s <pcap filename>" % sys.argv[0]
		sys.exit(2)

	parse_pcap_file(sys.argv[1])

Please note, I had to make a couple of changes to the dpkt library, which I have submitted back for review. Those changes can be found in the following patches 1 2 3. I will update this code if/when the patches get accepted.

Autoload symbols for FreeBSD kernel module

Sun, 11 Jan 2009 00:00:00 +0000

When debugging FreeBSD kernel modules with GDB, you have to tell GDB the correct symbols for the module, and the location the module is loaded in RAM. This is helpfully explained in the FreeBSD Developers’ Handbook. First you must load the module, then run kldstat, note down the address the module is loaded at, and finally execute a command in GDB that looks like the following.

add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0

However, I find this process tedious, so instead I wrote a quick python script which can be used with an experimental gdb built with python scripting support.

So here is the script:

import gdb
class FreeBSD_ReloadModuleSymbols (gdb.Command):
	"Reloads the symbol files for all loaded kernel modules"

	def __init__ (self):
		super (FreeBSD_ReloadModuleSymbols, self).__init__ ("reload-freebsd-module-symbols",
			gdb.COMMAND_FILES,
			gdb.COMPLETE_NONE)

	def invoke (self, arg, from_tty):
		link = gdb.parse_and_eval("linker_files->tqh_first")
		while link != 0:
			print link['filename'].string()
			if link['filename'].string() != "kernel":
				gdb.execute( "add-symbol-file " + 
					link['pathname'].string() + " " +  
					str(link['address'].address()) )
			link = link['link']['tqe_next']

FreeBSD_ReloadModuleSymbols ()

You load this by running the following command in GDB:

source freebsd_load_modules.py

Then the command reload-freebsd-module-symbols is magically added to GDB. Running this command will parse the linker table inside the FreeBSD kernel, determine which modules are loaded, and attempt to load their symbols.