I’ve spent a day tracking down a bug in a FreeBSD kernel module I’m developing, and to my surprise the bug was due to memcmp being broken! For those that don’t know, memcmp is used to compare two byte strings and returns 0 if they are identical, a negative number of the first string is less than the second, and a positive number of the first string is greater than the second.

However, the implementation of memcmp in the FreeBSD kernel looks like this:

static __inline int
memcmp(const void *b1, const void *b2, size_t len) {
    return (bcmp(b1, b2, len));
}

The problem with this is that bcmp is defined to return 0 if the strings are identical, otherwise returns a non-zero integer. This is not the same as memcmp, and you would only notice this if you are testing the signedness of the return value. I suspect this has not been noticed because traditionally FreeBSD has favoured bcmp, and in the few cases it does uses memcmp it only compares it with zero.

There is some good news, obrien@ noticed this problem in September 2008 and commited a patch (svn r183299). However, it looks like this fix won’t be included until FreeBSD 8.0. In the mean time I’m implementing a minor hack to fix this.

int my_memcmp(const void *s1, const void *s2, size_t n) {
	if (n != 0) {
		const unsigned char *p1 = s1, *p2 = s2;

		do {
			if (*p1++ != *p2++)
				return (*--p1 - *--p2);
		} while (--n != 0);
	}
	return (0);
}
#define memcmp my_memcmp

When debugging FreeBSD kernel modules with GDB, you have to tell GDB the correct symbols for the module, and the location the module is loaded in RAM. This is helpfully explained in the FreeBSD Developers’ Handbook. First you must load the module, then run kldstat, note down the address the module is loaded at, and finally execute a command in GDB that looks like the following.

add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0

However, I find this process tedious, so instead I wrote a quick python script which can be used with an experimental gdb built with python scripting support.

So here is the script:

import gdb
class FreeBSD_ReloadModuleSymbols (gdb.Command):
	"Reloads the symbol files for all loaded kernel modules"

	def __init__ (self):
		super (FreeBSD_ReloadModuleSymbols, self).__init__ ("reload-freebsd-module-symbols",
			gdb.COMMAND_FILES,
			gdb.COMPLETE_NONE)

	def invoke (self, arg, from_tty):
		link = gdb.parse_and_eval("linker_files->tqh_first")
		while link != 0:
			print link['filename'].string()
			if link['filename'].string() != "kernel":
				gdb.execute( "add-symbol-file " + 
					link['pathname'].string() + " " +  
					str(link['address'].address()) )
			link = link['link']['tqe_next']

FreeBSD_ReloadModuleSymbols ()

You load this by running the following command in GDB:

source freebsd_load_modules.py

Then the command reload-freebsd-module-symbols is magically added to GDB. Running this command will parse the linker table inside the FreeBSD kernel, determine which modules are loaded, and attempt to load their symbols.