FreeBSD on bramp.net

Installing packages into your user directory

Thu, 14 Jan 2010 00:00:00 +0000

Today I had the need to install some FreeBSD and Ubuntu packages inside my home directory because I did not have root permissions to install them.

It was quite simple to install the packages on FreeBSD

pkg_add -rRP /home/bramp nano

Where nano is the name of the package I wanted.

I should point out you don’t need to be root to do this!, but the installed packages only work for you, and in future could cause you hassle if the base OS is updated.

On Ubuntu I however failed :( I tried the “–root” option and did some Googleing but I didn’t find a solution. Unitl then I’ll just install from source.

Updating…/-\|/-|

Sat, 19 Dec 2009 00:00:00 +0000

Do people other than me find updating their servers to be relaxing? At every opportunity I run “apt-get update”. Today I’m updating 7 different machines simultaneously.

My Debian machines I simply issue the following commands:

apt-get update
apt-get upgrade

My FreeBSD machines:

portsnap fetch update
portmanager -u

I think the Debian process is easier (and clearly the fastest) but it obviously has to do less work because I’m downloading precompiled code. Portmanager is just as easy but seems to randomly ask me questions through the process whereas apt-get seems to get them out of the way early on. Either way I find this all very easy and relaxing… until I have to reboot the machines and find something is broken :/

Useful scripts/configurations

Sat, 28 Nov 2009 00:00:00 +0000

Make git colourful

git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto

Make nano colourful

cp /usr/local/share/examples/nano/nanorc.sample ~/.nanorc
# or
zcat /usr/share/doc/nano/examples/nanorc.sample.gz >~/.nanorc

# then
nano ~/.nanorc

To trim trailing whitespace from *.cc on Linux (taken from this blog):

find . -name '*.cc' -exec sed -i {} -e 's/[ \t]*$//' ';'

and on BSDs:

find . -name '*.cc' -exec sed -i '' -e 's/\ *$//' {} ';'

Linux style ‘ls’ colours on FreeBSD (taken from here):

export CLICOLOR="YES"
export LSCOLORS="ExGxFxdxCxDxDxhbadExEx"

To ensure the http_proxy environmental variable is passed to sudo. Edit the sudoers file by running visudo. Then add the following.

Defaults env_keep += "ftp_proxy http_proxy https_proxy"

Bash autocompletion on FreeBSD:

cd  /usr/ports/shells/bash-completion
sudo make install clean

Edit ~/.bashrc and add

if [ -f /usr/local/etc/bash_completion ]; then
    . /usr/local/etc/bash_completion
fi

Make svn output colorful
Edit ~/.subversion/config

[helpers]
diff-cmd = /usr/bin/colordiff

Create diffs with function names and ignoring whitespace

svn diff -x -uwp

To be continued…

FreeBSD Software Watchdog

Fri, 16 Oct 2009 00:00:00 +0000

Recently I’ve been doing some kernel hacking and I managed to deadlock my system. The system still responded to pings but the terminal was unresponsive, and it needed a hard reboot to fix. I would have loved to drop into a debugger or panic’ed the kernel to get a suitable back-trace.

I found that FreeBSD has a software watchdog feature, that will panic the kernel if a problem like this occurs. The watchdog does not seem to be built into the kernel by default, and I found the documentation a bit lacking. So here is what I did:

Firstly, build and install a custom kernel which includes: options SW_WATCHDOG

Secondly, edit the /etc/rc.conf file and add a line: watchdogd_enable="YES"

When you next reboot, the watchdogd daemon will run, enabling the watchdog feature. Every second the watchdogd will reset a timer within the kernel. If after 16 seconds the timer has not been reset, the kernel will print out some interrupt information, and panic.

Watchdogd seems to have some useful features, for example, you can configure it to execute a specific command every second, and if that command fails the timer will not be reset. The configuration option I use is: watchdogd_flags="-e /bin/ps"

Read WATCHDOGD(8) for more information.

memcmp broken in the FreeBSD kernel

Mon, 10 Aug 2009 00:00:00 +0000

I’ve spent a day tracking down a bug in a FreeBSD kernel module I’m developing, and to my surprise the bug was due to memcmp being broken! For those that don’t know, memcmp is used to compare two byte strings and returns 0 if they are identical, a negative number of the first string is less than the second, and a positive number of the first string is greater than the second.

However, the implementation of memcmp in the FreeBSD kernel looks like this:

static __inline int
memcmp(const void *b1, const void *b2, size_t len) {
    return (bcmp(b1, b2, len));
}

The problem with this is that bcmp is defined to return 0 if the strings are identical, otherwise returns a non-zero integer. This is not the same as memcmp, and you would only notice this if you are testing the signedness of the return value. I suspect this has not been noticed because traditionally FreeBSD has favoured bcmp, and in the few cases it does uses memcmp it only compares it with zero.

There is some good news, obrien@ noticed this problem in September 2008 and commited a patch (svn r183299). However, it looks like this fix won’t be included until FreeBSD 8.0. In the mean time I’m implementing a minor hack to fix this.

int my_memcmp(const void *s1, const void *s2, size_t n) {
	if (n != 0) {
		const unsigned char *p1 = s1, *p2 = s2;

		do {
			if (*p1++ != *p2++)
				return (*--p1 - *--p2);
		} while (--n != 0);
	}
	return (0);
}
#define memcmp my_memcmp

Threadnetperf v1.0

Mon, 01 Jun 2009 00:00:00 +0000

Today I am proud to announce that I’m releasing a tool I’ve used privately for well over a year now. It is a multi-threaded network testing tool which gives the user great control over how many connections, threads and processes the tool uses, and which CPU cores the threads/processes are allowed to execute on.

Currently the tool runs on Linux, FreeBSD, and Windows, and has performance as good as (if not better) than other similar tools, such as netperf, iperf, but with the added flexibility. Additionally, Threadnetperf has been completely written from scratch in C, using modern programming techniques and APIs (such as ePoll on Linux) which ensures the program runs as far as possible with minimal overheads.

You can download the source from GitHub, which contains instructions on how to build it.

So that you get an extra idea of it’s features here is the README.txt

****** Threadnetperf ******
A multi-threaded network benchmark tool
 by  Andrew Brampton (2007-2009)
 and Mathew Faulkner (2007-2009)

Threadnetperf is a highly customisable high performance network benchmarking
tool. The key difference from previous tools is that the user is able to control
how many threads threadnetperf uses, as well as which cores these threads are
pinned to. This tool was useful in our research for measuring the effect of
sending or receiving from one core, and having the OS network stack run on
another core. Additionally the tool can be configured to use an unlimited number
of connections, threads or processes, which scale considerably well due to the
use of the epoll API.

Threadnetperf has previously worked on Windows, Linux, and FreeBSD, however, we
do not test it very rigorously so one of these platforms may be broken. All
patches are welcome.

To build on a *nix style system just use the simple "Makefile" by executing the
command "make" or "gmake". To build on Windows just use the provided
threadnetperf.2008.sln file.

All the commands of Threadnetperf are documented within the application, but a
quick overview is here:

Usage: threadnetperf [options] tests
Usage: threadnetperf -D [options]
       -c level,interval   Confidence level, must be 95 or 99
       -D         Use daemon mode (wait for incoming tests)
       -d time    Set duration to run the test for
       -e         Eat the data (i.e. dirty it)
       -H host    Set the remote host(and port) to connect to
       -h         Display this help
       -i min,max Set the minimum and maximum iterations
       -m [t,p]   What programming model to use, [thread or process]
       -n         Disable Nagle's algorithm (e.g no delay)
       -p port    Set the port number for the first server thread to use
       -s size    Set the send/recv size
       -T         Timestamp packets, and measure latency (only on *nix)
       -t         Use TCP
       -r         Packets per second rate (default: ~0)
       -u         Use UDP
       -v         Verbose
       -V         Display version only

       tests      Combination of cores and clients
       tests      Core numbers are masks, for example 1 is core 0, 3 is core 0
and core 1
               N{c-s}   N connections
                        c client cores mask
                        s server cores mask

You can run tests locally, or across two machines. Here are some examples:

    threadnetperf -n -s 10000 1{1-1}

Will run a local TCP test, with Nagle's algorithm disabled, a send size of
10,000, and 1 TCP connection between cores 0 and 0.

    threadnetperf 10{1-1} 10{2-2} 10{4-4}

Will again run a local TCP test, however this time 10 connection from core 0 to
core 0, 10 connections from core 1 to core 1, and 10 connections from core 2 to
core 2

To run across a network just start the threadnetperf daemon on one of the
machines like so:

     threadnetperf -D

and on the other machine execute:

     threadnetperf -H serverIP 1{1-2}

which will conduct a TCP test from the client to the server, with one thread
executing on client's core 0, and the server's core 1.

There is no need to give the threadnetperf daemon any options, as the options
configured on the server will be passed over the network. This make running many
tests very easy as the daemon will continue to wait for new tests until it is
told otherwise.

If you find this tool useful, or you have any suggestions for improvements then
please contact us.

Compiler defined symbols

Fri, 06 Mar 2009 00:00:00 +0000

I found this neat little command to list all the GCC compiler defined symbols

gcc -dM -E - > /dev/null

This can help if you aren’t quite sure which defines are used on your systems. For example my FreeBSD 7.1 machine outputs this:

#define __DBL_MIN_EXP__ (-1021)
#define __FLT_MIN__ 1.17549435e-38F
#define __DEC64_DEN__ 0.000000000000001E-383DD
#define __CHAR_BIT__ 8
#define __WCHAR_MAX__ 2147483647
#define __DBL_DENORM_MIN__ 4.9406564584124654e-324
#define __FLT_EVAL_METHOD__ 0
#define __DBL_MIN_10_EXP__ (-307)
#define __FINITE_MATH_ONLY__ 0
#define __DEC64_MAX_EXP__ 384
#define __SHRT_MAX__ 32767
#define __LDBL_MAX__ 1.18973149535723176502e+4932L
#define __UINTMAX_TYPE__ long unsigned int
#define __DEC32_EPSILON__ 1E-6DF
#define __unix 1
#define __SCHAR_MAX__ 127
#define __USER_LABEL_PREFIX__
#define __STDC_HOSTED__ 1
#define __DEC64_MIN_EXP__ (-383)
#define __DBL_DIG__ 15
#define __FLT_EPSILON__ 1.19209290e-7F
#define __LDBL_MIN__ 3.36210314311209350626e-4932L
#define __DEC32_MAX__ 9.999999E96DF
#define __unix__ 1
#define __DECIMAL_DIG__ 21
#define __LDBL_HAS_QUIET_NAN__ 1
#define __GNUC__ 4
#define __MMX__ 1
#define __FLT_HAS_DENORM__ 1
#define __FreeBSD_cc_version 700003
#define __DBL_MAX__ 1.7976931348623157e+308
#define __DBL_HAS_INFINITY__ 1
#define __DEC32_MIN_EXP__ (-95)
#define __LDBL_HAS_DENORM__ 1
#define __DEC32_MIN__ 1E-95DF
#define __DBL_MAX_EXP__ 1024
#define __DEC128_EPSILON__ 1E-33DL
#define __SSE2_MATH__ 1
#define __amd64 1
#define __LONG_LONG_MAX__ 9223372036854775807LL
#define __GXX_ABI_VERSION 1002
#define __FLT_MIN_EXP__ (-125)
#define __x86_64 1
#define __DBL_MIN__ 2.2250738585072014e-308
#define __LP64__ 1
#define __DBL_HAS_QUIET_NAN__ 1
#define __DEC128_MIN__ 1E-6143DL
#define __REGISTER_PREFIX__
#define __DBL_HAS_DENORM__ 1
#define __NO_INLINE__ 1
#define __DEC_EVAL_METHOD__ 2
#define __DEC128_MAX__ 9.999999999999999999999999999999999E6144DL
#define __FLT_MANT_DIG__ 24
#define __VERSION__ "4.2.1 20070719  [FreeBSD]"
#define __DEC64_EPSILON__ 1E-15DD
#define __DEC128_MIN_EXP__ (-6143)
#define unix 1
#define __SIZE_TYPE__ long unsigned int
#define __DEC32_DEN__ 0.000001E-95DF
#define __ELF__ 1
#define __FLT_RADIX__ 2
#define __LDBL_EPSILON__ 1.08420217248550443401e-19L
#define __FreeBSD__ 7
#define __SSE_MATH__ 1
#define __k8 1
#define __LDBL_DIG__ 18
#define __KPRINTF_ATTRIBUTE__ 1
#define __x86_64__ 1
#define __FLT_HAS_QUIET_NAN__ 1
#define __FLT_MAX_10_EXP__ 38
#define __LONG_MAX__ 9223372036854775807L
#define __FLT_HAS_INFINITY__ 1
#define __DEC64_MAX__ 9.999999999999999E384DD
#define __DEC64_MANT_DIG__ 16
#define __DEC32_MAX_EXP__ 96
#define __DEC128_DEN__ 0.000000000000000000000000000000001E-6143DL
#define __LDBL_MANT_DIG__ 64
#define _LONGLONG 1
#define __DEC32_MANT_DIG__ 7
#define __k8__ 1
#define __WCHAR_TYPE__ int
#define __FLT_DIG__ 6
#define __INT_MAX__ 2147483647
#define __FLT_MAX_EXP__ 128
#define __DBL_MANT_DIG__ 53
#define __DEC64_MIN__ 1E-383DD
#define __WINT_TYPE__ unsigned int
#define __SSE__ 1
#define __LDBL_MIN_EXP__ (-16381)
#define __amd64__ 1
#define __LDBL_MAX_EXP__ 16384
#define __LDBL_MAX_10_EXP__ 4932
#define __DBL_EPSILON__ 2.2204460492503131e-16
#define _LP64 1
#define __GNUC_PATCHLEVEL__ 1
#define __LDBL_HAS_INFINITY__ 1
#define __INTMAX_MAX__ 9223372036854775807L
#define __FLT_DENORM_MIN__ 1.40129846e-45F
#define __FLT_MAX__ 3.40282347e+38F
#define __SSE2__ 1
#define __FLT_MIN_10_EXP__ (-37)
#define __INTMAX_TYPE__ long int
#define __DEC128_MAX_EXP__ 6144
#define __GNUC_MINOR__ 2
#define __DBL_MAX_10_EXP__ 308
#define __LDBL_DENORM_MIN__ 3.64519953188247460253e-4951L
#define __STDC__ 1
#define __PTRDIFF_TYPE__ long int
#define __DEC128_MANT_DIG__ 34
#define __LDBL_MIN_10_EXP__ (-4931)
#define __GNUC_GNU_INLINE__ 1

and my Debian Linux machine outputs this:

#define __DBL_MIN_EXP__ (-1021)
#define __FLT_MIN__ 1.17549435e-38F
#define __DEC64_DEN__ 0.000000000000001E-383DD
#define __CHAR_BIT__ 8
#define __WCHAR_MAX__ 2147483647
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1
#define __DBL_DENORM_MIN__ 4.9406564584124654e-324
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 1
#define __FLT_EVAL_METHOD__ 0
#define __unix__ 1
#define __x86_64 1
#define __DBL_MIN_10_EXP__ (-307)
#define __FINITE_MATH_ONLY__ 0
#define __GNUC_PATCHLEVEL__ 3
#define __DEC64_MAX_EXP__ 384
#define __SHRT_MAX__ 32767
#define __LDBL_MAX__ 1.18973149535723176502e+4932L
#define __UINTMAX_TYPE__ long unsigned int
#define __linux 1
#define __DEC32_EPSILON__ 1E-6DF
#define __unix 1
#define __LDBL_MAX_EXP__ 16384
#define __linux__ 1
#define __SCHAR_MAX__ 127
#define __DBL_DIG__ 15
#define __SIZEOF_INT__ 4
#define __SIZEOF_POINTER__ 8
#define __USER_LABEL_PREFIX__
#define __STDC_HOSTED__ 1
#define __LDBL_HAS_INFINITY__ 1
#define __FLT_EPSILON__ 1.19209290e-7F
#define __LDBL_MIN__ 3.36210314311209350626e-4932L
#define __DEC32_MAX__ 9.999999E96DF
#define __SIZEOF_LONG__ 8
#define __DECIMAL_DIG__ 21
#define __gnu_linux__ 1
#define __LDBL_HAS_QUIET_NAN__ 1
#define __GNUC__ 4
#define __MMX__ 1
#define __FLT_HAS_DENORM__ 1
#define __SIZEOF_LONG_DOUBLE__ 16
#define __DBL_MAX__ 1.7976931348623157e+308
#define __DBL_HAS_INFINITY__ 1
#define __DEC32_MIN_EXP__ (-95)
#define __LDBL_HAS_DENORM__ 1
#define __DEC128_MAX__ 9.999999999999999999999999999999999E6144DL
#define __DEC32_MIN__ 1E-95DF
#define __DBL_MAX_EXP__ 1024
#define __DEC128_EPSILON__ 1E-33DL
#define __SSE2_MATH__ 1
#define __amd64 1
#define __LONG_LONG_MAX__ 9223372036854775807LL
#define __SIZEOF_SIZE_T__ 8
#define __SIZEOF_WINT_T__ 4
#define __GXX_ABI_VERSION 1002
#define __FLT_MIN_EXP__ (-125)
#define __DBL_MIN__ 2.2250738585072014e-308
#define __LP64__ 1
#define __DECIMAL_BID_FORMAT__ 1
#define __DEC128_MIN__ 1E-6143DL
#define __REGISTER_PREFIX__
#define __DBL_HAS_DENORM__ 1
#define __NO_INLINE__ 1
#define __FLT_MANT_DIG__ 24
#define __VERSION__ "4.3.3"
#define __DEC64_EPSILON__ 1E-15DD
#define __DEC128_MIN_EXP__ (-6143)
#define unix 1
#define __SIZE_TYPE__ long unsigned int
#define __DEC32_DEN__ 0.000001E-95DF
#define __ELF__ 1
#define __FLT_RADIX__ 2
#define __LDBL_EPSILON__ 1.08420217248550443401e-19L
#define __SSE_MATH__ 1
#define __k8 1
#define __SIZEOF_PTRDIFF_T__ 8
#define __x86_64__ 1
#define __FLT_HAS_QUIET_NAN__ 1
#define __FLT_MAX_10_EXP__ 38
#define __LONG_MAX__ 9223372036854775807L
#define __FLT_HAS_INFINITY__ 1
#define __DEC64_MAX__ 9.999999999999999E384DD
#define __DEC64_MANT_DIG__ 16
#define __DEC32_MAX_EXP__ 96
#define linux 1
#define __DEC128_DEN__ 0.000000000000000000000000000000001E-6143DL
#define __SSE2__ 1
#define __LDBL_MANT_DIG__ 64
#define __DBL_HAS_QUIET_NAN__ 1
#define __k8__ 1
#define __WCHAR_TYPE__ int
#define __SIZEOF_FLOAT__ 4
#define __DEC64_MIN_EXP__ (-383)
#define __FLT_DIG__ 6
#define __INT_MAX__ 2147483647
#define __amd64__ 1
#define __FLT_MAX_EXP__ 128
#define __DBL_MANT_DIG__ 53
#define __DEC64_MIN__ 1E-383DD
#define __WINT_TYPE__ unsigned int
#define __SIZEOF_SHORT__ 2
#define __SSE__ 1
#define __LDBL_MIN_EXP__ (-16381)
#define __LDBL_MAX_10_EXP__ 4932
#define __DBL_EPSILON__ 2.2204460492503131e-16
#define _LP64 1
#define __SIZEOF_WCHAR_T__ 4
#define __DEC_EVAL_METHOD__ 2
#define __INTMAX_MAX__ 9223372036854775807L
#define __FLT_DENORM_MIN__ 1.40129846e-45F
#define __FLT_MAX__ 3.40282347e+38F
#define __SIZEOF_DOUBLE__ 8
#define __FLT_MIN_10_EXP__ (-37)
#define __INTMAX_TYPE__ long int
#define __DEC128_MAX_EXP__ 6144
#define __GNUC_MINOR__ 3
#define __DEC32_MANT_DIG__ 7
#define __DBL_MAX_10_EXP__ 308
#define __LDBL_DENORM_MIN__ 3.64519953188247460253e-4951L
#define __STDC__ 1
#define __PTRDIFF_TYPE__ long int
#define __DEC128_MANT_DIG__ 34
#define __LDBL_MIN_10_EXP__ (-4931)
#define __SIZEOF_LONG_LONG__ 8
#define __LDBL_DIG__ 18
#define __GNUC_GNU_INLINE__ 1

Autoload symbols for FreeBSD kernel module

Sun, 11 Jan 2009 00:00:00 +0000

When debugging FreeBSD kernel modules with GDB, you have to tell GDB the correct symbols for the module, and the location the module is loaded in RAM. This is helpfully explained in the FreeBSD Developers’ Handbook. First you must load the module, then run kldstat, note down the address the module is loaded at, and finally execute a command in GDB that looks like the following.

add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0

However, I find this process tedious, so instead I wrote a quick python script which can be used with an experimental gdb built with python scripting support.

So here is the script:

import gdb
class FreeBSD_ReloadModuleSymbols (gdb.Command):
	"Reloads the symbol files for all loaded kernel modules"

	def __init__ (self):
		super (FreeBSD_ReloadModuleSymbols, self).__init__ ("reload-freebsd-module-symbols",
			gdb.COMMAND_FILES,
			gdb.COMPLETE_NONE)

	def invoke (self, arg, from_tty):
		link = gdb.parse_and_eval("linker_files->tqh_first")
		while link != 0:
			print link['filename'].string()
			if link['filename'].string() != "kernel":
				gdb.execute( "add-symbol-file " + 
					link['pathname'].string() + " " +  
					str(link['address'].address()) )
			link = link['link']['tqe_next']

FreeBSD_ReloadModuleSymbols ()

You load this by running the following command in GDB:

source freebsd_load_modules.py

Then the command reload-freebsd-module-symbols is magically added to GDB. Running this command will parse the linker table inside the FreeBSD kernel, determine which modules are loaded, and attempt to load their symbols.

linuxoffsets for FreeBSD

Sun, 11 Jan 2009 00:00:00 +0000

VMWare 6 and above has this neat debugging functionality where you can attach gdb running on a host, to a guest running inside VMWare. This allows you to debug a running kernel, or on linux running processes.

However, VMWare seems to have coded a bit of a hack to allow gdb to understand what process/threads are inside the virtual machine. Now this hack involves using something called linuxoffsets, which provides the offset for certain fields in a linux kernel struct.

So I thought I could abuse these offsets, and hopefully make them work for FreeBSD. The Linux offsets are:

Linux Values	FreeBSD 7.1 Values	Notes
		Version number of linux kernel, ie 0x020609 (2.6.9)
task_struct->mm
task_struct->next_task (linux < 2.4.15)	struct proc->p_list->le_next	Pointer to next task struct
task_struct->tasks (linux >= 2.4.15)	0	Pointers to prev and next task struct
task_struct->comm	struct proc->p_comm	executable name
task_struct->pid	struct proc->p_pid	pid of the process
task_struct->thread
mm_struct->pgd
thread_struct->rsp0 (or thread_struct->esp0 32bit)	struct thread->td_sigstk->ss_sp
thread_struct->fs
0x2000 or sizeof(union thread_union->stack) (linux >= 2.6)
task_struct->group_leader (linux >= 2.6.11)	0
task_struct->thread_group (linux >= 2.6.11)	0
sizeof(struct task_struct->comm)	sizeof(struct proc->p_comm)	executable name’s max len

However, after some time I feel this isn’t going to be possible. Each value represents a offset into a Linux task_struct struct, however, nothing represents the location of the first task_struct in RAM. I suspect VMWare is figuring out the location via some other means. Since FreeBSD doesn’t have a task_struct it most likely won’t be able to find what it needs.

Regardless I’ve posted this entry in case someone wants to continue my work. However, not all is lost as my next solution involves playing with gdb-python.