dotCOMmie's Blog

I got myself one of the fancy shmancy netbooks. Due to a habit and some hardware issues I needed to compile a kernel. The problem here though is that it takes for ever to build a kernel on one of these things. No sweat I'll just build it on my desktop, it'll only take 5-10 minutes. But of course there is a catch. My desktop is 64bit and this new machine is an Atom CPU which only does 32bit.

The process for compiling a 32bit kernel on a 64bit machine is probably a lot easier if you don't compile it the Debian way. But this is not something I want to do, I like installing the kernels through the package manager and doing this type of cross compile using make-kpkg is not trivial. There are plenty of forum and email threads about people recommending to use chroot or virtual machines for this task, but that is such a chore to set up. So here is my recipe for cross compiling 32bit kernel on 64bit host without chroot / vm, the-debian-way.

1. Install 32bit tools (ia32-libs, lib32gcc1, lib32ncurses5, libc6-i386, util-linux, maybe some other ones)
2. Download & unpack your kernel sources
3. run "linux32 make menuconfig" and configure your kernel for your new machine
4. clean your build dirs "make-kpkg clean --cross-compile - --arch=i386" (only needed on consecutive compiles)
5. compile your kernel "nice -n 100 fakeroot linux32 make-kpkg --cross-compile - --arch=i386 --revision=05test kernel_image" for faster compilation on multi-CPU machines run "export CONCURRENCY_LEVEL=$((`cat /proc/cpuinfo |grep "^processor"|wc -l`*2))" first
6. At this point you have a 32bit kernel inside a package labeled for 64bit arch. We need to fix this, run "fakeroot deb-reversion -k bash ../linux-image-2.6.35.3_05test_amd64.deb". Open the file DEBIAN/control with vim/emacs and change "Architecture: amd64" to "Architecture: i386" exit the bash process with ctrl+d
7. That's it, now just transfer the re-generated deb to destination machine and install it.

Many if not all ideas for this process come from reading email threads the comments made by Goswin Von Brederlow were particularly helpful, thanks.

Every six months the internet lights up with stories that Canonical & Co has done the unthinkable they have increased the number following the word Ubuntu. In other words they have release a new version. This is a well understood concept to differentiate releases of software. As the version increases it is expected that new features are introduced and old bugs are removed (hopefully more are removed than added).

Versioning distributions and releasing the versions separately is a common practice, employed by most if not all distributions out there. Ubuntu has adopted the policy of releasing regularly and quite often. But there is a different approach it revolves around a concept I call "Versionless" where you do not have a hard release but instead let the changes trickle down. In the application world these releases are often called nightly builds. With distributions it is a little bit different.

First of all its worth noting that distributions are not like applications. Distributions are collection made up by applications and a kernel, the applications that are included are usually stable releases and so the biggest unpredictability comes from the combination and configuration there of. As a result one of the important roles for distro developers is to ensure that the combination of the many applications does not lead to adverse side effects. This is done in several ways, the general method is to mix all the applications in a pot, the so called pre-release and then test the combination. The testing is done by whole community, as users often install these pre-releases to see if they see any quirks through their regular use. When the pre-release becomes stable enough it is pushed out the door as a public release.

In an ideal world after this whole process all the major bugs and issues would have been resolved and users go on to re-install/update their distribution installations to the new release -- without any issues. The problem is that even if the tests passed with flying colors does not mean that on the user will not experience problems. The more complicated a configuration that a user has the more chances they will notice bugs as part of upgrade. This is particularly evident where there are multiple interacting systems. Doing a full upgrade of a distribution it is not always obvious what change in the update has caused this problem.

Versionless distributions are nothing new, they has been a staple of Debian for a while. In fact it is the process for testing package compatibility between release, but it is also a lot more. There are two Debian releases that follow this paradigm, Debian Testing and Debian Unstable. As applications are packaged they are added to Debian Unstable and after they fulfill certain criteria, IE they have spent some time in Unstable and have not had any critical bugs filed against them, they are then passed along to Debian Testing. Users are able to balance their needs between new features and stability by selecting the corresponding repository. As soon as the packages are added to the repositories the become immediately available to user for install/upgrade.

What it really comes down to is testing outside your environment is useful but it cannot be relied solely upon. And when upgrades are performed it is important to know what has changed and how to undo it. Keeping track of changes for 1000's of updates is nearly impossible. So update small and update often, use Debian. Good packages managers are your best friend, but only second to great package developers!

Apparently Linux does not have any monitor hotplugging support which is quite a pain. Every time you want to attach a monitor to laptop you have to reconfigure the display layout. This is a tad frustrating if you have to do this several times a day. And it doesn't help that KDE subsystems are a bit flaky when it comes to changing display configuration. I've had plasma crash a on me 1/3 times while performing this operation.

Long story short I got fed up with all of this and wrote the following 3 line script to automate the process and partially alleviate this head ache

#!/bin/bash
xrandr --output LVDS1 --auto --output VGA1 --auto
sleep 1
kquitapp plasma-desktop &> /dev/null
sleep 1
kwin --replace & &> /dev/null
sleep 1
kstart plasma-desktop &> /dev/null

You probably need to adjust the xrandr line to make it behave like you want but auto everything works quite well for me. Check man page for xrandr for details.

For further reading on monitor hot plugging I encourage you read launchpad bug #306735. Fortunately there are solutions for this problem, however they are on the other side of the pond.

Update: Added the kwin replace line to fix sporadic malfunction of kwin (disappearance of window decorations) during this whole operation.

I mentioned earlier that we netboot (PXE) our cluster. Before NFS-root begins, some things have to take place. Namely, the kernel needs to be served, IP assigned, DNS look-ups need to be made to figure out where servers are and so on. Primarily 3 protocols are in the mix at this time, TFTP, DHCP, DNS. We used to run 3 individual applications to handle all of this, they're all in their own right quite fine applications atftpd, Bind9, DHCP (from ISC). But it just becomes too much to look after, you have a config file for each of the daemons as well as databases with node information. Our configuration used MySQL and PHP to generate all the databases for these daemons. This way you would only have to maintain one central configuration. Which means you need to look after yet another daemon to make it all work. You add all of this together and it becomes one major headache.

Several months ago I had installed openWRT onto a router at home. While configuring openWRT I came across something called dnsmasq. By default, on openWRT, dnsmasq handles DNS and DHCP. I thought it was spiffy to merge the 2 services .. after all they are so often run together (on internal networks). The name stuck in my head as something to pay bit more attention to. Somewhere along the line I got some more experience with dnsmasq, and had discovered it also had TFTP support. Could it be possible what we use 4 daemons could be accomplished with just one?

So when the opportunity arose I dumped all node address information out of the MySQL database into a simple awk-parsable flat file. I wrote a short parsing script which took the central database and spit out a file dnsmasq.hosts (with name/IP pairs) and another dnsmasq.nodes (with MAC-address/name pairs). Finally I configured the master (static) dnsmasq.conf file to start all the services I needed (DNS, DHCP, TFTP), include the dnsmasq.hosts and dnsmasq.nodes files. Since the dnsmasq.nodes includes a category flag it is trivial to tell which group of nodes should use what TFTP images and what kind of DHCP leases they should be served.

Dnsmasq couldn't offer a more simple and intuitive configuration with 1/2 days work I was able to greatly improve upon on old system and make a lot more manageable. There is only one gripe I have with dnsmasq, I wish it would be possible to just have one configuration line per node that is have the name, IP, and mac address all in one line. If this was the case I wouldn't even need an awk script to make the config file (although it turned out to be handy because I also use the same file to generate a nodes list for torque). But its understandable since there are instances where you only want to run a DHCP server or just DNS server and so having DHCP and DNS information on one line wouldn't make much sense.

Scalability for dnsmasq is something to consider. Their website claims that it has been tested with installation of up to 1000 nodes, which might or might not be a problem. Depending on what type of configuration your building. I kind of wonder what happens at the 1000s of machines level. How will its performance degrade, and how does that compare to say the other TFTP/DHCP servers (BIND9 is know to work quite well with a lot of data).

Here are some configuration examples:

Master Flat file node database

#NODES file it needs to be processed by nodesFileGen
#nodeType nodeIndex nic# MACAddr

nfsServer 01 1
nfsServer 02 1

headNode 00 1 00:00:00:00:00:00

#Servers based on the supermicro p2400 hardware (white 1u supermicro box)
server_sm2400 miscServ 1 00:00:00:00:00:00
server_sm2400 miscServ 2 00:00:00:00:00:00

#dual 2.4ghz supermicro nodes
node2ghz 01 1 00:00:00:00:00:00
node2ghz 02 1 00:00:00:00:00:00
node2ghz 03 1 00:00:00:00:00:00
...[snip]...

#dual 3.4ghz dell nodes
node3ghz 01 1 00:00:00:00:00:00
node3ghz 02 1 00:00:00:00:00:00
node3ghz 03 1 00:00:00:00:00:00
...[snip]...

Flat File DB Parser script

#!/bin/bash

#intput sample
#type number nic# mac addr
#nodeName 07 1 00:00:00:00:00:00

#output sample
#ip hostname
#10.0.103.10 nodeName10
awk '
	/^headNode.*/ {printf("10.0.0.3 %s\n", $1)};				\
	/^server_sm2400.*/ {printf("10.0.3.%d %s\n", $3, $2)};			\
	/^nfsServer.*/ {printf("10.0.1.%d %s%02d\n", $2, $1, $2)};		\
	/^node2ghz.*/ {printf("10.0.100.%d %s%02d\n", $2, $1, $2)};		\
	/^node3ghz.*/ {printf("10.0.101.%d %s%02d\n", $2, $1, $2)};		\
	'									\
	~/data/nodes.db > /etc/dnsmasq.hosts

#output sample
#mac,netType,hostname,hostname
#00:00:00:00:00:00,net:nodeName,nodeName10,nodeName10
awk '
	/^headNode.*/ {printf("%s,net:%s,%s,%s\n", $4, $1, $1, $1)};			\
	/^server_sm2400.*/ {printf("%s,net:%s,%s,%s\n", $4, $1, $2, $2)};		\
	/^node2ghz.*/ {printf("%s,net:%s,%s%02d,%s%02d\n", $4, $1, $1, $2, $1, $2)};	\
	/^node3ghz.*/ {printf("%s,net:%s,%s%02d,%s%02d\n", $4, $1, $1, $2, $1, $2)};	\
	'										\
	~/data/nodes.db > /etc/dnsmasq.nodes

#output sample
#hostname np=$CPUS type
#nodeName10 np=8 nodeName
awk '
	/^node2ghz.*/ {printf("%s%02d np=2 node2ghz\n", $1, $2)};		\
	/^node3ghz.*/ {printf("%s%02d np=2 node3ghz\n", $1, $2)};		\
	'									\
	~/data/nodes.db > /var/spool/torque/server_priv/nodes

#Lets reload dnsmasq now
killall -HUP dnsmasq

dnsmasq.conf

interface=eth0
dhcp-lease-max=500
domain=myCluster
enable-tftp
tftp-root=/srv/tftp
dhcp-option=3,10.0.0.1
addn-hosts=/etc/dnsmasq.hosts
dhcp-hostsfile=/etc/dnsmasq.nodes

dhcp-boot=net:misc,misc/pxelinux.0,nodeServer,10.0.0.2
dhcp-range=net:misc,10.0.200.0,10.0.200.255,12h

dhcp-boot=net:headNode,headNode/pxelinux.0,nodeServer,10.0.0.2
dhcp-range=net:headNode,10.0.0.3,10.0.0.3,12h

dhcp-boot=net:server_sm2400,server_sm2400/pxelinux.0,nodeServer,10.0.0.2
dhcp-range=net:server_sm2400,10.0.0.3,10.0.0.3,12h

dhcp-boot=net:node2ghz,node2ghz.cfg,nodeServer,10.0.0.2
dhcp-range=net:node2ghz,10.0.100.0,10.0.100.255,12h

dhcp-boot=net:node3ghz,node3ghz.cfg,nodeServer,10.0.0.2
dhcp-range=net:node3ghz,10.0.101.0,10.0.101.255,12h

You all probably know that the most important thing on any multi user system is a pretty MOTD. Between some other things in the past couple of weeks I decided to refresh the MOTDs for the galaxy and Seawulf clusters. I discovered 2 awesome applications while compiling the MOTD.

First is a jp2a, it takes a JPG and converts it to ASCII and it even supports color. I used this to render the milky way as part of the galaxy MOTD. While this tool is handy it needs some assistance, you should clean up and simplify the JPGs before you try to convert them.

The second tool is a must for any form of ASCII-art editing. Its called aewan (ace editor without a name). It makes editing a lot easier, it supports coloring, multiple layers, cut/paste/move, and more. Unfortunately it uses a weird format and does not have an import feature, so its PITA to import an already existing ASCII snippet -- cut and paste does work but it looses some information -- like color.

Aewan comes with a sister tool called aecat which 'cats' the native aewan format into either text (ANSI ASCII) or HTML. Below is some of my handy work. Because getting browsers to render text is PITA I decided to post the art-work as an image.
Galaxy MOTD:

Seawulf MOTD:

I also wrote a short cronjob which changes the MOTD every 5 min to reflect how many nodes are queued/free/down

One more resource I forgot to mention is the ascii generator. You give it a text string and it returns in a fancy looking logo.

Finally when making any MOTDs try to stick to the max width of 80 and heigh of 24. This way your art work won't be chopped even on ridiculously small terminals.