XenServer/USBIP
I've migrated my server at home from a single physical box to multiple XenServer VMs (or domUs, as they're called in Xen), and i am really satisfied with the performance and ease of use of the whole XenServer/XenCenter setup. One thing i didn't think about, though, was how i would get my Asterisk back to work, as it used a Fritz!Card PCI for incoming and outgoing calls on my landline (ISDN, obviously). The problem with XenServer is, that it can't forward/pass-through PCI devices to its VMs. I read a lot about the problems, and altough there are some mixed reports of people getting it to work with (non-Citrix, regular) Xen, it would definitely not work with my (Citrix) XenServer.
The next idea was to use an USB device, instead of a PCI card, because some people had success with forwarding that from the dom0 to a domU. Well, turns out that only works for USB storage devices (memory sticks, etc.).
The only solution left to try out was USB over IP. It consists of 3 kernel modules (one common for client and server "usbip_common_mod.ko", one for the server "usbip.ko", and one for the client "vhci-hcd.ko") and 3 binaries ("bind_driver" and "usbipd" on the server, "usbip" on the client). If you're running on kernel 2.6.16 to 2.6.25, you have to use the kernel modules in the archive, kernel 2.6.28 and newer already have the modules in the staging tree of the kernel sources.
ProTip: If you want to use something like a Fritz!Card USB with USB/IP in your XenServer domU, make sure you also read this: XenServer/FritzUSB so you don't have to compile your kernel twice.
Preparations
My dom0 runs the kernel "2.6.18-128.1.6.el5.xs5.5.0.496.1012xen", so i had to compile the modules from the archive. Citrix provides a so-called DDK (Driver Developmen Kit) for this purpose, so you don't have to install all the development tools and sources on your production server. The DDK is just a domU you import into your XenServer, boot up, and start compiling ... more or less.
To download the DDK, you have to register an account for My Citrix, which is free and painless. After login in you can go to the Citrix Essentials for XenServer 5.5 page and all the way down you find the "Driver Development Kit" download link. While we're here, also grab the "Driver Development Kit Source" ISO under Source Code, a couple of lines above.
When the download of the DDK ISO has finished you can either copy it to the dom0 via SCP, or put it on a domU and mount the folder via NFS, or anything else, as long as you can mount the ISO like this:
mount -o loop /path/to/XenServer-5.5.0-DDK.iso /mnt
then run the following command to import the DDK VM:
xe vm-import filename=/mnt/ddk/ova.xml
When that's done you can boot the DDK and login.
XenServer modules
The DDK includes only the kernel headers, but for compiling the modules you also need the kernel sources. If you carefully read my instructions above, you already have downloaded the DDK Source ISO - if not, scroll up and do it now!
We need the kernel sources from the DDK Source ISO, so mount it and install the RPM, copy the extracted archive back to /usr/src and extract it ... like this:
mount -o loop /path/to/XenServer-5.5.0-source-ddk.iso /mnt rpm -Uhv /mnt/kernel-dom0/kernel-2.6.18-128.1.6.el5.xs5.5.0.496.1012.src.rpm cd /usr/src tar jxvf redhat/SOURCES/linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012.tar.bz2 cp redhat/SOURCES/kernel-2.6.18-128.1.6.el5.xs5.5.0.496.1012-i686-xen.config linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012/.config cd linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012/ make prepare && make scripts
Download the usbip sources from Sourceforge and unpack them:
cd /usr/src wget http://downloads.sourceforge.net/project/usbip/usbip/0.1.7/usbip-0.1.7.tar.gz tar xvf usbip-0.1.7.tar.gz cd usbip-0.1.7/drivers/2.6.18/
I had to comment out some of the code in stub_rx.c, else it would fail with errors. Open stub_rx.c with your favorite text-editor, and go to line 374.
Comment out the lines from 374 to 399 like this:
/*
#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,21)
static inline int usb_endpoint_xfer_bulk(
[...]
[...]
[...]
return ((epd->bmAttributes & USB_ENDPOINT_XFERTYPE_MASK) ==
USB_ENDPOINT_XFER_ISOC);
}
#endif
*/
Alright, compile the modules! Run the following command, it shouldn't result in any warnings or errors:
make KSOURCE=/usr/src/linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012
XenServer tools
Now on to the binaries. You will have to install some additional packages first:
cd /usr/src/usbip-0.1.7/src yum --enablerepo=base install autoconf libtool libsysfs-devel glib2-devel ./autogen.sh ./configure --prefix=/usr make && make install
That's it, you're theoretically ready to serve USB over IP now. Hang on, you're still in a domU...
Finalizing the XenServer
Keep in mind that you've compiled the driver and tools on the DDK, but i guess you'd rather use them on the dom0, where you can actually plug in USB devices, right?
You need to create two folders on the dom0:
mkdir /lib/modules/2.6.18-128.1.6.el5.xs5.5.0.496.1012xen/kernel/extra mkdir /usr/share/usbip
Copy everything to the dom0:
scp /usr/bin/usbip* dom0:/usr/bin/ scp /usr/bin/bind_driver dom0:/usr/bin/ scp /usr/lib/libusbip.* dom0:/usr/lib/ scp /usr/share/usbip/usb.ids dom0:/usr/share/usbip/ scp /usr/src/usbip-0.1.7/2.6.18/*.ko dom0:/lib/modules/2.6.18-128.1.6.el5.xs5.5.0.496.1012xen/kernel/extra
WARNING: I had to learn this the hard way, but you don't want to have any devices plugged into any USB port when you load the modules. The first one loads fine, and the second one ... well, hard to tell if it loads fine, because your XenServer will just hang. No Kernel Panic, no Oops, no Reboot - just freeze.
depmod -a modprobe usbip_common_mod modprobe usbip.ko bind_driver --list bind_driver --usbip X-Y usbipd -D
You shouldn't use X-Y, or course, use the values from bind_driver --list - have a look in the driver's README
Let's move on to the client...
domU modules
The domU is a Debian Lenny installation, running kernel version "2.6.29-xs5.5.0.14" provided by the XenServer Tools (the Linux Pack CD). Luckily for us, the usbip modules are already in the kernel source, although under the staging tree. To activate them you have to roll your own kernel, something i hadn't done in quite some time...
ProTip: I've set up a clean domU to compile all the drivers tools, in fact i keep a clean one backed up so i can always start from scratch and don't mess with the live domU. You could also export/clone the domU you want to use USB/IP in, and compile on that copy. Or you can just use our live domU, if you don't care about it.
First, add the GPG key for the Citrix repository, then install some tools and the kernel source:
wget -q http://updates.vmd.citrix.com/XenServer/5.5.0/GPG-KEY -O- | apt-key add - aptitude update aptitude install bzip2 build-essential libncurses5-dev kernel-package linux-source-2.6.29-xs5.5.0.14
Now extract the source and copy the currently running config and start configuring:
cd /usr/src tar jxvf linux-source-2.6.29-xs5.5.0.14.tar.bz2 cp /boot/config-2.6.29-xs5.5.0.14 linux-source-2.6.29-xs5.5.0.14/.config cd linux-source-2.6.29-xs5.5.0.14 make menuconfig
Now we have to add the modules we want:
Bus options (PCI etc.) --->
[*] PCI Support
Device Drivers --->
[*] USB Support --->
<M> Support for Host-side USB
[*] Staging Drivers --->
[ ] Exclude Staging drivers from being built
<M> USB IP support (EXPERIMENTAL)
<M> USB IP client driver
Here's the deal: you have to enable PCI support, or you don't get Host-side USB, and without Host-side USB you don't get the USB IP modules. Took me quite some time to figure out... I'm only enabling the USB IP client driver, because the host module wouldn't make much sense in the VM, which doesn't have any real USB controller, right?
ProTip: If you're planning on using a USB ISDN card, like i do, you enable the following:
[*] ISDN support --->
<M> CAPI 2.0 subsystem --->
[*] CAPI2.0 Middleware support (EXPERIMENTAL)
<M> CAPI2.0 /dev/capi support
[*] CAPI2.0 filesystem support
Now exit the interface and say YES when you're asked if "you wish to save your new kernel configuration". Let's compile the kernel!
make-kpkg --initrd kernel_image
WARNING: Wise people recommend that you also add something like "--append-to-version=-custom" to the make-kpkg command to keep the package from overwriting your current kernel, because when you install the kernel package we're compiling right now, it WILL overwrite your current, and probably only, kernel. In theory that's what we want, no fuzz with multiple kernels, right? In practice, things go wrong. And especially a kernel which doesn't boot isn't that much fun. Heed the warning! I didn't, and still everything was fine, but that doesn't mean you're as lucky as me. Just sayin'...
Now that you read the warning, you're kernel will probably already be compiled. Good job! Well, go ahead and install it anyway:
dpkg -i ../linux-image-2.6.29-xs5.5.0.14_165_i386.deb
When being asked if you want to stop installing the kernel, because it's the same as the one currently running, you should probably say "Yes" ... so go ahead and say "No". Fight the Power! Hit the "OK", and reboot. Seriously. It won't work otherwise.
After the VM is back up, login and enter the following to load the modules:
modprobe usb_common_mod modprobe vhci_hcd
If all went well, you should see something like this in when you run dmesg
usbip_common_mod: module is from the staging directory, the quality is unknown, you have been warned. usbip_common_mod: usbip common driver 1.0 usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb vhci_hcd: module is from the staging directory, the quality is unknown, you have been warned. vhci_hcd: vhci_hcd, 1.0 usbip: proving... vhci_hcd vhci_hcd: USB/IP Virtual Host Contoroller vhci_hcd vhci_hcd: new USB bus registered, assigned bus number 1 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 8 ports detected
Great, you just got a 8-port USB hub for free!
domU tools
There's only one thing left to do before you can finally use USB over IP, and that's compiling the tools, again. You could just copy them over from the DDK, but ... they're a bit picky. So let's finish this once and for all:
aptitude install autoconf libtool libsysfs-dev libwrap0-dev libglib2.0-dev cd /usr/src wget http://downloads.sourceforge.net/project/usbip/usbip/0.1.7/usbip-0.1.7.tar.gz tar xvf usbip-0.1.7.tar.gz cd usbip-0.1.7/src ./autogen.sh ./configure --prefix=/usr make && make install
Done! Wanna try?
usbip --list <your-dom0-IP>
There you go! Now go and finally read the USB/IP README for more info...