XenServer/USBIP

From fakedWiki
Jump to: navigation, search

I've migrated my server at home from a single physical box to multiple XenServer VMs (or domUs, as they're called in Xen), and i am really satisfied with the performance and ease of use of the whole XenServer/XenCenter setup. One thing i didn't think about, though, was how i would get my Asterisk back to work, as it used a Fritz!Card PCI for incoming and outgoing calls on my landline (ISDN, obviously). The problem with XenServer is, that it can't forward/pass-through PCI devices to its VMs. I read a lot about the problems, and altough there are some mixed reports of people getting it to work with (non-Citrix, regular) Xen, it would definitely not work with my (Citrix) XenServer.

The next idea was to use an USB device, instead of a PCI card, because some people had success with forwarding that from the dom0 to a domU. Well, turns out that only works for USB storage devices (memory sticks, etc.).

The only solution left to try out was USB over IP. It consists of 3 kernel modules (one common for client and server "usbip_common_mod.ko", one for the server "usbip.ko", and one for the client "vhci-hcd.ko") and 3 binaries ("bind_driver" and "usbipd" on the server, "usbip" on the client). If you're running on kernel 2.6.16 to 2.6.25, you have to use the kernel modules in the archive, kernel 2.6.28 and newer already have the modules in the staging tree of the kernel sources.

ProTip: If you want to use something like a Fritz!Card USB with USB/IP in your XenServer domU, make sure you also read this: XenServer/FritzUSB so you don't have to compile your kernel twice.


Preparations

My dom0 runs the kernel "2.6.18-128.1.6.el5.xs5.5.0.496.1012xen", so i had to compile the modules from the archive. Citrix provides a so-called DDK (Driver Developmen Kit) for this purpose, so you don't have to install all the development tools and sources on your production server. The DDK is just a domU you import into your XenServer, boot up, and start compiling ... more or less.

To download the DDK, you have to register an account for My Citrix, which is free and painless. After login in you can go to the Citrix Essentials for XenServer 5.5 page and all the way down you find the "Driver Development Kit" download link. While we're here, also grab the "Driver Development Kit Source" ISO under Source Code, a couple of lines above.

When the download of the DDK ISO has finished you can either copy it to the dom0 via SCP, or put it on a domU and mount the folder via NFS, or anything else, as long as you can mount the ISO like this:

mount -o loop /path/to/XenServer-5.5.0-DDK.iso /mnt

then run the following command to import the DDK VM:

xe vm-import filename=/mnt/ddk/ova.xml

When that's done you can boot the DDK and login.

XenServer modules

The DDK includes only the kernel headers, but for compiling the modules you also need the kernel sources. If you carefully read my instructions above, you already have downloaded the DDK Source ISO - if not, scroll up and do it now!

We need the kernel sources from the DDK Source ISO, so mount it and install the RPM, copy the extracted archive back to /usr/src and extract it ... like this:

mount -o loop /path/to/XenServer-5.5.0-source-ddk.iso /mnt
rpm -Uhv /mnt/kernel-dom0/kernel-2.6.18-128.1.6.el5.xs5.5.0.496.1012.src.rpm
cd /usr/src
tar jxvf redhat/SOURCES/linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012.tar.bz2
cp redhat/SOURCES/kernel-2.6.18-128.1.6.el5.xs5.5.0.496.1012-i686-xen.config linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012/.config
cd linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012/
make prepare && make scripts

Download the usbip sources from Sourceforge and unpack them:

cd /usr/src
wget http://downloads.sourceforge.net/project/usbip/usbip/0.1.7/usbip-0.1.7.tar.gz
tar xvf usbip-0.1.7.tar.gz
cd usbip-0.1.7/drivers/2.6.18/

I had to comment out some of the code in stub_rx.c, else it would fail with errors. Open stub_rx.c with your favorite text-editor, and go to line 374.

Comment out the lines from 374 to 399 like this:

/*
#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,21)
static inline int usb_endpoint_xfer_bulk(
[...]
[...]
[...]
        return ((epd->bmAttributes & USB_ENDPOINT_XFERTYPE_MASK) ==
                        USB_ENDPOINT_XFER_ISOC);
}
#endif
*/

Alright, compile the modules! Run the following command, it shouldn't result in any warnings or errors:

make KSOURCE=/usr/src/linux-2.6.18-128.1.6.el5.xs5.5.0.496.1012

XenServer tools

Now on to the binaries. You will have to install some additional packages first:

cd /usr/src/usbip-0.1.7/src
yum --enablerepo=base install autoconf libtool libsysfs-devel glib2-devel
./autogen.sh
./configure --prefix=/usr
make && make install

That's it, you're theoretically ready to serve USB over IP now. Hang on, you're still in a domU...

Finalizing the XenServer

Keep in mind that you've compiled the driver and tools on the DDK, but i guess you'd rather use them on the dom0, where you can actually plug in USB devices, right?

You need to create two folders on the dom0:

mkdir /lib/modules/2.6.18-128.1.6.el5.xs5.5.0.496.1012xen/kernel/extra
mkdir /usr/share/usbip

Copy everything to the dom0:

scp /usr/bin/usbip* dom0:/usr/bin/
scp /usr/bin/bind_driver dom0:/usr/bin/
scp /usr/lib/libusbip.* dom0:/usr/lib/
scp /usr/share/usbip/usb.ids dom0:/usr/share/usbip/
scp /usr/src/usbip-0.1.7/2.6.18/*.ko dom0:/lib/modules/2.6.18-128.1.6.el5.xs5.5.0.496.1012xen/kernel/extra


WARNING: I had to learn this the hard way, but you don't want to have any devices plugged into any USB port when you load the modules. The first one loads fine, and the second one ... well, hard to tell if it loads fine, because your XenServer will just hang. No Kernel Panic, no Oops, no Reboot - just freeze.

depmod -a
modprobe usbip_common_mod
modprobe usbip.ko
bind_driver --list
bind_driver --usbip X-Y
usbipd -D

You shouldn't use X-Y, or course, use the values from bind_driver --list - have a look in the driver's README

Let's move on to the client...

domU modules

The domU is a Debian Lenny installation, running kernel version "2.6.29-xs5.5.0.14" provided by the XenServer Tools (the Linux Pack CD). Luckily for us, the usbip modules are already in the kernel source, although under the staging tree. To activate them you have to roll your own kernel, something i hadn't done in quite some time...

ProTip: I've set up a clean domU to compile all the drivers tools, in fact i keep a clean one backed up so i can always start from scratch and don't mess with the live domU. You could also export/clone the domU you want to use USB/IP in, and compile on that copy. Or you can just use our live domU, if you don't care about it.

First, add the GPG key for the Citrix repository, then install some tools and the kernel source:

wget -q http://updates.vmd.citrix.com/XenServer/5.5.0/GPG-KEY -O- | apt-key add -
aptitude update
aptitude install bzip2 build-essential libncurses5-dev kernel-package linux-source-2.6.29-xs5.5.0.14

Now extract the source and copy the currently running config and start configuring:

cd /usr/src
tar jxvf linux-source-2.6.29-xs5.5.0.14.tar.bz2
cp /boot/config-2.6.29-xs5.5.0.14 linux-source-2.6.29-xs5.5.0.14/.config
cd linux-source-2.6.29-xs5.5.0.14
make menuconfig

Now we have to add the modules we want:

Bus options (PCI etc.) --->
   [*] PCI Support
Device Drivers --->
   [*] USB Support --->
      <M> Support for Host-side USB
   [*] Staging Drivers --->
      [ ] Exclude Staging drivers from being built
      <M> USB IP support (EXPERIMENTAL)
      <M> USB IP client driver

Here's the deal: you have to enable PCI support, or you don't get Host-side USB, and without Host-side USB you don't get the USB IP modules. Took me quite some time to figure out... I'm only enabling the USB IP client driver, because the host module wouldn't make much sense in the VM, which doesn't have any real USB controller, right?

ProTip: If you're planning on using a USB ISDN card, like i do, you enable the following:

[*] ISDN support ---> 
   <M> CAPI 2.0 subsystem --->
      [*] CAPI2.0 Middleware support (EXPERIMENTAL)
      <M>   CAPI2.0 /dev/capi support
      [*]     CAPI2.0 filesystem support

Now exit the interface and say YES when you're asked if "you wish to save your new kernel configuration". Let's compile the kernel!

make-kpkg --initrd kernel_image

WARNING: Wise people recommend that you also add something like "--append-to-version=-custom" to the make-kpkg command to keep the package from overwriting your current kernel, because when you install the kernel package we're compiling right now, it WILL overwrite your current, and probably only, kernel. In theory that's what we want, no fuzz with multiple kernels, right? In practice, things go wrong. And especially a kernel which doesn't boot isn't that much fun. Heed the warning! I didn't, and still everything was fine, but that doesn't mean you're as lucky as me. Just sayin'...

Now that you read the warning, you're kernel will probably already be compiled. Good job! Well, go ahead and install it anyway:

dpkg -i ../linux-image-2.6.29-xs5.5.0.14_165_i386.deb

When being asked if you want to stop installing the kernel, because it's the same as the one currently running, you should probably say "Yes" ... so go ahead and say "No". Fight the Power! Hit the "OK", and reboot. Seriously. It won't work otherwise.

After the VM is back up, login and enter the following to load the modules:

modprobe usb_common_mod
modprobe vhci_hcd

If all went well, you should see something like this in when you run dmesg

usbip_common_mod: module is from the staging directory, the quality is unknown, you have been warned.
usbip_common_mod: usbip common driver 1.0
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
vhci_hcd: module is from the staging directory, the quality is unknown, you have been warned.
vhci_hcd: vhci_hcd, 1.0
usbip: proving...
vhci_hcd vhci_hcd: USB/IP Virtual Host Contoroller
vhci_hcd vhci_hcd: new USB bus registered, assigned bus number 1
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected

Great, you just got a 8-port USB hub for free!

domU tools

There's only one thing left to do before you can finally use USB over IP, and that's compiling the tools, again. You could just copy them over from the DDK, but ... they're a bit picky. So let's finish this once and for all:

aptitude install autoconf libtool libsysfs-dev libwrap0-dev libglib2.0-dev
cd /usr/src
wget http://downloads.sourceforge.net/project/usbip/usbip/0.1.7/usbip-0.1.7.tar.gz
tar xvf usbip-0.1.7.tar.gz
cd usbip-0.1.7/src
./autogen.sh
./configure --prefix=/usr
make && make install

Done! Wanna try?

usbip --list <your-dom0-IP>

There you go! Now go and finally read the USB/IP README for more info...