User:Andy/Xen

The Xen virtual machine monitor is a set of tools and a patch to the x86 Linux kernel to enable it to host multiple virtual machines with close to native performance. Any x86 operating system can be ported to run as a Xen guest, and ports already exist for Linux, FreeBSD and NetBSD.

Why?
I've started playing with Xen for my own interests and because it has the potential to help with hosting issues of Linux User Groups UK. This page is for notes of my experiences with Xen.

Installation
We've already set Xen up once for lug.org.uk, on a Fedora Core host, but it seems rather buggy. This could be due to Xen, or the kernel used. I've recently installed Xen on my own Debian Sarge machine and this seems to be working much better, with a total so far of 6 unprivileged domains. Here's how I did that.

Xen kernel patch
Downloaded a snapshot of xen-testing from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0-testing-src.tgz and unpacked it into /opt/xen. This archive contains (amongst other things) a number of trees of kernel source where only files changed from the stock kernels are present. These are the so-called "sparse" trees. I deleted all spares trees that I was not interested in:

$ rm -fr freebsd-5.3-xen-sparse linux-2.4.30-xen-sparse netbsd-2.0-xen-sparse

and then generated a patch against stock 2.6.11 kernel:

[andy@curacao xen-2.0-testing]$ make mkpatches for i in linux-2.6.11 ; do make $i-xen.patch; done make[1]: Entering directory `/opt/xen/xen-2.0-testing' Cannot find linux-2.6.11.tar.bz2 in path .:.. wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2 -O./linux-2.6.11.tar.bz2 --14:09:14-- http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2 => `./linux-2.6.11.tar.bz2' Resolving www.kernel.org... 204.152.191.37, 204.152.191.5 Connecting to www.kernel.org[204.152.191.37]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 37,075,679 [application/x-bzip2]

100%[====================================>] 37,075,679  143.88K/s    ETA 00:00

14:13:06 (156.07 KB/s) - `./linux-2.6.11.tar.bz2' saved [37075679/37075679]

rm -rf tmp-pristine-linux-2.6.11 pristine-linux-2.6.11 mkdir -p tmp-pristine-linux-2.6.11 tar -C tmp-pristine-linux-2.6.11 -jxf linux-2.6.11.tar.bz2 mv tmp-pristine-linux-2.6.11/* pristine-linux-2.6.11 touch pristine-linux-2.6.11/.valid-pristine # update timestamp to avoid rebuild rm -rf ref-linux-2.6.11 cp -al pristine-linux-2.6.11 ref-linux-2.6.11 ([ -d patches/linux-2.6.11 ] && \ for i in patches/linux-2.6.11/*.patch ; \ do ( cd ref-linux-2.6.11 ; patch -p1 <../$i || exit 1 ) ; done) || true patching file drivers/char/agp/agp.h patching file drivers/char/agp/ali-agp.c patching file drivers/char/agp/amd-k7-agp.c patching file drivers/char/agp/amd64-agp.c patching file drivers/char/agp/ati-agp.c patching file drivers/char/agp/backend.c patching file drivers/char/agp/efficeon-agp.c patching file drivers/char/agp/generic.c patching file drivers/char/agp/hp-agp.c patching file drivers/char/agp/i460-agp.c patching file drivers/char/agp/intel-agp.c patching file drivers/char/agp/intel-mch-agp.c patching file drivers/char/agp/sworks-agp.c patching file drivers/char/agp/uninorth-agp.c patching file include/asm-alpha/agp.h patching file include/asm-i386/agp.h patching file include/asm-ia64/agp.h patching file include/asm-ppc/agp.h patching file include/asm-sparc64/agp.h patching file include/asm-x86_64/agp.h patching file drivers/char/agp/frontend.c patching file drivers/char/drm/drm_vm.c patching file drivers/char/drm/i810_dma.c patching file drivers/char/drm/i830_dma.c patching file drivers/char/hpet.c patching file drivers/sbus/char/flash.c patching file include/linux/mm.h patching file Documentation/SecurityBugs patching file MAINTAINERS patching file Makefile patching file REPORTING-BUGS patching file arch/ia64/kernel/fsys.S patching file arch/ia64/kernel/signal.c patching file arch/ppc/oprofile/op_model_fsl_booke.c patching file arch/ppc/platforms/4xx/ebony.h patching file arch/ppc/platforms/4xx/luan.h patching file arch/ppc/platforms/4xx/ocotea.h patching file arch/ppc64/kernel/pSeries_iommu.c patching file arch/sparc/kernel/ptrace.c patching file arch/sparc64/kernel/ptrace.c patching file arch/sparc64/kernel/signal32.c patching file arch/sparc64/kernel/systbls.S patching file arch/um/include/sysdep-i386/syscalls.h patching file arch/um/include/sysdep-x86_64/syscalls.h patching file arch/um/kernel/skas/uaccess.c patching file arch/um/kernel/sys_call_table.c patching file arch/x86_64/kernel/apic.c patching file arch/x86_64/kernel/ptrace.c patching file arch/x86_64/kernel/smpboot.c patching file arch/x86_64/mm/fault.c patching file arch/x86_64/mm/ioremap.c patching file drivers/block/ioctl.c patching file drivers/block/pktcdvd.c patching file drivers/char/drm/drm_ioctl.c patching file drivers/char/raw.c patching file drivers/i2c/chips/eeprom.c patching file drivers/i2c/chips/it87.c patching file drivers/i2c/chips/via686a.c patching file drivers/ide/ide-disk.c patching file drivers/input/serio/i8042-x86ia64io.h patching file drivers/md/raid6altivec.uc patching file drivers/media/video/adv7170.c patching file drivers/media/video/adv7175.c patching file drivers/media/video/bt819.c patching file drivers/media/video/bttv-cards.c patching file drivers/media/video/saa7110.c patching file drivers/media/video/saa7114.c patching file drivers/media/video/saa7185.c patching file drivers/net/3c59x.c patching file drivers/net/amd8111e.c patching file drivers/net/ppp_async.c patching file drivers/net/r8169.c patching file drivers/net/sis900.c patching file drivers/net/tun.c patching file drivers/net/via-rhine.c patching file drivers/net/wan/hd6457x.c patching file drivers/pci/hotplug/pciehp_ctrl.c patching file drivers/usb/serial/visor.c patching file drivers/video/matrox/matroxfb_accel.c patching file drivers/video/matrox/matroxfb_base.h patching file fs/binfmt_elf.c patching file fs/cramfs/inode.c patching file fs/eventpoll.c patching file fs/exec.c patching file fs/ext2/dir.c patching file fs/ext3/balloc.c patching file fs/hfs/mdb.c patching file fs/hfs/super.c patching file fs/hfsplus/super.c patching file fs/isofs/inode.c patching file fs/isofs/rock.c patching file fs/jbd/checkpoint.c patching file fs/jbd/transaction.c patching file include/asm-x86_64/processor.h patching file include/linux/err.h patching file kernel/exit.c patching file kernel/signal.c patching file lib/rwsem-spinlock.c patching file lib/rwsem.c patching file mm/mmap.c patching file mm/rmap.c patching file net/bluetooth/af_bluetooth.c patching file net/bridge/br_input.c patching file net/bridge/br_stp_bpdu.c patching file net/bridge/netfilter/ebtables.c patching file net/ipv4/fib_hash.c patching file net/ipv4/netfilter/ip_queue.c patching file net/ipv4/tcp_input.c patching file net/ipv4/tcp_timer.c patching file net/ipv4/xfrm4_output.c patching file net/ipv6/xfrm6_output.c patching file net/netrom/nr_in.c patching file net/rose/rose_route.c patching file net/sched/sch_netem.c patching file net/xfrm/xfrm_state.c patching file security/keys/key.c patching file sound/core/timer.c patching file sound/pci/ac97/ac97_codec.c patching file sound/usb/usbaudio.c patching file sound/usb/usx2y/usbusx2y.c patching file drivers/mtd/maps/nettel.c patching file kernel/rcupdate.c patching file net/ipv4/udp.c Hunk #1 succeeded at 737 (offset -1 lines). Hunk #2 succeeded at 747 (offset -1 lines). Hunk #3 succeeded at 847 (offset -1 lines). Hunk #4 succeeded at 1331 (offset -3 lines). Hunk #5 succeeded at 1342 (offset -3 lines). touch ref-linux-2.6.11/.valid-ref # update timestamp to avoid rebuild rm -rf tmp-linux-2.6.11-xen.patch cp -al ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch ( cd linux-2.6.11-xen-sparse && ./mkbuildtree ../tmp-linux-2.6.11-xen.patch ) diff -Nurp ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch > linux-2.6.11-xen.patch || true rm -rf tmp-linux-2.6.11-xen.patch make[1]: Leaving directory `/opt/xen/xen-2.0-testing' [andy@curacao xen-2.0-testing]$

That left me with a pristine 2.6.11 kernel archive in linux-2.6.11.tar.bz2 and Xen's patch to that in linux-2.6.11-xen.patch.

I then unpacked the kernel to /usr/src and applied the patch:

[andy@curacao xen-2.0-testing]$ cd /usr/src [andy@curacao src]$ sudo tar jxf /opt/xen/xen-2.0-testing/linux-2.6.11.tar.bz2 [andy@curacao src]$ sudo mv linux-2.6.11 linux-2.6.11-xen [andy@curacao src]$ cd linux-2.6.11-xen [andy@curacao linux-2.6.11-xen]$ sudo patch -p1 < /opt/xen/xen-2.0-testing/linux-2.6.11-xen.patch patching file arch/xen/boot/Makefile patching file arch/xen/configs/xen0_defconfig patching file arch/xen/configs/xenU_defconfig patching file arch/xen/i386/Kconfig patching file arch/xen/i386/kernel/cpu/common.c patching file arch/xen/i386/kernel/cpu/Makefile patching file arch/xen/i386/kernel/cpu/mtrr/main.c patching file arch/xen/i386/kernel/cpu/mtrr/Makefile patching file arch/xen/i386/kernel/entry.S patching file arch/xen/i386/kernel/head.S patching file arch/xen/i386/kernel/i386_ksyms.c patching file arch/xen/i386/kernel/ioport.c patching file arch/xen/i386/kernel/ldt.c patching file arch/xen/i386/kernel/Makefile patching file arch/xen/i386/kernel/microcode.c patching file arch/xen/i386/kernel/pci-dma.c patching file arch/xen/i386/kernel/process.c patching file arch/xen/i386/kernel/setup.c patching file arch/xen/i386/kernel/signal.c patching file arch/xen/i386/kernel/time.c patching file arch/xen/i386/kernel/timers/Makefile patching file arch/xen/i386/kernel/timers/timer_tsc.c patching file arch/xen/i386/kernel/traps.c patching file arch/xen/i386/kernel/vsyscall.S patching file arch/xen/i386/Makefile patching file arch/xen/i386/mm/fault.c patching file arch/xen/i386/mm/highmem.c patching file arch/xen/i386/mm/hypervisor.c patching file arch/xen/i386/mm/init.c patching file arch/xen/i386/mm/ioremap.c patching file arch/xen/i386/mm/Makefile patching file arch/xen/i386/mm/pageattr.c patching file arch/xen/i386/mm/pgtable.c patching file arch/xen/i386/pci/direct.c patching file arch/xen/i386/pci/irq.c patching file arch/xen/i386/pci/Makefile patching file arch/xen/Kconfig patching file arch/xen/Kconfig.drivers patching file arch/xen/kernel/ctrl_if.c patching file arch/xen/kernel/devmem.c patching file arch/xen/kernel/evtchn.c patching file arch/xen/kernel/fixup.c patching file arch/xen/kernel/Makefile patching file arch/xen/kernel/reboot.c patching file arch/xen/kernel/skbuff.c patching file arch/xen/kernel/xen_proc.c patching file arch/xen/Makefile patching file arch/xen/x86_64/kernel/early_printk.c patching file drivers/char/mem.c patching file drivers/char/tty_io.c patching file drivers/Makefile patching file drivers/xen/balloon/balloon.c patching file drivers/xen/balloon/Makefile patching file drivers/xen/blkback/blkback.c patching file drivers/xen/blkback/common.h patching file drivers/xen/blkback/control.c patching file drivers/xen/blkback/interface.c patching file drivers/xen/blkback/Makefile patching file drivers/xen/blkback/vbd.c patching file drivers/xen/blkfront/blkfront.c patching file drivers/xen/blkfront/block.h patching file drivers/xen/blkfront/Kconfig patching file drivers/xen/blkfront/Makefile patching file drivers/xen/blkfront/vbd.c patching file drivers/xen/console/console.c patching file drivers/xen/console/Makefile patching file drivers/xen/evtchn/evtchn.c patching file drivers/xen/evtchn/Makefile patching file drivers/xen/Makefile patching file drivers/xen/netback/common.h patching file drivers/xen/netback/control.c patching file drivers/xen/netback/interface.c patching file drivers/xen/netback/loopback.c patching file drivers/xen/netback/Makefile patching file drivers/xen/netback/netback.c patching file drivers/xen/netfront/Kconfig patching file drivers/xen/netfront/Makefile patching file drivers/xen/netfront/netfront.c patching file drivers/xen/privcmd/Makefile patching file drivers/xen/privcmd/privcmd.c patching file include/asm-generic/pgtable.h patching file include/asm-xen/asm-i386/agp.h patching file include/asm-xen/asm-i386/bugs.h patching file include/asm-xen/asm-i386/desc.h patching file include/asm-xen/asm-i386/dma-mapping.h patching file include/asm-xen/asm-i386/fixmap.h patching file include/asm-xen/asm-i386/floppy.h patching file include/asm-xen/asm-i386/highmem.h patching file include/asm-xen/asm-i386/io.h patching file include/asm-xen/asm-i386/mach-xen/irq_vectors.h patching file include/asm-xen/asm-i386/mach-xen/setup_arch_post.h patching file include/asm-xen/asm-i386/mach-xen/setup_arch_pre.h patching file include/asm-xen/asm-i386/mmu_context.h patching file include/asm-xen/asm-i386/msr.h patching file include/asm-xen/asm-i386/page.h patching file include/asm-xen/asm-i386/param.h patching file include/asm-xen/asm-i386/pci.h patching file include/asm-xen/asm-i386/pgalloc.h patching file include/asm-xen/asm-i386/pgtable-2level-defs.h patching file include/asm-xen/asm-i386/pgtable-2level.h patching file include/asm-xen/asm-i386/pgtable.h patching file include/asm-xen/asm-i386/processor.h patching file include/asm-xen/asm-i386/ptrace.h patching file include/asm-xen/asm-i386/segment.h patching file include/asm-xen/asm-i386/setup.h patching file include/asm-xen/asm-i386/synch_bitops.h patching file include/asm-xen/asm-i386/system.h patching file include/asm-xen/asm-i386/tlbflush.h patching file include/asm-xen/asm-i386/vga.h patching file include/asm-xen/asm-i386/xor.h patching file include/asm-xen/balloon.h patching file include/asm-xen/ctrl_if.h patching file include/asm-xen/evtchn.h patching file include/asm-xen/foreign_page.h patching file include/asm-xen/hypervisor.h patching file include/asm-xen/linux-public/privcmd.h patching file include/asm-xen/linux-public/suspend.h patching file include/asm-xen/multicall.h patching file include/asm-xen/queues.h patching file include/asm-xen/xen_proc.h patching file include/asm-xen/xen-public/arch-x86_32.h patching file include/asm-xen/xen-public/arch-x86_64.h patching file include/asm-xen/xen-public/COPYING patching file include/asm-xen/xen-public/dom0_ops.h patching file include/asm-xen/xen-public/event_channel.h patching file include/asm-xen/xen-public/grant_table.h patching file include/asm-xen/xen-public/io/blkif.h patching file include/asm-xen/xen-public/io/domain_controller.h patching file include/asm-xen/xen-public/io/netif.h patching file include/asm-xen/xen-public/physdev.h patching file include/asm-xen/xen-public/sched_ctl.h patching file include/asm-xen/xen-public/trace.h patching file include/asm-xen/xen-public/xen.h patching file include/linux/gfp.h patching file include/linux/highmem.h patching file include/linux/irq.h patching file kernel/irq/manage.c patching file mm/highmem.c patching file mm/memory.c patching file mm/page_alloc.c [andy@curacao linux-2.6.11-xen]$

At this point I had a 2.6.11 kernel with Xen patches in usr/src/linux-2.6.11-xen.

Domain 0 kernel
Then it was time to build a dom0 kernel.

I copied my old kernel config file from /boot as /usr/src/linux-2.6.11-xen/.config

I needed to add the following at the top of my .config otherwise menuconfig wouldn't work properly: CONFIG_XEN=y CONFIG_ARCH_XEN=y CONFIG_NO_IDLE_HZ=y


 * 1) XEN
 * 1) XEN

CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PHYSDEV_ACCESS=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y


 * 1) CONFIG_XEN_BLKDEV_FRONTEND is not set
 * 2) CONFIG_XEN_NETDEV_FRONTEND is not set

CONFIG_XEN_WRITABLE_PAGETABLES=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_X86=y

CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y
 * 1) CONFIG_X86_64 is not set

I configured and compiled a new kernel:

[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaoxen0 kernel_image

This brought up a menuconfig</tt> as normal, based on my normal kernel's config, but with some extra Xen options. I made sure to have the following settings:
 * XEN
 * Privileged Guest
 * X86 Processor Configuration
 * Kernel hacking
 * Magic SysRq key
 * Device Drivers
 * Multi-device support (RAID and LVM)
 * Device mapper support
 * Snapshot target
 * Networking support
 * Networking options
 * Network packet filtering

I later found it was very important to also disable anything related to AGP.

After this had finished compiling I was left with /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb</tt>

Finally I copied .config</tt> to /usr/src/config-2.6.11-xen0</tt> for safe keeping.

Unprivileged domain (domU) kernel
Cleaned out old compile and started another: [andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaotestxenu clean [andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaotestxenu kernel_image

The unprivileged kernel can be very stripped down, with no support for any physical devices. It shouldn't have loadable module support. My domUs would not have loadable module support, although that is possible. Aside from general stripping-down, the following options in menuconfig</tt> definitely needed to be changed:


 * XEN
 * (DISABLE) Privileged Guest
 * Network-device frontend driver
 * Block-device frontend driver
 * (DISABLE) Loadable module support
 * File systems
 * Pseudo filesystems
 * (DISABLE) /dev filesystem support

make-kpkg</tt> does not like the loadable module support being toggled while it is running, so the first build will fail. This is documented in the man page. I just did the clean</tt> and kernel_image</tt> steps again and it was fine. I was left with /usr/src/kernel-xen0-2.6.11curacaoxenu_1_i386.deb</tt>.

Finally I copied .config</tt> to /usr/src/config-2.6.11-xenu</tt> for safe keeping.

Xen packages and dependencies
I installed required dependencies and dom0 kernel:

[andy@curacao andy]$ sudo apt-get install iproute libatm1 \ python2.3-twisted python2.3-twisted-bin libcurl3 bridge-utils [andy@curacao andy]$ sudo dpkg -i /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb

Disabled thread-local storage as recommended in Xen docs:

[andy@curacao andy]$ sudo mv /lib/tls /lib/tls.disabled [andy@curacao andy]$ sudo touch /lib/tls [andy@curacao andy]$ sudo chmod 0 /lib/tls [andy@curacao andy]$ sudo chattr +i /lib/tls

Installed the xen kernel itself, and xen tools:

[andy@curacao andy]$ cd /opt/xen/xen-2.0-testing [andy@curacao xen-2.0-testing]$ sudo make xen tools

That installed xen.gz</tt> into /boot</tt>, and various other things around the system including xend and libxen.

Grub
Added a static stanza to my /boot/grub/menu.lst</tt>: title Debian GNU/Linux, Xen 2.6.11xen0, testing 25/5/2005 kernel /xen.gz dom0_mem=1966080 root (hd0,0) module /xen-linux-2.6.11curacaoxen0 root=/dev/sda2 ro console=tty0 console=ttyS0

Pray and reboot into the Xen kernel
First time around my praying wasn't enough as I forgot to remove the AGP stuff. This caused a nice kernel oops which locked up the boot process, and I needed to go and power cycle the machine. After building a kernel without AGP support it seemed to work fine.

xend
xend is the Xen control daemon. It should have already been started by sysv init. Commands are issued to it with the xm command: Name             Id  Mem(MB)  CPU  State  Time(s)  Console Domain-0          0     1915    0  r   1785.2
 * 1) xm list

Filesystems for first unprivileged domain
I already had one LVM volume group with free space (mainvg</tt>), so I just created two new logical volumes within this; one for root and one for swap, then initialised them:
 * 1) lvcreate -L1024M -n strugglersroot mainvg
 * 2) lvcreate -L256M -n strugglersswap mainvg
 * 3) mke2fs -j /dev/mainvg/strugglersroot
 * 4) mkswap /dev/mainvg/strugglersswap

debootstrap
For my first domain I decided to just install a minimal Debian Sarge. This was easily done with debootstrap:
 * 1) apt-get install debootstrap
 * 2) mkdir /mnt/xen
 * 3) mount /dev/mainvg/strugglersroot /mnt/xen
 * 4) debootstrap --arch i386 sarge /mnt/xen http://www.uk.debian.org/debian/

Domain config
Before doing anything I took an archive of the Sarge install to keep for any other domains I might want to make: 65M /data/xen-images/debian-sarge-root.tar.bz2
 * 1) mkdir /data/xen-images
 * 2) cd /mnt/xen
 * 3) tar jpcf /data/xen-images/debian-sarge-root.tar.bz2
 * 4) ls -sh /data/xen-images/debian-sarge-root.tar.bz2

Before the files in /mnt/xen</tt> can be turned into a bootable Linux install there are a few things that need to be configured. Note that all filenames are relative to <tt>/mnt/xen</tt>:

<tt>etc/fstab</tt>
We will be exporting <tt>/dev/mainvg/strugglersroot</tt> as <tt>/dev/sda1</tt> and <tt>/dev/mainvg/strugglersswap</tt> as <tt>/dev/sda2</tt>: /dev/sda1      /       ext3    defaults        0       1 /dev/sda2      swap    swap    defaults        0       0 proc           /proc   proc    defaults        0       0

<tt>etc/hostname</tt>
I haven't quite decided on a naming scheme yet but since this domain will be for strugglers stuff I decided to call it strugglers, and its hostname will be <tt>strugglers</tt> too. Its domain name will probably be something like <tt>domu.curacao.strugglers.net</tt>. strugglers

<tt>etc/hosts</tt>
Should contain at least: 127.0.0.1   localhost

<tt>etc/network/interfaces</tt>
I decided to give the IP <tt>212.13.198.70</tt> to this domain's eth0, which will be bridged to a virtual interface in domain 0: auto lo iface lo inet loopback

auto eth0 iface eth0 inet static address 212.13.198.70 netmask 255.255.255.224 gateway 212.13.198.65 dns-search strugglers.net dns-nameservers 212.13.198.69

<tt>etc/apt/sources.list</tt>
Eventually I will probably run my own apt mirror, but for now this is just a copy of what's in domain 0's file: deb http://the.earth.li/debian/ testing main deb-src http://the.earth.li/debian/ testing main deb http://ftp.uk.debian.org/debian/ testing main deb-src http://ftp.uk.debian.org/debian/ testing main deb http://security.debian.org/ testing/updates main

<tt>lib/tls</tt>
This needs to be renamed to <tt>lib/tls.disabled</tt> like in domain 0.

<tt>/mnt/xen</tt> should now be unmounted.

Xen domain configuration file
I wanted to:
 * create a domain called "strugglers"
 * with 128M RAM
 * 1 network interface
 * the two LVM logical volumes available as <tt>/dev/sda1</tt> and <tt>/dev/sda2</tt>

The following file saved as <tt>/etc/xen/strugglers.conf</tt> achieves this: name="strugglers" memory=128 kernel="/boot/xen-linux-2.6.10curacaoxenu" nics=1 disk=[ 'phy:mainvg/strugglersroot,sda1,w', 'phy:mainvg/strugglersswap,sda2,w' ] root="/dev/sda1 ro"

Free up memory
Currently I have domain 0 taking up all spare memory on the machine. Therefore to free up 128M for a new domain, I have to reduce domain 0's total by the same amount:
 * 1) xm balloon 0 1792

The above tells Xen to set the total memory allocated to domain 0 to 1792M.

Start it!
The moment of truth:

Using config file "/etc/xen/strugglers.conf". Started domain strugglers, console on port 9602 Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net) (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000008000000 (usable) 0MB HIGHMEM available. 128MB LOWMEM available. DMI not present. IRQ lockup detection disabled Built 1 zonelists Kernel command line: root=/dev/sda1 ro Initializing CPU#0 PID hash table entries: 1024 (order: 10, 16384 bytes) Xen reported: 3000.261 MHz processor. Using tsc for high-res timesource Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Memory: 126080k/131072k available (2228k kernel code, 4804k reserved, 574k data, 120k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... disabled NET: Registered protocol family 16 xen_mem: Initialising balloon driver. SCSI subsystem initialized Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Initializing Cryptographic API io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered loop: loaded (max 8 devices) elevator: using anticipatory as default io scheduler nbd: registered device at major 43 Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky Xen virtual console successfully installed as tty Event-channel device installed. xen_blk: Initialising virtual block device driver xen_net: Initialising virtual ethernet driver. register_blkdev: cannot get major 8 for sd NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 8Kbytes TCP: Hash tables configured (established 8192 bind 16384) ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack ip_tables: (C) 2000-2002 Netfilter core team ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. http://snowman.net/projects/ipt_recent/ Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver ip6_tables: (C) 2000-2002 Netfilter core team NET: Registered protocol family 17 NET: Registered protocol family 15 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 120k freed INIT: version 2.86 booting Activating swap. Adding 262136k swap on /dev/sda2. Priority:-1 extents:1 Checking root file system... fsck 1.35 (28-Feb-2004) /dev/sda1: clean, 13617/131072 files, 48382/262144 blocks EXT3 FS on sda1, internal journal hwclock is unable to get I/O port access: the iopl(3) call failed. System time was Mon Apr 25 12:02:00 UTC 2005. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. System local time is now Mon Apr 25 12:02:00 UTC 2005. Cleaning up ifupdown...done. Checking all file systems... fsck 1.35 (28-Feb-2004) Setting kernel variables ... ... done. Mounting local filesystems... Cleaning /tmp /var/run /var/lock. Running 0dns-down to make sure resolv.conf is ok...done. Setting up networking.../dev/shm/network/...done. Setting up IP spoofing protection: rp_filter. Configuring network interfaces... Disabled Privacy Extensions on device c0383b20(lo) done.
 * 1) xm create /etc/xen/strugglers.conf -c
 * REMOTE CONSOLE: CTRL-] TO QUIT ********

Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. Local time: Mon Apr 25 12:02:01 UTC 2005

Initializing random number generator...done. Recovering nvi editor sessions... done. INIT: Entering runlevel: 2 Starting system log daemon: syslogd. Starting kernel log daemon: klogd. Starting MTA: exim4. Starting internet superserver: inetd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron.

Debian GNU/Linux 3.1 strugglers tty1

strugglers login:

All of the hwclock complaints are because an unprivileged domain hasn't got access to the system's realtime clock hardware. They can be ignored, as all the time-related settings can be done once in domain 0.

Issuing <tt>CTRL-]</tt> exits from the xen console back to domain 0.

Logging in to a domain
Normally you'd use ssh, but the minimal Sarge install from above doesn't have that to begin with. You can connect to the xen console of a domain like this:

$ xm list Name             Id  Mem(MB)  CPU  State  Time(s)  Console Domain-0          0     1787    0  r   2176.5 strugglers        2      127    1  -b---      1.7    9602 $ xm console 2
 * REMOTE CONSOLE: CTRL-] TO QUIT ********

Debian GNU/Linux 3.1 strugglers tty1

strugglers login: root Last login: Mon Apr 25 14:29:50 2005 on tty1 Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux

The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. strugglers:~# uname -a Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux top - 15:18:55 up 3:17,  1 user,  load average: 0.00, 0.00, 0.00 Tasks: 24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0% us,  0.0% sy,  0.0% ni, 99.3% id,  0.7% wa,  0.0% hi,  0.0% si Mem:    126388k total,    13764k used,   112624k free,     1308k buffers Swap:  262136k total,        0k used,   262136k free,     6816k cached
 * 1) top

Note that even a normal user in dom0 can connect to the console of an unprivileged domain, although there would normally be a root password!

You can even reboot them from inside: strugglers:~# reboot

Broadcast message from root (tty1) (Mon Apr 25 15:20:07 2005):

The system is going down for reboot NOW! INIT: Sending processes the TERM signal strugglers:~# INIT: Sending procesStopping periodic command scheduler: cron. Stopping MTA: exim4. Stopping internet superserver: inetd. Saving the System Clock time to the Hardware Clock... hwclock is unable to get I/O port access: the iopl(3) call failed. Hardware Clock updated to Mon Apr 25 15:20:14 UTC 2005. Stopping deferred execution scheduler: atd. Stopping kernel log daemon: klogd. Stopping system log daemon: syslogd. Sending all processes the TERM signal...done. Sending all processes the KILL signal...done. Saving random seed...done. Unmounting remote and non-toplevel virtual filesystems...done. Deconfiguring network interfaces...done. Cleaning up ifupdown...done. Deactivating swap...done. Unmounting local filesystems...done. Rebooting... Restarting system. Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net) (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000008000000 (usable) 0MB HIGHMEM available. 128MB LOWMEM available. DMI not present. IRQ lockup detection disabled Built 1 zonelists Kernel command line: root=/dev/sda1 ro Initializing CPU#0 PID hash table entries: 1024 (order: 10, 16384 bytes) Xen reported: 3000.261 MHz processor. Using tsc for high-res timesource Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Memory: 126080k/131072k available (2228k kernel code, 4804k reserved, 574k data, 120k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... disabled NET: Registered protocol family 16 xen_mem: Initialising balloon driver. SCSI subsystem initialized Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Initializing Cryptographic API io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered loop: loaded (max 8 devices) elevator: using anticipatory as default io scheduler nbd: registered device at major 43 Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky Xen virtual console successfully installed as tty Event-channel device installed. xen_blk: Initialising virtual block device driver xen_net: Initialising virtual ethernet driver. register_blkdev: cannot get major 8 for sd NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 8Kbytes TCP: Hash tables configured (established 8192 bind 16384) ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack ip_tables: (C) 2000-2002 Netfilter core team ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. http://snowman.net/projects/ipt_recent/ Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver ip6_tables: (C) 2000-2002 Netfilter core team NET: Registered protocol family 17 NET: Registered protocol family 15 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 120k freed INIT: version 2.86 booting Activating swap. Adding 262136k swap on /dev/sda2. Priority:-1 extents:1 Checking root file system... fsck 1.35 (28-Feb-2004) /dev/sda1: clean, 13637/131072 files, 48407/262144 blocks EXT3 FS on sda1, internal journal hwclock is unable to get I/O port access: the iopl(3) call failed. System time was Mon Apr 25 15:20:25 UTC 2005. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. System local time is now Mon Apr 25 15:20:25 UTC 2005. Cleaning up ifupdown...done. Checking all file systems... fsck 1.35 (28-Feb-2004) Setting kernel variables ... ... done. Mounting local filesystems... Cleaning /tmp /var/run /var/lock. Running 0dns-down to make sure resolv.conf is ok...done. Setting up networking.../dev/shm/network/...done. Setting up IP spoofing protection: rp_filter. Configuring network interfaces... Disabled Privacy Extensions on device c0383b20(lo) done.

Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. Local time: Mon Apr 25 15:20:26 UTC 2005

Initializing random number generator...done. Recovering nvi editor sessions... done. INIT: Entering runlevel: 2 Starting system log daemon: syslogd. Starting kernel log daemon: klogd. Starting MTA: exim4. Starting internet superserver: inetd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron.

Debian GNU/Linux 3.1 strugglers tty1

strugglers login: