Xen Hypervisor

From ErikaWiki

(Difference between revisions)
Jump to: navigation, search
(...)
(Validating the)
Line 292: Line 292:
Please note that the version of the toolstack must be the same of the hypervisor. Using different version may make the system unstable.
Please note that the version of the toolstack must be the same of the hypervisor. Using different version may make the system unstable.
-
== Validating the ==
+
== Validating the toolstack==
To verify that the toolstack works correctly, just reboot the target board.
To verify that the toolstack works correctly, just reboot the target board.
Line 311: Line 311:
The only active domain should be the Linux "Domain 0".
The only active domain should be the Linux "Domain 0".
-
 
= ... =
= ... =

Revision as of 12:00, 29 January 2015

Contents

Introduction

This page describes the current support of ERIKA Enterprise for the Xen Hypervisor.

In particular, ERIKA can be run as a guest operating system inside a Xen domU.

Requirements for the working environment

The AllwinnerA20 CPU available in the cubieboard2 is directly supported by Xen; The Xen Wiki contains most of the documentation needed for building the software components.

This section summarizes the basic requirements of the development environment needed to start ERIKA Enterprise as a domU over a Linux dom0.

Firmware Requirements

The Hypervisor boot must happen in Hypervisor mode (EL2) (non secure).

Dupport for the EL2 boot mode is in the official repository since January 2014.

Moreover, to correctly init both cores of an SMP Platform, the bootloader needs to correctly handle the Power State Coordination Interface (PSCI); This support is not currently available in the official U-Boot repository, but a first working version is available in the development branch wip/psci of this repository.

dom0 requirements

Device tree.

All the platform features must be enumerated in a device tree, which must be compiled in the device tree binary format (also called device tree blob).

Starting with kernel 3.14, it is no more needed the creation of a dedicated device tree for the Linux boot on top of an hypervisor: it is in fact sufficient to [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/sun7ia20cubieboard2. dts use the device tree provided together with Linux]

In other words, the bootloader must provide to Xen a dtb for the dom0 kernel, plus an optional one for initramfs (in case the kernel needs it). The bootloader also needs to load the kernel and initramfs in memory, and also to note their location in the device tree blob; That information is included into a new node of the device tree, named chosen, that contains:

  • the image location and a classification per compatibility
  • additional parameters to be passed to the hypervisor or to the dom0 during boot.

The chosen node must be defined in the following way (as specified in docs/misc/arm/devicetree/booting.txt:

chosen {
  bootargs = "$xen_cmdline";
  modules {
    module@0 { /* dom0 kernel */
      bootargs = "$dom0_cmdline";
      compatible = "xen,linuxzimage", "xen,multibootmodule";
      reg = <$kernel_addr $kernel_size>
    };
    module@1 { /* initramfs */
      compatible = "xen,linuxinitrd", "xen,multibootmodule";
      reg = <$initramfs_addr $initramfs_size>;
    };
  };
};

The chosen node can be manually added to the device tree (e.g., by changing the sources and recompiling), or can be created dynamically at boot time using the U-Boot command fdt.

domU requirements

domU executable format.

Xen uses the zImage format to load a ARM kernel. However, the image should not be compressed; the only needed operation is the compilation of the kernel source code with a zImage header.

That header must have an offset 0x24 inside the zImage and must contain:

  1. a magic number (0x016f2818) that identifies the file as zImage;
  2. the starting address of the zImage;
  3. The end address of the zImage.

The following is an header example:

_start:
  @ zImage header
.rept 8
  mov r0, r0
.endr
  b _start_cubieboard2
  .word 0x016f2818 @ Magic numbers to help the loader
  .word _start_0 @ absolute load/run zImage address
  .word _end _
  start @ zImage size
  @ end of zImage header
_start_cubieboard2:

Please remember the zImage should not be compressed. To obtain the executable in the required format you can use the command objcopy as done during the compilation of a normal application for Cortex Ax (the hypothesys is that the objcopy path is for the ARMv7 toolchain is in the environment variable EE_OBJCOPY):

$(EE_OBJCOPY) -O binary myProject_master.elf myProject_master.bin

base address for the executable image of the domU

Th base address for the adress space reserved to a domU is 0x80000000. The [xen/include/public/archarm.h:372 linker script] used for the compilation of the ERIKA source code must be adapted to that value.

Privileged Instructions

A non-privileged domain cannot access to the System Control Register to enable the memory management unit (by setting bit M); If done, a domU exception is raised.

Moreover, the hypervisor protects register CP15 c15 to non privileged guest domains. This because on some ARM architectures using that register is possible to have a dump of caches L1/L2: having free access to that register would then bring memory isolation problems.

IMPORTANT: Commit to be reverted

If you compile a Xen Repository including commit 878ff4fe1816d7f808f11254b555b9e9c2f121fa, this will provoke a critical shutdown of the ERIKA Enterprise domU at boot time. We did not yet understand why this happens. the only solution is currently to revert the commit.

Creating a working system

The following steps will guide through the creation of a working system including a Linux dom0 and an ERIKA Enterprise domU that will be executed under the Xen hypervisor.

The hypothesis is that a armv7a (hardfloat) toolchain is installed in the system. That toolchain must include at least a gcc compiler, the ld linker as well as the objcopy tool. The hypothesis is that the executables will have a name starting with armv7a-hardfloat-linux-gnueabi- (Please note that this prefix is typical for the Gentoo distributions. Debian packages typically have the prefix arm-linux-gnueabihf-.).

The following sections describe how the environment can be installed into an SD card. Installing it over a USB flash drive is similar. This page does not contain instructions on using tftp or netboot.

Preparing the toolchain

Debian distributions as well as other distributions typically provide pre-compiled toolchains (on Ubuntu the package name is gcc-arm-linux-gnueabi).

However, the suggested procedure in our case is to use the crossdev application to generate a toolchain for the specific architecture:

crossdev -t armv7a-hardfloat-linux-gnueabi

Compiling and installing the bootloader

Use the following command to download and compile the U-Boot repository with support for PSCI:

# git clone https://git.kernel.org/pub/scm/linux/kernel/git/maz/uboot.git u-boot-psci
# cd u-boot-psci
# git checkout origin/wip/psci
# make CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- Cubieboard2_config
# make CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-

The bootloader should be stored at an offset of 40KB, whereas the symbol table at an offset of 8KB.

Supposing we are using a Linux host, and that the SD card can be accessed using the device file /dev/mmcblk0, you can use the following commands:

# dd if=spl/sunxispl.bin of=/dev/mmcblk0 bs=1024 seek=8
# dd if=uboot.img of=/dev/mmcblk0 bs=1024 seek=40

Creating the root file system

This section covers the creation of a single partition which will be formatted with an ext4 file system.

After that, an Arch Linux file system will be used for the partition (on a Debian host you can use debootstrap as an alternative to the download of a preconfigured root file system).

# fdisk /dev/mmcblk0

Then, you need to create a Linux partition. Please note that the first sector of the first partition in the SD card must be in a position after an offset of 40 KB (for the U-Boot installation) plus the size of the U-Boot image.

# mkfs.ext4 /dev/mmcblk0p1
# mount /dev/mmcblk0p1 /mnt
# wget http://archlinuxarm.org/os/ArchLinuxARM-sun7i-latest.tar.gz
# tar -xf ArchLinuxARM-sun7i-latest.tar.gz -C mnt
# umount /mnt

Compiling and installing of the dom0 kernel

Download the sunxi-devel branch from the official sunxi repository. That branch contains the drivers for the SD, USB ed ethernet.

# mkdir sunxi-devel
# cd sunxi-devel
# git init
# git remote add -t sunxi-devel -f origin https://github.com/linux-sunxi/linux-sunxi.git
# git checkout sunxi-devel

After that, please go on with teh kernel compilation. In the default configuration for the architecture and platform you need to enable the Xen options, as well as the peripherals and drivers that we want to use.

A complete list of the needed configuration options is available on the Mirage OS wiki.

Since ours is a minimal configuration, we disabled the support for dynamic loading modules.

# make ARCH=arm multi_v7_defconfig
# make ARCH=arm menuconfig
# make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- zImage dtbs -j4

The dtbs target is needed to compile the device tree in the binary format used during boot. Once the compilation is terminated, the Linux zImage can be found as arch/arm/boot/zImage, whereas the binary device tree to be used is arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb.

You need to copy both the zImage and the binary device tree in the root of the file system you created on the SD card containing the environment we are creating.

Validating the setup with only the Linux Kernel

To validate that everything works correctly, at this point it is already possible to boot the dom0 kernel, using a serial connection between the host and the target board. Once obtained the U-Boot prompt, please use the following commands to boot Linux:

# setenv bootcmd "ext4load mmc 0 45000000 zImage; bootm 45000000"
# boot

Compiling and installing the hypervisor

You need to download the latest Xen version using teh development repository, on the master branch.

# git clone git://xenbits.xen.org/xen.git
# cd xen
# git clean -xfd

You need to enable the diagnostic messages by modifying the Config.mk file, setting the option debug ?= y.

# make clean XEN_TARGET_ARCH=arm32 CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- CONFIG_EARLY_PRINTK=sun7i
# make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- CONFIG_EARLY_PRINTK=sun7i

If you want to run the ERIKA Enterprise domU with the GPIO driver, you need to apply the patchset that adds che hypercall XEN_DOMCTL_memory_mapping for the ARM architecture; That hypercall provides the complete access to an I/O memory region to a non-privileged domain.

Currently the patchset is not included in the Xen development repository, but is available in the mailing list archives (currently it is at it eighth version).

At the end of the compilation, the executable is the file xen/xen/xen.

You need to copy that file in the root of the SD card filesystem.

Preparing the boot scripts

To correctly boot the dom0 on top of the Xen Hypervisor, we need to tell U-Boot the right locations where the hypervisor and the dom0 should be stored.

Moreover, we need to add the chosen node needed for the hypervisor to the device tree, either statically at compile time or (easier) dynamically at boot time.

With the hyphotesis of a Linux configuration without modules and without initramfs, on an ext4 filesystem, the boot script is the following one (taken from this page, section "UBoot configuration", with some modifications).

# SUNXI Xen Boot Script
# Addresses suitable for 1GB system, adjust as appropriate for a 2GB system.
# Top of RAM: 0x80000000
# Xen relocate addr   0x7fe00000
setenv kernel_addr_r  0x7f600000 # 8M
setenv ramdisk_addr_r 0x7ee00000 # 8M
setenv fdt_addr       0x7ec00000 # 2M
setenv xen_addr_r     0x7ea00000 # 2M

setenv fdt_high       0xffffffff # Load fdt in place instead of relocating

# Load xen to ${xen_addr_r}.
ext4load mmc 0 ${xen_addr_r} /xen
setenv bootargs "console=dtuart dtuart=/soc@01c00000/serial@01c28000 dom0_mem=256M"

# Load appropriate .dtb file to ${fdt_addr}
ext4load mmc 0 ${fdt_addr} /sun7i-a20-cubieboard2.dtb
fdt addr ${fdt_addr} 0x40000
fdt resize
fdt chosen
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>

# Load Linux zImage to ${kernel_addr_r}
ext4load mmc 0 ${kernel_addr_r} /zImage
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linuxzimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "console=hvc0,115200n8 debug ignore_loglevel rw root=/dev/mmcblk0p1 rootwait earlyprintk=xen clk_ignore_unused"

bootz ${xen_addr_r} ${fdt_addr}

Given the boot script named boot.cmd, the compilation is run using the following command:

# mkimage -C none - A arm -T script -d boot.cmd boot.scr

The resulting file (boot.scr) must be put in the root of the file system storing the OS environment on the SD card.

How to exclusively assign a core to the dom0

Xen allows to assign one or more corse exclusively to the dom0 (CPU pinning).

The configuration option needed (from Xen Common problems) must be added to the command line used by U-Boot to boot Xen.

# Load xen to ${xen_addr_r}.
ext4load mmc 0 ${xen_addr_r} /xen
setenv bootargs "console=dtuart dtuart=/soc@01c00000/serial@01c28000 dom0_mem=256M dom0_max_vcpus=X dom0_vcpus_pin"

Validating the setup of Xen + Linux dom0

To validate the work done so far, you can connect to the serial line of the target board.

the boot script will automatically start Xen and Linux. once you reach the login prompt of the Linux dom0, press three times "Ctrl + A" to switch from the Linux console to the Xen console, thus verifying that Linux is really working under Xen. Just repeat the three chars to go back to the Linux console.


Compiling the toolstack

The following is the set of command needed to compile the toolstack directly on the target board. this is convenient when the environment is not Debian-based, without a debootstrap or other simple mechanisms to emulate an ARM system.

We suggest to use an external USB memory or a SATA hard disk to avoid burning the SD card due to the heavy disk load during the compilation process.

# git clone git://xenbits.xen.org/xen.git
# cd xen
# git clean -xfd
# make uninstall
# make clean
# ./configure
# make dist-tools
# make install-tools

Please note that the version of the toolstack must be the same of the hypervisor. Using different version may make the system unstable.

Validating the toolstack

To verify that the toolstack works correctly, just reboot the target board.

Once logged in, start the system daemons needed for Xen interfacing with the following command:

# /etc/init.d/xencommons start

Note that any message due to the absence of QEMU is correct, since the Xen toolstack for ARM is not based on QEMU emulation.

Finally use the xl command to list the domains which are currently active:

# xl list

The only active domain should be the Linux "Domain 0".

...

This page is not yet finished... please come back in a few days for more content...

Publications @ CloudCon

Arianna Avanzini, Integrating Linux and the Real-Time ERIKA OS Through the Xen Hypervisor - Arianna Avanzini, CloudCon 2014


Acknowledgements

This work has been done thanks to Arianna Avanzini as part of her Master Thesis done under the supervision of Prof. Paolo Valente of the University Modena Reggio Emilia. This page is mainly a translation of instructions coming from Arianna.

Special Thanks also to Bruno Morelli who provided the initial iMX6 support, as well as the integration of the complete package inside the ERIKA Enterprise virtual machine.

Personal tools