Introduction

1. Creating new VM.
2. Configuring existing VM.
3. Estimate cost of creating a new VM.
4. Attribute-value based querying of VMs.
5. Collect (or destroy) VM.

The figure above shows the architecture of VMShop Grid Service. It consists of the following components:

1. VMShop client which is any Grid middleware that requires a virtual machine (eg. InVIGO). These VMs provide native application environments for Grid applications.
2. VMPlant server which is a classical virtual machine (VMware/Xen/UML) server that handles the actual creation, configuration or destruction of virtual machines.
3. VMShop server which manages a number of heterogeneous VMPlant servers, and takes client requests. VMShop operates at the level of abstract virtual machines, and provides an API that hides the details of the virtualization technology. Upon receiving a create VM request, VMShop server picks the VMPlant where the VM can be created with least cost.

Describing VMs

The VMPlant servers maintain a library of cached VM images, from which new VMs can be cloned. The images are created by application installer and put in what we called the VM warehouse accessible to the VMPlants. These images are uniquely named by the application installer.

The new VM DAG starts with the node identifying the cached image (unique name) from which to clone, followed by nodes which may include configuring network, mounting application data files etc.

Some sample DAGs are available in the "VMShop/scripts" directory of the VMShop directory tree (See next section).

Downloading and Installation

1. Run the script called "configure_vmshop.py".The script asks you for the port on which VMShop server listens, and prepares a file called "configuration_vmshop.xml".
   
2. The VMShop directory also contains a directory called "addreses". There is a file in this directory called "vmplant_addreses.txt". This file contains IP address and port number of each of the VMPlant servers (one per line). Edit the file for your environment.

1. Run the script called "configure_vmplant.py".The script asks you for the following:
   • Virtualization technology (Xen/VMware).
   • Location of the VM warehouse. As mentioned earlier in this document, VMPlant maintain a list of VM images from which new VMs are cloned. The VM warehouse contains the locations of such VM images in the file system.
   • Location of VM clones (where all newly created VMs are placed).
   • Location of VM store (where information about new VMs is stored).
   • VMPlant listening port.
   • Location of network store. Both VMware (vmnet) and Xen (xen-br0) setup a number of virtual devices on the host to which VMs are plugged. The list of such devices is maintained in the network store.
   The script prepares a file called "configuration_vmplant.xml".
2. The VMShop directory also contains a directory called "addreses". There is a file in this directory called "vmshop addreses.txt". This file contains IP address and port number of each of the VMShop servers (one per line). Edit the file for your environment.
3. Setup the VM warehouse: Create a file called "index.txt" inside the VM warehouse directory, that contains absolute path names of the configuration files for VM images. These VM images should meet certain requirements which are described later.
4. Set up the network store. Change directory to the network store directory.
   • Create a file called "index.txt". Add an entry (one per line) for each virtual device on the host. The entry looks like the following for Xen
     $ cat index.txt
     1 # xen-br0
     2 # xen-br1
     $
     and for VMware it may look as follows:
     $ cat index.txt
     1 # vmnet0
     2 # vmnet1
     $
     The second column names the virtual device, while the first column is the name of the file that holds the classad describing the VM which may look as follows for Xen
     $ cat 1
     [DEVICE="/dev/vmnet0";TYPE="VMWare"; COUNT=0; ACCESS="public"] $
     and for VMware it may look like:
     $ cat 1
     [DEVICE="xen-br0";TYPE="Xen"; COUNT=0; ACCESS="public"]
     $
   • To build the network store, these classad files too have to be created inside the network store directory.
   Some more instructions on configuring a Xen VMPlant are available here. I would suggest you first go through the next section on VM warehouse setup.

VM warehouse setup

• VMware - Only Linux-based and file-backed VMs

  With a VMware VMPlant, VM configuration is achieved by connecting CD-ROM images holding configuration scripts and running them inside the VM guest.
  1. All VM files are inside the same directory (not scattered all over the file system).
  2. We assume that VMware images in the warehouse are "suspended VMs". Therefore, when we clone from those images and resume, we get a VM in post-boot stage.
  3. The disk type is non-persistent and the redo log directory is called "logs" relative to the VM directory name. No absolute names.
  4. The VM is configured with a CD-ROM device which is backed by a file called "cfg.iso" inside the VM directory. The CD-ROM starts disconnected. Moreover, there should also be a directory called "/mnt/cdrom" inside the VM guest, which serves as the mount point for CD-ROM.
  5. In case network settings are required, the ethernet device should again start disconnected. All the subsequent VM clones get resumed with the same MAC address, and hence we start disconnected. We only connect the device once we have configured a unique MAC. Therefore, the DAG should also have nodes for configuring network (MAC, IP), followed by a node called "START_NETWORK" having the command "/etc/init.d/network restart". The "START_NETWORK" is an indication that the ethernet card can now be connected safely.
  6. Upload the script called "vmAPIhost" in "VMShop/vmplant/vmware" (of the VMShop directory tree) into the VM and start the script before suspending the VM. This is a helper script for VMPlant for configuring the VM.
  7. Inside the VM directory, there should be a file called "description" which has the XML description of the image (unique name we talked about earlier).
  Looks like we are set. Questions welcome aganguly@acis.ufl.edu

• Xen - Only Linux-based and file-backed VMs

  Xen (unlike VMware) does not support disconnected devices , so we cannot do something like the virtual CD-ROM. Its is possible that we attach a startup configuration script in a separate VBD, which gets invoked on bootup. Now, a user may want to have the VM reconfigured, and it is not possible to connect a device with a different scripts to the domain (without rebooting).

  To configure the VM we use NFS shares between dom0 and domU over a host-only private network. All the domUs are also inside a host-only network with dom0 through a software bridge (linux bridge-utils). It is over this network that the NFS mounts happen.

  Here are the requirements for Xen images:

  1. Again assuming each VM has all its file in one directory. These files include root disk, swap disk, kernel image and Xen configuration file.
  2. In the Xen configuration file, all the file backings (disk, kernel image etc) should be specified as relative path names (relative to the VM directory). The advantage is that when we clone the VM directory to a different location, the same configuration file is still valid. However, we know Xen has complains unless we provide absolute path names for file-backed VBDs. The cloning script itself would change all file names to absolute names. The configuration file is a python script and can be loaded as a module. The pathnames should be specified like "%s/file_name". The people familiar with python probably got it. We want to replace the %s with the VM location to get an absolute pathname, which is so easy in python. Heres a sample:
     $ cat xen-config.py
     kernel = "%s/vmlinuz-2.6-xenU"
     memory = 64
     disk = ['file:%s/rootfs,sda1,w','file:%s/swapfs,sda2,w']
     vif = []
     root = "/dev/sda1 ro"
     $
     The kernel-arguments using "other" field should not be specified because this field is used by Xen plant.
  3. The logical device "/dev/sda3" is used by VMPlant to configure the VM. So use different names for all other logical devices.
  4. Inside the "/etc/fstab" create an entry to mount "/dev/sda3" at the mount point "/mnt/init".
  5. Create an "/etc/rc.local" which in turn calls "/mnt/init/config". This script "/mnt/init/config" achieves the following:
     1. Set up a host-only IP address for the VM.
     2. Mount a directory inside dom0 on "/mnt/nfs".
     3. Start a script in the background that keeps monitoring contents of "/mnt/nfs", and executes any new configuration scripts that it detects.
  Please read the next section on special instructions to setup a Xen plant in dom0.

Xen VMPlant - Special instructions

1. Create a directory called script directory and sub-directories inside it for each new domain. Create export entries ("/etc/exports") like:
   $ cat /etc/exports
   192.168.77.1 /script/1(rw,no_root_squash)
   192.168.77.2 /script/2(rw,no_root_squash)
   192.168.77.3 /script/3(rw,no_root_squash)
   $
   Every new domain that is created checks-out a private IP address and mounts the corresponding script directory.
2. Download the VBD init.dsk inside dom0. This device becomes the sda3 inside guest and contains a script that sets up NFS mounts in domUs.
3. Edit the file "VMShop/vmplant/xen/init.py". The file has the initialization values used by the Xen plant scripts located in the same directory. The following fields ned to be edited:
   • initDisk: Location of the init.dsk file in the dom0 fil system.
   • localhost: Private host-only IP address for dom0.
   • ipFile: Location of the file that has host-only IP addreses that can be assigned to domUs.
4. Creating an IP file. Look at the sample file "VMShop/vmplant/xen/ip.txt". It has the following fields:
   1. host-only IP address
   2. software bridge name to which the domain connects in the host-only network. To provide isolation, we may want to create separate bridges in domain0, and each domain connects to a different bridge in the host-only network. This is however not supported in the current implementation. There is a single host-only bridge to which all domains connect. It is called "test" in the sameple file.
   3. exported directory (the NFS share).
   4. copy-on-write (COW) directory (not supported). But continue to have this field.
   5. VFS proxy port nfsd for COW (not supported).
   6. VFS proxy port mountd for COW (not supported).
   7. server port (deprecated).
   8. Last field is be named "unused" depicting an unassigned host-only IP address. Once the IP address gets assigned this will be replaced by details of domU.
   Please do not delete any field that is not-supported or deprecated. Continue to have them, although they will not be used. This file is created with all "unused" IP address entries, and from then onwards is edited only by Xen plant scripts.

   The COW support is still not built into Xen, and we have been using redirect-on-write Virtual File System to achieve the same. This mechanism is under testing, and not in the current release. We are currently do full root disk copy.