Moving disks with LVM on them

Not really related to the theme of this blog, but I had quite a ride with LVM (Logical Volume Manager) recently. I upgraded to a new machine lately (Dell SC 1430) so that I can use all the cool virtualization on my desktop. My old machine was still hanging around because I have some old apps on it which need pinned Debian packages. Last week, the machine died: seems a motherbaord issue, because even swapping the power supply did not help. So I decided to virtualize the old machine on my new machine. For that, I first had to get the old disk working again. It's a 500GB IDE disk. Luckily, the newmachine still supports IDE. Only problem is that the cables (both data and power) are so short that I can't connect both the DVD and disk at the same time. I don;t use the DVD that often, so that should not be a problem. I unplugged the DVD, plugged in the disk, and rebooted. This was when the problems started: I did use the same volume group name (vg0) on both the old and the new machine, and there were also name clashes in logical volumes (eg vg0/home). Nicely enough LVM picked the LVs from the new disk instead of the old ones, so I had a working system. But I could not mount the old filesystems because of the name clashes.

The solution seemed quite simple: rename the old volume group from vg0 to vg1 and all should be fine. After some reading I found vgrename and also that it supports addressing volume groups using UUIDs. Trouble is that vgrename only works if all its logical volumes are offline. And lvchange does _not_ accept UUIDs as volume group names. After baning my head and some googleing, I decided I need to boot off a live CD. Remember that I had to disconnect my DVD... So I first had to unplug the disk again, plug in the DVD, and boot of the CD. Renaming the volume group on the old disk was ofd course not an option anymore. so I had to rename the volume group of the new disk. However, I could not even access the partition beause the LVM is installed on top of a SW-RAID1, and I did not have the correct mdadm.conf file. Some more googleing finally led me to "mdadm -A -a y --scan --auto -c 1 /dev/sda2 /dev/sdb2" which creates /dev/md1 out of the 2 partitions. After that, it was just a "vgrename vg0 vg1". Nest step was to powerofff again, unplug the DVD, replug the disk and boot. This resulted in booting from the old disk because grub had vg-root as root partition and the new disk now was vg1-root. So reboot again and adjust root partition in grub from /dev/mapper/vg0-root to /dev/mapper-vg1-root. This booted from the correct disk, but with the filesystems from the old disk becuase /etc/fstab also had vg0 in it. Luckily, /etc was on the root partition and perl as well, so I changed fstab using "perl -p -i -e 's/vg0/vg1/g' /etc/fstab /boot/grub/menu.lst" and booted again. Now I was in business: system booted off the new disk and mounted all file systems correctly, and no more name clashes.

I then mounted the old root filesystem onto /tmp/old and the other old filesystems onto that. "chroot /tmp/old" now gives me access to my old system. Overall, this took me like 2 hours (including googling), so bad, but not too bad. Would be really good if lvchange would accept VG UUIDs though, because then all woudl have been over within 10 minutes.

I can't simply virtualizse the complete old system because I don't want to waste so much space, so I will now set up a new virtual machine and copy the old app and it's dependencies over. Hopefully will be done over the weekend.

Keeping a tap on virtual machines

Ok, I'm close to admitting defeat now. I can't keep track of all of the virtual machines that are created around me. The fact that we are using VMware ESX, VMware Server, Xen, and KVM all together does not make the job easier. Just keeping an overview of what runs on which server, which IP addresses are used, and which credentials are needed to access them fills up a big table in a wiki page already. Just keeping that page updated is a struggle. There are of course specific tools (some nicer, some not so) for each of the virtualization solutions, but I still have to find the One Tool To Rule Them All, One Tool To Find Them, One Tool To Bring Them All And In The Darkness Bind Them!

So right now, I'm more often than not, accessing all the different server consoles and trying to find the machine that I'm looking for. Also, while the VMware virtualization center is nice and easy to use, it does not show you the IP addresses of the machines. Logging in to each of them through the console is time-consuming. Using IP addresses as names helps though, so that is what I started doing (obviously only for machines with static IPs).

Getting IP addresses for Xen guests is easier because the configuration files actually contain the IP addresses. There is a way to get current IP addresses from VMware guests as well, but they have to be running and they have to have VMware tools installed. VM tools are a high hurdle, especially for non-mainstream operating systems where you have to download, compile and install the tools yourself. An alternative is to look into the ARP and forwarding tables of the upstream switch and search for the IP address based on the MAC address (which you can get e.g. through the VC client). You have to make sure though, that the machine is in the ARP table, so that creates a chicken-and-egg problem. If you know at least the subnet of the machine and that subnet is not too big (/24 works fine), you can fill the table by fping-ing the range.

Rebooting virtual machines

Managing virtual machines is getting really easy these days with all the nice graphical user interfaces available. Although, unfortunately some (most notably the VI client) are only available for Windows right now and I learned the hard way that the nice "poweroff" buttons really do that: they cut the power to the virtual machine. That results in a lengthy disk check during the next startup and can result in lost files etc. Of course, we all know that you should always shutdown a machine and not just pull the plug, but logging in to every running VM and shutting it down cleanly is not fun if you have lots of them running, and for some of them, I did not even know the root/administrator password. So, wouldn't it be nice if the "poweroff" button would do the right thing? Seems the answer to this is to make sure the VMware tools are installed and running in the guests. The ESX server sends a message to the VMware toolbox running in the guest during power state changes (before poweroff or suspend, after startup or resume).

The trick to make clean shutdowns really working under Linux is to make sure that the toolbox is running as root or else the poweroff script won't have the permission to really shutdown the machine. However, running an unknown binary as root always makes me nervous - even more so if that binary is listening on the network ports. Thankfully, VMware open sourced their tools and recently moved from Subversion to a git repository, so I will spend one day of the upcoming memorial weekend to study the code before I install it on my virtual machines.

Network Fencing in VMware Lab Manager

VMware Lab Manager has a nice feature called "Network Fencing". The background for fencing is that you often want to run multiple instances of the same configurations (i.e., a group of VMs). This could entail different test cycles for a product or recreating a customer scenario. Normally, the VMs in this setup have fixed IP addresses, and changing them for every instance is painful because these IP addresses are also part of configuration files, etc. One solution would be to set up every instance with it's own vswitch without an uplink. This nicely isolates the instances, but unfortunately also prevents access to them from the outside (e.g., from your desktop).

Network fencing solves that problem by deploying a virtual router (VR) that connects the vswitch to the outside world. The VR is automatically configured to NAT the internal IP addresses of the "fenced" VMs to unique external IP addresses. This gives you the best of both worlds: the VMs of the configuration can talk to each other using their internal (but not system-wide unique) addresses, and you can still access the VMs from the outside using their external addresses.

The current version of Manager has the limitation that it only creates one VR per instance. Therefore, all instance members have to reside on the same host (i.e., an instance can't be spread accross multiple hosts). It would be interesting to see if it's possible to overcome the limitation by using TBD Networks VirtualFirewall (which does NAT as well as VLANs) instead of the VR.

Accessing network configuration of ESX servers

One of my recent tasks involved reading and processing networking configurations from an ESX server. As a concrete example, consider that you want to list all the VLAN tags currently used. There are multiple ways to do that. The first choice is the handy console program "esxcfg-vswitch -l", which prints information about all vswitches including portgroups with their VLAN tags . This works nicely, but needs some parser/regular expression matching to extract the required information.

Another tool is"esxcfg-info", which prints lots of information about the ESX server. By default, that prints the whole configuration as a (very long) formatted text. Reading through that is possible, but not very pleasant. So the first step is to reduce the amount of data by using the "-n" option, which results in printing only the network part (other options are e.g. "-s" for storage related information. "esxcfg-info -h" prints a list of all available options). But even "esxcfg-info -n" still prints a vast amount of information compared with "esxcfg-vswitch -l", so why even consider using that? The answer is the "-F" option that can change the output to either XML or Perl. Unfortunately, the XML output is currently broken (see this thread for details), so that leaves us with the Perl output format. "esxcfg-info -n -F perl" prints the same information as "esxcfg-info -n", but as one big Perl datastructure. To use it, simply evaluate the result within your Perl script and then navigate through the big nested hashmap. No more parsing needed!

Granted, if the goal would have been really to just list all the VLAN tags, parsing the output of "esxcfg-vswitch -l" is still simpler than navigating though the data. However, if you need to answer more complex questions about the network, with a single line of Perl you get a pre-populated, nice datastructure that contains all there is to know.

There are of course more options (aren't there always?). Leaving the crazy idea of parsing the HTML output of the web interface aside, there are SDKs provided by VMware that allow to access the same information remotely. More about this next time...

Port groups in VMware ESX

ESX servers have a whole virtual network within them: guest machines connect to virtual switches and uplinks from these switches to the outside. Although, one term that is not used outside of ESX are "port groups." After reading a bit about them and looking at the various tools the the ESX console provides, I think the best way for a network engineer to understand port groups is to see them as network hubs connected to a single vswitch port. This actually makes sense for multiple reasons:

• All members of a port group share common attributes like a VLAN tag
• All members of a port group can see all of the packets sent by other members of this port group
• A port group is always connected to a single vswitch
  

Actually, it even makes sense to think of the VLAN tag as being applied to the vswitch port that is connected to the uplink of the virtual hub. Therefore, a vswitch with a portgroup "PG1" that has two members "VG1" and "VG2" would be built using a pswitch and a 3-port hub. The uplink of the hub is connected to a pswitch port. Applying a VLAN tag on that port group then corresponds to configuring the VLAN on the pswitch port.

Port groups in ESX are identified by their name, which must be unique within an ESX server. Having the same port group names in different ESX servers, however, makes a lot of sense, especially when moving guests around between them. More on this later.