Sun released new servers and storage arrays

We had to wait for upcoming AMD Opteron servers from Sun a few months since the new quad-core AMD processors, code named Barcelona, were released. Now, it would be a few days when Sun officially announced here the availibility of the second generation of their AMD servers and new storage arrays, together called as "next generation open storage hardware". More about the Open Storage hardware and related projects, you can find here.

The new servers based on quad-core AMD processors are Sun Fire X4140, 4240 and X4540. At the storage field, there were introduced new Sun Storage J4200, J4400 and J4500 arrays. All of them are SAS JBOD arrays. For more details, look at the Sun System Handbook at SunSolve.

RHEL and Infiniband - software intro

Let's continue with software introduction. As I wrote the switch is equipped with the ALOM remote management. There is an universal set of commands for platform independent management like password, poweroff, setupsc, resetsc and so on and then a set of commands which are more specific to the platform. In the case of our IB switch there are two such commands:

1. setbp - for setting so-called blueprint of switch
   
2. showbp - for showing the current blueprint
3. there are five predefined blueprints:
• 9 node, 12 node, 18 node, none and unmanaged

The natural question is what does the blueprint mean? According to official documentation it seems to be like a predefined configuration of the switch. You can change it with the setbp command which asks you if you want to run IB management software, how many hosts will be in the subnet and what is the subnet identifier. By default, if you use the switch preconfigured from the factory then two switches will have the same subnet ID. The trouble is, if you intend to configure some level of redundancy between IB switches you will have to have them in different subnets with different subnet IDs. I think it strange because I had to disable the IB management software otherwise I wasn't able to see the nodes in the fabric. As we will see, the IB mangement software including IB subnet manager doesn't seem to like the OFED included in RHEL distro (more about RHEL and OFED I wrote here).

What about the servers? I preinstalled them with CentOS 5.1 distribution (which is binary compatible with RHEL 5.1). The distribution contains the OFED implementation in version 1.2. The complete OFED implementation in CentOS is divided in a set of RPM packages. The platform dependent part of OFED that means kernel modules are distributed with kernel package. Let's make a quick summary of basic packages:

1. kernel - contains IB hardware, IB core and IB ULP modules
• ULP means Upper Level Protocol
  
• everything is placed in the following directories:
• /lib/modules/`uname -r`/kernel/drivers/infiniband/hw
• /lib/modules/`uname -r`/kernel/drivers/infiniband/core
• /lib/modules/`uname -r`/kernel/drivers/infiniband/ulp
• currently there are supported only IB HCAs from Mellanox
• the supported ULPs are
  
• ipoib - IP over IB driver
  
• srp - IB SCSI RDMA initiator driver
  
• sdp - SDP driver
  
2. openib - this package contains a lot of useful documentation and the important part is the OFED configuration file /etc/ofed/openib.conf and the init script /etc/init.d/openibd which takes care of activating/deactivating the IB network interfaces. Simply, it loads the IB core modules and specified ULP modules in the config.
   
3. openib-diags - this package contains diagnostic tools for IB debugging, I will introduce them later.
4. opensm - here we have our IB subnet manager. The package provides the init script /etc/init.d/opensmd for starting it and the /etc/ofed/opensm.conf configuration file.
5. libibverbs - this package provides a library allowing userspace programs direct hardware access.
6. libibcommon, libibmad, libibumad, opensm-libs - and finally library dependencies for the above packages.
   

I need to add that the OFED packages belongs to the System Environment/Libraries RPM group and they are not installed by default apart from the openib and libibverbs and of course kernel package. That's all for now and next time I'm going to describe how to work with it.

RHEL and Infiniband - hardware intro

In my two previous articles, I summarized a few facts about the Infiniband support in RHEL distros and included protocols - you can go through them from the following links - RHEL and Infiniband support and Infiniband, RDP, SDP.... Let's be more particular now.

My scenario was based on two servers Sun Fire X4200 M2 and one Infiniband (IB) switch Sun IB Switch 9P. The servers had installed Infiniband host channel adapters (HCA) Sun Dual Port 4x IB HCA to be able to communicate over the IB fabric. The switch provides nine IB compliant ports at dual speeds of 4X/12X what means that each port is able to deliver of 10/30Gbit raw bandwidth. What surprised me was that the switch management is like at the SUN SPARC midrange servers. Yes, it is ALOM and it is perfect because you can use the same interface and similar commands you are used to. By the way, the switch chassis looks like a regular SUN server.

The switch is equipped with the IB subnet manager (SM) which is required to initialize the IB hardware and to allow the communication over the IB fabric. Each IB subnet has to have at least one and each has unambiguous identifier (ID) over the fabric. To be complete, the fabric comprises defined subnets. In my opinion, the IB SM seems to be working like ARP cache and DHCP server in LANs. Each HCA in a fabric is globally identified with so-called node GUID which is like WWN in FC or MAC in LAN. The switch has own GUID as well. The ports of HCA have so-called port GUID. Now, when one HCA or its port want to communicate with another one in the subnet we need to have assigned some network address. This address is called LID or local identifier and the IB SM is in charge of assigning it to the members of the subnet. The conclusion is the LIDs are available inside the subnet only and the GUIDs are routable over the subnets of fabric.

But one thing confused me a bit. When you configure the switch you will need to remember setting its blueprint otherwise you will ask for trouble. I'm going to write about it in the next part.

Red Hat prefers KVM to XEN?

Wow, the situation around Red Hat's attitude to the virtualization maze seems to be more clear now. I thought that Red Hat is going to support solutions based on XEN hypervisor. In 2007, they released RHEL 5.0 and it was their first distro with integration of XEN hypervisor. And I was looking forward to it.

But Red Hat considered the XEN to be immature as well. According to the article published at www.virtualization.info the main reason for such decision was acquisition of XenSource by Citrix and the collaboration between Microsoft and Novell and other vendors interested in XEN.

A few days ago, Red Hat unveiled their new virtualization strategy based on embedding the KVM hypervisor to their RHEL distro. The official announcement is published here and summarized some advantages of it.

I think it is interesting news but with many unanswered but important questions. Will they support both of hypervisors? Or are they going to support KVM only since now? What about their customers who already adopted the XEN in their environments? In my opinion, it will be quite difficult to make it mainstream. Let's wait and we will see...

Sun Fire X4600 and ILOM update

A few weeks ago, I have to prepare another Sun Fire X4600 M2 servers for our customer. The configuration was quite straightforward. Anyway, I had to solve a little pitfall with server ILOM and BIOS update. Let's take a look at it.

The servers were preinstalled with ILOM/BIOS from Sun Fire X4600 M2 Software 2.0 (or SW2.0) which was released during May, 2008. This release contains ILOM version 2.0.2.5 and BIOS version 87. At this time, I decided to downgrade it. Why? According to the EIS standard the new version wasn't included on the latest EIS DVD yet. There is only SW1.3a. I managed the downgrade via ILOM web interface from Maintenance menu.

I'm aware of the fact that the ILOM and BIOS are dependent on each other. It is like with ALOM and OBP on SPARC. But I was not aware of the fact that when I begin the downgrade process then only ILOM will be downgraded. So before the downgrade the ILOM was at version 2.0.2.5 and BIOS at 87, after it the ILOM was at 2.0.2.1 and BIOS still at 87. However, the BIOS should be at 50.

The effect of the previous unsuccessful downgrade was that I wasn't able to power the server off from the ILOM console correctly. Finally, when I unplugged all power cords and then returned them back and powered the server on the POST wasn't initialized and I wasn't able to access the BIOS menu. The server was suddenly unusable. The following screenshot shows the power off issue:


It seems that ILOM at version 2.0.2.1 and BIOS at 87 are not friends. You can try to do the backward upgrade. But in my situation it didn't help. It looked like the BIOS was changed somehow. Even when I upgraded to the original ILOM 2.0.2.5 the server was still dead. This led me to the opinion that BIOS is upgradable only in one direction - to the higher versions, naturally.

Unfortunately, you can install so-called force-recovery jumper P19 at the mainboard. The official documentation is here. In my opinion, it causes deactivation of the BIOS version check during the update. Here is a list of steps I did:

1. I powered the server off.
2. I installed the P19 jumper.
3. I powered the server on.
   
4. I was waiting until the service led was on and the locate button began blinking for a couple of seconds (five?).
5. When they stopped blinking I powered the server off, removed the jumper and powered it on.
   

Applying the five steps above leads to the BIOS force recovery activation. When I repeated the update process the BIOS version became "unknown". The BIOS was flashed successfully and server started working again. The following two screenshots show the state before the update with force-recovery applied and after the update:

Quickly - XEN hypervisor and full virtualization

An interesting article about XEN hypervisor and support of full virtualization (or HVM) was published in the last Red Hat Magazine release. It summarizes three useful commands how to find if your system supports it (run these commands from dom0):

• is your Intel CPU utilizing VT-x extension?
• grep -i vmx /proc/cpuinfo
• is your AMD CPU utilizing Secure Virtual machine (or SVM) extension?
• grep -i svm /proc/cpuinfo
• finally, is your XEN hypervisor capable of HVM?
• grep -i hvm /sys/hypervisor/properties/capabilities

The previous commands aren't RHEL specific only. They have to work on any XEN installation. By the way, the /sys/hypervisor/ directory contains many other useful information. If you would like to find out which processors support HVM take a look at XEN wiki pages.

VMware ESX server 3.x and USB support

It's a pity that VMware ESX server 3.x doesn't support USB bus and so your virtual machines are not able to access the USB devices directly while the VMware Server or VMware Workstation is. However, there are some applications which depend on such devices, for example they require to check a licence file stored in USB token. There are many threads published at VMware communities about it. Let's check a few of them:

• Need USB Support
• map a USB pendrive to a VM
• Tips and Tricks
  

So, do we have any way how to solve it? As you can see in the above discussion threads the only possible way is to use specialized hardware which provides USB bus services over TCP/IP stack. Such devices called AnywhereUSB hubs are available at Digi International company site. It is sure that VMware is aware of the problem and provides the same resolution. You can check it at VMware resources page.

The only drawback is that the presented solution is supported on MS Windows platform only. The reason is the AnywhereUSB hub requires device driver which is available for MS Windows only. So the problem is solved by one-half. I would like to know how to achieve this on GNU/Linux ...

VMware Infrastructure 3.5 review

If you don't have enough spare time to look for and go through some interesting papers about VMware Infrastructure 3.5 check the review published at www.virtualization.info. It's really useful reading containing everything you need to know at first glance. The document also contains this nice summary of available Virtual Infrastructure 3.5 editions:

VMware ESX server 3.x snapshots - capacity planning

So, as we said the snapshot is really useful stuff. But when you take one you need to keep in mind a few rules which should help you to maintain it.

At first, it is useful to know the behaviour of our virtual machine. Maybe, it is growing fast one and that means the snapshot will grow fast as well. The question is how fast. Let's think about a virtual machine whose virtual disk is 10GB large. Then, its snapshot can be as large as the disk but not more. Only 10GB data can be modified. So, at worst, the virtual machine size may grow twice.

But you need to remember another thing. If you take a snapshot of virtual machine's memory (RAM) it needs some disk space as well (it has own delta file). The size of it depends, naturally, on the size of memory and assigned values to the memory reservation and limit parameters. Again, the snapshot may be as large as the size of memory but not larger. So, if you preconfigure the machine with 1GB of memory it is convenient to have 1GB of extra disk space.

How do the reservation and the limit influence it all? Everything which is beyond the reservation and below the limit and that is allocated memory by the virtual machine can be stored in the virtual machine's swapfile if there is no free physical memory available. The swapfile is, by default, stored in the virtual machine's datastore. And because the reservation, by default, has zero value the swapfile has the same size as the memory. So we need another 1GB of extra disk space. Overall, we need , theoretically, 12GB of free disk space.

Let's make some summary of rules:

1. fast growing virtual machine means fast growing snapshot - e.g. RDBMS, SMTP servers
2. slowly growing machines has slowly growing snapshots - e.g. firewalls
3. snapshots can't be larger than the virtual machine's virtual disks
4. don't forget about virtual machine's memory and swapfile
5. don't forget about taken snapshots in the past because long time snapshot will take much more longer time to commit the changes
   

VMware ESX server 3.x snapshots - introduction

As everybody knows who is in charge of any Virtual Infrastructure installation VMware ESX server allows to do snapshots of virtual machines. Simply, a snapshot of virtual machine is a saving of its current state which provides a way how to return from a future state to the saved states (the wikipedia is a good source for more information). The snapshot of virtual machine is able to keep its memory state, its settings state and its disk state. It is possible to maintain multiple snapshots and so-called "snapshots of snapshots" as well.

What is it useful for? Snapshots can help, for example, with patch management of virtual machines. You take a snapshot, apply the required patches, check the machine behaviour and then you will decide to revert the changes if something goes wrong or you commit them which means to consolidate the snapshot with the virtual machine's virtual disk.

And how is it working at all? When you create a snapshot of virtual machine then new or changing data aren't written to the virtual machine's virtual disk but are redirected to the so-called snapshot delta file. The filename of this file has form of VM_DISK_FILE-*-delta.vmdk. Let's mention when you take a snapshot of virtual machine's memory it will have its own delta file. These delta files are described with metadata stored in files like VM_DISK_NAME.vmsd. Beside this, it is created a snapshot state file for each snapshot of name VM_DISK_NAME-*.vmsn. It stores the actual running state of virtual machine at the time of taking the snapshot (the file is similar to the suspend state file with *.vmss suffix when you suspend the machine).

In the next part we will discuss the question of proper capacity planning to get rid of troubles with growing snapshots.