Solaris 10 updates summary

The seventh update of Solaris 10 was released on May. It contains support of Intel Nehalem CPU and some ZFS enhancements. I added it to the summary of Solaris updates .Here it is:

1. Solaris 10 1/06 (u1) - GRUB bootloader, iSCSI initiator, fcinfo command
   
2. Solaris 10 6/06 (u2) - ZFS filesystem
3. Solaris 10 11/06 (u3) - Solaris Trusted Extensions, LDoms
4. Solaris 10 8/07 (u4) - full TCP/IP stack in zones, iSCSI target, branded zones (Linux in Solaris container), Samba AD, enhanced rcapd
5. Solaris 10 5/08 (u5) - Intel SpeedStep, AMD PowerNow!, Solaris 8/9 P2V (to Solaris 10 zones), CPU capping
   
6. Solaris 10 10/08 (u6) - ZFS boot support, many ZFS filesystem enhancements
7. Solaris 10 5/09 (u7) - performance and power management support for Intel Nehalem CPUs, support of ZFS clones when cloning zones, IPsec SMF services, SunVTS 7.0 update
   

For more details, click the particular release to read the official release notes.

New Sun Fire servers with Xeon 5500

Sun has released a new line of servers and blade modules based on Intel Xeon 5500-series processors. The new pieces are:

• Sun Fire X2270 (1RU, 1 or 2 CPUs)
  
• Sun Fire X4170 (1RU, 1 or 2 CPUs)
  
• Sun Fire X4270 (2RU, 1 or 2 CPUs, 16 2.5" disks)
  
• Sun Fire X4275 (2RU, 1 or 2 CPUs, 12 3.5" disks)
  
• Sun Blade X6270 (1 or 2 CPUs)
  
• Sun Blade X6275 (4 CPUs)

The official announcement of new servers with additional details is published at www.sun.com.

Sun VirtualBox 2.2 released

The next version of Sun's desktop hypervisor VirtualBox was released. The new version 2.2 brings the following important changes:

• OpenGL 3D acceleration for Linux/OpenSolaris guests
• OVF appliance import/export
• USB and shared folder support for OpenSolaris
• host-only networking mode
  

More details are in official changelog.

Sun xVM Server postponed

While management system Ops Center 2.0 was released recently, it seems Sun has some issues with their XEN based hypervisor. According to this article published by french magazine LeMagIT it is going to be released in second quarter 2009.

Licensing open source

I was considering to write this article a while because it doesn't fit in any type of article I have published before. And it isn't my primary business to discuss various open source licensing here. The thing is, it is useful to understand the role of them but it is often quite difficult to imagine what they just want to say. Sometimes, I have a feeling you need a lawyer education to understand them.

You know the obvious questions like "why it has to be GPLed?", "why this license is not compatible with that one?" or "why it can't be part of Linux kernel?". You know the open source license ensures the availability of source code which you can modify and redistribute. The true pitfalls begin appearing when you would like to integrate two products available under two different licenses. To make things clearer I borrowed these two comprehensive schemes from chandanlog at Sun blogs. The first one presents general attitude of open source licenses and classical EULA to source code. The second one explains differences of open source licenses. They are quite minor but may have out of sight consequences.

Let's try to apply the licensing rules to the problem of releasing ZFS filesystem with Linux kernel. What's the problem? First, Sun owns some patent rights which prohibit such action. Second, as Linux kernel is GPLized anything included has to be GPLized as well. ZFS is covered with CDDL license which requires to be preserved. From here, I see the main reason of incompatibility. But if I realize there are other binary only modules like video drivers from ATI or NVIDIA which are linked with kernel via some sort of GPLized open source wrapper why we can't do it the same way with ZFS?!? The question is if it is legal.

The two practical schemes makes me to understand the topic more deeply. The example with ZFS made the situation complicated and I need to find out something what shows me that it is not. I hope you will find these graphical explanations as useful as mine. And check the chandanlog who created them!

Sun xVM Server 1.0 delayed?

Sun released their unified multiplatform management system for physical and virtual servers a few months ago but they still lacks of their own hypervisor called xVM. The release of xVM Server is planned during the first quarter of 2009 and it seems to be delayed now. Nevertheless, we can make some conclusion about the product now:

• it supports MS Windows, Linux and Solaris guests
  
• it is vmware compatible so you can directly use available vmware appliances
• it has built-in web-based management
• it supports virtual SMP (2 virtual CPUs)
• it supports live migration
• it supports resource pools
• it should be released with GPL3 licence at no cost (that's not a surprise as it is based on XEN)

I believe it will be released as soon as possible. Today, there are available source codes of xVM only. Binaries will be available with the oficial release of xVM.

Sun VirtualBox 2.1 is out

Sun has released new major version of Sun xVM VirtualBox product recently. The version 2.1.0 contains some interesting enhancements like:

• experimental support of 64b guest on 32b host
• experimental 3D accelration via OpenGL
• VMware VMDK virtual disks are supported now
• experimental support of LsiLogic and BusLogic SCSI HBAs (used with VMDK disks)

And that's not everything. There are many other things which worthwhile to read. The more comprehensive list of all enhancements and bug fixes is here.

Sun VirtualBox 2.0 is out

Today, Sun released new major version of their desktop virtualization product Sun VirtualBox 2.0. Newly, it supports 64-bit guests (only Windows Vista and RHEL5), Microsoft VHD virtual disk format, AMD RVI or Python API.

It can be downloaded from www.sun.com. The official release notes is available as well.

Sun released new servers and storage arrays

We had to wait for upcoming AMD Opteron servers from Sun a few months since the new quad-core AMD processors, code named Barcelona, were released. Now, it would be a few days when Sun officially announced here the availibility of the second generation of their AMD servers and new storage arrays, together called as "next generation open storage hardware". More about the Open Storage hardware and related projects, you can find here.

The new servers based on quad-core AMD processors are Sun Fire X4140, 4240 and X4540. At the storage field, there were introduced new Sun Storage J4200, J4400 and J4500 arrays. All of them are SAS JBOD arrays. For more details, look at the Sun System Handbook at SunSolve.

RHEL and Infiniband - software intro

Let's continue with software introduction. As I wrote the switch is equipped with the ALOM remote management. There is an universal set of commands for platform independent management like password, poweroff, setupsc, resetsc and so on and then a set of commands which are more specific to the platform. In the case of our IB switch there are two such commands:

1. setbp - for setting so-called blueprint of switch
   
2. showbp - for showing the current blueprint
3. there are five predefined blueprints:
• 9 node, 12 node, 18 node, none and unmanaged

The natural question is what does the blueprint mean? According to official documentation it seems to be like a predefined configuration of the switch. You can change it with the setbp command which asks you if you want to run IB management software, how many hosts will be in the subnet and what is the subnet identifier. By default, if you use the switch preconfigured from the factory then two switches will have the same subnet ID. The trouble is, if you intend to configure some level of redundancy between IB switches you will have to have them in different subnets with different subnet IDs. I think it strange because I had to disable the IB management software otherwise I wasn't able to see the nodes in the fabric. As we will see, the IB mangement software including IB subnet manager doesn't seem to like the OFED included in RHEL distro (more about RHEL and OFED I wrote here).

What about the servers? I preinstalled them with CentOS 5.1 distribution (which is binary compatible with RHEL 5.1). The distribution contains the OFED implementation in version 1.2. The complete OFED implementation in CentOS is divided in a set of RPM packages. The platform dependent part of OFED that means kernel modules are distributed with kernel package. Let's make a quick summary of basic packages:

1. kernel - contains IB hardware, IB core and IB ULP modules
• ULP means Upper Level Protocol
  
• everything is placed in the following directories:
• /lib/modules/`uname -r`/kernel/drivers/infiniband/hw
• /lib/modules/`uname -r`/kernel/drivers/infiniband/core
• /lib/modules/`uname -r`/kernel/drivers/infiniband/ulp
• currently there are supported only IB HCAs from Mellanox
• the supported ULPs are
  
• ipoib - IP over IB driver
  
• srp - IB SCSI RDMA initiator driver
  
• sdp - SDP driver
  
2. openib - this package contains a lot of useful documentation and the important part is the OFED configuration file /etc/ofed/openib.conf and the init script /etc/init.d/openibd which takes care of activating/deactivating the IB network interfaces. Simply, it loads the IB core modules and specified ULP modules in the config.
   
3. openib-diags - this package contains diagnostic tools for IB debugging, I will introduce them later.
4. opensm - here we have our IB subnet manager. The package provides the init script /etc/init.d/opensmd for starting it and the /etc/ofed/opensm.conf configuration file.
5. libibverbs - this package provides a library allowing userspace programs direct hardware access.
6. libibcommon, libibmad, libibumad, opensm-libs - and finally library dependencies for the above packages.
   

I need to add that the OFED packages belongs to the System Environment/Libraries RPM group and they are not installed by default apart from the openib and libibverbs and of course kernel package. That's all for now and next time I'm going to describe how to work with it.

RHEL and Infiniband - hardware intro

In my two previous articles, I summarized a few facts about the Infiniband support in RHEL distros and included protocols - you can go through them from the following links - RHEL and Infiniband support and Infiniband, RDP, SDP.... Let's be more particular now.

My scenario was based on two servers Sun Fire X4200 M2 and one Infiniband (IB) switch Sun IB Switch 9P. The servers had installed Infiniband host channel adapters (HCA) Sun Dual Port 4x IB HCA to be able to communicate over the IB fabric. The switch provides nine IB compliant ports at dual speeds of 4X/12X what means that each port is able to deliver of 10/30Gbit raw bandwidth. What surprised me was that the switch management is like at the SUN SPARC midrange servers. Yes, it is ALOM and it is perfect because you can use the same interface and similar commands you are used to. By the way, the switch chassis looks like a regular SUN server.

The switch is equipped with the IB subnet manager (SM) which is required to initialize the IB hardware and to allow the communication over the IB fabric. Each IB subnet has to have at least one and each has unambiguous identifier (ID) over the fabric. To be complete, the fabric comprises defined subnets. In my opinion, the IB SM seems to be working like ARP cache and DHCP server in LANs. Each HCA in a fabric is globally identified with so-called node GUID which is like WWN in FC or MAC in LAN. The switch has own GUID as well. The ports of HCA have so-called port GUID. Now, when one HCA or its port want to communicate with another one in the subnet we need to have assigned some network address. This address is called LID or local identifier and the IB SM is in charge of assigning it to the members of the subnet. The conclusion is the LIDs are available inside the subnet only and the GUIDs are routable over the subnets of fabric.

But one thing confused me a bit. When you configure the switch you will need to remember setting its blueprint otherwise you will ask for trouble. I'm going to write about it in the next part.

Sun Fire X4600 and ILOM update

A few weeks ago, I have to prepare another Sun Fire X4600 M2 servers for our customer. The configuration was quite straightforward. Anyway, I had to solve a little pitfall with server ILOM and BIOS update. Let's take a look at it.

The servers were preinstalled with ILOM/BIOS from Sun Fire X4600 M2 Software 2.0 (or SW2.0) which was released during May, 2008. This release contains ILOM version 2.0.2.5 and BIOS version 87. At this time, I decided to downgrade it. Why? According to the EIS standard the new version wasn't included on the latest EIS DVD yet. There is only SW1.3a. I managed the downgrade via ILOM web interface from Maintenance menu.

I'm aware of the fact that the ILOM and BIOS are dependent on each other. It is like with ALOM and OBP on SPARC. But I was not aware of the fact that when I begin the downgrade process then only ILOM will be downgraded. So before the downgrade the ILOM was at version 2.0.2.5 and BIOS at 87, after it the ILOM was at 2.0.2.1 and BIOS still at 87. However, the BIOS should be at 50.

The effect of the previous unsuccessful downgrade was that I wasn't able to power the server off from the ILOM console correctly. Finally, when I unplugged all power cords and then returned them back and powered the server on the POST wasn't initialized and I wasn't able to access the BIOS menu. The server was suddenly unusable. The following screenshot shows the power off issue:


It seems that ILOM at version 2.0.2.1 and BIOS at 87 are not friends. You can try to do the backward upgrade. But in my situation it didn't help. It looked like the BIOS was changed somehow. Even when I upgraded to the original ILOM 2.0.2.5 the server was still dead. This led me to the opinion that BIOS is upgradable only in one direction - to the higher versions, naturally.

Unfortunately, you can install so-called force-recovery jumper P19 at the mainboard. The official documentation is here. In my opinion, it causes deactivation of the BIOS version check during the update. Here is a list of steps I did:

1. I powered the server off.
2. I installed the P19 jumper.
3. I powered the server on.
   
4. I was waiting until the service led was on and the locate button began blinking for a couple of seconds (five?).
5. When they stopped blinking I powered the server off, removed the jumper and powered it on.
   

Applying the five steps above leads to the BIOS force recovery activation. When I repeated the update process the BIOS version became "unknown". The BIOS was flashed successfully and server started working again. The following two screenshots show the state before the update with force-recovery applied and after the update:

Sun Fire V490 and RSC not responding

I was preparing Sun Fire V490 server for our customer and I wanted to configure RSC controller at first to be able to continue my work remotely. So I installed RSC packages and run the following command:

• /usr/platform/`uname -i`/rsc/rsc-config

This command, or better shell script which is calling rscadm command from the same directory, is responsible for initial setting of RSC controller properties like network interface and serial port settings, notifications, users and so on. After a few questions are answered the script will flash the RSC PROM to activate the provided values.

My trouble appeared in the phase of PROM flashing. It finished unconditionally with some errors mentioning a problem with network interface settings. Then I tried to run the command again after checking the entered settings and ended with this error message:

• rscadm: RSC firmware not responding

I tried to wait a few minutes at first because I supposed the controller is restarting. Then I tried to restart it with rscadm:

• rscadm resetrsc
  

But with the same result like above. So I rebooted the server to OK prompt and tried to restart the RSC controller from it with the command:

• reset-rsc

I booted the server to single user mode and tried the rscadm command again. Again without any response. Finally, I decided to flash the PROM by hand via:

• rscadm download ../lib/images/rscfw

I run that command from the directory:

• /usr/platform/`uname -i`/rsc/

Finally, the flashing began and finished after a few minutes with success and I was able to proceed with RSC reconfiguration. Successfully as well.

I' m surprised how rscadm behaves. Any option ended with no response but download option used for RSC PROM flashing was still working and solved the whole problem. More about RSC (version 2.2) is published at docs.sun.com. How to download and install RSC is placed at www.sun.com.

Sun Fire 4600 M2 and paired DIMM mismatch

I was working on configuration of Sun Fire 4600 M2 server and I was surprised when the message "Paired DIMM Mismatch" appeared during server's initial boot. I understood the warning but I wasn't sure with the cause of it. I received the server with 4 CPU boards each with 4 GB memory installed. Beside this, I received additional 24 2GB memory modules (or 12 4GB memory kits) and another 4 CPU boards. I populated the server with new CPU boards after installing additional memory modules onto them (together every board had 8GB memory installed) and powered on the server.

System boards installed in modern servers are often equipped with a BMC or "Board Management Controller". This circuit is responsible of managing the interface between system management software and hardware platform. BMC relies on many hardware sensors reporting such parameters as temperature, fan speeds, power mode and so on. The BMC is the intelligence in the IPMI or "Intelligent Platform Management Interface" architecture. In my opinion BMC is not the same as SC or "System Controller" in Sun terminology which is ILOM but it is separated controller.

Above is a screenshot of BMC response after BIOS POST startup. BMC warned me that some CPU boards' memory module pairs weren't combined well. This event can lead to degraded performance due to no optimal memory module pair interleaving.

According to the "Sun Fire X4600/X4600 M2 Servers Diagnostics Guide" documentation there exists a couple of identified errors related to memory modules:

• NODE-n Paired DIMMs Mismatch
• modules in pair aren't the same or the checksum are different
  
• NODE-n Memory Configuration Mismatch
• modules are not in pair (they are running in 64-bit mode instead of 128-bit)
• modules don't support ECC
• modules' speed is different
• modules are not registered
• modules' type/generation/organization or CL/T is mismatched
• the banks on a two-sided module are mismatched
• and the others ...
  
• NODE-n DIMMs Manufacturer Mismatch
• module's manufacturer is not supported

After removing affected CPU boards from the server and inspecting the memory module pairs the situation was clear to me. I forgot to check the memory modules vendors because I supposed they are all the same. So I reorganized them properly, put the boards back and then everything was working smoothly.