Thursday, March 27, 2008

RHEL and Infiniband support

The InfiniBand is a high speed, low latency switched fabric communications link. Its architecture specifies how to design interconnections between processor nodes (servers) and I/O nodes (storage devices). The interconnections are serial and bidirectional and have a point-to-point topology. More details are e.g. at

The article is a quick overview of InfiniBand and its support across RHEL distros. The InfiniBand technology preview was included in RHEL 4 update for the first time. It is based on OFED (OpenFabrics Enterprise Distribution) implementation from what is a validated version of open-source OpenFabrics software stack optimized for performance (with help of RDMA). The OFED is thoroughly tested and ready to be adopted by Linux vendors and their distros. The OFED contains required kernel modules and user space libraries and tools.

Back to the RHEL. More information about the first inclusion of InfiniBand implementatin in the RHEL 4 update 3 is written in its release notes. The most important is the notice that it is not supported for production environments due to a possibility of its API changes. The preview supports SDP (Sockets Direct Protocol), IPoIB (IP over InfiniBand) and RDMA (SCSI Remote Direct Memory Access) drivers. The implementation is splitted across a few RPM packages containing kernel modules, user space libraries and so on. Check the release notes or this FAQ entry.

Next, the update 4 of RHEL 4 contains updated OFED in revision 1.0. It is still not supported in production. The release 1.0 support wider range of hardware and iSCSI over InfiniBand driver. Check the release notes again.

The RHEL 4 update 5 is the first release of RHEL which supports InfiniBand in production. The OFED was updated to the revision 1.1. This release suppports only mthca-based (cards from Mellanox) InfiniBand HCA (Host Channel Adapter). Release notes are here. The latest revision of RHEL 4 contains updated OFED in version 1.2. Read the release notes please.

And what about the RHEL 5? It is quite similar. The initial release of RHEL 5 includes the OFED 1.1. Its first update contains the OFED 1.2. Both are considered stable and ready for production use. Their release notes are here for RHEL 5 and here for RHEL 5 update 1.

The latest stable revision of OFED is 1.3 and it was released at the end of february. I'm sure it will be included in the future updates of RHEL. Finally, here is a summary of included OFED revisions in RHEL releases:
  • RHEL 4 update 3 - technology preview of OFED
  • RHEL 4 update 4 - technology preview of OFED 1.0
  • RHEL 4 update 5 - fully supported OFED 1.1
  • RHEL 4 update 6 - fully supported OFED 1.2
  • RHEL 5 - fully supported OFED 1.1
  • RHEL 5 update 1 - fully supported OFED 1.2

No comments: