Raw devices on RHEL 5

I found out in a few discussion forums that many people are asking about raw devices configuration in the next generation of RHEL. A lot of them are thinking that support of raw devices was removed from the distro. They are right because Red Hat marked it as a deprecated and related configuration file and init script removed. It may be quite confusing for the people.

On the other hand, it introduced another way how to achieve the goal, newly via udev. I think it is more convenient than specifying raw device bindings in the /etc/sysconfig/rawdevices file which is then used by the /etc/init.d/rawdevices init script. Here it is a simple example of old bindings:

/dev/raw/raw1 /dev/sdc1
/dev/raw/raw2 8 34

The previous entries are passed to the /bin/raw command and corresponding bindings are created:

/bin/raw /dev/raw/raw1 /dev/sdc1
/bin/raw /dev/raw/raw2 8 34

To query created raw devices use the raw -qa command. To be more familiar with the raw command check the related man page (e.g. here). Here it is its output:

/dev/raw/raw1: bound to major 8, minor 33
/dev/raw/raw2: bound to major 8, minor 34

So, how to deal with the raw devices the new way? The udev "device manager" is a strong tool which is able to create device nodes on the fly according to kernel generated events. It can also run a set of defined commands if such a event happens.

The RHEL 5 introduced additional udev rules stored in /etc/udev/rules.d/60-raw.rules. This file can contain as many rules as you have bindings in your /etc/sysconfig/rawdevices file. Let's try to convert our two previous bindings to the new ones:

ACTION=="add", KERNEL=="sdc1",
RUN+="/bin/raw /dev/raw/raw1 %N"

ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="34",
RUN+="/bin/raw /dev/raw/raw2 %M %m"

Remember that udev rules cannot span multiple lines. The previous rules are splitted only due to its length (the RUN part is on the next line). If you specified a new rule and you want to activate it without reboot run udevtrigger command. Before you run udevtrigger check the rule with udevtest to see what's going on:

udevtest /block/sdc/sdc3

So, isn't it easy? From the presented examples it is clear that nothing changed. Only the /bin/raw command isn't called from the init script but directly from udev daemon. The syntax was changed the semantics remains the same.

RHEL and Infiniband support

The InfiniBand is a high speed, low latency switched fabric communications link. Its architecture specifies how to design interconnections between processor nodes (servers) and I/O nodes (storage devices). The interconnections are serial and bidirectional and have a point-to-point topology. More details are e.g. at wikipedia.org.

The article is a quick overview of InfiniBand and its support across RHEL distros. The InfiniBand technology preview was included in RHEL 4 update for the first time. It is based on OFED (OpenFabrics Enterprise Distribution) implementation from OpenIB.org what is a validated version of open-source OpenFabrics software stack optimized for performance (with help of RDMA). The OFED is thoroughly tested and ready to be adopted by Linux vendors and their distros. The OFED contains required kernel modules and user space libraries and tools.

Back to the RHEL. More information about the first inclusion of InfiniBand implementatin in the RHEL 4 update 3 is written in its release notes. The most important is the notice that it is not supported for production environments due to a possibility of its API changes. The preview supports SDP (Sockets Direct Protocol), IPoIB (IP over InfiniBand) and RDMA (SCSI Remote Direct Memory Access) drivers. The implementation is splitted across a few RPM packages containing kernel modules, user space libraries and so on. Check the release notes or this FAQ entry.

Next, the update 4 of RHEL 4 contains updated OFED in revision 1.0. It is still not supported in production. The release 1.0 support wider range of hardware and iSCSI over InfiniBand driver. Check the release notes again.

The RHEL 4 update 5 is the first release of RHEL which supports InfiniBand in production. The OFED was updated to the revision 1.1. This release suppports only mthca-based (cards from Mellanox) InfiniBand HCA (Host Channel Adapter). Release notes are here. The latest revision of RHEL 4 contains updated OFED in version 1.2. Read the release notes please.

And what about the RHEL 5? It is quite similar. The initial release of RHEL 5 includes the OFED 1.1. Its first update contains the OFED 1.2. Both are considered stable and ready for production use. Their release notes are here for RHEL 5 and here for RHEL 5 update 1.

The latest stable revision of OFED is 1.3 and it was released at the end of february. I'm sure it will be included in the future updates of RHEL. Finally, here is a summary of included OFED revisions in RHEL releases:

• RHEL 4 update 3 - technology preview of OFED
• RHEL 4 update 4 - technology preview of OFED 1.0
• RHEL 4 update 5 - fully supported OFED 1.1
• RHEL 4 update 6 - fully supported OFED 1.2
• RHEL 5 - fully supported OFED 1.1
  
• RHEL 5 update 1 - fully supported OFED 1.2

Quickly - PCI Express

Do you know how PCI Express work? Do you know how PCI Express slots look like or how do you figure how fast is a slot? I was looking for some picture or scheme of PCI Express slots and related speeds. Nothing more, just to be aware of it. I have found the following picture. More about it is written at computer.howstuffworks.com. Another comprehensive source of information is wikipedia.org.

Monitoring ASSP with monit

Do you know ASSP or Anti-Spam SMTP Proxy? I'm going to write some details about it in the near future. If you have deployed it on your servers to eliminate spams already, I will show you how to monitor it with the monit and restart it in case of a failure. The configuration was tested on Linux.

At first, I had to edit the init script of the service to be able to check its pid. The init script after the changes is below and the added lines are bolded:

#!/bin/sh -e
PATH=/bin:/usr/bin:/sbin:/usr/sbin

case "$1" in

start)
echo "Starting the Anti-Spam SMTP Proxy"
cd /usr/share/assp
perl assp.pl
ps ax | grep "perl assp.pl" | grep -v grep | awk '{ print $1 }' > /var/run/assp.pid
;;

stop)
echo "Stopping the Anti-Spam SMTP Proxy"
kill -9 `ps ax | grep "perl assp.pl" | grep -v grep | awk '{ print $1 }'`
rm -f /var/run/assp.pid
;;

restart)
$0 stop || true
$0 start
;;

*)
echo "Usage: /etc/init.d/assp {start|stop|restart}"
exit 1
;;

esac
exit 0

I know, there are many other ways how to do it better how to be compliant with the distro but I just want to show you how to configure the monit service. The monit service depends on it and it is used to define the service check block.

The assp service is listening at the TCP port 55555 by default which provides a simple configuration interface over HTTP protocol. The interface is authenticated so if you try to access it without proper authentication it will return a status code 401. It means client's authentication failure. You can get the whole error message via telneting to the port:

telnet localhost 55555
GET / HTTP/1.0

I used the HTTP protocol in version 1.0 and sent a GET request. If you want to use the version 1.1 you need to send the Host header as well. After pressing Enter and sending an empty line the request is processed and the following message is replied:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Anti-Spam SMTP Proxy (ASSP) Configuration"
Content-type: text/html
Server: ASSP/1.2.6()
Date: Mon, 25 Feb 2008 13:45:53 GMT
Content-Length: 49

...

We are going to be interested in the first line which contains already mentioned error code. The snippet of monit configuration code which monitors our service and the related process via pid file looks like:

check process assp with pidfile /var/run/assp.pid
start program = "/etc/init.d/assp start"
stop program = "/etc/init.d/assp stop"

It checks a pid of the process and if it is not running the service will be restarted. Now, we will extend it with the ability to check the connectivity to the port 55555:

check process assp with pidfile /var/run/assp.pid
start program = "/etc/init.d/assp start"
stop program = "/etc/init.d/assp stop"
if failed host 127.0.0.1 port 55555
then restart

But we would like to talk to the port with HTTP protocol. The above line is simple connectivity check over TCP protocol. Better is to do it via HTTP. The monit service support it and you can do it like:

check process assp with pidfile /var/run/assp.pid
start program = "/etc/init.d/assp start"
stop program = "/etc/init.d/assp stop"
if failed host 127.0.0.1 port 55555 protocol http
then restart

The above line is not the right one for us because it is suitable for unauthenticated environments. By default, it checks the return code only and it will be successful if it receive OK status or return code 200. To catch the return code 401 we need to redefine what we are going to expect. If you use a send/expect mechanism you need to omit "protocol http":

check process assp with pidfile /var/run/assp.pid
start program = "/etc/init.d/assp start"
stop program = "/etc/init.d/assp stop"
if failed host 127.0.0.1 port 55555
send "GET / HTTP/1.0\r\nHost: localhost\r\n\r\n"
expect "HTTP/[0-9\.]{3} 401 .*Unauthorized.*"
then restart

So, we construct the whole GET request and we expect the error code 401. If we receive anything else the monit service evaluates it as a connectivity failure and restarts the assp service. To be more fault tolerant it's better to check it twice, three times or more times to be really sure the service is not listening at the port 55555:

check process assp with pidfile /var/run/assp.pid
start program = "/etc/init.d/assp start"
stop program = "/etc/init.d/assp stop"
if failed host 127.0.0.1 port 55555
send "GET / HTTP/1.0\r\nHost: localhost\r\n\r\n"
expect "HTTP/[0-9\.]{3} 401 .*Unauthorized.*"
for 3 cycles then restart

That's everything. Why are we doing it like this? If the assp service is running the configuration interface should be accessible via TCP port 55555. Otherwise something is wrong and we should restart the service for sure.

SLES10 SP1 and pam session group errors

I was configuring a new backup server of our customer and I wanted to integrate it to the running LDAP infustructure. So I intented to configure it as a LDAP client and join it to the customer's LDAP server.

The backup server is based on SLES10 SP1 distribution and for such basic configuration tasks is equipped with YaST configurator. I can only recommend it if you don't want to waste time with simplicities! To configure a server as a LDAP client is really straightforward and you don't need to edit any config files like /etc/ldap.conf, /etc/nssswitch.conf or PAM config files and to remember exactly what to write where. Just fill in the proper options like address of LDAP server, LDAP base DN, if to use SSL/TLS, LDAP protocol version and confirm it. The screenshot ilustrates these options.


But sometimes troubles happen. After finishing the above process, remotely of course, everything seems to work as I expected. I was able to see LDAP users what was the main goal. Perfect!

The first problem which I noticed was that I wasn't able to connect to the server remotely via ssh once more. Debbuging of the connection didn't helped me. Why?!? Good question!

I had to inspect the server locally and I found the following errors in messages:

• sshd[8351]: Accepted publickey for root from A.B.C.D port 59203 ssh2
• sshd[8353]: pam_warn(sshd:session): function=[pam_sm_open_session] service=[sshd] terminal=[/dev/pts/0] user=[xxx] ruser=[] rhost=[yyy]
• error: PAM: pam_open_session(): Cannot make/remove an entry for the specified session

The errors are related to the sshd service and they were the result of unsuccesful connection. Another malfunctioned service was crond service and its errors were identical.

It is clear that something is wrong with PAM configuration of the services. In SLES10 and others distros the PAM modules are used for authentication, account and session processing of the most services. This behaviour of sshd daemon is affected with one option in the /etc/ssh/sshd_config config file:

• UsePAM yes

So I decided to try turning it off to make me sure I'm going in the right direction. The sshd service started working again but I wasn't still sure what's wrong with it. What should I do with the crond service to bypass the PAM modules?

I realized that the only thing I had changed before was the LDAP client configuration. I tried to bring the system to the previous state without it but it didn't helped. That means that when I had configured the LDAP client with YaST some operations weren't successful. Unfortunately, as I mentioned the LDAP client configuration is straightforward and you need to change only a few config files. Of course, you need to have installed the required packages with binaries and libraries.

I took a look over the configuration files and they seemed to be perfect. Only the /etc/pam.d/common-session file didn't contain any lines. This file is common for all other PAM config files and it is inluded from them. So, how to check its contents? Remember to use the rpm command in such situations. To check the validity of the pam package I run:

• rpm -V pam
  

It showed me that the file had to be changed:

• S.5....T c /etc/pam.d/common-session
  

The config file was different from the original one. The difference were these two missing lines:

• session required pam_limits.so
• session required pam_unix2.so

Finally, I tried to replace the modified file with the one from the installation source, put the support of PAM modules back and checked the services. They started to work again.

What's the result? Don't forget to use the strong tools like rpm and remember that simple things can go wrong too.

I need a platform for testing of clusters - a few screenshots

This post is only a continuation of the "I need a platform for testing of clusters" post and it contains some descriptive screenshots of actions taken during it.

• choose machine -> run "edit virtual machine settings" -> choose "add" button and follow "add hardware wizard" -> select "hardware type" to "hard disk"

• set "virtual disk type" to "SCSI" and check on "allocate all disk space now"

• select the disk from "hardware" tab after finishing "edit virtual machine settings" -> click "Advanced" button -> modify "Virtual Device Node" -> check it

Quickly - configuring VLANs on Linux

This post is similar to the one about configuring VLANs on RHEL but is applicable on any Linux distribution. VLANs support has been included into the Linux kernel since version 2.4.14. Earlier versions requires to be patched properly. More about it and patches is placed here. Again, configuration steps which requires to be done as root:

1. we have already configured eth0 interface which is accessible from the network and we want to add VLAN support to it, e.g. to accept packets tagged with VLAN ID 123 (avoid using VLAN ID 1, it is often used as administration VLAN)
2. we need to have support at the kernel level, try to load the proper module
• modprobe 8021q
3. you don't need to know how to modify configuration file's of interface eth0 because we are going to use the vconfig command to do it
4. to turn on VLAN ID 123 on the interface eth0, use the command:
• vconfig add eth0 123
5. check that the previous command was applied successfully, you can use ifconfig command of course:
• ifconfig eth0.123
6. finally, configure the remaining settings you want to set for the interface eth0, do it with ifconfig again
7. to make the changes persistent place the commands to some rc script or create your own init script and enable it during system boot
8. to check the status, kernel has exported some status information here:
• cat /proc/net/vlan/eth0.123
9. if you want to remove the interface, try these two:
• ifconfig eth0.123 down
• vconfig rem eth0.123

New and interesting features of VI 3.5 - Storage

You know, certainly that VMware released new version 3.5 of their flag product Virtual Infrastructure 3 recently. Let's take a glance what new features it brings and how they can help us in production environment.

As you could use VMware VMotion for hot migration of running virtual machines from one host server to another one, now you are able to migrate the virtual machine's disk from one storage to a second one. The technology is called SVM or Storage VMotion and it is useful in the maintenance period of your primary storage when you need to migrate stored virtual disks to your backup storage without disrupting your running virtual environment. The functionality is still new and you can take advantage of it only via Remote Command Line or RCLI. Until now, we were able to manage a virtual machine via console access or SSH (via ssh connection to an ESX host). RCLI is just another method how to do remote management. Unfortunately, Storage VMotion is available only for fibre channel storages.

The new version introduced support of N-port virtualization for fibre channel SANs which allows to each virtual machine to have its own WWN.

New VI3 supports new hardware as well. In my opinion, the most amazing one is support of SATA devices. It is recommended to check the new IO compatibility guide for supported SATA HBAs. Beside this, VMware includes technical support for InfiniBand (HCAs from Mellanox) and support for 10G ethernet.

The next feature is in experimental phase but I hope it will develop. Until know, multipathing with fibre channel was possible but only in a manner of failover. Now, you can test load balancing based on round-robin scheme.

To achieve better performance in your iSCSI SAN you can set jumbo frames and use TCP segmentation offload.

The last thing which merits our attention is to do VMotion of virtual machines without requirements to have their swap files stored on shared storage. Now, they can be placed locally.

I need a platform for testing of clusters

During last weeks I found out that I'm looking for a platform which could provide me a simple way how to test clusters and how to learn them in more details. I don't have spare money to buy high-cost hardware featuring high availability and shared storage by default. So I decided to choose another solution which is based on VMware server (or earlier VMware GSX server).

I don't want to spend many words for introducing VMware server. I suppose that everybody know it as well positioned virtualization technology suitable mainly for developers and testing purposes. The same holds for clustering. I intend to provide a simple way how to deploy high-availability cluster, nothing more. Additional information about VMware server can be found here and about clustering here. The documentation is accessible here.

Let's describe our initial configuration. I assume that VMware server is installed and that we have two virtual machines deployed. It's clear that two machines are the minimum for a cluster and that it doesn't matter if it is a windows or an unix machine. Each machine or better guest operating environment inside it is installed on a standalone virtual disk. Further, it never minds if it is a grow-able/preallocated or IDE/SCSI virtual disk.

Next, we need to reconfigure the machines and assign them a second virtual disk which we will use as our shared storage later. In fact, we will create only one new virtual disk, but we will assign it twice. So choose any from our two machines, run "edit virtual machine settings" and "add hardware wizard", select "hardware type" to "hard disk" and then follow the wizard. Here, we need to denote the first restrictions which we have to keep in mind during the new disk creation:

1. the new hard disk must be a SCSI one because the shared storage is implemented via SCSI reservation protocol
• it should be a SCSI virtual disk with preallocated disk space but the physical disk under it can be of any type (SCSI, SATA, IDE)
• it is possible to work with SCSI physical disk but its support is indicated as experimental so the previous point is the best choice
• the same holds for grow-able virtual disks it's experimental
2. it can't be realized via SCSI generic device
3. SCSI reservation is bounded to one SCSI bus
• SCSI bus used for shared disk should be different from the bus which is used for connection of system disks that's general recommendation
  

The point 1 and 2 is simple to pass with "add hardware wizard". We don't have to forget to set "virtual disk type" to "SCSI" and to check "allocate all disk space now" from the disk capacity pane. This is valid only for virtual disks. To pass the third restriction we need to edit advanced options of newly created disk so select the disk from "hardware" tab after finishing "edit virtual machine settings". Here we find an important attribute defining the SCSI disk address which is combination of SCSI controller ID and SCSI disk ID. The valid SCSI disk address looks like scsiX:Y where X identifies SCSI controller ID and Y disk ID. Controller ID of the system disk and of a shared disk should vary so if the first virtual disk or the system disk 0 has an address scsi0:0 and we add the second disk 1 which will have the address scsi0:1 we should change it to e.g. scsi1:0. We need to do it because SCSI addressing in VMware server works like that. If the last created disk has ID 10 and it resides on the controller 0, then the next one will have 11. Maximal SCSI ID is 15 and after 15 server will assign next SCSI controller ID that is 1 and will continue from disk ID 0 easily. The maximal SCSI controller ID is 3. Last obvious thing is that a chosen virtual machine have to be powered off otherwise we aren't able to create the new hard disk.

Now, we have a virtual machine with additional disk which is configured according to our restrictions and we need to assign it to the remaining machine. This is quite simple and corresponds with the process of adding new disk like above. The only difference is in the selection of "use an existing virtual disk" during the "add hardware wizard" if the created disk was a virtual disk and not a physical one. Then enter the file path of it and finish the wizard. For virtual machines in a cluster we have a few restrictions as well:

1. both virtual machines must reside on the same host. Yes, now you can understand why such solution is suited for testers preferably :-)
2. virtual machines support only SCSI-2 reservation protocol and not SCSI-3
   

The point 1 is evident and it implies from the restriction for SCSI bus boundary. Every SCSI bus is virtualized by VMware server and it is owned by it so it can't be spread over multiple servers. The point 2 is implementation restriction and it seems to be sufficient enough because all major clustering systems (e.g. MSCS, VCS) support SCSI-2 reservation protocol.

Let's continue with enabling SCSI reservation per our virtual machines. This step requires editing their configuration files. The configuration of a virtual machine is represented by a human readable configuration file with ".vmx" suffix. You can locate it with the "configuration file" entry at the summary tab of a virtual machine. The file is well defined and there are many options influencing machine's behaviour. A few of them are related to SCSI reservation so let's take a look at them. The options are disabled by default:

• scsiX.sharedBus = "virtual"

• scsiX:Y.shared = "true"

If you want to enable reservation globally per a SCSI bus on controller X, use the first parameter. Then everything or every hard disk attached to the bus will be considered as sharable. If you rather prefer to enable reservation for a selected hard disk only to global reservation choose the second one and the disk with address X:Y will be shared explicitly. If you combine both of them remember that the explicit sharing is not working because it is ignored then (restriction holds on the same bus). I must point out that the first method may be simpler but it is worse manageable. You aren't made to care about which disk on the bus should be shared because every disk will be shared. Further, every line you add to the configuration file of particular virtual machine have to be added to another one as well. The hard disk will be shared by all of them after all.

Enabling reservation isn't everything. Virtual machines are locking their hard disks to prevent concurrent access to them and to cause data corruption. There is a parameter which enable or disable this feature (it is true by default) and we need it for shared access:

• disk.locking = "false"
  

Beware of using it because it holds for every hard disk used by the virtual machine. The result depends on your proper planning and if you are using global SCSI bus sharing or explicit one. If you have configured global reservation then any virtual machine residing on the SCSI bus can possibly access that hard disk and change the data!

The last option is the possibility to redefine the reservation lock file name. The option is optional. It is used when reservation is enabled and it contains the shared state of the reservation of the current disk:

• scsiX:Y.fileName = "your_file_name_of_the_reservation_lock"

The thing is I don't know any useful reason to change it. And if so, it have to be changed on all virtual machines which are the members of the cluster. In conclusion, it will lead to troubles later.

The last thing is very important for keeping your "virtual data" healthy. It is recommended to turn off data caching at the level of virtual machines which are running in a cluster. Otherwise you risk their damage. The following option can achieve it (put the option to all ".vmx" files):

• diskLib.dataCacheMaxSize = “0”

The story is over and next time, I would like to describe particular usage of the previous scenario in some simple clustering environments. We have two virtual machines, they are sharing the virtual disk and we need to take advantage of it by installing some clustering aware software.

CentOS 4.6, Amavisd-new 2.5.3 update and troubles with Perl

Yesterday, I had a little trouble with one of my mail servers based on CentOS distribution enhanced with amavisd-new package from Dag Wieers's repository. The system had configured automated update and during the night the latest version of amavisd-new package was installed. It was 2.5.3 I think. Naturally, it led to the service restart and rereading required system libraries and dependent Perl modules.

The perl-MIME-tools package seemed to be the most critical part. It is responsible for disassembling MIME parts of messages containing multimedia attachments and it is used by amavisd-new content filter. I noticed that behaviour when the following error messages were appearing it the mail log:

• ... amavis[28144]: (28144-01) (!!)TROUBLE in check_mail: mime_decode-1 FAILED: Can't locate object method "seek" via package "File::Temp" at /usr/lib/perl5/vendor_perl/5.8.5/MIME/Parser.pm line 816 ...

I thought something strange had to happen to the File::Temp module. Perhaps its integrity could be broken. So I began checking the module! I was looking through the system update history but I didn't find anything updated recently. The module is part of the perl package and it wasn't updated recently nor there weren't any updates of it since the system's installation. Finally, I prove it by verifying the metadata of the package with command:

• rpm -V perl

It showed me the package wasn't broken and any file wasn't modified or corrupted. The File::Temp module was in version 0.14. You can check it like this:

• perl -le 'use File::Temp; print File::Temp->VERSION'
  

Further, I went through the update log again and noticed the package perl-MIME-tools was updated a few weeks ago as well. Its new version was 5.424.
The next phase was about trying to google through mailing lists to find anything related to the problem.

I found out a few people had the similar issues with the amavisd-new filter but nobody was sure how to solve it. The most denoted advice was to reinstall the File::Temp module.

I was deciding if I should use a latest version from the CPAN repository but Dag's repository contains standalone perl-File-Temp package with the module so I downloaded it. The package was in conflict with the perl package. The conflicting part was man page of the module. You don't have to install it or clearly you can bypass installing documentation of the package. Use this command to achieve it:

• rpm -Uvh --nodocs perl-File-Temp-0.20-1.el4.rf.noarch.rpm
  

I installed the module in version 0.20, restarted the service and nothing changed. The error was still there. So what next? The File::Temp module should be O.K., the perl package wasn't corrupted.

Let's try to check the MIME::Parser module which the error message mentioned as well. I installed the newest version from the CPAN which was 5.425. According to the changelog of the package this version solves some compatibility issues with tmp_recycling() method of the module. I restarted the service and it seemed to be working. The problem dismissed and the mail log was clean in the end.

To be sure with the previous steps I searched with Google once more. But now, I focused on misbehaviour between the File::Temp and MIME::Parser modules. I found this interesting article which mentions some incompatibilities between them and that new version of the File::Temp module is solving it.

The mail server and its content filter is healthy now. The troubles with Perl are away.