Quickly - persistent modules loading on RHEL

The kernel modules required for booting the system up are part of an initial  ramdisk which is automatically loaded into the memory by a boot loader. The ramdisk contains enough modules to mount the root filesystem and to initialize essential devices like keyboard, console or   various expansion cards.  The boot process  then continues with running the init process.

During the next phase, the other modules referenced by the operating system  are loaded automatically.  The modules are called by their aliases specified and set in the /etc/modprobe.conf configuration file. The typical alias is e.g. eth0 for a network interface card or usb-controller for an USB controller.

If we need  to load some specific module during the system boot and there isn't a way to reference it we have a few choices how to do it.

• Place a particular modprobe command to the /etc/rc.d/rc.local script which is called at the end of the whole boot process. But it is likely to be late at this phase.
• Or better, place the command in the  /etc/rc.modules file which is read and executed by the /etc/rc.d/rc.sysinit initialization script during the system initialization phase. It may be better to load the modules as soon as possible.

The /etc/rc.modules does not exist by default, so at first create it and make it executable. I think the first method is commonly used by many of us but the second one is in my opinion more systematical.

ESXi log files

What is the fastest way to retrieve log files from an ESXi host? In my opinion, the best way is to configure remote logging via syslog server but this requires host reboot to apply configuration changes (KB1016621). The alternative method is to forward log files to different datastore. 

If  you don't have prepared syslog server for remote logging you can use vsphere client and generate system log bundles for particular host. But this takes some time. 

The last method is I think the fastest one because it will allow you to access log files  directly with your web browser. You can use web interface of the ESXi host,  enter the following URL:

https://ESXi_HOST_ADDR/host

The output should looks like shown at this picture:

You can download ESXi log files  messages, hostd.log and vpxa.log  now.

vMA missing libraries

If you are using vMA (vSphere Management Assistant) for some specific management tasks like UPS monitoring  or running a scheduled backup script from cron daemon, you may experience an error similar to this one:

Can't load '/usr/lib/perl5/site_perl/5.8.8/libvmatargetlib_perl.so'
for module vmatargetlib_perl: libtypes.so: cannot open shared object
file: No such file or directory at
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.
at /usr/lib/perl5/site_perl/5.8.8/VMware/VmaTargetLib.pm line 10
Compilation failed in require at /usr/lib/perl5/site_perl/5.8.8/VMware/VIFPLib.pm line 10.

A reason for such  behaviour is typically caused by some misunderstandings how shell environment in vMA is configured. The most common mistake is testing the affected script with sudo which strips out some environment variables - especially LD_LIBRARY_PATH - due to some security restrictions. Otherwise, the error shouldn't appear because /etc/bashrc exports vmware SDK library path implicitly:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib

So in  case of sudo or other unspecified scenarios throwing the presented error try to create a wrapper script which explicitly exports a list of directories where to search for ELF libraries again:

#!/bin/bash

LD_LIBRARY_PATH=/opt/vmware/vma/lib64:/opt/vmware/vma/lib  
export LD_LIBRARY_PATH

/path/to/original-script "$@"

exit $?

VCB basic usage - debugging

During the series of articles about VCB usage I supposed that all the presented VCB command examples are running smoothly and without errors. But this is not always true. There can  be a lot of reason why it is not working as expected, e.g. wrong permissions assigned to VCB backup user, misconfigured SAN which doesn't allow to access VMFS volumes or  unspecified problem with creating virtual machine snapshot.

If something goes wrong all VCB commands can be run in more verbose mode with command line switch -L and verbosity level from 0 to 6. The next example illustrates it. We want to perform a full backup of a virtual machine named vcb-backup and it seems the provided user vcbadmin doesn't have required permissions to do it:

vcbmounter -h host -u vcbadmin -p pass -a name:vcb-backup -r c:\mnt\vcb-backup -t fullvm -m nbd -L4

[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Connected using API Namespace vim25.
[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Authenticating user vcbadmin
[2010-06-15 13:15:12.859 'vcbMounter' 360 info] Logged in!
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got VM MoRef
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got access method
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got coordinator object
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Attempting data access.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating mount directory
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] No snapshot info for this VM, nothing to do.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating snapshot
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Snapshot created, ID: snapshot-579
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Mount operation created snapshot.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Mount operation obtained backupinfo.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:19.328 'vcbMounter' 360 info] Successfully obtained instance lock.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] Error: No permission to perform this action.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] An error occurred, cleaning up...
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Successfully obtained instance lock.
[2010-06-15 13:15:29.437 'vcbMounter' 360 info] Remove clone disks successful.
Deleted directory c:\vcb\vcb-backup

The bold line helps us to identify the cause of the problem.

YUM download only mode

How many times I was in a situation I needed to update a server with RHEL installed but I wasn't at site and I didn't have a way how to reboot the server after installing a new kernel or glibc package on it reliably? Yes, I have a test environment and I'm testing the updates on it but many installations are too critical to just run yum update -y and then shutdown -r now. On top of that, there are well known Murphy's laws which are able to damage more than we are able to imagine.

Instead of remote resolution of why the server is suddenly unresponsive I'm trying to prepare some offline update archive (if there isn't an update server available but this is another situation) and then during a site visit to apply it.

As I'm talking about RHEL I'm using YUM or Yellowdog Updater Modified for it. This tool is able to download updates locally without installing them if we have RHEL 5.x system. It only requires to install a download plugin which is part of yum-downloadonly package. Try to install it with


yum install yum-downloadonly

The next lines contain common commands that I use for downloading updates:

yum install PACKAGE_NAME.rpm -y --downloadonly
yum update -y --downloadonly


If we have a RHEL 4.x server we don't have this package and we need to install another package called yum-tools which contains similar tool yumdownloader.


yum install yum-tools -y

Here it is how to use the tool

yumdownloader PACKAGE_NAME.rpm


If we wan't to download all the available updates with yumdownloader we need to get a list of all packages with yum check-update and then to pass it to yumdownloader. You can do it from shell with sed, cut or awk commands or what would you prefer:

for PKG in `yum check-update | cut -d' ' -f1`; do
   yumdownloader $PKG
done

For more detailed description of the tools and their parameters have a look at their man pages.

Time to continue

After almost a whole year, I would like to awake my blog from a sleep and begin to publish articles more regularly. During the previous period I was too busy at work and I couldn't afford to take care of it as I would like. On the other hand, I had an opportunity to participate on some interesting projects and I needed to solve many unusual tasks. Everything was written down and now it is waiting for readers and comments.

I hope the new articles will be interesting and helpful for you as the previous ones.

Red Hat Enterprise Linux 5.5 - what's new?

It's a few months since RHEL 5.5 was released (march, 2010). Despite this, I would like to point out the major changes and additions compared to the previous release RHEL 5.4. So what's new:

• Kickstart installation - it is possible to exclude package groups in the same way like single packages.
• KVM guests and Cluster Suite - management of KVM based virtual guests with Cluster Suite is supported.
• SPICE - RHEL 5.5 includes components of Simple Protocol for Independent Computing Environments which is competitor for VMware's PCoIP or Citrix's HDX.
• PCI passthrough - physical PCI devices attached to virtual guests are working better.
• Huge page support - it is extended to virtual guests with libvirt.
• Windows 7 support - new samba3x packages supporting Windows 7 are included.

For more details read the RHEL 5.5 official release notes.

VMware Workstation 7.1 enhancements

I have decided to update my current VMware workstation 7.0.1 installation I have on my desktop to the latest available release 7.1. Further follows a quick comparison of the most interesting features and enhancements between these two releases:

• WS7.1 officially support many latest versions of well-known Linux distros like RHEL/Debian/Ubuntu
• WS7.1 supports OVF1.1 via ovftool which is part of the installer
• WS7.1 has higher limits for guest's vCPUs - from 4 to 8 and guest's virtual disk - from 950GB to 2TB
• WS7.1 is capable of automatic downloads and updates of WMware Tools

Eventually, I would like to remember what new features VMware Workstation 7.0 came with:

• 32GB of guest's virtual RAM
• cross-platform license serial numbers
• pause running guest instead of suspending it or even powering it off
  
• import of Windows XP mode virtual machine
• GUI editor for virtual networks
• support for Windows 7 as guests
• support ESX hypervisor running as guest

That's not everything at all. The official release notes for VMware Workstation 7.1 and 7.0 are more comprehensive.

SLES 11 SP1 released

I have decided to write a brief summary of new features and enhancements which are available with the first service pack of Novell's SLES 11. I need to know the major differences between GA and SP1 release during my every day work and perhaps, it will help you in the same way like me. The original post about SLES11 GA is here. So, what's new?

• it is based on GNU/Linux kernel 2.6.32
• it provides web based YaST for remote management called WebYaST
• UEFI booting (useful with disks larger as 2TB) is supported on AMD64/Intel64 now
• it includes many driver updates (e.g., QLogic/Emulex HBAs, Broadcom NICs)
• it includes new XEN 4.0
• KVM hypervisor is now fully supported, good news
  
• it fully supports the latest enterprise Intel XEON 5600 and 5700 processors which are able to greatly improve performance of XEN hypervisor by decreasing latency of VMs
• finally, it contains all the latest security and bug fixes available since the release of GA

It seems there weren't so many changes included in SP1 but I think XEN 4.0 or KVM support from Novell is enough to move our GA installations to SP1. With XEN 4.0, we are able to benefit from these nifty features:

• fault tolerance
• memory overcommitment
• USB paravirtualization and VGA pass-through
  
• live snapshots and clones
• 64 vCPUs per VM
• 1TB RAM per XEN host
  

Now, it depends how fast and how successfully Novell will be able to implement and adjust these features into their management tools like YaST.

For more comprehensive contents about SLES11 SP1, check the oficial release notes at www.novell.com.

Second edition of VMware Site Recovery Manager is out

The second edition of VMware SRM, officially named as VMware vCenter Site Recovery Manger 4, was released recently. The product is responsible for automated disaster recovery of complex virtual environments. The recent version is fully compatible with VMware vSphere platform and provides these new important features:

• many-to-one failover - this means that one site is able to recover from multiple sites failures
• expanded support of storage vendors - those who provides storage replication solutions over FC, iSCSI or NFS, 12 generally. Among them belongs DELL, IBM, HP, EMC, LSI and others.

For more information on VMware SRM 4 visit the release notes and product home page.