VMware VDR licensing

Here I wrote about availability of VMware VDR in almost all the editions of vSphere - newly in Standard edition as well. VDR provides agent-less backups of  virtual machines and in-line block based destination deduplication.

This license change applies for all installations or upgrades to the latest release 4.1 update 1 of ESXi hypervisor. The  screenshot below shows the licensed features of one of my ESXi servers with  ESXi 4.1 update 1, build 348481, standard edition.

VMware Data Recovery

How do you backup your virtual infrastructure based on VMware platform? Do you still rely on VCB? Even if the VCB is still supported and it is compatible with the latest ESX/ESXi hypervisor at version 4.1u1 it should be discontinued from the next release. So what backup tool to use in the future? 

VMware vDR is considered to be its successor and in my opinion, the latest release is working pretty stable without any major issue. It's missing some features now like email reporting or GUI FLR restore tool but I think we can expect the future release will include them as the competing backup tools don't miss such basic things.

But what I consider the most important for better adoption of vDR in datacenters is that VMware corrected their licenses recently and vDR is now included in Standard edition as well. This means that vDR is available in all vSphere editions except vSphere Essentials kit.

ESXi log files

What is the fastest way to retrieve log files from an ESXi host? In my opinion, the best way is to configure remote logging via syslog server but this requires host reboot to apply configuration changes (KB1016621). The alternative method is to forward log files to different datastore. 

If  you don't have prepared syslog server for remote logging you can use vsphere client and generate system log bundles for particular host. But this takes some time. 

The last method is I think the fastest one because it will allow you to access log files  directly with your web browser. You can use web interface of the ESXi host,  enter the following URL:

https://ESXi_HOST_ADDR/host

The output should looks like shown at this picture:

You can download ESXi log files  messages, hostd.log and vpxa.log  now.

vMA missing libraries

If you are using vMA (vSphere Management Assistant) for some specific management tasks like UPS monitoring  or running a scheduled backup script from cron daemon, you may experience an error similar to this one:

Can't load '/usr/lib/perl5/site_perl/5.8.8/libvmatargetlib_perl.so'
for module vmatargetlib_perl: libtypes.so: cannot open shared object
file: No such file or directory at
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.
at /usr/lib/perl5/site_perl/5.8.8/VMware/VmaTargetLib.pm line 10
Compilation failed in require at /usr/lib/perl5/site_perl/5.8.8/VMware/VIFPLib.pm line 10.

A reason for such  behaviour is typically caused by some misunderstandings how shell environment in vMA is configured. The most common mistake is testing the affected script with sudo which strips out some environment variables - especially LD_LIBRARY_PATH - due to some security restrictions. Otherwise, the error shouldn't appear because /etc/bashrc exports vmware SDK library path implicitly:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib

So in  case of sudo or other unspecified scenarios throwing the presented error try to create a wrapper script which explicitly exports a list of directories where to search for ELF libraries again:

#!/bin/bash

LD_LIBRARY_PATH=/opt/vmware/vma/lib64:/opt/vmware/vma/lib  
export LD_LIBRARY_PATH

/path/to/original-script "$@"

exit $?

VCB basic usage - debugging

During the series of articles about VCB usage I supposed that all the presented VCB command examples are running smoothly and without errors. But this is not always true. There can  be a lot of reason why it is not working as expected, e.g. wrong permissions assigned to VCB backup user, misconfigured SAN which doesn't allow to access VMFS volumes or  unspecified problem with creating virtual machine snapshot.

If something goes wrong all VCB commands can be run in more verbose mode with command line switch -L and verbosity level from 0 to 6. The next example illustrates it. We want to perform a full backup of a virtual machine named vcb-backup and it seems the provided user vcbadmin doesn't have required permissions to do it:

vcbmounter -h host -u vcbadmin -p pass -a name:vcb-backup -r c:\mnt\vcb-backup -t fullvm -m nbd -L4

[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Connected using API Namespace vim25.
[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Authenticating user vcbadmin
[2010-06-15 13:15:12.859 'vcbMounter' 360 info] Logged in!
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got VM MoRef
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got access method
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got coordinator object
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Attempting data access.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating mount directory
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] No snapshot info for this VM, nothing to do.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating snapshot
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Snapshot created, ID: snapshot-579
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Mount operation created snapshot.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Mount operation obtained backupinfo.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:19.328 'vcbMounter' 360 info] Successfully obtained instance lock.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] Error: No permission to perform this action.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] An error occurred, cleaning up...
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Successfully obtained instance lock.
[2010-06-15 13:15:29.437 'vcbMounter' 360 info] Remove clone disks successful.
Deleted directory c:\vcb\vcb-backup

The bold line helps us to identify the cause of the problem.

VMware Workstation 7.1 enhancements

I have decided to update my current VMware workstation 7.0.1 installation I have on my desktop to the latest available release 7.1. Further follows a quick comparison of the most interesting features and enhancements between these two releases:

• WS7.1 officially support many latest versions of well-known Linux distros like RHEL/Debian/Ubuntu
• WS7.1 supports OVF1.1 via ovftool which is part of the installer
• WS7.1 has higher limits for guest's vCPUs - from 4 to 8 and guest's virtual disk - from 950GB to 2TB
• WS7.1 is capable of automatic downloads and updates of WMware Tools

Eventually, I would like to remember what new features VMware Workstation 7.0 came with:

• 32GB of guest's virtual RAM
• cross-platform license serial numbers
• pause running guest instead of suspending it or even powering it off
  
• import of Windows XP mode virtual machine
• GUI editor for virtual networks
• support for Windows 7 as guests
• support ESX hypervisor running as guest

That's not everything at all. The official release notes for VMware Workstation 7.1 and 7.0 are more comprehensive.

Second edition of VMware Site Recovery Manager is out

The second edition of VMware SRM, officially named as VMware vCenter Site Recovery Manger 4, was released recently. The product is responsible for automated disaster recovery of complex virtual environments. The recent version is fully compatible with VMware vSphere platform and provides these new important features:

• many-to-one failover - this means that one site is able to recover from multiple sites failures
• expanded support of storage vendors - those who provides storage replication solutions over FC, iSCSI or NFS, 12 generally. Among them belongs DELL, IBM, HP, EMC, LSI and others.

For more information on VMware SRM 4 visit the release notes and product home page.

VMware Server 1.0.x library dependency problem

In the beginning of the year, I wrote this article about some problems between older VMware server 1.0.x and newer Linux distributions. The problem is related to the vmware kernel modules whose source code are not compatible with newer Linux kernels.

I was surprised with one thing. When I upgraded VMware Server from version 1.0.8 to 1.0.9, VMware Server console stopped working. The new version was installed on the same system (OpenSUSE 11.1) as the old one, so I don't understand the reason. The important thing is I found a solution. The new version began producing these new error messages after trying to run vmware command:

/usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)
/usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)
/usr/lib/vmware/bin/vmware: symbol lookup error: /usr/lib/libgio-2.0.so.0: undefined symbol: g_thread_gettime

I have tried to unset this environment variable influencing behavior of GTK2 applications:

unset GTK2_RC_FILES

Otherwise, the variable is referencing related gtkrc files defining GTK2 user's environment. Try it and I hope it will help.

VMware vSphere hotplug

Hotplug of virtual hardware is attractive feature of VMware ESX 3.x/4.x. In case of ESX 3.x it is limited to hotadd of virtual disk to a running virtual machine only. With next generation of VMware vSphere hypervisor you are able to hotadd of memory or CPU to a machine if guest operating system supports it.

I was surprised during vSphere evaluation how it pretty works. I used to hotadd of virtual disks to my machines quite often. But when I upgraded 3.5 infrastructure to new vShpere 4.0, I became disappointed because it stopped working.

The reason why is simple. Hotadd feature is available from advanced edition only and I was upgrading to standard edition which doesn't contain license for it. You can check it in my previous post VMware vShpere 4.0 editions . Below is an error message which is complaining about missing license:

I think it wasn't a right decision to shift the feature to the higher editions. I think it would be better to leave things where they were because people are used to using them. And I hope that VMware will push back at least hotadd of virtual disk in some future release of vSphere.

VMware or Citrix?

Citrix released their virtualization solution named XenServer (from version 5, article XenServer is free) for free but only the time will show if it was a right decision. At first glance, it seems like a marvelous thing but there are some facts which should be investigated first. Together with XenServer, it was released central management solution XenCenter.

Let's have a look at their rival VMware (vSphere 4). XenServer is fully comparable to VMware ESX or ESXi. But what about XenCenter management? It's something more than VMware vSphere client but not so valuable as VMware vCenter Management Server. Citrix XenCenter is not the right choice in case of comparison to vCenter. The right one is Citrix Essentials but this one is not for free already. The main differences between Citrix XenCenter and Essentials are:

• XenCenter is missing alerting capabilities like send me an email when "CPU usage is too high" or when some error condition like "virtual machine power on failure" appears
• XenCenter is missing high availability support
• XenCenter is not able to show you performance data older than one day for physical or virtual servers
  

Now, let's try to propose a simple high availability (HA) solution based on Citrix/VMware products and compare their prices. Let's suppose we have 2 (3) entry level servers where each have 2 CPUs with max 6 cores per CPU (6 CPUs total). The servers are connected to a shared disk storage. The CPU speed or memory capacity is not important now. And we require HA solution to protect our virtual machines from hardware failure. Follows the analysis:

- Citrix Essentials Enterprise (1 license = 1 server):

• XenServer - 2 licenses = 0$ (3 lic = 0$)
• Essentials Enterprise - 2 lic = 5500$ (3 lic = 8250$)
• Essentials Preffered Support (optional) - 1 lic = 1500$
  
• Total cost = 7000$ or 9750$ for 3 servers
  
• Total cost without support = 5500$ or 8250$ for 3 servers

- VMware vSphere 4 Standard Edition (1 lic = 1 CPU):

• vSphere 4 Standard - 4 licenses = 3180$ (6 lic = 4770$)
• vShpere 4 Standard 1y Gold Support - 4 lic = 1092$ (6 lic = 1638$)
• vCenter 4 Foundation - 1 lic = 1495$
• vCenter 4 1y Gold Support - 1 lic = 545$
• Total cost = 6312$ or 8448$ for 3 servers
• Support is mandatory

- VMware vSphere 4 Essentials Plus Bundle (1 lic = 1 CPU)

• Licenses for 3 hosts plus vCenter Server for Essentials plus 1y Gold Support = 3624$
• Total cost = 3624$ for 2 or 3 servers
  

The prices of proposed solutions are quite different. In my opinion, the most valued solution is based on new VMware product line vSphere 4 Essentials.

There are rumors that VMware is the most expensive solution. I don't think so if I check the numbers above. Citrix's solution not covered by support is cheaper then VMware's solution with support but only for 2 servers. If I would like to add third server I would have to pay another license in case of Citrix. In case of VMware, I have still one spare license so I will use it. At first glance, XenServer seems to be free of charge but the price of added value by Citrix Essentials doesn't scale as well as in case of VMware vSphere 4 Standard Edition or vSphere 4 Essentials Plus. And what is your opinion to the topic?

VMware vSphere - OVF support

OVF or Open Virtualization Format is an open DMTF standard with intention to package and distribute virtual machines or virtual appliances among various hypervisors independently on hypervisor and CPU architecture.

VMware supports OVF format and actively participate on its development . It is supported on ESX 3.5 and VirtualCenter 2.5 but the implementation doesn't support full OVF feature set. (draft standard, version 0.9). VMware vSphere 4.0 has full native support of OVF format in version 1.0. Beside, there exists standalone VMware OVF Tool 1.0 which brings OVF support for products like VMware Workstation or VMware Server.

OVF is a packaging format for software appliances. For example, it may contain tested LAMP stack prepared for simple deployment in production. It is a way how to transport virtual machine templates portably. OVF package may contain single or more virtual machines which must be installed (deployed) before they can be run. It is not run-time virtual machine format like VMDK. Further, it provides content verification and integrity checking.

Compared to VMDK format, OVF defines complete virtual machine - virtual hardware configuration including CPU, memory, storage, networking and virtual disks. On the other hand, VMDK is in charge of virtual disks only.

Are there available any OVF packages? Yes, for example there exists OVF of VMware vCenter 2.5 for Linux or vCenter Admin Portal and many others at VMware Virtual Appliance Marketplace.

VMware vSphere - Fault Tolerance

VMware High Availability provides protection against physical servers failures running ESX hypervisors. If one host in HA cluster fails then failed virtual machines are restarted on another alive host from cluster. It ensures the host has enough resources to fulfill requirements of newly booted virtual machines. It is able to monitor virtual machine activity by checking its heartbeat as well and in case of its failure to restart it.

The next logical step is fault tolerant virtual environment. VMware vSphere 4 can do it . It provides zero downtime and data integrity of virtual machines in case of physical server failure.

When you configure a virtual machine to be fault tolerant a secondary duplicate machine is created on a different host. Then, any operation performed on the primary machine is recorded and replayed on its duplicate. If the primary fails the secondary takes over and continues running without interruption. However, current version is not able to monitor applications running inside virtual machines but it should be available in future.

VMware Fault Tolerance or VMware FT as it is denoted is cool and must have feature but to implement it means to meet these requirements:

1. VM (virtual machine) must be in HA cluster
   
2. esx host ssl certificates checking has to be enabled
   
3. VM has to be stored on shared storage
   
4. VM's virtual disks have to be in thick format, thin is not supported already
   
5. three VMkernel 1G ports are required, one for VMotion and two for FT
   
6. FT doesn't support Virtual SMP, only single processor VMs are supported!
7. physical hosts have to support hardware assisted virtualisation, no problem with recent servers

The most of requirements are common ones but points 4 and 6, for older servers 7 as well, are considerlable limitations. It's not so simple to implement VMware FT but I hope it will get better in next releases.

Finally, vSphere documentation is available at vmware.com.

VMware vSphere 4.0 editions

Yesterday, VMware uncovered new pricing and licensing model of vSphere 4.0 platform. In my opinion, VMware is trying to strictly split up the virtualization market into two parts - SMB and enterprise. I have a feeling from the table of features below that there is growing a hole between them. The competitors should catch the chance to fill it up.

• SMB editions - ESXi Single Server, Essentials and Essentials Plus
• Enterprise editions - Standard, Advanced, Enteprise and Enterprise Plus

Here is the mentioned table of features:

VMware ESX and SATA controllers

VMware ESX hypervisor has supported only SCSI internal drives for a long time. The third update of ESX hypervisor introduced support for some SATA controllers like Intel ICH-7. The newest fourth update contains support of ICH-9 and ICH-10 chipsets as well. The same holds for ESXi platform.

The big difference is what SATA mode is supported. For example, the ICH-7 chipset is supported in IDE/ATA mode only, so you can't use use connected hard drives but you can access connected optical drives. The rest of the chipsets is supported in AHCI or Advanced Host Controller Interface mode. In this mode, you can access internal SATA drives.

When IDE/PATA mode is used, you will be able to see internal SATA (or emulated PATA) drives but you can't use it as VMFS storage. VMFS filesystem can be created on SCSI-based disks only.

There exists a nice knowledge base article about the topic. To better understand it, I borrowed an image from the article which is quite self-explanatory:

VMware ESX/ESXi 3.5 update 4 released

The fourth update of VMware ESX platform was released. It contains many hardware enhancements like support of new Intel Xeon 5500 procesors, SATA controllers or network interface cards. It supports new guests as well like SLES 11 released recently. The official release notes provide more comprehensive information.

VMware ESX 4.0 aka vSphere 4.0 platform

The next major release of VMware ESX platform is being prepared. The platform newly called as vSphere 4.0 is going to be based on six stones which provide:

1. vCompute - virtualization layer, hypervisor, live migration
   
2. vStorage - storage management, replication
   
3. vNetwork - network management, distributed switch, Cisco Nexus switch
   
4. Availability - clustering, data protection
   
5. Security - VMsafe APIs, vShield Zones
   
6. Scalability - dynamic resource management, distributed power management
   

Furthermore, the new platform will support virtual machines with 8 virtual CPUs and 256 GB of virtual memory.

The second most important part of virtual environment is centralized management. Today, we know it as VMware VirtualCenter Server. In the future, it should be called vCenter Suite. The good news, it will be available for Linux servers as well so no more Windows licenses are required.

VCB, vcbMounter, vcbRestore ... updated

I have added another article dedicated to VMware VCB and backups over Samba or Windows shares. Here is updated list of them:

1. VM identification - how to identify a virtual machine you intend to backup? The command vcbvmname is the answer.
2. VM full backup - how to perform a full backup of the chosen virtual machine? The vcbmounter command can do it.
3. VM full backup data access - how to retrieve data from the virtual machine's full backup? It is possible to mount the backup image with the mountvm command.
4. VM file level backup - the vcbmounter command is able to perform file-level backup as well.
5. VM backup over NFS - this article describes a simple scenario of virtual machine backup over NFS protocol.
6. VM backup restore - it is important to know the process of restoring a virtual machine from the backup. You can use vcbrestore.
7. VM backup with Samba or Windows share - the other approach how to perform backups of virtual machines is to use Samba or Windows shares instead of NFS server.
   

VCB basic usage - VM full backup with Samba

In the previous article about VMware VCB, I wrote about full backups to NFS shares. For completeness, I decided to write another one dedicated to backups to Samba or Windows shares.

The idea of backup is the same. Let's have a Samba server available at IP address 192.168.1.1. The exported directory for backups is backup-smb and the user which has write access to this share is backup.

Before we will be able to continue we need to allow smbclient to access Samba server. You can perform it from VI client or directly from ESX service console via esxcfg-firewall command. First, let's check if smbclient is allowed:

esxcfg-firewall -q smbClient

The output of command should be by default:

Service smbClient is blocked.

To reconfigure ESX firewall to allow smbclient access use the next command:

esxcfg-firewall -e smbClient

Now, you should be able to browse the server (the command asks for user's password first):

 smbclient  -L 192.168.1.1 -U backup

The example command output follows (Samba server on SLES10):

Domain=[NAS] OS=[Unix] Server=[Samba 3.0.28-0.2-1625-SUSE-CODE10]

Sharename       Type      Comment
---------       ----      -------
profiles        Disk      Network Profiles Service
backup-smb      Disk
IPC$            IPC       IPC Service (Samba 3.0.28-0.2-1625-SUSE-CODE10)
Domain=[NAS] OS=[Unix] Server=[Samba 3.0.28-0.2-1625-SUSE-CODE10]

Now, we are ready to create a simple backup script:

#!/bin/sh

BACKUP_SERVER="192.168.1.1"
BACKUP_USER="backup"
BACKUP_PASS="backup"
SMB_SHARE="backup-smb"
MOUNT_DIR="/backup"

[ -d $MOUNT_DIR ] || mkdir -p "$MOUNT_DIR" || exit 1

VM_BACKUP="`vcbVmName -s any: | grep name: | cut -d':' -f2`"

if [ ! -z "$VM_BACKUP" ]; then
 smbmount //${BACKUP_SERVER}/$SMB_SHARE $MOUNT_DIR \
     -o username=${BACKUP_USER},password=$BACKUP_PASS || exit 1

 for VM in $VM_BACKUP; do
     vcbMounter -a name:$VM -r $MOUNT_DIR/$VM
 done

 umount $MOUNT_DIR
fi

exit 0

It is simple, isn't it? The code is almost the same as for backups over NFS. We added variables defining our Samba user and his password. The mount command was exchanged with smbmount which is CLI Samba client. If you insist on using the mount command replace the line mounting the backup-smb share with line:

mount -t smbfs //${BACKUP_SERVER}/$SMB_SHARE $MOUNT_DIR \
 -o username=${BACKUP_USER},password=$BACKUP_PASS || exit 1

That's all. In such simple backup scenarios I prefer NFS usage because it is simple to set and provides higher throughput than SMB protocol. On the other hand, SMB protocol provides basic authentication mechanism (if you don't disable it).

XenServer is free

It's unbelievable! Citrix decided to release their XEN based hypervisor and complete virtualization solution named XenServer for free a few hours ago (official announcement is here). The product was available in three editions until recently - Express, Standard, Enterprise and Platinum. The differences are outlined in the following table:

The Express edition was free of charge so far but it was missing some fundamental enterprise features like resource pools, live migration or central management console XenCenter. These features are paid. Or better, they were paid.

From now, we have only one edition of XenServer including features of enterprise edition. Everything is free and you can download it. Cool! You don't have to spend any money on virtual machines live migration, resource pools or central management stuff. What happens if we compare it with VMware ESXi? In my opinion, it seems the king might be dead. And the new king might be coming.

What do you think of it? What will be the answer from VMware? I think it is smart way how to show us that XEN based hypervisors are enterprise ready and how to spread it more. In connection with current economical situation they have the real chance to success.

Let me have final question. Who will need Microsoft Hyper-V now? If XenServer is free and because it is more mature and robust than Hyper-V what will be its new position? Today, the winner is Citrix. Tomorrow, the oponents might surprise us. But don't miss the opportunity today. Download XenServer and spread it!

VMware vCenter Converter 4.0 was released

The previous version of Converter was at 3.0.3 for a long time. The new standalone version is much similar to the one included in Virtual Infrastructure 3.5 (VI 3.5).

Before it, there were available two editions - Starter and Enterprise where the second one is part of VI 3.5. Here are the additional features of Enterprise edition compared to Starter:

• it supports multiple migration jobs
• it supports cold migration
  
• it is part of VI3.5 only (particularly VirtualCenter server)

What new brings us latest revision? It is free of charge, it has larger set of supported operating systems as source or it allows you to select the target virtual disks. Newly, it can migrate sources with RedHat, SUSE or UBUNTU Linux. Furthermore, it is able to power off the source after migration finishes. The more comprehensive comparison of the version 4.0 and the version included in VI3.5 is presented by this picture.