I am a huge fan of the VMUG organization.  I attend two different VMUG’s in central ohio.  From time to time I have been known to speak at the events. In fact I will be speaking on Feb. 25th on design at the Central Ohio VMUG.  So last week I attended the VMUG virtual day long event.  VMware has been playing around with this type of event for about six months now.  The concept is simple avoid the traveling show and create a virtual event.    There are live streams with Q and A sessions and of course vendor booths.

This type of medium is getting more popular by all companies to reduce expense.  Of course the key is they need to bring value to the table.  Traditionally they have these type of shows attract people for these reasons:

• SWAG (Stuff we all get)
• Get out of the office
• Chance to talk to a lot of vendors in one place
• Chance to learn about companies product (marketing)

The virtual event does offer prizes but not a lot of swag beyond white papers.   Personally, I enjoy going to the events because I can interact with other people.  I find the lunches at these events to be the most useful.   In that respect the virtual event is really missing out.  I can talk to vendors via chat or VMware employees but my ability to talk to other customers is limited.   I have always felt that VMUG is really about other customers.  I hope they find a way to create community at these events as well.  Here is the good news even if you could not attend you can get all the presentations streaming until Feb. 24th right here.

I did enjoy seeing more vSAN best practices.  I am really looking forward to seeing more vSAN deployments.

When I studied computer science it was not a raw science.  My training did not require knowledge of how transistors worked or even logic circuits.  It focused mostly on programming languages and how to configure a web server.   Why?  Because these were the skills most likely to be used by a computer scientist in the field today.   Very few people build computers from scratch.  Intel has a corner on that market.  Personally I wanted to understand all the under the hood components so I took a minor in electrical engineering.  It was worth my time and a great learning experience.   I find that a lot of technology is like this… which includes VMware snapshots.   I have had snapshots explained to me in every VMware course that I have attended and every answer is different.  I have cobbled together lots of KB articles and other sources into this article.  If something is missing or incorrect let me know so I can fix it.

What is a Snapshot?

• A snapshot file is only a change log of the original virtual disk
• A virtual machine uses the disk descriptor file to access the most current snapshot not the original disk.
• It is not a complete copy of the original disk
• Snapshots are combined dynamically with the original disk to form a current state of system
• The snapshot change logs are sometimes called delta files
• Think of them as chains to get the complete picture you need all the links in order.
• Snapshot files will grow forever unless deleted(re-integrated into original disk)
• Using a lot of snapshots can effect performance of the virtual machine

How do snapshots work?

The process on the surface seems simple.  When you initiate a snapshot the following process is followed:

1. A request to create (CreateSnapshot) snapshot for a virtual machine is forwarded to ESXi host running the virtual machine.
2. If memory snapshot is included the ESXi host writes the memory of virtual machine to disk
3. If quiesce is possible the ESXi host request the guest OS to quiesce the disks via VMtools
4. ESXi host changes the virtual machines snapshot database (.vmsd file) to denote snapshots
5. ESXi host calls a function to make changes to the child disks (-delta.vmdk via the .vmdk descriptor)

What is a .vmdk descriptor?

Due to the nature of file systems os’s don’t like to have file names change mid-access.  So VMware has implemented descriptors.  Or symbolic links that point to the real files.  This allows a snapshot to be created and access via descriptor to continue.

How can I identify snapshots?

• Select the Virtual Disk and check the Disk File. If it is labeled as VM-000001.vmdk , the virtual machine is running on snapshot disks.
• Run the following command from ESXi Shell:

find /vmfs/volumes/ -iname "*.vmx" -exec grep -Hie "-[0-9][0-9][0-9][0-9][0-9][0-9].vmdk" {} \;

• List currently open delta disks via command line:

ls -l /vmfs/devices/deltadisks

• Locate all delta disks on file system:

find /vmfs/volumes/ -iname "*delta.vmdk"

• In powercli

get-vm | get-snapshot | format-list

I ran into this issue last week while upgrading several ESXi environments to 5.5.  On these I needed to move the boot from SAN partition so I did fresh installs from the disks.  Everything was great until I tried to login to the ESXi host with domain credentials.  (By default ESXi looks for members of a AD group called ESX Admins)  All AD authentication requests failed.   At first I thought it was a issue with the HP version of ESXi from Sep. 2013:

VMware-ESXi-5.5.0-1331820-HP-5.71.3-Sep2013

Turns out it’s a issue with VMware’s base image.  So every fresh install could have this issue.   When you setup AD authentication a number of daemons are started:

netlogond, lwiod, and lsassd

First time they run they create a number of directories and create pid files in /var/lock/subsys.   Unfortunately this is where the error lies.  There is no /var/lock/subsys directory.  It was missed.  I confirmed the issue still exists on the latest patch set as well.  So here is the work around:

Log in to ESXi via SSH or vMA as root (since you cannot login as you)
mkdir /var/lock/subsys
/etc/init.d/netlogond restart; /etc/init.d/lwiod restart; /etc/init.d/lsassd restart;

And test AD login… all should be good.  VMware should post a KB article in the near future.