Clear VCSA DNS cache

The command to clear the DNS cache on vCenter Server Appliance is

systemctl restart dnsmasq

and not

systemctl restart systemd–resolved.service




vCenter to VCSA Migration

I was trying to migrate from a Windows based vCenter 5.5 install to VCSA 6.5 and after encountering several errors trying to run the migration tool I discovered that I could disconnect and remove a host from vCenter and add it to VCSA w/o interrupting the VMs.

  1. Spin up new VCSA and recreate EVC clusters. (Make sure to have a DNS A record for VCSA before spinning it up.)
  2. Stop DRS on the vCenter EVC cluster.
  3. Enter maintenance mode for a host. (Don’t evacuate VMs)
  4. Disconnect host from vCenter.
  5. Remove host from vCenter.
  6. Add host to new VCSA.

I lost all the historical perf data, but that’s better than trying to fix SSL certs.


ESX disconnected from vCenter

One of my esx 5.5 u3 hosts was disconnected from vCenter. The VMs were still running but reconnecting it to vCenter would result in an vmodl.fault.HostCommunication error.


I logged into the console and looked at the logs and saw that the ramdisk /tmp was full. I verified that it was full with esxcli system visorfs ramdisk list [1]

A simple ls -al /tmp showed that an error log file was filling up the entire ramdisk. I deleted the file and the host automatically reconnected to vCenter.

Reclaiming space on VMs

Here are some great posts about reclaiming unused space in esxi.




To get automatic space reclamation you’ll need:

  • ESXi 6.5 Update 1
  • iSCSI volume formatted with VMFS6. (I think VMFS5 works but there are caveats.)
  • Thin provisioned disks.
  • discard option enabled on mount points for your Linux guests.

My new esxi 6.5 host connecting to iscsi volume on FreeNAS 11.0 works like a charm at least when I force and UNMAP with sudo fstrim /

I haven’t had it enabled long enough to watch it work it’s magic, but apparently this means no more filling the drive with zeros, then punching a hole while the machine is offline.

UPDATE 2018-01-20: The magic happens immediately. You can see the updated datastore size immediately after deleting a large file in a linux VM.

It also means that as much as I hate setting up iSCSI, I’m not going to go back to NFS if it doesn’t have this.

Convert VMDK to Thin without vMotion

Storage vMotion is wonderful but vMotion isn’t available without vCenter. bah!

This was a nice guide:

  1. Enable SSH to the host.
  2. Browse to the VM
    cd /vmfs/volumes/<datastore>/<vm>
  3. Create a thin version of the VM.
    vmkfstools -i d <vharddrive>.vmdk -d thin <vharddrive>-thin.vmdk
  4. Remove the thick .vmdk
  5. Attach the thin .vmdk
  6. Delete the old .vmdk and *-flat.vmdk


Can’t format VMFS5 partition

While trying to add an ssd to my esxi 5.5 server I encountered a bug when it tried to format and add the storage:

Error:A specified parameter was not correct.
Error Stack
Call “HostStorageSystem.ComputeDiskPartitionInfo” for object “storageSystem” on ESXi “x.x.x.x.” failed.

I found a blog post describing the same issue and found it was the existing partitions that was causing it to fail. But instead of dropping into the CLI to delete the partitions manually and continue, the comments had another idea: Create a VMFS3 partition then upgrade to VMFS5.

I ended up creating a VMFS3 partition, but deleted it instead of upgrading it. Then I created a VMFS5 partition successfully. I don’t care for upgrading filesystems. I always prefer to start fresh.

Windows Hosts Losing Static IP After vmotion

I’ve been fighting a very annoying bug with a couple of my Windows servers. They would lose their static IP address after being vmotion’d and go back to DHCP. The easy workaround was to disable DRS on those specific VMs which we did, but that doesn’t fix anything.

I thought for sure that the issue was at the VM level, but it turns out that after upgrading from 5.1 Patch 2 (1743533) to 5.1 Patch 7 (2583090) the issue has cleared up.

Here is a very useful list of all the versions and release dates from VMWare.

Update Manager had an error

I ran across this little gem of a bug on VMWare’s Update Manger while trying to scan any of our hosts:

VMware Update Manager had an unknown failure: Check task and events

It was probably caused from our last upgrade. We upgraded from vCenter 5.1 on a Windows 2003 32-bit to 5.5 on Windows 2008 R2 (64-bit). VMWare’s supported helped me through the upgrade and it went smoothly. Until I had to use Update Manager. I couldn’t find anything online to help fix it so I ended up uninstalling Update Manager, re-installing it, and this time choosing to go with a blank database. Luckily, our environment is small enough that I can get away with that.

Upgrading vCenter 5.1 to 5.5 (+OS)

I was finally bitten by the vCenter 5.1 Single Sign On (SSO) bug. Yikes, that was bad. I had to restart the SSO service nearly everyday in order to login to vCenter. I was just leaving the vCenter Client open all the time to avoid it. Well, I finally upgraded vCenter to 5.5 to fix it, and it looks like it has so far (knock on wood).

However, in order to upgrade vCenter to 5.5 I first needed to get it off the Windows 2003 box it was on. vCenter 5.5 doesn’t support 2003. Also, I wanted to virtualize the machine. So how do I go from a physical Win2k3 box running 5.1 to a virtual Win2k8R2 box running 5.5? Actually, it’s easier than you think. (A big thank you to Murali M. at vmware. He walked me through this.)

  1. Build up the new VM.
  2. Install SQLExpress (we have a small deployment so Express is more than adequate.)
  3. Use SMSS to backup and restore the db’s from the old sqlexpress instance to the new one.
  4. Create an ODBC connection on the new server to the vCenter db.
  5. Do a custom install for vCenter 5.5. Install each component separately and in order: SSO, Webclient, Infrastructure. The installer will walk you thru it. Skip vCenter Server for now.
  6. Shut down vCenter Server service on the old server.
  7. Install vCenter Server on the new server and use the ODBC connection to use the restored db. This will upgrade the 5.1 db to a 5.5 db.

If anything goes wrong, just shutdown the new server, turn on the old vCenter service and we’re back.

My first plan was to just install 5.1 on the new box, migrate the db’s and then do an in-place upgrade. This new method is cleaner, simple, and quite easy.