Migrate a vCenter Server and Change its IP Address

I needed to migrate a vCenter Server between datacenters. A new IP address was required for the vCenter Server at the destination. The process of changing a vCenter Server’s IP address became a straightforward process in vSphere 6.5. However, to add a complication in my situation, I also needed to migrate the vCenter Server. Unfortunately, the destination network was not available at the source. Here are the steps I went through and then an issue I ran into. This was for a 6.7 vCenter Server appliance with an embedded PSC.

  • Backup the vCenter Server
  • Shutdown vCenter Server
  • Clone vCenter Server
    • This was only a failsafe if the vCenter Server does not work at the destination to avoid restoring from a backup
  • Power on the vCenter Server
  • Run and save an export of RVTools
    • I always like to do this before big vCenter Server work so that I know where all my VMs are at
  • If only using a vDS, verify you have a port group with ephemeral binding
    • I did not need it, but you might depending on your destination
  • Change the IP address of the vCenter Server
    • The new IP address was displayed in the vCenter Server console
  • Shutdown the vCenter Server
  • Migrate the vCenter Server with VMware vCenter Converter Standalone to the destination ESXi host.
    • Ensure to verify and/or make the following changes in Converter
      • Required VM Hardware Version
      • VMXNET3
  • Change the DNS records for the vCenter Server
  • Power on the vCenter Server
  • Verify if the vCenter Server vNIC is connected
  • Reboot the vCenter Server
  • Verification
  • Delete the original and cloned vCenter Servers at the source

I thought everything went well. All ESXi hosts and VMs appeared to be happy. However, a user reported his remote console for some VMs would freeze after about 30-45 seconds and I also noticed some vDSs had sync issues. I did some research and found out about 90% of my hosts had the old vCenter Server IP address in vpxa.cfg.

VMware has KB1001493 which covers this issue. There are two methods to resolve the issue in the KB article. Every host with the issue needed to be touched so a lot of tedious work required for both methods. At first, I went with method 1 and tried it out on one host. The host was not responding after restarting the management agents and required to restart the vCenter Server service. I did not want to have all of my hosts not responding for a long period of time nor did I want to restart the vCenter Server service after restarting managements agents on each host. Therefore, I went with method 2.

Method 2 had more steps, but seemed to be cleaner. Essentially, each host is removed and added back to the vCenter Server one at a time so that seemed like a better approach. Hosts with a vDS required a little more work and documentation since a host will not have access to the vDS when removed. That meant to first put each host into maintenance mode before starting the first step. Then go through the steps and repeat on the next host. This greatly reduced the risk of VMs hitting any road bumps since VMs were not on a host that was being worked on. Then, of course, add each host back to the vDS. If no vDS and only standard switches, than no need to follow my extra step since network connectivity will be fine when the host is removed from the vCenter Server. Keep in mind performance data, permissions (depending what level they are set), VM placement in a folder, tags, and events/tasks (host level) are lost when removing a host from a vCenter Server. By the way, I did not do step 4 (Reinstall the VMware vCenter Server agent). The referenced KB article only mentioned up to 6.0, but seemed to worked well.

Perhaps, there’s a cleaner way to do this process. However, all things considered, the IP address change and migration went well. No outages and the desired outcome was achieved. I did two tasks at once so I, at first, thought that’s why I ran into the host issue. Though, I expect many people may have had this same issue, without combining the two tasks I did, since there is a KB article on how to resolve the issue.

Migrate resources to a new vCenter Server – Unregister old vCenter Servers (Part 4)

Now that all resources have been migrated, verify that nothing was forgotten at the source vCenter Server. Refer back to part 1. Make sure all permissions have been set on the destination resources in case you need to go back to the source to double check. Would not be fun to roll back once the single command to unregister a vCenter Server is ran.

KB2106736 has the steps for the appliance and Windows versions to unregister a vCenter Server from a PSC. In the addition to the KB article, I have two suggestions. Take a snapshot of all PSCs before running the command just to be safe.  I did run into a small issue at first, but nothing negative happened because of it. The vCenter name in the cmsso-util unregister command is case sensitive. “Could not find a host id which maps to NAME INPUTTED in Component Manager, Failed!!!” was the error message I received. If the vCenter Server is not found when running the command, I suggest you instead use the IP address of the vCenter Server.

CaseError1

The unregister command went well for me after I entered the vCenter Server in the correct case. It took a few minutes to run before I received the success output.

CaseError2

That completed the major steps in my migrations. I spent time afterwards to put datastores back in datastores clusters, import host profiles, etc. Documentation is important to assist in making your clusters look how they did before.

Migrate Resources to a New vCenter Server (Part 1)
Migrate resources to a new vCenter Server – Methods for Migrations (Part 2)
Migrate resources to a new vCenter Server – Cross vCenter vMotion Utility (Part 3)
Migrate resources to a new vCenter Server – Unregister old vCenter Servers (Part 4)

 

Migrate resources to a new vCenter Server – Cross vCenter vMotion Utility (Part 3)

I used a different method of migration for a cluster that was migrating to a new vCenter Server on a new SSO domain. The destination vCenter Server was using the vDS so I had the same limitation as the previous migration. Though, I had a different idea for this one. I had enough resources in one of the clusters to first bring over half of hosts to the destination vCenter Server. Having this configuration made the process of live migrating my VMs easy by using the Cross vCenter Workload Migration Utility.

vMotion is not bound by a SSO domain and vMotion does not even know the concept of an SSO domain. I heard this discussed on the Virtually Speaking Podcast with William Lam describing his work on the Cross vCenter Workload Migration Utility. vMotion appears to only be within the same SSO domain because of the limitation in the GUI. However, there are APIs to migrate a VM to a different SSO domain. The utility is a technically a fling, but I see no problem with using it in a production environment.  After all, the APIs are there and the utility is just giving an easy to use interface.

Here’s the process I went though for this migration.

  1. Export the vDS from the source vCenter Server
  2. Import the vDS on the destination vCenter Server
  3. Place host in maintenance mode
  4. Document and then remove VMkernel ports
    1. Of course, migrate the management VMkernel to a standard port if it is on the vDS.
    2. VMkernel ports need to be removed before a host can be removed from a vDS.
  5. Remove host from vDS
  6. Disconnect host and then remove host from source vCenter Server
  7. Add host to vCenter Server
  8. Add host to vDS
  9. Create and/or migrate  VMkernel ports that were previously deleted
  10. Exit maintenance mode
  11. Repeat steps 3-10 for additional hosts
  12. Use the Cross vCenter Workload Migration Utility to vMotion VMs

Migrate Resources to a New vCenter Server (Part 1)
Migrate resources to a new vCenter Server – Methods for Migrations (Part 2)
Migrate resources to a new vCenter Server – Cross vCenter vMotion Utility (Part 3)
Migrate resources to a new vCenter Server – Unregister old vCenter Servers (Part 4)

Migrate resources to a new vCenter Server – Methods for Migrations (Part 2)

My first walkthrough is for a cluster migrating to a different vCenter Server. This is the traditional vDS to standard switch to vDS. The reason for this way is to maintain up time for my VMs and supportability. Moving a host to a new vCenter Server without first going to a standard switch is not supported. I will go over each step. In my case, management and vMotion VMkernel ports are already on dedicated standard switches. Though, there is also a method to migrate the VMkernel ports with no downtime that is very similar. Only VM network traffic is on my vDS.

Each host needs to already have at least two physical NICs to avoid downtime for VMs and VM traffic. If you keep a continuous ping on a few VMs during this process, you will only see one ping drop during each major step. Make sure you verify the root password before removing hosts from vCenter.

Source vCenter Server

  1. Export the vDS configuration
    1. Go to the vDS -> Right click the vDS
    2. Export Configuration
    3. Select the Export the distributed switch configuration and all port groups option
  2. Create a Standard Switch and the virtual machine port groups
    1. No physical uplinks on every host at this moment
    2. A new port group that matches every port group that’s on the vDS
  3. Remove one of the physical NICs from all hosts
    1. Go to the vDS -> Manage Hosts
    2. Select all of the hosts
    3. Remove one of the two physical NICs
    4. Next -> Next -> Finish
  4. Add the physical NIC that was removed to the new standard switch on each host
  5. Migrate VMs in bulk in each port group
    1. Go to the vDS -> Migrate Virtual Machine Networking
    2. Select the source vDS port group
    3. Select the destination standard switch port group
    4. Select all VMs
    5. Finish
    6. Repeat these steps for each port group
  6. Remove the last physical NIC on the vDS
    1. Ensure no VM is connected to the vDS
    2. Go to the vDS -> Manage Hosts
    3. Select all of the hosts
    4. Remove the last physical NICs
    5. Next -> Next -> Finish
  7. Remove the hosts from the vDS
    1. Go to the vDS -> Click the host tab
    2. Right click each host and click remove
  8. Remove the hosts from the vCenter Server
    1. Right click a host -> Disconnect
    2. Right click a host -> Remove
    3. Repeat for each host

Destination vCenter Server

  1. Create a new cluster
  2. Add hosts to the new cluster
    1. Right click on the cluster
    2. Add host -> next through wizard
    3. Repeat for each host
  3. Import the vDS configuration at the destination vCenter Server
    1. Do not preserve the original vDS and port group identifiers
    2. Preserving the vDS and port group identifiers is used when restoring the vDS within the same vCenter Server
  4. Remove one physical NIC from the standard switch on each host
  5. Add the hosts to the vDS
    1. Go to the vDS -> Manage Hosts
    2. Select all of the hosts
    3. Add the physical NIC that was previously disconnected
    4. Next -> Next -> Finish
  6. Migrate VMs back to the vDS
    1. Go to the vDS -> Migrate Virtual Machine Networking
    2. Select the destination standard switch port group
    3. Select the source vDS port group
    4. Select all VMs
    5. Finish
    6. Repeat these steps for each port group
  7. Ensure no VMs are on the standard switches
  8. Remove one physical NIC from the standard switch on each host
  9. Add the last physical NICs to the vDS
    1. Go to the vDS -> Manage Hosts
    2. Select all of the hosts
    3. Add the physical NIC that was previously disconnected
    4. Next -> Next -> Finish
  10. Delete the unused standard switches

Now all VMs are running in the new vCenter Server on the vDS with no downtime. I will go over another method in my next article that I used for a different cluster.

Migrate Resources to a New vCenter Server (Part 1)
Migrate resources to a new vCenter Server – Methods for Migrations (Part 2)
Migrate resources to a new vCenter Server – Cross vCenter vMotion Utility (Part 3)
Migrate resources to a new vCenter Server – Unregister old vCenter Servers (Part 4)

Migrate Resources to a New vCenter Server (Part 1)

I recently had to move all resources under two vCenter Servers to two different vCenter Servers.  Then retire the source vCenter Servers. Both source vCenter Servers were in the same SSO domain. Fortunately, the source did not have many hosts and VMs and I wanted to bring over as much as I could.

I wanted to do my best not to leave anything behind at the destination vCenter Server. Most are obvious in my list below, but some may be easily forgotten if not used often. Below is a list of what I moved over and other settings to note in case moving to a completely new vCenter Server. A few have export/import wizards in vCenter and can of course be scripted for several others.

  • vDS – export/import
  • Host profiles – export/import
  • Customization specification – export/import
  • Custom attributes – output in RVTools
  • Folders – output in RVTools
  • Permissions
  • Roles
  • vCenter Settings – SMTP, database retention policy, etc
  • Alarms – custom/edited alarms
  • vCenter Server License
  • Cluster settings – HA, DRS, and EVC
  • Datastore clusters
  • Tags
  • Content libraries
  • Scheduled tasks

There are some third party apps to keep in mind. Make sure to reconfigure backups to point to the new vCenter Server. Third party plug-ins will need to be registered to the new vCenter Server.

Here are somethings not to worry about since they pertain to the hosts and remain with or without a vCenter Server.

  • Standard switches
  • Access to fiber channel storage
  • VM settings and notes

Events and tasks for a VM will persist when the VM is in the new vCenter Server. However, performance data for VMs is lost. Also, performance data, events, and tasks for hosts will be lost when a host is added to a new vCenter Server.

The next articles will focus on two different ways I migrated the hosts to the new vCenter Servers.

Migrate Resources to a New vCenter Server (Part 1)
Migrate resources to a new vCenter Server – Methods for Migrations (Part 2)
Migrate resources to a new vCenter Server – Cross vCenter vMotion Utility (Part 3)
Migrate resources to a new vCenter Server – Unregister old vCenter Servers (Part 4)

VMware vCenter SSO 5.5 Migration to PSC 6.0 Error

I am in the process of upgrading three vCenter Servers on 5.5 to 6. SSO was embedded on all three and SSO has been recently externalized to Windows Servers. The next step is to use the vCenter Server Migration Tool to migrate each SSO 5.5 server to PSC 6 appliance.

I went through the migration wizard and the migration was on its way. The PSC was deployed and the progress bar on the migration was moving along. However, when I opened the console for the PSC, there was an error; Upgrade EXPORT failed. Then the migration never finished.

UpgradeExportFailed

I ran the migration again with VMware support since they did not know what could be causing the issue. I opened up the console for the PSC as soon as it was deployed and there was a quick message to look at UpgradeRunner.log. However, there was nothing useful in that log. Then checked out upgrade-export.log.

There were network related errors in upgrade-export.log. I knew I inputted everything correctly into the migration wizard and there was nothing that would block communication between anything involved. The IPv6 address in the log stuck out to me. The SSO Windows Server had a IPv6 address, but the wizard never asked for anything IPv6 related. I disabled IPv6 on the SSO server, ran the migration again, and everything went well.

UpgradeExportFailed_Shell2

Here’s one way to view the PSC’s logs. At the PSC’s console, hit Alt+F1. Then type the commands below.

shell.set –enabled True
shell
cd /var/log/vmware/upgrade
less upgrade-export.log

Moral of the story is to disable IPv6 on all VMware related servers before using the vCenter Server Migration Tool. Then enable it after everything is on 6. This is just to be safe in case any other of the upgrades or migrations have similar issues. VMware support said they will have a knowledge base article on this issue. When they do, I will edit this article with a link so everyone can check out the latest directly from VMware.