Jumbo Frames

Jumbo frames are the way to go for vSAN, iSCSI, and vMotion VMkernels for increased performance if able to support the larger MTU on all networking devices. MTU of 9000 is supported on standard switches and distributed switches. A mismatch of MTUs can happen and network traffic may seem to be fine when it’s not. vmkping can be used to help test everything is configured correctly for MTU size.

The MTU size can be changed when the VMkernel, vSS, or vDS are in use. Though, there will be a short loss of network connectivity. No worry for vMotion. Can just temporarily disable DRS so no vMotions kick off. Though, careful planning is needed for vSAN, iSCSI, and the virtual switches they are connected to. If a host is using a vSS, the host can be placed in maintenance mode. Then change the MTU for the VMkernels and vSS. However, if using a vDS, the vDS MTU change effects all hosts in the cluster. Also, the VMkernels need to be changed for each host. I tried it on a 5 host vSAN cluster and the vDS change took 72 seconds for the MTU change to apply. VMs lost connectivity during the change and then were fine when the change was fully applied. Of course, this is not recommended to do in a production environment.

My Top Picks for Sessions at VMworld 2021

VMworld is virtual for the second straight year. The event is October 5-7 and will have content around the clock to accommodate various time zones. There are two types of passes available; the general pass is free and the Tech+ Pass has a charge. Some of the sessions are only available with the Tech+ Pass. The content catalog has been released so it’s nice to get an idea of what’s happening to plan ahead. I like to focus mostly on sessions that will help me add value to the work I do today. Some sessions will be pre-recorded and some will be live via Zoom. As always, there’s a lot of sessions to choose from. I went through them all and below are my top picks.

Deep Dive: VM Performance and Best Practices [VI2158]$
Extreme Performance Series: Performance Best Practices [MCL1635]
Extreme Performance Series: vSphere Advanced Performance Boot Camp [MCL2033]$
It’s good to see sessions based on the fundamentals of vSphere and then taking those fundamentals to the next level. All three sessions sound similar and rated at the highest technical level. Both extreme performance sessions are hosted by Mark Achtemichuk and Valentin Bondzio, and one of them is free. Mark is a VCDX, and whenever a VCDX speaks, I pay attention. These sessions should deliver useful information for all VMware admins. Mark and Valentin both work for VMware so I am sure they have seen a million different problems and can help people better troubleshoot when they face these problems.

Core Storage Best Practices Deep Dive [MCL2071]$
VMware vSAN – Dynamic Volumes for Traditional and Modern Applications [MCL1084]
VMware Storage Queue Tuning [VMTN2862]
The first session I listed above will be hosted by Cody Hosterman and Jason Massae. I have watched Cody present before at a local VMUG and other events, and he always delivers. He works for Pure, but don’t let that fool you into thinking he’s there to only promote his product. He always gets to the crux of storage that applies to nearly everyone. Duncan Epping, VCDX, and Cormac Hogan are presenting the vSAN session. They will be discussing the newer vSAN File Service and more cutting edge integration with vSphere Container Storage Interface. I am very interested in what storage tips all of these rock stars will give that I can apply at my job.

The Clock is Ticking on NSX Data Center for vSphere [EDG2721]$
NSX Data Center for vSphere to NST-T Data Center – Migration Approaches [NET1211]
NSX Data Center for vSphere to NSX-T: Strategies to Make the Move [EDG1515]
NSX-V reaches end of support on January 16th, 2022 and that’s the basis of these panels. Jayson Block, VCDX, is hosting the third session linked above. If you are still using NSX-V, you have to attend at least one of these sessions. All three sessions will give good advice on developing a migration plan. Hopefully, this will help to jumpstart your planning and migrating to NSX-T before support ends.

A Field Guide to Health Check vSAN to Operate, Upgrade and Transform [MCL1825]$
A technical Review of PSC. VMDIR, Replication and Snapshots [VI2447]$
Troubleshooting with vRealize Operations and vRealize Log Insight [MCL1286]$
All three of these sessions are in the meet the expert category and will give deep dives in their respective areas. Paul McSharry, VCDX, is presenting the vSAN session. This session looks valuable for anyone using vSAN to hear from an architect that has many deployments under their belt. The PSC has come a long way, but I still feel I don’t know everything I need to know about it. I always come across some sort of problem that support needs to help with. The PSC session looks to give the deeper understanding that I have been looking for. A lot of companies have vROps and vRLI deployed, but I rarely see anyone use the two products to their full potential. Using one or both of these products together is very useful for troubleshooting. I am looking forward to fine tuning my troubleshooting skills with this session.

Solid sessions make a virtual event and VMworld delivers again. I rather it be in person. However, it being virtual has some benefits, such as there’s a free option and more accessible to a wider audience. I will be again missing out on collecting a suitcase full of swag, but there will be giveaways from VMware and vendors, even during a virtual event. I won an official VMworld backpack last year!

vExpert 2021 and VMworld

I earned vExpert status for the forth year in a row. I was selected for my blog and contributions on the VMTN forums. I will continue my mission, which is writing articles that will assist someone that is facing the same issue I had.

VMworld will be only online again this year. It will be October 5-7, 2021. Not much has been announced so far for this event. I hope it is more interactive than last year’s online event. I definitely miss attending in person VMUGs and VMworld.

VMware vSAN 6.7 Specialist 2019 Exam Preparation and Experience

I recently passed the VMware vSAN 6.7 Specialist 2019 Exam (5V0-21.19) with a scare of 440. Despite the exam’s name, I earned the VMware Specialist – vSAN 2021 badge. Fortunately, VMware is now keeping up with the names of the badges and certifications by matching it with the calendar year, even if the test wasn’t yet updated. This is a nice change because I previously took the vSAN specialist exam in 2019 and earned the 2017 badge, which immediately looked out of date.

The exam was very straight forward. A lot of it is what engineers do on a weekly or monthly basis for many of the exam objectives. I had more than enough time to finish the exam. I don’t think there was any surprises so I recommend to know the exam objectives inside and out. Some exam objectives are not deep so don’t think too much into them. For example, objective 7.5, describe the effect of maintenance mode options; only need to know the three maintenance mode options and what they do. Though, there are some questions that were not worded the best, but that shouldn’t make it or break it for anyone.

The exam is based on vSphere 6.7 and vSAN 6.7 U1. Keep that in mind as there was a lot of changes in newer versions, especially in 6.7 U3. You are probably running a newer version at work and VMware’s HOL are no longer running the exam version of vSAN from what I saw. Though, VMware still has the lab manual for their retired vSAN 6.7 HOL. I found a lot of good information and screenshots in there.

There are only two requirements for achieving the vSAN 2021 badge; pass the 5V0-21.19 exam and have one of the pre-requisite certifications. The badge page says it needs to be a current VCP. However, there are many other certifications that also count, such as VCAP and VCDX certifications. Check out your certification manager on VMware Learn and you can see more details on what’s covered. Also, I wrote about this in a previous article about the vSAN 2017 Specialist badge.

vRealize Log Insight Not Connected to vCenter Servers After Upgrade

This is just a quick article that I have been meaning to do. I upgraded my Log Insight server. After the upgrade, it wasn’t collecting logs from my vCenter Servers. I had to accept a certificate for each vCenter Server and then it worked again. The setting is in Log Insight, under administration, and vSphere. I forgot to take a screenshot when I saw the error. Below is the location to accept the certificate.

Migrate a vCenter Server and Change its IP Address

I needed to migrate a vCenter Server between datacenters. A new IP address was required for the vCenter Server at the destination. The process of changing a vCenter Server’s IP address became a straightforward process in vSphere 6.5. However, to add a complication in my situation, I also needed to migrate the vCenter Server. Unfortunately, the destination network was not available at the source. Here are the steps I went through and then an issue I ran into. This was for a 6.7 vCenter Server appliance with an embedded PSC.

  • Backup the vCenter Server
  • Shutdown vCenter Server
  • Clone vCenter Server
    • This was only a failsafe if the vCenter Server does not work at the destination to avoid restoring from a backup
  • Power on the vCenter Server
  • Run and save an export of RVTools
    • I always like to do this before big vCenter Server work so that I know where all my VMs are at
  • If only using a vDS, verify you have a port group with ephemeral binding
    • I did not need it, but you might depending on your destination
  • Change the IP address of the vCenter Server
    • The new IP address was displayed in the vCenter Server console
  • Shutdown the vCenter Server
  • Migrate the vCenter Server with VMware vCenter Converter Standalone to the destination ESXi host.
    • Ensure to verify and/or make the following changes in Converter
      • Required VM Hardware Version
      • VMXNET3
  • Change the DNS records for the vCenter Server
  • Power on the vCenter Server
  • Verify if the vCenter Server vNIC is connected
  • Reboot the vCenter Server
  • Verification
  • Delete the original and cloned vCenter Servers at the source

I thought everything went well. All ESXi hosts and VMs appeared to be happy. However, a user reported his remote console for some VMs would freeze after about 30-45 seconds and I also noticed some vDSs had sync issues. I did some research and found out about 90% of my hosts had the old vCenter Server IP address in vpxa.cfg.

VMware has KB1001493 which covers this issue. There are two methods to resolve the issue in the KB article. Every host with the issue needed to be touched so a lot of tedious work required for both methods. At first, I went with method 1 and tried it out on one host. The host was not responding after restarting the management agents and required to restart the vCenter Server service. I did not want to have all of my hosts not responding for a long period of time nor did I want to restart the vCenter Server service after restarting managements agents on each host. Therefore, I went with method 2.

Method 2 had more steps, but seemed to be cleaner. Essentially, each host is removed and added back to the vCenter Server one at a time so that seemed like a better approach. Hosts with a vDS required a little more work and documentation since a host will not have access to the vDS when removed. That meant to first put each host into maintenance mode before starting the first step. Then go through the steps and repeat on the next host. This greatly reduced the risk of VMs hitting any road bumps since VMs were not on a host that was being worked on. Then, of course, add each host back to the vDS. If no vDS and only standard switches, than no need to follow my extra step since network connectivity will be fine when the host is removed from the vCenter Server. Keep in mind performance data, permissions (depending what level they are set), VM placement in a folder, tags, and events/tasks (host level) are lost when removing a host from a vCenter Server. By the way, I did not do step 4 (Reinstall the VMware vCenter Server agent). The referenced KB article only mentioned up to 6.0, but seemed to worked well.

Perhaps, there’s a cleaner way to do this process. However, all things considered, the IP address change and migration went well. No outages and the desired outcome was achieved. I did two tasks at once so I, at first, thought that’s why I ran into the host issue. Though, I expect many people may have had this same issue, without combining the two tasks I did, since there is a KB article on how to resolve the issue.

VMware Training and VCPs/VCAPs Stats

VMware is giving away a free 6 month premium subscription to VMware Learning Zone for everyone. You need to register by November 6, 2020 and your subscription will expire 6 months after you register. That makes it easier to plan ahead to take advantage of a time frame that works best for you. You can even sign up with a personal email address. Check out the official article from VMware for more details and how to register.

VMware released details on the amount of VCPs and VCAPs. I like looking at statistics and it’s fascinating when vendors release numbers on certification holders. VMware last did this a few years ago. Unfortunately, I could not find the past numbers directly from VMware, but I did find it on vInfrastructure Blog. I will reference the data from the aforementioned sites in this article.

There are only 3,850 VCAPs in the United States and a total of 13,580 around the world. There are 98,576 VCPs in the United States and a total of 335,371 VCPs around the world. VCAPs have nearly tripled in the past few years and over 6 times more VCPs. I wonder if that is due to VMware changing their strict 2 year expiration on VCP certs.

The United States is on par with the world total of only roughly 4% of VCPs going ahead to achieve the VCAP certification. There are some overachieving countries like Egypt, which has 10% of VCPs getting their VCAP. Hopefully, VMware will give a list of totals in all countries. I made a comment requesting additional data and I will edit this article if I receive it.

VCAP-DCV Deploy 2020 Exam Experience

I passed the VMware Certified Advanced Professional 2018 – Data Center Virtualization Deployment exam (3V0-21.18) with a score of 400. I am glad the certification is considered 2020 so it doesn’t look as old already. I passed the VCAP-DCV Design exam a couple years ago. Oddly enough, the exam I just took earned me the VCIX badge for 6.5 and 2020. The Design exam also gave me the VCIX 6.0 badge a couple years ago when paired with my VCAP5. Funny how that worked out with the extra badge.

I received my results and badges the following day after I took the exam. I was pleasantly surprised to receive my results that fast, especially considering VMware says it can take 4-6 weeks. I received emails from VMware and Acclaim. I read some exam takers who didn’t receive an email from VMware and never even found out their score. They simply knew they passed only because they received the badge.

The exam begins as soon as you hit start and loads. You see what looks nearly identical to a VMware HOL lab and you can start reading the question manual before everything finishes loading. Then you have 210 minutes to complete everything, which is actually slightly longer than what the exam blueprint says. By the way, VMware says you cannot press the control key and right-click. Both worked for me.

I made my first run through of all 17 questions with about one hour left. I was only stumped on one question and a part of another. When I first took the VCAP exam four years ago, I didn’t have enough time and didn’t answer all the questions. Though, I still passed. I am not sure if the exam got easier or just my added experience got me through it much quicker. Anyway, I spent a half hour double checking my work and then finally clicked what to do with the one question I was stumped on. I was only stuck on one part of a question. I was fairly positive I did very well when I ended the exam. I don’t mean to sound braggadocious, but I honestly thought I would have had scored more than 400.

I might have lost some points on typos. The question manual was visually clear, but everything else was a little fuzzy within the environment. This made a lot of the text difficult to read and I couldn’t figure out a way to make the text sharper. Another problem was almost every time I hit a key, multiple letters were outputted. It was very annoying. I tried to drag and drop from the manual as much as possible. Other than that, the exam ran well. Much better than the past version of it.

I am sure most people practicing for the exam are using one vCenter Server. That’s fine. It’s easy to next your way through a wizard. However, remember that the exam has multiple vCenter Servers so you have to be very careful you are always working on the vCenter Server you are required to for the question. I knew this coming into the exam and I still had to go back to correct a mistake I made.

Here are a couple links to practice VCAP questions to try out. I forgot to post them in my last article. They were helpful to prepare for the exam to get in the mindset of possible questions. The HOLs they refer to are no longer available, but you can pull up other related HOLs to run through most of the questions or use your own lab.
https://www.jortechnologies.com/vcap-dcv-deploy-hol-based-simulator-free http://virtualg.uk/vcap6-dcv-deploy-exam-simulator-free

The exam sounds more daunting than it actually is. Most people with a lot of VMware experience can pass it with some studying and focusing on features they have never used. Be ready to stay focused for a few hours and you will be fine. Good luck!