Jumbo Frames

Jumbo frames are the way to go for vSAN, iSCSI, and vMotion VMkernels for increased performance if able to support the larger MTU on all networking devices. MTU of 9000 is supported on standard switches and distributed switches. A mismatch of MTUs can happen and network traffic may seem to be fine when it’s not. vmkping can be used to help test everything is configured correctly for MTU size.

The MTU size can be changed when the VMkernel, vSS, or vDS are in use. Though, there will be a short loss of network connectivity. No worry for vMotion. Can just temporarily disable DRS so no vMotions kick off. Though, careful planning is needed for vSAN, iSCSI, and the virtual switches they are connected to. If a host is using a vSS, the host can be placed in maintenance mode. Then change the MTU for the VMkernels and vSS. However, if using a vDS, the vDS MTU change effects all hosts in the cluster. Also, the VMkernels need to be changed for each host. I tried it on a 5 host vSAN cluster and the vDS change took 72 seconds for the MTU change to apply. VMs lost connectivity during the change and then were fine when the change was fully applied. Of course, this is not recommended to do in a production environment.

My Top Picks for Sessions at VMworld 2021

VMworld is virtual for the second straight year. The event is October 5-7 and will have content around the clock to accommodate various time zones. There are two types of passes available; the general pass is free and the Tech+ Pass has a charge. Some of the sessions are only available with the Tech+ Pass. The content catalog has been released so it’s nice to get an idea of what’s happening to plan ahead. I like to focus mostly on sessions that will help me add value to the work I do today. Some sessions will be pre-recorded and some will be live via Zoom. As always, there’s a lot of sessions to choose from. I went through them all and below are my top picks.

Deep Dive: VM Performance and Best Practices [VI2158]$
Extreme Performance Series: Performance Best Practices [MCL1635]
Extreme Performance Series: vSphere Advanced Performance Boot Camp [MCL2033]$
It’s good to see sessions based on the fundamentals of vSphere and then taking those fundamentals to the next level. All three sessions sound similar and rated at the highest technical level. Both extreme performance sessions are hosted by Mark Achtemichuk and Valentin Bondzio, and one of them is free. Mark is a VCDX, and whenever a VCDX speaks, I pay attention. These sessions should deliver useful information for all VMware admins. Mark and Valentin both work for VMware so I am sure they have seen a million different problems and can help people better troubleshoot when they face these problems.

Core Storage Best Practices Deep Dive [MCL2071]$
VMware vSAN – Dynamic Volumes for Traditional and Modern Applications [MCL1084]
VMware Storage Queue Tuning [VMTN2862]
The first session I listed above will be hosted by Cody Hosterman and Jason Massae. I have watched Cody present before at a local VMUG and other events, and he always delivers. He works for Pure, but don’t let that fool you into thinking he’s there to only promote his product. He always gets to the crux of storage that applies to nearly everyone. Duncan Epping, VCDX, and Cormac Hogan are presenting the vSAN session. They will be discussing the newer vSAN File Service and more cutting edge integration with vSphere Container Storage Interface. I am very interested in what storage tips all of these rock stars will give that I can apply at my job.

The Clock is Ticking on NSX Data Center for vSphere [EDG2721]$
NSX Data Center for vSphere to NST-T Data Center – Migration Approaches [NET1211]
NSX Data Center for vSphere to NSX-T: Strategies to Make the Move [EDG1515]
NSX-V reaches end of support on January 16th, 2022 and that’s the basis of these panels. Jayson Block, VCDX, is hosting the third session linked above. If you are still using NSX-V, you have to attend at least one of these sessions. All three sessions will give good advice on developing a migration plan. Hopefully, this will help to jumpstart your planning and migrating to NSX-T before support ends.

A Field Guide to Health Check vSAN to Operate, Upgrade and Transform [MCL1825]$
A technical Review of PSC. VMDIR, Replication and Snapshots [VI2447]$
Troubleshooting with vRealize Operations and vRealize Log Insight [MCL1286]$
All three of these sessions are in the meet the expert category and will give deep dives in their respective areas. Paul McSharry, VCDX, is presenting the vSAN session. This session looks valuable for anyone using vSAN to hear from an architect that has many deployments under their belt. The PSC has come a long way, but I still feel I don’t know everything I need to know about it. I always come across some sort of problem that support needs to help with. The PSC session looks to give the deeper understanding that I have been looking for. A lot of companies have vROps and vRLI deployed, but I rarely see anyone use the two products to their full potential. Using one or both of these products together is very useful for troubleshooting. I am looking forward to fine tuning my troubleshooting skills with this session.

Solid sessions make a virtual event and VMworld delivers again. I rather it be in person. However, it being virtual has some benefits, such as there’s a free option and more accessible to a wider audience. I will be again missing out on collecting a suitcase full of swag, but there will be giveaways from VMware and vendors, even during a virtual event. I won an official VMworld backpack last year!

vExpert 2021 and VMworld

I earned vExpert status for the forth year in a row. I was selected for my blog and contributions on the VMTN forums. I will continue my mission, which is writing articles that will assist someone that is facing the same issue I had.

VMworld will be only online again this year. It will be October 5-7, 2021. Not much has been announced so far for this event. I hope it is more interactive than last year’s online event. I definitely miss attending in person VMUGs and VMworld.

VMware vSAN 6.7 Specialist 2019 Exam Preparation and Experience

I recently passed the VMware vSAN 6.7 Specialist 2019 Exam (5V0-21.19) with a scare of 440. Despite the exam’s name, I earned the VMware Specialist – vSAN 2021 badge. Fortunately, VMware is now keeping up with the names of the badges and certifications by matching it with the calendar year, even if the test wasn’t yet updated. This is a nice change because I previously took the vSAN specialist exam in 2019 and earned the 2017 badge, which immediately looked out of date.

The exam was very straight forward. A lot of it is what engineers do on a weekly or monthly basis for many of the exam objectives. I had more than enough time to finish the exam. I don’t think there was any surprises so I recommend to know the exam objectives inside and out. Some exam objectives are not deep so don’t think too much into them. For example, objective 7.5, describe the effect of maintenance mode options; only need to know the three maintenance mode options and what they do. Though, there are some questions that were not worded the best, but that shouldn’t make it or break it for anyone.

The exam is based on vSphere 6.7 and vSAN 6.7 U1. Keep that in mind as there was a lot of changes in newer versions, especially in 6.7 U3. You are probably running a newer version at work and VMware’s HOL are no longer running the exam version of vSAN from what I saw. Though, VMware still has the lab manual for their retired vSAN 6.7 HOL. I found a lot of good information and screenshots in there.

There are only two requirements for achieving the vSAN 2021 badge; pass the 5V0-21.19 exam and have one of the pre-requisite certifications. The badge page says it needs to be a current VCP. However, there are many other certifications that also count, such as VCAP and VCDX certifications. Check out your certification manager on VMware Learn and you can see more details on what’s covered. Also, I wrote about this in a previous article about the vSAN 2017 Specialist badge.

vRealize Log Insight Not Connected to vCenter Servers After Upgrade

This is just a quick article that I have been meaning to do. I upgraded my Log Insight server. After the upgrade, it wasn’t collecting logs from my vCenter Servers. I had to accept a certificate for each vCenter Server and then it worked again. The setting is in Log Insight, under administration, and vSphere. I forgot to take a screenshot when I saw the error. Below is the location to accept the certificate.

Migrate a vCenter Server and Change its IP Address

I needed to migrate a vCenter Server between datacenters. A new IP address was required for the vCenter Server at the destination. The process of changing a vCenter Server’s IP address became a straightforward process in vSphere 6.5. However, to add a complication in my situation, I also needed to migrate the vCenter Server. Unfortunately, the destination network was not available at the source. Here are the steps I went through and then an issue I ran into. This was for a 6.7 vCenter Server appliance with an embedded PSC.

  • Backup the vCenter Server
  • Shutdown vCenter Server
  • Clone vCenter Server
    • This was only a failsafe if the vCenter Server does not work at the destination to avoid restoring from a backup
  • Power on the vCenter Server
  • Run and save an export of RVTools
    • I always like to do this before big vCenter Server work so that I know where all my VMs are at
  • If only using a vDS, verify you have a port group with ephemeral binding
    • I did not need it, but you might depending on your destination
  • Change the IP address of the vCenter Server
    • The new IP address was displayed in the vCenter Server console
  • Shutdown the vCenter Server
  • Migrate the vCenter Server with VMware vCenter Converter Standalone to the destination ESXi host.
    • Ensure to verify and/or make the following changes in Converter
      • Required VM Hardware Version
      • VMXNET3
  • Change the DNS records for the vCenter Server
  • Power on the vCenter Server
  • Verify if the vCenter Server vNIC is connected
  • Reboot the vCenter Server
  • Verification
  • Delete the original and cloned vCenter Servers at the source

I thought everything went well. All ESXi hosts and VMs appeared to be happy. However, a user reported his remote console for some VMs would freeze after about 30-45 seconds and I also noticed some vDSs had sync issues. I did some research and found out about 90% of my hosts had the old vCenter Server IP address in vpxa.cfg.

VMware has KB1001493 which covers this issue. There are two methods to resolve the issue in the KB article. Every host with the issue needed to be touched so a lot of tedious work required for both methods. At first, I went with method 1 and tried it out on one host. The host was not responding after restarting the management agents and required to restart the vCenter Server service. I did not want to have all of my hosts not responding for a long period of time nor did I want to restart the vCenter Server service after restarting managements agents on each host. Therefore, I went with method 2.

Method 2 had more steps, but seemed to be cleaner. Essentially, each host is removed and added back to the vCenter Server one at a time so that seemed like a better approach. Hosts with a vDS required a little more work and documentation since a host will not have access to the vDS when removed. That meant to first put each host into maintenance mode before starting the first step. Then go through the steps and repeat on the next host. This greatly reduced the risk of VMs hitting any road bumps since VMs were not on a host that was being worked on. Then, of course, add each host back to the vDS. If no vDS and only standard switches, than no need to follow my extra step since network connectivity will be fine when the host is removed from the vCenter Server. Keep in mind performance data, permissions (depending what level they are set), VM placement in a folder, tags, and events/tasks (host level) are lost when removing a host from a vCenter Server. By the way, I did not do step 4 (Reinstall the VMware vCenter Server agent). The referenced KB article only mentioned up to 6.0, but seemed to worked well.

Perhaps, there’s a cleaner way to do this process. However, all things considered, the IP address change and migration went well. No outages and the desired outcome was achieved. I did two tasks at once so I, at first, thought that’s why I ran into the host issue. Though, I expect many people may have had this same issue, without combining the two tasks I did, since there is a KB article on how to resolve the issue.

AWS Certified Solutions Architect – Associate Certification Exam Experience

aws-certified-solutions-architect-associate

I recently took the exam at home for the AWS Certified Solutions Architect – Associate Certification (SAA-C02). I passed with a score of 775. The exam was harder than I anticipated and I had a very tough time with the exam’s check in process.

I used a few resources to prepare for the exam: A Cloud Guru’s course on Udemy, practice tests on WhizLabs, and AWS FAQs. The A Cloud Guru course was a good overview of most of the exam objectives, but not thorough enough. The practice tests were very easy that came with the course. WhizLabs’s tests were nearly as hard as the actual exam and good to learn from. I recommend to read the following AWS FAQs: EC2, EBS, S3, Aurora, RDS, DynamoDB, VPC, Auto Scaling, and ELB.

The system check used by Pearson VUE is almost worthless. It does not do comprehensive enough checks on your computer and network. It appears to only check your Internet speed, webcam, and microphone. I did it a few days before the exam, the day of the exam, and every time it said I was good to go. However, I had an issue that prevented me from completing the check-in process and I barely got it working in time.

I got through the cell phone steps where I had to take photos of my ID, my room, and myself. Then Pearson VUE’s OnVUE gave me the following error message.

You have lost your internet connection. OnVUE is disabled until your connection is restored

My computer was obviously still connected to the Internet. I did the process again and got stuck at the same spot again. I got a hold of Pearson VUE’s support after waiting 18 minutes. They were not helpful at all. I did some troubleshooting and googling. I found a lot of people with the same problem and no resolution. Then I finally figured out what was the problem for me. I use Sophos XG Firewall Home Edition. Very rarely, Sophos’s IPS and/or virus scanner can cause odd issues with websites not working. I disabled both of those services and then I was able to move to the next step of my check-in process with one minute to spare. Pearson VUE’s system check should be much more thorough and catch issues like this so that the exam taker can work out potential issues beforehand.

VMware Training and VCPs/VCAPs Stats

VMware is giving away a free 6 month premium subscription to VMware Learning Zone for everyone. You need to register by November 6, 2020 and your subscription will expire 6 months after you register. That makes it easier to plan ahead to take advantage of a time frame that works best for you. You can even sign up with a personal email address. Check out the official article from VMware for more details and how to register.

VMware released details on the amount of VCPs and VCAPs. I like looking at statistics and it’s fascinating when vendors release numbers on certification holders. VMware last did this a few years ago. Unfortunately, I could not find the past numbers directly from VMware, but I did find it on vInfrastructure Blog. I will reference the data from the aforementioned sites in this article.

There are only 3,850 VCAPs in the United States and a total of 13,580 around the world. There are 98,576 VCPs in the United States and a total of 335,371 VCPs around the world. VCAPs have nearly tripled in the past few years and over 6 times more VCPs. I wonder if that is due to VMware changing their strict 2 year expiration on VCP certs.

The United States is on par with the world total of only roughly 4% of VCPs going ahead to achieve the VCAP certification. There are some overachieving countries like Egypt, which has 10% of VCPs getting their VCAP. Hopefully, VMware will give a list of totals in all countries. I made a comment requesting additional data and I will edit this article if I receive it.