Fun and Learning at VMware Explore 2022

VMware Explore US is this month! I am excited to be at an in person VMware event for the first time in three years. I have been planning out my schedule and it’s jammed packed as always. There are a lot of great sessions to chose from during the day and a lot of entertainment at night (party list).

Content catalog has 814 sessions, at the moment, spread across the four days and a VCDX workshop on Monday. Hard to do it, but I narrowed down what I could squeeze into my schedule. Keep in mind only up to three ‘Meet the Expert’ sessions can be added. I like to focus on sessions that I can apply what I learn as soon as I get back to work. Below is what I have on my schedule.

A First Look at Building Your Own Management Pack without Writing Code
Advanced Troubleshooting of ESXi Server 7.x for vSphere Gurus
Networking and Security Analytics, and the Evolution of the NSX Platform
Advanced Topics in VMware vRealize Operations
Automated VM Rightsizing with Ease
Advanced Topics in VMware vRealize Log Insight
PowerCLI and REST APIs – A New Beginning

The sessions I am most looking forward to are the VMware Code sessions that involve a NUC. The sessions are similar to what they did in 2019 with the Raspberry Pi sessions. You get to work on a NUC focusing on two different topics. There is limited availability for each session so register soon. If you are waitlisted, still show up. Worse case is you sit on the side line and listen to the presentation. If you are a vExpert and answered the program survey on attending, you will be given a NUC to take home.

VMware {code} Intel NUC Home Lab with Smart Sensors
VMware {code} VMware Tanzu on Intel NUC Lab Environment Setup and Application Building

The online events over the past two years were good filler, but it’s hard to match the networking that is done at this event. I am looking forward to learning from and hanging out with the best in the industry. I am sure the expo area will be full with vendors and a great party on Wednesday night hosted by VMware. I wonder what big announcements will be made and what artists will be at the official party.

VMware Skyline Advisor Pro Technologist Badge

VMware is offering free Skyline training and a Skyline badge. There are training modules and questions after each module. I spent around three hours to get through it all. Score at least 80% on the questions and you will earn the VMware Skyline Advisor Pro badge. The offical article for it called it a certification, but it’s deiniftely a badge. It took 6 days after I completed the course for it to show up on my Credly account. The course showed up on my course transcript summary in Customer Connect Learning. However, the badge did not display there and did not on myTransript at All of my other certificates and badges are there.

Should receive a check mark after each module is completed.

The first three modules are unnecessarily repetitive. The last three modules dive into more areas: API, security, and vROps intergration. I got the most out of the vROps intergration module since I did not know this was possible. vROps can display a dashboard with a subset of Skyline data. Though, this module doesn’t go deep enough to be able to implement this intergration and the documentation I found is spotty. I will do a separate article on this topic.

I recommend all VMware users to go through this course. It’s definitely worth the time. I am sure everyone will get something out of it and receive a new, shiny badge.

vExpert 2022 and VMworld

That time of the year again for this article. I am now a vExpert for five years in a row. After two years of only online events, VMworld is back in person and will have some sort of virtual option. Will be interesting to see how the king of hybrid cloud will handle a hybrid event. Starting August 29 – September 1 in San Fran and then November 7-10 in Barcelona. I imagine proposals should be starting soon and then more details to get everyone pumped up.

Where are those vCLS machines hiding?

Everyone has noticed the tiny (1CPU, 128MB memory) vCLS machines (vSphere Cluster Services) that deployed in our environments after the 7.0 U1 upgrade. I like the concept of them that remove the dependency of vCenter being up for DRS to function. I envision that VMware will add more features that no longer depend on vCenter in future releases. There’s a little quirky thing that only an admin account in vCenter can see the vCLS machines. Searching with a non-admin account will only find the vCLS tags. Not really many scerionos that someone needs to interact with them, but one is to migrate them off a datastore you don’t want them to use.

vRealize Password Trouble

Even if you document every password you created, you may still run into password related issues. Root password expirations are easy ones to miss, especially when you don’t know when the password is going to expire. Most vRealize products expire root accounts password after 365 days. I recommend disabling password expiration for root and admin passwords for vRealize products if you can. Of course, still rotate the passwords. That way you are not in a tough spot if you miss the chance to easily change it. Either way, make a reminder in your PAM to change the passwords if you have a policy to rotate passwords. Below are some tips if you cannot log in to one of your vRealize products.

For example, if you are sure you know you are using the correct password to SSH with root and it’s saying your password is wrong, start off with rebooting each node one at a time. Then SSH into each node with the current password. Hopefully, you’ll be prompted that the password expired and to change it. This also works for NSX Manager nodes. If it’s an account that was recently locked out, waiting about one hour worked for me in the past.

VMware has documentation for each vRealize product to reset the root password. It’s generally booting into single user mode when using Photon OS. It also has ways in documentation to disable the password expirations.

If you are using vRealize Lifecycle Manager, make sure to update passwords in its Locker. Your accounts will lock out if you don’t do it.

Configure SMTP for everything that has the ability and set email addresses for all accounts. Password reset links are emailed out for Log Insight. Therefore, best to have this squared away ahead of time in case a user needs their password reset.

End of Year

Saying the end of this year has been busy is the understatement of the year. The log4j vulnerbilities hit an extreme amount of products from various vendors across most enterprises. I have applied workarounds to nine VMware products to remediate Log4Shell. It has seemed almost neverending with new vulnerabilities related to log4j being discovered as the past couple of weeks have unfolded. It was nice when VMware offered scripts to do most of the work. I am happy that ESXi was not affected.

Make sure to apply the workaround to your NSX Intelligence Appliance. I think it’s an easy one to forget about. Also, KB87150 for this workaround has a small typo in step 6. Easy to miss it when copying and pasting. Will get an error message; command not found. Below is the correct syntax. I reported the typo so, hopefully, it will be fixed soon.

Step 6: dpkg -i zip_3.0-12_amd64.deb

I am glad to be a part of the vExpert program for the past four years. I have gotten a lot out of it from discussions on the private vExpert Slack to licenses for nearly every VMware product. The program has expanded every year I have been a part of it and I am excited to see what’s in store for next year.

Jumbo Frames

Jumbo frames are the way to go for vSAN, iSCSI, and vMotion VMkernels for increased performance if able to support the larger MTU on all networking devices. MTU of 9000 is supported on standard switches and distributed switches. A mismatch of MTUs can happen and network traffic may seem to be fine when it’s not. vmkping can be used to help test everything is configured correctly for MTU size.

The MTU size can be changed when the VMkernel, vSS, or vDS are in use. Though, there will be a short loss of network connectivity. No worry for vMotion. Can just temporarily disable DRS so no vMotions kick off. Though, careful planning is needed for vSAN, iSCSI, and the virtual switches they are connected to. If a host is using a vSS, the host can be placed in maintenance mode. Then change the MTU for the VMkernels and vSS. However, if using a vDS, the vDS MTU change effects all hosts in the cluster. Also, the VMkernels need to be changed for each host. I tried it on a 5 host vSAN cluster and the vDS change took 72 seconds for the MTU change to apply. VMs lost connectivity during the change and then were fine when the change was fully applied. Of course, this is not recommended to do in a production environment.

My Top Picks for Sessions at VMworld 2021

VMworld is virtual for the second straight year. The event is October 5-7 and will have content around the clock to accommodate various time zones. There are two types of passes available; the general pass is free and the Tech+ Pass has a charge. Some of the sessions are only available with the Tech+ Pass. The content catalog has been released so it’s nice to get an idea of what’s happening to plan ahead. I like to focus mostly on sessions that will help me add value to the work I do today. Some sessions will be pre-recorded and some will be live via Zoom. As always, there’s a lot of sessions to choose from. I went through them all and below are my top picks.

Deep Dive: VM Performance and Best Practices [VI2158]$
Extreme Performance Series: Performance Best Practices [MCL1635]
Extreme Performance Series: vSphere Advanced Performance Boot Camp [MCL2033]$
It’s good to see sessions based on the fundamentals of vSphere and then taking those fundamentals to the next level. All three sessions sound similar and rated at the highest technical level. Both extreme performance sessions are hosted by Mark Achtemichuk and Valentin Bondzio, and one of them is free. Mark is a VCDX, and whenever a VCDX speaks, I pay attention. These sessions should deliver useful information for all VMware admins. Mark and Valentin both work for VMware so I am sure they have seen a million different problems and can help people better troubleshoot when they face these problems.

Core Storage Best Practices Deep Dive [MCL2071]$
VMware vSAN – Dynamic Volumes for Traditional and Modern Applications [MCL1084]
VMware Storage Queue Tuning [VMTN2862]
The first session I listed above will be hosted by Cody Hosterman and Jason Massae. I have watched Cody present before at a local VMUG and other events, and he always delivers. He works for Pure, but don’t let that fool you into thinking he’s there to only promote his product. He always gets to the crux of storage that applies to nearly everyone. Duncan Epping, VCDX, and Cormac Hogan are presenting the vSAN session. They will be discussing the newer vSAN File Service and more cutting edge integration with vSphere Container Storage Interface. I am very interested in what storage tips all of these rock stars will give that I can apply at my job.

The Clock is Ticking on NSX Data Center for vSphere [EDG2721]$
NSX Data Center for vSphere to NST-T Data Center – Migration Approaches [NET1211]
NSX Data Center for vSphere to NSX-T: Strategies to Make the Move [EDG1515]
NSX-V reaches end of support on January 16th, 2022 and that’s the basis of these panels. Jayson Block, VCDX, is hosting the third session linked above. If you are still using NSX-V, you have to attend at least one of these sessions. All three sessions will give good advice on developing a migration plan. Hopefully, this will help to jumpstart your planning and migrating to NSX-T before support ends.

A Field Guide to Health Check vSAN to Operate, Upgrade and Transform [MCL1825]$
A technical Review of PSC. VMDIR, Replication and Snapshots [VI2447]$
Troubleshooting with vRealize Operations and vRealize Log Insight [MCL1286]$
All three of these sessions are in the meet the expert category and will give deep dives in their respective areas. Paul McSharry, VCDX, is presenting the vSAN session. This session looks valuable for anyone using vSAN to hear from an architect that has many deployments under their belt. The PSC has come a long way, but I still feel I don’t know everything I need to know about it. I always come across some sort of problem that support needs to help with. The PSC session looks to give the deeper understanding that I have been looking for. A lot of companies have vROps and vRLI deployed, but I rarely see anyone use the two products to their full potential. Using one or both of these products together is very useful for troubleshooting. I am looking forward to fine tuning my troubleshooting skills with this session.

Solid sessions make a virtual event and VMworld delivers again. I rather it be in person. However, it being virtual has some benefits, such as there’s a free option and more accessible to a wider audience. I will be again missing out on collecting a suitcase full of swag, but there will be giveaways from VMware and vendors, even during a virtual event. I won an official VMworld backpack last year!