Transfer Latency

Dell Compellent’s Storage Manager has many tools for managing, monitoring, and reporting. I will focus on one performance metric which is transfer latency. Of course, monitoring displays the performance of many aspects of the SAN. Transfer latency monitors more than just the SAN. It measures latency from every point, which is the NIC on the host and everything in between connecting to the back end of the SAN.

Tracking the source of the transfer latency can be difficult since there are many places to check. However, somethings can be done to narrow it down. Storage Manager can drill down to many levels such as the disks to monitor latency. If the transfer latency follows directly with write latency, then the problem is most likely not with Compellent. ESXTOP can be used on the host side, but can be difficult to know exactly what is causing the latency.

Compellent has documentation that pointed me in another direction which was network switch configurations. Compellent has best practice documents available for many switches. Link level flow control needed to be enabled on each switch port the Compellent controllers are connected to on a Cisco Nexus 5548UP switch. This change can be made with no down time to VMs. Change one port’s configuration at a time as a precaution. Below is the configuration provided by one of Compellent’s documents. Transfer latency went dramatically down as soon as the ports were reconfigured.

Switch Configuration Guides for Compellent and EqualLogic SANs

switch# configure
switch(config)#interface ethernet 1/1-32
switch(config-if-range)#priority-flow-control mode off
switch(config-if-range)#flowcontrol send off
switch(config-if-range)#flowcontrol receive on
switch(config-if-range)#exit

VMware vCenter SSO 5.5 Migration to PSC 6.0 Error

I am in the process of upgrading three vCenter Servers on 5.5 to 6. SSO was embedded on all three and SSO has been recently externalized to Windows Servers. The next step is to use the vCenter Server Migration Tool to migrate each SSO 5.5 server to PSC 6 appliance.

I went through the migration wizard and the migration was on its way. The PSC was deployed and the progress bar on the migration was moving along. However, when I opened the console for the PSC, there was an error; Upgrade EXPORT failed. Then the migration never finished.

UpgradeExportFailed

I ran the migration again with VMware support since they did not know what could be causing the issue. I opened up the console for the PSC as soon as it was deployed and there was a quick message to look at UpgradeRunner.log. However, there was nothing useful in that log. Then checked out upgrade-export.log.

There were network related errors in upgrade-export.log. I knew I inputted everything correctly into the migration wizard and there was nothing that would block communication between anything involved. The IPv6 address in the log stuck out to me. The SSO Windows Server had a IPv6 address, but the wizard never asked for anything IPv6 related. I disabled IPv6 on the SSO server, ran the migration again, and everything went well.

UpgradeExportFailed_Shell2

Here’s one way to view the PSC’s logs. At the PSC’s console, hit Alt+F1. Then type the commands below.

shell.set –enabled True
shell
cd /car/log/vmware/upgrade
less upgrade-export.log

Moral of the story is to disable IPv6 on all VMware related servers before using the vCenter Server Migration Tool. Then enable it after everything is on 6. This is just to be safe in case any other of the upgrades or migrations have similar issues. VMware support said they will have a knowledge base article on this issue. When they do, I will edit this article with a link so everyone can check out the latest directly from VMware.

AWS Public Sector Summit 2017

AWSsummit2017a

I attended the AWS Public Sector Summit in Washington, DC this week. I usually do not attend so many conferences. This should be my last one for a while. I just happen to live nearby so I didn’t want to pass on the opportunity to learn more about AWS.

The keynotes were not really what a typical keynote address is at a conference. They were only a few minutes of announcements and then a few customer stories each day. The biggest announcement was GovCloud East is coming in 2018. Though, a more specific time frame would have been better received.

AWSsummit2017c

There were a lot of break sessions to choose from. There was a session on VMware Cloud on AWS. I don’t think there was any big news and sadly still no date for GA. Here’s a few things I got out of the session. Customers will purchase this service from VMware. AWS services will be billed separate from VMware by Amazon. All AWS services can be accessed from a customer’s VMware cluster, which will be hosted at a AWS data center. ESXi will boot from a EBS volume and no root access to the host for customers. The underlying storage for the VMs will be vSAN, which is a minimum of 4 hosts. Elastic DRS will be able to bring up a new host fast. I think within minutes. However, removing a host will take more time as data needs to be moved from the disks on the host.

I attended a fascinating session hosted by Kevin Murphy, Program Executive Earth Science Data Systems at NASA. He talked about and showed some of the projects NASA has been working on and how they leverage AWS. Some of NASA’s projects pull in petabytes of data ever day from satellites. The data is available for free for every person and company in the world. Kevin demonstrated one of his projects called Worldview, which I provided a description below. I highly recommend everyone to check it out. You can add an assortment of layers with various information, such as all major fires around the world and by date.

This tool from NASA’s EOSDIS provides the capability to interactively browse global, full-resolution satellite imagery and then download the underlying data. Most of the 150+ available products are updated within three hours of observation, essentially showing the entire Earth as it looks “right now”. This supports time-critical application areas such as wildfire management, air quality measurements, and flood monitoring. Arctic and Antarctic views of several products are also available for a “full globe” perspective. Browsing on tablet and smartphone devices is generally supported for mobile access to the imagery.

I’m sure Amazon will be looking to expand the event next year. I heard the attendance was at least 7,500 before Tuesday afternoon, which seemed to catch Amazon by surprise. Despite the crowds, I had a good time and learned a lot during my two days at the event .

 

My Home Setup

I think the equipment an IT professional uses in their home and the time they take at home for technology is a reflection of their dedication. Keeping up with technology is hard to do solely at work. Using various devices and software at home is one way to help broaden one’s knowledge.

The cornerstone of my home network is a box running Sophos XG Firewall Home Edition. Sophos is more than just a firewall. It can do anti-malware scanning, IPS, VPN, and many other services. I can talk a lot about my experiences with Sophos. I will write deep drive of Sophos XG Firewall in a future article.

My wired devices are connected to a 8 port gig Meraki switch, MS220-8. I use a Meraki MR33 for a 802.11ac wireless access point. Both devices are cloud managed. I was a little skeptical at first, but I can now really say that I like Meraki’s dashboard that allows management of my devices. The dashboard allows a lot of configuration to be done before a customer even receives the devices. Then the device will pull down the config once it’s connected to the network.

I do not work in networking so I enjoyed seeing a different side then I am use to at work. I am not taking advantage of all features, but I enjoy testing what I can do. A wifi guest network can be easily isolated. Also, packet capture can be easily ran from the dashboard against an AP or port of the switch.

I, of course, build my own computers and enjoy the process. Though, I’m really due for building a new one. My computer is showing it’s age with an i7 Ivy Bridge, but it still gets the job done. I use VMware workstation when I want to spin up some VMs.

Here’s a couple device a have for fun. I have a Raspberry Pi, which I currently run RetroPie on it. RetroPie is loaded with emulators and makes it easy to play some classic games. I have a Steam Link that allows me to stream games from my computer. I got it on sale for $30 and it can stream anything from my computer so even non-gamers may find a use for it.

The most odd ball set of devices I have are MoCA network adapters, ECB2500C. MoCA is fun to say and an easy way to expand a home network over existing coaxial cable. For example, I live in an apartment that already had coaxial ran to a room that I wanted a wired connection. I just added couple of the adapters to easily extend my network.

My home setup is small, but I consider it elaborate and to the point. My goal is for it to be secure and easily functional for home related devices. I am not one for having devices for the sake of having them. I like to make sure I use everything I have.

 

VeeamON 2017 – Part 3

This will be my last article on VeeamON 2017, which will be a mix of my thoughts from the event. I meant a lot of great people throughout the entire event. Everyone was eager to offer their ideas and help. Customers and partners traveled from all over the world and I saw some as far as Singapore. The Canadians were my favorite to hang out with!

Veeam had a nice lounge area filled with Veeam employees. Attendees could ask technical questions directly to the support staff at the event. If they didn’t know the answer or needed logs, they would give priority to a ticket once it was created.

0517171358cc

Above is a photo of the show floor with the vendors and dining area in the back. Does not look like many attendees around, but that is because this photo was taken in between back to back breakout sessions. The back side of each vendor you see actually has an additional company behind it. There was a good mix of vendors from big names, such as Microsoft and VMware, to smaller cloud based companies.

I couldn’t miss talking about the food and parties. Veeam made sure to keep everyone busy all day and night. If all that wasn’t enough, there was always Bourbon Street for some late night fun. A lot of the food during the event was traditional New Orleans dishes and it was delicious. There was a pub crawl Wednesday night that spanned across four venues and each venue was unique. Then the Veeam party on Thursday night was amazing. The party was filled with live music, food, and drinks at a beautiful venue called Generations Hall.

I had a blast at VeeamON and I have enjoyed looking back as I wrote these articles. I’ve always been a fan of Veeam and now I am an even bigger one. If all this talk got you excited for attending next year, then you can now start making your plans. Veeam already announced VeeamON 2018, which is scheduled for May 14-16 in Chicago.

 

VeeamON 2017 – Part 2 – Breakout Sessions

I’m still amazed Veeam and it’s partners squeezed in 85 break out sessions throughout the event. I had a tough decision most of the time on which session to attend as many sessions shared the same time slot. Fortunately, the schedule listed details for each session saying if it was business, partner, or technical. Then technical had a few levels so that helped to describe how deep the session will go.

My favorite session was How to Back Up and Restore VMware vCenter Server Appliance (vCSA) and Platform Controllers Properly. The session was hosted by a Emad Younis, Technical Marketing Engineer from VMware, and Mike White, Technical Evangelist from Veeam Software. They started off talking about VMware Platform Services Controller. The PSC was new to vSphere 6 so it’s been around for a little while now, but good to get a refresher. Then they went into backing up and restoring the vCSA and PSC. Backing up the vCSA in 6 is the same as previous versions. Do not bother backing up the vCSA’s databases. Use Veeam or another product to back up the vCSA and PSC VMs. Then restore the vCSA VM directly to a host. However, the PSC is a little different if using Enhanced Linked Mode, which means there will be more than one PSC. Deploy a new PSC and join to the existing SSO domain. Replication will simply kick in and bring the PSC up to speed. If only one PSC, then just restore it.

I thought there would have been more love for Veeam ONE. There were actually only two sessions! I know it’s the sidekick to Back and Replication, but more sessions on it would have been appreciated. I attended one of the two. The second one wasn’t relevant to me since it was about how to scale Veeam ONE. My Veeam ONE server is fine. The session I attended,  Take Out the Guesswork with Veeam ONE & Chargeback, went into some reports and alarms. A couple of the reports, powered off VMs and VMS with no archive copy are ones that I never looked. There are a great many of reports so nice to get some tips on others to look at outside of my routine ones. They talked about a useful tip on creating an alarm for a VM that has not be backed up for defined amount of time. This is good to catch a bug that sometime stops backing up VMs within a vApp with no warning or error.

VeeamON2017-breakout

The photo above was from What’s New in v10: A Deeper Dive, hosted by Anton Gostev. His sessions seemed to be the most popular by far. People were lining up for his sessions that were in the smaller rooms, but at least the deep dive was in the big hall.

The breakout sessions were hit and miss as some of the presenters did a better job than others with going through their topics. However, I certainly understand that every presenter puts in a lot of time and effort. I got something out of each session I went to. Perhaps one day I can think of an interesting topic to present and start small at a local VMUG.

VeeamON 2017 – Part 1

VeeamON2017-keynote

I had an amazing time attending VeeamON last week. The event during the day was packed with many presentations and nights were filled with parties. There was a lot going on from a company that doesn’t have many products. However, they are the best at what they do and have a strong following.

The keynote on Wednesday focused on availability. Peter McKay, co-CEO, described data being as valuable as oil was last century. I understand where he was going with the comparison, but hard to compare a natural resource to data. He talked about how Veeam has been growing at a steady rate since the company formed 10 years ago. Though, they added 1,000 employees in the last year and will add hundreds more the next year.

I always wondered how the name Veeam was created. Turns out it is basically how saying the letters V and M sound. I thought that was interesting for how simple it is. After all, they are, of course, all about virtual machines from backing up and monitoring so makes sense.

John Metzger, VP product marketing, made announcements for version 10 of Backup and Replication, which is the next major release. There will be a new feature called continuous data protection. RPO can be set to seconds if someone has the bandwidth for it. There will be native object storage support to take advantage of a service, such as AWS’s S3. There will be a new Veeam agent for Windows. Backing up to a NAS will be supported. He said that a lot of customers have requested it, which I didn’t realize. New APIs for storage vendors; IBM, Lenovo, and Infinidat. A Microsoft fail over cluster will now be able to be backed up. Check out Veeam’s official page on version 10 for more details.

I’ve never used VMware’s Log Insight, but I have always wanted to Implement it. Veeam will have integration with Log Insight. I’m sure customers with Back and Replication, and Log Insight will appreciate one place to review logs with products that work so closely together.

The keynote had a few live demos. Veeam had their engineers discussing and showing the audience some of the new features. One of the first things I noticed when the version 10 client was opened was finally a tab on the left of the client for history. In the current version, it was a gear with no label, which made it look like it was for some sort of settings. Anyway, just glad they finally made where to view the history more obvious as everyone I worked with surprised that’s the history button.

To be honest, I wasn’t too interested in the keynote on Thursday. The speakers focused  mostly on Azure. I have nothing against Azure. Looks better than AWS in many ways. Just something I haven’t worked with yet. Anyway, nice that everyone that attended received the book Zero Day: A Jeff Aiken Novel.

Everything still looks green to me as if I have a filter on my glasses. Veeam loves the color green and makes everything a shade of green. Some of the speakers even joked they must wear at least 3 things that are green. I’m still slowly catching up at home. I will make my next article on the breakout sessions.