Rescuing A Poor Performing VMware Virtual Infrastructure

Rescuing Slow VM Performance

Does your VMware vSphere Environment require First Aid?

Rescuing A Poor Performing VMware vSphere Virtual Infrastructure

About 4 years ago I was hired by a firm to help rescue their VMware infrastructure. Though they had everything any top performing VMware environment needed for success, including a VMware TAM, it just wasn’t meeting customer requirements for performance.

The job began with me interviewing frustrated developers who demanded I move them off slow VMs back on to physical servers ASAP, “Get me off this VMware crap, now!” It was my job to calm them down and reassure them we were going to fix the slow VM problems.

Helps On The Way!

After about a week the problems were getting clear to me: mixed matched combinations of server hardware, poorly configured storage, oversubscribed network with backup running on the same switches (VLANs) and poor VM configuration and lack of knowledge and expertise.

Basically it was the typical story I’ve seen many times when the free version of ESXi turns deadly.

VMware is really an awesome technology if it’s set up right.

Unfortunately most companies for years have been deploying into an already badly designed infrastructure then expect VMware to somehow improve server performance while at the same time saving money. Adding VMware or any other hyper-visor product will only highlight the issues that already exist: bad storage, network, design and build practices. An unwillingness to change the way things are done in storage and network management will doom your virtual infrastructure real fast.

  1. First start by getting help from someone with expertise in the virtual infrastructure space. Normally this will be a 3rd party VMware Engineer that is not defending their own bad practices.
  2. Do an inventory and technical analysis on everything that VMware leverages: how networks are setup, VLAN configuration, firewalls, backup traffic, storage types, configuration, spindle count, drive sizes, RAID configurations, LUN sizes, Brocades, what shares the same storage and network equipment. Then start on server hardware: model, type, brand, CPUs, memory, blade or not blade, firmware, HBA, NICs. Finally – what VMware ESX/I versions are being used: patch level, VMtools and VM configuration, etc.
  3. Now decide what the long-term expectations are for this VI environment. For example, do you plan to virtualize everything, possible? If so, then it’s time to think about dedicated resources that are only used for virtualization. No more shared networks and storage environments.
  4. Redesign for the future and start migrating to a new tested, validated and standard virtual infrastructure that uses best practices for virtualization. Note: Ignorance is the enemy and allowing system, network or storage administrators to do foolish things will cause poor performance, outages and down-time.

A well performing virtual infrastructure is standardized, and in my experience, is scalable by adding more of the same blocks of capacity (example only: cluster of 4 hosts, using 4 terabytes, with networks the same and VM configurations following a documented and test design). Note: Try to stay away from one-off requests, if possible.

An unwillingness to change the way things are done in storage and network management will doom your virtual infrastructure real fast.

Diagnosing Slow VMware VMs

The worst environments are normally a result of leaders and teams doing dumb things requested by ignorant managers or executives. Here’s my suggestion on how to solve this.

  1. Get your vendor partners to come in and do lunch-and-learns or training sessions to help educate staff on products.
  2. Make sure you ask them  plenty of good thought-out questions about what the risks, or disadvantages are, of doing things that are questionable.

Hopefully your vendors will be honest when answering your questions (some may hold back because they sense what you are doing).

SOW for a Health Check

If worst comes to worst, call VMware and get a SOW worked out to have someone do a health check on your vSphere environment. I never understood this but executes find bad news more believable when it comes from a 3rd party.

Your turn to share tips and comments on how to tune and tweak a poor performing VMware and get rid of slow VMs?

 

Leave a Reply

vSphere Data Protection EOA
Finding The Best vSphere Backup Replacement For VDP (3 Alternatives)

Important Notification: vSphere Data Protection (VDP) End of Availability (EOA) That...

vSphere 6
8 Updates That Make vSphere 6 Better – Keith Barker

Editor’s note: Keith Barker has been a CBT Nuggets Trainer since 2012. Some...

VMware Interview Questions
25 VMware Interview Questions And Answers: Tough & Technical (Download PDF)

Free VMware vSphere Interview Guide In this VMinstall Guide, I’ll share...

Project Photon
VMware Project Photon: Technical Review for Linux Admins

My quick and dirty review of Project Photon. I was very happy...

DevOps Plan
Best DevOps Strategy Hack (Winners Start With Why)

What is DevOps? It’s a model. It’s a culture. It’s a...

DevOps Tools Download
DevOps Toolchain: Download 3 Free Tools Used By Pros (Ansible, Git, Jenkins)

Hack Your Resume with DevOps Skills Lately, I’ve been getting a...

DevOps Guide
The Best Microsoft DevOps Skills: 25+ For Windows SysAdmins

What is Microsoft DevOps? This is an interesting question because first off,...

continuous integration tools
The Best Automation Training Video Courses (DevOps Tools)

When I was just starting out as a Jr SysAdmin, I...

Assessing Your Company's Cloud Readiness
13 Cloud Readiness Assessment Tips To Guide Your Migration Success (Updated)

Planning your migration to the cloud? If you’re planning to move...

How Bare Metal Virtualization Made Cloud Computing A Reality
How Bare Metal Virtualization Made Cloud Computing A Reality

An Overview of Bare Metal Virtualization Let’s start by explaining what...

Advantages of Cloud Computing
Advantages of Cloud Computing (Private versus Public)

Advantages of Cloud Computing Whether you’re an enterprise IT department planning...

Free Offical Amazon VPC & EC2 Cloud Guides for Beginners

Free Cloud Guides for Beginners Are you researching Amazon’s AWS Virtual...

DesTechAZ
What’s So Interesting About Woz U?

On 10/12/2017, I had the privilege of attending the AZ Tech...

tensorflow training
TensorFlow Tools for Beginners (7 Easy Takeaways To Get You Started)

Introduction To Deep Learning As a follow-up to my book review...

job rut
How To Get MOVING When You’re Stuck In A Career Rut (Video)

You worked hard to get your degree and after graduation you...

Storage Engineer Skills
Why Storage Engineer Skills Are HOT! (Can You Say Big Data?)

3 BIG Skills New Storage Engineers Are Missing Out On… Behind...

VULTR Reviews
VULTR Review And Comparison: Best VPS Platforms Now Thru 2018
Best Hosting for WordPress
Case Study: Best VPS For Easy WordPress Migration
White Box Storage
White Box Storage that Rocks! Great for SME
cloudways review updated
11 Cloudways Review Pros & Cons From My Deep Dive Testing (VPS Series: Part 2)
Website for Small Business
Best Small Business Website Packages (6 Alternatives That Rank!)
Click here to learn how to Optimize WordPress Speed
WordPress Speed Optimization: Learn To Rank Higher In Google