Best Practices | VM Install

P2V – Step by Step Removing Physical Hardware

It’s been a while since I had to P2V a Windows server but recently I found myself on a project doing P2Vs of business critical servers, again.

FYI: The new version of VMware VM Converter works great for converting a physical Windows and Lunux server into a virtual server.

Once the conversion is completed, the new VM is online, VMtools are installed and video acceleration is set, here’s a couple more steps I learned about at a VMUG meeting that I recommend doing to finish the P2V job.

First – Remove Old Physical Hardware from the New Windows VM (picture shows “grayed” disk drives)

Step 1 - Log into the VM

Step 2 - Open a command prompt
-    Start > Run
-    CMD
-    Enter

Step 3 – At the prompt type: set devmgr_show_nonpresent_devices=1
-    Enter
-    Type: start devmgmt.msc
-    Enter

Step 4 – Wait for the Device Manager to open, then click on View > Show Hidden Devices
- Expand each category and search for “grayed” hardware
- Right click and click uninstall when “grayed” hardware is located
- Don’t worry about the System Devices category
- When all the hardware devices are gone, close the Device Manager and reboot

Second – Uninstall Old Hardware Specific Software

Step 1 – Log into the VM

Step 2- Start > Control Panel > Add/Remove Programs

Step 3 – Carefully find and uninstall programs from IBM/Dell/HP/other that are used for drive arrays, networking, ILO or any other similar application that is specific to the physical hardware.

Note: These items are not required now that the server hardware has been virtualized.

Done – When all the applications have been uninstalled, close the Control Panel and reboot

Q: Why do this?

A: The reason I do these steps is because over the years I’ve had to troubleshoot VMs that I found were P2V’d by someone else and not cleaned up. This normally causes poor performance and issues with ghost hardware that conflict with the virtual hardware devices, more specific – network adapters.

There – now all you need to do is make sure all your best practices are applied to the VM and it’s ready for production.

Originally posted 2009-09-19 07:22:18. Republished by Blog Post Promoter

10 Biggies to Help Managers and Admins Avoid Virtualization Pit-Falls

I wish this information was available 10 years ago when I started working with virtualization, but then again, like many reading this, I thought I was an expert and didn’t need it. Now I see myself as a student because of how fast virtualization is changing.

1. First and most important, look at the big picture for why you are implementing virtualization. Most managers look solely at VMware, XenServer, Hyper-V or any other virtual server product for ROI (return on investment). Bad way to make IT decisions! Look at the big picture. How will virtualization affect everything and everyone it makes contact with? For example: How will your storage be affected when you start sharing it with hungry VMs, will the I/O hold up? Or, how will your system administrator handle the new responsibilities? How will you handle the users when they start complaining that everything is slow because you didn’t consider I/O and new responsibilities?

2. Once you have a big picture view of what you want to do with virtualization then consider that you’re still probably missing a few things that you will learn along the way. Just look at these challenges as growing pains, and unavoidable. Virtualization dynamically changes as the environments grow and upgrade. First there’s the experimental ESX or free ESXi, Hyper-V and XenServer host that gets you started. Then when the experimental host(s) get filled up there’s the small farm of host servers that get landed when you actually start purchasing new hardware and the full infrastructure licenses. Beware! of the “wow we can virtualize everything” period that happens from 50 – 200 virtual servers. At this point everything seems to work fine because you haven’t saturated your SAN I/O, or host memory and CPUs. But then there’s that point that happens at VM number 201 (201 is a relative number, it could be more or less depending on a number of factors) where panic is unavoidable if you haven’t prepared properly. That’s why you need to read the rest of this post.

3. Now that consideration 1 and 2 are out of the way, which are mainly to make IT managers think, I’ll get to the good stuff. Have a backup strategy in the beginning that is made for backing up the VM images. Don’t rely on your legacy backup software for physical servers. Yes NetBackup or whatever can still do agent backups of files of a VM, this is a no brainer. However, how are you going to do a full system restore? Unless it’s just data, 5 hours after your backup administrator begins the system restore he is still going to be trying to solve this riddle. You want a good solution that makes an image backup. Solutions: VCB, vRanger Pro, Veeam Backup, Avamar. These are all specific backup tools for virtualization. Avamar can work on any type of virtual environment including Sun containers.

4. Know your storage limits. Capacity is just one part of the storage requirement. The other part is I/O or OIPS (Input/Output Per Second). VMs have different I/O needs. One hungry database or SharePoint VM on a LUN that shares it’s disk parity with multiple LUNs can cause performance problem across all the LUNs in the disk parity group. The best way I have found to avoid this is to design your storage with the biggest I/O pool available. I/O begins at the disk and 15K disks have roughly 200 IOPS where 10K disks have 150 IOPS (SATA have 30 – 50 IOPS). Do the math, which is better? After capacity and I/O is considered, then there is the pathing, which needs to be manually configure to split I/O down multiple paths to the SAN/NAS cache. I’ve seen million dollar equipment brought to its knees because this stuff was overlooked. It’s usually not the equipment (HP, NetApp, EMC) that is causing the problem, its configuration. Whether you plan to use FC, NFS or iSCSI, this is important for your storage administrator to consider. Otherwise, you will be playing VM storage Tetris and I guarantee you will lose.

5. This is in conjunction with 4, VM template configuration. If you’re planning to have a huge pool of I/O then you will never know your template configuration is poor. VM configuration is important and is easy to overlook. Most will find out how important when I/O runs out… I’ve read this best practice on many blogs – “put data and OS, and even swap files on separate LUNs.” I agree this is a good best practice, but I am taking it even further and adding a criteria. “Separate LUN on separate disk parity groups.” Here’s why, ten – 15K disks will give you roughly 1500 IOPS across each LUN it is carved into. Depending on the size of the drive you may have various LUN sizes of 200 to 500 GB (each with 10 – 20 I/O hungry VMs) sharing the same IOPS. Splitting data, OS and swap onto more spindles will give you more IOPS and possibly an alternate path to the 2nd storage processor (active/active), or more cache that is assigned to another FC or NIC port. Make sure data store names include what the LUN is for (Data, OS or swap) and odd even disk parity (data goes on odd, OS go on even).

6. Clean up your messes. Don’t leave old proof of concept (POC) VMs or equipment running after the POC is done. Nothing is harder to do then to clean up a VM environment 2 years after everyone who was on the original project team has left and your VM inventory now has 500 VMs in it. The first place you need to look when you hit your host and storage limits is here. Out of 500 VMs you can bet there are at least 50 VM zombies that are idly running and using up precious resources. Then there’s the clean up of zombie VM folders that are from VMs that were improperly deleted and the files were left on the data store (you know the VM you said you’d delete later – that was 2 years ago). Clean up also helps control “Sprawl”. Sprawl is a fancy word for out of control.

7. You probably didn’t hear me the first time so I’m saying “Backups” again. I’m putting this down again to make sure you have a backup solution that backs up the complete VM image. It’s no easy task to change backup process 2 years and 500 VMs later so make sure you do this right from the start.

8. Establish standards for your environment. All hosts will be on the certified version of ESX or whatever hypervisor you use. Once you allow old hosts to say around after you have decided to build new host on the current ESX version, it won’t be long before your virtual infrastructure is fragmented. Remember, virtualization is evolving almost daily and new features are on each new version of ESX and Hyper-V. Live migration didn’t work on the old Hyper-V version but it work on R2, but it doesn’t work across R1 to R2 or R2 to R1. Get all those R1 upgraded to R2 so all are the same and live migration works. Keeping the standard isn’t easy because VM administrator are also system administrators, they have to land the servers, configure the host, as well as deploy the VM and configure the VM. It’s the same people doing both jobs and in some cases they are storage and network administrators too. Make sure you have enough staff to maintain your standards. I’ve known more than a few overworked, underpaid and miss-understood VM administrators in my time.

9. I hate this one as much as any true IT professional but someone has to keep doing the job if you leave and take a better paying job somewhere else. Make sure you keep good documentation. If it’s required, cool Visio’s of everything is nice for management, but even more important for day to day support staff are “How To” documents. How to land and provision a host (hardware and hypervisor). How to deploy a VM. How to add additional disk space to the “C” drive of a VM. How to P2V a system. How to properly request more storage. How to decommission a VM. How to schedule a VM backup. How to recover a VM from a backup. Also keep the “How To” documents up-to-date. You need a new “How To” for each version of ESX because they are not the same; customization to the SWAP volume for example is different on 2.5, 3.0 and 3.5. Hyper-V and XENserver have their own little tweaks as well.

10. Don’t buy every tool out there thinking its going to fix everything I have spent the last 2 hours writing about. Listen to what I am saying. Listen to your support staff. Carefully listen to vendors who want to sell you something because there is no silver bullet for poor planning. And, while on the subject of vendor, any consult recommendation with direct connection with equipment vendors should also be scrutinized. I’ve seen the best SAN money can buy collapse under 25 VMs because it was haphazardly used (VM storage Tetris). Many of the problems I have warned you about can be avoided if you plan. Read number 1 and 2 again until this makes sense. To the VM administrators who are fighting the daily battles because most of what I have written about is already occurring in your virtual environment, I feel your pain. To all the new bright-eyed IT managers and system administrator who are licking their chops because they are finally getting a budget to start virtualizing, I warn you and say, “Consider the big picture and plan, plan, plan!”

Hopefully this post has been helpful. Other items that were not covered are: How to monitor VM and host servers, disaster recovery DR of virtual environments, capacity planning, forecasting and hardware (servers, network and storage) brands and types. These can be topics for the next 10 biggie list. My final note is “Backups” will challenge traditional thinking so heed my warnings.

Originally posted 2009-03-29 10:38:28. Republished by Blog Post Promoter

Why you need a step by step guide for installing ESX

For more than 10 years I’ve been installing server technology and it’s never been easy to get things operating correctly on the first go-round. Even with the best planning, there are always unexpected things that come up and cause delays. That’s why you need a step-by-step guide for installing VMware ESX Server and Virtual Infrastructure Technology.

http://www.vminstall.com/step-one-start-your-vmware-esx-install-creating-checklist-important-information/

Originally posted 2008-05-23 17:26:39. Republished by Blog Post Promoter

Best Fit Virtualization Criteria Checklist

Finding servers which are the “Best Fit” canadates for virtualization can be tricky. I have put together the following criteria checklist to help determine whether a new server provision or a P2V should use virtualization.

The Best Fit checklist uses my own criteria and weight recommendations but it can be modified with more liberal values. However, remeber the “Best Fit” for virtualizing a server should always consider the user’s experience, too liberal will produce poor performance and complaining users.

The “Best Fit” file has been attached for you to use it as-is or you can make your own. Open file: best-fit-virtualization-criteria-checklist-vminstall

Best Fit Virtualization Criteria Checklist

Requestor: ____________________________________________ Date: ___________________

Server Name: __________________________ Server Role: ______________________________

Primary Application: ______________________________ Database: ______________________

Purpose:

The purpose of this checklist is to evaluate new server provisions for server virtualization. The checklist criteria are designed to only allow “Best Fit” usage for virtualization. A weight of 3 or more will waive the server’s provision for virtualization.

Instructions:

Each item has a weight assigned. While going down the checklist place the weight value on the line to the left of the items that apply. Once you are complete, total the weights.

Checklist:

____	1. Is the primary application supported when hosted on a VMware VM? Yes/No (If no add 3 Pt.) Self explanatory.
____	2. If this is a database server will clustering be required? Yes/No (If yes add 3 Pt.) Caveat: If clustering is required, high availability is hindered on the host server.
____	3. Is this server considered a fail-over for a physical server? Yes/No (If yes add 3 Pt.) Caveat: Fail-over servers should be provisioned on an equivalent platform.
____	4. Can recommended system requirements be scaled back (CPU/Memory)? Yes/No (If no then use items 5 and 6 to calculate weight.
____	5. Will memory requirement be greater than 2 GB? Yes/No (If yes add 1 and .5 for each additional 512 MB)
____	6. Will disk requirements grow beyond a 20 GB (C: or Root disk) and 50 GB of combined additional disk space. Yes/No (If yes add 1 and .5 for each additional 50 GB) How much? _____GB
____	7. Will this be a temporary installation/deployment? Yes/No How many months? _____ (If yes can Lab Manager be used?) (0 Pt.)
____	8. Will a database be hosted on the server (SQL/MySQL/Oracle)? Yes/No Other ___________ (If yes add 1 Pt.)
____	9. _______________________________________________________
____	10. ______________________________________________________

( ) A server with a total 3 and greater should not be virtualized.

Evaluation Summary: _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Originally posted 2009-02-21 09:54:36. Republished by Blog Post Promoter

ESX Swap File Size Best Practice

Do a custom installation of ESX and make your swap partition 1024 MB instead of the default 542 MB. Later you will need the room when you have to increase your service console memory from 272 MB to 512 (800 MB Max) to keep your systems from crashing when you access them from within a service console window.

If you have a VMware best practice that will help someone, please post it in our best practice forum! Thanks!

Revised on 1-3-09:

The above is only for ESX 3.0.x. VMware has redesigned the volume structure for ESX 3.5.x to include the larger volumes. The large SWAP volume is only standard on clean installs of 3.5.x, upgrades of previous versions will have the smaller volumes. Best practice for upgrading ESX is to do a clean install to remove old version limitations.

Then when running the installation make the following changes and additons to the default volumes:

/boot | ext3 | 250 MB
/swap | swap | 2000 MB (largest accessible size) Once this is changed you must increase service console memory in VirtualCenter 2:1 (swap:ram)
/ | ext3 | 5120 MB
/var | ext3 | 1024 MB (core files)
/var/log | ext 3 | 1024 MB (logfiles)
/tmp | ext3 | 1024 MB (where updates get downloaded to)
/opt | ext3 | 2048 MB (where HA log go)
/ home | ext3 | 1024 MB (user files)

Originally posted 2008-05-23 17:26:22. Republished by Blog Post Promoter

Naming Standard for Virtual Infrastructure

Like a new mom and dad, have you given any thought to what naming standards you plan to use for your new virtual environment? Right, I said naming standards because you will have to have names for the VirtualCenter or VMM, datacenter, clusters, data stores (LUNs), ESX or Hyper-V host servers, VLAN tags, VM guests, user groups, alert rules and possibly few more services and roles.

I kid you not – six months after you setup your virtual infrastructure it will dawn on you that you should have thought this out sooner, uh, why did we call the data stores datastore-hp-8000-01 thru datastore-hp-8000-128 even the NetApp ones? And, why did we call all our VLAN tags “temp-v120” through “temp-v897” when they are not “temp”? Now you have to go back and change all that stuff and that could break scripts if you have any running, not to mention it is a slow process and what about documentation?

What are your best practices for naming?

Nip the naming standards in the bud ASAP because your VM environment will grow very fast and soon it will look like a spam email with upper-case and lower-case names, and host name like Viagra and Mickey Mouse, get the point?

I recommend a naming convention that uses a tree: Datacenter>cluster>host>datastore

Ex. Datastore = DC1-CL1-((server name) (rack id))-((LUN ID)-(Datastore ID))

Ex. VLAN tag = VLAN1 (all upper-case)

Remember to keep names simple and don’t let someone get lazy and cut corners or it will be you cleaning up the mess.;-)

Originally posted 2009-02-05 16:47:37. Republished by Blog Post Promoter

To V or Not to V is the Question?

Buzz Word

Has virtualization become a buzz word in your strategic plan? Take the advice from someone who has spent years supporting VMware in three different enterprise environments – think before you V.

If all you plan is a small experimental cluster of virtual servers, it’s no big deal. The software is even free. VMware ESXi is the best way to go. But, remember, it’s only a seed for future growth. Growth which will evolve into a nest of unexpected problems if you don’t think before you start to roll out ESXi hosts for your DEV, QA and PROD environments.

V is not for Vendetta

Cynical as this may sound, it’s more truth than fiction! Five months from the day the first ESXi host is deployed in your datacenter you will have users complaining about poor performance, and your smartest system administrators will begin many long hours of research to solve riddles for virtualization backups, migrations, storage and other strange unknown glitches that are attributed to the “V” word.

Fix for SCSI Errors Filling Up Logs

If you are dabbling, like I said earlier go for it. Virtual servers are fun to work with – that is – until you get too many on one host and you need to reboot the host but you can’t because there is a DEV server that somehow has become a production server and is running a business critical application that absolutely cannot be bounced! Now your are caught between complaining users, worried managers and not knowing for sure if the fix applied by your system administrator will be the solution for the SCSI errors that are filling up the logs and killing performance. All you need to do is bounce the host to see if changing the queue depth does the job. It should because the ESX command was found by your system administrator when Googling “Fix for SCSI Errors Filling Up Logs”

and reading through 50,000 threads about SCSI errors and VMware.

I am not trying to be sarcastic; I am trying to make you think! The incident about the SCSI error is a real situation that has played out in two of the three VMware virtual environments I have worked in. Google “SCSI error” + VMware and see for yourself.

Ignorance and Arrogance

The two biggest problems I have noticed since I started supporting VMware are ignorance and arrogance. Here’s what I mean.

“I” is for Ignorance

First, just about every virtual environment starts off with ignorant managers who do not understand what’s required to build a successful virtual environment. They just know everyone else is doing it and it’s supposed to save a lot of money. Unfortunately, all the future saving will go out the windows with support and license costs and lost labor. Support and license costs are easy enough to understand, but what do I mean by lost labor? Well, lost labor is the many hours, days and weeks that your top system administrators spend trying to solve VM riddles. Even when you purchase expensive support the problems don’t go away. In most cases someone just finds a work-around because the solution for the root cause is purchasing a new SAN or more powerful hardware because someone didn’t bother to check the HCL (hard compatibility list).

“A” is for Arrogance

Oh, then there’s the problems associated with arrogance. Here’s how this one plays out. You and the stakeholders decide to deploy a virtual infrastructure to save money, right? You start with the basic package of three hosts and a virtual center which quickly grows to 16 and 32 hosts (whistles and bells included are: vMotion, HA, DRS). Then you schedule a 30 minute meeting with your systems team and pick your best system administrator to lead the “Virtualization Project” (maybe even send him to training). He/she has built a ton of servers and even has a MCSE or RHCE. VMware is easy to setup. Just load the hosts, install the virtual center, and start building and P2Ving servers into the environment. It’s so easy, anybody can do it, right?

A week later, Wow! Everyone thinks you have the VMware god working for you. Pretty soon the VMware god gets vMotion running and HA to work. Next, scripts are running to reboot and auto-config VMs. Another 30 minute meeting is scheduled to plan out migrating servers into the virtual infrastructure; this plan is called the “Server Consolidation” Project. The system administrator has become a VMware expert in two months and this is his/her virtual environment…. Quoting from the good book, “Pride comes before the fall” .

I am a firm believer that history repeats itself so here’s what I am predicting happens next. Beneath the surface of your virtual world trouble is brewing. First the problems are small and your VMware expert can figure them out, then the problems get bigger and like any normal person would do, the expert doesn’t tell anyone because they’ve gotten used to being called the VMware god and they don’t want anyone to know they are really mortal. Soon emails starting trickling into your inbox and you are the CC:. The subject states “why is my VM so slow logging in?”. Soon it’s no longer a trickle and you are now the “To:” from all the users and the VMware god is at the end of the list which includes everyone in the GAL and your boss. The cat’s out of the bag, there’s a problem with VMware and everyone now knows the VMware god is mortal and they come looking for you. No this is not a Steven Spielberg novel it’s real world stuff…

How Do You Measure Success – Invisible

The worst thing that could’ve happened to your virtual infrastructure has happened. Technically things are fixable but what will not be fixed any time soon is the bad user experience that has affected hundreds or even thousands of users who work on servers that were virtualized. All your effort to sell virtualization to the business units has just been destroyed and users don’t want virtual servers any more. Managers even demand that all their applications are moved off your virtual infrastructure because they can’t handle bad performance and complaining workers any longer.

Rule of thumb: The measurement of a successful VMware deployment is not how many VMs can be hosted per ESX host, its how many users can be satisfactorily serviced without them knowing they are using virtual technology. Virtualization should be invisible. Once users start noticing foot prints in the snow, it’s over before a shot is fired.

Do It Right the First Time

If to V or not to V is still the question then here’s my recommendations? Don’t even setup a single host, even with the free version ESXi if you don’t plan to do it right. It will just snowball from there into a monster with unexpected results and in the end users will think all virtual environments are the same – crap!

If I haven’t deterred you and you still want to virtualize, then here’s the right way to do it. (Pardon my arrogance).

First, work closely with vendors. Listen to them and let them guide you. Yes, they will try to sell you better equipment, but not because they want to take advantage of you. They can’t tell you this but they know you will have problems if you buy the cheapest equipment. My guess is you will still be tempted to buy the cheap stuff then later you will blame the vendor when the users start emailing you about their bad experience. As I stated earlier, it will be too late and you are to blame.

For those who do listen to their vendor’s advice, the next item is more important. Hire an experienced VMware administrator. Don’t try to turn a system or network administrator into a VMware administrator. VMware administrators are specialists just like a DBA. Here’s why, VMware is an infrastructure technology, not a server technology. Sure a system administrator can become a skilled VMware administrator, I did. Here’s why you need someone from the outside, they will be objective about things like storage, networking, and quality. Existing internal staff will cut corners because they always do, you just don’t know about it. And, chances are likely that you will trust an outsider’s discretion more than a buddy or subordinate. If it’s done right the users will never know the difference, if it’s done wrong, everyone all the way up to the CEO will be sending you an email.

Third, virtualization requires total collaboration with storage, networking, developers, operations and business unit managers. If you can’t enlist them to work together as the “Virtualization Team” forget it. There’s nothing more frustrating than trying to figure out a problem when storage administrators will not let you verify they are using “Best Practices” for zoning your ESX host server storage fabric, or when a network administrator will not do what you ask when you tell them why curtain VLANs need to be trucked on all four data ports for the same host. The list of people hassles goes on…

Forth, this is more important than food and water. Please, heed my warning. After everyone has done the best they can do to make sure your virtual infrastructure will work and be a success. Do not come out of left field with your own ideas and change everything after weeks and months of planning. Don’t change the SAN from tier 2 to tier 3. Don’t buy a cheaper model of server. Don’t redo how the servers will be backed up. Don’t change anything… The time for making changes is during the planning process or when an unexpected problem surfaces. The problem should not be you at the last minute throwing a wrench into the mix. Please!

Finally, virtualization is a great technology that will eventually find it’s way into your datacenter whether you want it or not. The way you choose to go is simple: let it evolve into a monster that everyone will hate or, plan and architect its deployment into something nobody knows exists. Remember, virtualization should be transparent to the users. When they know it’s there, someone hasn’t done their job right.

Conclusion!

To virtualize or not to virtualize is the question? My answer to this question is in the form of a question, “How important are people to you?” If their satisfaction is the most important thing on your list and if happy people pecking away on the keyboard is your top priority then make sure when you begin to assemble your virtualization team and you start scheduling 30 minute meetings, everyone knows from the first day until the project is completed that the objective is “virtualization should be invisible to the user”.

Originally posted 2008-11-11 13:04:10. Republished by Blog Post Promoter

VMware P2V Checklist for Servers

In November I visited the VMware users group for the Phoenix area and one of the big topics was P2V.

Not enough can be said about all the challenges there are to successfully convert physical servers to virtual servers (P2V), especially when your data center has hundreds of servers with various configurations and operating systems.

Sometimes a P2V is done in about 30 minutes, while other times it’s almost impossible because of all the planning due to dependencies like databases, unsupported applications, unsupported operating systems and finally unforeseen gotchas that happen.

Something I got from the VMUG was a P2V check list that Matt Mancini created to make his server migrations easier. I am attaching the file for your use as well as a link where you can visit the VMUG website. <link>

It’s always a best practice to plan your P2V, especially if the server is a production or mission critical server that can’t be powered off for very long.

Kudos to Matt for sharing the P2V checklist file with the group and I look forward to the next VMUG meeting.

Link to download P2V checklist –> Download

Originally posted 2009-01-01 14:41:41. Republished by Blog Post Promoter

VM Install