Let’s Talk Dirty Shutdowns (NetApp ONTAP Update Problems!)

Are Dirty Shutdowns causing your NetApp ONTAP Update to Loop?

In this post I’ll share what I learned recently about NetApp ONTAP v8.x. Let’s talk “Dirty-Shutdowns!”

Beginner Alert:

Mark this page because some day you might need it. This is a must know for newbies managing NetApp storage.

What are Dirty-Shutdowns?

A dirty-shutdown can happen for a number of reasons, for example:

  • A software bug has caused a filer to fail-over and trigger a loop. The “dirty-shutdown” is now keeping you from updating ONTAP to fix the bug.
  • Or a loop was caused by a forced take-over during a normal ONTAP update. The “dirty-shutdown” is now keeping you from completing the normal ONTAP update.

Both cases put the filer head into dirty-shutdown mode and force ONTAP to continue booting to the primary partition.

Here’s the Scenario:

You’ve decided to update your NetApp software to a new version of ONTAP v8.x and while going through the documented process of clicks, take-overs, and reboots something goes terribly wrong and now your filer head is looping back to the old firmware.

After reviewing your steps and freaking out because this is a production system and now you are stuck failed-over on one filer, you call technical support.

From here you go through the rigmarole of redoing all the steps you just did, again.

Then after about an hour you get transferred again and now you rehash all the steps again with someone else. But this time they decide they want you to pull down the update on a web-server so you can do a NETBOOT which installs the upgrade over the primary OS partition.

From here things really get hairy because you don’t have a web-server available to do a NETBOOT and time is ticking away. Now it’s 3AM and your change window is quickly running out.

Jumping ahead…you finally set up a web-server, pull down the file, run the update and reboot. Success. You are back up on the new version. Then you do the take over and rerun the update on the second filer.

But this should have been non-disruptive and quick but instead it took 3 – 4 hours and a call to NetApp support.

Why did this happen?

Here’s the Explanation:

I’ve had to endure this problem on production systems in the past and what I found out is this. The new version of ONTAP, v8.x, now runs on BSD. There is a trigger now that wasn’t there in v7.x that detects what is known as a dirty-shutdown which forces ONTAP to boot to the primary flash partition.

Here’s the Solution:

When your filer is looping after a dirty-shutdown you can try this option before doing the NETBOOT.

Run this command with the environment variable set to LOADER.

For example:  “LOADER-B >boot_backup” to boot to the secondary partition or “boot_primary” for the primary partition.

If this works you will bypass the primary and boot to secondary partition where the updated OS should be sitting.

If this fails then the NETBOOT is the next option.

Note: Always call your technical support when you are not sure.

Wrap-up

Storage is a big part of virtualization and knowing how to resolve problems fast is important for managing your virtual service.

Remember, no matter how much you spend on hardware and software — outages will still happen, guaranteed!

To be indispensable you need to learn how to get your services back up with the least amount of disruption.

I hope this “Dirty-Talk” has helped you?

Leave a Reply

vSphere Data Protection EOA
3 Alternative vSphere Backup Replacements For VDP

Important Notification: vSphere Data Protection (VDP) End of Availability (EOA) That...

vSphere 6
8 Updates That Make vSphere 6 Better – Keith Barker

Editor’s note: Keith Barker has been a CBT Nuggets Trainer since 2012. Some...

VMware Interview Questions
25 VMware Interview Questions And Answers: Tough & Technical (Download PDF)

Free VMware vSphere Interview Guide In this VMinstall Guide I’ll share a...

Project Photon
VMware Project Photon: Technical Review for Linux Admins

My quick and dirty review of Project Photon. I was very happy...

DevOps Plan
Best DevOps Strategy Hack (Winners Start With Why)

What is DevOps? It’s a model. It’s a culture. It’s a...

DevOps Tools Download
DevOps Toolchain: Download 3 Free Tools Used By Pros (Ansible, Git, Jenkins)

Hack Your Resume with DevOps Skills Lately I’ve been getting a...

DevOps Guide
Microsoft DevOps: 25+ Powerful Windows SysAdmin Skills (NEW Guide!)

What is Microsoft DevOps? This is an interesting question because first off,...

continuous integration tools
DevOps Training Courses That Will Give You The Edge (Perfect For Beginners Who Want Skills That Are Noticed By Google & Facebook…)

When I was just a fledgling starting out as a Jr...

Click here to learn how to Optimize WordPress Speed
WordPress Speed Optimization: Part 4 – VPS Case Study To Make VMinstall.com Rank Higher In Google Through Faster Hosting & High Availability @ Lower Cost

This is going to be a quick and dirty part 4...

Best Hosting for WordPress
Case Study: Best VPS For Easy WordPress Migration

Why I moved my WordPress blog off slow shared hosting… Let’s...

cloudways review updated
11 Cloudways Review Pros & Cons From My Deep Dive Testing (VPS Series: Part 2)

Which is a better cPanel alternative of a VPS – Cloudways...

ServerPilot Review
ServerPilot Review: Installing WordPress On DigitalOcean (VPS Series: Part 1)

How to install WordPress on a cloud server like a Pro...

DesTechAZ
What’s So Interesting About Woz U?

On 10/12/2017, I had the privilege of attending the AZ Tech...

tensorflow training
TensorFlow Tools for Beginners (7 Easy Takeaways To Get You Started)

Introduction To Deep Learning As a follow-up to my book review...

job rut
How To Get MOVING When You’re Stuck In A Career Rut (Video)

You worked hard to get your degree and after graduation you...

Storage Engineer Skills
Why Storage Engineer Skills Are HOT! (Can You Say Big Data?)

3 BIG Skills New Storage Engineers Are Missing Out On… Behind...

Open Source Storage Software
3 Open Source Storage Software Alternatives For Beginners

Build Your Own Storage Solution! Reflecting on my trip to VMworld...

VMware VCP FOR Hire eBook
Read Excerpts from VCP for Hire eBook

BIG Thanks… Before I share the excerpt of VCP for Hire,...

How to Self Publish Your VMware eBook KDP
How to Self Publish Your VMware eBook KDP

Do you have a VMware book you’d like to self publish...

free chef for 5 servers
Free Hosted Opscode to Try Chef On 5 Servers Before You Buy

Is Chef the configuration management and automation tool for your operations team?...