Behind every successful business is data, lots of data. And behind all that data are lots and lots of gigabytes of storage.
Guess who’s behind the technical scene making sure all the data crunching is going on without any interruptions – THAT’S RIGHT – storage engineers (A.K.A. SAN Engineer)!
In this lesson we’ll go over 3 critical jobs many storage engineers do behind the scene that most CEOs don’t know are happening. We’ll break each job down into skillsets and as a bonus I’ll share related tough storage engineer interview questions you can use…
First off, let me be fair to hard-working storage engineers across the world because there are a lot of jobs they handle daily that nobody else can do. And this is why these 3 groups of skills are perfect for this lesson.
- Designing, Building & Supporting Storage Solutions (Big and Small)
- Data Storage Capacity Planning (Big and Small)
- Monitoring Data Storage Systems (Big and Small)
Alright, are you ready to dive into the life of a SAN engineer?
Designing, Building & Supporting Data Storage Solutions
Whether a business is operating on spinning disks, solid state (SSD), or flash, someone has to design and build the storage infrastructure (and yes, let’s not forget storage provisioned in the cloud).
Every server from the smallest VM or Docker container hosting a web application all the way up to the largest database needs storage. And without someone with the right skills managing the hardware and software properly, you can bet there are painful problems with poor performance, data loss, and crashes. Ouch, right?
Don’t forget to take notes or save this page as a shortcut so you can come back over and over until this lesson sinks in!
Now is the time to take action if you’re thinking about becoming a SAN engineer because this urgent skillset is only going to grow as Big Data grows. Especially with more businesses starting to use deep learning frameworks such as Tensor Tools and Keras to develop A.I. software.
And this urgent need brings us to the most important point of this lesson, which is “RIGHT NOW” businesses are looking for highly skilled and experience storage engineers who can dynamically adjust and tweak gigabytes, luns, aggregates, and flash pools around with surgical precision without impacting business services.
And get this. They will pay a premium to hire the best SAN engineers!
You see, a quality storage solution is not just about buying the newest storage technology to replace the old oversubscribed, poorly designed, and abused storage system; it’s about understanding the need and asking the right questions before meeting with any storage vendor.
Why have a meeting if nobody knows what storage problems or challenges need to be solved?
This unfortunately is where many businesses go wrong. The CTO normally meets with vendors at a conference, agrees to buy something shiny, and then the storage engineer tries to make it fit.
And this is why many of the storage infrastructures I’ve seen over the years are just patched together storage solutions from multiple vendors – a bit of EMC, some NetApp, and now Pure because flash is the buzz that solves all the bad practices from the past. Also, if you’re a VMware shop let’s not forget VSAN because someone on the shop floor is already building a VSAN whether it’s approved or not.
How Google handles storage…
Look, I could be way off but here’s what I look for when I interview storage engineers.
The skillset of a solid storage engineer is someone who can adapt to the environment both via their skills and their mindset.
But the problem I’ve had quite often filling this criteria is because storage engineers, like network engineers, are religious. Here’s what I mean. Many storage engineers have become overly zealous about brands such as Pure or Cisco.
So unless the plan for your shop is to convert every gigabyte of storage to Pure, or every network switch to Cisco, then I avoid hiring brand zealots.
On VMinstall, I always tell it like it is and teach my readers to keep an open mind – learn to be diverse (agnostic). I love VMware products but I also suggest you learn about OpenStack, XenServer, and Docker, too.
Being objective and having an open mind allows for more career opportunities, as well as possible involvement in cool or challenging projects that only include people who can think outside the box. (Nothing against Pure or Cisco, just making a point! The same could be said about EMC or HP.)
I know this is touchy ground but it needs to be said out loud! Don’t be religious about brands or technologies because they can change 6 months from now when your CIO or CTO goes to their next Tech conference.
Yes, you read that right…
It’s not uncommon to see a storage engineer job posted on Indeed with skill requirements for 5 – 6 storage brands. Click here to take a look for yourself.
So having a broad data storage skillset is important for your career success. But you also need to be able to think objectively and constructively about what you have to work with because sometimes you will have to turn lemons into lemonade.
Think about this example.
What if you’re hired by a big brand, exciting right?
But on your first day you find out they’re recovering from a major storage disaster that has been going on for several weeks.
Then you learn you have just replaced the guy or gal who quit or was fired over the crash – now what?
Think I’m exaggerating? I’m not.
Learn to design, build and support all types and brands of storage solutions. And don’t get hung-up on one flavor of storage unless you plan to become a SME for a brand – but even then knowing the good and bad about the competition is important for a SME.
Good SAN engineers in my book focus on great services, even in the worst storage environments. And believe me when I say there are some incredible storage messes behind many of the hottest brands.
The secret to designing a solid storage solution is to ask relevant questions:
- What will the application be used for?
- How many users will the application host?
- Who are the users of the application?
- How fast will the data grow?
- Is there any special requirement for PII data?
- Are there any special security requirements?
- What is the expected SLA of the application?
- Is there a budget for storage?
By defining a scope up front, you’ll “hopefully” avoid problems and rework later.
If there are any red flags or if the existing storage infrastructure cannot meet the requirements of the project, then raise your concerns before moving forward. Always try to avoid designing and building a data storage solution that will not work long-term. All too often short-term solutions become long-term problems that storage engineers have to babysit.
Got it? Now let’s move on to the next topic…
Data Storage Capacity Planning
Every storage Rock Star I’ve worked with tracks capacity trends for daily, weekly and/or monthly usage.
Outages at Amazon don’t happen because of a storage crashes, especially when SLAs are measured in millions of dollars per second. Which is why storage capacity planning is an important skill when designing scalable storage solutions for big eCommerce databases.
But you know something? Even small business expects the same careful planning.
Do you agree data is important? OK, then if data is so important why do IT shops operate on the premise where DBAs send an email to storage admins with a request for “x” GBs of storage? And a few minutes later the response email says, “Done”.
Then 3 months down the road all hell is breaking loose because the storage admin did not ask any questions about what the storage was going to be used for and now a new web service that the company’s sales and marketing team have been planning a grand opening for months is showing a 500 server unavailable error because the database can’t keep up with all the hits the marketing campaign has generated.
Am I getting through to you?
These are real experiences I am sharing.
The world needs storage engineers who can think ahead and who are not just brand evangelists. Technical thinkers who can also work with other teams to unravel chaos.
Capacity planning is a key skillset for a storage engineer and anyone desiring to become a storage engineer should spend a lot of time testing and developing skills with tools and scripts that can help generate useful trend reports.
Once a storage capacity problem is happening it’s too late to undo poor planning…
Monitoring Data Storage Systems
I saved the best for last.
All joking aside, storage monitoring has to be the most difficult monitoring of all infrastructure technologies.
Why? Because most storage manufactures have their own storage tools, which are not cheap so often storage engineers are fending for themselves with freeware, trialware or homemade scripts.
Albeit storage vendors have gotten better over the years adding tools and dashboards, however there are still gaps between brands that a good storage engineer will need to plug. Which is why, once again, I suggest developing the skills now for monitoring capacity and performance on big and small storage systems.
I know monitoring storage sounds easy but I’ve seen storage systems fail unexpectedly and take weeks to restore all services back to normal.
Remember, at any time and without any warning, a drive, controller, or SFP can fail. Or firmware can get buggy, heck any number of hardware or software issues can happen that cause storage across multiple services to flip out.
Monitoring may not stop a failure but it’s better to know a drive has failed so it can be replaced before a second and third drive fails and you only have 2 spares. Or getting an alert that an SFP has failed, before the redundant path fails, too.
Sorry if this is not a step by step list of skills a good storage engineer should have, but if you are able to see the big picture I have just painted for you, and if you’re able to wrap your head around why storage engineer are important, then you get the value this lesson is designed to create.
Just like every other lesson I have written about on VMinstall, my goal is to forge a “Think Service First” mindset…
Does this sound like you?
Tough Data Storage Interview Questions (BONUS)
In this lesson we covered 3 important skillsets that data storage engineers should cultivate. Now, let me share some insanely practical storage engineer interview questions that managers can use during your next interview (screening) or interviewees can use to prepare for your next storage job interview.
- As a SAN engineer, provide some examples how you manage storage capacity?
- What are the key performance metrics you track that show how the storage is performing? Explain why?
- How would you troubleshoot reports from a DBA that storage is causing poor database performance? Step by step, what would you do?
- What would be the storage performance thresholds you would use for alerts? Explain the why?
- Share an example with details of a time when you worked with a DBA to design a storage solution for a tier 1 application? Step by step what did you do?
- Share an example with details of a time when you worked with a virtualization team to design a solution for presenting storage to vSphere or another virtualization platform? Step by step what did you do?
- Describe a time when you had to solve a difficult storage problem caused by improperly configured SAN hardware or software?
- Describe a time when you had to recover from a major storage outage, and what you did you do to resolve the issue?
- This is a 3 part question. How would you diagnose a storage problem where the storage keeps disconnecting – give an example for iSCSI, NFS and Fiber channel (FC) attached storage? Give as much detail as possible about what you would do for each scenario.
- Out of all the storage solutions on the market which do you prefer to use and why?
- Describe how you would design a storage solutions for big data, virtualization, and file storage? High-level, step by step how would each be step up?
- There’s a lot of ways this can be done but how do you prefer to scale (grow) storage pools for databases, virtualization, and file shares? Explain why your way is better.
- How would you handle working with another storage engineer who you didn’t agree with you on storage related work such as: design, capacity planning, and monitoring?
- What are your thoughts about deduplication and thin-provisioning? Give examples for your thoughts.
- Explain how/what you do to keep your storage skills sharp and up-to-date?
The SAN Engineer Interview questions listed above are taken from years of related real-life examples working with storage engineers. Feel free to add them to your more detailed technical questions related to your environments and processes.
Let’s Wrap Up…
There’s a lot to be said about storage engineers we didn’t cover and even more about all the different data storage technologies now available. But that said, I can’t stress enough how BIG the 3 skillsets we covered are for your success.
- Learn to design, build and support all types of storage. Don’t get hung-up on one brand.
- Develop skills and learn techniques for capacity management (Writing your own scripts).
- Test and evaluate all storage monitoring solutions until you find the right one for your use case.
Never let a disk fill up and crash a system! (Make sure you emphasize this point in your storage job interview)
Still want more? Read about storage strategies.