RAID Classic

Early storage systems were revolutionary, in physically removing storage from the CPU, in enabling sharing of storage between multiple CPUs, and especially in virtualizing disk drives using RAID. When Patterson, Gibson, and Katz proposed the creation of a redundant array of inexpensive disks (RAID) in 1987, they specified five numbered “levels”. Each level had its own features and benefits, but all centered on the idea that a static set of disk drives would be grouped together and presented to higher-level systems as a single drive. Storage devices, as a rule, mapped host data back to these integral disk sets, sometimes sharing a single RAID group among multiple “LUNs”, but never spreading data more broadly. Storage has remained stuck with small sets of drives ever since.

The core insight of the 1980s remains true: More spindles means better performance. Although additional overhead dulls the impact somewhat, the benefit of spreading data across multiple drives can be tremendous. A typical RAID set offers much better performance than the drives alone, and can handle a mechanical failure as a bonus.

Cracks are appearing in the RAID veneer, though. Double drive failures are much more common than one would expect, leading to the development of hot spare drives and dual-parity RAID-6. If four drives perform well, then forty drives perform much better, leading to the common practice of “stacking” one RAID set on others. Caches and specialized processors were introduced to overcome the performance issues related to parity calculation.

But traditional RAID cannot overcome today’s most critical storage issues. As drives have become larger, the tiny chance of an unrecoverable media error compounds, becoming a certainty. Even dual-parity will not be able to guarantee data protection on the massive disks predicted for the near future – statistics cannot be denied. The latest disks contain so much data, without commensurate improvements in throughput, that rebuild times have skyrocketed, resulting in hours or days of reduced data protection.

RAID is also ill-suited to the demands of virtualized systems, where predictable I/O patterns become fragmented. It cannot provide tiered storage or account for changing requirements over time. It cannot take advantage of the latest high-performance solid state storage technology. It cannot be used in cloud architectures, with massive numbers of small devices clustered together. It interferes with power-saving spin-down ideas. Most RAID implementations cannot even grow or shrink with the addition or removal of a disk. In short, traditional RAID cannot do what we now need storage to do.

RAID is Dead

Although most vendors still use the name, nearly every one has abandoned much of the classic RAID technology. EMC’s Symmetrix pioneered the idea of sub-disk RAID, pairing just a portion of each disk with others to reduce the impact of “hot spots”. HP’s AutoRAID added the ability to dynamically move data from one RAID type to another to balance performance. And NetApp paired disk management so closely with their filesystem that they were able to use RAID-4 and the flexibility it brings.

Today, a new generation of devices has even evolved beyond RAID’s concept of coherent disk sets. Compellent, Dell EqualLogic, and others focus on blocks of data, moving portions of a LUN between RAID sets, disk drive types, and even inner or outer tracks based on access patterns. With these devices, a single LUN could encompass data on every drive in the storage array. And the latest clustered arrays can spread data across multiple storage nodes to scale performance and protection.

These innovative devices point the way to a future in which virtual storage is serviced and protected very differently than in the past. Perhaps software like Sun’s ZFS serves to illustrate this future best: It unifies storage as a single pool, intelligently protecting it and presenting flexible storage volumes to the operating system. Although Sun calls its data protection scheme “RAID-Z”, it has little in common with its namesake. Like NetApp’s WAFL, the copy-on-write ZFS filesystem is totally integrated with the layout of data on disk, allowing mobility and efficient use of storage. A single pool can include striping, single- or dual-parity, and mirroring, and disks can be added as needed. Importantly, ZFS also checksums all reads, detecting disk errors.

Long Live RAID

The post-RAID future will see these concepts spread across all enterprise storage devices. Disks will be pooled rather than segregated into RAID sets. Tight integration between layout and data protection will allow for much greater flexibility, integrating tiering and differing data protection strategies in a unified whole. Storage virtualization will allow mobility of data within these future storage arrays, and clustering will enable massive scalability.

Two things will likely remain to remind us of Patterson, Gibson, and Katz, however. First, the core principle that multiple drives working as one yields dividends in terms of performance and data protection. And second, that whatever we use should be called RAID, even though the definition of that term has changed beyond recognition in the last two decades.

Convenience Killed The Cat?

As I’ve stressed before, I feel this convergence should be focused on the physical medium with the most potential like fiber cable rather than twisted pair copper.  Regardless, we must not forget the importance of convenience.  Several markets such as cell phones and even websites forego performance in favor of convenience.  I would like to think that we will continue to stave off that rationale in the IT storage industry, but it does pull a lot of weight.  Convenience is good, but an even bigger factor is the reality that consumers want all of their information right away.  We all need to be a bit more curious, look at the big picture, and question the status quo.

Our Fiber Diet

I could see cabling convenience being huge in the data center.  I was recently rerouting core switch cabling in our data center and fighting to protect the precious fiber cable while moving the huge, bulky and ubiquitous UTP copper.  Though I favor the fiber tremendously, I was looking at our patch panels and contemplating the rapid deployment of our FC SAN.  I started to fumble when I thought about how nice it would be to have fiber patch panels, and ideally it would be used for everything, but I thought about what a risk that was with everything favoring copper these days.  Sure the cost of copper is rising and fiber is dropping.  Indeed fiber has so much more performance potential, can be pulled at higher tension than copper, is smaller than copper, however we cannot overlook the convenience factor which has caused us to restrict fiber from our collective diets.

More Questions Than Answers

Why is it that copper seems to be winning the race?  Is it because more people know how to terminate UTP than polish fiber ends?  Will they be just as good with STP when they have a shield to deal with and even thicker cable?  When will copper become cost-prohibitive and reach the end of its roadmap?

Often I feel the IT industry and the big-vendors out there are focused too much on the road right in front of them without looking at the map first.

We’ve spent time in the trenches with folks in the Enterprise space asking for our advice on whether SATA-based storage is ready for prime time in the datacenter or if it’s still really just a consumer level technology that has no business in mission critical applications.  We always give them a simple, straight-forward answer that to be honest, sounds like a cop-out but in reality
it couldn’t be more true; it all depends on your application. Are you really dealing with mission critical data or can you live with an iota of down time?

The fact of the matter is if you’re working with an enterprise application and cost isn’t the only driving factor but uptime and rebuild times are critical, there’s a really strong case for SAS technology, though SATA is a tempting low-cost alternative to be sure.  In practice though,SATA drives don’t have the same level of status handling and error reporting that SAS drives do.  SAS drives utilize SCSI commands, while SATA drives rely on ATA Smart Monitoring reports. 

The SCSI command set for error reporting and handling is significantly more robust.  If a drive starts
going bad in a RAID volume with SATA, the controller may or may not alert you before the drive goes completely offline.  SAS on the other hand will give you a much more detailed view into what’s going on with a RAID volume, potentially allowing you to get that hot spare in for a rebuild sooner.

For the average consumer, SATA has it all over SAS with its cost per gigbyte ratio that absolutely blows SAS drives out of the water.  However, if you’re a performance freak, you might actually consider a 15K or 10K RPM SAS drive if you’re running a single drive system or something with relatively low
overhead like RAID 0 or 1. The performance benefits can be pretty impressive and there are some much lower cost SAS controller cards hitting the market today from vendors like LSI Logic. 

On the flip side, for you Datacenter types, there’s the new Seagate line of Barracuda ES2.1 SAS drives that are actually high density SATA drive platforms outfitted with a SAS interface.  Here you get the best of both worlds, SATA-like cost structures with hefty 750G and 1TB drives, and
the robust status handling features of the good ol’ SCSI interface in SAS. Well then, that’s a few more options than you can shake a tired, overworked IT Help Desk Engineer at, isn’t it?

“The FC incumbent has a huge advantage in being the FCoE vendor of choice. Like today’s FC networks, we do not expect mixed vendor FCoE-FC networks.”

Storage is a serious business, and everyone is very concerned about compatibility and interoperability. So much so, that they will buy more of the vendor that they already have and ignore the interoperability. Brocade knows that and is planning for it. Now, to me, that sounds like an opportunity to charge more, and make more profit, for very little effort.

Those storage folks must love being bunnies.

A deeply cynical observer would conclude that Brocade feels that FCoE locks customers in to their product and can’t wait to get their hands on the money. In fact, they believe it enough to suggest it to their investors.

How is this going to work ?

The “operability lock in” means they charge extra.  Because you aren’t going to mix your “standardised” FCoE and FC vendors are you ? Why ? Because support would be a nightmare! Its not a truly open market, competition is low so Brocade can have a go at gouging you.

Bingo, Brocade is profit assured, good for bonuses and the share price!

What is the bet that Brocade can make sure of this by promoting and marketing compatibility and interoperability ? (Cisco is already making a lot of loud noises about their upcoming interop sessions). Let me also ask how many IT people are really going to deploy multiple vendors ? Anyone, anyone ? Bueller ?

Give it some thought people, when you take FCoE onboard you just might getting stitched up by some clever marketing.

You can download the Brocade Tech Day briefing http://media.corporate-ir.net/media_files/irol/90/90440/TechDayJune2008.pdf and read it for yourself. It is quite educational.

At least that’s the attitude that most companies will have when they’re asked if they will implement a new-fangled technology for their existing SANs.  Most firms have their critical systems on a SAN, and unless  there is a real need to, will not forklift upgrade their existing SANs.  That leaves the new technologies to attract  new clients, such as small business and small-to-medium enterprises (SME).  iSCSI is catching their eye now, and sure maybe FCoE will catch it later.  The key is still having devices that more than one protocol and interface, such as  those that use FC and iSCSI in order to attract the new but retain customers with existing investments.  Later perhaps, it will shift to FCoE and iSCSI, possibly Infiniband.  As a sidenote, I see the most bandwidth potential with fiber and Infiniband rather than copper.  Plus the cost of fiber continues to go down while copper costs are going up.  Fiber is smaller, but then again so was “Betamax”.

Evolution vs. Revolution
No one should expect the storage industry to make a drastic change unless a  start-up company comes along with revolutionary technology that really takes off and others take notice.  I believe that is unlikely since storage networks also rely on network technology which isn’t going to change overnight just for storage.  Afterall we’re moving back towards commonly used network technologies like IP and Ethernet for our Storage Networks!

I also feel it’s not accurate to think the storage industry is not improving daily.  We are not merely recreating what we started with, we are improving its reliability, performance, resiliance, flexibility, and utilization.  We are not mimicking evolution, there is no such thing.  To ignore all of the progress in hopes of a revolutionary shift seems to me as useful as questioning why we still use processors.  Granted quantum computing would be great, but how do you write code for something you don’t understand, let alone something that changes only once it’s observed?  Do you want your storage to be defragmented only when you observe it?

Thoughts on recent topics

Cloud computing will not take off in the financial industry or many other conservative industries.  They simply will  not rely on it to get critical data or services.  If your Internet is gone, it’s ALL GONE.  They just won’t tolerate that.  I’ve seen the Internet go down in all 3 of the biggest cities for an extended period, and so did a lot of people that will never be convinced to invest too much in “cloud computing”.

Deduplication is not simply compression.  It is inherently different.  It does so much more than just looking for repeated strings or bits and replacing them with a shortcut for “1000 spaces” for example.  It doesn’t end with each file or with each archive or backup.  Deduplication looks at every single block of data and determines if it has ever seen that block before, and if so replaces it with a reference to the original block.  After that you could even compress the deduped data.

As far as synchronization of files at home goes, a cloud with silver lining is called NAS with sync features enabled.  Have everything do a 2-way sync with the giant NAS.  If simplicity is desired, having 1 PC per person and a one-way upload from a “control device” such as an iPOD would be the way to go.

We Are The Industry
And for those of you who haven’t noticed — we ARE the industry, the big guys are asking us experts what we see coming around the corner, and they want to invest in developing and marketing that product.  No company wants to make the next “Betamax”, despite it being a better product than the others.  Sites like this are catching vendors’ attention and will change their minds.  Remember first comes expert opinions and recommendations, then vendor investment and production, and last comes recognition from customers.

Virtualization Changes Everything

I keep hearing this, and to some extent, it’s true, but on the other hand, if you have been around storage you also see a lot of similarities to the issues that we always had to deal with as storage admins/managers. The main difference with VMWare is that you might find that it’s even more important that you address the following three issues:

1. Application performance is dependent on storage performance. This isn’t news for most storage administrators. However, what’s different is that since VMWare can combine a number of different workloads all talking through the same HBA(s), the result is that the workload, as seen by the storage array, turns into a highly random usually small block I/O workload. These kinds of workloads are typically very sensitive to latency much more than they require a great deal of bandwidth. Therefore the storage design in a VMWare environment needs to be able to provide for this type of workload across multiple servers. Again, something that storage administrators have done in the past for Exchange servers, for example, but on a much larger scale.
2. End to end visibility from VM to physical disk is very difficult to obtain for storage admins with current SRM software tools. These tools were typically designed with the assumption that there was a one-to-one correspondence between a server and the application that ran on that server. Obviously this isn’t the case with VMWare, so reporting for things like chargeback becomes a challenge. This also effects troubleshooting and change management as well since the clear lines of demarcation between server administration and storage administration are now blurred by things like VMFS, VMotion, etc.
3. Storage utilization can be significantly decreased. This is due to a couple of factors, the first of which is that VMWare requires more storage overhead to hold all of the memory, etc. so that it can perform things like VMotion. The second reason that VMWare uses more storage is that VMWare admins tend to want very large LUNs assigned to them to hold their VMFS file systems and to have a pool of storage that they can use to rapidly deploy a new VM. This means that there is a large pool of unused storage sitting around on the VMWare servers waiting to be allocated to a new VM. Finally, there is a ton of redundancy in the VMs. Think about how many copies of Windows are sitting around in all those VMs. This isn’t new, but VMware sure shows it to be an issue.

So, the question is, what can we do about these things? Well, some new technology that’s just coming into the market will help. things like thin provisioning, block storage virtualization, new SRM tools that can do correllization between servers, networks, and storage will help. Another thing that we are storating to see are virtual HBAs which will help as well on the reporting end of things.

So, as I see it, server virtualization (VMWare) will drive storage virtualization. I would even go so far to say that you won’t be able to get the full promise of server virtualization unless you impliment some storage virtualization. If I’m right about that, then we should see a significant uptick in the sales of virtualization very soon…

The data will be collected on a quarterly basis for one year, through June 2009. The plan is for Energy Star to analyze the data and then launch the new rating system in January 2010. (It is unclear if the one-month extension to garner additional participants will impact the January 2010 delivery date.) Data centers with a rating of at least 75 will be offered the Energy Star label.

Channels, Blocks, and Files

Most innovation in the 1980s and early 1990s focused on moving storage out of the server. SCSI allowed disk to exist in a separate cabinet, RAID allowed multiple physical disks to become a virtual one, and these were mixed to become the prototype storage array. Although SCSI allowed one-to-many connectivity, it was never a true peer-to-peer network, even once it was mixed with network concepts in the form of Fibre Channel.

Even today, SAN storage is focused on providing faster, more flexible, and feature-packed direct-attached storage. A modern virtual SAN hides a complex arrangement of caching, data protection, tiered storage, replication, and deduplication, masquerading the lot as a simple, lowly disk drive. It is sad but true that all of our work as an industry has been dedicated to recreating what we started with.

Networked file-based storage is no better. Although NAS devices have all the advanced features of their SAN cousins, they must present a simple file tree to the host to retain compatibility. File virtualization merely presents a larger homogenous tree.

Inside the server, too, features and complexity are hidden to retain a familiar file system format. Volume managers can do anything a virtualization device can, but must present their output as a simple (though virtual) disk drive. File systems, too, have added features but still present a familiar tree of mount points, inodes, and files. Even ZFS, possibly the most advanced combination of volume management and file system technology yet, must present a simple tree of storage to applications.

The Metadata Roadblock

This outdated paradigm, of disks and file trees, is ill-suited to today’s storage challenges. Data must be categorized so actions can be taken to preserve or destroy it based on policies. Data must be searchable so users and applications can find what they want. Data must be flexible so it can be used in new ways. Our antiquated notions are not capable of meeting these challenges.

One simple problem is that we lack context for our data. Most file systems merely assign to a file a name, location, owner, and security attributes. The most advanced can contain extended metadata, but this is rarely seen in practice since many applications cannot agree on how to use this data. Microsoft’s Office suite can store and share extended file attributes, for example, but these live inside the file rather than in the file system. The promise of expanded Office attributes is only realized in conjunction with a content management system like SharePoint which lies above the lowly file system.

What if the storage system could keep this data instead? What if it could logically group files according to project or client, mining keywords and authors, and maintaining revisions? These concepts are not new, having been implemented in content management systems for years, and certain elements appeared in file systems, like Apple’s HFS and VMS’ Files-11, for decades.

Cut Down the Tree

File metadata would allow advanced features, but truly taking advantage of them requires a more fundamental shift in the way applications access files. Rather than sticking to a traditional hierarchy of directories in a tree (which was, after all, simply a primitive metadata system), we should remove the tree altogether. Allow files to become data objects, identified by arbitrary attributes and managed according to an overarching policy.

This future vision is decidedly different from our current notion of storage, but is not so far off. Many organizations now rely on central data warehouses based on SQL-language relational databases. As many storage managers have grumbled, databases tend to ignore storage management concepts entirely, managing their own content independently.

But not all applications need a database back-end, so another initiative seeks to provide generic object storage for wider use. Called content-addressable storage or CAS, these devices have traditionally been used only for archival purposes, since that was their first market application. As vendors break free of proprietary interfaces in favor of open ones like XAM, CAS could transform storage itself by eliminating both file and block storage at once.

Similar concepts are already at work in the so-called Web 2.0 world. Non-traditional databases like Google BigTable, Amazon S3, and Hadoop allow massive scalability for object storage. API-sharing initiatives with many Web 2.0 companies can be seen as similar prototypical object storage frameworks. Any of these could be leveraged to provide a new world of data storage, and many are gaining traction even now.

Although traditional block storage is here to stay for disk drives, and tree-type file systems are likely to remain the foundation of operating system storage, new object-based concepts could change the world in fundamental ways. As applications become “web aware”, they also become object aware, increasing the likelihood of such a storage revolution. For the majority of applications, this new world would be a welcome one indeed.

Cloud Computing High-Level Diagram, Courtesy: Wikipedia

The term “cloud” was originally coined by networking technologist to explain a group of resources connected together as one.  These days, coupled with the word “computing”, the cloud takes on a whole new meaning.  The concept is quite simple actually but it’s the forces at play that leave us all speculating how the landscape will take shape.  As the internet’s capacity has scaled and enabled powerful new functionalities, real-time virtual resource access and billing/metering of those resources, the Cloud Computing model gained serious traction.  The market has evolved from offering not only enterprise and educational institutions, services like Amazon’s EC2, but also services like Google Apps are being offered all the way down to the average end user.

At a high level, there are a few usage models of Cloud Computing that have begun to take shape.  SaaS or Software as a Service, is what is generally thought of when you consider services like Google Apps or Salesforce.com.  Let’s face it, from the end user and SOHO crowd, to big businesses that are making use of services like Salesforce, you can see the huge value in outsourcing certain applications, not only from a cost standpoint but for ease of use and collaborative efforts.  Then there’s Utility Computing, another variation on the Cloud concept, that offers raw virtual server resources to corporate entities, IT departments, and even educational institutions.  Need more capacity or throughput from your Data Center?  Just provision more resources with a place like Liquid Computing and kick back for a coffee break.  Not to mention, you’ll probably sleep better at night because someone else has built the infrastructure for you already that will cover your assets, so to speak, should some sort of catastrophic hardware or other failure take place.

In the final analysis, there’s no question that Cloud Computing, Grid Computing, Utility Computing or whatever else you’d like to call it, is definitely the wave of the future for many applications and usage models.  Granted, the average power user or enthusiast will likely still have a powerful desktop or notebook system for many years to come.  However there’s something taking shape on the horizon that looks a little like Web 2.0 for computing hardware and compute resources — a shared, collaborative and leased commodity, always accessible and shaped by the customer. It’s a compelling business case to be sure with lots of competition filing into the market place.   The questions that remain are many and various.  What will the dominant player be for the various service types?  What will the killer apps look like?  And probably more importantly, will the transport providers supporting the internet backbone — the Ciscos of the world — be able to scale quickly enough with it?  Some rather smart folks better have a few precision crystal balls at work for these questions because it’s coming fast and furious but there are many of us that can’t quite make out what it looks like just yet.

Here’s a brief review of four popular integrated solutions for high performance video editing as well as other large file sharing applications such as broadcast media, film, and medical imagery:

Global SAN: http://www.studionetworksolutions.com/products/product.php?pci=10

Workgroup ability to edit audio/video projects and large files directly over RAID-protected storage.  High-throughput iSCSI/IP SAN.    Advantage:  Avoiding the expense of Fibre Channel storage, cards and switches.

Avid Unity: http://www.avid.com/products/unitymedianetwork/

AVID’s experience in Video Editing is reflected in their high number of high end global installs.    AVID Unity’s advantages  are support for collaborative HD workflow, huge storage capacity of up to 40 TB, and support for both Windows and  Macintosh.   Another Avid SAN advantage is this active online forum community:
http://community.avid.com/forums/p/13609/76665.aspx

Apple Xsan 2: http://www.apple.com/xsan/
At $999 Apples’s Xsan 2 offers low cost and high performance. Key features of Xsan2 for the Macintosh are
Powerful collaboration tools and pooling of storage resources to increase capacity.   Also, Apple’s Spotlight search feature allows users to search *within* video files for content relevant to their query.  This type of video search will be increasingly important to SAN environments as the number of video files skyrockets.

Metasan: http://www.rorke.com/av/metasan.cfm
metaSAN is designed for heavy bandwidth workgroup environments such as those found with film editing and healthcare applications.   Simultaneously access of groups of files such as video clips, satellite imagery, medical data, and CAD files.