There is a dark cloud looming in storage. Over the last decade,
conventional storage platforms have been able to keep up with the
demand for ever higher capacity systems at a lower cost per GB,
however, the real specter on the horizon is severe and inevitable
performance degradation. This degradation of performance is
critical because most organizations rely on a scalable facility for
servicing I/O to rapidly deliver information to aid in revenue
generation. Indeed, resolving the storage I/O performance
bottleneck becomes even more critical in a soft economy, when
profits are the most elusive.
What's Causing the Storage I/O Bottleneck?
Multi-tenant workloads are at the heart of the problem.
Multi-tenant workloads are also known as concurrent/aggregate
workloads in which data is shared between multiple users or
applications and accessed concurrently by multiple users of the
same shared storage resource. These shared workloads
and resources are no longer relegated to a few isolated companies
whose performance demands are on the fringe of mainstream data
center environments. In fact, any organization that is deploying a
server virtualization project has by definition a multi-tenant
workload demand.
There are three basic elements of performance in a data center; the
processing power harnessed by servers, the network harnessed by
switches and routers, and the storage which consists of the disks
harnessed by SAN and NAS controllers also referred to as "NAS
Heads". Each of these elements are
under constant strain to keep up with the digital demands of their
users. Servers and networks have kept pace through added
performance and intelligently utilizing that performance, but
storage has not kept pace and has become the bottleneck of the
enterprise. Now this storage bottleneck has moved beyond being an
IT problem and has created a perilous situation for the
organization as a whole.
Of the two elements that have kept pace with the growing digital
demand, compute capability has kept pace via increased performance
and increased core density, as well as increased intelligence
through server virtualization and scale-out clustering or grid
infrastructures. Networks similarly have kept pace with increased
bandwidth capacity and intelligent use of that capacity through
QoS, prioritization, and efficient use of wide area
connectivity.
Meanwhile, storage performance has not kept pace. Instead it has
remained frozen in the same architectural design for at least a
decade; a high performance SAN or NAS controller pair that drives
an increasing number of disks. While increasing the number of
drives can improve performance, there is a limit to the number of
drives these controller pairs can support as well as a limit to the
amount of inbound traffic they can sustain. This controller (SAN)
or head (NAS) is now the primary bottleneck, limiting improved
storage performance.
Storage I/O vs. Multi-tenant Workloads
To
compound this problem, the workload is now changing. Workloads are
now multi-tenant with multiple shared servers and networks trying
to access storage in this out-dated model. Prior to multi-tenant
workloads, a single application coming from a single server could
only create a limited number of
requests. Multi-tenant workloads, running either through multiple
virtual machines on a single physical server or through a single
application scaled across many physical servers in a cluster or
grid, can now generate hundreds if not thousands of requests for
storage I/O.
The impact is that these requests saturate the storage controller
(or head) and the applications or servers have to wait for it to
catch up, which in turn delays processing, eventually costing
organizations money and limiting productivity.
A multi-tenant workload is one that typically has multiple owners
or users at any given point in time. The presence of these
multi-tenant workloads is increasing in quantity and in capacity.
They are no longer uniquely restricted to a limited number of
enterprises but are in fact very common in some form in almost
every enterprise today. Many enterprises now have multiple sources
of these workloads.
At a minimum any organization implementing server virtualization
today can have multi-tenant workloads; in some cases 20 or 30
virtual servers coming from a single physical server. NAS storage
systems have become one of the preferred methods for delivering
storage services to the virtual hosts and the access patterns of
the virtual machines are inherently random. Storage performance
scaling in virtual environments becomes critical as one or more
virtual machines begin to consume all the available storage I/O
resources which then adversely affects performance across all the
other virtual machines on that host, creating a domino effect of
lowered performance and lowered confidence in the virtualization
project.
Beyond the very common virtual server use case, there is also a
rise in the more traditional case of multi-tenant workloads;
multiple processing servers grinding through a job. These workloads
are not limited to the common example of simulation jobs similar to
those found in chip designs or processing SEG-Y data in the energy
sector. There are many others; DNA sequencing in bioinformatics,
engine and propulsion testing in manufacturing, surveillance image
processing in government, high-definition video in Media and
Entertainment, and many Web 2.0-driven projects.
Storage I/O performance is critical in these environments because
work essentially stops while the processing or simulation job
completes. When these jobs stop, so typically does the
organization’s ability to create revenue. To get around these
delays timing of job runs becomes critical to minimize user impact
but even with the best planning possible, user productivity will
suffer. When that productivity suffers, so does organizational
profitability.
Another compounding factor is that all of these data sets have
increased in complexity in recent years, becoming more granular,
shifting to three dimensions, or significantly increasing color
depth. This granularity not only increases the physical size
required to store this data but also the processing and storage I/O
required to create, modify, analyze or test the data.
In all cases reliable, predictable, scalable storage I/O
performance is critical. For example, an integrated circuit
designer may need to run a simulation on a particular chip design.
As with other environments, this data set is becoming significantly
more complex and detailed. In the case of chip design, the chips
become smaller or the number of functions on the chip increase.
There is a tremendous need in these environments for synthesis and
regression testing. As a result the time required to process a
simulation of the chip takes longer and longer. It is not uncommon
for this type of job to take from three days to an entire month to
run. There are two bottlenecks in this process; the time required
for the CPU to process the data and the time for the storage to
read and write the simulation scenarios.
In the virtual server example, the VMs are almost purely random by
nature. While the virtual machines don't have to wait for a
particular VM to finish its task, if one VM becomes busy it can
dramatically impact performance of the other systems. As in the
case of simulation-type workloads, the storage I/O pattern on these
systems is as large as it is random.
The Storage I/O Bottleneck
While it is
necessary to address all of the performance bottlenecks, computer,
network and storage, often the most challenging for these
environments is handling the storage bottlenecks. The compute
bottlenecks are well understood and can be dealt with by allocating
a higher quantity of faster processors
through techniques like clustered and grid computing or simply
leveraging Moore's law:
In 1965, Intel co-founder Gordon Moore noticed the number of
transistors per square inch on integrated circuits had doubled
every year since their invention. Moore predicted the trend would
continue for the foreseeable future. Since then the power of
microprocessor technology doubles and its costs of production fall
in half every 18 months.
In similar fashion, networking has increased bandwidth via
techniques like trunking or multi-homing. These techniques will
adequately handle the compute and network element of the bottleneck
and are also well understood.
Storage on the other hand has not benefited by a similar Moore's
law. While capacities have continued to increase, speed of the disk
system has not. This leads to the use of more disk per controller
or head and results in the storage I/O bottleneck.
What is lacking from most storage manufacturers is a similar
scale-out model, as the current dual controller systems quickly
become saturated by these workloads, especially many NAS-based
systems.
Because of the shared nature of these systems, Network Attached
Storage (NAS) should be an ideal storage platform for multi-tenant
workloads. Unfortunately because of these workloads’ highly
random data access patterns and very high number of storage I/O
requests, either from a single server with multiple requests in the
virtual server example, or a single application making requests
from multiple physical servers, single or even clustered, NAS heads
and ports can quickly become a severe bottleneck.
The result is that many organizations turn to a shared SAN, which
is not by its nature shared, nor is it as easy to manage as a
single NAS file system. It too will still lead to bottlenecked
storage performance, which again not only slows the business down,
limits employee productivity, and eventually loses the organization
money, but also adds greater complexity to an already complex
environment.
In either the SAN or NAS case, job runs tend to produce a
significant number of sequential and random writes requiring an
equally large amount of very random reads. This is a deadly
combination that renders most cache on these storage systems
useless because they are too small to have a high degree of cache
hits. The result is that in addition to these bottlenecks most if
not all requests have to come from the drive mechanism, not the
cache that supports it, further lowering performance.
Solving the Storage I/O Problem
As these
workloads become more prevalent across the enterprise, the ideal
solution is to solve the NAS bottleneck and establish an easy to
manage, high-performance NAS infrastructure -- for many
organizations it has become an absolute imperative.
One potential solution is to apply the same methodology behind
clustered computing to the storage I/O platform. Build a scale-out
NAS solution that increases both storage I/O performance and
storage I/O bandwidth in parallel to each other. This would allow
for the scaling of the environment, as the workload demanded it.
Additionally it would also allow coherent use of memory within the
NAS solution creating a very large but cost-effective cache.
Finally it would keep the inherent simplicity of a NAS environment
as opposed to the more complex shared SAN solution.
An increasing number of customers are searching for ways to solve
the challenge brought on by multi-tenant workloads and are being
challenged first to identify where the performance problem is, and
then to solve the problem. In short they are struggling with using
legacy storage to address a modern challenge.
Using Legacy Storage to Address a Modern
Challenge
While the demand for capacity is being
met, although not perfectly, the need for a scalable I/O model for
storage is not keeping pace. This is particularly true when
compared to the scalable models that are being implemented for
compute and network performance. As a result, storage professionals
are spending an inordinate amount of time trying to design
solutions to address these performance issues but are being
severely limited by legacy solutions.
Confirming a Storage I/O Problem
The first
step before creating a solution to a storage I/O performance
problem is to validate exactly where in the environment that
problem exists.
When identifying a storage bottleneck it makes sense to first look
at overall CPU utilization within the compute infrastructure. If
utilization is relatively low (below 40%), then the compute
environment is spending 60% of its time waiting on something. What
it is typically waiting on is storage I/O.
To confirm the existence of a storage bottleneck, system utilities
like PERFMON provide metrics which offer insight into disk I/O
bandwidth. If there is not much variance between peak and average
bandwidth, then storage is likely the bottleneck. On the other
hand, if there is a significant variance between peak and average
disk bandwidth utilization but CPU utilization is still low, as
outlined above, then this is a classic sign of a network
bottleneck.
In the legacy model of a single application to a single server
environment, the first step is to add disk drives to the array and
build RAID groups with a high population of drive spindles. The
problem is that a lone application on a single server can only
generate so many disk I/O requests simultaneously
and to perform optimally, each drive needs to be actively servicing
a request. In fact, most manufacturers recommend that an
application generate two requests per drive in the RAID group to
ensure effective spindle utilization. As long as there are more
simultaneous requests than there are drives, adding more drives
will scale performance until you saturate the storage system
controller or NAS head.
The legacy model, in most cases, can't generate enough I/O requests
to feed drive mechanisms and saturate the performance capabilities
of the controller or NAS head. In fact, the traditional storage
manufacturers are counting on the legacy model, because any other
scenario exposes a tremendous
performance-scaling problem with their systems.
Storage Performance Demands of the Modern Data
Center
The modern data center no longer resembles
the legacy model. In general, the modern data center consists
mainly of high performance virtualization servers that participate
in what is effectively a large compute cluster for applications.
Each of the hosts within these clusters has the potential to
house
multiple virtual servers; each with its own storage I/O demands.
What initially began as servers merely hosting 5 to 10 virtual
machines has led to servers with the potential to hold 20 to 30
virtual machines per server.
Today, 30 very random workloads per host in the virtualized
cluster, easily scaling up to 300+ physical machines in a virtual
cluster supporting 1,000 plus virtual workloads is not uncommon,
driving multiple storage I/O requests. Consequently, current VM
environments can easily demand high drive counts within storage
arrays. This significantly heightens the risk of saturating the
controllers or heads of current storage platforms.
Furthermore, in many environments there are specialized
applications that are the inverse of virtualization -- a single
application is run across multiple servers in parallel. As stated
earlier these applications are not limited to a few commonly cited
examples of simulation jobs but cover a broad range of projects.
Many, if not most, companies now have one or multiple applications
that fall into this category. As is the case with server
virtualization, these applications can create hundreds if not
thousands of storage I/O requests. Once a high enough concentration
of disk drives is configured in an array, the limiting performance
factor shifts from drive resource availability to a limit on the
number of I/O requests that can be effectively managed at the
controller level.
The Key Issues
The modern data center now
faces two key issues. First, because of the high storage I/O
request nature of these environments, even if adding drive count to
the system scales performance, there is often a limitation on the
size of the file system. This limitation forces the use of very low
capacity drives which considerably increases the expense of the
system. Alternatively, if drives that strike a better price for
capacity ratio are used, the file system size limitation also
limits the population of drives that can be added to a file system
assigned to a particular workload, thus limiting the overall
performance potential.
Neither option is ideal. The challenge is that storage systems have
to continually evolve to provide ever higher drive counts and
capacities to achieve efficiencies that make sense. Likewise, file
systems must be able to leverage increasingly higher disk drive
counts in order to play into the efficiencies of that model.
Regardless, higher drive counts will eventually lead to saturation
of the storage controller or NAS head. This is the reason that
there is typically a performance bell curve on storage system
specification. While a given storage system may support 100 drives,
it may reach its peak performance at only half its published
capacity -- 50 drives.
Searching for a High Performance Storage
Solution
As discussed, throwing drive mechanisms at
the problem quickly exposes a more difficult bottleneck to address
-- the storage controller or NAS head. The traditional solution to
this bottleneck was to add increasingly more horsepower in a single
chassis. If it was eventually going to require a V12 engine to fix
the storage controller, buy the V12. With the equivalent of a V12
storage controller or NAS head engine in place, disk drive count
would have a chance to scale to keep up with storage I/O requests.
This V12 storage engine often has additional storage I/O network
ports connecting the SAN or NAS to the storage
infrastructure.
There are several flaws with this technique. First, especially in a
new environment or for a new project, there may not be a need for
that much processing power upfront. If all that is required is a
V4, why pay for a V12 now? This is especially true for technology
where the cost of additional compute power will decrease
dramatically in price over the next couple of years. In essence
purchasing tomorrow’s compute demand at today's prices results in
a dramatic waste of capital resources.
Second, it is likely that as the business grows and the benefits to
revenue creation and CAPEX controls of server virtualization or
compute clustering are realized, there will be a need to scale well
beyond the current limitations of the V12 example. The problem is
there is no way to simply add two more cylinders to a V12. Instead,
a whole new engine must be purchased.
This requirement may come from the need to support additional
virtualization hosts with even denser virtual machines or increased
parallelism from a compute grid. It could also come from the
storage controller or NAS head being saturated by the number of
drives it has to send and receive requests.
Unfortunately, the upgrade for most storage systems is not
granular. If all that is needed is more inbound storage
connectivity, the whole engine must be thrown out.
“Throwing the engine out” has a dramatic impact well beyond the
additional cost of a new engine. Now work must all but stop while
decisions are made as to what data to migrate, when to start the
migration and then wait for the migration to complete. This,
especially in some of the simulation environments where data sets
can be measured in the hundreds of TB's, could take weeks if not
longer to migrate.
Finally, even if the organization could justify the purchase of a
V12 storage engine, rationalize the eventual need to buy a V16 in
the future and deal with the associated migration challenges, there
is still the underlying problem of file system size. Most file
systems are limited to a range of 8 to 16TB's. While some have
expanded beyond that they do so at the risk of lower performance
expectations.
As stated earlier, the impact of limited file system size is
manifested in both the inherent limit itself as well as the limited
number of spindles that can be configured per file system. Again if
higher capacity but similar performing drives are purchased, it no
longer takes many drives to reach the capacity limitations of a
file system.
File system limitation also impacts the speed at which new capacity
can be brought online and made available to users. While many
storage solutions feature live, non-disruptive disk storage
capacity upgrades and some even allow for uninterrupted expansion
of existing LUNs or file systems, these
niceties break when a file system reaches its limit.
When a file system is at its maximum size, the new capacity has to
be assigned to a new file system. Then time has to be spent
deciding which data to migrate to the new file system. In this
instance, down time often is incurred while the data move takes
place.
One potential work around for limited file size is virtualized file
systems that logically sew two disparate file systems together,
even when the components of the file systems are on different
storage controller heads. While these solutions work well to help
with file system and storage management, they do little to address
storage performance challenges. This is because the level of
granularity is typically at a folder level or lower. As a result,
the individual heads or storage controllers cannot simultaneously
provide assistance to a hot area of a particular file system and
once again the single head or controller becomes the
bottleneck.
As stated earlier NAS is potentially a preferred platform for these
applications but many customers look to a SAN to address some of
the performance problems. Reality is that both storage types have
the similar limitation of being bottlenecked at the data path going
into the storage controller or NAS head, or being bottlenecked at
the processing capability of the controller/head itself.
The Answer is in front of us
The answer for
solving the storage I/O problem is to leverage the same technology
that moved the bottleneck to the storage in the first place.
Scale-out the storage environment similarly to the infrastructure
now common in the compute layer. By developing the clustered
approach pioneered by
Isilon's Scale-out NAS, infrastructure drive spindles can be added
to match the number of requests by the compute platform on a pay as
you grow basis without worrying about hitting the performance wall
of legacy storage systems.
Solving the Storage I/O Performance Bottleneck with
Scale-out NAS
A performance bottleneck caused by
multi-tenant workloads and server virtualization, that users have
to "live with", can cost companies revenue, customers, or a
competitive advantage, all of which may adversely affect profits
and long-term viability.
As stated earlier, for many storage engineers the gold standard for
improving performance is simply adding more hard disk drive
mechanisms to the storage system. This approach only works,
however, as long as there are more requests from the storage system
than there are drives to service those requests.
As a result, storage performance will continue to scale as drives
are added. In this scenario, most server applications eventually
become their own storage bottleneck because at some point they will
not be capable of generating enough requests to the storage system
to sustain drive additions. The challenge that multi-tenant
workloads introduce is they can easily generate more requests than
conventional storage systems can support---regardless of drive
count. Essentially the bottleneck moves from a lack of available
disk drive mechanisms for servicing I/O to the storage controller
or NAS head itself.
The solution for this problem is found within the very same high
I/O workloads that created it to begin with -- server
virtualization and/or grid compute environments. These environments
allow multiple tenant applications to either live on a single
physical server or allow a single application to scale across many
servers. The same architecture design is now available for storage.
In fact, companies like Isilon Systems are providing scale-out NAS
built on a clustered architecture that allows for a scalable, high
IOPS NAS to address both short term and long term storage I/O
performance bottlenecks.
Symmetric Architecture
The first step in
designing scale-out NAS storage is to base it on a symmetrical
architecture that enables a series of nodes to be grouped together
to act as a single entity. In this clustered architecture, multiple
industry standard servers can be equipped with SAS drives and
network connections to form a node.
Each node can be bonded together through either an Infiniband
Network or an IP Network.
Symmetrically Aware File System
Individual
nodes are united through software intelligence to create a
symmetrically aware file system capable of leveraging disparate
components into a single entity. When this file system is applied
to the hardware nodes, it creates a high IOPS NAS cluster that can
address the challenges of today's – and
tomorrow's – multi-tenant workloads.
Eliminating the Storage Compute Bottleneck
As stated earlier, with the emergence of multi-tenant workloads, no
matter how large a traditional storage system is scaled, no matter
how many drives are used, eventually the I/O capabilities of the
storage compute engine become the bottleneck. The value of a
scale-out NAS configuration is that the storage compute engine is
no longer confined to a single system and a single set of
controllers or heads, as is the case in a traditional NAS.
With a symmetrically aware file system, such as OneFS® from
Isilon, each node in the cluster provides storage compute
resources. In fact, it can also ensure that all the nodes in the
cluster actively participate.
By comparison, some clustered storage solutions must designate a
primary set of nodes, typically two, for each request. While these
systems benefit from the redundancy of a cluster, they often have
the same performance bottleneck of a traditional NAS.
With a cluster-aware file system, each file is broken down into
small blocks and those blocks are distributed throughout nodes on
the cluster. As a result, when a file is needed, multiple nodes in
the cluster are able to deliver the data back to the requesting
user or application. This dramatically improves
overall performance, especially when hundreds, if not thousands, of
these requests are made simultaneously from a multi-tenant
application.
Compared to traditional storage solutions where performance
flattens out long before the system reaches its theoretical maximum
drive count, the symmetrical design of a scale-out NAS system
allows performance to scale linearly as nodes are added. Each node
in the cluster delivers additional storage capacity in the form of
drives, additional cache memory, storage network I/O in the form of
inter-cluster connections, and additional connections out to the
users. What’s more, each node contains additional processing
power capable of addressing both external and internal (such as
replication, backup, snapshot management and data protection)
requests.
Beyond Enterprise Reliability
Since
multi-tenant environments often support hundreds of applications,
it is critical that they actually provide higher levels of
reliability beyond the standard “five 9’s” offered by
enterprise-class storage systems. A failure in storage can affect
hundreds of applications or the performance of a mission-critical,
revenue-generating compute cluster. These workloads also can't be
subject to a one size fits all RAID protection scheme. Some
applications or even specific files may demand specialized data
protection so they can remain operational beyond multiple drive or
node failures.
The symmetrical nature of a clustered high IOPS NAS typically
delivers beyond enterprise-class reliability.
First, there is the inherent value of any clustered environment due
to the redundant nodes. When coupled with a file system, like
Isilon's OneFS that is fully storage cluster-aware, the platform
can deliver granular levels of protection at an application or even
file level. This allows for not only data availability and
accessibility, in the case of multiple drive or node failures, but
also rapid recovery.
Conventional storage systems that are limited to two controllers or
heads must process the rebuilding of a failed drive in conjunction
with its other tasks. As a result, these systems can take 10+ hours
to recover today's high capacity drives. Furthermore under a full
load, the time to recover can increase to 20 hours or more.
Multi-tenant workloads by their very nature are almost always under
a full load and as a result, incur the worst case rebuild
times.
Cost-Effective - Pay as you Grow Performance
Finally, performance has to be cost-effective and justifiable.
Theoretical high-end storage systems provide a top level of
performance by utilizing specialized and expensive processors.
Until the environment's storage performance demands scale to match
the capabilities of these processors, the investment in them
represents wasted capital. Ironically, soon after the
environment’s performance demands have matched the capability of
the specialized and expensive processor, they quickly scale right
past the capabilities of that processor. What is needed is a
solution that can start with a small footprint and scale modularly
to keep pace with the growth of the environment.
Scale-out NAS is the embodiment of a pay as you grow model. The
cluster can start at the precise required to meet the performance
and capacity demands of the environment, allowing the investment to
match the current need. Then as the environment grows, nodes can be
added which
improve each aspect of the storage cluster -- capacity, storage
performance and storage network performance.
Most importantly, like their compute cluster brethren, NAS storage
clusters can take advantage of industry-standard hardware to keep
costs down and enable the NAS cluster vendor to focus on the
abilities of the storage system, not on designing new chips.
The challenge of multi-tenant workloads can ideally be addressed by
a highly scalable but cost-effective NAS. A scale-out NAS storage
system allows you to leverage the simplicity of NAS – maximizing
IT efficiency while at the same time outpacing the performance
capabilities of most legacy storage
architectures. Any organization planning to deploy a multi-tenant
workload, whether it is as common as a virtualized server
environment or a more specialized revenue-generating compute
cluster or application, should closely examine a scale-out NAS
solution to fulfill their storage needs.
