Math for Storage Design: Part One – Performance

By | April 16, 2014

harddrive

Introduction

In the past 15 years virtualization has transitioned from a novelty to the de facto standard for data center operation. Until recently few people understood the real implications of this transition, especially in the realm of storage.

We’ve seen servers go from almost all direct attached storage, to complex SAN and NAS arrays, and now back to direct attached with the trend in node based hyper-convergence technologies like VSAN.

No matter what you choose for your storage there are certain things you need to consider. Most folks plan for capacity, but throughput and performance are just as important. Here are some rough guides to help you with sizing your storage.

Performance, the IOPS Story

Input Output Operations per Second, or IOPS for short, is the main measurement of storage performance. Unfortunately IOPS can be co-opted by less than truthful marketing because all IOPS are not the same. The discussion of performance cannot be isolated to the driver or array. Workload matters. It is important to understand the percent of reads vs. writes, randomness of the drive access, and block size.

When researching storage options be cautious with vendor reported numbers. Most vendors report IOPS based on 4K random reads, your workload is most likely much different so be sure to test the hardware yourself to verify.

Formula for IOPS of a single non-SSD hard drive:

IOPS = 1 / Average Seek + Average Latency

Example: Seagate Constellation ES.3 ST4000NM0023 4TB 7200 RPM 128MB Cache SAS 6Gb/s 3.5″ 

Average Latency: 4.16 ms

Average Seek Read: 8.5 ms

Average Seek Write: 9.5 ms

Read IOPS: 79.617

Write IOPS: 73.206

IOPS for a single Solid State Drive

Calculating IOPS for an SSD drive is not nearly as clean cut. Drives range from two thousand to several tens of thousands of IOPS depending on the technology, size, and power consumption.

Your best bet when sizing the IOPS for an SSD is to research the drive on a third party website. The IOPS entry on Wikipedia has a good list. As always your mileage may vary when using a resource like this so when possible test the drive yourself using a tool like IOMeter.

Multiple Drives, RAID, and Write Penalty

Performance calculations for a single drive are pretty straight forward, at least for spinning disks. Things get a little more complicated when you start to create aggregates of disks. Let’s look at some common ways disks are grouped and how this affects the performance.

JBOD: Just a Bunch of Disks

JBOD is the least sophisticated disk aggregate. A volume is simply spanned across multiple disks. There is no data protection and if one drive fails the entire volume is destroyed.

Lately JBOD has made a surprising come back due to distributed systems like Hadoop that are failure tolerant due to the structure of the data. There are several other applications and frameworks that also started to embrace the “Let It Fail” mindset pioneered by folks like Netflix. As more of these apps are deployed in the Enterprise data center I expect to see JBOD grow even more in popularity.

Read IOPS in a JBOD array are extremely close to the total combined read IOPS of all disks in the array.

Write IOPS are slightly less, however they improve closer to the Read IOPS with more disks are added. The corollary to this is that adding more disks improves the likelihood of the volume failing.

RAID 0: Striping Without Parity

RAID 0 is slightly better than JBOD from a performance aspect. It still provides no data protection, however the Read IOPS is identical to the combined total of drives in the array due the fact that data is striped evenly across all drives.

This is also the only aggregate where that is also true for Writes. There is statically zero write penalty for RAID 0.

RAID 1 and RAID 10: Mirroring

To this point we have seen almost no Write Penalty when creating disk aggregates. Data Protection changes this dramatically.

All RAID arrays have no READ Penalty, meaning that once again the Read IOPS for the aggregate are equal to the total number of Read IOPS of all drives in the array.

Mirroring disks provides excellent protection, however the cost is HALF of Write IOPS. We refer to this as having a Write Penalty of 2.

Formula for calculating Write IOPS for Mirrored arrays is:

Let X be the number of drives, and let N be the total WRITE IOPS for the drive

(X * N) / 2

RAID 5 and 6: Striping with Parity

Again here the Read IOPS are the same as above, however Write IOPS take an even larger hit.

The Write Penalty for RAID 5 is 4 and RAID 6 is 6. This means that RAID 5 has a quarter or the available Write IOPS compared to RAID 0. RAID 6 has an even less with a comparative sixth.

Formula for calculating Write IOPS for RAID 5 arrays is:

Let X be the number of drives, and let N be the total WRITE IOPS for the drive

(X * N) / 4

Formula for calculating Write IOPS for RAID 6 arrays is:

Let X be the number of drives, and let N be the total WRITE IOPS for the drive

(X * N) / 6

Ok, so now we know how to calculate the Read and Write IOPS for a drive or aggregate, now we need to blend them together. For this we need to analyze the workload.

Workload Analysis

Almost no production workload is 100% reads or 100% writes. It is always a blend of the two. To properly size an array, you need to know the workload that is going to be running against it. Discovery of the workloads current metrics is the fastest way to learn this.

In vSphere there are a number of tools to measure workloads. Performance charts, esxtop, vscsiStats and others can get you the information. For the our needs here the question to ask is:

For a given period of time, what is the total disk IO and what percent are Reads and what percent are Writes?

With this question answered we can use the following:

Formula to calculate the IOPS requirements for a workload:

%Read *TotalIOPS = Read IOPS

%Write*TotalIOPS = WriteIOPS

The goal is to now use these numbers as one of the requirements of sizing the array. There are others, but this is where we need to start.

Next Steps

After we understand our performance needs we can think about throughput, capacity, advanced technologies, and cost. Look for continued discussion on those topics in future posts.