When you are a system administrator in a large environment you are often involved in investigating problems with applications performing poorly due to latency problems. From my experience, the problem is that in many cases system administrators don’t know the difference between latency and throughput. In some cases they know from the grapevine (such as MCP courses) that the disk queue should not exceed 2, and that hosting databases on a RAID 5 is not a good idea because of its congenital bad write performance. They can also see that everyone in the storage market is obsessed by IOPS and that many storage vendors claim millions of IOPS on their systems, but can’t really tell what affects the advertised figures.
That's why I have decided to walk you through all the necessary steps to better understand concepts like latency and throughput, and how to measure them properly using modern tools and of course PowerShell. Take this very long series of posts as a quest. We will perform a lot of tests following a trial and error methodology, which is perfect since we can assume that we have low-to-no-knowledge of such a vaste subject.
As you will see, I have added many links to external resources that can be used to dig deeper into the subject, and I suggest you to read them carefully.
WHAT YOU WILL LEARN
You will learn here that measuring storage performance is NOT a hard task, as long as you know the terminology, you have the right tool for the job and you have a few guidelines to follow.
The terminology is pretty easy. You are trying to measure the performance of a storage subsystem as a whole, which is composed of three things: a disk, a controller and a host bus adapter (HBA).
To test your storage, you need a tool.
IOMETER: A TOOL FOR A SYNTHETIC WORKLOAD
The best known tool for checking your storage subsystem (controller and disks) performance is for sure IOmeter, which uses an executable named Dynamo.exe to generate the desired synthetic workload.
Its interface has five tabs: Disk Targets, Network Targets, Access Specifications, Results Display and Test Setup:
The main tab for testing is the one named 'Access Specifications', which includes a list of possible I/O profiles to measure your storage subsystem capability.
If you look inside an access specification, you'll see the following window:
Basically, what IOmeter is asking you here, is to input what kind of workload you want to measure your storage subsystem against, in terms of:
- Transfer request size
- Percent Read/Write Distribution
- Percent Random/Sequential Distribution
- # of Outstanding I/Os, (which by the way appears on another tab)
Once you have chosen an appropriate access pattern, you have to attach it to a workload, knowing that Intel (the original editor of IOmeter) recommended using one worker per processor.
Once you have run a workload, you can retrieve the results on a dedicated tab, where by default you can find four key values:
- Total I/Os Per Second (aka IOPS)
- Total MBs Per Second
- Average I/O Response Time
- % CPU Utilization
WHAT'S WRONG WITH IOMETER
There are three major problems with IOMETER:
- The first problem with IOMeter is that it doesn't get updated since 2006, so it's getting dated. Nonetheless it is a good idea to start by having a quick look at it because it is still one of the most complete tool around.
- The second problem is that, as you can see, IOmeter interface is pretty complex, so it can be a little scary for first time users, and it is not looking at it that you will get less confused on the subject: IOmeter has its own terminology for everything (Address specifications, Transfer request size or Average I/O response Time), and you have to make a strong effort to match their keywords to general concepts used by other tools.
- Third problem, a major one, IOmeter doesn't allow automation of the task, which is something very very bad, especially when there is PowerShell around to help you with the implementations of tool.
That's why I will not talk about IOmeter anymore in this series of posts. What I want here is a tool which can help us better understand what we are doing, because setting up a synthetic workload representing your real workload is a challenging task. The main lesson that you have to retain after trying IOmeter is that we are trying to produce a workload which gives us meaningful results:
If you are eager to learn more about IOmeter, there are a couple of articles out there which are well written:
- The first is a beginner level how-to which explains the interface and how to run workloads on a Windows or a Linux computer
- The second is a post giving suggestions on IOmeter settings
Ok, the road is still pretty long and we still need to find a tool which can help us understand all the ins and outs involved in the measurement of the performance of a storage subsystem. Let's move to part two of this series.