You never know what something will lead to.
I took a quantitative analysis chemistry course back in my undergrad days at Purdue University. Had to take it, I should say. I wouldn’t have chosen it, but I needed a course with associated laboratory session from the science side of the school, as opposed to engineering.
What an influential course that turned out to be!
I don’t know how much pure chemistry I really learned in the course, but that wasn’t really its point. It was about the quantitative analysis, and the same approach can (and should) be applied to biology, to physics, or any area of science you want to call a discipline.
It made me an inconvenient skeptic in the robotics lab in electrical engineering grad school. You see, just because you got an industrial robot to perform some assembly task once, doesn’t mean that it will always do that. Not even if you managed to capture the event on video, ”QED à VHS” as we facetiously called it. The lone success only shows that it can do it once in a while.
Let’s say you try four runs of the assembly process and the robot succeeds on three of them. That will have taken ages given the ponderous speeds to which you have limited the robot’s motion in the interest of safety, which followed the even slower rates of data collection. Everyone wants to call it a day. Three out of four, it works 75% of the time.
But we can’t say that.
Three successes in four trials, and 3/4 = 0.75. But…
One of the main points of that quantitative analysis course is that it takes a lot of tests to legitimately add another significant digit to your final number.
Bayes’ theorem tells us that we can’t say what the real success rate is. We can only estimate that the real rate probably lies within some range. We can’t be absolutely certain that our estimate is correct. Maybe we happen to have observed a really strange sequence of events, not representative of the usual behavior.
Let’s say that we want to have a 95% confidence in the truth of our estimates.
Bayes’ theorem tells us that observing 3 successes in just 4 trials only allows us to estimate that there is a 95% probability that the real success rate lies in the range of 32.986–97.397%. Wow. Anywhere between one-third of the time and almost always.
Ten times the measurement work and 30 successes out of 40 trials means that we can estimate the real success rate as lying within the range 60.539–86.448%.
Ten times more work, 300 success out of 400, lets us say with 95% confidence that the real success rate lies within the range 70.613–79.069%. Hundreds of tests and still almost 10 percentage points wide.
Learning Tree’s Linux optimization and troubleshooting course shows you how to take measurements of various performance issues. Disk reads, disk writes, local file systems, network file systems, and so on.
We cover a lot of ground during that 4-day course. On top of that, the nature of training courses means that we have to take our measurements in an unusual setting, on top of Type 2 virtualization running on top of a full Windows operating system. We only take a few measurements, and they are done in an unusual setting.
The point of the performance tuning course is to teach you the concepts and show you how to make tuning changes and carry out measurements. You need to run those tests in your data center.
But more than that: You need to make several measurements. More tests make for better estimates of actual performance. Don’t make a few tests and jump to an inappropriately precise estimate of performance numbers!