Let' say you are a systems project manager at a financial institution and you called on numerous systems developers to adopt grid computing. I assure you, you'll get really excited listening to all the explanations and rosy outlook from eager sales people and consultants. That's because unlike academic grid computing and systems like SETI@HOME - in which an unspecified large number of people take part - grid computing for private-sector businesses prioritizing security takes place in enclosed networks. And most require the installation of a large number of blade servers. Think about it. For a sales person at a manufacturer who was told to "Go out and sell XX units of blade servers this fiscal year," your project is an easy approach to bringing in several tens of millions of yen to several billions of yen in sales. That sales person can sell several hundred blade servers from one negotiation. This person would be worthy of Employee of the Month award.

So in this column, we'd like to mention some issues that IBM, HP, Gartner and other companies could never reveal (or are probably forbidden from revealing from their companies.) In other words, we are offering you knowledge you really need to know before adopting grid computing.

 

Reasons Why Systems Developers Love Grid Computing

Is grid computing the latest technology? Not at all. Talk of parallel computing, decentralized processing have been around since the invention of computers. These topics cyclically become a fad in commercial negotiations. (You may not remember the last time these topics became a fad, but it was more than 10 years ago.) Then why is the topic of grid computing the focus of so much attention now? To understand the reason, please refer to the following chart.

 

Changes in Intel CPU Clock Frequency (Unit: MHz, logarithmic scale)

If you look at the chart, you can see that the processing capacity of CPUs have stopped improving around 2004. Don't you feel that the processing speeds on computers you've bought recently don't feel like they have improved as much as compared with your past computer purchases? That's because the processing speed of CPU's has hit a ceiling recently. There are both physical and economic reasons behind this plateau. What's important is that the technical walls we face now are huge, and that perhaps for years (maybe even for more than 10 years) CPU processing speeds will continue to be stuck at current levels. For detailed information, please refer to other documents. (For example, W.W. Gibbs' "Multicore Chips" in the February 2005 edition of Nikkei Science, page 98)

This means that as long as the processing speed is the main marketing feature, computer makers won't be able to encourage consumers to buy new computers. That's why US developer Intel and other CPU manufacturers began marketing in-chip parallel-processing called multi-core, and systems developers invented a new word called "scale out" - insisting that the overall processing speed will speed up when many servers are connected. One could accurately say that grid computing is the cool version of the scaling out method. (Academic grids are a different story compared with financial grids even though some may say that interconnect technology developed because of the grid boom.)

Scaling out can produce an extremely economical system. Please refer to the following chart.

 

Standard Server Prices by Memory Volume and CPU Clock Frequency

(Source: Numerical Technologies research into the market price of 2CPU servers equipped with 73GB x 4HDD, 64-bit Windows as of June 2007)

This chart shows that server prices surge when servers exceed a certain size. The single CPU has little effect over the price of servers. What jacks up the price are the number of CPU and the memory volume. At the time I was writing this document, server prices skyrocketed when the number of CPU exceeded 2CPU (sockets) or the memory volume exceeded 16GB. In other words, if you compare prices based on the processing ability, the 2CPU (8 core at this point) 16GM memory machine is the best buy. (That's related to the problem of CPU=inter-memory interface and the DRAM market cycle, but I will refrain from explaining the reason for this because it is unrelated to our main topic.) Don't you think it would be much wiser to connect many of these 2CPU machines (= scale out) rather than buy a huge server from Sun or HP (= scale up)?

Unfortunately, it's not that simple. Let me point out two issues you have forgotten.

The first problem is who is going to write software that works on such parallel hardware?

Sometime around 2005, experts were avidly discussing the theme of "a free lunch in terms of software." The details are explained in an article titled "The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software by Herb Sutter." (Herb Sutter is as prominent as Professor D.E Knuth in the industry and currently serves as a consultant at Microsoft Corp.) It is very difficult to write parallel-processing software that operates smoothly. This is another reason why there aren't as many game software compatible with Sony's PlayStation 3 game console (which uses multi-core CPU) compared with Nintendo's Wii or Microsoft's Xbox 360. Pricing isn't the only reason for Sony's failure in its game machine strategy.

The second problem is that grid computing systems break down.

There are words called Mean Time Between Failures (MTBF) and Mean Time to Failure (MTTF) to show how easily a system breaks down. Some unbelievable numbers pop up when you look at a manufacturer's catalog. For example, they say that the MTTF is 1 million hours. For those of you who thought that this means systems only break down once every 114 years, a seasoned engineer who has worked in machine rooms will give you a glimpse of what happens in the real world. The following chart demonstrates the reality of systems.

Average Hardware Failure Rate per Year of User at US Internet Firm Google

(Source: Eduardo Pinheiro, Wolf-Dietrich. Weber, and Luiz André Barroso, "Failure Trends in a Large Disk Drive Population." Appears in the Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST'07), February 2007.

These statistics show the hardware failure rate at US Internet firm Google. The hard disk is one of the parts of computer that is most susceptible to failure. The abbreviation AFR used in the chart stands for "Annualized Failure Rates" and refers to the rate at which failures occur in a certain year. This chart shows that the failure rate was the lowest in the first year at 1.7%. However, the frequency of failure surges in the second year, and peaks at 8.6% in the third year. One might say that hard disks fail in the early stages as well as in the third year and onward.

There are similar reports. A thesis by a Carnegie Mellon University researcher unveils the following facts after studying a massive amount of disk failures. (The thesis is by Bianca Schroeder and Garth A. Gibson. It is titled "Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?", which appeared in the 5th USENIX Conference on File and Storage Technologies Feb. 13-16, 2007.)

• Hard disk failure rates are as high as failure rates for memory
  
  CPUs have a 250% lower failure rate than hard disks, and motherboards have a 50% lower failure rate compared with hard disks. In other words, the failure rate levels can be expressed as HDD=RAM>> motherboard >> CPU.
  
  
• The Annual Replacement Rate (ARR) for hard disks is 3%, much higher than the manufacturer's estimate.
  
  In addition, there is no scientific proof that expensive SCSI or FC are better than low-cost SATA. On the other hand, the highest Annualized Failure Rate (AFR) noted on a manufacturer's data sheet is 0.88%.
  
  
• Hard disks break down one after another
  
  

What do these things have to do with grid computing? Well the answer is obvious when you calculate what happens to the failure rate when many servers are connected. The parts of a computer that can fail are the hard disks, memory, power source, motherboard, CPU, network, fans and others. Please refer to the following chart.

Failure Rates of Grid Computing Systems

Let's say you adopted a grid computing environment of about 1000 nodes. The manufacturers will probably be very happy (and the sales person who sold you this might win the Employee of the Month award.) The grid computing system starts operating smoothly and the systems integrators are very happy. Then the following year, and the year after - when the manufacturer and integrators are all gone - the failure rate continues to rise. What do you do?

What this chart shows is the plight of the maintenance hell in which workers are busy exchanging failed parts everyday. In reality, even the academic grid TSUBAME (655 nodes in early 2007) at the Tokyo Institute of Technology was considered normal (for a calculation node for a super computer of this size) when breaking down at the rate of one had disk a week, according to Hisashige Ando's comment in the January 8, 2007 edition of Mainichi Communications Journal titled, "The Flight of Japan's Fastest Super Computer - the Tokyo Institute of Technology's TSUBAME."

If there is a sales person who says that reliability will improve if you switch to RAID, ask them, "Even if data is saved by RAID, wouldn't failures increase because the number of parts will increase?" If a consultant says that the fault tolerance will improve if you use a middleware in the grid, you should point out that, "Even if the middleware is fault tolerant, it doesn't help if the application running on that middleware is not fault tolerant." In a nut shell, if you don't use common sense like this, you will end up regretting the purchase of a grid computing system.