Moore Linux Power

Before Beowulf began in 1993, High Performance Computing Clusters (HPCC) belonged to the world of elite organizations with large, very specialized projects and multi-million dollar budgets. Then along came Intel exploiting its founder’s vision of exponential increases in compute power coupled with corresponding decreases in cost. Suddenly box vendors like Dell were able to make supercomputers into a mainstream enterprise computing item. Even IBM and HP could see the possibilities for new ways of profiting - no longer primarily from the raw hardware, but in the design, services and support of large-scale, complex systems.

From Academia to Enterprise

High performance clusters are groups of computers connected together to operate in parallel on discrete parts of a large compute problem. By executing a high number of calculations simultaneously and minimizing memory and communications latency, clusters deliver enormous computing capacity.

HPCC on Open Source originated in labs and academia due to budget restrictions as well as to an abundance of high quality programming talent. Scientific research, weather forecasting, genomics, geochemistry and others have achieved great success with clusters of cheap Intel PCs and OSS clustering technologies. Now, HPCC is steadily moving from academic applications into the enterprise with automobile, bioinformatics, entertainment, and oil exploration companies leading the way.

A new level of organization - grids of clusters and compute resources - is emerging in the form of on-demand distributed environments where complex data is available over the same networks as the ones that deliver email to end users’ desktops. The combination of high performance clusters, supercomputers and desktop systems has led to the formation of “grid communities”, which can target a variety of massive computation problems cost-effectively.

Great Examples of Linux HPCC

HPC at PNNL
The U.S. Department of Energy’s Pacific Northwest National Laboratory (PNNL) grid-enabled Linux cluster cost a mere $24.5 million. It is a 11.4-teraflop, 1900 node Intel Itanium based cluster for research in nuclear science, biology, genomics, and material sciences. It’s 45 times faster than PNNL’s earlier supercomputer, an IBM SP/2 which was the 19th fastest supercomputer in the world when deployed in the dinosaur age of 1997. PNNL selected a Linux based cluster because of robust scalability, high reliability and superior access to applications and tools.

HPC at AIST
IBM has recently deployed a Linux based solution for Japan’s National Institute for Advanced Industrial Science and Technology (AIST). The eServer supercomputer has 2116 Opteron 246 processors, and 520 Intel Itanium processors. Its 11.2 teraflops would rank the AIST HPCC just above the current Linux king, a cluster based on Intel’s Xeon processor at the U.S. Lawrence Livermore National Laboratory (LLNL). AIST will link this cluster with others in a supercomputer grid that will explore grid technology, life sciences, bioinformatics and nanotechnology.

High Expectations

The economies of Open Source HPC have been based upon harnessing cheap hardware to solve complex problems. But a close look at HPCC reveals that not only can commodity hardware change the way the game is played, the forces of commoditization actually apply to the software as well. Software commoditization is now beginning to take place everywhere. Just as the bulk of scientific computing can be handled, often better, using Open Source HPC technologies, the day is nearing (some say it’s already here) when the bulk of all computing (scientific, medical, office, factory, retail, embedded) can be handled by the winning combination of increasing capabilities and decreasing cost in the form of open standards and an evolving platform of Open Source Software and tools: the gathering power of Linux might just demonstrate a kind of Moore’s law of software. Exciting times are ahead!