Distributed Systems Group -- Research

Extreme Scale Cluster Architecture

A well-known trend today is that data is growing at a much higher rate than processing power. For example, the amount of data in GenBank (a genomics database) doubles every 9 months compared to the increase in processor speed (roughly doubles every 18 months by Moore's law). Where all this leads to is that in the near future we are going to need much more processing power that single CPU's can offer. A cheap solution to this problem could be clusters of commodity PC's in extremely large numbers (thousands or tenths of thousands of PC's connected together). However, little work has been done in solving the scaling problems that clusters of that size have. How should the data be send between the nodes so that congestion and latency problems do not arise? How do you deal with node and network failures that are inevitable in a system of that size? What's the most effective way to interconnect such a big network without spending the bulk of the budget in exotic switches? And last but not least, how can one program and debug such large scale applications?

On a first thought, many of these problems seem orthogonal to each other; a connection between, e.g. the structure of the network and how to program efficiently large-scale applications is not obvious. However, we believe that an integrated solution to these problems does exist: designing the whole system based on streams. Our thesis is that by using streams one can: