Another Amazon EC2 Beowulf Cluster Joins The Grid
March 21, 2008 – 10:21 pmHave you ever been working with a dataset, started crunching some numbers and said to yourself, "damn, I should distribute this across the cluster," only to realize that your cluster is already saturated with your last job and will be for the next day or two? If you answered yes, then we probably share the same data-craving/slicing/mining sickness.Well the above scenario happens to me and often enough for me to pose the question to others. I could simply invest in a larger cluster --- an expensive investment, especially since the scenario often only requires bursts of compute time. This would make an on-demand cluster a perfect solution. On-Demand Beowulf I had heard some chatter about Peter Skomoroch's ElasticWulf and found myself walking through his series on creating an on-demand beowulf cluster using Amazon's EC2. You can find his very helpful posts here and here (with another on the way). ElasticWulf is ...
Tags: Distributed Computing, Python