Clustering: Coming soon to a PC near you!
MacDevCenter.com: Integrating Xgrid into Cocoa Applications
Integrating Xgrid into Cocoa Applications, Part 1
by Drew McCormack
Wow - my thoughts and deliberations on Xgrid and what it means for clustering have been echoed on MacDevCenter, by Drew McCormack no less!
I'm very new to clustering, not having access to the resources that were once required, but I do recall reading technical pipe dreams about how spare computing cycles, which amount to massive quantities around the world, could be used to decentralise the power of supercomputers. If only there was someway to harness spare computer processing power which was simple and non-intrusive, then high performance computing - the potential to run simulations and do calculations beyond the reach of desktop computers today, would be in the hands of everyone. Such a situation, so the dreamers said, could revolutionise the way we think of computing, and what we could achieve.
Well, being immature in this brave new world, I kept my excitement about this computing dream actually materialising to a dull rumbling. Drew McCormack, possibly partly in the interests of sensationalism, but more likely based on witnessing more of the same revolutions that I have, is roaring about it from the rooftops. Or at least, from the pages of a MacDevCenter article.
My opportunity to dive into the world of clustering came about because part of my presentation for the Apple Developer Roadshow would in fact, be on clustering. It has been a speedy learning curve, but it is amazing what a few years have done to mature this technology and make it digestable and applicable.
As I presented on Apple's Xgrid, and what it means to us, I remarked that Apple have once again, like they did with desktop publishing in the 80's and with video production in the 90's, brought a technology once solely the domain of a niche and made it available to the masses. After reading some more after the first week of the roadshow I was a little more confident that Apple were onto to something big, and emphasised more the significance of bringing clustering to the layman. McCormack goes a little further though:
"Admittedly it is still in an early stage of development, but the silence from Apple was almost deafening for a technology that may well prove to be one of the most significant in years. Yes, you read right, Xgrid may well be as big a revolution as the iMac was for home computing, and iMovie was for home movies."
And later:
"So why is Xgrid important?
It all comes down to the Apple touch. Take something that already exists, like an MP3 player, see how it could be useful to real people, and then produce an iPod. Take a good idea from some geeks in a research lab, like a Graphical User Interface, realize its potential for real people, and produce a Macintosh. Apple has always been the computer company with vision, and it is about to realize the potential of grid computing for real people.
Apple has taken something that is typically very complex � as you would know if you had ever tried to get any other grid software up and running � and made it excruciatingly simple to install and use. At the same time, they are beginning to blur the definition of what constitutes a single computer. In the future, computers will seek each other out, sharing the computational load, distributing work to where resources are idle. If an application can perform tasks on any networked Mac in the world, where does one computer end, and the next begin?"
In the second half of the road show I had a little more ammunition, having taken a little more interest in the clustering news over the weekend, and took the opportunity to do a little rumourmongering, suggesting that Xgrid is likely to be a star at WWDC 2004. My guesses were that it would be opened up, have a little more fanfare, and perhaps API's would become available to incorporate the technology into Cocoa applications - that's really where this stuff seems to be headed. That is, now we have the simplicity and seemlessness, critical mass will follow and people will start to exect clustering capabilities in their applications. The analogy that comes to mind is the introduction of the network to the personal computer. There was a time when networks were in common contact with the general populace, let alone prevalent in homes. The technology became easier and started appearing everywhere, so people built applications for the network. Now network capabilities in an application are common place - you don't build an application for a network, instead, applications have features which use the network. The difference is that the network is no longer a specialist technology - people expect that their application will run on the network. And I see the same happening with clustering. In the past you set about building a clustering application solely for running on a cluster. Now we are starting to see the shift to ordinary applications having the ability to take advantage of your ad hoc cluster. An Intergrated Development Environment (IDE) like Xcode, is not a clustering application. However, we can now incorporate clustering ability to speed up things like compiling large applications. I only see this becoming more commonplace. At first, it will be unusual and gimmicky, then people will see the benefits and we'll see more attempts. The next stage will be applications which compete on their clustering ability. That is, applications which do not traditionally have anything to do with the cluster will feature differentiate by adding support for the cluster. Once people accept this new tool and appreciate the potential (or even simply buy the marketing speak that it is "good and new so give us your money") applications will simply have clustering support, just as they have network support now.
Well, I didn't quite get across the entire previous paragraph in the Developer Roadshow Seminars, but honestly, that's what I was thinking ;-)
Being a newbie to the clustering world and still finding my feet, I wasn't quite confident to back myself saying all that a week ago. But the more I read, the more it sounds plausible. McCormack again:
"We are witnessing the beginning of the Democratization of Distributed Computing, and it didn't even warrant a mention in January's keynote. But don't worry, it will come. Apple knows what it has. If Mac OS X 10.4 doesn't have Xgrid built in by default, ready to blur the edges of personal computing, I will swim ten laps around Manhattan Island with only an Xserve cluster for buoyancy."
And later:
"Unless I am gravely mistaken, once Xgrid has stabilized somewhat, Apple will publish a Cocoa API, and start pushing the technology to non-scientific developers."