Modern Ninja :: Migrating Servers

Migrating Servers Mon, 20 Apr 2009 14:41:45 -0400

Okay, so I'm nearing a transition in life. Not only is my first baby due 1 month from yesterday, but I also have to migrate one of the largest site's I've ever managed. I can't give any details atm for what it is, or why. But I am moving everything to a cluster of servers configured in Amazon's EC2 Cloud using a few parts of their technologies, which will hopefully expand to some others, that should end up totally rockin, and way cheaper. Currently the binary database for this system is approaching 100 gigabytes. And there are other migration issues at hand in regards to the nature of the network and the data it maintains, but sufficient to say that there is some new data being imported at least every 10 minutes or less.

Each day has shown increase in load, as well to the fact that the imported data stands around 20 million records a day currently and that now we will expand the abstraction of said data from 1 table to 250+ tables. Thank goodness with EC2 I can quickly throw up more instances to handle any load and scale horizontally.

TBH this is probably the first technology that I've been really impressed with in a long time, now the only question is if it can stand up to it's promise.

This isn't my first time messing with the concept of shared resource processing networks. When I was 16 I built a beowulf cluster using computers that were being thrown away by my high school. Mostly Pentium's and 486, but considering I didn't have anything better and the electric bill was paid by my parents at the time I enjoyed the challenge. Granted I never was able to find a use for the network. Then a couple of years ago with 3tera launching their grid I began to research and came up with a couple of theories of how they had done this. I concluded they used a set of common GNU tools, the names of which have long escaped me. C'mon it was like 3 years ago!

Anyway, it's very cool and change is a comin so you better get ready for it!

:: ModernNinja (dot) com ::

Be the first to comment!

Post Comment