Throughout life there are a few things that are consistent. Three of those that I readily acknowledge are death, taxes, and change. Change is the one that I’ve realized over the years that you can embrace it or get owned by it. Change is a lot like death and taxes in that sense. You ignore change and eventually you’ll die or get slammed with a huge bill by the IRS who will then demand payment on their timeline, not yours. There are also times where you want to change to preserve something that is valuable. Technology is one of those areas that changes so rapidly that when you don’t keep on top of those changes you end up with a big bill to pay and possibly even pieces of hardware that die on you. Around here we call it technical debt and it currently has outstanding balances.
The Clemens house, in Huntsville, was built in 1835 and lived on the same corner for 169 years. In 2004 Huntsville Utilities was in the process of expanding their facilities and needed more space to accommodate the growing traffic in and through their new central office. The Clemens House was standing right in the way of this growth. Rather than destroy the house, Huntsville Utilities along with other teams in Huntsville decided to preserve the house, but this meant it had to be moved roughly 1 mile to a new location. The uprooting of the house from its foundation, sharp turns, road closings, bystanders, and people involved to make this happen was truly a spectator sport in Huntsville for weeks in 2004. The tech team here at Solid Earth is right in the middle of a project much like the moving of the Clemens house. One group is laying new foundations while another is prepping the application to move across town and finally be placed onto the foundation. This has been a long term plan for List-It, which is coming to fruition. We have been working steadily for the past 7-8 months planning, purchasing, configuring, and implementing the new virtual network at our Colo here in Huntsville. This build out has some fascinating pieces too it:
- Redundant Hosts for application deployment. These aren’t your typical hosts. Each has 192 GB of RAM and dual quad-core processors
- Storage with Redundant, hot-swappable controllers, power supplies / cooling fans, and disks. A disk array totaling 26TB +/- of storage.
- Networking equipment boosted to GB ethernet.
- Backup facilities outside of the “Dixie Alley“
- Monitoring down to the fan speed on each host.
With the hardware in the rack, now the primary focus of the team is to set List-it onto this new foundation. We’ve come around corners thinking everything is going well only to find that we are embracing change thrown our way when critical pieces of the network decide to drop on us. However, much like Dale Earnhardt, we’ve wiped the mud off the windshield and kept on racing. The DNS outage on 3/11/2012 was painful and couldn’t have been a more perfect storm in the life of this aging system. However, in continuing to race we now have two brand new DNS servers sitting just on the outside of our network hosting all zones for Solid Earth. In the near future we’ll be spinning up two more name servers to run behind the main authority servers just because we don’t want to see anymore perfect storms… or hear about them… or well you get the picture.
Back to the virtualization, We are currently working to move multiple sites into the virtual environment. We are also working to improve the database machines that support List-It. These machines will remain physical, but are going to be blazing fast. This move requires configuration, testing and some maintenance windows that will be coming your way in the near future. You’re probably asking when? We are shooting to have a few List-it sites virtual by the end of April. The first ones should be ready to move over soon and once we’ve completed our testing and monitoring we’ll be contacting you for the final stages of testing and scheduling maintenance.
So what, right? The what is this new hardware will be putting us into a place where if we lose a disk on the storage array we’ll have the ability to pull it and replace it with no downtime or loss of data. If we lose an application host, there are two more sitting beside it that will immediately pick up where the failed host left your VM at failure. Within the Huntsville colo we will have fully redundant systems for List-it. Pairing this with the backup sites living in Chicago now we believe the stability of List-It will be noticeable. When all List-It virtualization is complete we’ll be shifting our focus towards the planning of the next phase in Chicago. More on that later, but think Huntsville in Chicago and they talk to each other.
I can speculate and guess about what might happen once List-It is virtualized, but I think it is best to wait and let the system speak for itself. I’m reminded of something my friend Scott used too say when we were going fast and hitting it hard. Keep in mind that while we’re driving 90 mph down the road changing the tires on List-It, we will hit bumps and have setbacks, but hold on we’re almost there. So be ready when we call for the change coming to your List-It system. It’s coming and we’re working hard and fast to get it done and when it is within our control we are doing this with the focus of mitigating risk and impact on your end.
Excellent deep dive Robb. I hope our customers understand the value you’re creating in the new network. Your post really helps provide some visibility into the hard working but invisible parts of the network, keep ‘em coming! I know some of our customers really love hearing about the details! I competitors do too probably but that’s OK. If you’re a competitor and you’re reading about us on our website, that’s cool too! Hi! Leave a message and maybe we’ll see each other at a trade show!