Linux for Suits - Making IT Work
When Hurricane Katrina slammed New Orleans and the Gulf Coast in August 2005, it quickly became the worst natural disaster in US history. After the levees broke and the Gulf of Mexico poured into New Orleans, law and order failed along with the civic graces we call infrastructure. As the waters rose, electric service went out, along with phone, cell, cable and tap water. TV and radio stations went silent. So did communications among police, fire and emergency services. Hospitals struggled to survive along with their patients. On the roofs of submerged houses and the shores of drowned parishes, citizens waited for help from services rendered just as helpless—while looters emptied stores and snipers shot at rescuers. For the next fortnight, New Orleans was Mogadishu.
Yet one beacon of infrastructural persistence stood like a lighthouse above the storm and the tides: DirectNIC, the domain name registrar and site hosting service operated by Intercosmos Media Group, Inc., a company founded by Sigmund “Sig” Solares, who is also its CEO. Sig was born, raised and educated in New Orleans, earning his BA, law degree and MBA from Loyola University there. He is also a determined and resourceful businessman with a loyalty to his company, his customers and his city that verges on the absolute.
In the course of keeping DirectNIC up, Sig and his crew also opened a lifeline to the world, serving as a firsthand source of zero-BS news and photojournalism while also providing much-needed technical help. The survival of DirectNIC is a triumphant story of open source in every meaning of the phrase. It may also be the best DIY-IT (Do-It-Yourself IT) story of all time: a shining example of how to survive a true worst-case scenario.
DirectNIC isn't a small IT potato. According to Netcraft (see the on-line Resources), DirectNIC is the world's #11 domain name registrar, with more than 400,000 customers and 1.1 million domains. It's also a large hosting service taking up 6,800 square feet on the 9th and 10th floors of an office building on Lafayette Square, close to the Superdome and the shores of the city's drowned districts.
Not coincidentally, DirectNIC runs on Linux. As soon as I saw that was the case, I began trying to get in touch with Sig. He finally found time to give me a 45-minute interview on September 7, 2005, while the disaster was still going on. (To place it in a timeline, the levees were still breached, the city was still under water, chaos still ruled and five days would pass before FEMA director Michael Brown resigned.) We did that interview as background for this column. We also made it a podcast you can download from IT Conversations (see Resources).
“When we first started out, we had to go with Linux because we didn't have the money for any other options”, Sig explained. He continued:
We hired programmers who were familiar with Linux. That's just the direction we went. It's made it easy to find employees. We use Red Hat. Everything of course is Apache, MySQL, PHP. We have 17 programmers, and we try to get everything done with PHP and MySQL. If fact, back when they were small, MySQL had a couple of meetings each year. They used our training facility and did their training here. We're big on hiring their training people to get our programmers up to speed.
Thus, what was already clear on the Web became vivid on the phone: Sig and his team are resourceful and determined in large and equal measures. “We have our own 6,800 square-foot data center. It was built for us by Enron, which went in to bankruptcy. We went in to bankruptcy court and said 'We'll give you 5 cents on the dollar for it.' And they said, 'We'll take it.'”
As for dependencies, DirectNIC is closer with its customers than with its vendors:
We've always bought HP servers. We used to have white boxes; but we had some problems, so since 2000 we've been using HP. We've been very happy with those servers. MySQL has always helped us with technical questions. We have a support package with them. But in this case, we really didn't need help from them. In fact, in this case, we didn't turn to any of our vendors. We actually turned to a couple of our customers who were in the area, and we needed diesel, so she went and bought a (Hummer) H1. She's like, “Look, I'm going to make sure you can get diesel in there. And you'll also have a cool vehicle afterwards.”
As Katrina approached and orders came to evacuate, Sig made his position clear:
I told all my IT guys, “I'm not asking anybody to stay. If you're going to stay, and want to come to the office, great. I'm going to be here.” Myself, I stayed. Donny Simonson, a Senior Vice President—he's in charge of all our programmers—he stayed. All of our net ops guys left town. Then Michael Barnett, a friend of mine—he never worked for the company before—he wanted to stay at the office. He already had a Live Journal blog where he called himself the Interdictor.
In fact, Michael Barnett is a former Green Beret who clearly felt his experience and skills would come in handy. His title became Crisis Manager, and his early Katrina entries make clear that managing was exactly what he intended to do:
Sunday, Aug. 28, 12:01 p.m. We're on the 10th and 11th floor of a corporate high-rise on Poydras Ave., right near St. Charles. We have generators and tons of food and water. It is five of us total. I am not sure how the Internet connection will be affected. I have a camera and my gun.
Monday, Aug. 29, 5:12 a.m. Welcome to ground zero.
An hour later, Hurricane Katrina made landfall in Louisiana as a Category 4 storm with winds over 145 mph. That same morning, the 17th Street Canal levee was breached. And the Gulf of Mexico poured into New Orleans through Lake Pontchatrain.
DirectNIC also took a direct hit, as rain and wind broke through the windows of DirectNIC's facility. “You never expect to have water come into your data center when it's on the 10th floor of a building”, Sig said. Yet, they were prepared. Sort of. Sig explains:
We're always looking for angles.
Any time we make T-shirts, it's a sure sign it's going to be a failed business project. Luckily we had a failed project so we had 2,000 of these T-shirts. They were perfect for sandbags....
First, we created a barricade of T-shirts. We grabbed buckets and mops. We also grabbed a wet-vac to suck up the water. So we had six or seven people who were doing nothing but sucking up the water. Meanwhile we could hear the hurricane break the glass behind the wall that we had....And any time the glass breaks, that's going to allow more water through the walls...and head toward our servers.
Today we have cardboard over the vents just to keep the air conditioning in. The storm shutters aren't going to keep the AC in.
The generator is on the 9th floor and we're on the 10th floor. We also have a 1,500-gallon diesel tank on the 10th floor. We have 120 tons of coolers for the air conditioning, and blowers, on the 11th floor.
DirectNIC could stay up only by keeping the generators going and the AC cooling the servers. Also by making sure their fiber optic connections to the Net were working. “We knew we'd lose OC-3s from time to time”, Sig said. “So we had four OC-3s when we knew we could get by with only two.” (OC-3 stands for Optical Carrier level 3, the highest currently provided. An OC-3 line carries three times the base OC-1 rate of 51.84Mbps, or 155.52Mbps.) He continued:
We had to shut a couple things down, but we had at least one OC-3 in operation all through the entire time....
We were relying on three different providers for four OC-3s. Two of them went down. The third was close to running out of fuel, so we had to find a way to get fuel to their facility. One of my customers got on a fuel truck with the National Guard and went down the street to make sure the fuel got to the facility....The provider told us they were going to run out of fuel if they didn't get it by Saturday. So...Brian got on the truck with the National Guard and made sure the fuel got to the building. It was near the Superdome, where the flooding was pretty bad. There was no way we could take a truck there. You had to have one of the military high water vehicles to get the diesel there.
The fiber is all underground, but was fine. Most of the generators were above ground and were fine; but the problem was getting fuel to the generators. Our generator is on the 9th floor. Another one was a couple floors up. So the biggest problem was just getting the fuel to the building, and also getting past all of the military, to convince them that there was enough reason. You have to get approval from City Hall to get fuel moved where you need it.
Meanwhile, the absence of net ops workers put a squeeze on the small remaining staff:
During this whole process, we did have a lot of questions. And one of the problems we had was all of our employees who left....We couldn't reach them by cell phone. The main method of communication was through the blog, and through e-mail and stuff like that. We'd say, “Okay, here are problems that we don't know how to figure out because the guys who usually figure this stuff out are far, far away.” So we were able to get a lot of help in that manner.
For example, “We were so worried that we would not get diesel for several weeks that we had to figure out, 'How can we lessen the load so that we can stretch out our diesel for as long as possible?' We got a lot of information on that, straight from comments in the blog.”
Meanwhile, Michael Barnett's Interdictor became a virtual port in a real-world storm. While the blog served as a half-duplex one-to-any help line for DirectNIC, it also served as on-the-ground zero-BS source of Solid Information about what was actually happening in deepest New Orleans. Michael did most of the posting and had no time for small talk. Every post was a punch to the fat belly of the media beast that was caught no less flatfooted than the local, state and federal offices quickly proved to be. For example, this entry on August 30:
I keep being told that CNN and the Slate reported our “moods” as something other than upbeat. The city is falling apart, no doubt. The looting is rampant. Just take a look at the cam and you'll see them breaking into that hotel and taking everything. The water is still creeping in. But you know what? My team's mood is not negative. We're focused. We've got things that need doing and we're gonna get them done. That's all there is to it. We need diesel. We'll find some. We have people depending on us and we are not going to let them down. That's all there is to it.
And if that's not enough to put our situation in perspective, just remember we live here.
And, the next day:
If you're watching the cam you'll note that there are WHITE people, BLACK people, and HISPANIC people looting. It's interesting to note that I see no ASIANS looting, but I'll leave that observation to the sociologists to explain. Back to the point, don't bring your PC bullshit to this blog. This crisis is not about race, but about inhumanity. Got it?
As a result, The Interdictor took on a media life of its own. On September 1, six days into the Katrina disaster, Michael wrote, “This blog was never intended to be some kind of springboard to fame. I had no idea it would be... well, be what it is now. It's nothing short of stunning....” That post alone had more than 500 comments. Many came in via links from the Times of London and other bigtime publications, as well as thousands of other blogs.
Yet media stardom did nothing to distract from DirectNIC's primary mission. Here's the first thing Michael posted on September 2:
Outpost Crystal has had a long day. Alpha Team is gonna grab a few hours of sleep. You guys are great and all your support is pure motivation to keep plowing ahead. We won't quit, I promise you. We're expecting today (Friday) to be our most physically demanding day so far. Sig and I will be pushing 18 55-gallon drums of diesel up a steep parking garage incline to the 9th floor generator.
And the next day:
The three guys you just saw on the Webcam are Jeff and Hank of Data Protection Services (one of our customers who swore to their customers that they would not go down because of this storm), and Doctor Tom, an anesthesiologist. These three men heroically found a way to deliver us fuel and supplies into this disaster zone. I say heroically, because the amount of effort it took them to coordinate a way to get a container and fuel and a route in to the city in the face of persistent danger on the streets was absolutely off the charts. Their customers need to know the lengths that these men went to in order to get the job done. I listened to their accounts and I was absolutely impressed by the initiative and resourcefulness of these guys.
The result, one month later (as I'm writing this, in early October 2005), is a growing body of wisdom about DIY-IT and both preparing and recovering from natural disasters. Sig explains:
We've been through hurricanes before. We said “Okay, we need to hurry up and get some diesel, and have diesel on standby, and have the standby tank completely filled.” The first time we had only one pump. And a diesel pump is going to burn out, and you have to wait for it to cool off. So we learned a lot because we've already been around a couple of times. But nothing nearly this scale. So this time it was definitely a trial by fire.
We've done migrations before. But when we've migrated from one data center to another, the biggest migration we had was five racks. We did five racks across the country. This time we had a 6,800 square-foot data center. Things just end up so much more complicated. Especially when you throw in the fact that now most of your employees you can't reach. We did have seven to ten guys who do a lot of telecommuting work. That's one of the things that really helped us out. One of the owners lives in Naples, Florida. He handled all of the coordination of migrating stuff away from this facility. So he just worked around the clock for a week, starting with the most critical stuff and working his way down. We have a lot of customers who aren't worried about having a solution redundant across two data centers. They'll pay us $15 for a domain and we host it for free. So in those cases we don't provide two different solutions. So if this data center were to go down, they'd be down. But once we were faced with the prospect of the data center going away, we wanted to do everything we could to prevent that from happening.
I've just been focused on making sure everything has been under control.
Sig is as plain about his company's role as a news source as he is about its role in the marketplace: “We've definitely tried to put out as much information as possible while doing everything that has to be done.”
DirectNIC also worked to help communications inside New Orleans as well. Sig reports:
There are so many different organizations. I've talked with Alabama police, Florida police, reserves....The communications between the different groups are not very good. [This is one] of the things we're trying to help. The city's phones (mayor's office and so on) are handled by one of our customers. So if our facility were to go down, the city wouldn't have any phones. NOPD lost so much of their capacity to communicate as a result of Katrina. We had a police officer who stayed with us and went 24 hours without hearing from command.
Exactly one month after I spoke with Sig, he published an open e-mail on the Interdictor, which he and Michael Barnett had transformed into the Survival of New Orleans blog, with a new URL: mgno.com. That e-mail contains such valuable and hard-won lessons that I can find no way to edit it down. Here it is:
Recently I received an e-mail from a customer who was worried about domain names that he registered at DirectNIC because of the problems in New Orleans. Let me make one thing perfectly clear: Every other business in New Orleans may fail; nevertheless, Intercosmos Media Group, Inc., operators of DirectNIC, will continue on successfully in New Orleans and/or elsewhere without interruptions to our clients.
New Orleans has never been known as a pro-business location. Despite this, we have thrived in the Crescent City. Prior to Katrina, we were one of the most successful businesses in New Orleans. A few days after Hurricane Katrina hit, only two offices had their lights powered on the main street (Poydras Street), in the heart of the CBD. Those two offices were Bell South and Intercosmos Media Group, Inc. It was no accident that we continued to operate.
At Intercosmos, we did nothing special to prepare for Hurricane Katrina. We had already prepared for dozens of prior hurricanes and we already had our disaster preparations in place. The following is a very short summary of how we have prepared for any catastrophe:
1. Require Excellence from Employees. A small excellent team can accomplish anything. If you have the right team, then you can survive and actually thrive during any disaster. We have spent years putting together the right team of employees to operate under any circumstance.
2. Built an Excessively Large War Chest. At Intercosmos, we have never borrowed money to facilitate growth. We always focused on increasing liquid assets while maintaining steady profitable growth. For this strategy, we just followed Bill Gates who likes to keep a large amount of cash on hand for a rainy day. Because of this we are well positioned to weather any storm or series of storms.
3. Secured a $3 Million Line of Credit. We secured this line of credit despite the fact that we had and still have no intention of using this line of credit. The line of credit is merely a safety net.
4. Lots of Insurance. We are insurance junkies at Intercosmos. Nevertheless, as a result of Hurricane Katrina and our preparedness, we will have fairly limited claims to make.
5. Telecommuter Plan in Place. Prior to Hurricane Katrina hitting, close to 50% of our employees had telecommuting agreements in place with the company. When any hurricane hits and employees evacuate, we plan for employees to telecommute through the use of Internet-based (VoIP) phones, wireless Internet and laptops.
After Hurricane Katrina hit:
1. Increased levels of redundancy. Now we have servers in four secure, distant facilities instead of just in two facilities.
2. Increased profitability and cash on hand. Our business does not rely on areas hit by Katrina. Well over 99% of our business comes from individuals and businesses outside of Louisiana. As a result, we were actually more profitable in the month after Katrina hit than we were in the month prior to Katrina hitting.
3. We started to use an 8,000-square-foot building that we have in another state as a re-location work space for employees. Employees who cannot work in one of our buildings in New Orleans can work as a telecommuter or from our other office.
What can you do to help?
We are passionate about our city. We want to see New Orleans rise again. However, our fate is not tied to New Orleans. We are trying to raise awareness of problems in New Orleans because we want the rebuilding effort to succeed and not get tied down in politics as usual. If you would like to help us on this mission, we ask that you join in the conversations at the message board we launched at nola.us.
Sincerely,
Sigmund Solares Chief Executive Officer Intercosmos Media Group, Inc.
My wife divides the world into “savers” and “spenders”. Her theory is that most of us are born as one or the other. She's a saver. I'm a spender. Our four kids include two of each. Phil Hughes, our founder and publisher here at Linux Journal, is a saver. Near as I can tell, so is Linus Torvalds. After hanging out with Andrew Morton (the kernel maintainer that Linus calls his “right-hand man”) on last fall's Linux Lunacy Geek Cruise, I'm convinced that he's a saver too.
So, clearly, is Sig Solares.
As my wife explains it, the difference between savers and spenders is debt. Especially credit-card debt. She didn't want to marry me until I carried no credit-card debt and began paying my bills in their entirety every month. I've done that for so long now that it gives me chills to think about going back to my old ways. I may be a born spender, but I'm not a practicing one. At least not when it comes to debt.
Linux, I believe, is the saver's operating system. The values of saving are also embodied in the development and application practices of every free software and open-source project.
Surviving disasters is about saving too. That's the lesson that transcends politics, race, class and every other issue that divides us. When the going gets tough and the tough get going, the ones that go farthest will be the ones who saved most in the first place.
Resources for this article: /article/8637.
Doc Searls is Senior Editor of Linux Journal.
