blog.Adonis.net

Yesterday was a bad network day. It started with a slow link from one of our websites out to our most obnoxious clients. Those are always the tough ones; if a link’s dead, it’s usually pretty obvious where things are screwed up. When they’re just intermittently slow, it takes a little more detective work and some actual thinking to pin down what’s going on. Fortunately, I’d received some good data from the client (a very rare occurrence) and was pretty sure I had it pinned down to one of our ISPs’ routers.

Anyone who’s worked in networking can tell you that’s where the battle starts. No provider anywhere, regardless of how big or small they are, wants to admit the problem is on their equipment. It’s just one of those rules of the Internet: it’s always the next guy’s problem. So, even though I had evidence that their router was hosed, the ISP jerked me around for the entire day trying to prove I was wrong. They finally surrendered at around 6:00pm and actually looked at the router I’d told them was broken at 10:30am, and whaddaya know? It was hosed.

Then there was the other gigantic client who’s been trying to get a point-to-point line working with us for months. I’ve had the thing set up and ready to go for so long I don’t even remember when I did it, but I get pulled into conference calls every few days where they try to tell me I’ve either put in incorrect firewall rules, bad routes, or any number of other things I haven’t done. There was another one of those calls yesterday, and I watched another couple hours of my life drain away while I sat on a call with their firewall guys, their network guys, their QA guys, and a handful of random managers answering all the same questions yet again. At this point I should just have them dial into my voicemail, and tell them “Press 1 for me to tell you the routes are there again. Press 2 to hear me say the firewall holes are all open. Press 3 if you want me to beg you to check things out on your own damn network again before dragging me into another boring conference call.” Fortunately, this time someone with a clue was actually on the call, and whaddaya know? Their routing was hosed.

Just when I thought I was done, though, I got home and the potential new employer needed some firewall configuration done. I hopped on the boxes as soon as I got home and started poking through them to see how much work was going to be involved. About 15 minutes into flipping through the configurations, my sessions locked up. Then the pages started coming. Apparently, the box I was looking at had just crashed, and took their website down with it. I made a few phone calls, trying to explain that I knew what had crashed, and that even though I’d been on the box at the time, I hadn’t done anything to crash it. That’s not a position I like to be in — it just doesn’t look good. After that it took another hour to get things working again, and in the end it looks like their equipment is underpowered, and I’d just been lucky enough to be logged into the firewall at the same time someone else was bring up enough new equipment to cause a hiccup on the circuit that sent my box into a seizure. It took about an hour of phone calls, reboots, and crossed fingers to get everything working again.

You know, it’s days like yesterday that I start reconsidering the decision to become a computer guy instead of an English teacher. At least when you’ve got a pissed off teenager threatening to shoot you, you know exactly what you’re dealing with.