It started with a phone call from Tony Diaz to inform me that the server was down. “Great” I thought, “…another DSL outage, or program crash. My server has been up for months, so it probably just needs a reset”. That’s what I was telling myself as I headed downstairs to my home office. My rusty but trusty server had been rock-solid reliable for over a year.
I could hear the beeping sound before I opened the door. “That doesn’t sound good”, I said to myself. I was wondering if it was the UPS or the server? As it turned out, the beeping was coming from the RAID subsystem; it had failed! More than that, it failed because someone or something had exploited an unknown security hole and trashed most of the contents of the site. The damage to the data on the server was pretty extensive.
How did they get in? It might have been a security flaw in the host operating system, or an exploit in the blog software I was running (a few versions out of date, I’ll admit). It could have been one of the plug-ins or a vulnerability in the PHP code. Maybe my security just sucked. I couldn’t tell exactly what had happened, but in the end, the box was seriously pooched; in Monty Python terms it was dead, deceased… it was an ex-parrot.
Fortunately I was able to recover quite a bit of the deleted data and I did have most of the site backed up. What wasn’t backed up recently, Tony Diaz was quickly able to salvage. A2Central wasn’t going to be a total loss as I had initially feared. Tony offered to help me get back online and I gratefully accepted.
We divided up tasks; Tony took the web site, I took the BBS software/server and we got busy rebuilding and restoring. We decided to move the main web site to a hosting service (something I was already in the process of doing anyway) but the BBS would stay where it was. It became Project: A2-deCentralized.
Within days, people became concerned with the sudden disappearance of the A2Central web site. I received a few inquiries, and upon relating my site woes, I received many generous offers of help and sympathy from the Apple II Community. Bill Martens of Call-A.P.P.L.E. offered to host us temporarily (thanks Bill!) and others offered to help to rebuild our data. Tony and I were already well into our restoration effort by then and could see our efforts paying off, but it was reassuring to know I had so many friends willing to lend a hand. Thanks everyone!
My RAID problems were all soft errors, so the server was functionally intact and thus I began the process of starting over… wipe, install and patch (repeat) for the operating system and BBS applications. Tony, among his many other talents, is a machine when it comes to (re)building web sites. Before I knew it, he had the database and presentation of the site almost back to normal. A little testing, a lot of tweaking (some still ongoing) and presto, A2Central is back.
This post wasn’t intended to be a lecture, or to remind readers of the importance in taking security precautions, keeping software current or maintaining good backups. Everyone already knows all about that stuff (right? good!). I just wanted to let everyone know what happened to A2Central.
I’m not posting to whine about getting hacked either. When I realized the site had been compromised, I didn’t get mad… I just felt a little frustrated. All along, I’ve assumed it was only a matter of time before something like this might happen. Keeping up with security is difficult, and this wasn’t the first time the site had been attacked; splogbots, spambots, link-spam… it’s happening all the time. The scriptkiddies, bots and scammers are probing away at A2Central (and all the other web sites the world over), looking for an opening to vandalize, add sites to their zombie nets, pimp dubious snake oil or compromise privacy. I say screw ‘em. We’re moving forward despite their shenanigans.
And so that brings us to the nickel tour on the changes we’ve made around here. Most services are restored and we’ve made a few DNS changes to make things easier (for us, hopefully for you too). Please make note of of the following:
Both of these resolve to the site you’re reading now. The old /portal kludge is being phased out but a permanent redirect is in place to minimize disruptions and keep broken links to a minimum. So, while changing any old links would be nice, it’s not necessary. New link references should be made directly, as they appear now.
This resolves to the BBS web interface of A2Central. Previously that site was running on a secondary web server answering on port 8000. That secondary server has been sacked.
The file library is online BUT be advised that I plan to overhaul and expand it. I’ve never liked how I had it organized. Anonymous logins are still allowed, so don’t have feel like you have to join the BBS to download anything. What’s ours is yours etc.
IRC is open for chat, as usual on port 6667.
If you used A2Central for e-mail, your current settings are broken. You need to use mail.a2central.com for pop3 and smtp access.
If you used A2Central for usenet, again, your current settings are broken. Use nntp.a2central.com or news.a2central.com instead.
There you have it. We’re still busy fixing a few things but all the major kinks have pretty much smoothed out by now. I want to thank everyone for their patience while we got back on our feet. I also want to extend my personal gratitude to Tony Diaz for jumping in without a second thought to help me get A2Central online.