The Notepad is old and grody

Site Incident Report

Scribbled  · PDF · ◊Pollen source

Important notice: last Tuesday night all my sites went offline. In the interests of extreme transparency, I present this complete incident report and postmortem.

Impact

  1. All of my websites that still use Textpattern were broken from 11pm Jan 31 until about lunchtime the next day. (The Notepad does not use Textpattern and was mostly unaffected, except for #2 below.)
  2. All the traffic logged on all sites during that time was lost. So I have no record of how many people used my websites during the time that my websites were unusable.

Timeline

(All times are my time zone.)

  1. Tuesday Jan 31, late evening: I logged into my web server and and noticed a message about an updated version of my Ubuntu distribution being available. I was in a good mood and ran the do-release-upgrade command, even knowing it would probably cause problems. Because breaking your personal web server’s legs every once in a while is a good way to learn stuff. If I’d noticed that this “update” proposed to take my server from version 14.04 all the way to 16.04, I’d have said Hell no.
  2. In about half an hour the process was complete and sure enough, all my DB-driven sites were serving up ugly PHP errors.

Recovery

  1. Soon determined that my Apache config referred to non-existant PHP5 plugin. Installed PHP7 because why the hell not.
  2. More errors. The version of Textpattern I was using on all these sites doesn’t work with PHP7. Installed the latest version of Textpattern on one of the sites.
  3. Textpattern site still throwing errors because a few of my plugins didn’t like PHP7 either. Logged into the MySQL cli and manually disabled them in the database.
  4. Textpattern’s DB upgrade script kept failing because it doesn’t like something about my databases. I began the process of hand-editing each of the tables in one of the affected websites.
  5. Sometime around midnight my brother texted asking me to drive over and take him in to the emergency room. I judged it best to get over there in a hurry so I closed up my laptop and did that. His situation is a little dicey right now; it was possible that when I got there I’d find him bleeding or dying. That wasn’t it, thankfully. By four in the morning they had him stabilized and I was able to drive home.
  6. Morning of Feb 1st: I got out of bed at around eight on the morning of Feb 1st, made myself some coffee and emailed my boss to tell him I wouldn’t be in the office until nine-thirty.
  7. After driving in to work, I remembered almost all of my websites were still busted. I started to think about the ramifications. I wondered if anyone had noticed. I opened Twitter for the first time since before the election and closed it again, appalled.
  8. At lunchtime I drove to the coffee shop for some more caffeine and a sandwich. I remember it got up to 30º F that day so I almost didn’t need a coat. After I ate my sandwich I pulled out my laptop and resumed poking around the same database and trying to swap in all the mental state from before the hospital trip.
  9. Towards the end of my lunch hour I decided that this wasn’t fun anymore. Maybe I could poke this one database until Textpattern would stop whining about it, but there was still the matter of the broken plugins, and then I’d have to go through the same rigmarole for the other three sites.
  10. Sometime between noon and 1pm I logged into my DigitalOcean dashboard and clicked a button to restore the automatic backup from 18 hours ago. In two minutes it was done and all the sites were running normally.

Problems Encountered

  1. In-place OS upgrades across major releases will always break your stack
  2. Textpattern 4.5.7 doesn’t support PHP7
  3. Textpattern 4.6.0 needs a bunch of hacks to work with newer versions of MySQL
  4. Emergency rooms always have so much waiting time in between tests and stuff

Post-Recovery Followup Tasks

  1. Leave the goddamn server alone
  2. Revisit shelved projects that involve getting rid of Textpattern and MySQL.