I once encountered a small web development team that kept all its source code (Classic ASP) on a shared network drive. No version control. No organized backup. When someone was going to work on a file, he announced it verbally. If someone made a mistake that had to be rolled back, the sysadmin would grab the latest version of the ASP file from the production server. If the erroneous code had already been deployed, the sysadmin would have to find the last pristine version from the server backups.
Another development team coded against their live production database. If a new developer writes a buggy stored procedure? Go grab the backups and hope you can fix things!
Then there’s the time a small typo (one missing character in a script) obliterated all the data in a production data store.
In all these cases there was an emergency procedure to recover data but not enough of a safety net to protect against downtime and customer impact. These were all well established development shops with experience and viable software running in production.
It surprises me that more developers don’t crave safety nets. We claim laziness as a virtue when it comes to automating tasks but overlook its vice when protecting ourselves from, well, ourselves. Software developers as a population tend to be highly intelligent and competent. It’s mostly impossible to invent and control virtual environments without being highly intelligent. And so it confounds me when highly intelligent professionals make remarkably poor decisions and easily avoidable mistakes. Perhaps it is hubris that only wanes over time as the awkward postmortems and recovery crises tally up.
As professionals we owe it to our users and our stakeholders to establish safety nets. It’s part of what separates the professionals from the hobbyists.
Photo: https://flic.kr/p/nRvfME