A Memorable IT Migration Experience
I was contacted by a construction company, and the person I spoke with was Greg, the sole IT person for the firm. My task was to assist him in migrating a physical server to a VMware virtual environment. Greg had learned about me from the company’s accountant, who had attended a holiday training session at my previous workplace (I later found out that the accountant was not entirely confident in Greg’s skills). In my assessment, Greg’s knowledge and experience were more on par with a general technician, yet he carried the responsibilities of a system administrator.
The company owned a powerful server dedicated to financial operations, and they wanted to migrate the existing operating system to a virtual machine and then run it there. Since Greg was not very familiar with Linux and VMware, he asked for my help in reinstalling the server. On a Saturday morning around 9 AM, we outlined a rough plan, and the work began. I suggested a physical-to-virtual (P2V) conversion, but Greg preferred to start fresh, reinstall the server, the accounting application, and then restore the database.
Pre-Migration Discussions and Decisions
Before starting, I insisted that Greg make another backup of the data to ensure we wouldn’t lose anything. He assured me that he had already backed up the data the previous night and showed me the tape. However, I still felt it was necessary to create an additional backup for safety. Greg, perhaps concerned about the cost, stubbornly refused.
Reluctantly, I proceeded to format the server, install a basic Debian system, and set up the VMware environment. Then, Greg started installing the virtual server and the accounting application.
The Unexpected Happened
When it came time to restore the data from the tape, disaster struck. We discovered that the tapes were almost empty, as if they had never been used. The reason became clear: Greg had labeled his five backup tapes for Monday through Friday and configured the backup software to write to the tape for the current day. Each day, he would insert the tape labeled for that day. However, the backup process ran at midnight, backing up the next day’s non-existent data. For nearly a year, none of the backups had been successful, and Greg had never checked the logs.
Severe Consequences
Just a week earlier, Greg’s boss and the accounting team had worked late into the night to finalize all the financial data and close the fiscal year. All their hard work was now lost, along with the rest of the year’s efforts. This incident not only caused significant damage to the company but also likely ended Greg’s career as a system administrator.
The Lesson Learned
This experience reiterates a fundamental principle in IT: never assume, always ensure absolute safety. Whenever a computer needs to be reinstalled, I always insist on a full image backup, even if the user assures me they have all the data. I can’t count the number of times, after we’ve started working, users have rushed to ask if it’s too late, and how relieved they are when I tell them, “We made a full backup just in case.”
This story serves as a reminder to every IT professional that no detail is too small to overlook, especially when dealing with critical business data. Backups are not just a technical requirement; they are a responsibility to the business.