System upgrades, whether we like them or not, are a necessary part of our technological world. In an effort to save costs however, many organizations are holding back from rolling out new technology. The downstream impact of this is that once an upgrade is finally approved, multiple complicated systems are upgraded all at once—sometimes involving hardware, software and other IT infrastructure items. “Might-as-well syndrome” is a good name for this, as in “As long as we’re upgrading our [insert your IT item here], we might as well add some [insert the next IT item your organization needs]. And while we are adding …, we might as well get some new …”
Does this scenario sound familiar? If so, you probably know that the bare-bones, already-harried IT staff are expected to heroically complete an already complicated upgrade and bring everything up to date at the same time. Multiple upgrades increase the risk of complicating the execution or lost user productivity because of delays—at worst is the loss of critical data during the upgrade.
Below is a case study of an organization upgrading too many IT systems and assets at once because they did not have the resources to keep up with regular IT maintenance projects. In dealing with a network infrastructure that had expanded only as needed and was lacking any documentation of configuration or settings, this IT department not only had to upgrade the foundations of the network, but it also demanded focus on completing the primary part of the project—upgrading the e-mail server.
A large service and product organization in the IT sector with multiple international offices took on an Exchange 2010 migration project for all of its regions. The migration scope was 20 offices, three network domains, and over 6,000 user mailboxes. The migration project included: a Blackberry Enterprise Server (BES), Outlook Web Access (OWA) services, and the introduction of Microsoft Forefront servers to the infrastructure.
The biggest roadblock to planning the project was a delayed Microsoft Windows Server Domain Active Directory (AD) consolidation project that had been in progress for almost a year. Due to the nature of the growing business, many of the satellite offices had set up their own domains with only a trust relationship with the domain of the company’s headquarters. Users in those satellite offices could not log on to the headquarters domain and access corporate resources, even when those users had been added with access rights to those resources.
Over the years, shortcuts were taken to get users access to business systems. Some of these quick-fixes increased the complexity of user accounts and even introduced IT security risks. In one example, a foreign office was constantly having access issues but rarely got enough of IT’s time to resolve the issues because of a time zone difference. The office manager was so frustrated that he took matters into his own hands and ordered a second Internet line to the office, setting up a connection with the company’s headquarters and remote controlling a computer in the sales group in order to access the organization’s CRM business system.
At great risk, the low encryption connection was going onto the Internet from the foreign office and re-entering the organization through an unused network port. How the office manager found this open port in the corporate firewall and who worked with him in the sales area has remained a mystery. When the organization’s IT security department found the unauthorized connection, it was shut down immediately and again, this satellite office was without business system resources and had to re-open the original job ticket with IT.
There were so many undocumented fixes like this to the network infrastructure that the IT engineer working on the Active Directory design wanted to start over and rebuild the systems from the ground up. Unfortunately, with the number of user accounts and dependencies the organization had within its network, a complete reconstruction was not an option.
The IT department started a systematic collection of all of the server systems in each office and began documenting an upgrade process that would complete the AD move and lay the foundation for the Microsoft Exchange 2010 migration. The impacts were bigger than originally thought and the entire project took 18 months—six months longer than was planned.
One of the more intricate migrations was for the BES system. There was not enough IT staff to perform the migration for the employee population, and as a solution, specific instructions were written for users to manually complete the mailbox update on their mobile phones.
If the process was not performed exactly as written, then the BES mailbox synchronization process would work in reverse and all of the user’s archived e-mail data would be lost. Inevitably, some users lost data because of not following the instructions. Thankfully, prior to the upgrade, the IT department’s business continuity plan accommodated for user error and multiple backups were completed at various stages in an effort to minimize data loss.
This project was long and tedious, and it required diligence and patience to work through all of the phases and milestones. The biggest complaint came from dozens of users whose accounts had been rigged to work within the domain by using duplicate accounts. Most of those complaints revolved around not being able to access the corporate calendar and scheduler. After an eventual domain migration, these users were down to a single account with marked improvement in speeding up their access times. A surprise result was the ease with which the satellite offices were merged into the new domain and Exchange server.
A great deal of acknowledgment should go to the organization’s project planners and IT experts in reaching out for expert advice to prepare for the project. Early on, they consulted Ontrack Data Recovery to determine the risks to their data before proceeding; however their services were never required due to the development of a concise business continuity plan after discussing their migration plans with Ontrack Data Recovery engineers. To assist in meeting the project’s disaster recovery plans, this organization used Ontrack PowerControls to quickly access the archived and/or back up tape media of Information Stores for those users who had not followed the BES migration instructions.