Production Change Management
A Process Improvement Initiative
By Lynn C. Graves
click here for video presentation
Business Case
Problem Statement:
An Information Technology department supporting a large infrastructure, including hospitals, clinics and 1000’s of users with a generous mix of platforms had no formal change management process in place. This was a recipe for disaster!
With no leadership enforced change management process in place, each area had their own way of implementing change. Some areas had very detailed processes to include changes made in non-production environments and comprehensive supporting documentation. Others, as you might guess, not much, if any change management process was utilized.
Change Management –versus- Change Announcement:
The only change management uniformity that existed was the use of change announcements. Each approved change was posted to a common change calendar. A resource representing the proposed change would attend a weekly change meeting attended by all areas to review proposed changes for the week. The intended purpose of the meeting was to review all proposed changes, identify and vet any proposed change that might have negative or unintended consequences for the organization. This change management process was not always implemented for a number a reasons:
- The Change Management Meeting was not always attended by the appropriate resources.
- The change owner was not present to talk about the details of their proposed change.
- The change owner was not fully aware of the impact of their proposed change on other processes.
- The area representative of the change was quite often unaware of the proposed change details.
- The area representative of potentially affected system(s)/process(es) was not present to identify any potential negative impact.
- The change manager had no authority to decline a change request.
- There were unintended consequences once a change was implemented, due to inadequate or no testing.
With no way to formally enforce and monitor change – whether or not changes were posted to calendar or declined in the meeting -changes could still be implemented.
Drivers of Change
Documentation? You want documentation?
Due to the highly regulatory and critical nature of the organizations’ business, auditors were a frequent visitor. The lack of appropriate documentation put the organization at considerable risk. In addition, the IT environment was very dynamic and fast paced in part due to the many demands placed on the organization. To require adequate documentation would mean slowing down long enough to properly think through changes needed. A culture change was very difficult to affect. As stated earlier, no formal process for submitting a change had been established including a defined scope for any required documentation:
- Who is requesting this change?
- What is the justification for the proposed change?
- What is the risk of implementing or not implementing the proposed change?
- What is the impact to the organization of the proposed change?
- What are the specifications for building and testing the change, once approved?
- Has it been adequately tested?
- What was the testing outcome?
- What is the plan for implementation of the approved change?
- Has the change been communicated to all stakeholders?
In the initial discovery of what documentation was being collected by various entities within the department, we found that some groups were compiling records based on rigid guidelines set by their individual group(s) while others were not documenting anything useful.
If you can’t measure it, you can’t improve it!
This was no doubt a high-performing IT department that met all the common criteria of like IT organizations, to include providing an exceptional level of service and support. And not unlike other organizations of its kind, there was no unified methodology for managing change. Multiple tools were used to document changes:
- Microsoft® Excel Spreadsheets
- Microsoft® Word documents
- The legacy database
- Handwritten notes stored in a notebook
When attempting to gather metrics on how productive this department was, we researched:
- How many routine changes have been implemented in the last 12 months?
- How many high-risk changes have been implemented in the last 12 months?
- How many IT resources were consumed for the changes implemented?
- How many emergency changes have been implemented in the last 6 months?
- How much system uptime did they have?
- How much system downtime did they have by application?
Suffice it to say, there was no easy way to obtain this kind information, and no way at all to obtain it comprehensively. Further, if each area had been documenting necessary information, but not using the same methodology, providing information for the same set of questions, using the same set of guidelines and language, would these metrics really be meaningful?
Time to electrify the fence!
“What is often overlooked is that if one person can single-handedly save the ship, that one person can probably single-handedly sink the ship, too.” (The Visible Ops Handbook, 2007)
One word – cowboys. Not unlike other IT shops, this IT shop had its share of well-intended resources that would take it upon themselves to tackle a problem, not realizing the multi-threaded nature of the infrastructure and the importance of collaboration and communication. These “cowboys”, often brilliant in their area of expertise, are a very much needed resource, but without leadership enforced change management controls in place, can be dangerous to the organization. Executive Leadership, more than once, had to explain why a critical application went down unexpectedly or face harsh criticisms about the competency of their staff. It was time to electrify the fence!
A Unified Approach
A core group of resources within the IT department were convened to look at the problem and craft a solution. The resources represented in the group included:
- Executive Leadership
- Security
- Regulatory
- Internal Auditing
- Project Management
- Operations Management
- Infrastructure Support Management
This group researched current best practices from respected sources such as:
- Information Technology Infrastructure Library (ITIL),
- The Gartner Group,
- Evergreen Systems, Inc.,
- Microsoft Operations Framework, and
- Like IT organizations that were willing to share their policies.
Based on research, this group developed a standard that outlined the guidelines by which production change management should be implemented, monitored, controlled and reported. The standard change management process was vetted among Directors and Managers of all groups within the IT department, as well as with the Executive Leadership Team.
The Objective:
The primary objective of the Formal Change Management Process is to provide standardized methods and procedures to meet the production change management requirements supporting the organizations operations. Following these guidelines will ensure all information technology changes satisfy the Change Management Control Objectives. This will further ensure day-to-day IT functions performed to provide effective change management satisfy all regulatory and administrative requirements. In addition to meeting all of the requirements, this process will facilitate a method for efficient and prompt handling of all IT changes.
Key Goals:
- Establish clearly defined best practice change management processes and a common methodology.
- Improve efficiency through the use of automated tools and a centralized data repository.
- Improve change management communication through automated escalations and notifications.
- Ensure there are proper approval levels.
- Reduce the risks associated with completing changes.
- Reduce the impact of changes on the IT and business organizations.
- Establish and produce data metrics for executive review and audit reporting.
Challenges:
Change is difficult. Outside the expected challenges of invoking a new process and changing the culture, one major challenge encountered was incorporating workflow in an area that was very change intensive and already had a stellar change process. The problem faced was not using the same tool, (e.g., not answering same questions or using the same guidelines). The hope is to find a way to incorporate this group and convince them to use the new process.
Conclusion:
As of March, 2010 greater than half of total department has been trained and is using the new change management process. Next steps include:
- Resolve issues with the group using different tools.
- Engage internal auditing to provide a gauge on how we are performing from a documentation stand-point.
- Continue to refine and improve our change management process.
So far, the Change Advisory Board has been successful and meeting weekly to review all changes submitted, police changes submitted via the tool for things like documentation for building, testing and post-validation comments. This has facilitated making the larger Change Management meeting less painful, more productive and informative.
In short, the success of this entire effort has required:
- Standards and guidelines that can be generalized across the IT body
- Desire to do the right thing and the diligence to make it happen
- Leadership enforced controls and accountability
