Broadband Service Providers: Using Network Outages and Change Management Strategies to Improve Customer Experience
Stephen Finn
September 19, 2023

The effects of unexpected and poorly communicated service interruptions
Seven years ago, when we started working with greenfield and early-stage growth telcos, we noticed an overlap in the engineering and operations role. They acted as one, as they were small teams with limited resources. Where is the issue with this? When you think of the functions of these specific areas, engineering is to build the network and often build it fast yet reliably. Operations are to maintain the network, with customer experience as the number one priority. Building fast usually takes precedence over the customer experience when the lines of these groups are blurred. This is no one's fault, simply the result of the need to move quickly and no one person or group being solely responsible for protecting the customer. An example of blurred responsibilities breaking down is during scheduled or emergency change management. Without having someone with the responsibility to push back on work that could harm the customer experience, there can be severe consequences for not only your customer but the individuals on your team who have to deal with the backlash that usually comes with unexpected (and poorly communicated) service interruptions.
As with many growing companies, there are limited human resources, and separating the engineering and operations roles may be challenging. Implementing a change management framework will keep the customer's experience front and center during planned and unexpected network changes, regardless of the size of your team. Change Management adds operational rigor and allows your team to take a second look before implementing. Think of the carpenter's motto, measure twice and cut once! This telecom best practice is a sign of a mature company regardless of the size of your team.
Before sharing some ideas to help you build your change management strategy, let's further break down the two scenarios of change management that we observe working with utility and municipality-owned regional communications service providers (CSPs).
Types of Change Management
Scheduled Change Management
Scheduled change management is the type of change management we want. It is (or should be) tied to our product Service Level Agreements (SLA) or customer SLA. These are often written within the RFP that you may have won from an enterprise organization or critical service.
Emergency Change Management
These are the ones we secretly wish we were on a deserted island on vacation when they happen. We can all attest that these are not fun events. It is when something has broken, and we must immediately make network changes. In this environment, we must create an outage more manageable than an unpredictable, more catastrophic one. In my first NOC job, we had a predictable memory leak in the router (Our NMS would fire off a threshold that would tell the NOC team to reboot the router manually). We did this every few days, early in the morning, for a month, which would cause all of our cable modems to go offline at 3 am instead of some random time when the router would unpredictably fail. Let's hope you don't have that level of change management!
Should you implement a change management strategy?
Three reasons to implement change management
You have enterprise customers who demand that you notify them of network outages affecting their services. Downtime is costly in lost revenue or, worst, life-threatening if your customers are suppliers of critical services such as Police, Hospitals, and Fire Stations.
The second reason is to provide structure to your team so that engineering, operations and sales are aware of changes in the network services, resulting in minimal disruptions and a better customer experience.
The third and most often overlooked is that it slows down the installation of new critical equipment, which is a good thing! We often build our networks, and in our haste to bring new customers online, we may negatively impact the current customers. We build so fast that we forget to peer review or see the damage that we may be causing customers.
Here are some suggestions to get you started if you are brand new to the process or looking to further enhance your current Change Management procedures.
Getting Started with Change Management
Define the ticket system to track the change management process. There are a plethora of systems available. We use Znuny (formerly OTRS) on the backend; it is an open-source system that provides excellent functionality, including the facilities to implement change management approval. The ticket system is secondary.
Define the roles and responsibilities. Who are the creators and managers of the engineering change management content? If you use roles, you should define them. I am often in conversations where people use titles and tiers, expecting that the others listening knows what they mean. Everyone in the room has their definition. For example, suppose you create a MOP (Method of Procedure) and a rollback. In that case, you need to define the role responsible for making it—for example, the junior network engineer, senior network engineer, or facilities person. Don't use vague functions like Tier 1 and Tier 2 because everyone has their definition of who is Tier 1/2/3.
Define the approval process. Change management needs to be peer-reviewed and signed off by someone responsible for managing the customer experience. For example, a junior engineer may need a senior engineer to review their work. This provides an excellent opportunity for feedback and, in a short period, will alleviate the senior network engineer of work. It also may be necessary for manager approval; I reserve manager approvals for Emergency Maintenance. See the definition of both Schedule and Emergency Maintenance above.
Define the communication process. I look to the NOC to manage the customer experience as they should have an operational mindset, but with the number one job of protecting the customer experience. If your organization is small, you may designate someone to wear the "NOC" hat. However, be careful to look at this as a junior employee or "Helpdesk." The NOC should be empowered to decline an outage that violates a predefined Service Level Agreement. This can be overridden by a CTO or other senior employee who can make a decision weighing both the technical and business requirements.
As a partner in developing, rolling out, and dispatching change management for our customers, we've seen how having even a few of these strategies in place can increase your team and customer confidence in your ability to effectively manage and mitigate associated risks with making changes. Collaboration and communication between different groups and departments are improved, resulting in more efficient and effective technology operations.
Sign up to receive more operations insights for communication service providers.
Similar stories
Here’s what we’ve been up to recently.
Get our stories delivered
From us to your inbox weekly.

