Engineering Service Assurance for Telecom Providers

Gartner survey data shows that over 50% of communication service provider (CSP) CIOs expect to reach 70% automation level of operations by 2024. The main activities target business processes, along with business and network operations. Another key finding of Gartner research is that most enterprises lack a coherent network operations tool architecture strategy. This absence of a proper architectural approach perpetuates a vertically siloed view of tool acquisitions, and critically undermines network automation initiatives. Additionally, network automation initiatives can stall by improper tool architecture approaches – most commonly when enterprises design too many interdependencies on vendor network management tools instead of taking an abstraction layer approach.

Growing demand for automation in network and service assurance

With the increasing complexity of communication networks, expanded virtualization and cloudification, as well as limited human resources in the area of ​​network operations, the role of automated OSS systems is growing. Additionally, the growing need to analyze huge amounts of data (e.g. from IoT, 5G, FTTx elements) in real time puts enormous pressure on telecommunications networks and service management systems to deal with such high complexity and scale.

The benefits of deploying a centralized assurance system

• Increased operational efficiency – effectively manage services implemented on heterogeneous multi-vendor, multi-domain and multi-technology networks and delivery platforms serviced by partners.
• Faster response time – unified management of consolidated network and platforms reduces the time to respond to issues and resolves problems more efficiently.
• Reduced network down-times and outages – thanks to a single-pane of glass approach, network outages can be predicted more accurately, which helps reduce down-times.
• Lower costs – by automating repetitive tasks and ensuring they are resolved in an efficient manner; costs can be reduced while staff and resources are freed up to use in other valuable areas.
• Energy saving – leveraging automation in a network makes it possible to detect energy efficiencies and address them in a timely manner. As well, channeling energy to hardware only when it is needed achieves more efficient energy usage, without impacting performance.

Let’s look at a few real examples of how Software Mind supported leading CSPs to achieve significant improvements in their assurance domain.

Tailored made, cost-effective approach

For a Polish Tier-1 convergent (mobile, fixed and cable) operator we’ve delivered a tailor-made Umbrella Fault Management solution. The main drivers of such an approach were the need for specific, highly complex functionalities (including automation), as well as cost-efficiency and maintaining full control over the source code and its future development by third parties.

The project covered all phases, starting from HLD, through LLD, development and deployment with many integrations done towards the network (Huawei U2000, NCE, ENM Ericsson, OpenNMS) with CORBA, SNMP and HTTP based adapters and connectors.

Other technologies included Elastic Search, MongoDB, Apache Kafka MQ, Apache NIFI and Drools.

The main technical benefits of the deployed solution include:

• All alarms in one dashboard – by managing alarms in one tool, network and IT engineers can save time compared to situations with several unrelated monitoring tools
• An advanced correlation engine that enables the grouping of alarms by derived root cause
• An automation rules engine – that makes automatic actions possible (like executing a script on NMS/devices, acknowledging alarms etc.)
• Alarm enrichment with network topology and geographical structure views (with Google Maps and OpenStreetMap support)
• Advanced reporting engine – together with a notification engine used to synchronically populate vital information to operational teams and management
• Highly scalable integration layer-based on hexagonal concept and abstraction layer to unify protocol and alarm structure data, implemented in cloud-native, microservices-based architecture (Quarkus 3).

COTS reference project

Software Mind has been engaged with another Polish Tier-1, (mobile and fixed) CSP to modernize its OSS landscape by bringing the umbrella approach to the fault management domain and replacing an existing solution that went out-of-support.

The main functions of the system are:

• Aggregating, processing, correlating and storing fault management data (alarms and events) collected either from domain-specific network management systems or directly from telecommunication network devices,
• Providing users with a graphical user interface which allows them to perform various tasks on collected alarms (browsing, searching, filtering, editing, acknowledging and executing external actions) including integration with Trouble Ticketing system.
• Providing an API which allows external applications and automation tools to exchange and enrich alarm data.

The main challenges of the project were multivendor and multi-technology environments (both legacy and virtualized) that had been deployed over many years of operations, along with the need to introduce the required automation via a web-based single pane of glass (360° view) portal, all in a high-availability architecture.

Software Mind decided to cooperate on the project with UMBOSS and selected their award-winning product portfolio of Network and Service Assurance.

The solution processes all incoming events and alarms according to configurable sets of rules that govern how the alarms are filtered, correlated, enriched with external data, and presented to the user. The administrator can define a separate set of alarm processing rules for each alarm source (network management system instance, SNMP trap receiver, etc.).

Alarm deduplication, correlation, and suppression mechanisms will be applied to raw alarms based on the requirements provided by the customer. These mechanisms can significantly reduce the number of raw active alarms. Deduplication, correlation, and suppression criteria are fully customizable by an administrator.

The solution includes predefined rule sets for a large number of network management systems and devices out-of-the-box. All rule sets not included out-of-the-box will be created during the implementation phase of the project. Standard IETF MIBs such as IF-MIB, BRIDGE-MIB, SNMPv2-MIB, as well as MIBs of major vendors are supported out of the box.

A user with administrative privileges can manage the majority of rules from the FM administration, web-based application. An interactive graphical query builder supports manipulation with alarm parameters using different operators such as AND, OR logical operators, string concatenation, numerical and string comparison, etc.

Alarms are enriched using reference information about the network resources and locations. Network resource information is stored in the database and periodically synchronized with external network resource inventory systems using a discovery module. The alarm processing engine produces a detailed, timestamped log of all incoming events and alarms, and their processing details, including raw values of all fields, for troubleshooting purposes.

The main benefits of the deployed solution include:

• All alarms in one dashboard – By managing alarms in one tool, network and IT engineers manage can save time compared to situations with several unrelated monitoring tools.
• Faults localized and isolated – Enrichment enables fast automatic fault localization, while RCA and cross-domain correlation ensure rapid fault isolation and swift response to any issue.
• Unified alarm management – Fault management administration, as well as management of alarms, is done through a unified user interface.
• Visibility across domains – The umbrella approach combines alarms from various networks and platforms, making it seamless to detect issues across multiple domains.
• Visibility of impacted customers and services – Alarms are automatically associated with customers and their services in case the affected devices or functions are associated with them.

FTTH monitoring

Our client, a Polish subsidiary of one of the world’s largest telecommunications corporations, is a significant FTTH Internet Service Provider, who was dealing with end customers’ complaints about bandwidth and streaming quality and needed to gather the correct data to resolve network stability issues.

The Software Mind team designed and developed a unique and flexible FTTH monitoring system – three customized probes capable of running crucial performance, stability and diagnostics tests (G1/G3/XG). One of the three probes was a reliable and secure Raspberry Pi-based embedded system – tailor-made hardware assembled by our team. Independent hardware probs are necessary to make the test results independent of the hardware platform and operating system used by the user or when testing a co-located node for multiple clients.

The designed probes’ key functionalities and use cases are: commissioning tests for telecommunications partners, assuring quality tests of services provided to end customers, monitoring key elements of the operator’s network continuously and gathering the correct data to resolve network stability issues. Probes are a crucial element of a rBSA (revers-Bitstream Access) model, where a carrier uses a third-party operator FTTH access network.

The core system was implemented in a cloud environment (Kubernetes) with open-source tools (Grafana, Ansible AWX, Rabbit MQ etc.). The results are aggregated and visualized in a consolidated web-based GUI allowing operational teams to quickly check the status of the network and diagnose problems in a more efficient way. It also allows for the probe’s configuration management and provisioning for a specific use-case, their stock-management, as well as advanced reporting.

Consequently, the probes safeguard bandwidth quality and help to achieve an upgraded FTTH network stability, better bandwidth and upgraded streaming quality enabled by gathering the necessary data during testing. With an improved capacity to test network stability in users’ households, Internet Service Providers can increase clients’ satisfaction.

Stay tuned for an upcoming article about leveraging AI in Network Operations.

The telecommunications industry trusts Software Mind

Software Mind expertise supports several assurance domains and helps clients from the consultancy & design phase (to avoid common mistakes mentioned above), through the deployment either with customized software development of a specific solution or by the introduction of the COTS product from one of our partners (UMBOSS, Splunk etc.), to optimization and tuning of the deployed systems and its support and maintenance.

If you’re interested in hearing how Software Mind can support your business growth, use this contact form to get in touch with one of our experts.