These roles are crucial to ensure the smooth operation and development of telecommunication networks and services by fulfilling network design and architecture, network monitoring and maintenance, customer support, capacity planning, quality assurance, emergency response, prototyping and testing, research and development, implementation of new technologies, and more.

All these areas require reliable data and thus robust voice monitoring is becoming the industry’s linchpin, ensuring seamless performance and meeting user expectations. It’s clear that the industry needs to vigilantly monitor VoIP services, with the understanding that issues in this sphere can impact the bottom line. 

With this growing need for visibility, I decided to ask several leading telecom professionals— extremely experienced ones, people whose opinions I highly value—for their insights on  effective VoIP monitoring. Here’s what they had to say (plus my personal two cents at the end). 

Khalil Ayman Badwan, Project Engineer/Digital Platforms & Customer Experience

Khalil is an experienced solution/system engineer working in service delivery, project delivery, 3G, 4G, 5G and cloud computing, and AI. He has been working on network intelligence and network analytics since early 2016, with a rich experience in network protocols. This is what he chose to share with us:

From what I’ve seen, it’s super important for any service provider to have and collect insights about their network performance. Having said that, bear in mind that traditional Network Management (NM) tools are very limited when it comes to providing such details.

In terms of data you have around call quality, it should allow you to get to the root cause of problems. It’s essential to collect the exact metrics and counters that describe the issue and the reason behind it. This capability allows the team to identify both the issue and the solution. Unfortunately, ‘generic’ voice monitoring solutions are simply not enough; it can’t be enough to have insight on each aspect or case faced by the subscriber and internally in the network.

Many industry players are switching their voice services to the cloud. It’s very important to have cloud compatibility since cloud computing is the core technology for modern network architecture nowadays. Having such a step will accommodate various options and capabilities to expand our services. Whether it’s on the cloud or a classic physical network, the most important characteristic of a network is to be visible and monitored at all times. 

The most important voice service KPIs to keep track of, in my opinion, are call drop ratio, call setup ratio, Mean Opinion Score (MOS), packet drop, packet loss, and jitter.

Silent calls/one-way-audio is definitely one of the problems my colleagues and I have to deal with the most. In my opinion, the ideal setup is to have a functioning two-directional channel. The second best action is to terminate the call and let the call be re-established between the customers again.

When it comes to VoIP signaling and media monitoring, it’s crucial to care about both and not only one of the two. Let’s assume that your media plane is a bus carrying passengers (which is your VoIP subscriber media) and the signaling plane is the route that the bus is supposed to take to deliver those passengers so both are very critical. The quality of the bus alone is not sufficient if you can’t ensure the quality of the road.

I’m often asked what kind of voice quality information I should expect to get from a voice monitoring solution. My answer to this question is very simple. Make sure it offers:

• An end-to-end analysis capability as well as packet status (transport, loss, delivered, etc.) 
• Call information (termination, release cause, drop rate, setup time, etc.)

Finally, when you’re looking for a voice monitoring solution, these are the most important questions you need to ask the different vendors to help you make a decision:

• How would I know exactly what the actual customer experience is (as a real value) using your solution?
• How would you help me maximize and increase my customers’ experience? 
• How can we prevent issues and problems from happening in the first place, using your tool? 

Jan Holub, Professor, Head of Department of Measurement at Faculty of Electrical Engineering, Czech Technical University

Prof. Holub’s research interests span various aspects of Measurement Technology, with a focus on subjective quality, intelligibility, and usability testing. He has been an active member of several professional organizations and has chaired organizing and program committees for numerous conferences. In addition to his research pursuits, he has shared his knowledge and expertise through teaching various courses at CTU. He shared the following with us:

While there are many angles to VoIP monitoring I can discuss and hopefully give a piece of useful advice on, I chose to focus my two cents for the telecom community on the role I see for monitoring data in future AI/ML-driven service automation.

I see its potential in multiple domains, primarily in predictive maintenance, dynamic resource allocation, and network security, but also in single-user-tailored QoE.

AI and ML algorithms can analyze RTP data to assess the quality of voice and video streams in real time. By continuously monitoring RTP metrics like jitter, packet loss, and latency, AI can detect performance issues and adjust network configurations or routing to optimize QoS. Also, AI can use historical RTP data to predict when network or service issues are likely to occur. By identifying patterns in RTP metrics that precede high-load periods or even service disruptions, AI can dynamically allocate resources like bandwidth or processors, optimize processes like voice codec or video compression, or even trigger preventive measures or automated maintenance tasks to avoid or at least minimize downtimes. When RTP issues occur, AI-driven systems can automatically perform root cause analysis. By correlating RTP metrics with network configurations and other data sources, AI can rapidly identify the source of problems and suggest corrective actions. AI-based systems that detect anomalous traffic patterns or security threats by monitoring RTP data are already in place. These systems already not only generate alarms but directly respond with security measures.

One could say that all these applications were already available in existing monitoring systems, but the major benefit of AI/ML deployment is opening the possibility of continuous improvement and automated self-learning by using RTP data to identify areas for enhancement in real-time communication services, leading to ongoing service optimization.

Another fruitful AI application is perhaps user behavior and preferences analysis based on tracked user interactions and/or sentiment analysis from audio and video data where patterns acquired from RTP monitoring play an essential role. Also here AI can provide personalized recommendations or features to enhance the user experience.

Liju Mathew, Director—Solutions & Systems

With over 15 years of professional experience in the telecom industry, Liju has a robust background spanning various sectors, including manufacturing, software development, and telecom service provision in 2G, 3G, LTE, and NGN technologies. His expertise encompass a deep understanding of signaling, switching, network planning, billing systems, revenue assurance, service provisioning, network management, service costing, OSS/BSS, risk assessment, and mitigation. Here is what he told us:

It is critical to have the ability to generate data-driven insights through your voice monitoring solution. Here’s why: Voice over 4G and 5G technologies will remain one of the main services offered to the customers and it is critical and very important to be able to monitor and measure the technical parameters that influence and impact the setup, call quality, speech impairments like noise, silence, one-way call, etc. Effective monitoring of the signaling and RTP media is required to ensure top-quality voice services demanded by your customers.

It is also very important to make sure the data you have around call quality allows you to get to the root cause of problems. The first thing is to detect that there is poor quality in voice service experienced by the customers. With effective monitoring of different legs of the network (access, core) and with proper knowledge of the network architecture and flow of the traffic within the network it is possible to determine the source of quality issues and conclude the root cause analysis.

The most important voice service KPIs I highly recommend you keep track of are latency and packet loss potentially resulting in MOS value degradation, one way speech, and silent calls, as those are the ones that are impacting your customers’ experience the most.

Unfortunately, with Voice over IP networks in 4G and 5G mobile networks, the generic signalling-based voice monitoring solutions are not enough and it is absolutely necessary to invest in media quality monitoring solutions.

Livio Pogliano, Head of Roaming & Carrier Operations

With 28 years of experience within global telecommunications companies, Livio is a highly skilled and accomplished executive known for pioneering new technologies, conceptualizing innovative services, and advocating continuous efficiency enhancements. A proactive change agent, he specializes in implementing innovative strategies that optimize resource allocation while upholding cutting-edge quality benchmarks. According to him:

In a world where IP-protocol is driving many types of communications from data to voice, it becomes very important to have a clear and detailed description of the service perceived by the customer, and it has to be possible to do that in real-time. Service monitoring moves to the next level where it is not enough to verify that network nodes are up and running, but multiple sets of KPIs can provide the real description and potentially create “service alarms.”

A wider and open approach is required to create KPIs and to potentially detect subjective effects, such as voice call quality, in an objective way. Great effort is required to have a consistent and accepted-by-standards Mean Opinion Score algorithm to verify that traffic quality can reach good levels for the human ears. It means that working on media contents, and not just IP signaling, is needed, taking always into account privacy constraints.

Moreover, migrating the infrastructure into the Cloud increases the focus on service performance for Operations teams, driving engineers to a new learning phase in their careers.

At last, all these service evolutions and new requests are highly demanding when it comes to computation and analytical algorithms: a strong knowledge of the service implementation and technical architecture is needed to create the right and customizable tools.

This future-oriented and high expertise is really the key factor for the definition and choice of providers for voice-over-IP service measurement systems.

Markus Monka, Hacking telco since 2003

Our long-time customer and friend Markus from sipgate GmbH told us this:

Generating data-driven insights through a voice monitoring solution is paramount to our continuous improvement in the telecommunications industry. These insights allow us to spot trends, pinpoint quality issues, and proactively enhance customer satisfaction. This data-driven approach is pivotal to our service optimization.

Our ability to track problems down to their root causes is made possible by the wealth of call-quality data at our disposal. It’s not just about addressing surface-level issues; it’s about eradicating underlying causes for sustainable service quality and minimal disruptions.

As more industry players migrate their voice services to the cloud, careful consideration is required. Robust voice monitoring becomes essential to ensure the performance of cloud-hosted services meets expectations.

In my opinion, key voice service KPIs to monitor include call quality, delay, jitter, packet loss, and call drop rates. These metrics are crucial for ensuring smooth communication and customer satisfaction.

Both VoIP signaling and media monitoring deserve attention as problems at either level can affect service quality. A comprehensive approach that considers both aspects is advisable.

Silent calls and one-way audio are serious issues affecting customer satisfaction. Vigilant monitoring and testing are crucial for early detection and swift resolution of such problems to maintain uninterrupted communication.

From a Voice Monitoring solution, we expect detailed information on call quality, including MOS scores, packet loss, jitter, and latency. This data is instrumental in our continuous quality enhancement efforts.

Michael Wallbaum (that’s me), Director of Product Marketing

What can I add to the insights of the seasoned telco professionals who contributed to this article? In 1999 I was part of a research team that demonstrated voice and data transmission over an HSCSD mobile data connection at the Telecom ‘99 in Geneva. Back then, voice over IP was new and exciting, but also flakey and we were happy for every demo that went without a glitch. A long time has passed since and the telco industry has now almost completed its transition to IP. Any form of “glitch” is no longer acceptable as we all rely on phone services for our daily lives. Still, everyone has experienced the inability to set up a call, call drops, silent calls, or simply bad voice quality. How can this happen and why does it often take so long to fix? 

Having worked with many different CSPs I would argue that the main reason for today’s “glitches” is lack of information in many different forms. It’s not that there is a lack of data—quite the opposite. There are, however, issues with the data itself and how it’s organized and processed. 

First of all, while CSPs have tons of data, it is often inadequate. Traditionally, there is a strong focus on signaling performance monitoring, but the in-call user experience is frequently neglected. Still, every monitoring solution vendor and CSP will claim that “MOS is available,” but few actually understand (or admit) that this media quality information is not accurate, because it is not obtained from direct measurements of the RTP streams. One symptom of this data quality issue, is that only in 2019 ETSI proposed a set of timeslice media quality KPIs in TR 103 639 to complement well-known signaling KPIs. Only now is it possible to calculate media quality for calls, trunks, routes and other aggregates in a meaningful and standardized way.

Secondly, even if information on the voice service performance is accurate, comprehensive and reliable, all too often relevant data is not accessible to all stakeholders. Data is still stored in silos, defined either by the licensing and export policies of vendors or by organizational boundaries. Typically, the larger the CSP the more fragmented the data space. Identifying the stakeholders and putting all the puzzle pieces together can be cumbersome and time-consuming, which greatly adds to the stress of incident management. Exporting all available data into a so-called data lake is the recommended approach to break down the silos, but I have yet to see more than measly data ponds.

Lastly, even when all the data is accessible, relevant and accurate, there is a lack of tools and best practices helping to make sense of the vast amounts of data, in addition to a lack of human resources caused by low margins and stiff competition in the telecommunications market. Artificial intelligence and machine learning technology are promising to automate the process of filtering, correlating and analyzing the available data to speed up root cause analysis and other operational tasks. The holy grail is zero-touch automation (ZTO), i.e., fully automated voice service management. However, without reliable data from many different sources ZTO remains a pretty bold vision.

To wrap it up: the industry needs to focus on the quality and accessibility of data, to enable more automation leading to better service quality. 

Final Words

In the ever-evolving landscape of telecommunications, industry leaders have shared invaluable insights concerning the complexities, necessities, and benefits of effective VoIP monitoring. These professionals, with years of experience under their belts, emphasize the importance of generating data-driven insights.

As professionals deeply entrenched in the industry, we now all recognize that voice monitoring goes beyond the conventional checks of network node functionality; it should delve into the intricate details of the service quality experienced by our users, examining issues in both VoIP signaling and media quality. Generating insights through comprehensive data analysis is not merely a best practice; it’s the cornerstone of service optimization strategies. 

We should meticulously track an array of key performance indicators, from call quality and delay to jitter, packet loss, and call drop rates. By deciphering this data, we not only address surface-level issues but also uncover the root causes, ensuring sustained service quality and enhancing customer satisfaction. On top of that we can lay the foundation for trustworthy service automation technologies.

Through continuous monitoring efforts, we can swiftly identify and resolve challenges such as silent calls and one-way audio, maintaining uninterrupted communication for our users. Voice monitoring solutions available in the market should all provide us with detailed insights, including MOS scores, packet loss, jitter, and latency, enabling us to drive continuous quality enhancement initiatives. 

In this transformative landscape, our commitment to accurate monitoring and proactive resolution remains unwavering, ensuring superior user experiences and driving the future of telecommunications.

Trying to track the call path through multiple networks, we uncovered something truly unexpected. As we followed the packets from the German operator, through a wholesale operator’s network to the subscribers’ Swedish home network and all the way back to Germany, we were startled to see our first international end-to-end VoIP call. In a twist of fate, the RTP packets never arrived at their destination. The culprit? The packets’ hop count was set too low for the planned journey, leading to the abrupt termination of communication at the wholesale carrier. Starting an 80 hop journey with a maximum hop count of 64 is like trying to drive from Berlin to Stockholm on a single charge of your car’s battery. You just won’t make it.

Now, why does this tale matter? For one, it defied the old norms of international calls, highlighting the unpredictability lurking within global networks. More importantly, it underscored the need for detailed and relevant data on RTP streams with unparalleled precision. Telecom professionals, aware of the havoc that silent calls wreak, understand the importance of such insights.

This gripping story encapsulates the essence of why your voice monitoring solution must offer comprehensive, data-driven insights. ‘Generic’ solutions fall short, unable to unveil the issues faced by subscribers and service providers alike.

In the upcoming article, I’ll unravel the limitations of conventional tools like Wireshark and Homer, paving the way to a carrier-grade solution. Let’s go on a journey into a world where every voice is heard, and mysteries are unraveled.

Steering Clear of Pitfalls: The Reality of Voice Monitoring

Ever found yourself in this scenario? “Our voice monitoring tool shows we’re good, but customers keep complaining about voice quality.” If you’ve been there, you’re not alone. We’ve had numerous CSPs reach out with similar frustrations. But let me be clear: in cases like these, ignorance is certainly not bliss.

Let’s dive into some pitfalls you need to be aware of and actively avoid, drawn from real-life cases that could be happening on your network right now.

Core Network Issue

A leading Israeli CSP faced connectivity problems between central and remote sites. While standard tools hinted at issues, it took in-depth RTP monitoring to reveal the root cause—a broken glass fiber cable. Imagine the productivity losses and customer dissatisfaction before this was discovered!

One-Way Voice 

Consider a medium-sized enterprise plagued by one-way voice troubles for months. The cause? “Zombie Streams” generated by a bug in their IP-PBX:

• RTP streams of previous calls sometimes just don‘t stop
• These “Zombie Streams” contain silence and persist for weeks and months (until the reboot of the IP-PBX)
• The receiver picks the stream that arrives first
• Packets of the other stream are discarded

These lingering streams caused chaos, with callers unable to hear the other side. Think about the impact on client relationships and operational costs.

Ghost Voices

Customers startled by unknown voices in their conversations—a nightmare scenario, right? This perplexing issue remained unresolved until packet recording exposed Cisco MGWs as the culprits. The vendor’s response? A nonchalant acknowledgment of a known bug. Imagine the frustration of both customers and the CSP involved.

Now, these examples barely scratch the surface. Countless other incidents plague networks daily, causing businesses significant harm. The sad reality is that most standard tools fall short of addressing such complexities. They merely skim the surface, focusing on RTCP while ignoring the actual RTP.

Let me be frank—many CSPs unknowingly invest in near-useless tools, wasting valuable time attempting to identify issues, all while frustrating their customers and hurting their bottom line.

Inadequate voice monitoring isn’t just a technical glitch; it’s a direct blow to your business. It hinders productivity, increases operational costs, and erodes customer trust. It’s time to shift gears and invest in solutions that dig deeper, tools that unravel the intricate layers of voice communication, ensuring seamless experiences for your customers and safeguarding your business’s reputation.

The Quest for Something Better: Elevating Voice Monitoring to New Heights

If you got to this point of the article, you’re keenly aware that ensuring top-notch voice service isn’t just a challenge; it’s a competitive necessity. Call quality, encompassing seamless service, pristine audio, and a lack of disruptions, stands as a pivotal competitive advantage. In today’s crowded market, customer satisfaction is the ultimate differentiator. If they’re not content, they’ll swiftly move on to a provider who delivers.

Now, the question lingers: have you considered a solution that comprehensively covers every facet of voice monitoring? It’s not just about availability; it’s about delving deep into the realms of call management and in-call quality. Imagine having access to precise data on any standards-based voice service, empowering you to ensure seamless operations through in-depth analysis of SIP and RTP traffic. Picture auto-detection of impairments like dropped and silent calls, enabling efficient performance optimization.

So, what should this ideal solution embody? Here are some fundamental characteristics:

• Dual visibility: It seamlessly merges cutting-edge media analytics with advanced signaling monitoring capabilities, fostering highly effective network operations at a minimal total cost of ownership (I’ll delve deeper into this in a minute!).
• Versatility: Imagine having a unified view of your entire voice network and service quality, facilitating applications from root cause analysis to comprehensive monitoring and reporting. Compatibility with all standard VoIP services, including VoLTE, VoWifi, IMS, IPX, NGN, and enterprise services, is non-negotiable.

• Accurate data: Relying on its own measurements, independent of potentially unreliable data from third-party systems, it provides a reliable foundation for strategic decisions.

• Network segmentation: Swiftly isolating network elements that are causing impairments, accelerates troubleshooting. It also enables the creation of customized statistics for individual customers, interconnection points, and network elements.

• Automated analysis: Root cause indicators are automatically identified, drastically reducing troubleshooting efforts and enhancing operational efficiency.

• KPIs for media analytics: Deep insights into actual RTP transmission quality and user experience provide invaluable data for continuous improvement.

• Scalability: A solution that’s not only scalable but also robust, ensures that your capabilities grow with your business demands.

Sounds hard to find? Couldn’t agree with you more. In fact, this is exactly why Voipfuture was established. Siemens engineers discovered that there were no tools to automatically detect RTP issues, so they set off to create their own tool. But that’s a story for another article. 

Adding One-to-One Results in a Value Greater Than Two

Ensuring crystal-clear calls isn’t just a priority—it’s a necessity. But here’s the catch: I’m not just talking about the standard fare of checking call establishment; I’m delving deep into a realm where call connectivity meets in-call quality (just as I promised in the previous section). Enter the concept of dual visibility—a game-changer in the world of voice services.

Traditionally, VoIP analytics often fixate on the control plane, focusing primarily on SIP signaling and call connectivity. It’s like ordering a product online—sure, the order is placed, but does the product meet expectations? Dual visibility, which encompasses both SIP signaling and RTP media control, paints the full picture of VoIP.

So, what exactly is dual visibility all about? Close one eye. You can still see your surroundings, but you have a reduced scope and everything appears “flat”. Now open both eyes. Not only do you now have a wider field of view, but you also get a sense of depth. With dual visibility on SIP and RTP, similar magic happens.

Here’s why dual visibility matters.

1. Understanding the evolution: In the realm of telephony, history tells a story of circuit switching and its inherent simplicity. A physical connection ensured audio transmission. Enter IP technology, and suddenly, data is split into packets, paving the way for more complexity. Monitoring each 20ms audio packet separately is a challenge. This complexity created the need for monitoring RTP media transmission. However, the industry’s focus on circuit-switching legacies led to a gap in media monitoring.

2. SIP signaling: SIP signaling, governed by Session Initiation Protocol, forms the backbone of call establishment in VoIP. It connects all parties over the internet, streamlining the call setup process. Monitoring SIP signaling provides critical insights such as call setup time and post-dial delay, ensuring successful call establishment and simplifying troubleshooting. When talking about SIP signaling, I feel it’s the right place and time to share with you the Phoneburner SIP story, where they wondered about failure to set up new calls during peak hours. In the end, they found that they didn’t buy enough channels from their service provider. While the traditional monitoring tools they used, such as Homer and Wireshark, were not able to explain their call quality issues, the cut-off effect was clearly visible in Qrystal, a carrier-grade advanced solution.

3. RTP media control: While SIP ensures connections, Real-time Transport Protocol (RTP) transports audio over the internet. RTP media monitoring becomes pivotal in scenarios like silent calls, where the call is established, but audio is lost in transit. Dual visibility allows us to pinpoint such issues, understand where data is lost, and resolve problems efficiently.

4. The power of dual visibility: Dual visibility isn’t just about seeing both planes—it’s about understanding the interplay between them. While SIP signaling offers a broader context, RTP media control zooms in on specifics. Combining these perspectives equips us with a holistic understanding of the voice services we offer and the end-user experience. It’s not merely the sum of its parts; it’s an amplified understanding that simplifies issue detection and resolution.

Click here to uncover the full story of dual visibility and revolutionize your understanding of VoIP monitoring.

Before We Part: The Eureka Moment

Amidst the various insights explored in this article, there’s one that stands out—the pivotal role of superior voice quality for communication providers and their clientele. It’s not just a facet; it’s the very essence of seamless communication and customer loyalty.

In a landscape brimming with diverse voice monitoring tools, the ability to distinguish and ask the right questions becomes imperative. Not all solutions were created equal. This article underscores the necessity of advanced tools, a point vividly illustrated through real-life examples. The emphasis isn’t merely on sophistication, but on efficacy.

However, a solution is not just about its technical capacity. It’s also about simplicity, flexibility and scalability.

Crucially, voice monitoring solutions don’t just resonate technically; they directly impact the balance sheet. The return on investment materializes swiftly, often within a year. Moreover, the CAPEX remains low, thanks to standard carrier-grade server hardware or virtual machines, ushering in an era of cost-effective and operationally efficient voice monitoring. It’s not just about hearing; it’s about listening to the pulse of business, resonating with savings, efficacy, and unparalleled customer satisfaction.

Insufficient voice monitoring transcends mere technical glitches; it can be a direct blow to your business. Its repercussions extend to hindering productivity, escalating operational costs, and corroding customer trust. Achieving top-notch voice service performance isn’t just a daunting challenge; it stands as a fundamental competitive necessity.

In this article, I will delve into real-life examples and case studies that illuminate instances of voice quality issues, how to uncover the root causes and facilitate quicker resolutions and  preventive measures. These cases serve as compelling illustrations of how the right data can streamline the troubleshooting process, resulting in significant time and effort savings. In some instances, the time required for resolution has been reduced by an impressive 80%, a figure that might even underestimate the potential efficiency gains.

By leveraging insightful data, businesses can rectify current problems more expediently as well as fortify their systems against future occurrences. This strategic approach not only enhances operational efficiency but also fosters a more resilient and reliable voice service infrastructure, ultimately contributing to sustained customer satisfaction and loyalty.

Case #1: My Monitoring Tool Says Everything Is Fine…

Let’s kick off with a classic scenario from the early days of our company. We received a call from someone requesting a Proof of Concept (POC), and the game-changing line in that conversation was, “My monitoring tool insists everything is fine, so why are my customers still grumbling?”

Now, take a peek at the image below – the existing voice monitoring system flashed green for good quality in both directions of this call example. Yet, the complaints persisted.

The data is the same in one direction, but different in the other. Qrystal didn’t sugarcoat it. Instead of the expected 4.2, it showed a dismal minimum MOS of just 1.42. That’s a massive gap, and I’m sure you’d all nod in agreement. A MOS of 1.42 is outright awful, and even though the average seems okay, it actually validates customers’ complaints.

Now, how come we see such a difference? Most tools don’t bother with the nitty-gritty of RTP traffic. Therefore, there’s a huge gap between the measurements or the data that they present and the reality that customers experience.  

If your tool is saying that everything is fine, the investigation on this particular tool is pretty much over and you now need to invest a lot of time and effort to further investigate the case by yourself. Now imagine you had a different tool that provides you with plenty of additional data that saves you all this detective legwork. It not only confirms the service degradation during the call but also pinpoints exactly when it happened and possibly even where and why. This makes the journey to successful troubleshooting a whole lot simpler and quicker.

Without going too deep into the techy stuff, just know that when you use time slicing, you’re not just averaging things out. Instead of looking at the entire call, you slice it into bits. This way, you know precisely when the glitch hit, how intense it was, how long it lingered, and the impact it had on your users’ experience. Unfortunately, many tools out there lean on unreliable or even inaccurate data from endpoints or network elements, which don’t provide any meaningful insights.

Case #2: When You Don’t Even Know There Is Something to Fix

This case, looking back, is very funny, especially considering it happened during a POC at a major Tier-1 Communication Service Provider (CSP). Our mission was to set up a probe at a crucial site, monitoring traffic to and from other Points of Presence (POPs). The CSP anticipated a dashboard painted in a serene dark green, symbolizing flawless quality. However, reality painted a more vibrant picture – a canvas of red, yellow, and light green, hinting at an anomaly in one specific site.

Upon closer inspection, we discovered that the Real-time Transport Protocol (RTP) streams between the two POPs were plagued by frequent bursts of packet losses, impacting all streams simultaneously.

Each line mirrored an RTP stream, featuring the familiar traffic light pattern. The expectation was a sea of green, but reality unfolded in hues of red and yellow. The red line cascading from top to bottom signaled a pervasive issue affecting all streams consistently.

When we shared these revelations with the CSP’s operations team, their response was defensive: “It’s our core network; we don’t believe you. We would have noticed if something was indeed wrong.” Fortunately, armed with more detailed data, we substantiated our initial findings and exposed the root cause: severe loss without jitter, indicating a physical issue such as cracked fibers, dirty fibers or defective Small Form-Factor Pluggables (SFPs). The investigation ultimately unveiled a broken link, and the problem was promptly fixed.

Case #3: Three Days Instead of Three Months

Remember I mentioned at the beginning that an 80% reduction in fix time can be underestimated? This case vividly exemplifies this notion, revealing a scenario resolved in just three days with the right data, whereas a consortium of vendors struggled for three months without success. The challenge at hand was a formidable one – the perplexing issue of one-way audio. Without the right data, tackling such a problem is akin to navigating uncharted waters.

One of the largest CSPs in Europe, with millions of customers, onboarded an enterprise customer. The enterprise client voiced persistent complaints from the outset of their onboarding. The complaint centered around the calling party frequently being inaudible, with no discernible pattern related to the time of day or caller identity. The enterprise diligently reported and documented incidents, prompting a task force comprising vendors like Cisco, Acme, and incumbent monitoring solutions to grapple with the enigma.

The below image showcases statistics from December 12 to December 20, revealing a concerning frequency of dozens of reported calls per week. (Spoiler alert: once the pattern was identified, it became clear that the reported calls only amounted to about 4% of the total cases).

Upon our intervention, a glaring anomaly surfaced merely by inspecting the reported calls. As it turned out, each reported call had an accompanying unexpected second stream. Both streams shared identical start times, durations, destinations, and port numbers.

Further investigation exposed a bug in the Cisco Call Manager. In some cases, when an earlier call concluded, the associated RTP stream persisted, transmitting only silence. Dubbed “zombie streams,” these silent streams perpetuated until the Call Manager rebooted, never truly ending. If a new call is initiated to the same media gateway on the same port, the zombie stream could slip through the firewall or Session Border Controller (SBC). The receiver, unaware of what to expect, erroneously selected the first stream received, typically the zombie stream. While signaling appeared flawless and proper speech was exchanged in both directions, the receiver’s selection issue resulted in a one-way audio scenario.

From a data and monitoring standpoint, the critical insight lies in these zombie streams being invisible to conventional tools. Traditional monitoring tools recognize RTP packets only within a call context. Yet, the call context for these zombies had long vanished, leaving only RTP and nothing else in their wake.

Case #4: The Missing Link

This case stands out as one of the most intricate and unconventional challenges I’ve ever tackled, and, interestingly, it also revolves around a one-way audio issue.

The story unfolds at a top-ranked German radio station, a prominent client of a Tier-1 service provider. Renowned for its effective promotions, this station’s wide audience significantly contributes to its success. Audience reach, measured semi-annually by an external agency, plays a pivotal role in determining ad rates. Hence, during these measurement periods, maximizing audience reach becomes paramount.

In February 2018, the radio station initiated a riddle game, promising a €50K prize for callers who could solve it. The catch? Callers were instructed to dial a phone number explicitly excluded from use in mass-calling events. Complaints flooded in as listeners struggled to establish connections, prompting the service provider to start an investigation. Disturbingly, severe degradations of service availability occurred precisely five times a day – the slots designated for the riddle game. The overload not only affected those calling the radio station but also everyone connected to the impacted switch, creating a system-wide strain.

It just so happened that an unrelated government agency, employing 100K people, faced one-way audio and call drop issues around the same time. Frustration mounted as the agency bombarded the CSP with numerous tickets.

Unaware of any link between the incidents, two independent teams at the CSP launched investigations. The radio team traced the overload to calls to the same problematic number, prompting a resolution by recommending a different number for the radio station. Case closed. Or so it seemed.

Concurrently, the agency team delved into reported issues, suspecting network problems but perplexed by one-way audio issues during the same specific time slots. The absence of ticket bursts post-number change raised more questions – after all, the agency team wasn’t even aware of the issue with the radio station. This triggered a deeper investigation with Voipfuture’s support, which uncovered the link caused by a cascade of issues. Massive call attempts led to switch and media gateway overloads, resulting in “service 503 unavailable” error messages. The confused media gateway then emitted non-standard RTP streams, causing one-way audio for some clients.

Why some? Why was the agency hit so hard while others connected to the same media gateway faced fewer issues? The critical factor was the occurrence of sender restarts within these perplexed media gateways. A sender restart refers to a situation where the sequence number fails to increment systematically with each RTP packet. In the midst of the data flow, the sequence number resets, causing a simultaneous impact on all streams.

All streams experience the sender restart, initiating sequence numbers from one. While some longer calls persist, others are swiftly terminated. This inconsistency poses a challenge because the standard RFC 3550 doesn’t provide guidance on how a receiver should respond to such a sender restart. Typically, receivers utilize these sequence numbers to identify packet loss; if there’s a gap in the series, the receiver recognizes lost packets and can initiate packet loss concealment. However, in the absence of a wraparound mechanism for overflowing numbers, uncertainty arises when the sender restarts in the middle of the stream. The receiver faces the ongoing dilemma of whether to play out the audio or disregard the packet, awaiting the one with the correct sequence number.

The unique twist in this tale was the agency’s severe impact, attributed to devices from a specific vendor being overly discerning and ignoring packets during sender restarts. This incident underscores the critical role of data for saving time and resources. This complex case serves as a compelling reminder of the indispensability of comprehensive monitoring tools in resolving intricate network issues efficiently.

Final Thoughts

The journey through the intricacies of voice monitoring has illuminated the critical role it plays in the efficiency and resilience of communication networks. From uncovering hidden voice quality issues to swiftly resolving perplexing one-way audio challenges, the right data proves to be the linchpin in troubleshooting and preemptive measures. The cases presented underscore the tangible impact of advanced monitoring tools.

In a landscape where customer satisfaction and loyalty hinge on seamless communication experiences, the insights shared in these real-life scenarios serve as a beacon, guiding businesses toward a future where proactive monitoring ensures uninterrupted and superior voice services.

In this article I will delve into real-life case studies where customers encountered significant challenges, each one woven with the common thread of protracted struggles with network element vendors unwilling to concede their errors until backed into a corner – confronted with irrefutable evidence, often provided by Qrystal, but I’ll circle back to that a bit later. No more spoilers for now.

Let’s explore the vital lessons learned from these encounters and understand why blind trust in vendor data can lead to extended downtimes, heightened frustration, and potential harm to your business.

Case #1: One Morning in a Major Bank – An Unfolding Disaster

It’s 8:00 AM, the perfect time for the CEO to make an announcement. And not just to a selected group of people, but to all 1,500 HQ employees. Picture this – back in the day when Zoom and Teams were just futuristic dreams, we relied on good ol’ telephone conferences. No video calls, just voices buzzing through the phone lines. Now, here’s where it gets interesting. Fifteen minutes into the call, disaster strikes. The audio goes haywire, and the CEO’s important message is lost in a sea of technical glitches. If you’ve ever been on the operations side of things, you can imagine the chaos. Not fun.

Once the call ends, everything seems to have gone back to normal.

Back to the operations people. They started to scramble and dig into the problem.

The waves of issues hit at 5, 10, 12, and 15 minutes into the call. Everyone stuck around, trying to hang on to every word the CEO was saying. Little did they know that everyone was suffering from the same problem, and little did the CEO know that he was increasingly talking into the void. 

It’s evident that packets arrived in 20 millisecond intervals, so no jitter. But there’s a substantial loss still: burst loss with 10 packets or more lost in sequence and very few packets arriving in between. This severe loss resulted, as we already know, in no one being able to hear the CEO’s important announcement.

Time to call in the cavalry – the firewall vendor, CheckPoint. The vendor’s response came in quickly: “Yes, the issue is known. The workaround is to just switch off the SIP Application Layer Gateway”!! Say what? A feature they charged for, and no heads up about potential hiccups.

That’s when the head of operations reached out to us, thankful for still having his job, after the CEO realized that he was practically talking to himself. This case could have taken a different turn from the monitoring perspective if there was no hard evidence to provide to CheckPoint, who had no other choice but to admit there was indeed a problem on their end. Lesson learned – whether it’s CheckPoint or any other tech wizard, where there’s software, there’ll be bugs. Always arm yourself with hard evidence to make them take responsibility.

Case #2: Ghost Voices

This one is about a tier-1 fixed network provider dealing with a barrage of tickets from seriously concerned enterprise customers. Why? Because these folks were hearing voices. No, not the psychotic kind, but the voices of complete strangers in their calls. You can imagine the annoyance: if they can hear others, who knows if the tables are turned? To top it off, one of the peeved customers was a political party, so you can guess how fast things went from 0 to 100 in the complaint department.

That’s when the network provider reached out to us. At first, we were scratching our heads. Ghost voices, cross-talk, whatever you want to label it, was a regular issue in the analog world. Electric signals could spill over, and you could accidentally eavesdrop on someone else. But in the IP world? How does a voice signal encoded in one packet magically spill into a different call? Total head-scratcher. But hey, we love a good mystery, so we took it on.

We dove into all the reported calls, expecting some SIP and RTP data anomalies, but no dice. All the theoretical mumbo-jumbo about “VoIP crosstalk” didn’t hold up. Then came the lightbulb moment – Cisco MGWs were the common denominator in 100% of the reported convos.

The CSP then called Cisco. Guess what? Cisco played hard to get. No action until there’s proof, they said. They wanted a WireShark trace, or no ticket, no investigation. So, we rolled up our sleeves, set up a packet recording job in Qrystal to record calls of a friendly customer and patiently waited. One day we finally struck gold: Ghost voices – or the voices of another conversation – loud and clear.

Another call to Cisco, armed with hard proof this time. Lo and behold, they admitted it was an old bug that hasn’t been fixed and wasn’t even planned to be fixed. Their grand solution? “Just reset your media gateways every 466 days.”

Case #3: An IMS Full of Surprises

Alright, buckle up for the saga of a customer diving into the world of a new IMS from Mavenir, and boy, did they get more than they bargained for. And Mavenir’s response to the plethora of issues? They basically shrugged off every problem like it was no big deal. Let’s jump into the details:

So, this mobile operator rolls out Mavenir’s shiny new IMS core in their private cloud. Picture this – subscribers migrating from the old Ericsson system to the fresh Mavenir setup, and lucky us, we’re monitoring both. Midway through this techno tango, VoLTE users on the Mavenir platform start griping about silent and dropped calls. Mavenir’s response? Classic denial mode. Independent investigations were launched by both Mavenir and the operator, but surprise, surprise, Mavenir’s data doesn’t align with the actual issues reported in the support tickets.

Here’s what our investigation came up with:

Issue #1 – Silent calls: 

Subscribers were complaining about audio going silent in one direction. Investigation reveals a distinctive pattern. We were monitoring the private cloud that appears in blue in the image below, on the access side and on the core side. 

Party A placed a call, got early media from the media gateway (SSRC1), passed through the IMs cloud, and transferred to the VoLTE subscriber. 

Next, Party B, who is a VoWiFi subscriber sent SSRC2 toward the IMS private cloud, to be transferred to Party A. 

Same thing with the SSRC3, coming from Party A, now that the call was established. 

But this SSRC is terminated at the media gateway and should lead to a new RTP stream being sent from the media gateway to Party B. While the media gateway does that (SSRC4), that RTP stream never popped out from the IMS private cloud. Now the data was clear. We had several examples of this pattern. We had the receipts, but Mavenir kept insisting everything was working well.

Issue #2 – Wrong data on dropped call ratio: 

Mavenir reported a Dropped Call Rate (DCR) of around 0.1%, based on their measurements. But our own measurements told a different story:

• ~ 0.4% for Mavenir IMS and
• ~ 0.3% for Ericsson IMS

Our numbers matched up with the user complaints, not Mavenir’s story.

Issue #3 – Call drops due to wrong bearer:

Why were there more dropped calls with Mavenir than Ericsson? Enter the ‘RTP timeout’ party crashers. Turns out, the media was using the wrong bearer, wreaking havoc on the calls. If you’re familiar with VoLTE, you know that the SIP signaling goes over the default bearer, between the PGW and the UE. To establish a call, a dedicated bearer is set up, which is used exclusively for exchanging the RTP media between the two call parties. That’s the expected scenario. 

What we saw in reality was different. A dedicated bearer was indeed set up, but the RTP was sent through the default bearer. That caused the network elements to declare an RTP time out because nothing was coming out from the expected dedicated bearer’s end of the tunnel. 

Mavenir confirmed the wrong bearer but denied any real impact on users.

This case is a classic reminder of the importance of having the hard facts, the undeniable proof. Service providers and vendors might not always see eye to eye, especially when it comes to critical KPIs. Sometimes, it’s a clash of interests, and having that evidence handy is your golden ticket.

Case #4: Side-Effects of a Power Outage

Alright, let’s dive into the nitty-gritty of this last tale. A quick power hiccup hits one of the mobile network operator’s Points of Presence (POP). The blackout messes with the cooling system, but kudos to the tech squad – they fix it up within hours, and the site is back on its feet. Easy peasy, right? Well, not quite.

Now the plot thickens. The operations team spots a surge in issues post-power outage. Subscribers aren’t shy about airing their grievances either. Check out the image below – thousands of calls are taking a beating with all sorts of packet losses, due to the fallout from the power hiccup.

Now, the bars in the image? They’re like a visual map of misery, showing the number of minutes things went haywire. The colors? Green is all cool with a MOS better than 4.0, but when it hits red, it’s like a signal for downright awful voice quality.

Ericsson was called in to inspect the media gateway. Surprise, surprise – no issues found. But armed with our data, we play detective and point them straight to the troublesome media gateway card out of the 12, wreaking havoc. Card swapped, and just like magic, service quality bounced back to normal.

This was a simple story, but it could’ve easily been a real brain teaser. Picture this: a massive media gateway, making it harder to play detective and pinpoint the source of the call chaos. From the Ericsson Media gateway’s point of view, everything was rainbows and sunshine – calls accepted and terminated like clockwork. No alarms. But customers? Oh, they were ringing the complaint bell. The kicker here? Only when you peek at the quiet metric does it hit you – the chances of having bad call quality were playing a wild 1/12 game of roulette. Crazy, right?

Wrapping It Up

As we draw the curtain on these tales from the telco operations trenches, one thing becomes crystal clear – the road with network element vendors is often paved with bumps, twists, and unexpected turns. From audio glitches in major bank announcements to ghostly voices haunting fixed network providers, and the surprising hiccups in the world of a new IMS.

What stands out, though, is not just the unique challenges each vendor posed, but the common theme weaving through all these narratives. No matter the vendor, no matter the hardware, bugs and issues are the unwelcome guests at this party.

In a world where network disruptions and downtime can mean serious trouble, blind trust in vendor data is akin to walking through a tech minefield. The stories shared here emphasize a crucial takeaway – having the right data at your fingertips is not just a luxury; it’s your shield against prolonged struggles, potential business harm, and heightened frustration.

As telco operators, you’re often at the mercy of vendors, and when things go south, getting them to own up can feel like pulling teeth. The hard evidence is what can turn vendor denials into admissions and pave the way for solutions.

It’s not just about making vendors take responsibility; it’s about ensuring your network stays resilient, robust, and ready for whatever surprises the tech world throws your way.

These roles are crucial to ensure the smooth operation and development of telecommunication networks and services by fulfilling network design and architecture, network monitoring and maintenance, customer support, capacity planning, quality assurance, emergency response, prototyping and testing, research and development, implementation of new technologies, and more.

All these areas require reliable data and thus robust voice monitoring is becoming the industry’s linchpin, ensuring seamless performance and meeting user expectations. It’s clear that the industry needs to vigilantly monitor VoIP services, with the understanding that issues in this sphere can impact the bottom line. 

With this growing need for visibility, I decided to ask several leading telecom professionals— extremely experienced ones, people whose opinions I highly value—for their insights on  effective VoIP monitoring. Here’s what they had to say (plus my personal two cents at the end). 

Khalil Ayman Badwan, Project Engineer/Digital Platforms & Customer Experience

Khalil is an experienced solution/system engineer working in service delivery, project delivery, 3G, 4G, 5G and cloud computing, and AI. He has been working on network intelligence and network analytics since early 2016, with a rich experience in network protocols. This is what he chose to share with us:

From what I’ve seen, it’s super important for any service provider to have and collect insights about their network performance. Having said that, bear in mind that traditional Network Management (NM) tools are very limited when it comes to providing such details.

In terms of data you have around call quality, it should allow you to get to the root cause of problems. It’s essential to collect the exact metrics and counters that describe the issue and the reason behind it. This capability allows the team to identify both the issue and the solution. Unfortunately, ‘generic’ voice monitoring solutions are simply not enough; it can’t be enough to have insight on each aspect or case faced by the subscriber and internally in the network.

Many industry players are switching their voice services to the cloud. It’s very important to have cloud compatibility since cloud computing is the core technology for modern network architecture nowadays. Having such a step will accommodate various options and capabilities to expand our services. Whether it’s on the cloud or a classic physical network, the most important characteristic of a network is to be visible and monitored at all times. 

The most important voice service KPIs to keep track of, in my opinion, are call drop ratio, call setup ratio, Mean Opinion Score (MOS), packet drop, packet loss, and jitter.

Silent calls/one-way-audio is definitely one of the problems my colleagues and I have to deal with the most. In my opinion, the ideal setup is to have a functioning two-directional channel. The second best action is to terminate the call and let the call be re-established between the customers again.

When it comes to VoIP signaling and media monitoring, it’s crucial to care about both and not only one of the two. Let’s assume that your media plane is a bus carrying passengers (which is your VoIP subscriber media) and the signaling plane is the route that the bus is supposed to take to deliver those passengers so both are very critical. The quality of the bus alone is not sufficient if you can’t ensure the quality of the road.

I’m often asked what kind of voice quality information I should expect to get from a voice monitoring solution. My answer to this question is very simple. Make sure it offers:

• An end-to-end analysis capability as well as packet status (transport, loss, delivered, etc.) 
• Call information (termination, release cause, drop rate, setup time, etc.)

Finally, when you’re looking for a voice monitoring solution, these are the most important questions you need to ask the different vendors to help you make a decision:

• How would I know exactly what the actual customer experience is (as a real value) using your solution?
• How would you help me maximize and increase my customers’ experience? 
• How can we prevent issues and problems from happening in the first place, using your tool? 

Jan Holub, Professor, Head of Department of Measurement at Faculty of Electrical Engineering, Czech Technical University

Prof. Holub’s research interests span various aspects of Measurement Technology, with a focus on subjective quality, intelligibility, and usability testing. He has been an active member of several professional organizations and has chaired organizing and program committees for numerous conferences. In addition to his research pursuits, he has shared his knowledge and expertise through teaching various courses at CTU. He shared the following with us:

While there are many angles to VoIP monitoring I can discuss and hopefully give a piece of useful advice on, I chose to focus my two cents for the telecom community on the role I see for monitoring data in future AI/ML-driven service automation.

I see its potential in multiple domains, primarily in predictive maintenance, dynamic resource allocation, and network security, but also in single-user-tailored QoE.

AI and ML algorithms can analyze RTP data to assess the quality of voice and video streams in real time. By continuously monitoring RTP metrics like jitter, packet loss, and latency, AI can detect performance issues and adjust network configurations or routing to optimize QoS. Also, AI can use historical RTP data to predict when network or service issues are likely to occur. By identifying patterns in RTP metrics that precede high-load periods or even service disruptions, AI can dynamically allocate resources like bandwidth or processors, optimize processes like voice codec or video compression, or even trigger preventive measures or automated maintenance tasks to avoid or at least minimize downtimes. When RTP issues occur, AI-driven systems can automatically perform root cause analysis. By correlating RTP metrics with network configurations and other data sources, AI can rapidly identify the source of problems and suggest corrective actions. AI-based systems that detect anomalous traffic patterns or security threats by monitoring RTP data are already in place. These systems already not only generate alarms but directly respond with security measures.

One could say that all these applications were already available in existing monitoring systems, but the major benefit of AI/ML deployment is opening the possibility of continuous improvement and automated self-learning by using RTP data to identify areas for enhancement in real-time communication services, leading to ongoing service optimization.

Another fruitful AI application is perhaps user behavior and preferences analysis based on tracked user interactions and/or sentiment analysis from audio and video data where patterns acquired from RTP monitoring play an essential role. Also here AI can provide personalized recommendations or features to enhance the user experience.

Liju Mathew, Director—Solutions & Systems

With over 15 years of professional experience in the telecom industry, Liju has a robust background spanning various sectors, including manufacturing, software development, and telecom service provision in 2G, 3G, LTE, and NGN technologies. His expertise encompass a deep understanding of signaling, switching, network planning, billing systems, revenue assurance, service provisioning, network management, service costing, OSS/BSS, risk assessment, and mitigation. Here is what he told us:

It is critical to have the ability to generate data-driven insights through your voice monitoring solution. Here’s why: Voice over 4G and 5G technologies will remain one of the main services offered to the customers and it is critical and very important to be able to monitor and measure the technical parameters that influence and impact the setup, call quality, speech impairments like noise, silence, one-way call, etc. Effective monitoring of the signaling and RTP media is required to ensure top-quality voice services demanded by your customers.

It is also very important to make sure the data you have around call quality allows you to get to the root cause of problems. The first thing is to detect that there is poor quality in voice service experienced by the customers. With effective monitoring of different legs of the network (access, core) and with proper knowledge of the network architecture and flow of the traffic within the network it is possible to determine the source of quality issues and conclude the root cause analysis.

The most important voice service KPIs I highly recommend you keep track of are latency and packet loss potentially resulting in MOS value degradation, one way speech, and silent calls, as those are the ones that are impacting your customers’ experience the most.

Unfortunately, with Voice over IP networks in 4G and 5G mobile networks, the generic signalling-based voice monitoring solutions are not enough and it is absolutely necessary to invest in media quality monitoring solutions.

Livio Pogliano, Head of Roaming & Carrier Operations

With 28 years of experience within global telecommunications companies, Livio is a highly skilled and accomplished executive known for pioneering new technologies, conceptualizing innovative services, and advocating continuous efficiency enhancements. A proactive change agent, he specializes in implementing innovative strategies that optimize resource allocation while upholding cutting-edge quality benchmarks. According to him:

In a world where IP-protocol is driving many types of communications from data to voice, it becomes very important to have a clear and detailed description of the service perceived by the customer, and it has to be possible to do that in real-time. Service monitoring moves to the next level where it is not enough to verify that network nodes are up and running, but multiple sets of KPIs can provide the real description and potentially create “service alarms.”

A wider and open approach is required to create KPIs and to potentially detect subjective effects, such as voice call quality, in an objective way. Great effort is required to have a consistent and accepted-by-standards Mean Opinion Score algorithm to verify that traffic quality can reach good levels for the human ears. It means that working on media contents, and not just IP signaling, is needed, taking always into account privacy constraints.

Moreover, migrating the infrastructure into the Cloud increases the focus on service performance for Operations teams, driving engineers to a new learning phase in their careers.

At last, all these service evolutions and new requests are highly demanding when it comes to computation and analytical algorithms: a strong knowledge of the service implementation and technical architecture is needed to create the right and customizable tools.

This future-oriented and high expertise is really the key factor for the definition and choice of providers for voice-over-IP service measurement systems.

Markus Monka, Hacking telco since 2003

Our long-time customer and friend Markus from sipgate GmbH told us this:

Generating data-driven insights through a voice monitoring solution is paramount to our continuous improvement in the telecommunications industry. These insights allow us to spot trends, pinpoint quality issues, and proactively enhance customer satisfaction. This data-driven approach is pivotal to our service optimization.

Our ability to track problems down to their root causes is made possible by the wealth of call-quality data at our disposal. It’s not just about addressing surface-level issues; it’s about eradicating underlying causes for sustainable service quality and minimal disruptions.

As more industry players migrate their voice services to the cloud, careful consideration is required. Robust voice monitoring becomes essential to ensure the performance of cloud-hosted services meets expectations.

In my opinion, key voice service KPIs to monitor include call quality, delay, jitter, packet loss, and call drop rates. These metrics are crucial for ensuring smooth communication and customer satisfaction.

Both VoIP signaling and media monitoring deserve attention as problems at either level can affect service quality. A comprehensive approach that considers both aspects is advisable.

Silent calls and one-way audio are serious issues affecting customer satisfaction. Vigilant monitoring and testing are crucial for early detection and swift resolution of such problems to maintain uninterrupted communication.

From a Voice Monitoring solution, we expect detailed information on call quality, including MOS scores, packet loss, jitter, and latency. This data is instrumental in our continuous quality enhancement efforts.

Michael Wallbaum (that’s me), Director of Product Marketing

What can I add to the insights of the seasoned telco professionals who contributed to this article? In 1999 I was part of a research team that demonstrated voice and data transmission over an HSCSD mobile data connection at the Telecom ‘99 in Geneva. Back then, voice over IP was new and exciting, but also flakey and we were happy for every demo that went without a glitch. A long time has passed since and the telco industry has now almost completed its transition to IP. Any form of “glitch” is no longer acceptable as we all rely on phone services for our daily lives. Still, everyone has experienced the inability to set up a call, call drops, silent calls, or simply bad voice quality. How can this happen and why does it often take so long to fix? 

Having worked with many different CSPs I would argue that the main reason for today’s “glitches” is lack of information in many different forms. It’s not that there is a lack of data—quite the opposite. There are, however, issues with the data itself and how it’s organized and processed. 

First of all, while CSPs have tons of data, it is often inadequate. Traditionally, there is a strong focus on signaling performance monitoring, but the in-call user experience is frequently neglected. Still, every monitoring solution vendor and CSP will claim that “MOS is available,” but few actually understand (or admit) that this media quality information is not accurate, because it is not obtained from direct measurements of the RTP streams. One symptom of this data quality issue, is that only in 2019 ETSI proposed a set of timeslice media quality KPIs in TR 103 639 to complement well-known signaling KPIs. Only now is it possible to calculate media quality for calls, trunks, routes and other aggregates in a meaningful and standardized way.

Secondly, even if information on the voice service performance is accurate, comprehensive and reliable, all too often relevant data is not accessible to all stakeholders. Data is still stored in silos, defined either by the licensing and export policies of vendors or by organizational boundaries. Typically, the larger the CSP the more fragmented the data space. Identifying the stakeholders and putting all the puzzle pieces together can be cumbersome and time-consuming, which greatly adds to the stress of incident management. Exporting all available data into a so-called data lake is the recommended approach to break down the silos, but I have yet to see more than measly data ponds.

Lastly, even when all the data is accessible, relevant and accurate, there is a lack of tools and best practices helping to make sense of the vast amounts of data, in addition to a lack of human resources caused by low margins and stiff competition in the telecommunications market. Artificial intelligence and machine learning technology are promising to automate the process of filtering, correlating and analyzing the available data to speed up root cause analysis and other operational tasks. The holy grail is zero-touch automation (ZTO), i.e., fully automated voice service management. However, without reliable data from many different sources ZTO remains a pretty bold vision.

To wrap it up: the industry needs to focus on the quality and accessibility of data, to enable more automation leading to better service quality. 

Final Words

In the ever-evolving landscape of telecommunications, industry leaders have shared invaluable insights concerning the complexities, necessities, and benefits of effective VoIP monitoring. These professionals, with years of experience under their belts, emphasize the importance of generating data-driven insights.

As professionals deeply entrenched in the industry, we now all recognize that voice monitoring goes beyond the conventional checks of network node functionality; it should delve into the intricate details of the service quality experienced by our users, examining issues in both VoIP signaling and media quality. Generating insights through comprehensive data analysis is not merely a best practice; it’s the cornerstone of service optimization strategies. 

We should meticulously track an array of key performance indicators, from call quality and delay to jitter, packet loss, and call drop rates. By deciphering this data, we not only address surface-level issues but also uncover the root causes, ensuring sustained service quality and enhancing customer satisfaction. On top of that we can lay the foundation for trustworthy service automation technologies.

Through continuous monitoring efforts, we can swiftly identify and resolve challenges such as silent calls and one-way audio, maintaining uninterrupted communication for our users. Voice monitoring solutions available in the market should all provide us with detailed insights, including MOS scores, packet loss, jitter, and latency, enabling us to drive continuous quality enhancement initiatives. 

In this transformative landscape, our commitment to accurate monitoring and proactive resolution remains unwavering, ensuring superior user experiences and driving the future of telecommunications.

Google Cloud, a leading global enterprise delivering enterprise-grade solutions that leverage Google’s cutting-edge technology, and Voipfuture, provider of market-leading voice service monitoring technology and analytics solution, announced today their partnership, under which Voipfuture’s Qrystal solution is now available on Google Cloud, thereby creating a joint visibility offering for the telecommunications industry.

Communication service providers are increasingly adopting public cloud infrastructure offered by Google Cloud, for their core voice services. It allows them to scale operations quickly and efficiently, to improve reliability and availability and it offers them robust backup and disaster recovery mechanisms, protecting voice services against data loss or system failures.

One of the most challenging aspects of delivering an exceptional customer experience has always been visibility into the network. Traditional on-premise deployments required installing taps and aggregators and rolling out expensive hardware to all sites. Voipfuture and Google Cloud now enable voice service providers to focus on delivering a superior customer experience.

Google Cloud, in collaboration with Voipfuture, offers a unique solution to maintain call quality throughout all stages of the service lifecycle. The following capabilities are available through Qrystal for Google Cloud, a comprehensive monitoring solution:

1. Analyze Google Cloud call-related traffic, and if necessary, extend the analysis to on-premises infrastructure as well.
2. Dropped calls or cases of silent calls can be detected and identified automatically.
3. Provide accurate and rapid root cause analysis, allowing problems to be resolved quickly.
4. Provide informed decision-making through the generation of valuable statistics and Key Performance Indicators (KPIs).
5. By maintaining high service quality, ensure a superior user experience that meets customer expectations.

“Partnering with Google Cloud is an important milestone for Voipfuture,” said Eyal Ullert, Director of Sales and Marketing of Voipfuture. “Qrystal for Google Cloud stands as a carrier-grade and vendor-agnostic solution, providing a proven assurance platform for voice services on Google Cloud. Qrystal delivers accurate and actionable data that is an essential data source for the future of AI/ML-assisted zero-touch operations.”

• Voipfuture Qrystal and AWS Lookout delivered convincing results in a first joint trial
• Showcase will be demonstrated at MWC Barcelona 2021

Voipfuture and Amazon Web Services (AWS) enable Communication Service Providers to benefit from artificial intelligence (AI) and machine-learning (ML) for 5G voice services in the AWS cloud. Qrystal’s deep data on voice service performance and Amazon Lookout proved to make a perfect match. The demonstrator showed how quickly anomalies and their root causes can be found. This illustrates the potential of automated service management based on AI/ML models trained with accurate and reliable data.

Voipfuture Qrystal delivers accurate service performance and user experience data to all CSP stakeholders

• Deep analysis of SIP and RTP traffic, with high temporal detail and automatic detection of call drops, silent calls and many other impairments, for effective and efficient operations,
• Aggregate data on trunks, routes, eNBs, interconnections and other relevant entities enabling trending, alarming and benchmarking for performance optimization teams,
• Unique key performance indicators, e.g. for in-call user experience, used by sales/marketing teams to attract and retain customers, e.g. through self-service portals

This data forms the foundation for service automation based on AI/ML.

Amazon Lookout for Metrics uses machine learning (ML) to automatically detect and diagnose anomalies (i.e. outliers from the norm) in business and operational data. Lookout makes it easy to diagnose detected anomalies related to the same event and sending an alert that includes a summary of the potential root cause. It also ranks anomalies in order of severity so that you can prioritize your attention to what matters the most to your business.

Having worked closely together with the AWS Telco Network Transformation team Dr. Michael Wallbaum, Director Product Marketing at Voipfuture says: “I’m really excited to see our data in action. For over a decade operations and performance management teams have successfully used Qrystal for their daily work, but using Qrystal’s data to train AI/ML models is taking it to a whole new level. We soon hope to see first examples of automated service management powered by Qrystal and AWS AI/ML services.”

• The annual MWC Barcelona is one of the most influential events of the communications industry. This year it will take place from June 28^th to July 1^st 2021
• Voipfuture was invited by Amazon Web Services (AWS) to the AWS Virtual Village to showcase 5G voice monitoring and AI/ML applications in the cloud

Voipfuture is proud to announce participation in the Mobile World Congress 2021. Voipfuture was selected by AWS to showcase 5G voice analytics at the AWS Virtual Village.

During the MWC Barcelona AWS will provide a deep dive into the latest strategic thinking, innovative approaches, and use cases that are accelerating business outcomes for telecom operators and the consumers and enterprises that depend on their digital services. The AWS Virtual Village will show how AWS enables communication service providers to transform and offer an unlimited range of new services based on 5G networks and beyond. The AWS Virtual Village will feature interactive demo showcases, spotlight sessions and a Partner Showcase area.

Voipfuture contributes to Amazon’s vision of reinventing communications and was invited by AWS to join the Partner Showcase. Voipfuture’s Qrystal for AWS is the carrier-grade, vendor-agnostic solution for VoIP service monitoring and analytics in the cloud.  Qrystal for AWS provides reliable and accurate performance data for voice service providers. It addresses the need for operational efficiency and helps to ensure that service quality meets customer expectations. Qrystal provides control over voice quality and reduces costs by shortening the time-to-fix by up to 80% through automatic root cause detection, timeslicing and behavioral analysis of users matching the actual customer experience. Qrystal data together with Amazon’s artificial intelligence and machine learning services paves the way to reliable and efficient service automation for 5G networks. As part of Amazon’s Partner Showcase, Voipfuture will demonstrate Qrystal’s benefits to mobile network operators and the integration with AWS AI/ML services.

“I am excited to showcase our capabilities as part of the AWS Virtual Village. Qrystal for AWS is a strategic product line for Voipfuture. Qrystal will help CSPs migrating to AWS to increase their operational efficiency and reduce costs by shortening the time-to-fix by up to 80%“, said Leor Penn, VP Sales at Voipfuture.

The MWC Barcelona, is the world’s most influential exhibition for the communications industry. After last year’s cancellation, the Mobile World Congress 2021 will take place between the 28th of June and 1st of July in hybrid form. Attendees can visit the MWC Barcelona in-person or virtually.