How to Produce Research-Grade Benchmarking Data on Conferencing Platform Performance

Benchmarking data on conferencing platform performance is only as credible as the infrastructure used to produce it, and most testing setups are not built to the standard that credible research requires.

For research organizations, enterprise procurement teams, and technology analysts who need to compare conferencing platforms across realistic conditions, the critical constraint is methodological. Standard QA tooling is designed for pass/fail functional validation, not for capturing the perceptual quality metrics, like video PSNR, VMAF scores, and POLQA audio ratings that differentiate platform performance under degraded or complex conditions. Producing data that can withstand the scrutiny of sophisticated buyers requires test infrastructure engineered from the ground up for the specific demands of the research brief.

This article addresses that challenge directly: what does it actually take to design and execute a rigorous, multi-platform conferencing benchmarking study, and why does the quality of the test setup determine the value of the results?

This article draws on TestDevLab's engagement with Signals Research Group (SRG), a US-based research and consulting firm whose independent analysis of wireless telecommunications technology is relied upon by mobile operators, equipment suppliers, and the financial community. When SRG set out to benchmark the audio and video performance of Cisco Webex, Zoom, Google Meet, and Microsoft Teams across a wide range of devices and network conditions, credible results required purpose-built test infrastructure. Read the full Signals Research Group case study for complete details.

Why can't standard QA infrastructure produce research-grade conferencing benchmarks?

Standard quality assurance setups answer a different question than benchmarking research does. QA testing asks: does this feature work as specified? Benchmarking research asks: how does this platform perform relative to competitors under the conditions users actually encounter? These questions require fundamentally different infrastructure.

Functional QA validates pass/fail behavior under controlled, typically optimal conditions. Research-grade benchmarking must capture quantitative perceptual quality metrics—PSNR, VMAF, POLQA—across a matrix of platforms, devices, network conditions, and call configurations that reflects the full range of real-world deployment scenarios. A setup designed for the first question cannot produce reliable answers to the second.

The specific demands of a conferencing benchmarking study compound quickly. Multi-participant call configurations—1v5 and 6v8 sessions, not just 1v1—require video separation and processing capabilities that do not exist in off-the-shelf QA tooling. Network condition variability across multiple ISPs and bandwidth states requires custom simulation infrastructure. Noise suppression testing requires calibrated audio injection at loudness levels precise enough to produce meaningful between-platform comparisons. And extending the study to a new platform mid-engagement requires real-time problem solving that generic test frameworks are not designed to accommodate.

The result of using inadequate infrastructure is not a failed study. It is a study that appears to succeed but produces data that cannot be trusted, precisely because the methodology cannot withstand scrutiny from the sophisticated buyers who would use it.

What metrics and configurations does research-grade conferencing testing require?

A rigorous conferencing benchmarking study must cover several distinct measurement domains, each requiring specific technical capability.

• Video quality metrics. PSNR (Peak Signal-to-Noise Ratio), VMAF (Video Multi-Method Assessment Fusion), visual impairment classification, and FPS monitoring each capture different dimensions of video quality degradation under varying conditions. Collecting all of these across multiple platforms and devices requires a processing pipeline capable of handling large volumes of video output consistently and without introducing measurement artifacts.
• Audio quality metrics. POLQA (Perceptual Objective Listening Quality Analysis) scoring provides a standardized, perceptually grounded measure of audio quality that maps reliably to user experience. Applying POLQA across platforms and conditions requires precise test audio injection and a consistent capture methodology.
• Multi-participant call configurations. 1v1 configurations are the simplest case and the least informative for enterprise buyers who need to understand platform behavior at scale. 1v5 and 6v8 configurations reflect the meeting scenarios that actually matter, and require engineering to separate and process multiple simultaneous source video streams in ways that standard tooling does not support.
• Network condition variability. Testing under optimal network conditions tells buyers nothing about platform behavior when conditions degrade. A credible study must test across multiple ISPs and bandwidth states, using controlled simulation that can be reproduced consistently across every platform-device-condition combination.
• Noise suppression behavior. Evaluating noise suppression requires calibrated audio injection across multiple scenarios: no background noise, background noise without suppression enabled, and background noise with suppression enabled. The calibration of the injected noise level is not a minor detail. If it is wrong, the between-scenario comparisons are meaningless, and the resulting POLQA scores reflect the noise injection artifact rather than genuine platform behavior.
• Cross-platform and cross-device consistency. Research-grade data requires that every metric be captured under conditions that are consistent enough across platforms and devices to support direct comparison. Any variability in test conditions that is not itself a controlled variable becomes a confound that undermines the comparability of results.

What does purpose-built conferencing benchmarking infrastructure look like?

The engineering challenge in a research-grade conferencing study is not executing tests. It is designing infrastructure that can execute tests at the required scale while maintaining the methodological consistency that makes results comparable and publishable.

• Custom participant configuration setups. 1v1, 1v5, and 6v8 configurations each require different engineering. The 6v8 configuration in particular demands a novel approach to separating and processing six simultaneous source videos within a 30-user call grid. A capability that must be developed specifically for the engagement rather than drawn from existing tooling.
• ISP virtualization and bandwidth simulation. Replicating the network variability that enterprise users and consumers experience requires virtual network configuration across multiple ISPs, combined with a custom bandwidth limitation script capable of delivering consistent, reproducible network conditions across every platform, device, and bandwidth state in the test matrix.
• Calibrated noise suppression testing. Getting the pink noise injection level right requires iterative calibration work before testing begins. The noise must be loud enough to engage the suppression algorithms, but not so loud that it overwhelms them and renders between-scenario comparisons meaningless. This calibration work is methodologically essential and cannot be skipped.
• Large-scale video processing pipeline. A study producing hundreds or thousands of video files requires a processing pipeline capable of handling that output systematically,  applying VMAF and PSNR analysis consistently across every file and delivering results in a format that supports the analytical work the research organization needs to do.
• Real-time scope extension capability. Research studies frequently encounter scope changes as findings emerge, such as a new platform to add, a new condition to test, an additional question from a client. Infrastructure that can accommodate these extensions without disrupting the overall test matrix is not a luxury; it is an operational requirement for a three-month engagement with sophisticated clients.

What did purpose-built benchmarking infrastructure make possible for a real research organization?

SRG's engagement with TestDevLab demonstrates what research-grade infrastructure enables at scale. The study covered 360 individual tests across four platforms, produced 2,304 video files, and operated across 14 real devices, three ISPs, and three bandwidth conditions. This scope was only achievable because the test setups were engineered to handle it from the outset.

The 6v8 participant configuration required TestDevLab's video processing engineers to develop a novel approach to separating and processing six simultaneous source video streams within a 30-user call grid. This capability did not exist before the engagement and was built specifically to meet the research requirement.

Network variability testing across three ISPs with a custom bandwidth limitation script produced a dataset that reflected how each platform behaves when conditions degrade, not just when they are optimal. This is the data that matters most to SRG's clients—operators and enterprises who need to understand performance at the margins.

Noise suppression calibration required iterative methodological work to ensure that POLQA scores captured genuine differences in suppression effectiveness across all four platforms, rather than an artifact of the noise injection itself.

When Chromebook was added as a test platform mid-engagement, introducing issues across device performance, connectivity, application support, and script compatibility, TestDevLab resolved each issue in turn, maintaining the integrity of the overall test matrix and delivering Chromebook data to the same standard as every other platform.

From the full dataset, SRG was able to draw conclusions about which platforms perform best on which devices, which hold up better under constrained network conditions, which are better suited to large-scale calls, and which are appropriate for specific deployment contexts such as schools. These are findings that directly inform the purchasing and deployment decisions of SRG's operator and enterprise clients.

As SRG President Michael Thelander noted of the engagement, TestDevLab's team went above and beyond to ensure every data requirement was met, fulfilled additional requests throughout the three-month project, and delivered all information in a timely and professional manner.

Read the full Signals Research Group case study for the complete methodology and findings.

The bottom line

For research organizations and enterprise teams that need conferencing platform benchmarking data capable of driving real purchasing and deployment decisions, the rigor of the test infrastructure is not a methodological detail. It is the single variable that determines whether the results are worth trusting.