Core principles of competitive analysis & benchmarking

There are some core principles of conducting such project which are important to take into consideration and remember:

• The testing needs to be conducted in the most objective way possible (unless agreed otherwise) to ensure comparability of the products and eliminate human biases from the process. For non-functional testing numerical metrics are the best option, for functional testing - stating facts.
• When analyzing data, use appropriate language to communicate findings that do not discriminate against other competitors or showcase opinions about specific applications - avoid using “application had bad quality” or “this competitor has good audio quality”. Opt to use “application video quality is lower by x scoring points compared to the competitor” or “this competitor has higher POLQA score compared to your product”.
• When conducting such a project, it is highly common to work with different applications that might be covered by other projects or even be our clients - make sure you are familiar with NDA policies and public information in order to not disclose anything that might compromise other companies. Also, ensure that even with public client information, the type of project (unless also public) and what is being done for that specific client is never disclosed.
• Have knowledge of the industry. The clients can prepare their own list of competitors, however, you should be ready to understand the potential applications that might be considered competitors of the client. You need to state them in a way that does not disclose if they are our clients. You have to consider security policies when communicating such information.