Can Your Favorite App Keep Up? [Part 1] Testing Video Cold Startup Using VPN

Picture this: You wake up in London, catch a few Shorts over breakfast, hop on a flight to Berlin for a business meeting, and then jet off to New York. Sound familiar? Today, frequent location changes are a regular part of business travel. But as you move from one continent to another, can your favorite apps keep up? Are they capable of delivering a smooth, reliable video experience regardless of where you are?

At TestDevLab’s Audio and Video Testing Lab, we set out to answer these questions. We conducted a comprehensive study on cold-start video playback performance across leading short-form video apps like Pinterest, TikTok, YouTube, Snapchat, Triller, and X. By simulating diverse VPN locations — including the UK, Germany, UAE, Brazil, Mexico, India, Indonesia, Japan, and New York — and testing under both 10 Mbps and 2 Mbps network conditions, we aimed to uncover how quickly each app initiates playback under varying circumstances.

For this experiment, we used NordVPN, a tool that allowed us to route internet traffic through different servers worldwide. But why a VPN? Typically, VPNs are associated with privacy, security, and bypassing region-locked content. However, for developers, QA engineers, and testers, VPNs offer another invaluable function: simulating user experiences from different global regions. In our tests, NordVPN enabled us to assess how these apps perform under varying server locations and bandwidths, making it an underrated yet crucial tool for optimizing the user experience worldwide.

This is the first article in our three-part series exploring video cold-startup performance using VPN. Stay tuned for the next articles in this series, where we will dive deeper into our findings, analyzing the impact of startup time and video playback quality across different server locations when using a VPN.

Preparing the test setup

To effectively evaluate app performance, we aimed to replicate realistic user scenarios using different network conditions. From airport lounges to remote road trips, our testing environment was carefully curated to simulate everyday situations where users might encounter connectivity challenges. Let's take a closer look at the devices, network conditions, and video content we used to assess app performance.

1. Devices & environment

We created and used the following environments and devices:

• Mobile device used: Samsung Galaxy S21, Android 14
• Device used: Apple MacBook Pro 13.3, macOS Sequoia 15.0.1 
• Network conditions: 
  • Bitrate limited to 10 Mbps
  • Bitrate limited to 2 Mbps
• Network: NordicVPN

Why did we choose these network limitations? Here are some real-world examples of network limitation use cases:

• Bitrate 10 Mbps (moderate-speed connection, minor delays possible): A user is using mobile data (4G) while sitting in a busy airport lounge. They open the YouTube app, which hasn’t been used in the last days. The app cold-starts and loads the home feed within a couple of seconds. Videos autoplay smoothly in good quality, with only a brief delay.
• Bitrate 2 Mbps (very low-speed connection, frequent buffering likely): While on a road trip in an area with a weak signal, a user opens the Snapchat app for the first time that day. The app cold-starts, but loading the feed is noticeably slow. Stories and short videos load in low resolution, with visible delays and intermittent buffering due to poor network conditions.

2. Apps tested

We tested a selection of popular apps that support short-form video content. 

• Pinterest, 13.5.0
• TikTok, version 38.7.3
• YouTube, version 20.06.36
• Snapchat, version 12.29.0.49
• Triller, version v55.2b54
• X, version versionName=10.80.1-release.0

3. Video uploaded 

To ensure a fair comparison across different apps, we created a new user profile and uploaded the same video to all applications. Video complexity is crucial in evaluating performance, as various apps may handle high spatial complexity differently.

Video specifications

• Duration: 15 seconds
• Frame rate (FPS): 30
• Bitrate: 13,969 Kbps
• Resolution: 1080x1920
• Codec: H.264 (High) (avc1 / 0x31637661)
• Color format: YUV 4:2:0 (Progressive)

Video complexity

• High spatial complexity – Contains highly detailed scenes, intricate textures, and high contrast.
• Low temporal complexity – Features a static camera and background, with only the subject (a dancing girl) in motion.

Testing methodology

We conducted this experiment in the period between February 18 and March 26, 2025, with a total test count of 2,848. The precondition for the user scenario is that the app is set in a cold start-up state (clear cache and data). Below are the steps we took for each test scenario:

• User opens the app and logs in
• User clicks search and searches for the account
• User opens the account
• User clicks on the second short/video
• User watches the video for some time 

Recording length

• 15 seconds for 10 Mbps tests
• 30 seconds for 2 Mbps tests

Tools used for data processing and collection

• FFmpeg—a free and open-source software designed for processing video and audio files.
• Text detection for startup time measurement
• VMAF—full reference video metric; additionally used in-house built video processing argument to ignore color differences (darker overall for some of the apps)
• Ntrex—TestDevLab’s in-house built tool for processing network
• Tempero—TestDevLab’s in-house built tool for data collection
• Viqubox—TestDevLab’s in-house built tool for audio, video, and network processing
• Speed test app
• What is my IP

Key performance metrics

The following performance metrics were gathered and analyzed:

• Startup + buffer time - The time it takes for the video to load after clicking on a video link, plus the duration of buffering events during recording time.
• Out-of-scope tests - Metric used for poor latency, when video playback does not start within the recording time (either 15 or 30 seconds, depending on network limitations). Out-of-scope tests are excluded from the average data calculation as video playback has not started, and no performance data has been gathered.
• VMAF - This full reference video quality metric was developed by Netflix. For this particular experiment, an ignore color difference argument is used when processing video quality because most of the apps have a darker overlay at the top or bottom of the screen.
• Startup time - The time it takes for the video to load after clicking on a video.

Results & findings

When considering a scenario where a user has a moderate-speed connection—for example, using mobile data (4G) while sitting in a busy airport lounge and opening an app that hasn't been used for a while—we observed that Pinterest delivered some of the fastest video playback start times among the apps tested, initiating playback within 0.12 to 0.26 seconds across all VPN server locations.

Depending on the VPN server location, the next fastest apps to start video playback are TikTok, YouTube, and X. In contrast, Snapchat and Triller lag behind the competition in all VPN test cases conducted under a 10 Mbps connection.

Note: TikTok is currently banned in India; therefore, no tests were conducted using a VPN connection to India for this app.

The graph below shows video cold startup performance on 10 Mbps. Apps are listed in alphabetical order.

Tests using VPN connections to London (UK) and Berlin (Germany) show lower startup times across all apps than other server locations. The difference is particularly noticeable for apps such as TikTok, YouTube, Triller, and X, where the startup time is significantly faster in these two European locations.

In contrast, for locations like Mexico City (Mexico), Jakarta (Indonesia), São Paulo (Brazil), Dubai (the United Arab Emirates), Tokyo (Japan), and Mumbai (India), these same apps tend to have slower video playback start times. Interestingly, Snapchat demonstrates a different pattern—its startup time in these regions is comparable to London (UK) and Berlin (Germany), indicating a more consistent performance across geographically diverse locations.

Although a VPN connection to São Paulo (Brazil) presents challenges for most apps, YouTube appears to struggle the most. The startup time for YouTube in São Paulo (Brazil) is highly inconsistent — video playback can begin in as little as 1.7 seconds, but there are also significant outliers, with some instances taking as long as 14.3 seconds. This wide variation results in an average startup time of 7.15 seconds, the highest among all tested locations on a 10 Mbps connection.

Below is an example of app performance with the  VPN set to São Paulo (Brazil).

When analyzing startup + buffer Time on a 10 Mbps connection, we observed that most apps ensure smooth video playback once the video has started, with no additional buffering required. However, during tests using a VPN connection to a server in Japan, YouTube exhibited a slight increase in buffering, requiring on average an extra 0.1 seconds.

Many short-form video apps use dark overlays at the top and bottom of the screen, which can potentially impact video quality metrics such as VMAF. For this research, we applied an "ignore color difference" setting when analyzing video quality to account for the consistent use of overlays across most apps.

Under a 10 Mbps connection, YouTube leads in video quality, though this can vary by server location. Pinterest, Snapchat, and TikTok follow closely behind in terms of visual clarity, while X and Triller show comparatively lower scores. Overall, video quality remains stable across all platforms, with the exception of YouTube showing the highest consistency, and Triller displaying minor instability.

👉 For a deeper dive into startup time and video quality variations across different regions at 10 Mbps, check out this blog post.

When the network is limited to 2 Mbps, simulating real-world situations such as being on a road trip in an area with a weak signal, users may experience varied performance across different apps. Based on the startup time data, Pinterest demonstrates strong performance under these constrained network conditions, consistently showing quick video startup times across all tested locations.

Specifically, Pinterest can initiate video playback in just 0.17–0.20 seconds in locations like New York (USA), Jakarta (Indonesia), London (UK), and Berlin (Germany). Even in more distant or network-challenging regions such as Mexico City (Mexico), Dubai (the United Arab Emirates), Mumbai (India), and Tokyo (Japan), Pinterest maintains solid startup performance, with times ranging from 0.26 to 0.36 seconds. In São Paulo (Brazil), where network constraints are typically more pronounced, Pinterest still manages a startup time of 0.83 seconds, outperforming its competitors.

All other apps tested lag behind Pinterest in this scenario, indicating that Pinterest’s optimization for low-bandwidth environments gives it a clear edge in delivering faster content access even under limited network conditions.

With a great gap in performance, the next quickest apps to start video playback are TikTok, YouTube, and X. Triller shows the poorest performance, starting the video playback in 10 seconds. Additionally, it requires extra buffering of around 12+ seconds for all locations. Although Snapchat is behind in startup time, there is almost no buffering needed for all locations, making its overall user experience much better than Triller.

When analyzing weak network conditions, we should pay attention to the percentage of tests that fall out of scope. This metric indicates how many video playbacks actually start within the test time window, 30 seconds for 2 Mbps tests.

In our research, we did not observe any specific location where all apps consistently failed to start video playback within 30 seconds. However, we did notice that certain apps struggle to start playback in multiple locations. This suggests that out-of-scope results are more closely related to the overall stability and performance efficiency of the app itself, rather than any particular test location.

The apps that showed higher rates of out-of-scope results include TikTok, YouTube, Snapchat, and Triller.

👉 For a deeper dive into out-of-scope tests, startup time, and video quality variations across different regions at 2 Mbps, check out this blog post.

When conducting tests on a 2 Mbps connection, X prioritizes video playback over video quality and drops the video quality the most from the tested scope. Pinterest, TikTok, YouTube, and Snapchat have similar video image quality. Overall, minimal to no video quality difference between the tested locations was noticed.

Below is an example of app performance using VPN to São Paulo (Brazil) in a 2 Mbps limitation test. When video playback starts, the video is tinted green.

Key findings

Pinterest showed strong performance across regions and network conditions, particularly in terms of startup time and buffering. Even under constrained 2 Mbps network conditions, it maintained relatively quick startup times, ranging from 0.16 to 0.83 seconds. This trend was also observed in high-latency regions such as São Paulo (Brazil), where Pinterest was among the faster apps to initiate playback, often staying under 1 second.

Overall, Pinterest proved to be a reliable option for users on limited or moderate networks, delivering consistently responsive playback experiences with minimal buffering.

TikTok and YouTube generally offer fast startup and acceptable buffering at both bandwidth levels. However, they show more performance variability depending on server location and network quality.

On 2 Mbps, Snapchat has a unique performance pattern. Despite slower startup times compared to other competitors, Snapchat exhibits minimal buffering across all regions, offering a smoother user experience than apps like Triller, which suffer from both slow startup times and extensive buffering. Snapchat demonstrates more consistent performance across all regions. Triller consistently shows the slowest startup times, especially on 2 Mbps tests, with buffering times exceeding 12 seconds in some locations, which negatively impacts its overall performance. Triller struggles to maintain a smooth and reliable video playback experience, both in terms of startup time and buffering, making it less optimized for users with lower internet connections.

YouTube has the highest VMAF (video quality) across all locations and network limitations, though its startup time can be inconsistent in some regions, such as Brazil, under 10 Mbps.

In terms of regional performance, most apps achieve faster and more stable playback when connected to servers in the UK and Germany. In contrast, locations such as Mexico City (Mexico), Jakarta (Indonesia), São Paulo (Brazil), and Dubai (the UAE) often experience higher latency and slower startup times across multiple apps.

Conclusion

In the research testing different VPN server locations, we observed minimal to no difference in video image quality across locations for short-form video apps such as Pinterest, TikTok, YouTube, Snapchat, Triller, and X. However, differences did emerge in the speed at which apps initiated video playback.

While some app-specific performance variations were noted depending on the VPN server location, at 10 Mbps, video playback generally started faster when connected to servers in London (UK) and Berlin (Germany), across all tested apps.

Under more constrained conditions — a 2 Mbps connection — this trend became less consistent. App behavior varied more significantly across VPN server locations, suggesting that performance in low-bandwidth environments depends heavily on each app’s optimization and adaptability to such conditions.

Is your video app fast enough for your users? Contact us and learn more about our video testing services and how we can help optimize your platform's video performance.

Appendix

This appendix includes:

• Screenshots of speed tests, demonstrating network limitations in all tested locations.
• Screenshots of What is My IP Address, showing IP addresses and corresponding locations.

Speed tests at 10 Mbps

Speed tests at 2 Mbps

What is my IP address