How we test VPNs
Nowadays, using the correct digital security tools is crucial because so much of our daily life is intertwined with the web. While we must go online for work or entertainment, the internet is rife with all manner of malicious software waiting to ruin our lives. And even if we do everything right to avoid digital viruses, we can’t stop an unexpected data leak of a service we use, which can expose our personal information.
Thankfully, plenty of cybersecurity products are available that can stop malware, mask our online activities, and notify us about data breaches. To be precise, you need to employ at least a VPN (Virtual Private Network), antivirus software, and a password manager to stay safe online. However, the variety of software options makes finding the most effective tool for the job difficult. Most importantly, using unreliable or untested cybersecurity products will create a false sense of security. Plus, it’s well-known that many online services tend to overpromise and underdeliver.
As such, we at Cybernews conduct a wide range of hands-on tests to uncover the best digital protection tools currently available on the market. We aim to verify whether the most popular security software is as good as it claims. In this article, we’ll focus on how we test VPN services, which features we prioritize, and which criteria you should pay attention to when choosing your next potential purchase.
To accurately evaluate VPN services, we run extensive tests to determine connection speed results, leak prevention capabilities, kill switch effectiveness, unblocking abilities, and ease of use. We rate VPNs by their overall features, but you will find articles more tailored to specific user needs, such as VPNs for streaming, torrenting, or businesses.
Our reviews aim to combine methodological research and personal insights for a comprehensive, all-rounded analysis. Read on to learn more about how the Cybernews research team evaluates specific fields so you can be sure you can make an informed decision after reading our VPN articles.
Why you should trust our reviews
Before we begin, you might wonder why you should read Cybernews over other online publications and whether we’re a trustworthy source for getting the latest scoop about the cybersecurity industry. Well, here’s a brief explainer of why you should trust us and our findings:
- Independent research. We conduct various in-house tests and evaluations to get hands-on experience with the products we review. Additionally, we consult various reliable third parties, like dedicated cybersecurity testing labs, to corroborate our findings.
- Expert reviewers. We have a dedicated and experienced team that focuses on conducting research and evaluating the products we review. They have access to various devices and virtual testing environments to get a well-rounded picture of every product’s overall effectiveness.
- Original research. We try to develop new and innovative ways to improve our testing efficiency, effectiveness, and reliability.
- Up-to-date information. We constantly re-evaluate existing products to ensure all the information we provide is up to date.
Introduction to what we evaluate
VPNs are complex tools that require thorough evaluation from multiple angles. For starters, we begin by evaluating a VPN’s foundational features, such as encryption standards, tunneling protocols, and surface-level features. After that, we can see if the provider has potential and it’s worth conducting further research into its real-life connection speed results, privacy practices, geo-block removal capabilities, ease of use, and other areas. Here's a quick breakdown of what specific areas we test:
The foundation of VPN security
Our evaluation process begins with an overview of the VPN’s essential technical specs. These details are available to the public via security white papers and user manuals. We begin by checking these aspects because they ensure everyday online security while using the VPN:
- Encryption. VPNs, by definition, must employ encryption to keep user activities secure and confidential. Nowadays, the most widely accepted standard is AES-256, which provides excellent protection against intrusions without impeding performance. On the other hand, ChaCha20 is an upgraded version that grants similar protection levels with better performance. Any VPN that uses weaker encryption is considered insecure by modern standards.
- Hash. Hashing goes hand-in-hand with encryption to provide solid connection privacy. Like encryption, it comes in varying degrees of effectiveness, with SHA-2 being the modern-day standard for overall security. The SHA-2 standard can be further categorized into SHA-256, SHA-384, SHA-512, etc. Overall, we consider SHA-256 to be enough for comprehensive digital protection.
- Server type. The two most common VPN server types are traditional hard drive servers or RAM-only ones. The latter are superior because they store information in volatile memory, making it almost impossible to keep anything for the long term. To be precise, most of the information on RAM servers disappears once the servers are unplugged or rebooted. Meanwhile, hard drive servers store information indefinitely or until it gets overwritten by other data. These are crucial factors in rare cases when the authorities seize VPN server hardware for investigation.
- Server quantity. Regarding VPN servers, having more is better because it grants more IP address variety and ensures users don’t overload a handful of servers with excess traffic. Similarly, having more country options is beneficial because users can access more foreign content and find servers near their actual location for optimal speed.
- Tunneling protocols. Tunneling protocols are the means of communication between different networks. In short, they define how internet data packets travel from one place to another. The VPN tunneling protocol also determines speed, data integrity, security, and other factors. There are many protocol options in the VPN industry, and while some are reliable (WireGuard, OpenVPN, IKEv2), others (L2TP/IPSec, LLTP) are considered obsolete. Additionally, some services develop proprietary protocols based on open-source variants, while others build something from scratch. It’s difficult to determine whether in-house protocols are better or worse because their source code is hidden. However, some are audited by independent third parties, thus making them more reliable.
- Dark web monitoring. This feature is uncommon among VPN services but is essential for overall cybersecurity. As the name implies, it scans the dark web for leaked databases with the customer’s account credentials and notifies if anything is exposed. Naturally, the feature also suggests a course of action for fixing this issue.
- Additional security features. A robust VPN service must include extra features to protect its customers from various security issues. For starters, a kill switch is necessary to keep your actual IP address hidden in case the VPN connection drops. Additionally, you’ll need DNS leak protection to ensure your whereabouts are 100% masked while the VPN is enabled. And let’s not forget all the other non-essentials that can vastly improve your online experience, such as ad and tracker blockers, split tunneling, specialty servers, and more.
Leak protection and kill switch testing methodology
The IP, DNS, and WebRTC leak tests are done on the IPLeak website with the internet kill switch and other related features like DNS leak protection turned on.
We usually perform these tests using the DNS leak test website by choosing the Extended test, which performs more queries (36) than the Standard one (6) and works by sending several domain names for the VPN to resolve.
The efficiency of the kill switch is evaluated by manually terminating the VPN connection and imitating unexpected trouble with the server (for example, using Task Manager to end the VPN process). After that, we use the same online tools to check if the internet connection is terminated or the data is leaked.
The results here don’t have any numeric evaluation as the only answers to the question [Does a VPN leak IP/DNS/WebRTC?] are yes or no.
VPN privacy essentials
VPNs are primarily a privacy product because they hide your online activities from unwelcome onlookers, such as your ISP, network administrators, public Wi-Fi patrons, threat actors, and more. However, it’s crucial to ensure the VPN provider you use isn’t tracking your activities in any way. Therefore, we must check its overall privacy details, such as:
- Independent audits. VPNs can employ third parties to evaluate their hardware, software, infrastructure, privacy practices, and other areas to prove their commitment to security and privacy. It adds a layer of reliability to a VPN provider because an independent entity verifies its no-logging or other security claims.
- VPN provider’s country. Where a VPN service is located impacts the data retention laws (or lack thereof) it must follow. Some countries, such as Fourteen-Eyes Alliance member states, may require online businesses and ISPs to store long-term data so government authorities can access it when necessary. However, neutral or privacy-conscious regions don’t have such laws and thus are ideal jurisdictions for VPN services. However, these data-retention laws aren’t that relevant if the service uses technology that can’t store data long-term.
- Anonymous payment methods. Some privacy enthusiasts will appreciate the option to purchase a VPN anonymously to cover their digital tracks. Cryptocurrencies are the most popular confidential payment method, although some VPNs even support cash payments under specific circumstances.
How we perform VPN speed tests
We test VPN provider speeds by comparing our default internet download/upload speeds and latency (ping) with the rates obtained with the established VPN connection.
We usually connect to the server closest to the original location and, from this formula [VPN speed (Mbps) / original speed (Mbps) * 100% = x%], get the speed retention percentage. Naturally, we measure connection speeds of several different servers in more popular countries (like the US, UK, Canada, Germany, France, Australia, and others) to check if the performance is more or less consistent across regions.
For speed testing, we use an online tool called Speedtest by Ookla. We are looking for 75% and more for speed retention to ensure the speed performance of a VPN is sufficient.
Unblocking geo-restricted content
As you can imagine, we test VPN provider geo-block removal capabilities by connecting to VPN servers in different countries and attempting to access region-exclusive content on various online services. Unsurprisingly, we prioritize popular streaming services like Netflix, Hulu, Amazon Prime Video, BBC iPlayer, and many more. During this time, we considered how often we had to switch servers before unlocking our desired movies and TV shows. Additionally, we perform these tests on various devices to ensure everything works well across all platforms.
We track the number of foreign content libraries a VPN can unblock and rank the providers by that data. However, it’s crucial to note that a VPN’s geo-unblocking performance can change over time due to various factors. Therefore, our ranking metrics aren’t very strict.
Besides movies and TV shows, we also check if a VPN manages to unblock various websites that are blocked or censored by certain governments. This includes online news publications, social media sites, and messaging apps. Obviously, we can’t travel to countries like China and Russia to manually check if the VPNs work as intended. Instead, we check user feedback on sites like Reddit to get a general performance overview.
When testing VPN software usability, we separate the tests into desktop apps, mobile apps, and browser extensions. We also split the mobile and desktop apps based on the operating system because implementations can vary drastically.
During our tests, we looked into how easy it is to download and set up the applications. After setting up the software, we move on to the interface, how easy it is to navigate, and whether the apps are attractive, modern, or filled with unnecessary graphics.
Only then can we move further into the features available within the app and how well they work. That also includes app settings, customizability, if there’s a map interface, server selection, and other features.
To test the VPN service customer support, we start by checking what support channels are available. We primarily look for live chat, email, an extensive knowledge base, and FAQs.
After noting down all communication channels, we test the quality of customer support responses/information and the time it takes to receive a reply (when talking about live chat or email support).
To evaluate the user experience of the VPN apps, we use these criteria:
- Accessibility. A good VPN should be approachable to users of all skill levels. That means the UI should be straightforward, while more complex features must include explanations of what they do.
- Map interface. We found that including a map interface makes the server connection process easier and more appealing to the average user. After all, there’s no need to type in the country name since you can simply navigate to your preferred area.
- Feature descriptions. As mentioned previously, the VPN’s tools should be accompanied by descriptions explaining what the feature does. Naturally, the explainer should be simple enough so that even tech-illiterate users can understand.
- Settings. The settings menu should be extensive so power users can customize the VPN experience to suit their specific needs. At the same time, the default configuration should be good enough so the VPN would secure users who don’t make any changes.