How we test antivirus and security software

Antivirus software is essential in providing you with robust security from malware and other online threats. However, to cut through the noise in the multitude of antiviruses on the market, it’s important to know how effective the antiviruses are in fighting malware and other modern threats and how they affect your device’s performance. For this, we run multiple in-house tests using various tools to help you make an informed decision.

The testing process isn’t identical for every antivirus because the software differs from one another and has a unique feature package available. Therefore, the reviews may vary in content. However, all antiviruses are tested for malware detection.

When reviewing antivirus software, we look through the eyes of consumers and consider the following main factors:

• In-house testing results
  • Real-time protection
  • Virus scans
  • Web protection
  • Ransomware protection
  • Anti-phishing tools
  • Firewall
• Independent lab results
• Advanced detection techniques
• Additional features and protection
• Ease of use and user interface
• Pricing

Below you’ll find detailed explanations of the methodologies we use, how we gather data, and how we check and test each area.

Why trust Cybernews?

With many comparison and review sites online, it can be hard to find a trusted source of the most recent information. Our researchers maintain transparency, clearly outlining the criteria, methodology, and sources used. All affiliations are disclosed upfront to uphold transparency and accountability. The Cybernews team comprises seasoned professionals with vast experience in cybersecurity, software testing, and information technology.

We conduct reviews independently, free from undue influence or bias from external parties, ensuring impartial assessments of various security software. Our evaluations are based on objective criteria and empirical evidence rather than subjective opinions or personal preferences.

We follow strict testing protocols covering performance, security, usability, and value for money. Our methodologies are designed to provide a comprehensive assessment of all product’s capabilities.

Our comprehensive evaluations delve deep into each aspect of the software tested, leaving no stone unturned, including features, performance, security measures, and usability.

We maintain consistency in our review process, applying the same standards and criteria to all products tested. Users can rely on our reviews to provide consistent and reliable information, enabling them to make decisions confidently over time.

We stay in line with the latest developments in the cybersecurity landscape, ensuring that our reviews reflect the most current trends, threats, and advancements in security software.

Read more about how we test and review different types of cybersecurity products: How we test at Cybernews

Our in-house testing and methodology

Our Cybernews research team conducts real-life testing and analysis to evaluate how well the antivirus software performs. The testing covers multiple areas, including real-time protection, scanning options, usage of system resources, web protection, ransomware detection and removal effectiveness, phishing protection, firewalls, and additional features.

Testing environment

Based on the product and its compatibility, we test the software on different devices. Here’s what we use:

• Windows – we test on Virtualbox VM with Windows 11 (10 GB RAM, 120 GB dynamic HDD, and 2 CPU cores) installed on a Lenovo ThinkPad T14s Gen2 device (AMD Ryzen 5 PRO 5650U processor, 16 GB RAM, 256 GB SSD, Windows 11 Pro OS). Or, Virtualbox VM with Windows 11 (20 GB RAM and 4 CPU cores) on Lenovo ThinkStation (AMD Ryzen Threadripper PRO 3945WX 12-core processor, 3.99 GHz, 64 GB RAM, 3TB SSD, Windows 11).
• macOS – we test on Virtual Machine Apple M2 Pro (8 GB RAM, Ventura, 268 GB storage) setup on Apple M2 Pro (10 Cores, 16 GB RAM with latest Sonoma OS, and 500 GB SSD storage).
• Android – we test on Nokia 6.2 (model TA-1198) with Android 11, 64GB internal storage, 4GB RAM, CPU Octa-Core (4x1.8 GHz Kryo 260 Gold & 4x1.6 GHz Kryo 260 Silver).
• iOS – we test on iPhone 7 (model MN8X2), with iOS version 15.7.1., 32GB internal storage, 2GB RAM, CPU Quad-core 2.34 GHz (2x Hurricane + 2x Zephyr).

Real-time protection

Reat-time protection (RTP) is one of the most important features of antivirus software. It is responsible for catching threats instantaneously in real time, so it is crucial to test its capabilities. When testing RTP, we use various popular malware samples (ransomware, keyloggers, spyware, adware, etc.) residing in the wild and malicious files prepared in advance.

The live (in the wild) samples come from malicious URLs, which, by opening the link, triggers the malware to download.

The samples prepared in advance are downloaded by our research team from online malware libraries and are set up in a controlled environment imitating the World Wide Web (for this, we use the XAMPP tool to configure a virtual host.)

It’s important to mention that before we run our tests, we turn off all local malware-catching tools like Windows Security. If the antivirus software has a feature that is supposed to block malicious websites, it’s also turned off.

The process is simple – we download the malicious files (usually 10) and take notice of how many of these samples the antivirus in question can detect and remove.

Suppose the antivirus in question has different levels of real-time protection (in our case, only the NordVPN Threat Protection does – basic and cloud-based options). In that case, we investigate, compare, and evaluate those separately.

The score of these tests is the percentage of the malware that the antivirus detected and prevented from entering the system (device), so if 8 out of 10 malicious files have been cached, the score is 80% of the malware detection rate. It’s worth mentioning that a perfect antivirus does not exist, and a score of 100% is pretty rare.

If the software detects no less than 80% of threats, we classify this antivirus as good and efficient at catching malware.

It’s worth mentioning that we do not execute any malicious files (we don’t click on them) as you’d usually encounter them, but we use a safe virtual machine environment.

Scanning options

Another important feature of any AV is on-demand system scanners. The Cybernews research team investigates and evaluates all available scan options with the particular antivirus software – quick/smart, full, custom, etc.

Before testing, real-time protection is turned off on both the antivirus and the local system. Then, malware samples are put in a testing folder on the system, and separate antivirus scans are run.

We then evaluate each scanning option on:

• How long the scan took
• How many files in total have been scanned
• How many malicious samples have been detected
• How the antivirus scan impacted the device’s performance and CPU usage

We repeat quick scans more than once as these are developed to scan only those system directories that have the highest probability of being infected. So, the malicious files are planted accordingly (for example, in desktop, download folder, etc.)

For full system scans, the samples are usually placed deeper in the system (for example, somewhere on a Local Disk C:). We also consider whether the antivirus instantly removes or quarantines the detected malware from the system or leaves this choice to the user. The problem with the latter option is that it gives extra time for the malware to act and potentially cause more damage to the system.

The score is measured in the same way as with real-time protection – the percentage of detected malicious samples.

Web protection

Most antiviruses contain web protection features or additional browser extensions that protect from accessing dangerous or potentially malicious websites.

For the analysis of web protection features, we again use the previously mentioned malicious URLs. However, in this case, we investigate two types of links – URLs containing domain names and IPv4-based URLs. We click on these links and see how effectively the antivirus software responds.

We then evaluate how many of these websites have been detected and blocked by the antivirus. And, as with the other tests, the score is computed as percentages.

Ransomware protection

Ransomware protection is a crucial part of a strong antivirus solution. To test the ransomware protection effectiveness, we perform penetration testing (pen testing). We download ransomware files to test the antivirus software capabilities for detecting ransomware. We then run an antivirus scan on the system to see how well the AV has performed.

The antivirus performance is evaluated based on the percentage of ransomware files detected and removed, as well as overall ransomware protection-related features included in the antivirus software package. For example, Bitdefender has a Ransomware Remediation function, which backs up user’s files and reverses any damage done by ransomware by restoring the encrypted files.

Phishing protection

Phishing is another type of online threat you need strong protection from. Our in-house research team evaluates phishing protection by testing the anti-phishing features available within the antivirus suite. We test the antivirus’ capability to protect you from dubious phishing websites.

For example, Bitdefender offers a phishing protection feature, which checks web pages for phishing behavior – whether the page tries to obtain sensitive user data, like login credentials or credit card details.

Another feature we test is email or spam filters to see how successful they are with removing threats that come via emails. For example, ESET has an Antispam protection feature, which detects and removes spam and potential phishing emails.

Firewall

To test the antivirus firewall by evaluating how customizable it is, how it monitors network traffic to prevent unwanted access from the outside, and if there is an option also to safeguard the outgoing traffic. A good example is Norton’s smart firewall feature.

We test the monitoring of local applications, whether there is no network resource misuse, and whether the firewall can prevent port scanning (a common technique used by cybercriminals to look for potential weak points in the network.)

Independent lab tests

In addition to our in-house tests, we analyze the independent lab test results that regularly test different antivirus software on various platforms.

The three trusted independent labs we use for assessing antivirus products are:

• AV-Tests – independent organization testing and evaluating security software for Microsoft Windows, MacOS, and Android OS
• AV-Comparatives – an independent Australian company assessing antivirus software and regularly providing reports freely available to the public
• SE labs – a private and independently run testing company evaluating security services and products

These labs conduct comprehensive assessments using standardized methodologies to provide objective insights into the capabilities of different antivirus products. The independent labs test the antivirus software capabilities to detect malware, including known and zero-day threats, false positives, and effects on the device’s performance.

Every antivirus is rated and graded based on how they performed in protecting from malware and then grouped based on their results.

We use these results together with our in-house tests to come to a conclusion for an objective and real-life comparison. We also see whether our results match the independent labs’ ones. This way, we get a complete overview of how secure the antivirus software is and how much impact it has on your device.

Advanced detection techniques

We evaluate antivirus solutions that employ advanced detection techniques, such as signature-based scanning, heuristic analysis, and behavior monitoring. Signature-based scanning involves comparing files to a database of known malware signatures for detection. Heuristic analysis proactively identifies suspicious behaviors in files, such as code patterns or actions, even if they don’t match known signatures. Behavior monitoring observes real-time activities on the system to detect unusual behavior indicative of malware, such as unauthorized file modifications or network connections.

These methods complement each other to provide comprehensive protection against both known and emerging threats in antivirus software. Therefore, we check whether the antivirus program implements all of these methods or just some.

Evaluation of other features

Many antivirus services provide additional useful features on which we also perform tests. For example, some have system optimization features, like Bitdefender’s OneClick optimizer.

We evaluated it on how effectively it detected and removed redundant files from the device to improve its performance, how long it took, and how much space it freed up on the system. This is just one example of how we test additional antivirus features.

Another good example is McAfee and its Secure Apps feature, designed to find and install updates for Windows and local applications. We evaluate its performance by checking if the applications that require an update are detected and updated.

We also check and evaluate other extra antivirus features, like VPNs, password managers, or ad blockers. All of them are tested the same as if these services were separate. We test VPNs for their encryption, unblocking capabilities, server count, and many other features. Password managers are evaluated based on their encryption levels, multi-factor authentication options, logging policies, etc. Ad blockers are tested on their effectiveness in blocking ads and trackers on different platforms, like games, social media sites, or streaming services.

Testing user experience

When testing security products, we always consider the overall user experience and decide if the tool in question is, for example, convenient for beginners or non-technical users.

We evaluate the arrangement and accessibility of tools and features on the interface. For example, we check whether all the features can be found on the same application or if it’s required to go to the provider’s website for some of them. We check if there are any descriptions or tutorials provided on how to use the features and how complicated the settings are to configure.

We analyze the applications to see whether the interface is clear, not too crowded, or complicated. And vice versa – if the service is aimed at more tech-savvy users, we look for more extensive configuration options and setups.

Pricing

Here at Cybernews, we believe it’s important to know the pricing structure and transparency of ToS. After testing the feature and overall service effectiveness, we evaluate whether the antivirus service provider offers a good quality-to-price ratio based on its features, performance, and level of protection.

We also check whether the subscription options are flexible, have annual plans, how many multi-device connections are offered, etc. We check if the service provides a discount and if the price increase after the deal ends is clearly indicated before subscribing.

We always investigate if the security software provides a free trial or a money-back guarantee, as consumers often look at how to test the service before committing long-term. What’s more, we check how clearly it’s explained on how to stop the free trial before being charged or how easy it is to claim the refund.

Based on all of the above evaluations regarding pricing and subscription plans, we then rank the antivirus services. For example, if the service performs poorly but is fairly expensive, it will get a lower rating. And vice versa, if the service is cheap and also offers excellent protection rates, it will be one of the top providers on our list.

Our researchers

At Cybernews, our in-house research team of information security experts and tech enthusiasts conducts antivirus software tests. The team uses proprietary tools and industry-standard methods to evaluate various antivirus tools and their effectiveness. During testing, they also gain valuable insight into the user experience that will be extremely valuable to our readers.

The research team and writers work closely together to ensure our articles are accurate, up-to-date, and understandable to a broad audience. If you notice something missing, let us know.