A Complete Guide to Negative Testing in Software Testing

Good QA means not just covering the basic requirements but also thinking of the unexpected. If focus is only on happy-path testing, we might run into a situation where a critical bug that is outside of the normal testing area is missed and released to production for users to discover. Such missed areas might make the software crash, cause failures, or unexpected vulnerabilities, leading to major financial or reputational consequences.

In an industrial experiment, it was reported that negative test cases make up only 29% of the test cases, but might reveal as much as 71% of the defects found by all test cases, highlighting the importance of negative testing and positive testing bias. 

In the following guide, we will explore and learn more about negative testing, the different types and techniques that can be used, as well as practical examples that can be applied in your future projects. Applying the methods covered in this article should give you more confidence in the software that is being tested and its ability to smoothly handle improper behaviors and circumstances.

Negative testing 101

Before we dive deeper, it’s important to understand why negative testing is a crucial part of the QA process. In the following sections, we’ll explore its purpose, practical examples, and how it helps ensure software remains reliable and secure even when users—or bad actors—try unexpected or incorrect actions.

How does negative testing differ from positive testing?

In simpler terms, in positive testing, we are checking if the software is able to perform its intended functionality using valid inputs and normal conditions. Such testing confirms “happy path” scenarios.

Whereas in negative testing, we’re approaching this from a different perspective - we’re verifying that the software can handle invalid or incorrect inputs and conditions outside of the norm without the software crashing or exposing various vulnerabilities. The main aim is to gain confidence in error handling and ensure that the software rejects invalid inputs, provides useful error messages, or prevents incorrect behaviors.

Why do we need negative testing?

Negative testing is just as important as positive testing. Let’s imagine a banking application where users are able to send money between accounts. Testers have done their “happy path” testing, but as it turns out, users can send money requests with negative amounts, leading to incorrect bank statements and possible application crashes. In this scenario, some negative tests regarding money amount input fields accepting only positive integers could help cover this gap.

As we are checking if users are able to use the main functionalities and fulfill various expected tasks using the software, it’s equally important to ensure that the product does not break upon first use outside of the “happy path” - users don’t always follow our expected behaviors.

As we know, users might try to log in with incorrect credentials, buy something that’s already out of stock, or submit a contact form without providing all of the necessary details. These are more common scenarios where negative testing might provide the necessary confidence that the software is able to properly handle such situations, but we could also go into more advanced negative tests to prevent possible security exploits from bad actors.

Main benefits of negative testing 

When done right, negative testing can make a huge difference in how your software performs and how users experience it. Here’s why.

1. Improves software quality: Negative testing reveals defects that might have been missed with just positive testing.
2. Uncovers weaknesses in the software: These are ways someone could exploit the software using various invalid inputs, file and SQL injections, etc.
3. Gives confidence that the software can perform well in real-life situations: Negative testing ensures software can withstand a big load, or incomplete submissions.
4. Improves the user experience: Negative testing gives the chance to improve UX by verifying that errors are handled correctly and that proper guidance for the user is in place.

Different types of negative testing

We can get really creative when it comes to negative testing - this will vary depending on the software that is being tested, the defined requirements, and the time that is available for the testing. 

The most common negative testing types are:

• Invalid inputs: Using invalid inputs such as text in phone number fields, unsupported file types, special characters, leaving fields empty, etc., to see if the software handles such inputs properly - software remains stable and possibly displays an error message regarding the invalid input and what the user’s next steps should be.
• Stress testing: Checking how the software behaves when it needs to process an unusually high amount of data, especially high levels of network traffic, etc.

Getting started with negative testing

For positive test cases, we rely on the requirements to specify the expected results. For negative test cases, on the other hand, often such requirements are not defined, but are more implicit, deriving what should not be allowed. For example, users should not be able to log in to their profile if the password field is left empty—a scenario where we might rely on common sense and industry best practices for test case design rather than strictly defined requirements.

To achieve good negative test coverage, follow these steps:

1. Identify potential test scenarios 

Reviewing the existing “happy path” scenarios and deriving test cases from those, exploring areas where uncovered issues might bring the highest impact. Generally, defects tend to cluster, which indicates that a large number of software defects (close to 80%) can be found in a small number of software modules (around 20%), so it's good to focus efforts on areas that tend to be more problematic.

2. Define expected outcomes

Once we’ve sketched out the scenarios, we’re able to determine the expected outcomes - either from requirements documentation or using our experience in testing similar products. We can establish that an error message should be displayed if the user provided incorrect input for a field, or that a PDF file should not be accepted for a profile picture, and the user should be informed about the unsupported file type.

3. Design test cases

During negative test case design, we should keep in mind two things: how the user is informed and how the system behaves, so test cases should contain information on both of these aspects. It is also important to keep in mind the scope of testing and existing time constraints, as it won't be possible to design a test case for every possible combination. The desired coverage could be to cover every type of invalid condition, not every possible invalid value. 

4. Choose inputs

Based on previously identified test scenarios, we need to prepare the test data. These might be inputs that exceed or disregard the limitations that were defined in requirements, preparing files for upload and injection, preparing users with specific permission levels, etc.

Techniques for negative testing

Many techniques can be used in negative testing, and choosing which one is more appropriate and useful will depend on the software being tested, the testers' experience, available documentation, and other factors.

Error guessing

This method relies on the tester's experience and domain knowledge. The tester defines specific conditions under which an error should be produced.

Advantages:

• Offers a lot of flexibility.
• Good when there’s limited time, and documentation might not be the best.
• Can uncover issues that might be missed by more formal techniques.

Disadvantages and difficulties:

• It might be more difficult for less experienced testers.
• Needs some prior knowledge or experience with the software under test.
• Not the best documentation or repeatability.
• Tester bias - as the tester might only check the areas they deem problematic.

Practical applications:

• Focusing on areas that had the most issues in previous releases or in similar software;
• Checking various input errors for the field when the requirements have not clearly defined acceptable values or limitations;
• Checking for behavior when a user uploads a corrupted file.

Boundary value analysis 

Testing involves checking if the software handles inputs that are at the border of the boundary correctly. In negative testing, we should focus on values just outside of the set boundaries.

Advantages:

• Provides clear coverage metrics (there is a set amount of boundary values).
• Can uncover edge cases where values just outside of the boundary are improperly handled.

Disadvantages and difficulties:

• Boundary values need to be correctly identified.
• If there are no clear boundaries, the technique is hard to apply.
• Can only be applied to numeric or ordered ranges.

Practical applications:

• Checking if the user can set a password that is 1 character too short or too long than required;
• Checking if it’s possible to upload a file that exceeds the upload limit size by 1MB.

Equivalence partitioning

Inputs are divided into valid and invalid partitions. During negative testing, focus is on values from the invalid partitions.

Advantages:

• Reduces the number of tests needed - we can pick one value from the invalid partition and assume that all values would be handled the same way.
• Good coverage metrics.
• Repeatable with clear documentation.

Disadvantages and difficulties:

• Need to correctly define partitions.
• Might miss incorrect handling at boundary values.

Practical applications:

• Checking if the phone number field accepts text.
• Checking if the user can upload an Excel file for their profile picture.
• Checking if an invalid email format is accepted.
• Checking if a user can log in to their account with an incorrect password.

Fuzz testing

Checking how the software acts when random, malformed, or unexpected inputs are given.

Advantages:

• Can highlight various crashes/security issues.
• Might reveal bugs that would be missed with the standard approach.
• Best for parser, memory, and input sanitization bugs.

Disadvantages and difficulties:

• Not the best at finding logic or semantic bugs.
• Resource and time costs.
• Sometimes reproducibility could be difficult.

Practical applications:

• Uploading a file with a malicious payload in metadata.
• Generating thousands of random uploads via a script to stress-test.

State transition testing

Checking various state changes and sequences. During negative testing, focus is on invalid transitions and invalid states.

Advantages:

• Can uncover more real-life defects (connected to how users use the software).
• Provides confidence that users can’t perform invalid transitions or that such scenarios are handled without breaking the software.
• Good for sequence-based validation.

Disadvantages and difficulties:

• Need to correctly define valid and invalid state transitions (state model).
• It can’t be used for static input validation.

Practical applications:

• Checking if the user can access the bank account balance after inputting an incorrect PIN.
• Checking if the user can access their profile after they’ve logged out (‘back button’).

Exploratory testing

Exploratory testing is checking the software under test without predefined test cases, mostly relying on domain knowledge.

Advantages:

• Good when there’s limited time, and documentation might not be the best.
• Very adaptive to software changes.
• Might uncover real-life user behavior-related defects.

Disadvantages and difficulties:

• Unstructured approach, low repeatability.
• Effectiveness depends on the tester’s skill and knowledge.
• Coverage isn’t always clear.

Practical applications:

• Checking what happens if the file upload is canceled midway.
• Trying to edit input fields after clicking the ‘Save’ button.
• Rapidly submitting the contact form multiple times.

Fault injection testing

Checking how the software handles intentional errors, for example, different environment failures, network partitions.

Advantages:

• May expose more hidden system-level failures.
• Helps validate the system’s robustness under various failures.

Disadvantages and difficulties:

• Needs tool support.
• Setup and maintenance might be difficult and expensive.
• It might cause real outages of the system.

Practical applications:

• Check network failures during ‘form’ submission.
• Simulate database outages and how that affects different use cases.
• Uploading when disk storage is full.
• Simulating slow and unstable network connections during file upload.

Best practices and common pitfalls in negative testing

It is important to keep the scope in mind. As there are endless possibilities for negative tests and different inputs that could be used, we need to clearly define the goals and objectives for the negative testing that we wish to perform. 

The objectives in this case can be to cover the negative scenarios and edge cases that might have the biggest impact on the software if left unchecked. By focusing our efforts on more high-risk areas, we’re able to improve the system's reliability and stability.

Negative testing could have saved the day (real-world example)

Unfortunately, there are real-world examples when a lack of negative testing has caused issues for the end user when critical bugs outside of the “happy path” have reached production. The positive in this is that we can learn from such instances, and it highlights the importance of including negative testing in product releases. Let’s look at one of the biggest and most recent examples - the CrowdStrike outage.

Approximately 8.5 million Windows devices were affected by this fault in the update. This outage affected many critical industries such as healthcare, emergency services, financial, and transportation industries, and others. Users were unable to use their machines as they were stuck in a boot loop or entered recovery mode.

Based on their root cause analysis, the main cause of the incident was a mismatch in input field count - their template required 21 inputs, while only 20 inputs were passed. Originally, their systems used wildcard matching for the 21st field, which masked the bug. On the newest template update, they included a non-wildcard matching criterion on the 21st field. This triggered an out-of-bounds memory read, leading to a kernel-level crash of Windows hosts. 

After this incident, CrowdStrike has introduced preventative measures, one of which is additional testing procedures using testing types such as fuzz testing and fault injection, which were previously covered. They also hired third-party security and QA reviewers to audit both the affected code and their end-to-end development and deployment process to detect any gaps.

Negative testing could have been the exact safety net to catch the CrowdStrike issue before it went global, because the problem wasn’t with “normal” inputs, but with an unexpected input count. 

Conclusion

When creating test plans, we can’t only focus on positive or negative test cases. There needs to be a good balance between both. It is important to not just check that the software provides basic functionality but is also reliable and resilient to various out-of-the-box conditions and scenarios. Both of these are tools that testers need to utilize to provide a product that meets the acceptable standards of functionality and reliability.