1. Introduction
Testing is an essential stage of the modern software development process. Regardless of the adopted development methodology, whether it is a traditional one, such as cascading, or an agile one, such as eXtreme Programming, tests are a mandatory part of the process.
However, not all tests are the same. There exist multiple categories of tests. These categories differentiate, for example, how the tests operate in a software (black-box and white-box) or which to test in the software (unit, integration, system, and acceptance).
In this tutorial, we’ll specifically study the black-box and white-box tests. First, we’ll have a brief introduction to tests in software development. Thus, we’ll specifically investigate what are and how white-box and black-box tests work. In the end, we’ll outline the explored concepts and compare white- and black-box tests in a systematic summary.
2. Testing in Software Development
Testing consists of the process of evaluating the functionalities of a software program. The testing process’s main objective is checking if the developed software behaves as it is supposed to.
This process typically inspects low-level units, checking if they were implemented as per their technical specification; and the integration of such units into a complete system, analyzing how they cooperate. Moreover, developers also test high-level routines, acting as the end users of the developed system.
Technically, we can execute tests manually or program them for automatic execution. The benefit of using automated tests is that they work as a checklist of the implemented functionalities and their correct operation in the system. Some development methodologies, such as Test Driven Development (TDD), consider creating automated tests even before starting to program the aimed system itself.
There are four clear benefits of testing software/systems during their development process:
- Cost-effectiveness: tests help to save money in the long term since errors or incompatibilities found in the early stage of a software development process are much cheaper to solve compared to identifying them in the late-development stage or after its release
- Security improvement: tests improve the trustability and security of a system. Executing them reduces the risk of backdoors, bugs, or other vulnerabilities being present in a released software
- High-quality products: executing tests avoids releasing software or systems with broken functionalities or misplaced configurations
- High customer satisfaction: employing UX/UI-driven tests to analyze the experience of the users, adjuting the software according to the results, is a manner to improve overall satisfaction with a released system
Taking all the presented background concepts into account, we’ll deeply explore white-box and black-box tests in the following sections.
3. White-box Testing
White-box testing is a technique that explores the input, output, and processing flows of an algorithm. It means that white-box testing requires access to the source code: the tests consider the implementation details of the operational routines of a given software. Due to that reason, white-box testing is also called transparent-box testing, glass-box testing, and code-based testing.
In short, white-box testing considers an internal perspective of software, commonly accessible only by the development team.
White-box testing evaluates several software aspects. Let’s see some of them:
- Execution flows (according to the inputs)
- Software results (taken as outputs)
- Potential security breaches
- Potentially broken data structures
- Loops stopping criteria
- The functionality of each implemented operation (methods or functions)
Thus, preparing and executing white-box testing routines requires much knowledge regarding the examined software. In addition to understanding the internal flows, the tester must also define the expected results and verify if they match the obtained ones with the tests.
Relevant advantages of white-box testing are the code optimization enabled by the results, the easy automation of tests, and the possibility of testing all the operational paths of an algorithm, even the ones not available for the end users. But, they are typically expensive and complex, and only the ones involved with the software development process can efficiently create and execute them.
There are multiple techniques of white-box testing. We’ll briefly investigate some of them in the following subsections.
3.1. Statement Coverage
The main idea behind statement coverage is executing every statement of a particular algorithm. Thus, a statement coverage test may require several executions of the same code in different scenarios and heterogeneous inputs.
The following figure shows an example of the statement coverage testing process through a simple flowchart:
3.2. Branch Coverage
In the branch coverage technique, every path from every decision point in an algorithm is taken. Again, it may require several executions of the same algorithm (potentially with different inputs) to complete a branch coverage test.
The figure next exemplifies the process of branch coverage testing:
3.3. Condition Coverage
The condition coverage tests the results of each conditional structure in a code, making the result true or false by manipulating the inputs. An alternative to condition coverage is multiple condition coverage, which tests all the input combinations for a logic expression in a conditional structure.
The flowchart sequence next exemplifies condition coverage testing cases:
4. Black-box Testing
Black-box testing is a technique to analyze software or systems through a code-agnostic perspective. It means that the tester does not have access to the algorithm (source code) that composes the software on tests, operating it typically through some interface. Due to these characteristics, we can also call such testing process behavioral testing.
In summary, black-box testing gets an external perspective of software, focusing on the outputs received given particular inputs.
There exist several types of black-box testing, but the ones that focus on the functional and non-functional requirements of the software are the most relevant ones:
- Functional Testing: the objective is to verify if the software accomplishes all the functional requirements defined for it in the correct way
- Non-functional Testing: the objective consists of analyzing if the software on tests has an adequate level of performance, scalability, and usability, among other non-functional requirements
- Regression Testing: the objective is to guarantee that new functionalities added to the software are fully compatible with the other functionalities already on it, not generating any problems for them
So, to execute black-box testing, the tester must have much knowledge of the software specification, thus being able to evaluate if the results for specific inputs correspond to the expected ones (and are satisfactorily generated).
Some advantages regarding black-box testing are that the tester does not need much low-level technical knowledge to execute them (we use the end-users perspective to accomplish tests). Moreover, the tests are typically cheaper than white-box ones and easily reproducible. But, black-box testing may not reveal the causes of a failure, and we may not detect errors in the control structure of the software.
Let’s briefly explore some of the most know black-box testing techniques in the following subsections.
4.1. Equivalence Class Testing
A technique to identify the inputs that return the same results when submitted to test specific functionalities of software and systems. So, the idea is to cluster such inputs. We then organize the tests considering a sample of each generated cluster instead of all the input possibilities.
The figure next depicts, on a high level, this technique:
4.2. Boundary Value Testing
Boundary value testing is especially useful to test functions that receive inputs in a range of values. So, the idea here is to create a set of tests with the boundary values of such ranges: the minimum and maximum ones.
The following figure shows an example of the boundary value technique:
4.3. Cause-Effect Graphing
Cause-effect graphing establishes the relationship between provided inputs and resulting outputs. In this case, we consider inputs as causes and the outputs caught by providing a set of inputs as consequences.
The main idea is to connect inputs to outputs in a cause-effect graph. Next, the cause-effect graph is transformed into a decision table, which is employed to create a set of representative test cases for the software or system on tests.
The figure below depicts the previously presented technique:
5. Systematic Summary
As we saw in the previous sections, testing is essential to guarantee cost-effective programming projects, high-quality software products with operational security, and high-level customer satisfaction.
There are many categories of tests. In particular, white-box and black-box testing refer to how we interpret software and systems for executing tests on them.
White-box testing deeply investigates the processing flows defined in the software source code. Black-box testing, however, is source code-agnostic, analyzing the software through its operational behavior given a set of inputs.
Thus, white-box testing strongly relates to the developer’s perspective of software and systems. On the other hand, black-box testing assumes a perspective closer to the final user of a software program.
The following table summarizes and compares some relevant characteristics of white-box and black-box testing:
White-box Testing
Black-box Testing
Objective
Analyzing the operational behavior of the software being aware of how exactly the operations are executed
Analyzing the operational behavior of the software agnostically of how the operations are executed
Source Code
Required
Not required
Execution
Mostly done by the software developers
Mostly done by the software testers
Focus
On the algorithm logic
On the software behavior
Relevant Techniques
Statement Coverage, Branch Coverage, Condition Coverage
Equivalence Class Testing, Boundary Value Testing, Cause-Effect Graphing
6. Conclusion
In this tutorial, we studied white-box and black-box testing. Initially, we got a brief review of general concepts on testing. Thus, we analyzed details and techniques regarding white-box and black-box testing categories. Finally, we outlined and compared these categories of software/system tests in a systematic summary.
We can conclude that testing is crucial to guarantee high-quality software products. White-box and black-box testing are both really important to achieve this high quality. So, the discussion is not about which category of testing to adopt but which techniques of each category are the best ones to be executed in particular projects.