What should be Taken into Account? Important Considerations for Software Testing
Black or White Box Testing
- Black Box Testing: Testing is limited to creating input values and evaluating the output values of the software.
- White Box Testing: Block box testing with reasonable effort is sometimes not possible. White box testing (modifying or checking internal information of the software) can help in this case.
Quality of the Test Cases
The definition of the test cases of software-, integration- or unit-tests is not trivial. In the end, the quality of a test does heavily dependent on the "engineering judgment" of the author.
However, there are methods that allow a structured, repeatable creation and assessment of the quality of tests:
Structured Definition of Test Cases based on Guidelines/ Check Lists
The tests are defined based on guidelines. These specify, for example, the aspects to be tested. Examples are:
- Define test cases based on the analysis of the functional requirements
- Define test cases to ensure function under all expected conditions (keywords: Boundary Values, Equivalence Classes)
- Define test cases to ensure robustness (e.g. bugs in the software or by users, unexpected environmental conditions)
- Define test cases based on expected failures (test critical aspects based on experience)
Analysis of Requirements Coverage
Establish traceability of test cases to requirements to analyze the coverage of requirements by tests in a structured way.
Analysis of Code Coverage
Modern test tools allow to capture the code coverage during test execution. This analysis is used to check whether the defined test cases will test the entire implemented functionality or not.
Depending on the type of tests and the desired quality, different metrics can be used., e.g.:
- Call- and Caller-Coverage: Proportion of the called functions respectively of the function calls during the test execution.
- Statement coverage: Percentage of statements/commands executed by the software during test execution.
- Decision-Coverage: Percentage of decisions made during test execution
- Modified condition/decision coverage (MC/DC): Proportion of active conditions for all decisions during test execution.
The Safety Net - the Combination Makes the Difference!
Basically, it must be assumed that errors will happen when creating software and that errors will also be overlooked when implementing a verification method.
Therefore, it makes sense to apply multiple verification methods at different levels (e.g. unit-level
software) in order to have a "safety net" that can detect potential errors. Depending on the required quality, this "safety net" can be coarse- or fine-meshed.