Software Testing Techniques

Once the software is developed it should be tested in a proper manner before the system is delivered to the user. For this, two techniques that provide systematic guidance for designing tests are used. These techniques are discussed here.

1. Once the internal working of software is known, tests are performed to ensure that all internal operations of the software are performed according to specifications. This is referred to as white box testing.
2. Once the specified function for which the software has been designed is known, tests are performed to ensure that each function is working properly. This is referred to as black box testing.

White Box Testing

White box testing (also called structural testing or glass box testing) is performed to test the program internal structure. To perform white box testing, the tester should have a thorough knowledge of the program internals along with the purpose of developing the software. During this testing, the entire software implementation is also included with the specification. This helps in detecting errors even with unclear or incomplete software specification.

The goal of white box testing is to ensure that the test cases (developed by software testers by using white box testing) exercise each path through a program. That is, test cases ensure that all internal structures in the program are developed according to design specifications. The test cases also ensure the following.

1. All independent paths within the program have been exercised at least once.
2. All internal data structures have been exercised.
3. All loops (simple loops, concatenated loops, and nested loops) have been executed at and within their specified boundaries.
4. All segments present between the control structures (like ‘switch’ statement) have been executed at least once.
5. Each branch (like ‘case’ statement) has been exercised at least once.
6. All the logical conditions as well as their combinations have been executed at least once for both true and false paths.

Various advantages and disadvantages of white box testing are listed in Table.

              Table Advantages and Disadvantages of White Box Testing

The effectiveness of white box testing is usually measured in terms bf test or code coverage metrics, that is, the fraction of code exercised by test cases. Various types of testing, which occur as part of white box testing are basis path testing, control structure testing, and mutation testing.

Basis path testing enables to generate test cases such that every path of the program has been exercised at least once. This technique is used to specify the basic set of execution paths that are required to execute all the statements present in the program. Note that with the increase in the size of the software the number of execution paths also increase, thereby degrading the effectiveness of basis path testing.

Creating Flow Graph

A flow graph represents the logical control flow within a program. For this, it makes use of a notation.

A flow graph uses different symbols, namely, circles and arrows to represent various statements and flow of control within the program. Circles represent nodes, which are used to depict the procedural statements present in the program. A sequence of process boxes and a decision box used in a flowchart can be easily mapped into a single node. Arrows represent edges or links, which are used to depict the flow of control within the program. It is necessary far every edge to end in a node irrespective of whether it represents a procedural statement. In a flaw graph, the area bounded by edges and nodes is known as a region. In addition, the area outside the graph is also counted as a region while counting regions. A flow graph can be easily understood with the help of a diagram.

Note that a node that contains a condition is known as predicated node, which contains one or more edges emerging out of it.

Finding Independent Paths

A path through the program, which specifies a new condition or a minimum of one new set of processing statements, is known as an independent path. For example, in nested ‘if’ statements there are several conditions that represent independent paths. Note that a set of all independent paths within a program is known as its basis set.

A test case is developed to ensure that while testing all statements of the program get exercised at least once.

P1: 1-9

P2: 1-2-7-8-1-9

P3: 1-2-3-4-6-8-1-9

P4: 1-2-3-5-6-8-1-9

Where P1, P2, P3, and P4 represent different independent paths present in the program.

To determine the number of independent paths through a program, the cyclomatic complexity metric is used that provides a quantitative measure of the logical complexity of a program. The value of this metric defines the number of test cases that should be developed to’ ensure that all statements in the program get exercised at least once during testing.

Cyclomatic complexity of a program can be computed using any of the following three methods.

• By counting the total number of regions in the flow graph of a program. For example, there are four regions represented by R1, R2, R3, and R4; hence, the cyclomatic complexity is four.
• By using the following formula.

CC = E – N + 2

Where

CC = the cyclomatic complexity of the program

E = the number of edges in the flaw graph

N = the number of nodes in the flaw graph.

For example, E = 11, N = 9. Therefore, CC = 11 – 9 + 2 = 4.

• By using the following formula.

CC= P + 1

Where

P= the number of predicate nodes in the flow graph.

For example, P = 3. Therefore, CC = 3 + 1 = 4.

Deriving Test Cases

In this, basis path testing is presented as a series of steps and test cases are developed to ensure that all statements within the program get exercised at least once while performing testing. While performing basis path testing, initially the basis set (independent paths in the program) is derived. The basis set can be derived using the steps listed below.

1. Draw the flow graph of the program: A flow graph is constructed using symbols previously discussed. For example, a program to find the greater of two numbers is given below.

procedure greater;

integer: x, y, z = 0;

1. enter the value of x;
2. enter the value of y;
3. if x > y then
4. z = x;

else

5. z = y;
6. end greater

2. Compute the cyclomatic complexity: Thecyc10matic complexity of the program can be computed using the flow graph depicted.

CC = 2 as there two regions R1 and R2

or

CC 6 edges – 6 nodes + 2 =2

or

CC 1 predicate node + 1 = 2.

3. Determine all independent paths through the program: For the flow graph depicted the independent paths are listed below.

P1: 1-2-3-4-6

P2: 1-2-3-5-6

4. Prepare test cases: Test cases are prepared to implement the execution of all independent paths in the basis set. The program is then tested for each test case and the produced output is compared with the desired output.

Graph matrix is used to develop software tool that in turn helps in carrying ‘out basis path testing. It is defined as a data structure used to represent the flow graph of a program in a tabular form. This matrix is also used to evaluate the control structures present in the program during testing.

Graph matrix is a square matrix of the size NxN, where Nis the number of nodes in the flow graph. An entry is made in the matrix at the intersection of i^th row and j^th column if there exists an edge between i^th and j^th node in the flow graph. Every entry in the graph matrix is assigned some value known as link weight. Adding link weights to each entry makes the graph matrix a useful tool for evaluating the control structure of the program during testing.

In the flow graph, numbers and letters are used to identify each node and edge respectively. A letter entry is made if there is an edge between two nodes of the flow graph. For example, node 3 is connected to the node 6 by edge d and node 4 is connected to node 2 by edge c, and so on.

Control Structure Testing

Control structure testing is used to enhance the coverage area by testing various control structures (which include logical structures and loops) present in the program. Note that basis path testing is used as one of the techniques for control structure testing. Various types of testing performed under control structure testing are condition testing, data-flow testing, and loop testing.

Condition Testing

In condition testing, the test cases are derived to determine whether the logical conditions and decision statements are free from errors. The errors presenting logical conditions can be incorrect Boolean operators, missing parenthesis in a Boolean expression, error in relational operators, arithmetic expressions, and so on.

The common types of logical conditions that are tested using condition testing are listed below.

1. A relational expression such as E1 op E2, where E1 and E2 are arithmetic expressions and op is an operator.
2. A simple condition such as any relational expression preceded by a NOT (~) operator for example, (~E1), where E1 is an arithmetic expression.
3. A compound condition, which is formed by combining two or more simple conditions using Boolean operators. For example, (E1 & E2) | (E2 & E3), where El, E2, and E3 are arithmetic expressions and & and | represent AND and OR operators respectively.
4. A Boolean expression consisting of operands and a Boolean operator such as AND, OR, or NOT. For example, A|B is a Boolean expression, where A and B are operands and | represents OR operator.

Condition testing is performed using different strategies, namely, branch testing, domain testing, and branch and relational operator testing. Branch testing executes each branch (like ‘if statement) present in the module of a program at least once to detect all the errors present in the branch. Domain testing tests relational expressions present in a program. For this, domain testing executes all statements of the program that contain relational expressions. Branch and relational operator testing tests the branches present in the module of a program using condition constraints. For example,

if a > 10

then

print big

In this case, branch and relational operator testing verifies when the above code is executed, it produces the output ‘big’ only if the value of variable a is greater than 10.

In data flow, testing, test cases are derived to determine the validity of variables definitions and their uses in the program. This testing ensures that all variables are used properly in a program. To specify test cases, data-flow-based testing uses information such as location at which the variables are defined and used in the program.

For performing data-flow testing, a definition-use graph is built by associating the program variables with nodes and edges of the control flow graph. Once these variables are attached, test cases can easily determine which variable is used in which part of a program and how data is flowing in the program. Thus, data-flow of a program can be tested easily using specified test cases.

Loop Testing

Loop testing is used to check the validity of loops present in the program modules. Generally, there exist four types of loops, namely, simple loop, nested loops, concatenated loops, and unstructured loops.

• Simple loop: Refers to a loop that has no other loops in it. Consider a simple loop of size n. Size n of the loop indicates that the loop can be traversed n times, that is, n number of passes are made through the loop. The steps followed for testing simple loops are listed below.

1. Skip the entire loop.
2. Traverse the loop only once.
3. Traverse the loop two times.
4. Make ill number of passes through the loop, where ill < n.
5. Traverse the loop n – I, n, n + 1 times.

5. • Nested loops: Loops within loops are known as nested loops. The numbers of tests required for testing nesting loops depends on the level of nesting. More is the level of nesting, more will be the number of tests required. The steps followed for testing nested loops are listed below.

1. Start with the inner loop and set values of all the outer to minimum.
2. Test the inner loop using the steps followed for testing simple loops while keeping the iteration parameters of the outer loops at their minimum values. Add other tests for values that are either out-of-range or are eliminated.
3. Move outwards and perform tests for the next loop while holding other nested loops to ‘typical’ values and the iteration parameters of the outer loops at their minimum values.
4. Continue performing tests until all the loops are tested.

1. Concatenated loops: The loops containing several loops that may be dependent or independent In case the loops are dependent on each other, the steps in nested loops are followed. On the other hand, if the loops are independent of each other, the steps in simple loops are followed.
2. Unstructured loops: Such loops of difficult to test; therefore, they should be redesigned so that the use of structured programming constructs can be reflected.

Mutation testing is a white box method where errors are ‘purposely’ inserted into a program (under test) to verify whether the existing test case is able to detect the error. In this testing, mutants of the program are created by making some changes in the original program. The objective is to check whether each mutant produces an output that is different from the output produced by the original program.

In mutation testing, test cases that are able to ‘kill’ all the mutants should be developed. This is accomplished by testing mutants with the developed set of test cases. There can be two possible outcomes when the test cases test the program either the test case detects the faults or fails to detect faults. If faults are detected, then necessary measures are taken to correct them.

When no faults are detected, it implies that either the program is absolutely correct or the test case is inefficient to detect the faults. Therefore, it can be concluded that mutation testing is conducted to determine the effectiveness of a test case. That is, if a test case is able to detect these ‘small’ faults (minor changes) in a program, then it is likely that the same test case will be equally effective in finding real faults.

To perform mutation testing, a number of steps are followed, which are listed below.

1. Create mutants of a program.
2. Check both program and its mutants using test cases.
3. Find the mutants that are different from the main program. A mutant is said to be different from the main program if it produces an output, which is different from the output produced by the main program.
4. Find mutants that are equivalent to the main program. A mutant is said to be equivalent to the main program if it produces the same output as that of, the main program.
5. Compute the mutation score using the formula given below.

M = D/ (N – E)

Where

M = Mutation score

N = Total number of mutants of the program.

D= Number of mutants different from the main program.

E = Total number of mutants that are equivalent to the main program.

6. Repeat steps 1 to 5 till the mutation score is ‘1’.

However, mutation testing is very expensive to run on large programs. Thus, certain tools are used to run mutation tests on large programs. For example, ‘Jester’ is used to run mutation tests on java code. This tool targets the specific areas of the program code such as changing constants and Boolean values.

Black Box Testing

Black box (or functional) testing checks the functional requirements and examines the input and output data of these requirements. When black box testing is performed, only the sets of ‘legal’ input and corresponding outputs should be known to the tester and not the internal logic of the program to produce that output. Hence to determine the functionality, the outputs produced for the given sets of input are observed.

The black box testing is used to find the errors listed below.

1. Interface errors such as functions, which are unable to send or receive data to/from other software.
2. Incorrect functions that lead to undesired output when executed.
3. Missing functions and erroneous data structures.
4. Erroneous databases, which lead to incorrect outputs when the software uses the data present in these databases for processing.
5. Incorrect conditions due to which the functions produce incorrect outputs when they are executed.
6. Termination errors such as certain conditions due to which a function enters a loop that forces it to execute indefinitely.

In this testing, tester derives various test cases to exercise the functional requirements of the software without considering implementation details of the code. Then, the software is run for the specified sets of input and the outputs produced for each input set is compared against the specifications to conform the correctness. If they are a23pecified by the user, then the software is considered to be correct else the software is tested for the presence of errors in it. The advantages and disadvantages associated with black box testing are listed in Table.

              Table Advantages and Disadvantages of Black Box Testing

Various methods used in black box testing are equivalence class partitioning, boundary value analysis, and cause-effect graphing. In equivalence class partitioning, the test inputs are classified into equivalence classes such that one input checks (validates) all the input values in that class. In boundary value analysis, the boundary values of the equivalence classes are considered and tested. In cause-effect graphing, cause-effect graphs are used to design test cases, which provides all the possible combinations of inputs to the program.

This method tests the validity of outputs by dividing the input domain into different classes of data (known as equivalence classes) using which test cases can be easily generated. Test cases are designed with the purpose of covering each partition at least once. A test case that is able to detect every error in a specified partition is said to be an ideal test case.

An equivalence class depicts valid or invalid states for the input condition. An input condition can be either a specific numeric value, a range of values, a Boolean condition, or a set of values. The general guidelines that are followed for generating the equivalence classes are listed in Table.

                       Table Guidelines for Generating Equivalence Classes

To understand equivalence class partitioning properly, let us consider an example. This example is explained in the series of steps listed below.

1. Suppose that a program P takes an integer X as input.
2. Now, either X < 0 or X >0.
3. In case X < 0, the program is required to perform task T1; otherwise, the task T2 is performed.
4. The input domain is as large as X and it can assume a large number of values. Therefore the input domain (p) is partitioned into two equivalence classes and all test inputs in the X < 0 and X > 0 equivalence classes are considered to be equivalent.
5. Now, as shown in Figure independent test cases are developed for X < 0 and X > 0.

In boundary value analysis (BVA), test cases are derived on the basis of values that lie on an edge of the equivalence partitions. These values can be input or output values either at the edge or within the permissible range from the edge of an equivalence partition.

BVA is used since it has been observed that most errors occur at the boundary of input domain rather than at the middle of the input domain. Note that boundary value analysis complements the equivalence partitioning method. The only difference is that in BVA, test cases are derived for both input domain and output domain while in equivalence partitioning test cases are derived only for input domain.

Generally, the test cases are developed in boundary value analysis using certain guidelines, which are listed below.

1. If an input condition specifies a range of values, test cases should be developed on the basis of both the values at the boundaries and the values that are just above and below the boundary values. For example, for the range -0.5≤X≤0.5, the input values for a test case can be ‘-0.4’, -‘0.5’, ‘0.5’, ‘0.6’.
2. If an input condition specifies a number of values, test cases should be designed to exercise the minimum and maximum numbers as well as values just above and below these numbers.
3. If an input consists of certain data structures (like arrays), then the test case should be able to execute all the values present at the boundaries of the data structures such as the maximum and minimum value of an array.

Equivalence partitioning and boundary value analysis tests each input given to a program independently. It means none .of these consider the case of combinations of inputs, which may produce situations that need to be tested. This drawback is avoided in cause-effect graphing where combinations of inputs are used instead of individual inputs. In this technique, the causes (input conditions) and effects (output conditions) of the system are identified and a graph is created with each condition as the node of the graph. This graph is called cause-effect graph. This graph is then used to derive test cases. To use the cause-effect graphing method, a number of steps are followed, which are listed below.

• 1. List the cause (input conditions) and effects (outputs) of the program.
  2. Create a cause-effect graph.
  3. Convert the graph into a decision table.
  4. Generate test cases from the decision table rules.

In order to generate test cases, all causes and effects are allocated unique numbers, which are used to identify them. After allocating numbers, the cause due to which a particular effect occurred is determined. Next, the combinations of various conditions that make the effect ‘true’ are recognized. A condition has two states, ‘true’ and ‘false’. A condition is ‘true’ if it causes the effect to occur; otherwise, it is ‘false’. The conditions are combined using Boolean operators such as ‘AND’ (&), ‘OR’ (I), and ‘NOT’ (-). Finally, a test case is generated for all possible combinations of conditions.

Various symbols are used in the cause-effect graph. The figure depicts various logical associations among causes c_i and effects e_i. The dashed notation on the right side in the figure indicates various constraint associations that can be applied to either causes or effects.

To understand cause-effect graphing properly, let us consider an example. Suppose a triangle is drawn with inputs x, y, and z. The values of these inputs are given between ‘0’ and ‘100’. Using these inputs, three outputs are produced, namely, isosceles triangle, equilateral triangle or no triangle is made (if values of x, y, z are less than. 60°).

1. Using the steps of cause-effect graphing, initially the causes and effects of the problem are recognized, which are listed in Table.

                                              Table Causes and Effects

2. A decision table (a table that shows a set of conditions and the actions resulting from them) is drawn as shown in Table.

                                                            Table Decision Table

3. Each combination of conditions for an effect in Table is a test case.

Differences between White Box and Black Box Testing

Although both the testing techniques are used together to test many programs, there are several considerations that make them different from each other. Black box testing detects errors of omission, which are errors occurring due to non accomplishment of user requirements. On the other hand, white box testing detects errors of commission which are errors occurring due to non-implementation of some part of software code. Other differences between these two techniques are listed in Table.

                    Table Differences between White Box and Black Box Testing

In this testing, it is checked whether the calculator is working properly by giving inputs by pressing the buttons in the calculator.

Gray Box Testing

Gray box testing technique is often defined as a mixture of black box testing and white box testing techniques. This technique is especially used in web applications. In this technique, the complete knowledge of the internal structure of the program is not required; rather partial knowledge that is, how the components of the program operate and interact is sufficient. This knowledge is used in deriving test cases. Some points noted in gray-box testing are listed below.

1. Gray box testing is platform and language independent.
2. The current implementation of gray box testing is heavily dependent on the use of a host platform debugger(s) to execute and validate the software under test.
3. It can be applied in real-time systems.
4. It utilizes automated software testing tools to generate test cases
5. Module drivers and stubs are created by automation means, thus saving time of testers.