Orthogonal Array for Testing
The job of a software tester is to try break the system in every possible way so that all faults are detected, thereby increasing the likelihood of delivering fault-free software to the customer. In a fast paced industry, the main challenges to the testing community are:
• To reduce cycle time of the test phase
• To find maximum defects with minimal test cases
• To offer maximum coverage with minimum test cases
The testing process/ test strategy should be efficient and effective to help face these challenges. Orthogonal Array testing strategy is one statistical method pioneered by Prof Taguchi that helps achieving a comprehensive and representative suite of test combinations.
2. ORTHOGONAL ARRAY TESTING STRATEGY (OATS)
OATS is a systematic statistical way of testing pair-wise interactions. It has been found useful in testing combinations of configurable options (variables).
An orthogonal array has specific properties. First, an OA is a rectangular array or table of values, presented in rows and columns. Each column represents a variable or parameter. Here is some terminology for working with orthogonal arrays:
Runs The number of rows in the array. This directly translates to the number of test cases that will be generated by the OATS technique.
Factors The number of columns in an array. This directly translates to the maximum number of variables that can be handled by this array.
Levels The maximum number of values that can be taken on by any single factor. An orthogonal array will contain values from 0 to Levels-1.
Orthogonal arrays are most often named following the pattern - LRuns (LevelsFactors)
2.1. How Does OATS Help Optimize Testing?
• Creates an optimized test suite with lesser test cases, and uncovers most of the bugs
• Detects all single mode, double mode, and multimode faults*.
• Guarantees testing the pair-wise combinations of all the selected variables.
• Exercises some of the complex combinations of all the variables.
• Is simpler to generate and is less error prone than test sets created manually.
• Helps in productivity improvement with cycle time reduction.
• Is independent of platforms and domains.
* Detect and isolate all single-mode faults: A single-mode fault is a consistent problem with any level of any single parameter. For example, if all cases of factor A at Level A1 cause error conditions, it is a single-mode failure.
Detect all double-mode faults: If there exists a consistent problem when specific levels of two parameters occur together, it is called a double-mode fault. Indeed, a double-mode fault is an indication of pair-wise incompatibility or harmful interaction between two test parameters.
Multi-mode faults: Many multimode faults are also there and can be understood by studying the geometric view of the test cases.
2.2. How to Use This Technique?
The OATS technique is simple and straightforward and can be customized based on available time and known problems. The first step in constructing an orthogonal array to fit a specific case is to calculate the minimum number of tests that must be performed -- this is also called degrees of freedom. (The degrees of freedom associated with interaction between two factors A and B, are given by the product of the degrees of freedom for each of the two factors, [degrees of freedom for A] x [degrees of freedom for B])
1. Decide how many independent variables will be tested for interaction. This will map to the Factors of the array.
2. Decide the maximum number of values that each independent variable will take on. This will map to the Levels of the array.
3. Find a suitable orthogonal array with the smallest number of Runs. (Suitable array is one that has at least as many Factors as needed and has at least as many levels for each of those factors as decided in.)
4. Map the Factors and values onto the array.
5. Transcribe the Runs into test cases, adding any particularly suspicious combinations that are not generated.
For example, assume that we have a system with 5 independent variables (P, Q, R, S, and T). Variables P and Q each have three possible values (P1-3 and Q1-3). Variables R and S each have four possible values (R1-4 and S1-4), and variable T has six possible values (T1-6). To test all of the possible combinations, it would take 864 test cases (3 x 3 x 4 x 4 x 6 = 864). Using an L18(3661) OA can significantly reduce the number of test cases to just 18 to test all the pair-wise combinations.
3. CASE STUDIES USING OATS
3.1. Example :
Application that contains a Configure the scheme section with three main subsections Scheme selection, Start & End date and Contribution to the fund. The Configure the scheme section also takes an instance of Investment calculation section with subsections Investment Amount, Number of units and Term of retirement.
To exhaustively test all combinations of the above sections in Administrator Application, it would take 27 test cases (for validating the three sections with three subsections). In real time project scenario, the assigned representative has developed approximately 26 test cases to test this functionality. Average time taken for each test case generation is 20 minutes. Also, this interaction is probably a very small portion of the entire system being tested.
1. Three independent variables are selected for interaction. These are the Factors of the array.
2. A maximum of three values exist that each independent variable can take on.
3. An L9 (34) orthogonal array is decided for this testing.
An L9 (34) Orthogonal array –
(Where three levels of each factor is represented as 0, 1, 2)
Factor 1 Factor 2 Factor 3 Factor 4
Run 1 0 0 0 0
Run 2 0 1 1 2
Run 3 0 2 2 1
Run 4 1 0 1 1
Run 5 1 1 2 0
Run 6 1 2 0 2
Run 7 2 0 2 2
Run 8 2 1 0 1
Run 9 2 2 1 0
4. When mapping the factors and values onto the array,
An L9 (34) orthogonal array –
(Where three levels of each factor is represented as 0, 1, 2)
Configure the scheme (A) Issuing the policy – (B) Investment calculation – (C) Factor 4
TC-1 A0 B0 C0
TC- 2 A0 B1 C1
TC- 3 A0 B2 C2
TC- 4 A1 B0 C1
TC- 5 A1 B1 C2
TC- 6 A1 B2 C0
TC- 7 A2 B0 C2
TC- 8 A2 B1 C0
TC- 9 A2 B2 C1
5. There are no "left over" Levels; however there was an extra Factor in the original array. This factor can simply be ignored; it does not change the properties of the test set generated from the array.
6. Taking the test case values from each run, there should be nine test cases. Any particularly suspicious combinations that are not generated can be included additionally.
Results: OATS technique significantly reduces the number of test cases to 9 in this project, thereby considerably reducing generation, execution, and overall cycle time by around 65%. Also, in traditional method the coverage is mapped on the basis of Bidirectional Traceability Matrix (BDTM), and this might involve the chance of missing some of the combinations. But orthogonal array based testing employs the method of pair wise combinations and maximizes the coverage.
Orthogonal array based testing considerably reduces the overall test cycle time and product cycle time. This method is found effective in arriving at an efficient and effective test strategy, producing an optimum suit of representative but exhaustive combination of test cases, and optimizing the productivity.
• Phadke, M.S. "Planning Efficient Software Tests" CrossTalk Journal of Defense Software Engineering
• Taguchi, Genichi. "System of Experimental Design" Edited by Don Clausing. New York: UNIPUB/Krass International Publications, Volume 1 & 2, 1987
• Jeremy M. Harrell, “Orthogonal Array Testing Strategy (OATS) Technique" Seilevel, Inc
• Sloane, Neil J. A. A Library of Orthogonal Arrays. Information Sciences Research Center
Can you please elaborate the process for finding the number of runs in the above example for mixed element array.
Please can you explain how do we find number of runs for a mixed element array. An what is the formula to calcutale the permutation of levels in different columns.