CHALLENGES OF REGRESSION TESTING: A PRAGMATIC PERSPECTIVE

: The prevailing scenario of ever increasing dependency of human beings on software applications has built pressure on software organizations to produce quality software. The quality of software is determined by many factors. One of the vital factors in deciding the software quality is optimized usage of testing tools and techniques employed during regression testing. Extremely high complexity of regression testing makes it necessary to utilize optimized way of running a selected and minimized test suite. Still, various challenges like redundancy, repetition ratio, Recurrence ratio or missing functionalities during regression testing are common. This paper discusses about the problems and challenges encountered during regression testing.


I. INTRODUCTION
The significance of software can be easily perceived as it emerged as a powerful tool which finds its application in every field ranging from mobile phones to medical treatment covering almost all electronic gadgets on earth up to space i.e. satellite communication. Thus, the software has made a way of changing and facilitating the life of humans. If it has such an involvement, it must be developed in a well-defined and sequential way; otherwise, its quality will be compromised leading to an unreliable software. The software is thus developed after undergoing various phases of SDLC.
To ensure the quality of a software, testing is performed. The various testing techniques find numerous faults/errors, which are thus removed by various debugging approaches. One of the approaches to testing is exhaustive testing. It has a limitation that it can be performed only on very small programs. Running exhaustive testing on large programs keeping in mind time and cost constraint is not a feasible idea. Regression testing is about running the entire test ensemble again to ensure that amendments do not negatively affect the system.
Verification and validation of a software product is an essential step towards building quality software. Verification and validation process is also known as software testing. Software testing is performed both statically (verification) and dynamically (validation) to ensure that the software meets the customers' demands and expectations. [1][2] [3]. "Software testing is a process of executing a program with the intent of finding the software bugs".
Test cases are executed during software testing to find the failures (bugs). A test case is a collection of set of input, behavior and output conditions. A test case with same expected and observed output is considered as "pass". Any mismatch or discrepancy between expected and observed output makes a test case "fail" [4] [10]. Software testing still can show the presence of bugs or software failures, but it can never confirm regarding the absence of bugs because it is nearly impossible to completely test a software exhaustively due to resource and time constraints.

A. Regression Testing
Regression Testing is necessary during software development as the developers alter the code to rectify the bugs that have been reported during software testing; and regression testing ensures to capture and mitigate the undesired sideeffects of code amendments.  Figure 2 describes the importance of regression testing and depicts that the testing team needs to re-run the test suite after code amendments to make sure that the previously failed test cases are now passed after bug fixation (or code change) and status of previously passed test cases is still "pass" so as to capture the undesired side effect of software code change [5][9]. Regression Testing makes sure that "fixing one bug do not introduce several new bug" due to code change as shown in fig. 3   Test case prioritization approach increases test feasibility in the testing of software [3]. Software evolves with time, so the size of software test suite also increases which often makes it costly to execute. Numerous researchers have shown that regression testing is a costly process and therefore it requires most of the collective expenditure of the software.

A.1 Test suite minimization
This approach intends to eliminate the test cases, which becomes redundant with regard to coverage of some set of program requirement and therefore decreasing the number of test cases in a regression test suite. It clearly states that only a subset of the test suite is actually economical to use. Minimization is at times called "test suite reduction" denoting that the exclusion of test cases are everlasting.

A.2. Test case selection
This approach targets to pick the test cases from the original test suite, that focuses on testing the customized part of the software program. It does not remove test cases, rather it filters the test cases which are related to the customized portion of source code.

A.3. Test case prioritization
Prioritization approach is followed because execution of all the test cases is not feasible due to resource constraints. In this, the test cases are ordered such that those with higher priorities are run earlier than those with lower priorities based on some criterion [10]. Prioritization techniques are usually preferred because prioritization deals with the original test suite and no test cases are eliminated from the initial test suite.

B. Complexity of Regression Testing
The complexity of regression testing can be understood from the fact that it is impossible to completely test a software using a test suite collection even once due to resource constraints. Regression testing requires to re-run the entire ensemble of the test suite after every new build of software code is launched due to code change for fixing software bugs. Hence, one can easily judge that running the entire test suite again for every build is not feasible. [  Maintaining a balance between the ever-growing test suite size and limited constraints it-self is the biggest challenge in regression testing.
 Choosing a right automation tool based upon the nature of software application and availability of resources is one of the common challenges [25][26][27] [28].

A. Minimizing Recurrence Ratio
This metric is used to measure the quality of the regression testing procedure. Value of this metric depicts the percentage of those bugs (failures) whose fixation introduced some "newer" bugs in the software system that were not present earlier.
It measures the level of quality of the regression testing as how well the bugs have been fixed in response to the bug reports. Fixing a bug must be performed by altering software code in such a way that it introduces least or no negative impact on rest of the "passed" test cases. Recurrence ratio determines the degree to which previously correct functionality is being negatively affected by the altered software code for bug fixation [29][30] [31][32] [33].

B. Minimizing Repetition Ratio
Most of the test cases that constitute a test suite are actually testing some kind of functionality. The basic challenge during regression testing is that there are many test cases that are testing the same functionality or path again and again. This leads to drastic wastage in the time and resources. Hence, utmost care must be taken to ensure that redundancy in terms of repetition of functionality test must be avoided by removing un-necessary test cases [34][35] [36].

C. Maximizing Functionality Coverage
Test cases are executed again during regression testing after every new build is launched. Optimization of the test suite to be executed can cause loss of functionality coverage or code coverage or can minimize the defect detection capability of a test suite [37][38] [39][1] [2]. So, it must be made sure that a regression test suite that has been optimized does not compromise with functionality coverage of the original test suite as shown in fig. 6.

D. Focusing Defect Cluster Based Testing
This is one of the most important factor which can play a vital role in regression testing. As per the Pareto Rule of software testing: "Not all the software modules are equally buggy. There are nearby 20 percent of software modules which contributes towards 80 percent of the software bugs." Therefore, regression test suite optimization must ensure to focus on the modules where defect clusters are highest [2][3][4] [5].

E. Avoid Pesticide Paradox
New test cases must be added to the existing original test suite to avoid the "pesticide paradox" rule for software testing which states that if the same type of test cases is executed repeatedly, then they can no longer find new bugs or defects. So, test case optimization during regression testing must ensure to enhance the fault detection capacity by exercising focus on different parts of software system.

F. Context Based Regression Optimization
Every software application is different from other software application. Testing different types of application requires different type of skills, tools, technology and strategy. So, testing is actually context dependent. Testing a safety critical application requires completely different strategy than testing a commercial website [5] [6][8].

G. Absence of Defects Fallacy
Software having least number of defects or bugs can still becomes useless for end user as it does not provide the intended functionality intended to user. Therefore, testers must make sure that focus for testing must be utilized constructively for testing only the correct functionalities [5][7][8].

H. Update Regression Pack Regularly
Test suite collection which is executed after every software build or update is known as regression pack. Test cases in a regression pack must be updated regularly as per the changing user requirement.

I. Performance Measurement
Time to time measurement of the performance of the test procedures or tools employed for regression testing is a big challenge as there are not sufficient metrics available to measure the performance of a regression technique. Most commonly used metrics are APFD ("Average Percentage of Faults Detected"), PTR ("Percentage of Test Suite Executed for Complete Fault Coverage") etc. [24] [26].

J. Balancing Between Large Test Suite Size and Limited Resources
The major challenge in regression testing is to maintain a balance between the ever-increasing size of the test suite and the limited resources as shown in fig. 7. As the size of the test suite keeps on growing at a faster pace with every new build and the time to achieve the quality testing keeps on decreasing due to hard deadlines [27] [22].

III. CONCLUSION
The paper discussed about the major challenges and issues during regression testing in a pragmatic way. Any regression testing technique or tool does not provide any golden solution to all the challenges associated with regression process. The study of literature related to regression testing revealed some of the critical and vital challenges and issues, which have been described in detail in the paper.