The improved dynamic slicing for spectrum-based fault localization

Statistical fault localization

Statistical fault localization technology locates faults by calculating the suspiciousness of program statements. Jones & Harrold (2005) proposed the Tarantula technique, which computes the suspiciousness of statements and ranks them according to the suspiciousness. Similarly, Abreu, Zoeteweij & Van Gemund (2006) proposed another statistical fault localization technique, Ochiai, which finds the location of the fault by calculating the percentage of passing and failing tests of the executed statement. Specifically, fault program executes test cases. According to the result of the run, it further divides test cases into failed test cases and passed test cases. Finally, collecting the coverage information of test cases.

Suppose a program S = {s1, s2,…,sn} with n statements. The corresponding test suite T = {t1, t2,…,tm} consists of m test cases. After the test cases T are executed by the program S, we can obtain the program execution traces and the execution results. Finally, we use the execution traces and the execution results conveniently to get a program coverage matrix of m × n. There are a lot of approaches to statistical fault localization using the calculation formula of suspiciousness, among which the representative ones are Tarantula (Jones, Harrold & Stasko, 2002), Ochiai (Abreu, Zoeteweij & Van Gemund, 2007), etc.

Although there are great differences in the form of SBFL methods, the specific construction of SBFL methods is based on some common assumptions (Steimann, Frenkel & Abreu, 2013).

• Faulty statements may be covered by failing test cases and may be covered by passing test cases.

• Every failed test case executes at least one fault whose execution causes the failure.

• The distribution of faults in a program cannot be predicted. The prior probability distribution of faultiness is unknown.

• For the purpose of measuring the performance of a fault locator, we assume that upon inspection, a programmer always recognizes a faulty statement as such.

Xie et al. (2013) summarized additional assumptions when they theoretically analyzed the fault localization method:

• The fault localization method knows in advance how the test cases will perform in the program.

• The test case will always produce the same running result on the program regardless of the environment settings.

• The defective program can be 100% covered by the selected set of test cases, and contains at least one pass test case and one failing test case available.

The research and analysis of fault localization in this article are also based on the above assumptions.

Program slicing

Program slicing is an important program analysis and understanding technology, which mainly analyze the data dependencies and control dependencies of the program to obtain the relevant features inside the program (Korel & Yalamanchili, 1994). Then understanding the entire program by analyzing the sliced program. At the same time, program slicing can be used to remove statements unrelated to program faults to narrow the scope of bug search, thereby reducing the cost of fault localization. Program slicing is widely used in software fault localization research.

Control Flow Graph: Control Flow Graph (CFG) is a directed graph, represented by CFG (N, E, Entry, Exit), where N is the set of nodes, representing program statements; E is a set of edges that represent the control flow relationship between nodes; Entry and Exit represent the entry and exit nodes of the program, respectively.

Program Dependence Graph: Program Dependence Graph (PDG) is a directed graph formed by removing the control flow edge from the control flow graph CFG and adding the control dependency and data dependency edges, which is expressed in the form of a two-tuple PDG (S, E). Where S is the node set of CFG, E is the set of edges, and edges represent the data dependency or control dependency between two nodes.

System Dependence Graph: System Dependence Graph (SDG), SDG = {GPDGs, EInterpro} of multi-process program P is a directed graph. Where GPDGs is the PDG set, and each process is represented as a PDG; EInterpro is the set of inter-procedure calling edges and dependent edges. Data-dependent edges represent data flow between actual and formal parameters.

The Static Slicing: Given a slicing criterion $C=<s,V>$, where s is the interesting points in the program and V is the set of variables. A static slice of a program P is the set of all statements affecting the set of variables V in a point of interest s (a statement or block of statements).

The Dynamic Slicing: Given a slicing criterion $C=<I,s,V>$, where I is the input to the program, the s is the interest in the program, and V is the set of variables. Based on the input I, a dynamic slice is the set of all statements of the program P that affect the variable set V in the point of interest s (a statement or block of statements) in this execution.

The concept of program slicing was first proposed by Weiser in his doctoral dissertation in 1979 (Weiser, 1979). Subsequently, Weiser (1984) proposed a program slicing algorithm based on the control flow graph equation by analyzing the data flow in the program, which provided theoretical support for the development of program slicing technology. Subsequently, many researchers began to pay attention to the study of program slicing technology, and achieved some important research results. The researchers found it difficult to establish data flow equations for multiple processes using Weiser’s method.

To solve this issue, Ottenstein & Ottenstein (1984) mapped the program statements to the nodes of the directed graph when analyzing the dependencies between the program statements, and added the edges of the directed graph if there were dependencies between the statements, and finally abstracted the source program. is the program dependency graph. Ferrante, Ottenstein & Warren (1987) implemented single-process program slicing and proposed a graph reach ability slicing algorithm that uses program dependency graphs to represent programs. Horwitz, Reps & Binkley (1990) implemented program slicing for multiple processes on the basis of (Ferrante, Ottenstein & Warren, 1987), and proposed a two-stage graph reach ability slicing algorithm using a system dependency graph.

At present, the typical program slicing algorithm is based on the data flow equation and the graph reach ability algorithm, and other algorithms are extended or improved from these two types of algorithms. The following briefly introduces the program slicing algorithm based on data flow and the program slicing algorithm based on graph reach ability.

• Dataflow-based program slicing algorithm. The basic idea is to first calculate the control dependencies in the program, and then according to the data flow transfer in the control flow analysis program, iteratively construct slices by calculating the statements of the direct and indirect dependencies of the variables. The first program slice proposed by Weiser (1982) belongs to static backward slice, which is obtained based on the control flow graph and through two-layer iterative data flow analysis. The first layer calculates data dependencies, and calculates the variables that directly and indirectly depend on the variables in the slicing criterion; the second layer calculates control dependencies, by tracking the transferable data flow, calculating its control dependent statements, iteratively analyzing the newly obtained statement set; finally, the desired slice is obtained.

• Graphreach-based program slicing algorithm. The basic idea is to obtain program slices by traversing the constructed Program Dependency Graph (PDG) or System Dependency Graph (SDG), and finding other nodes that the program can reach from the slicing criterion according to the dependencies. For example, Ottenstein & Ottenstein (1984) solved intra-procedural slices using a graph reach ability algorithm with the help of a constructed program dependency graph. Horwitz, Reps & Binkley (1990) constructed a system dependency graph, and also implemented program slicing between procedures on SDG using a graph reach ability algorithm.

By program slicing, all statements can be classified into correct set of statements and suspicious set of statements. Therefore, most of the slicing-based fault location techniques use different slices (Zhang, Gupta & Zhang, 2004), execution slices (Xu et al., 2011) to locate faults in order to minimize the scope of suspicious statements. Singh & Mohapatra (2018) presented a context-sensitive dynamic slicing technique for concurrent programs. To effectively represent the concurrent aspect-oriented programs, Singh & Mohapatra (2018) proposed an intermediate graph called the multithreaded aspect-oriented dependence graph. Li & Orso (2020) presented a technique that computes memory-address dependence and represents them on standard dynamic dependence graphs for better supporting software debugging. Zhang (2021) attempted to study a light-weight approach of static program slicing, which works as a dataflow analysis on low-level virtual machine.

The improved dynamic slicing

Dynamic slicing is an effective debugging technique, which has been widely used in fault localization. When program execution results in a fault output or exception, the debugger usually wants to quickly ascertain the range of possible fault statements. By calculating the backward dynamic slice of the fault output statement, the dynamic slice can narrow the range of fault localization. Compared with the backward calculation, after the occurrence of the fault, the forward approach can generate a dynamic slice, which is according to the direct dynamic dependency of bug output statement. The forward approach does not need to traverse the dynamic dependency in the back, so it can calculate the dynamic slice result more quickly when a program runs incorrectly.

Dynamic slicing technology (Korel & Yalamanchili, 1994) reduces the fault search scope by removing statements that are not related to program faults. However, less consideration is given to variables actually referenced in variable definition during dynamic slice calculation, which results in the inclusion of slices of some unrelated variables and the redundancy of slice results. Therefore, we propose a dynamic slicing approach, which is based on the defined variable influence set. When calculating dynamic slicing, this approach finds out the variables actually referenced by the current variable at the time of definition and excludes the unreferenced variables, in order to get more accurate dynamic slicing.

Therefore, the forward calculation approach is used to calculate the program’s dynamic slicing, and the influence set analysis of the defined variables is carried out to improve the slicing accuracy, and the obtained dynamic slicing results are used for fault localization. The forward calculation method obtains the dynamic slice of interest points in the program through the direct dynamic dependencies of the interest points in the program. This method needs to maintain the slice results of each execution instance of the program (Korel & Yalamanchili, 1994; Zhang, Gupta & Zhang, 2004). The forward calculation method can generate a dynamic slice of the fault output statement according to the direct dynamic dependency of the fault output statement after an error occurs, and does not need to traverse the dynamic dependency backward. When the program fails so the dynamic slice result can be calculated more quickly. The following are the relevant definitions.

Definition1. Influence set Influence[v]. For a given statement s, the variable $v\in Def[s]$, the influence set Influence[v]= {v′| The v′ is the variabl e actually referenced when v is defined, $v^{\mathrm{\prime }}\in Use[s]$}. Where, the definition set Def[s] is the set of variables defined in statement s, and the reference set Use[s] is the set of variables referenced in statement s.

In this section, the influence set analysis is performed on the defined variables to remove irrelevant variables in order to reduce the redundancy of the slicing results.

Definition2. Hybrid Slicing Spectrum Matrix (HSSM). For a program P consisting of n statements s and a given m test cases, each execution of the test case yields a dynamic slice (slice) and the corresponding execution result. The mixed slice spectrum matrix is a two-dimensional matrix composed of multiple dynamic slices and execution results.

(1) $$\mathrm{H}\mathrm{S}\mathrm{S}\mathrm{M}=\left[{\mathrm{M}}_{\mathrm{m}\times \mathrm{n}}{\mathrm{N}}_{\mathrm{m}\times 1}\right]\mathrm{\forall }{\mathrm{a}}_{\mathrm{i}\mathrm{j}}\in {\mathrm{M}}_{\mathrm{m}\times \mathrm{n}},\mathrm{\forall }{\mathrm{b}}_{\mathrm{i}\mathrm{j}}\in {\mathrm{N}}_{\mathrm{m}\times 1}$$

(2) $${\mathrm{a}}_{\mathrm{i}\mathrm{j}}=\{\begin{array}{cc}1\hfill & \mathrm{s}\in \mathrm{s}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{e}\hfill \\ 0\hfill & \mathrm{s}\notin \mathrm{s}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{e}\hfill \end{array}1\le \mathrm{i}\le \mathrm{m},1\le \mathrm{j}\le \mathrm{n}$$

(3) $$b_{ij}=\{\begin{array}{cc}\mathrm{T}& \mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\\ F& \mathrm{f}\mathrm{a}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{d}\end{array}1\le i\le m,j=n+1$$

This hybrid slice spectrum matrix uses the program dynamic slice information to extend the program coverage information and combines it with the corresponding program execution results to generate the hybrid slice spectrum matrix.

Program slicing can extract a set of statements from a program according to a certain slicing criterion, we use the fault-related criteria of failed runs by program slicing to collect the slices. Our approach computes the slices according to the Program Dependence Graph (PDG) and the slice of statements with a forward computing algorithm. Algorithm 1 describes the forward computing dynamic slicing algorithm based on the influence set of defined variables. Algorithm 1 takes as input the current execution instance $s^i$, current execution method call statement (call), dynamic control dependency of $s^i$ ( $dynCD\left[s^i\right]$), dynamic data dependence of $s^i$ ( $dynDD\left[s^i\right]$) and slice of predicate pre (slicePred). The final output is the statement s refers to the dynamic slice of the variable use and the statement s defines the dynamic slice of the variable def. Lines 1–3 calculate the dynamic slice[use] of the variable use in the reference set Use[s] of statement s. $dynDD\left[s^i\right]$ obtains a dynamic slice[use] of the referenced variable based on the dynamic data dependency of the current execution instance $s^i$. Lines 4–10 calculate the influence set of the defined variable def according to definition 1: Influence[def]. Lines 11–18 compute the dynamic slice[def] of the variable def in the definition set of Def[s]. Line 11 executes the dynamic control dependency $dynCD\left[s^i\right]$ of the instance si to get the slicePred of the control dependent predicate pred. Lines 12–18 calculate the dynamic slice slice[def] of the defined variable, which is consist of four parts: the current execution statement s, the call statement call, the control dependent predicate slice slicePred, and the slice of the variables influence in the defined variable influence set Influence[def].

Fault localization

Dynamic slicing of programs can narrow the search for software fault localization by obtaining statements related to the slicing criterion. Then, we compute the suspiciousness of entities in the dynamic slice by the formulas of fault localization to generate a fault localization report. For passed tests and failed tests, more uncovered entities with passing executions and covered entities with failed executions should be assigned a higher degree of suspiciousness. Moreover, the covered entities by more passed executions and the uncovered entities by the failed executions should be assigned a lower suspiciousness. Therefore, the number of covered entities should have more weight than the number of uncovered entities in the suspiciousness calculation. Over the past few decades, dozens of doubtful formulas have been proposed and evaluated through empirical research. In our approach, we apply Tarantula (Jones & Harrold, 2005; Jones, Harrold & Stasko, 2002), Naish1 (Naish, Lee & Ramamohanarao, 2011), GP02 (Yoo, 2012), Ochiai (Abreu, Zoeteweij & Van Gemund, 2007), Jaccard (Chen et al., 2002) and GP13 (Yoo, 2012) to generate statement checking ranking based on coverage information. The suspiciousness value formulas of Tarantula, Naish1, GP02, Ochiai, Jaccard and GP13 as follows.

(4) $$Tarantula(\mathrm{s})={\displaystyle \frac{n_{ef}(s)/n_f}{n_{ef}(s)/n_f+n_{ep}(s)/n_p}}$$

(5) $$Naish1(\mathrm{s})=\{\begin{array}{cc}-1& n_{ef}(s)<n_f\\ n_p-n_{ep}(s)& n_{ef}(s)=n_f\end{array}$$

(6) $$GP02(\mathrm{s})=2\left({\mathrm{n}}_{\mathrm{e}\mathrm{f}}(\mathrm{s})+\sqrt{{\mathrm{n}}_{\mathrm{p}}}+\sqrt{{\mathrm{n}}_{\mathrm{e}\mathrm{p}}(\mathrm{s}}\right))$$

(7) $$Ochiai(\mathrm{s})={\displaystyle \frac{n_{ef}(s)}{\sqrt{n_f\times \left(n_{ef}(s)+n_{ep}(s)\right)}}}$$

(8) $$Jaccard(\mathrm{s})={\displaystyle \frac{n_{ef}(s)}{n_f+n_{ep}(s)}}$$

(9) $$GP13(\mathrm{s})=n_{ef}(s)\times \left(1+{\displaystyle \frac{1}{2n_{ef}(s)+2n_{ep}(s)}}\right)$$where, $n_f$ and $n_p$ denote the number of failed and passed test cases respectively. $n_{nf}(s)$ and $n_{np}(s)$ denote the number of failed and passed test cases that are not covered entities. $n_{ef}(s)$ and $n_{ep}(s)$ denote the number of failed and passed test cases that are covered entities. The coverage information of the entity, we can get the corresponding suspiciousness of the entity through the suspiciousness formula.

Experiment subjects

In this Section, 15 Java programs are selected as the benchmark for testing, and the detailed description is shown in Table 2. Where columns 1–3 are the program name, description and lines of code (excluding blank lines and comment lines), and columns 4 and 5 are the number of fault versions and test cases used in the experiment. The first six programs in the table are the Siemens Test Suite. Siemens Suite (except jtcas) is a program that converts a C version to a Java version by Santelices et al. Jtcas, Nanoxml, and XML-security with the corresponding test cases are all derived from SIR (Do, Elbaum & Rothermel, 2005). The latter three projects JFreeChar, Joda-Time and Mockito are from the Defects4J library (Just, Jalali & Ernst, 2014). The numbers of executable lines of all the subjects are ranging from hundreds of lines to tens of thousands of lines. This allows us to evaluate across a very broad spectrum of programs and allows us to have more confidence in the ability to generalize our results.

Our approach applied to single-bug version programs to locate fault. We removed two versions with no test case execution failure and eight versions with multiple faults from the 41 fault versions of jtcas, and selected the remaining 31 versions for testing. We randomly selected 10 of the 23 incorrect versions of the tot $\mathrm{\_}$info program for testing, and selected nine fault versions of schedule. Selected 8 of the 10 fault versions of Schedule2 to test, removed the v4 version (fault is to remove the comment statement at 350 lines) and the v6 version with multiple faults. Test with seven fault versions of print $\mathrm{\_}$tokens and five fault versions of print $\mathrm{\_}$tokens2. In addition, a total of 28 fault versions of NanoXML and 12 fault versions of XML-security (abbreviated XML-sec) were selected for testing. In Defects4J, the part of the fault version of the program does not output a fault. We removed the v2 and v19 version of Time, and selected the remaining 25 versions for testing. We removed the v6 and v10 version of Mockito, and selected the remaining 36 versions for testing. Finally, we tested 197 single-fault versions of programs.

As shown in fourth column of Table 2, in the first six programs, all the faults were manually seeded by other researchers and the considered fault types include predicate based fault and assignment based fault. All creation faults are semantic bugs that do not incur crashes. These creation faults have been shown to simulate realistic faults well and provide reliable, trustworthy results. The latter nine projects have the real faults. Also packaged with each of these programs was a set of test cases and faulty versions. For all of our subject programs, any faulty versions that did not lead to at least one test case failure in our execution environment were excluded. In this way, we assure that each fault examined would be revealed by at least one test case.

Evaluation metrics

In order to evaluate the effectiveness of various fault localization approaches, we use the following four criteria to evaluate the effectiveness from different perspectives.

Cumulative Number of Statements Examined: The number of cumulative examined statements for each program is the number of total statements that need to be checked when m faults are located in m fault versions. For m fault versions of any given program, M(i) and N(i) represent the number of statements that need to be checked when approach M and approach N locate a fault. If $\sum _{\mathrm{i}=1}^{\mathrm{n}}\mathrm{M}(\mathrm{i})<\sum _{\mathrm{i}=1}^{\mathrm{n}}\mathrm{N}(\mathrm{i})$, it indicates that the approach M is more effective than the approach N.

Cost Score: Cost criterion is to evaluate the effectiveness of fault localization from the perspective of relative index, which was first proposed by Renieres & Reiss (2003). Cost is the ratio of the percentage of the number of statements (Rank of faults) that need be examined to the total number of all executable statements when finding errors in the program version in, as shown in Eq. (10).

(10) $$\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}={\displaystyle \frac{\mathrm{R}\mathrm{a}\mathrm{n}\mathrm{k}\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{a}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{s}}{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{e}\mathrm{x}\mathrm{e}\mathrm{c}\mathrm{u}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}}\times 100\mathrm{\%}}$$

Because the faulty statement may share the same suspiciousness with others, the programmer might need to examine only one of these statements fortunately (best cases) or to examine all these statements to locate the real fault unfortunately (worst cases) (Jones & Harrold, 2005). In the experiments, we obtain the average fault-localization costs per subject by the best fault-localization costs per subject and the worst fault-localization costs per subject. Finally, we use the average fault-localization costs as an evaluation indicator.

EXAM Score: To more intuitively compare the effectiveness of different approaches, we also use EXAM indicators for evaluation. The EXAM indicator is the ratio of the fault detection rate (% of faults located) to the code examined rate (% of code examined) (Renieres & Reiss, 2003), as shown in Eq. (11).

(11) $$\mathrm{E}\mathrm{X}\mathrm{A}\mathrm{M}={\displaystyle \frac{\mathrm{\%}\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{a}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}}{\mathrm{\%}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{e}\mathrm{x}\mathrm{a}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}}$$

In the early research, Renieres & Reiss (2003) used the test score EXAM to evaluate the efficiency of fault localization. Jones, Harrold & Stasko (2002) also used this benchmark to compare the efficiency of the fault localization approaches. In the effectiveness of the evaluation in locating fault, the higher the suspicious score is, the better effectiveness fault localization is.

Recall at Top-N: This dependent variable measures the number of faults with at least one faulty element within Top-N in the ranked list. This metric emphasizes earlier fault detection and has been widely used in fault localization work (Le, Oentaryo & Lo, 2015). Kochhar et al. (2016) reported that developers usually only inspect top-ranked program elements during fault localization. Therefore, following prior work, we use Top-N (N = 5, 10, 20, 30) in our experimental study.

Experiments using the cumulative number of statements examined

We can observe the cumulative number of statements examined between our technique and the original dynamic slice in Table 3. The improved dynamic slices contributed to the improvement of the accuracy in locating faults. From the comparison of Table 3, for each program, the cumulative number of statements examined by the improved dynamic slice is smaller than the original dynamic slice on five techniques (Tarantula, Nasish1, GP02, Ochiai, Jaccard, and GP13). For example, on the most popular Ochiai the number of examined statements by the improved dynamic slices to locate faults for jtcas program was reduced by 66 than the original dynamic slice, similarly, the number of examined statements for tot $\mathrm{\_}$info programs was reduced by 197. In the last row of Table 3, Total Cumulatives, the total number of statements for the overall topic that each method needs to examine is stated. We can also observe that DS-Naish1 performs best and has only 85,534 total check statements, which is much less expensive than all the comparison techniques over 15,000 lines.

Furthermore, we can observe the cumulative number of statements examined between our technique and ADBS in Table 3. The improved dynamic slices better than ADBS in locating faults. From the comparison of Table 3, for each program, the cumulative number of statements examined by the improved dynamic slice is smaller than ADBS on six techniques (Tarantula, Nasish1, GP02, Ochiai, Jaccard, and GP13). For example, on the most popular Ochiai the number of examined statements by the improved dynamic slices to locate faults for jtcas program was reduced by 41 than ADBS, similarly, the number of examined statements for tot $\mathrm{\_}$info programs was reduced by 93. In the last row of Table 3, Total Cumulatives, we can also observe that our approach performs better than ADBS in all formula. Therefore, it is clear that our proposed technique improves the effectiveness of fault localization.

Experiments using the average cost metric

As shown in Table 4, we compare the average cost of fault localization our method and the original dynamic slice on each topic. From Table 4, we can observe that for all subjects, the average cost of fault localization of our method is much smaller than that of the original dynamic slice. The last row of Table 4 illustrates the average cost across all topics for each technique. It shows that, for all subjects, the average cost of our improved dynamic slice is less than that of original dynamic slice on Tarantula, Naish1, GP02, Ochiai, Jaccard and GP13 which are 5.09%, 4.79%, 4.87%, 4.53%, 4.76% and 5.08%, respectively. Furthermore, the cost of DS-Tarantula is much smaller than that of D-Tarantula on schedule2, which is 6.27%, whereas the cost of DS-GP02 is much smaller than that of D-GP02 on schedule2, which is 8.52%.

Than, we compare the average cost of fault localization our method and ADBS on each topic, as shown in Table 4. The last row of Table illustrates the average cost across all topics for each technique. From Table 4, we can observe that for all subjects, the average cost of fault localization of our method is smaller than that of ADBS. Furthermore, the cost of DS-Tarantula is smaller than that of ADBS-Tarantula on JFreeChart, which is 1.73%, whereas the cost of DS-Ochiai is smaller than that of ADBS-Ochiai on JFreeChart, which is 1.17%. That is to say, our approach is actually more effective than the original dynamic slice and ADBS in the experiments.

Experiments using EXAM metric

Additionally, to more intuitively show the improvement of our approach over the compared techniques, we use the Exam metric to evaluate the effectiveness. As shown in Fig. 3, the horizontal coordinate indicates the code examined rate and the vertical coordinate indicates faults located rate. For the same code examined rate, a higher percentage of faults located rate means more effective. In Fig. 3, shows that the line of the improved dynamic slice technique is always beyond the line of the original dynamic slice and ADBS techniques, which indicates that DS-Tarantula, DS-Naish1, DS-GP02, DS-Ochiai, DS-Jaccard and DS-GP13 outperform the original and ADBS approaches in the effectiveness of fault localization. Jones & Harrold (2005); Jones, Harrold & Stasko (2002) concluded that the lower code examined rates better reflects the effectiveness of fault localization approaches. In this article, we compare the percentage of faults located by different approaches with 20% code examined rate. The faults located rate of DS-Tarantula is 85.26% compared to 68.98% of D-Tarantula and 64.06% of ADBS-Tarantula. DS-Naish1 is 88.24% compared to 61.73% of D-Naish1 and 63.70% of ADBS-Naish1. DS-GP02 is 87.10% compared to 50.12% of D-GP02 and 62.12% of ADBS-GP02. DS-Ochiai is 85.19% compared to 68.21% of D-Ochiai and 69.20% of ADBS-Ochiai. DS-Jaccard is 81.53% compared to 67.94% of D-Jaccard and 67.19% of ADBS-Jaccard. DS-GP13 is 86.26% compared to 66.97% of D-GP13 and 69.61% of ADBS-GP13. This improvement in the accuracy of fault localization is mainly due to the improved dynamic slice to narrow down the range of fault localization. It can be obviously seen that our technique has improved the accuracy of fault localization than these compared techniques.

Experiments using Top-N metric

As shown in Table 5, Five metrics (Top-5, Top-10, Top-20, Top-30) are used to verify the accuracy and effectiveness of the D-SBFL, DS-SBFL, and ADBS-SBFL approaches. Specifically, there are 24 programs that DS-Tarantula succeeds in locating all faults by considering only the top five positions in ranked lists. When considering only the top five positions in the ranking list, DS-Naish1, DS-GP02, DS-Ochiai, DS-Jaccard13, and DS-GP13 locate 21, 9, 19, 5, and 19 programs, respectively. Moreover, the number of programs in which all faults are successfully located by DS-SBFL is always more than the others when using the other three Top- N metrics. Figure 4 shows that DS-SBFL outperforms other baseline approaches in locating all faults when using four Top-N metrics in a more intuitive way.

The comparison of time cost

In this section, we evaluate the time cost of the improved slice, the ADBS slice, and the original slice fault localization using the tarantula formula. As shown in Table 6, the time comparison between the improved slice, the ADBS slice, and the original slice fault localization method when running all fault versions on each test program. Column 2 “Collecting slice” represents the time to build slice information and collect coverage information on all test cases, the unit is second (s); Column 3 “Computing suspiciousness” represents The time of the calculation of suspiciousness; Column 4 “Total” represents the total time cost. From the data in column 2, the time of the computing slice based on the original slice is slightly longer than that of the improved slice. As can be seen from the last column of Table 6, the total time of improved slice-based fault localization is shorter than the original slice-based fault localization methods, and the difference between the improved slice and the ADBS slice is small in time overhead.

Threats to validity

In this subsection, we discuss the potential threats to the validity of our empirical study.

Threats to external validity are about whether the observed experimental results can be generalized to other subjects. To guarantee the representativeness, we choose a large number of programs from the widely used Simens, Linux, and Defects4J suites. Our experiments consider both C and Java programs and include both seeded and real bugs. We realize that there is no perfect empirical study and there must be some tricky programs and bugs not considered in our experiments. We plan to enlarge our subjects in our future work.

The internal validity of this work lies in the accuracy of the slicing result computed by our approach. To avoid faults in our tool implementation, we prepared our data carefully and tested our approach with simple programs.

Threats to construct validity are about whether the performance metrics used in the empirical studies reflect the real-world situation. In the experiment, we used the cumulative number of statements examined, Cost, EXAM, and Top-N other indicators. We realize some indicators were not considered in our experiments. Therefore, using more widely metrics is our future work.

Slicing-based fault localization techniques

After Weiser (1982) first proposed program slicing for fault localization. Researchers have proposed various fault localization methods based on program slicing methods. The first type of method is that the program slicing results are directly used for fault localization. For example, Zhang et al. (2005) proposed a dynamic slicing-based fault localization method, which used forward computation to calculate and compare the fault localization effects of data slicing, full slicing and correlation slicing, and experimentally proved that the fault localization effects of full slicing and correlation slicing are better. Jiang et al. (2012) used the application slicing technique to locate null pointer exceptions by combining real-time stack information and performing null pointer and alias analysis on sliced programs. The second type of method is the combination of program slicing and program spectrum methods for fault localization. For example, Wen et al. (2011) used dynamic slicing to extract dependencies between program elements, reduce program execution traces, and then constructed a Tarantula-like formula to count statement suspiciousness on statement spectrum information. Alves et al. (2011) used dynamic slicing and modification impact analysis to improve the efficiency of Tarantula’s fault localization. In contrast to the literature (Wen et al., 2011; Alves et al., 2011), this article uses an improved dynamic slicing technique to improve the dynamic slicing. Yu et al. (2011) used the Tarantula formula to generate the priority order of suspiciousness statements based on coverage information and static control flow graphs. Yu et al. (2011) constructed the execution flow graph based on coverage information and static control flow graphs to obtain semi-dynamic slicing. Lei et al. (2012) used the intersection of static backward slicing and execution slicing to form an approximated dynamic slice, and then counted the suspiciousness of the statements. Ghosh & Singh (2021a) presented using context-sensitive slicing to reduce time and be more precise for spectrum fault localization.

Spectrum-based fault localization approaches

Spectrum-based tuning is one of the most popular tuning techniques in the automated tuning process. The spectral information of program execution is used to calculate the suspiciousness of each statement (Yoo, 2012), Ochiai (Abreu, Zoeteweij & Van Gemund, 2007), Jaccard (Chen et al., 2002). The goal of all these methods is to assign the flawed statement the highest possible suspect value and the correct one the lowest possible suspect value. Suspiciousness ranking can help programmers save debugging time (Naish, Lee & Ramamohanarao, 2011). Renieres & Reiss (2003) proposed the nearest neighbor query error localization method. The method assumes that there is one failed run and multiple successful runs, and uses the distance metric method to count the most similar program spectrum from the successful run to the failed run, compare the differences, remove the statements executed by both failed and successful runs, and generate error locations Report. By constructing N-length execution sequences, Nessa et al. (2008) used statistical information association to mine the association between N-length execution subsequences and execution results, and then proposed an error localization method combining N-length sequences and association rules. Ribeiro et al. (2018) proposed a spectrum-based Java program fault location tool JAGUAR. It relies on data flow strategies for fault location and visualization of suspicious statement lists. Zhang et al. (2017) proposed a PageRank-based fault location, which uses the PageRank algorithm to improve the technique of spectrum-based fault location. Wong et al. (2013) proposed DStar, a technique that computes a suspiciousness score for each program statement. DStar is a well-known statistical formula used in SBFL techniques to reveal problematic statements. Xiaobo et al. (2021) proposed an entropy-based framework Efilter to filter unlabeled test cases, constructing a sentence-based entropy and a test-suite-based entropy to measure the localization uncertainty of a given test suite. To improve the effectiveness of fault localization, Wang, Wu & Liu (2021) used a clustering-based technique to identify coincidentally correct test cases from passed test suites and empirically quantified the accuracy of identifying coincidentally correct test cases to assess its effectiveness. A framework for fault localization using a multivariate logistic regression model that combines static and dynamic features collected from the program being debugged by Ju et al. (2020). Ghosh & Singh (2021b) proposed an automated framework using chaos-based genetic algorithm for multi-fault localization based on SBFL technique.

In SBFL, ranking is based solely on the suspiciousness score of each element. Elements with high scores are placed at the beginning of the list, and elements with low scores are placed at the bottom. Heiden et al. (2019) argue that the SBFL technique only exploits the program spectrum as an abstraction for program execution without considering any other useful contextual information. Vancsics et al. (2021) addresses this problem by using method call frequency. He refines the standard SBFL formula with the frequency with which the investigated method appears in call stack instances. Zou et al. (2019) considered the execution time of different techniques and thus combined different techniques. He proposes a combined technique, CombineFL, which can be configured according to time cost, and CombineFL significantly outperforms independent techniques. Furthermore, in their work, they obtain fault localization reports on program coverage information, but we propose an improved dynamic slicing that can narrow the fault scope to improve the effectiveness of fault localization.

To enhance the impact of failed test cases on fault localization, Xie et al. (2022) used a universal data augmentation method that generates synthesized failing test cases from reduced feature space for improving fault localization. Zeng et al. (2022) introduced a probabilistic approach to model program semantics and utilize information from static analysis and dynamic execution traces for fault localization, which balance could be reached between effectiveness and scalability. Wang et al. (2022) investigated the performance of Mutation-based fault localization with First-Order-Mutants and Higher-Order-Mutants on single-fault localization and multiple-fault localization.