Systems & Security Group (SSG)

In order to produce reliable software, it is important to have knowledge on how faults and errors may affect the software. In particular, designing efficient error detection mechanisms requires not only knowledge on which types of errors to detect but also the effect these errors may have on the software as well as how they propagate through the software. Here we presents the Propagation Analysis Environment (PROPANE) which is a tool for profiling and conducting fault injection experiments on software running on desktop computers. PROPANE supports the injection of both software faults (by mutation of source code) and data errors (by manipulating variable and memory contents). PROPANE supports various error types out-of-the-box and has support for user-defined error types. For logging, probes are provided for charting the values of variables and memory areas as well as for registering events during execution of the system under test. PROPANE has a flexible design making it useful for development of a wide range of software systems, e.g., embedded software, generic software components, or user-level desktop applications.

PROPANE is designed to run on a desktop system and consists of the PROPANE Campaign Driver (PCD), the PROPANE Library (PL), and the PROPANE Data Extractor (PDE). An overview is shown in Fig. 1.

The PL is used by the target system to gain access to the probing and injection functionality of PROPANE and is written in the C programming language. The PCD is responsible for handling the actual execution of experiments and is in a sense the main administrator of PROPANE. It has a user interface through which the user can control and follow the experiments. The PDE may be used during analysis to extract specific data from the experiment readout files. The PCD and the PL are integrated with each other, whereas the PDE is a stand-alone component of PROPANE. The environment simulator and target software are provided by the user. The environment simulator will act as a stimuli generator for the target software and may be partially controlled by the output generated by the target software (e.g., as in a control loop). The interactions between these two sub parts of the target executable are left to the user to define.

An FI-experiment is set up with a number of description files containing details on which faults and errors to inject, which variables and events to trace and so on. There are three types of description files: database description files, campaign description files, and experiment description files. A database is a set of campaigns, which in turn are sets of experiments.

For each experiment specified in the description files, the PCD spawns a new process running an executable file containing a complete specification for conducting one experiment. This executable contains the
PL which performs the actual injection of errors and logging of variables. The executable also has to contain everything necessary to run the target system and the environment simulator. The reason for running experiments in individual processes is twofold;

1. Parallel execution may shorten the total time for executing sets of experiments. This is especially true for systems that have excess computing capacity that there is a lot of idle time during the execution of a single experiment, primarily multi-processor systems.
2. Every experiment begins execution from identically specified initial conditions. This ensures that there are no residual artefacts that make one experiment affect another experiment.

During the execution of the experiments, log files and readout files are created. The log files contain information regarding the execution of the experiments, i.e., how PROPANE itself performed, and does not contain any readout data gathered from the target software. If the experiment could not be executed successfully for some reason, the log files provide hints to the potential problems. The readout files contain the data obtained by the inserted probes and the performed injections and are the basis for subsequent error propagation analysis. The environment simulator is designed by the user of the PROPANE tool, hence it may or may not use description files and may or may not create log files and/or readout files as per user-specified requirements. Also, the format of the files read and/or written by the environment simulator is user-defined.

The PDE will extract traces of the various logged variables and memory areas and can conduct Golden Run Comparisons to detect whether errors have occurred due to fault injection. The fact that the extraction and analysis of data are not performed by the part of the tool handling experiment execution supports writing customised analysis tools which may take into account desired experiment specific information and/or aims, such as coverage estimation of error handling mechanisms, failure classification or other activities which may be target specific.

Working process for experiments in PROPANE

The typical work process when using PROPANE can basically be divided into three main phases, namely: 1) the Setup phase, 2) the Injection phase, and 3) the Analysis phase (as illustrated in Fig. 2).

Setup phase

In the Setup phase, description files are written and the target system is instrumented. The inputs to this phase include the original source code of the target software, information on distribution and nature of faults and/or errors, and information about target system usage. The fault and error information is used for determining the fault and error sets to be injected in the experiments. The usage information forms the basis for determining the test cases used during the injections in order to provide the target system with a realistic operational profile. In the setup phase, the target software is also instrumented with probes for logging variables, memory areas and events, as well as high-level software traps for injecting faults/errors. The description files contain information on which faults are to be injected, which errors are to be injected at which locations, and which test cases are to be used by the environment simulator during the execution.

Injection phase

During the Injection phase, the PROPANE Campaign Driver (PCD) is set up with the description files generated in the Setup phase. The PCD invokes the target executable as an individual process and generates readout files containing detailed information on the results of the experiments. During the experiment, the specified faults and/or errors are injected and the specified variables and events are logged. Log-files are generated recording the actions of the PROPANE tool itself.

Faults are injected when the corresponding fault-triggers are activated. Fault injection at this level means that a faulty piece of code is executed instead of the correct piece.

Errors are injected based on the built-in error types, or on user-implemented error types. Thus, it is possible to implement error models which are not included in PROPANE. For example, if some parts of a system work unreliably under extreme temperatures, a user error type could take this into consideration.

Error-triggers are boolean expressions and an error is injected when its corresponding error-trigger is evaluated to true. Error-triggers may be based on time, frequency or a probability distribution. In addition to the built-in error-triggers, PROPANE also supports user-implemented error-triggers. As was the case for user error types, a user error-trigger may take into account target specific information, such as system state or the environment. In the example with the temperature-induced error type, a corresponding error-trigger may evaluate to true when the temperature is below a lower threshold and above an upper threshold.

Analysis phase

The readout files generated in the Injection phase are analysed in the Analysis phase to evaluate metrics for the target systems. These metrics may include coverage values, propagation information, etc. One aspect of analysis is to compare traces from two different runs with each other (e.g., compare a golden run with an injection run). The PROPANE Data Extractor which generates a set of data-files containing specific extracted data, such as Golden Run Comparisons and injection information, which may be used for propagation analysis at the variable-level.