Verification and validation of integrated vehicle health management system

Wang Zhaobing1 , Qing Xinlin2 , Liu Zhifang3 , Wang Yi4

1, 2, 3, 4Beijing Aeronautical Science and Technology Research Institute of COMAC, Beijing, China

1Corresponding author

Vibroengineering PROCEDIA, Vol. 4, 2014, p. 324-328.
Accepted 7 October 2014; published 3 November 2014

Copyright © 2014 JVE International Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Creative Commons License

The verification and validation (V&V) has been identified as a key step in quantifying and building confidence of IVHM system to inform about equipment or component health status and ensure safe and reliable application of IVHM system to military and industrial areas. The paper introduces the definition of IVHM and main methods of V&V. Besides, the key aspects of V&V are reviewed including the data sources and performance metrics as well as the challenges.

Keywords: V&V, IVHM, data source, metrics.

1. Introduction

The Integrated Vehicle Health Management (IVHM) system is being attached more and more importance as the significant basis of the autonomic logistics system adapting to high-tech local wars in the 21st Century. Having gone through anomaly detection, diagnostics and prognostics and immerged with the state of arts reasoning technologies, IVHM system monitors and controls the function of critical systems and subsystems/components in order to improve the effectiveness of the maintenance and decision-support system, reduce the cost of run and logistics, and decrease probability of the occurrence of catastrophic incidents.

There are more and more attentions in the architecture, algorithms and prototypes of IVHM recently. However, due to generally no rigorously universal methods and no statistically adequate data available, V&V technology remains a challenging open problem. With the support of Defense Advanced Research Projects of America, NASA and other university institutes have obtained some research achievements already. In order to verify and validate specific IVHM system, various types of data sources with different levels of fidelity can be utilized. Also, universal metrics for fault detection, isolation and prediction are important for the effectiveness assessment of IVHM system.

The paper is organized as follows. Section 2 presents the methods of V&V. In Section 3, data sources and metrics are indicated. Conclusions and challenges are followed in Section 4.

2. V&V Methods of IVHM

As shown in Fig. 1, IVHM can monitor the condition of the outside environment, structure, Engine and airborne systems. Besides, IVHM contains seven function layers such as sensor, data processing, monitoring, assessment, prediction, decision and expression. It is a comprehensive system to use all the information including aircraft operation monitoring data, fault data, ground-test data, maintenance data and design documents to diagnose the failure, predict the RUL (residual useful life) and make maintenance decision to prevent accidents and reduce the cost [1].

The framework architecture employed for IVHM V&V is shown in Fig. 2. Major elements are the physical system, diagnosis and prognosis algorithms, scenario-based experiments, and performance metrics. The physical system description and sample data (nominal and faulty) are provided to algorithm and model developers to build the models. System documentation in special format specifies the components, connections, and high-level mode behavior descriptions, including failure modes. A diagram with component labels and connection information is also provided. The fault catalog, part of the system documentation, establishes the failure modes that may be injected into experimental test scenarios and diagnosed by the algorithms. The output of the experiment is used to be quantitatively evaluated according the predefined metrics which will be introduced in the next chapter [2].

Fig. 1. The scale and function of IVHM

The scale and function of IVHM

Fig. 2. V&V methods

 V&V methods

3. Data sources and metrics for V&V

The evaluation process will be capable of assessing IVHM technologies in terms of their ability to detect, diagnose and predict the failure progression of faults. The key factors of IVHM V&V include the system interface, data source and metrics, as shown in Fig. 3. The set of metrics are in conjunction with a database of data from testing, simulation or in-service according the interface standards. The database and performance metrics will be discussed.

3.1. Data sources for V&V

In order to perform validation and verification of specific IVHM algorithms or technologies, various types of data sources with different levels of fidelity can be utilized. From a PHM (Prognostic and Health Management) algorithm designers perspective, some of the most desired data sets would come from either seeded fault testing, accelerated mission testing or actual field data from the entire system where detailed loading profiles are known and selected parameters were monitored that are correlated with the progression of the failure. However, these types of data sets are typically expensive to obtain and hence only typically represent less than 10 % of the data sets required for performing a comprehensive V&V program [3]. In order to supplement these more comprehensive data sets, subsystem or component rig tests can be performed that is more cost effective but limited in applicability and realism. Also, dynamic models with different levels of fidelity can be used to simulate fault signatures and provide data at any desired sampling rate. There are obviously many benefits to the simulate-based approach but many drawbacks as well. Table 1 details these pros and cons of these various data sources and their applicability for V&V of IVHM.

Fig. 3. Key aspects of V&V

Key aspects of V&V

Table 1. Description of different data sources

Simulation models and fault generation
Signals, noise and fault signatures simulated mathematically
Ideal for generalizing signals and noise, partially overcomes small sample problem
Data may not always be realistic, model may be expensive to develop to reliably simulate.
Component and LRU fault and failure tests
Use data collected on component or subscale test rigs
Real seeded fault, natural fault, and fault progression data available at little cost to program
Limited applicability for actual system fault observability
Subsystem/System fault characterization tests
Tests on stands using actual subsystem hardware
Somewhere realistic data, usually well controlled and documented with ground truth information
Seeded faults may not be entirely realistic of natural fault
Limited vehicle data evaluation
Dedicated missions for normal and off-normal test and evaluation
Realistic data of normal and some performance faults.
Expensive, doesn’t cover full range of potential conditions, no critical faults or progression
Technology Maturation Field Program
Data collection and tests during vehicle service
Covers full range of vehicle mission, operation, and environmental conditions
Limited to opportunistic fault occurrence. Delays use and implementation in field

3.2. Performance metrics for V&V

Fault detection, diagnostic and prognostic technology metrics should reflect all of the performance and effectiveness characteristics that influence their value to the overall IVHM system. Performance metrics need to quantify how well a technology responds to changes in normal operation in terms of the ability to detect anomalies, isolate a root cause fault or failure mode or predict the time to a given fault/failure condition.

Table 2. Performance metrics

Fault detection metrics
Fault detection rate
Fault Detection Rate is the percentage that the number of detected fault correctly occupies the number of fault occurred of the measure unit. The metric can indicate the detection capability of the system/equipment
Mean detection time
It is the mean time to detect and alarm when fault happens. The metric can indicate the effectiveness of detection
Max detection time
It is the max time to detect and alarm. The metric can indicate the worst case of detection
Fault detection rate of a single fault
It is the percentage that the number of correct detection occupies the number of a single fault occurred
False alarm rate
Within the prescribed time, the false alarm rate is the percentage that the number of false alarm occupies the total number of fault at the same time. False alarm is divided into misstatement (indicate B when A failure) and false declaration (indicate failure when no failure occurs)
Fault isolation metrics
Fault isolation rate
It is the percentage that the fault number isolated to the specified number of LRUs occupies the fault number detected at the same time
Mean time of isolation
It is defined as the mean time from the beginning to the end of isolation. Fault isolation time is usually the longest and difficult to predict during the artificial or off-line maintenance
Max isolation time
It is defined as the max time from the beginning to the end of isolation. The metric can indicate the effectiveness of isolation
Max fuzzy set
Fuzzy set is the units with fault. The metric can indicate the worst case of isolation
Fault prognosis metrics
Accuracy is a measure of how close a point estimate of failure time is to the actual failure time. The metrics include mean error, mean absolute error, variance absolute error, mean absolute value relative error, relative error of variance and correlation coefficient
Precision is a measure of the narrows of an interval in which the remaining life falls. The accuracy is the highest when the predicted failure time is equal to the actual failure time
Robustness is defined as the sensitivity which measures how sensitive the prognostic algorithm is to input changes or external disturbances
Timeliness of the predicted time to failure
It is defined to account for the impact of the timeliness of the predicted time to failure (TTF), that is, the probability density function’s relative position along the time axis with respect to the occurrence of the actual failure event
Calculate metrics
It is important for online prediction. The metrics include mean of the prediction-to-failure time and max of prediction-to-failure time
Effectiveness metrics
The benefits achieved through accurate detection, fault isolation, and prediction of critical failure modes are weighed against the costs associated with false alarm, inaccurate diagnoses/prognoses, licensing costs, and resource requirements of implementing and operating specific techniques

Specifically, prognostic metrics should be capable of evaluating prediction time accuracy in terms of remaining useful life or system degradation levels. Overall performance of an IVHM system that consists of diagnostic and prognostic modules may then be estimated as the weighted sum of performance metrics and cost/benefit analysis considerations. Table 2 illustrates the main performance metrics of IVHM [4].

4. Conclusions

V&V has been identified as a key step in quantifying and building confidence of IVHM system to inform about equipment or component health status. The main method, data sources and metrics were illustrated in the paper. However, there are several challenges for V&V technology. Firstly there is no set of methods and tools developed to provide a comprehensive approach. Secondly, there is no statistically adequate data and perfect metrics for V&V in military and industrial areas. Thirdly, it is difficult in establishing standards to perform V&V for IVHM.

Given the current state that we are still at the design stage of IVHM, it is important to develop V&V as soon as possible to speed up the process of IVHM on our civil aircraft. It is hoped that the review presented in the paper will be useful to the researchers in V&V area for IVHM.


This work was supported by Beijing Municipal Science and Technology Commission Fund (Z121100000212020) and China Postdoctoral Science Foundation (No. 2013M541575).


  1. Wang Yi, Zhao Guangtao, Liu Zhifang, Qing Xinlin The review of the integrated vehicle health management. Civil Aircraft Technology Highlights, Vol. 3, 2012, p. 4-6. [Search CrossRef]
  2. Wu Mingqiang The integrated engineering environment of IVHM. Computer Measurement and Control, Vol. 19, Issue 5, 2011, p. 98-101. [Search CrossRef]
  3. Roemer J. M., Dzakowic J., Orsagh R. F. Validation and verification of prognostic and health management technologies. Proceedings of IEEE Aerospace Conference, 2005, p. 3941-3947. [Search CrossRef]
  4. Saxena A., Celaya J., Saha B. Evaluating algorithm performance metrics tailored for prognostics, Proceedings of the Aerospace Conference, 2008. [Search CrossRef]