Evaluation of experiences from applying the PREDIQT method in an industrial case study

Abstract

We have developed a method called PREDIQT for model-based prediction of impacts of architectural design changes on system quality. A recent case study indicated feasibility of the PREDIQT method when applied on a real-life industrial system. This paper reports on the experiences from applying the PREDIQT method in a second and more recent case study – on an industrial ICT system from another domain and with a number of different system characteristics, compared with the previous case study. The analysis is performed in a fully realistic setting. The system analyzed is a critical and complex expert system used for management and support of numerous working processes. The system is subject to frequent changes of varying type and extent. The objective of the case study has been to perform an additional and more structured evaluation of the PREDIQT method and assess its performance with respect to a set of success criteria. The evaluation argues that: 1) the PREDIQT-based analysis facilitates predictions providing sufficient understanding of the impacts of architectural design changes on system quality characteristics, so that informed decisions can be made; 2) the PREDIQT-based analysis is cost-effective; 3) the prediction models are sufficiently expressive to adopt the relevant architectural design changes and analyze their effects on quality; 4) the prediction models are sufficiently comprehensible to allow the domain experts to be actively involved in all phases of the PREDIQT process and achieve the goals of each phase with a common understanding of the results; and 5) the PREDIQT-based analysis facilitates knowledge management and contributes to a common understanding of the target system and its quality. Moreover, the study has provided useful insights into the weaknesses of the method and suggested directions for future research and improvements.

Oppdragsgiver: Research Council of Norway