A Systematic Literature Review on Software Reliability Modeling

Document Type : Research Paper

Authors

Abstract

Nowadays software systems have very important role in a lot of sensitive and critical applications. Sometimes a small error in software could cause financial or even health loss in critical applications. So reliability assurance as a nun-functional requirement, is very vital. One of the key tasks to ensure error-free operation of the software, is to have a quantitative measurement of the software reliability. So far, many models have been proposed for quantification of the software reliability. Each of these models has its advantages and limitations, so selecting the best proper model for various applications is very crucial. In this article, first, we define the basic concepts of reliability, then we classify different approaches of software reliability modeling and finally, based on the analysis of the advantages and limitations, compare different approaches and mention some challenges and issues.

Keywords


[1] I. Sommerville, (2010), "Software Engineering", 9th ed., Harlow, England: Addison-Wesley
[2] R. S. Pressman, (2009), "Software Engineering-A Practitioner’s Approach-Required", 7th ed., New York, USA: McGraw-Hill
[3] International Organization for Standarization,(2001), "ISO Standard 826: Software Engineering – Product Quality, parts 1, 2 and 3", Geneve, Switzerland
[4] E. Hull, K. Jackson, and J. Dick,(2005), "Requirements engineering", 2nd ed., London, England: Springer
[5] H. Pham, (2007)، "System software reliability", New York, USA: Springer-Verlag
[6] F. Febrero, C. Calero, M. A. Moraga, (2014), “A systematic mapping study of software reliability modeling,” Information and software technology, Elsevier, vol. 56, no. 8, pp. 839-849.
[7] M. Rahmani, A. Azadmanesh,(2011), "Exploitation of Quantitative Approaches to Software Reliability", Tech. Rep. cst-2011-002, Computer Science, University of Nebraska, Omaha.
[8] Lyu, M. R., M. Xie, (1996), "Handbook of Software Reliability Engineering", 1st ed., New York, USA: McGraw-Hill.
[9] B. Kitchenham,(2004), “Procedures for performing systematic reviews”, Keele, UK, Keele University, vol. 33, pp. 1-26.
[10] S. Keele,(2007), “Guidelines for performing systematic literature reviews in software engineering”, Technical report, EBSE Technical Report EBSE-2007-01, pp. 1-57.
[11] P. Brereton, B. Kitchenham, D. Budgen, M. Turner, M. Khalil,(2007), “Lessons from applying the systematic literature review process within the software engineering domain”, Journal of systems and software, vol. 80, no. 4, pp. 571-583.
[12] B. Kitchenham, P. Brereton, D. Budgen, M. Turner, J. Bailey, S. Linkman,(2009), “literature reviews in software engineering–a systematic literature review”, Information and software technology, vol. 51, no. 1, pp. 7-15.
[13] B. Kitchenham, R. Pretorius, D. Budgen, P. Brereton, M. Turner, M. Niazi, S. Linkman,(2010), “Systematic literature reviews in software engineering–a tertiary study”, Information and Software Technology, vol. 52, no. 8, pp. 792-805.
[14] F. Q. Da Silva, A. L. Santos, S. Soares, A. C. C. França, C. V. Monteiro, F. F. Maciel,(2011), “Six years of systematic literature reviews in software engineering: An updated tertiary study”, Information and Software Technology, vol. 53, no. 9, pp. 899-913.
[15] J. Pan, (2014), “Software Reliability”, http://users.ece.cmu.edu/~koopman/des_s99/sw_reliability/#metrics.
[16] C. V Ramamoorthy and F. B. Bastani, (1982), “Software Reliability - Status and Perspectives”, Softw. Eng. IEEE Trans., vol. SE-8, no. 4, pp. 354–371.
[17] L. Shanmugam and L. Florence, (2012),“An Overview of Software Reliability Models”, in International Journal of Advanced Research in Computer Science and Software Engineering, vol. 2, no. 10, pp. 36–42.
[18] J. G. Shanthikumar,(1983), “Software reliability models: a review”, journal of Microelectronics and Reliability, Elsevier, vol. 23, no. 5, pp. 903–943.
[19] C. A. Asad, M. I. Ullah, and M. J. Ur-Rehman, (2004),“An approach for software reliability model selection”, in Proceedings ofthe 28th Annual International Computer Software and Applications Conference, pp. 534–539.
[20] S. S. Gokhale, P. N. Marinos, and K. S. Trivedi,(1996), “Important milestones in software reliability modeling”, in 8th International Conference on Software Engineering and Knowledge Engineering (SEKE), Nevada, USA, pp. 345–352.
[21] A. L. Goel,(1985), “Software reliability models: Assumptions, limitations, and applicability”, Softw. Eng. IEEE Trans., no. 12, pp. 1411–1423.
[22] X. Li, (2002), “Software reliability measurement: a survey”, MSc. Thesis, Dept. Computer Science & Software Engineering, Concordia University.
[23] Z. KRAJČUŠKOVÁ,(2007), “Software reliability models,” Proceeding of 17th International Conference on Radioelektronika, IEEE, pp.1-4.
[24] H. Pham, M. Pham,(1991), “Software reliability models for critical applications,” Technical report, Idaho National Engineering Laboratory, EG&G-2663.
[25] P.-H. Seong, (2009), "Reliability and Risk Issues in Large Scale Safety-critical Digital Control Systems", Springer, 1st ed., Berlin, Germany, pp. 85–87.
[26] P. C. J. P. K. Kapur, H. Pham, A. Gupta, (2011), "Software Reliability Assessment with OR Applications", 1st ed., London, England: Springer.
[27] A. K. Pandey, N.K. Goyal, (2013), "Early Software Reliability Prediction a Fuzzy Logic Approach", Springer.
[28] N. Fenton, M. Neil, and D. Marquez,(2008), “Using Bayesian networks to predict software defects and reliability”, Proc. Inst. Mech. Eng. Part O J. Risk Reliab., vol. 222, no. 4, pp. 701–712.
[29] Q. P. Hu, Y.-S. Dai, M. Xie, and S. H. Ng, (2006),“Early software reliability prediction with extended ANN model”, in Computer Software and Applications Conferenc, COMPSAC’06. 30th Annual International, vol. 2, pp. 234–239.
[30] S. Mohanta, G. Vinod, A. K. Ghosh, and R. Mall,(2010), “An approach for early prediction of software reliability”, ACM SIGSOFT Softw. Eng. Notes, vol. 35, no. 6, pp. 1–9.
[31] C. Smidts, M. Stutzke, and R. W. Stoddard, (1998),“Software reliability modeling: an approach to early reliability prediction”, Reliab. IEEE Trans., vol. 47, no. 3, pp. 268–278.
[32] Rome Laboratory, (1987),“Methodology for Software Reliability Prediction and Assessment”, Tech. Rep. RADC-TR-87-171.
[33] A. Immonen, E. Niemela, (2007),“Survey of reliability and availability prediction methods from the viewpoint of software architecture,” Software & System Modeling, Springer, vol. 7, no. 1, pp. 49-65.
[34] S. S. Gokhale and K. S. Trivedi,(2006), “Analytical models for architecture-based software reliability prediction: A unification framework”, Reliab. IEEE Trans., vol. 55, no. 4, pp. 578–590.
[35] R. C. Cheung, (1980),“A user-oriented software reliability model”, Softw. Eng. IEEE Trans., no. 2, pp. 118–125.
[36] P. Kubat, (1989),“Assessing reliability of modular software”, Oper. Res. Lett., vol. 8, no. 1, pp. 35–41.
[37] B. Littlewood, (1979), “software reliability model for modular program structure”, IEEE Trans. Reliability, vol. 28, no. 3, pp. 241-246.
[38] R.H. Reussner, H.W. Schimidt, I.H. Poernomo, (2003), “Reliability prediction for component-based software architectures” , J.Sysetem Softw., vol. 66, no. 3, pp. 241-252.
[39] G.N. Rodrigues, D.S. Rosenblum, S. Uchitel,(2005), “Using scenarios to predict the reliability of concurrent component-based software systems ” ,in Proceedings of the 8th international conference on Fundamental Approaches to Software Engineering, pp. 111-126.
[40] M.G. Thomason, J.A. Whittaker, (1999),“Rare failure-state in a Markov chain model for software reliability” ,in Proceedings of the 10th international symposiom on Software Reliability Engineering, pp. 12-19.
[41] S. S. Gokhale and K. S. Trivedi,(2002), “reliability prediction and sensitivity analysis based on software architecture”, in Proceedings of the 3rd international symposiom on Software Reliability Engineering, pp. 64-75.
[42] M. L. Shooman,(1976), “Structural models for software reliability prediction”,in Proceedings of the 2nd international conference on Software engineering, pp. 268–280.
[43] S. Krishnamurthy and A. P. Mathur, (1997),“On the estimation of reliability of a software system using reliabilities of its components”, in Proceedings of 8th International Symposium on Software Reliability Engineering, pp. 146–155.
[44] S. M. Yacoub, B. Cukic, and H. H. Ammar,(1999), “Scenario-based reliability analysis of component-based software”, in Proceedings of 10th International Symposium on Software Reliability Engineering, pp. 22–31.
[45] K. Goševa-Popstojanova and K. S. Trivedi, (2001), “Architecture-based approach to reliability assessment of software systems”, Journal of Performance Evaluation, Elsevier, vol. 45, no. 2, pp. 179–204.
[46] M. Xie and C. Wohlin,(1995), “An additive reliability model for the analysis of modular software failure data”, in Proceedings of 6th International Symposium on Software Reliability Engineering, pp. 188–194.
[47] W. W. Everett,(1999), “Software component reliability analysis”, in Proceedings of 3rd IEEE Symposium on Application-Specific Systems and Software Engineering and Technology, pp. 204–211.
[48] M. Klein, et al., (1999),“Attribute-based atchitecture styles”, in Proceedings First Working IFIP Conference on Software Architecture, pp. 225-243.
[49] L. Chung, B. Nixon, E. Yu, J. Mylopoulos,(2000), "Non-functional Requirements in Software Engineering". Kluwer Boston, Dordrecht 439.
[50] R. Kazman, G. Abowd, L. Bass, P. Clements, (1996),“Scenario-based analysis o software architecture ” , IEEE Softw., vol. 13, no. 6, pp. 47-55.
[51] N. Lassing, D. Rijsenbrij, H. van Vilet, (1999),“On software architecture analysis of flexibility, complexity of changes: size isn't everything”, in Proceedings of the 2nd Nordic Software Architecture Workshop.
[52] G. Molter,(1999), “Integrating SAAM in Domain-centric and Reuse-based development process”, in Proceedings of the 2nd Nordic Software Architecture Workshop.
[53] R. Kazman, et al., (1999),“The architecture tradeoff analysis method”, in Proceedings of the 4th IEEE International Conference on Engineering of Complex Computer Systems, pp. 68-78.
[54] S. S. Gokhale and K. S. Trivedi, (1999),“A time/structure based software reliability model”, Annals of Software Engineering, vol. 8, no. 1–4, pp. 85–121.
[55] S. S. Gokhale, P. N. Marinos, and K. S. Trivedi,(1996), “Important milestones in software reliability modeling”, in 8th International Conference on Software Engineering and Knowledge Engineering (SEKE), Nevada, USA, pp. 345–352.
[56] H. A. Stiber, (2007), “A family of software reliability growth models”, in Proceeding of 31th Annual International Computer Software and Applications Conference, IEEE, vol. 2, pp. 217-224.
[57] H. Pham, (2003),“Software reliability and cost models: Perspectives, comparison, and practice”, Eur. J. Oper. Res., vol. 149, no. 3, pp. 475–489.
[58] H. Pham and X. Zhang,(1997), “An NHPP software reliability model and its comparison”, Int. J. Reliab. Qual. Saf. Eng., vol. 4, no. 03, pp. 269–282.
[59] Z. Jelinski and P. Moranda, (1972), “Software reliability research”, Statistical Computer Performance Evaluation, pp. 465–484.
[60] A. L. Goel and K. Okumoto, (2009),“Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures”, in IEEE Transactions on Reliability, pp. 206 – 211.
[61] S. Yamada, (2014), "Software reliability modeling Fundamental and Applications", Japan, Springer.
[62] Jinxia An, Jihong Zhu, (2010),“ Software reliability modeling with integrated test coverage,” in Proceeding of Fourth International Conference on Secure Software Integration and Reliability Improvement, IEEE, pp. 106-112.
]63[ دگ یسی زاده ه حگ هانگیری، ت 1386 ، "مدلسازی دماری قا لیت ایمینان نرم افزار"، ر اهلی ن رانس ی المللی تحقیق در ملیاک
ایران، زیره یم، ایرانگ
[64] T. A. Thayer, G. R. Craig, L. E. Frey, W. L. Hetrick, and M. Lipow, (1976), “Software reliability study”, Tech. Rep. ADA030798.
[65] S. Bologna and W. Ehrenberger, (1978),“Applicability of statistical models for reactor safety software verification”, ENEA Internal report: ENEA RT 47/78, Roma, Italy.
[66] J. R. Brown and M. Lipow, (1975),“Testing for software reliability”, in ACM SIGPLAN Notices, vol. 10, no. 6, pp. 518–527.
[67] M. H. Halstead, (1977),“Elements of software science”, Elsevier, New York, USA.
[68] E. Nelson, (1978), “Estimating software reliability from test data”, Microelectron. Reliab., vol. 17, no. 1, pp. 67–73.
[69] S. N. Weiss and E. J. Weyuker, (1988),“An extended domain-based model of software reliability”, Softw. Eng. IEEE Trans., vol. 14, no. 10, pp. 1512–1524.
[70] F. Grigorjev, N. Lascano, and J. L. Staude, (2003),“A Fault Seeding Experience”, in Proceedings of the 4th Argentine Symposium on Software Engineering (ASSE), Buenos Aires, Argentina.
[71] H. D. Mills, (1972),“On the statistical validation of computer programs”, IBM Fed. Syst. Div. Rep., pp. 72–6015.
[72] K.-Y. Cai, (1998),“On estimating the number of defects remaining in software”, J. Syst. Softw., vol. 40, no. 2, pp. 93–114.
[73] Y. Tohma, H. Yamano, M. Ohba, and R. Jacoby,(1991), “The estimation of parameters of the hypergeometric distribution and its application to the software reliability growth model”, IEEE Trans. Softw. Eng., vol. 17, no. 5, pp. 483–489.
[74] V. Volovoi,(2004), “Modeling of System Reliability Using Petri Nets with Aging Tokens”, J. Reliab. Eng. Syst. Saf., vol. 84, pp. 149–161.
[75] M. Xie, K.-L. Poh, and Y.-S. Dai, (2004), "Computing System Reliability: Models and Analysis", 1st ed., New York, USA: Springer.
[76] M. Ohba, (1984),“Software reliability analysis models”, IBM J. Res. Dev., vol. 28, no. 4, pp. 428–443.
[77] N. B. Fuqua, (2014), “The Applicability of Markov Analysis Methods to Reliability, Maintainability, and Safety”. [Online]. Available: http://theriac.org/DeskReference/viewDocument.php?id=201.
[78] G. J. Schick and R. W. Wolverton,(1973), “Assessment of software reliability”, in Proceedings of the Vortrage der jahrestagung dgor/papers of the annual meeting, Springer, New York, NY, USA, pp. 395–422.
[79] M. L. Shooman, (1972),"Probability Models for Software Reliability Prediction", Statistical Computer Performance Evaluation, W. Freiberger (ed.). New York: Academic Press.
[80] J. D. Musa, (1975),“A theory of software reliability and its application”, Softw. Eng. IEEE Trans., no. 3, pp. 312–327.
[81] G. J. Schick and R. W. Wolverton,(1978), “An analysis of competing software reliability models”, Softw. Eng. IEEE Trans., no. 2, pp. 104–120.
[82] P. B. Moranda,(1975), “Prediction of software reliability during debugging”, in Proceedings of Annual Reliability and Maintainability Symposium, no. JAN 28, pp. 327–332.
[83] C. V. Ramamoorthy and F. B. Bastani, (1980),“Modelling of the software reliability growth process”, in COMPSAC, pp. 161–169.
[84] B. Littlewood and J. L. Verrall,(1973), “A Bayesian reliability growth model for computer software”, IEEE Symp. Computer Software Reliability, New York, USA, pp. 70-77.
[85] L. K. Singh, A. K. Tripathi, and G. Vinod, (2011),“Software reliability early prediction in architectural design phase: Overview and Limitations”, J. Softw. Eng. Appl., vol. 4, p. 181.
]86 [ غگ شاهمحمدی ه گ لیلی، ت 1386 ، "ارزیا ی می سن های معماری نرمافزار از دید قا لیت ا تماد ا رهیکرد مدل سازی دیاهرام
لو ی"، سیزدهمی ن رانس ملی انجم امپیوتر ایران، زیره یم، ایرانگ
[87] “Markov Modeling for Reliability”، http://www.mathpages.com/home/kmath232/part1/part1.htm.
]88 [ حگ ندالملکی، مگ رزازی ه اگ فراهی، ت 1390 ، "ارایه رهشی رای ارزیا ی قا لیت ایمینان معماری نرم افزار منتنی ر مثال سن
تر ینی لایهای ه لوله ه فیلتر ا نگاشت مدل شنکه پتری درزنجیره مار ا"، اهلی همایم تبصصی سیستمهای هوشمند
امپیوتری ه ار ردهای دنها، تهران، ایرانگ
[89] W.-L. Wang and M.-H. Chen, (2002),“Heterogeneous software reliability modeling”, in Proceedings of 13th International Symposium on Software Reliability Engineering, pp. 41 – 52.
[90] R. Rana, M. Staron, C. Berger, J. Hansson, M. Nilsson, F. Torner, W. Meding, C. Hoglund,(2014), “Selecting software reliability growth models and improving their predictive accuracy using historical projects data”, System and Software, Elsevier, vol. 98, pp. 59–78.
[91] R. Lai and M. Garg,(2012), “A Detailed Study of NHPP Software Reliability Models (Invited Paper)”, J. Softw., vol. 7, no. 6, pp. 1296–1306.
[92] P. Lakey and A. Neufelder, (1997), “System and software reliability assurance notebook”, Rome Lab., New York, USA, Tech. Rep. RL-TR-97-XX.
[93] K. Goševa-Popstojanova and K. S. Trivedi,(2003), “Architecture-based approaches to software reliability prediction”, in international journal of computer mathematics with applications, vol. 46, no. 7, pp. 1023–1036.
[94] S. S. Gokhale and M.-T. Lyu,(2005), “A simulation approach to structure-based software reliability analysis”, Softw. Eng. IEEE Trans., vol. 31, no. 8, pp. 643–656.
[95] K.-C. Chiu, Y.-S. Huang, and T.-Z. Lee, (2008), “A study of software reliability growth from the perspective of learning effects”, in international journal of Reliability Engineering & System Safety, vol. 93, no. 10, pp. 1410–1421.
[96] B. Anniprincy and S. Sridhar, (2014),“An Improved Software Reliability Growth Model”, Int. J. Comput. Eng. Res., vol. 4, no. 2, pp. 22–28.
[97] K. M. Cheol, J. S. Cheol, and J. J. Ha, (2007),“Possibilities and Limitations of Applying Software Reliability Growth Models to Safety- Critical Software”, in journal of Nuclear Engineering and Technology, vol. 39, no. 2, pp. 129–132.
[98] V. Almering, M. van Genuchten, G. Cloudt, and P. J. M. Sonnemans, (2007), “Using software reliability growth models in practice”, Software, IEEE, vol. 24, no. 6, pp. 82–88.
[99] C. Stringfellow and A. A. Andrews,(2002), “An empirical method for selecting software reliability growth models”, Empir. Softw. Eng., vol. 7, no. 4, pp. 319–343.
[100] DOES Inc., (2014),“Statistical Design of Experiments (DOE)”, http://www.doesinc.com/knowledge.htm.
[101] S. Söhnlein and F. Saglietti,(2014), “Software Reliability Assessment by Statistical Analysis of Operational Experience”, http://ercim-news.ercim.eu/en75/special/software-reliability-assessment-by-statistical-analysis-of-operational-experience.
[102] M. A. A. Boon, E. Brandt, I. C. Ramos, A. Di Bucchianico, and R. Henzen,(2007), “A New Statistical Software Reliability Tool”, in Proceedings of VVSS2007-verification and validation of software systems, pp. 125–139.
[103]B. Cukic, E. Gunel, H. Singh, and G. U. O. Lan, (2003), “The theory of software reliability corroboration”, IEICE Trans. Inf. Syst., vol. 86, no. 10, pp. 2121–2129.
[104] I. Karanta, (2006),“Methods and problems of software reliability estimation”, VTT WP, vol. 63, p. 57.
[105] “Deterministic simulation”, 2014. http://en.wikipedia.org/wiki/Deterministic_simulation.
[106] K. Sharma, R. Garg, C. K. Nagpal, and R. K. Garg, (2010),“Selection of Optimal Software Reliability Growth Models Using a Distance Based Approach”, in IEEE Transactions on Reliability, pp. 266–276.
[107] R. V. Melnyk, (2014), “Petri Nets: An Alternative to Markov Chains”. [Online]. Available: http://www.theriac.org/DeskReference/viewDocument.php?id=80.