KINGSTON UNIVERSITY, LONDON

Faculty of Computing Information System and Mathematics

Software Certification
Amarpreet Singh Saini

K1051678

M.Sc Networking and Data Communication

Research methods - II - CIM951

Submission Date -27th May 2011

Software Certification

1.

Abstract

Software certification is an arduous process that deals with certifying the software packages. Software is a very essential commodity in todays times. The vast scope that computer software deals with means there is a vast number of possibilities of unreliable code and bugs. This can lead to loss in productivity and is a major cost issue. Software certification comprises of a wide range of techniques to provide an assurance of quality of software when it is deployed on a system.

2.

Introduction

Software quality is something that is very hard to gauge. Building reliable software is a big challenge especially in todays time when there are so many different platforms to support and much architecture to cover. Rushing deadlines amongst other things can result in poor code which can manifest as bugs, holes and errors and so on. Software can thus exhibit behavior that is undesirable and is a hindrance to productivity. That undesirable behavior may be traced to a defective specification, defective software, incorrect usage, or a combination of these.*1] All of this can lead to serious issues such as financial losses, loss of privacy and a large magnitude of other security issues. Hence software certification as argued by proponents is a very vital process in software development in todays times. Software certication comprises a wide range of formal, semi -formal, and informal assurance techniques, including formal verication of compliance with explicit safety policies, system simulation, testing, code reviews and human sign offs, and even references to supporting literature. The main goal of the certification process is to provide a high level of assurance of quality of the software when it is in deployment. And that the system will perform expectedly in the environment and any documented situations. Not only that, but the software can aptly perform when an unexpected situation arises and can provide good information as to why that specific behavior took place. Although the above statement holds true for systems that can be identified by discrete states and thus one can follow some formal guidelines and processes to certify software for such systems, it is difficult to do so for systems with a wide dynamic behavior. In other words software certification gauges the correctness of code so that the adopter of the software need not worry about the procedures and tools used to certify the software system. The certification process can be carried out by an independent authority which can be recognized on an industry wide level. This is akin to a doctor referring to another doctor for advice specializing in a particular area of the science which he can then take without hesitation. One such example is the ESRB or the Entertainment Software Rating Board which rates video games in the United States based on their content indicating the appropriate age group it is meant for. The nature of the medium here is interactive entertainment and hence it requires a different form of certification. However we can see that the basic principle is same as consumers can simply verify the age rating of the software and buy it without hesitation for their children etc. An SCMS or a software certification management system is a framework used by the certification agencies to enable various services on their part. A software certicate management system (SCMS) provides a range of certication services. It maintains the links between different system artifacts (e.g., design documents, engineering data sets, or programs) and different varieties of certicates, checks the validity of certicates, provides access to explicit audit trails, enables browsing of certication histories, and enforces system -wide certication and release policies. [3]

Amarpreet Singh Saini

3.

Challenges

Certification occurs at the end of the softwares development lifecycle. However certification is considered from the very beginning of the software. The currently existing software verification systems have several weaknesses and flaws that are not addresses. The below figure describes a basic certification process lifecycle.

[2] Here a software publisher provides the release candidate version of its software to a software certification lab. The lab embeds a residual testing tool and checks the product while it is operating. The results are then compared to the requirements of the market the software is intended for and this determines the certification that will be granted to the software. We look at some of the challenges faced during the certification process. Some of the popular certification mechanisms are process based which are problematic as mentioned before due to the dynamic nature of some software. It is difficult to maintain a high amount of reliability during the process as there are a large number of tools and techniques that are available for testing of software. Some products are part of a family and interconnected to other products which can prove to be a challenge. The time taken to certification can be a problem especially if it needs to be carried out multiple times which are very common in todays times. Linking with components such as design documents, engineering data sets, certificates etc. Providing useful information about the process and documenting the certification process. Time and effort required to complete the certification is a major challenge in software certification as there is no procedure that can provide a correct answer to this problem due to the vast nature and scope of software systems. The field which the software is intended for. This is a problem as the certification agency not only has to verify the inputs provided by the developers of the code but also look for bugs and vulnerabilities outside the known and established working of the software which can only be determined on a probabilistic basis.

Software Certification

There is no uniform certification process which is a major headache for many organisations which use software that is very niche and not mainstream. Ultimately certification does not guarantee that the software is free of bugs and defects. Rather it is only a rating to describe that the software meets certain standards. Lack of metrics has made the field of certification a bit murky as the various different techniques are hard to gauge. This is because the software is intricately linked to its operating environment amongst other factors which can result in software which is coded well but still functions poorly due to its interaction with the operating system.

4.

Certified Software

Certied software consists of a machine-executable program C plus a rigorous formal proof P (checkable by computer) that the software is free of bugs with respect to a particular dependability claim S. The proof and the specification are written in a general purpose mathematical logic language. The proof can be checked by a computer using software such as proof checking theorems and certified compilers. In such a method machine executables are checked rather than the high level code of the software. It is possible to certify the high level code and then convert the certifying checker along with the rest of the software using certified compilation.

Amarpreet Singh Saini

In the figure above we have 5 components. The certified software which includes the formal proof and the machine level code, the formal machine model, Formal dependability claim, the proof checker and the underlying mechanized metalogic. [15]

5.

Certification methodologies and techniques 5.1 High Assurance Pipeline method by Jefrey Voas ( Reliable Software Technologies Corporation ) This model is a high assurance certification model independent of the processes used in the requirement phase and the development phase of the software lifecycle. Here each certification process is based on the operation of the software rather than testing the software for expected inputs or some other process driven method.

[1]

In this process it is important to note that the infrastructure of the organization isnt taken into consideration along with its profile and past performance while certification. This can be bit of an issue with some as others would champion this clause. The granting of high assurance certification for undefined system states is strictly avoided here. Also it is to be noted that even though this method is strict on undefined system states which the author readily admits is quite a cumbersome and in cases impossible task it is still not good practice to ignore the behavior of the software in these states and that they should be taken into consideration while certifying the software. Finally like all other certification models certification using this model cannot guarantee a highly assured product and it is only done so on a probabilistic case.

Software Certification

5.2 Separation Logic Separation logic is a system used for checking a programs correctness. Separation logic is kind of a substructural logic. It is an extension of Hoares logic which uses expressions called Hoares triples to verify correctness of a program. Separation logic adds two expressions to Hoares logic to improve upon it namely separating conjunction and separating implication. Robert Frohardt in his paper A Brief Introduction to Separation Logic specifies separation logic if it is used to be working with programming heaps. [13]

5.3 Security Security assessment during software development is a very key issue. The security of the software entirely depends on the development process, as bad coding or design decisions will lead to bugs in the software which can manifest as exploits and security holes. Software systems depending on their development models are chopped into various components which are separately coded. This is done to produce deliverables in a fast and efficient manner. However a bad design choice can lead to some kind of exploit that can be very dangerous to the life of the product. This is very serious as if the bug that is found in the system during deployment or certification is traced to early stages of the development then it can be very expensive to fix. For e.g. A bad decision can cost millions of pounds to be fixed for high assurance software if the bug is traced to a design flaw in the requirements phase of the lifecycle of the product. Security is therefore given utmost priority in all the phases of the software development lifecycle with verification and validation being done at the end of each step in the lifecycle. The below figure describes an attempt by Anup K Ghosh and Gary McGraw to certify security of software components.

Amarpreet Singh Saini

[10] Component models of software are dependable on other components of the system and their behavior in addition to their own. Security assessment occurs at two levels for any given software i.e. component level and the system level. Hence firstly certification of components is required so as to describe that a component will not exhibit undesirable behavior when implemented into the system. And also at the system level, even though a component may behave secure individually by itself, however due to integration it might result in instability and unwanted side effects into the system. After the components have been tested they have to individually be certified in some way so that the consumer can be sure of their quality. Digital signatures serve this purpose well. These are encrypted pieces of code that are well implemented in browsers of todays times. Also after the individual certification of the components a system wide analysis is needed to be performed in order to check for any unwanted behavior that may have been introduced by the integration of the component. IPA or interface propagation analysis is a fault injection technique proposed by Voas, Graw, Ghosh and k. Miller that perturbs the data at the component interfaces and check for any unwanted behavior or anomalies. Hence the certification pipeline is used to certify the components and IPA is used to analyze the system for any anomalies that may have been introduced by the integration of components.

Software Certification

4.5 Verification and Validation techniques. Verification and validation techniques are a very important part of the software engineering process. Due to the large complexity of software systems today V&V has become a necessary step that in integrated in all the stages of a software development lifecycle. Unlike in previous times where it was a highly informal process V&V process today are highly formalized and are closely linked with certification as the quality of verification and validation, and the techniques used to perform them play a major role in determining the quality of its certification. Early in development of the software a certification liaison is agreed upon between the developer and the other party if present. Next a verification plan is provided to the certifier for approval which determines the techniques and framework that the verification process. After that the certification agency can call upon the developers for a discussion where any misunderstandings in the submitted plan be solved. This continues for the whole lifecycle of the development process where documentation is continually supplied to the certifier to ensure that the steps are been followed properly. One more important document that is given a high priority during certification process is called safety case document. It is checked by the certifying authority to see that all the hazards have been pinpointed and measures taken to mitigate their occurrence and proper recovery measures are in place. Safety case document is dynamic in nature as risks and other hazards are uncovered during the whole development lifecycle. Items that should be included in the safety case include, but are not limited to the following: specification of safety requirements, results of hazard and risk analysis, verification and validation strategy, and results of all verification and validation activities. The CONTESSE Test Handbook, which is applicable in the United Kingdom, lists a number of items that should be included in a safety case[12][11] Verification as defined by IEEE is "The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase." Hence it means that the software confirms to the requirements of that phase at the beginning [11] Validation is defined as "The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements." Validation hence means that the system produces correct output. Various Verification techniques exist which can broadly be classified into 2 categories. These are Dynamic and Static testing. Dynamic testing includes the testing of system or its composite components. Static testing on the other hand does not include testing of the working of the system and its components. Dynamic testing Functional testing - This is a form of blackbox testing. It involves testing of all the functions of the system or the individual component. Structural testing - This is a form of whitebox testing and uses the information of the internal structure of the component or the system to test its operation. Random testing - Here random test cases are chosen that are taken as inputs to the system. This is a very good way to detect faults as a complete testing using all sets of inputs is practically impossible in todays times due to size and time constraints.

Amarpreet Singh Saini

Static Testing Consistency techniques - This refers to various techniques that are used to verify various different properties of a system. These include correct syntax of expressions, specifications, parameters b/w expressions and values, typing etc. Measurement techniques - These techniques are used to measure various properties of a system such as error proneness, overall structuredness etc.

Validation techniques are performed at the end of a phase where the entire system is tested for discrepancies.

4.6 Formal methods Formal methods include tools and techniques based on the principles of mathematics and formal logic for the investigation of the properties of a computer hardware and software system. Formals methods can be used for both verification and validation. There is a need for formal methods to be used due to the nerve breaking issue of maintaining reliability in large and complex software systems in todays times. Using formal methods one can establish and prove correctness of a system mathematically. Formal methods include Formal specification, Formal proofs, Model checking and Abstraction. Formal specifications are used to describe the external behavior of the system without taking a look at the internal structure. There are 2 broad categories of formal specifications- Property oriented which are used to describe the behavior in a purely declarative way and include algebraic and axiomatic, and model oriented which are used to describe the behavior in a more direct way and include abstract and state machines. Formal proofs are complete arguments which argue about a certain aspect of a system. These are mathematically proved and eliminate any informal derivation. Basically a series of small aspects are proved in steps first which are then used to derive the result of a larger problem. Model checking uses tools called model checkers. Hence this is rather an operational form of method where a state machine model of a system or a component is checked if it conforms to the requirements for that particular component or system. A model checker traverses all reachable paths that are described in the state machine model. Abstraction refers to the processes that eliminate the unwanted details and generate a core basic design of the system. Here the major needed properties are focused on so that proper design choices can be made while beginning to work on the system. Lightweight formal methods were introduced due to the complexity of describing the entire software into a specification language. Here it is proposed that the formal specification methods be applied to only the areas where they are useful i.e. specific areas. When such a methodology is applied different specifications might be used to describe various parts of the software subsystems while ignoring the ones where they will not be useful. A few examples are LARCH which provides two levels of specifications, SML or Standard Meta Language which is used to describe situations related to type theory, HOL or High Order Logic which is used to prove simple theorems and hardware logic.

Software Certification

4.7 Fault Injection Fault injection is a method where faults are inserted into the system and then the results of the executions are checked. This is very helpful and an important technique as the behavior of a system is checked to see if it can handle faults in modules well if it ever arises due to a situation such as component upgrade etc. Fault injection can be both hardware and software. Squeeze play process is a white box software reliability verification method which is used to determine if adequate testing has been done so that the smallest predicted fault that was inserted into the software was detected. The squeeze play model on a probabilistic basis provides data that the software for that particular predicted fault size is correct. [9]

4.8 Dependability analysis Dependability analysis includes the identification of hazards and discussing of techniques to mitigate them.

4.9 Hazard analysis Hazard analysis involves using guidelines to identify various hazards and the reasons of their occurrences.

4.10 Risk analysis

10

Amarpreet Singh Saini

Risk analysis proposed by kopetz takes into consideration the impact a hazard can have and the probability of the occurrence of the hazard. Here financial impact of the hazard is also considered and accordingly the hazards are classified according to a risk factor.

6.

Certified Compilation

A compiler compiles the source code from a language into the assembly code which can be executed by the processor. Compilers are a very essential part of the software engineering process. It was seen that production quality compilers we so complex even in 1998 that they eluded any form of verification attempts at them. [4] Compilers are hugely complex programs which perform complex symbolic transformations. Bugs in the compiler can make a program that was correctly coded become a buggy executable due to no fault of its own. Compiler correctness is a big issue in high assurance complex software. For low assurance software testing done on the final executable can uncover the bugs in the compiler as well but this changes as we turn out attention to large software. In large and high assurance software we always apply formal methods on the source code itself. However, all this labor can be a waste due to bugs in the compiler. The source code can have been formally verified and so can the processor be. The compiler can however prove to be a weak link which can be very frustrating. This is alleviated by manual assembly code checking done on the part of the software developers which is hugely time consuming and expensive. Most of the ti me any object code that cant be directly traced to the source is properly verified which can be quite a process. Compilers are debugged using large numbers of test cases, some of which are distilled at considerable expense from large application programs. [4] The effectiveness of testing is badly affected and is a huge drain on resources as it is almost impossible to test all the executions paths of the compiled programs. A certifier automatically checks the type safety and memory safety of any assembly language program produced by the compiler. The result of the certifier is either a formal proof of type safety or a counterexample pointing to a potential violation of the type system by the assembly-language target program. The ensemble of the compiler and the certifier is called a certifying compiler.[4] One of the approaches is to check the compiler itself for errors and bugs using the formal methods such as model checking and program proof, in order to ensure that the semantics of the source are properly translated by the compiler into machine code. This formal verification of a compiler consists of establishing certain correctness properties between the source code and the compiled code. [5] Some examples of this are-

[5]

11

Software Certification

Various research ideas have furthered the notion of a certified compiler. George C. Necula Peter Lee described a method that includes type safety and employs symbolic evaluation to fully verify the output code with the input source. Morrisett, Walker, Crary and Glew proposed a way to implement a certifying compiler that uses a rened type system culminating with a typed assembly language (TAL). [7] Proposed by Andrew W. Appel Edward W. Felten Zhong Shao FLINT is a language and platform independent representation of mobile code. FLINT is intended as a typed common intermediate format for object-oriented, functional and imperative languages. [8]

7.

Conclusion

There are a vast variety of issues that software certification faces. The primary issue is the balancing of certifying either the components of the software system or certifying the methodology or the process of development. When a certified method is used to make different products the different products will not be needed to be certified. However this can lead to other coding related bugs in the system. If the components of the system are certified, then one can change the methodology at free will without having to submit changes and thus forcing a recertification for the product. For more complex IT systems it is more feasible to certify the processes as bugs in the system will be across wide range. Additionally certification is only a rating and does not prove that the system is free of bugs and defects. Certification only depicts that a product has met certain criteria which signify it managing to reach some standards. One of the biggest issues faced by certification methods is a lack of standardization of the certification processes and methodology. Various stands such as ISO 9000, IEC 1508 etc exist which are a major barrier to adoption by various different sectors of the industry and lead to confusion. Recertification is also a major issue due to the time it takes to certify a product over again. If certification tools are improved and can provide estimates such as time and effort required for the certification process then there can be a widespread adoption of certain standards which can be followed industry wide.

8.

References

[1] Voas, J. ;( 1999) "Certifying software for high-assurance environments," Software, IEEE, vol.16, no.4, pp.48-54, Jul/Aug 1999[Accessed 23 May 2011]. URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=776948&isnumber=16869

12

Amarpreet Singh Saini

[2] Voas, J.(2000) "Developing a usage-based software certification process," Computer, vol.33, no.8, pp.32-37, Aug 2000[Accessed 23 May 2011]. URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=863965&isnumber=18714 [3] Ewen Denney and Bernd Fischer. (2005) "Software Certification and Software Certificate Management Systems". In Proceedings of 2005 ASE Workshop on Software Certificate Management . Long Beach, CA, pp. 1-5, Nov. 2005. [Accessed 23 May 2011]. URL: http://ti.arc.nasa.gov/profile/edenney/ [4] George C. Necula and Peter Lee. ( 1998). The design and implementation of a certifying compiler. In Proceedings of the ACM SIGPLAN 1998 conference on Programming language design and implementation (PLDI '98), A. Michael Berman (Ed.). [Accessed 23 May 2011]. URL: http://doi.acm.org/10.1145/277650.277752 [5] Xavier Leroy. (2006). Formal certification of a compiler back-end or: programming a compiler with a proof assistant. In Conference record of the 33rd ACM SIGPLAN-SIGACT symposium on Principles of programming languages (POPL '06). [Accessed 23 May 2011]. URL: http://doi.acm.org/10.1145/1111037.1111042 [6] George C. Necula. (2000). Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation (PLDI '00). [Accessed 23 May 2011]. URL: http://doi.acm.org/10.1145/349299.349314 [7] Greg Morrisett, David Walker, Karl Crary, and Neal Glew. (1999). From system F to typed assembly language. ACM Trans. Program. Lang. Syst. 21, 3 (May 1999), 527-568. [Accessed 23 May 2011]. URL: http://doi.acm.org/10.1145/319301.319345 [8] Andrew W. Appel, Edward W. Felten, Zhong Shao (1999) Scaling Proof-Carrying Code to Production Compilers and Security Policies [Accessed 23 May 2011] URL: http://flint.cs.yale.edu/flint/publications/pccwhite/index.html

[9] Jeffrey Voas and Jeffery Payne. (2000). Dependability certification of software components. J. Syst. Softw. 52, 2-3 (June 2000), 165-172. [Accessed 23 May 2011] URL: http://dx.doi.org/10.1016/S0164-1212(99)00143-0

[10] Ghosh, A.K. and McGraw, G., (1998). An Approach for Certifying Security in Software Components. Reliable Software Technologies. [Online]. [Accessed 23 May 2011] URL: http://csrc.nist.gov/nissc/1998/proceedings/paperA4.pdf

[11] Eushiuan Tran(1999) Verification/Validation/Certification Spring 1999 [Accessed 23 May 2011] URL: http://www.ece.cmu.edu/~koopman/des_s99/verification/ [12] Storey, Neil, Safety Critical Computer Systems, Harlow, England: Addison-Wesley, 1996 [13] Robert Frohardt (2009) A Brief Introduction to Separation Logic December 4, 2009 [Accessed 23 May 2011] URL: http://www.cs.colorado.edu/~frohardt/SepLogic.pdf

13

Software Certification

[14] Ewen Denney and Bernd Fischer. (2005) "Software Certification and Software Certificate Management Systems". In Proceedings of 2005 ASE Workshop on Software Certificate Management. Long Beach, CA, pp. 1-5, Nov. 2005. [Accessed 23 May 2011]. URL: http://ti.arc.nasa.gov/m/profile/edenney/papers/Denney-Fischer-softcement05.pdf [15] Shao, Z., (2007). Certified Software. [Online]. [Accessed 23 May 2011]. URL: http://flint.cs.yale.edu/flint/publications/certsoft.pdf.

14

Amarpreet Singh Saini

Software certification

Software certication comprises a wide range of formal, semi-formal, and informal assurance techniques, including formal verication of compliance with explicit safety policies, system simulation, testing, code reviews and human sign offs

Tracing bugs

Certifying Software

Kingston University
15

Software Certification

Wide range of formal, semi-formal, and informal assurance techniques,

including formal verication of compliance with explicit safety policies,

system simulation, testing, code reviews and human

sign offs , and

even references to supporting literature. safety policies, system

Lack of metrics High maintenance of reliability is difficult Long wait for the process to complete No uniform certification process Learning Casual relationships among node

Challenges in Certifying Software

Certification Process
16

simulation, testing, code reviews and human

sign offs

Amarpreet Singh Saini

Most Popular approaches for Software certification

CURRENTLY USED APPROACH Process Based CERTIFIES THE PROCESS OTHER DEVELOPERS CAN FOLLOW SUIT

BASED ON TESTING THE PRODUCT Product Based IMPLEMENTED IN THE SOFTWARE DEVELOPMENT LIFECYCLE ITSELF

RELIABLE CERTIFIED SOFTWARE LESS BUGGY PRODUCTIVITY IMPROVEMENT

Kingston University

17

Software Certification Certified Software

Framework for building certified software

Certied software consists of a machine-executable program C plus a rigorous formal proof P (checkable by computer) that the software is free of bugs with respect to a particular dependability claim S.

Kingston University

18

COMPILER CERTIFICATION
Amarpreet Singh Saini

What is Compiler Certification? A certifier automatically checks the type safety and memory safety of any assembly language program produced by the compiler. The result of the certifier is either a formal proof of type safety or a counterexample pointing to a potential violation of the type system by the assembly-language target program. The ensemble of the compiler and the certifier
Formal verification of a compiler consists of establishing certain correctness properties between the source code and the compiled code. Some examples of this are-

is called a certifying compiler

Kingston University

19

Software Certification

RECENT ADVANCES
PROOF CARRYING CODE HOARES LOGIC LOCAL REASONING AND SEPERATION LOGIC DOMAIN SPECIFIC LOGICS AND CERTIFIED LINKING CERTIFIED GARBAGE COLLECTORS AND THREAD LIBRARIES

Kingston University

20