Experiences with Formal Verification in the Design of High Integrity Embedded System & DAE Initiatives A.K.

Bhattacharjee BARC
April 8, 2013 NAL, Bangalore 1

Computing and Control

In DAE, some regulators feel it is not right to use computer/ software for safety systems of NPPs
Fear is built by computer world jargons like
Bug, Exceptions, NaN, RMA, hacking, cyber attacks

Their fear is also aided by media hyped infamous computer systems failures

April 8, 2013

NAL, Bangalore

Infamous Computer Caused Errors

July 28, 1962 -- Mariner I space probe 1982 Blast in Trans-Siberian Soviet gas pipeline 1985-1987 -- Therac-25 medical accelerator. 1988 -- Buffer overflow in Berkeley Unix finger daemon 1988-1996 -- Kerberos Random Number Generator January 15, 1990 -- AT&T Network Outage (ESS-4) (Domino Effect) 1994 -- Intel Pentium floating point divide 1995/1996 -- The Ping of Death 1996 -- Ariane 5 Flight 501.. 1999 Mars Polar Landar
(http://news.bbc.co.uk/2/hi/8505402.stm)

2010 Toyota Recall of 200,000 Prius

April 8, 2013

NAL, Bangalore

Subsystem Propulsion Guidance and navigation Electrical Operational ordnance Structures Software and computing Pneumatics and hydraulics Unknown

1980s 42% 6% 6% 2% 4% 0% 4% 37%

1990s 38% 16% 8% 8% 6% 8% 2% 16%

2000s 54% 4% 8% 0% 0% 21% 0% 13%

Source: DEVELOPING SAFETY-CRITICAL SOFTWARE REQUIREMENTS FOR COMMERCIAL REUSABLE LAUNCH VEHICLES, FAA/NASA, 2006 (www.faa.gov)

April 8, 2013

NAL, Bangalore

Evolution of Computer Application in Indian NPPs

Data Acquisition and health monitoring of hardware systems in Dhruva reactor at Trombay and FBTR at Kalpakkam (1980-85) Safety related : Reactor power Control in 220 MWe Reactor at Narora, UP, 1987 Safety: Reactor Protection in 220 MWe Reactor at Kakrapar, Gujrat 1989 All major Control, Monitoring, Test, Surveillance and Operator Information functions in TAPP-3&4 are computer based systems. 2004
April 8, 2013 NAL, Bangalore 5

Late Eighties and early Nineties

I&C Designers started moving towards application of Computers
Qualification Issues Lack of Standard Review Process DAE Initiated IV&V practices
Initial Challenges in Verification of Reactor Protection System
Design reviews, Code Inspection Verification Tool Unavailability Massive manual efforts

Need for Safety Classification

Development of AERB Guides D-1,D-10,D-20

Identification of Software Process Standards

Inputs from IEEE 7.4.3.2, IEC60880, US NRC Development of AERB D-25

Development of In-house Static Analysis and Visualisation Tools (Assembly and C) (1989-95) Identified Thrust Areas in Formal Methods, V&V

April 8, 2013

NAL, Bangalore

Issues with Computer based I&C Systems

April 8, 2013

NAL, Bangalore

Software & Hardware Caveats: Technology & Regulatory Issues

Complex Requirements (SW & HW)

How to check Inconsistency of requirements?

Software

Infinite State System

Difficult to test for Corner case bugs: How to reduce dependency on low level testing ?(Effectiveness depends on Human factor) Internal States difficult to access : How to design for Effective Monitoring to make systems fail safe and resilient? Software interfaces are conceptual: How to conduct Effective Reviews?

Trustworthiness of Compiler & Operating System

Output of a compiler is put onto the controller running a RTOS

NAL, Bangalore 8

April 8, 2013

Software & Hardware Caveats: Technology & Regulatory Issues

Hardware

Trustworthiness of Processors

Programmable Hardware Devices like FPGA

Implementation of the Instruction Set Architecture (has it been implemented correctly?) Robustness from security (Has it been evaluated from perspectives of system security e.g. IEC62443?) Software Issues (Challenges & Issues with Design Process & Tools are same as in software) Device Failure : Can we monitor their internal states to predict failure?

Analysis of Software Hardware Interactions (Can we model sw-hw interaction for analysing complex interactions to validate behavioral and performance aspects?)
NAL, Bangalore 9

April 8, 2013

Software & Hardware Caveats: Technology & Regulatory Issues

Security Vulnerabilities & Evaluation of Embedded Systems (Evaluation of

Architecture, Software, Hardware & Communication and Need for Security Plan)

Human factors in Design & Verification.

What is the state of the Art?
(e.g. IEC60880: Methods which are applied purely manually are highly error prone. Therefore they should be supported by tools which use mathematical techniques to reveal the structure and internal function relationships of the software and to check for internal consistency, consistency with some prior model, desirable/undesirable properties, etc)

April 8, 2013

NAL, Bangalore

10

Software System Safety

Safety-Critical Software Functions are those software functions failure of which can directly or indirectly, in consort with other system component behaviour or environmental conditions, contribute to the existence of a hazardous state

April 8, 2013

NAL, Bangalore

11

Safety and I&C Systems

When I&C systems perform functions important to safety, these systems must be demonstrated to be safe and reliable with appropriate degree of confidence. Safety critical functions must be identified based on Postulated Initiating Events (PIE)

April 8, 2013

NAL, Bangalore

12

Generic Requirements of Software Performing Safety Critical Functions

Upon detecting an anomaly or failures, the software should remain in or revert to a safe state (Runtime Monitoring?) Override commands should require multiple operator actions The software should notify the controlling executive during or immediately after transiting to an unsafe state This is in addition to the application logic Hence Requires a thorough Design Verification
April 8, 2013 NAL, Bangalore 13

Computation Errors
Calculation or computation errors (incorrect algorithms, calculation overflow, etc.) Data errors (out of range data, incorrect inputs, large data rates, etc.) Logic errors (improper or unexpected commands, failure to issue a command, etc.) Interface errors (incorrect messaging, poor interface layout and design, etc.) Environment-related errors (improper use of tools, Compilers, changes in operating system, etc.) Hardware-related errors (memory errors, SEUs, unexpected computer shutdown, etc.)
April 8, 2013 NAL, Bangalore 14

Assessment

Evidence that the

software is correct (with respect to

specifications), Safe (functionally, security) completely implements the requirements.

In other words, the software in these systems must be demonstrated to be safe and to have high level of integrity.
April 8, 2013 NAL, Bangalore 15

Assessment

Integrity should be assured by developing system/software using systematic, technically appropriate, carefully controlled, fully documented and reviewable engineering process, which is suitably interfaced with V and V activities. Emphasis on Process

April 8, 2013

NAL, Bangalore

16

Why we need Process Standards?

Assessment of software is guided by a regulatory standard. The documentary evidence is recorded as a safety case. Since dependability cannot be derived concretely from an assessment, we need an assurance on the development process. Because we cannot demonstrate how well we have done, we demonstrate how hard we tried Dr. John Rushby, SRI
April 8, 2013 NAL, Bangalore 17

Formal methods in Design and Verification: Industrial Initiatives in Safety Systems

The process for getting a formal proof for the certification process is not well documented. European documents (such as IEC 61508) recognize formal methods/ proofs. Advances have been made in the area of formal methods that are now more practical and viable in "real life" than when standards like IEC60880/ DO178B were written. DO178C has recommendation on formal methods.
April 8, 2013 NAL, Bangalore 18

IEC 61508 SIL

Safety-integrity : probability of a safety related system satisfactorily performing the required safety functions under all the stated conditions within a stated period of time. Safety integrity level : discrete level (one out of a possible four) for specifying the safety integrity requirements... where SIL 4 has the highest level of safety integrity and SIL 1 the lowest.

April 8, 2013

NAL, Bangalore

19

IEC 61508, Risk Analysis and SIL

Risk analysis guides risk reduction.
By the allocation of development resources.

A Class 1 (Intolerable) risk usually

requires software designed to SIL3/4 (highest)

level.

A Class 2 (Undesirable) risk might

Require software designed to SIL2/3 levels.

Higher SILs require more resources

NAL, Bangalore 20

April 8, 2013

Verification of Software Performing Safety Functions

Architecture : Event Triggered/Time Triggered Functional and performance requirements

Functional Logic Control flow, data flow, information flow, Timing

Interrupt handling and exceptions handling in embedded systems Appropriateness of Finite Arithmetic, pointers and Buffer usage Communication protocols Compliance to quality standards and programming guidelines Absence of malicious programs.
April 8, 2013 NAL, Bangalore 21

Testing Is Complex

April 8, 2013

Dependability Assessment should not guided by Testing Alone

NAL, Bangalore

22

Checking compliance to MisraC is not enough

April 8, 2013

Need for Rigorous and Precise Program Analysis to detect data flow anomalies, RTE NAL, Bangalore

23

Need to think beyond Testing

Verification of High Level Requirements (HLR) Arsenals: HL Modelling and Model Checking
(Statecharts, HMSC, Model checking, SCADE)

Verification of Low Level Requirements (LLR) Arsenals: Verification of Code

(Assertion Checking Environment (ACE))

Verification of Absence of Runtime Errors (RTE) Arsenals: Automated Rigorous Program Analysis
(ACE-II, Astree)

But What about Tool Assessment?

April 8, 2013 NAL, Bangalore 24

Assertion Checking Environment

LLR

LLR
HLR

LLR
Call Tree

pre {program} post

checkTrip

readRTD

applyLogic

genSignals

writeDevice

April 8, 2013

NAL, Bangalore

25

April 8, 2013

NAL, Bangalore

26

Experience
Verification of Reactor Trip Logic Verification of Function Blocks (~80) of an Inhouse PLC Type safeness and verifying against Runtime Errors Translation Validation Also used for external projects from ADA, VSSC, DRDL
April 8, 2013

Used PVS as backend Theorem Prover Not Automatic and difficult to be used by Design Engineers

NAL, Bangalore

27

Model based Design

Developed few controllers with SCADE and verified controller properties If the loss_of_electric_load or turbine_trip signal is true and reactor power exceeds 20% Full Power then Anticipatory Action (AA) should start and should get completed in time T1+T2, where T1 and T2 are predefined constants. AA lowers the operational set point (OPSP) in proportion to reactor power based on the following equation OPSP = NLSP - K * T

April 8, 2013

NAL, Bangalore

28

Rigorous Program Analysis

Code is what runs and controls real world Guarding against Run Time Errors due to modular & finite precision arithmetic and heaps Rigorous program analysis to detect possible computation errors Issues are with False positives and False Negatives
More precise analysis is required
April 8, 2013 NAL, Bangalore 29

Field Programmable Devices (FPGA)

Requirement Specification Manual Programming HDL Design Simulation

Synthesis

Place & Route Tool Certification

Bit Stream Generation

April 8, 2013

NAL, Bangalore

30

Formal Verification Tool for VHDL Designs (CFDVS-BARC)

Symbolic Constraints

Property

Property Translator
Abstraction/ Refinement Manager

Abstraction Refinement Hints Transition Relation Verification Condition Generator

Property Satisfied/ Counter example

VHDL Design

IR Translator

IR

Symbolic Simulator

Constraint Solver

Achieves scalable verification of designs with both data and control dominated sub-parts Combines symbolic simulation, abstraction-refinement, and bounded model checking with word-level constraint solving

 Used for the functional verification of VHDL designs used in in-house developed Hardware
boards used in C&I Applications

Functional Verification of Hardware Design

Properties in PSL extracted from FPGA Requirements Specification and submitted for
verification

Counterexamples produced by the tool helped in increasing precision of specification Papers in CAV'11 and TACAS'13
April 8, 2013 NAL, Bangalore 32

Few Other Formal Verification Projects

Verification of FIT Logic of ECCS for Dhruva Reactor using Esterel, Auto and Duration Calculus Internal Communication Protocol by Spin Object Code Verification for ADA (with TIFR) Required large implementation

April 8, 2013

NAL, Bangalore

33

Who will verify Tool?

April 8, 2013

NAL, Bangalore

34

Tool Assessment
Independent Output Assessment: Can the output of the tool be verified to be correct through an independent means? Some possibilities include manual review of tool output, comparison with a second equivalent tool . Relevant History: Does the tool have a welldocumented history of usage where it has consistently produced acceptable results? The history of usage may include similar applications.

April 8, 2013

NAL, Bangalore

35

Can tool insert error in software?

No No Qualification Necessary

Yes
Will the tools output be verified as per applicable standard?

Yes

No
Who will do?

No
Are processes of standard Eliminated, reduced or automated by use of tool?

Tool Qualification Requirement

April 8, 2013 NAL, Bangalore

Yes

Tool must be Qualified

36

Some Discussions

April 8, 2013

NAL, Bangalore

37

Effort Reduction for Better Design Review

Better Models of higher level specification

Data Flow Equations
State Transition Diagrams Hierarchical Designs

Can be reviewed by domain experts

Invest efforts in validating automated code generators

Verify once and use many Reduce Efforts in Programming and Unit Testing Admissibility of Regulating Norms to be seen

Promote Component based Designs

Reuse Verified components with due care of environment

and rigorous traceabilty.

April 8, 2013 NAL, Bangalore 38

Promote Reviewable System Design

Design Tools:
Domain Specific Modelling Language, Automatic code generator (Verify Once) Proof obligation generator (For Safety

Functions)

Analytical Tools:
Compliance analyzer (Rigorous Program

Analysis) Formal proof generator (For Safety Functions) Security Properties

April 8, 2013 NAL, Bangalore 39

New Issues
Can we be future ready?
Distributed Heterogeneous System Handling Multicore processors Failsafe Programmable FPGA designs Languages
Model based, Type safe languages, Functional Compilation issues Rigorous Program Analysis

Vulnerabilities on modern architectures

April 8, 2013 NAL, Bangalore 40

How to convince end user?

Formal proof is (significantly but not exclusively) about showing you got it right (and hence part of the argument for dependability claims) but, even then, proof is just formal reasoning about some properties of the specification and design, and proofs rarely compel a sceptic to become a believer. So proofs are more important to the engineer than to the customer or public. Prof. Bev Littlewood, UK,
Safety Expert
April 8, 2013 NAL, Bangalore 41

Soundness For Formal Methods?

Of the formalism Of the software tools Of Axioms and assumptions How to convince Certification Authorities What should be provided to certification authorities about soundness? Operational Semantics, Proof system, Theories of bit vectors? How long the proof lasts! If I do not understand your equations, I may not certify Formal Methods inspired Debugging and Testing to build confidence
NAL, Bangalore 42

April 8, 2013

DAE Contributions through Extramural Research Setting up of CFDVS at IIT Bombay to promote research in Formal Verification Techniques. Development of Tools and Techniques with improved precision and scalability in collaboration with CFDVS. Development of Automated Program Testing Techniques at IIT Kanpur.
April 8, 2013 NAL, Bangalore 43

Thank You

April 8, 2013

NAL, Bangalore

44