Professional Documents
Culture Documents
Taking On The Verification Challenge With Better Performance, Capacity, and Productivity Avery Design Systems
2 Atwood Lane Andover, Ma 01810 978 689 7286 www.avery-design.com
Seminar Agenda
Industry Trendlines Averys perspective and vision Avery solutions and products
10X for ASICs 100X for ASIC-based systems and SOCs which include embedded software
Verification Complexity = (Architectural Complexity, Amount of Reuse, Clock Frequency, System Software Worst-Case Execution Time, Engineer Experience)
Time to Market
Increasing design complexity imposes a thorough functional verification process Verification needed at all levels : Block - Module - System
How SOCs and Embedded Systems Exponentially Expand the Scope and Scale Embedded System Host System of Verification ? /SOC
System Applications Operating System Device Drivers Hardware Devices Embedded System Application Embedded OS/Firmware Device Drivers Hardware Devices System Verification Target
Host System
ASB-PCI Bridge
1. 2. 3. 4.
Multiple core functions On-chip interfaces protocols Multiple off-chip interface protocols Embedded processors, firmware, OS
Algorithm
Implementation verification
Circuit-level
Formal Equivalency
Gate/Structure HW Emulation
Timing RTL
HW Emulation
Manufacturing verification
At-speed
Manufacturing defects
RTL Modelling
System verification:
Does the overall design work? - Run application : check end result Does the communication between blocks work? - Check blocks synchronization, latency Does an input stream generate the expected output behavior? - Check block functionality and corner cases.
Module verification:
Block verification:
Module
Block
Chip NREs increasing making respins an unaffordable proposition RESPIN Average ASIC NRE ~$122,000 SOC NREs range from $300,000 to $1,000,000
Self-checking methodology is standard At the end of the system verification a high confidence level is reached
It is felt that too much work is involved and little pay-back Nobody is really responsible for module verification
10
Often stimuli-only and visual inspection of simulation results Structural coding style instead of behavioral Testbench derived from RTL implementation and not specification
Too much verification / Debug needs to be done at system level Resulting in a long verification process (70% of the design cycle time)
11
System verification
Not suited for finding RTL implementation bugs Do I get the correct end state results? The sequence of operation is not relevant Signal activity is not relevant Run application / test case and check that at the end I get the expected memory / register image It is too hard to define the sequence of operation; a cycle accurate circuit model is typically not available. Application specialist; not necessarily RTL design / VHDL literate.
Typical approach
Engineering type
12
Test plan
Application requirements Overall performance tests Finding / debugging RTL bugs at system level is extremely time consuming RTL designers might not be available to support debug process Long iteration cycles
Re-verify block and module functionality before restarting system verification Implies new synthesis runs
Finding Bugs late in the design process can substantially delays tape-out. Bugs will/should be found but their number should be minimized by extensively verifying Blocks and Modules
13
Module Verification
M 1 M 3
M 2
B1 B5 B3 B2 B4
...
M n
14
sufficient cache-memory and FIFOs size sufficient inter-block bus bandwidth optimal processing power partitioning
15
Block verification
Data flow2 Data flow1 Rese t Wai t Acces s Rea d Writ e
Block
Checker 1 Checker 2
16
Verification goal
Debug RTL implementation Verify implementation is according to spec Verify block post-synthesis netlist RTL designer Possibly not the same person who did the RTL implementation simple data flow one process for each data stream define test cases before RTL implementation develop test cases before or during RTL implementation
Engineering type
Testbench architecture
Process
17
Easy and fast to write for RTL designer use HDL language at behavioral level Self-checking : mandatory for efficient verification Visual inspection is error-prone especially in debug cycle Hierarchically structured allows abstraction: dataflow - operation - signal Reusable direct reuse of (protocol) generators and checkers from block to modules reuse from project to project when same basic protocol is used : ARM AMBA Bus, ATM-UTOPIA, PCI, SONET,...
18
1010
107
105
G.7XX, G.168, DSL, IP 1-2M gates 40 Gbps per wavelength 3.2 Tbps fiber capacity 622 MHz (SONET/SDH 192), 322K bits/frame (STS-192) 25M gates / wavelength OC-3 to OC-48, ATM 30 M gates 45 M transistors
1013
1010
107
Terabit router
1012 1016
109 1012
106 106
Intel P4 microprocessor
19
20
Simulation-based
Directed testing
Generate testcases for each test assertion and related test parameters Interfaces exercised independently Block, module, and system-level Generate testcases for one or more test assertions concurrently Test parameters selected randomly from enumerated list Multiple interfaces exercised concurrently Block, module, system-level Automatic functional test generation Deterministic process using SAT solver engine(s) Guided by functional coverage targets
Pseudo-random testing
Auto-directed testing
Formal methods
Prove design complies with design properties Or not, generate counter examples Verifies specific assertions/properties in the context of actual simulation Plus, searches limited state space around seed state (assertion triggers) for noncompliance and generates counter examples
21
22
23
Design productivity
Faster simulation through abstraction Using the same language for HW and SW must make HWSW co-design and verification easier Software engineers can lend a hand designing the hardware
Verification productivity
High-level Verification Languages (HVLs) are the only way to implement transaction-level verification Formal model checking is on the verge of becoming practical
25
Withering?
Draft standard Co-designs Superlog combines C constructs and Verilog together by June 2003
SystemVerilog Language Standard builds on IEEE-1364-2001 Averys TestWizard adds verification features to Verilog IEEE-1364-1995, VHDL 1076 (VCS, NC-Verilog, MTI)
26
27
Multi-bit operators can be easily be overloaded as C++ operators You dont care about timing, X, and design contains no bus
Fast simulation, some people claim that simulation is 100X faster than Verilog
But the design description level was different between the two There is no inherent speed advantage if models are described at the same level Difficult to handle large amount of parallelism
Pay nobody
Difficult to correlate design bugs to C or C++ model Difficult to do incremental refinement of the design Synthesis and other tools are not mature Cannot accept legacy code
28
No backwards compatibility
Verification-centric enhancements do not have same priority as design and synthesis-centric enhancements
29
Verilog and C are at the same abstraction level with only data structure limitation
Designs today only use a very limited subset, restricted by Synopsys synthesis subset
30
Innovate new verification technologies and flows addressing RTL, HW-SW systems, and IP Build on customers investments in tools, language standards, and engineering know how
31
Transaction-oriented verification
Raise test generation abstraction and tools to match rise in architectural complexity Standards-based design increases design and interoperability and designer productivity System-level simulation performance/capacity Early HW-SW integration Design management Use only existing standard implementation languages
Reusable verification IP
32
5-10 X increase in performance and capacity vs. stand-alone HDL simulators through parallel simulation processing
33
Hardware-Software co-verification
Earlier system SW integration and compliance validation Integrated HDL and C/C++ verification flow for IP
34
TestWizard Is
35
Transaction-based Verification
Decouple testcase content and signal-level interfaces Apply pseudo-random test case generation to generate more thorough set of testcases Eliminate manual result checking and use only dynamic selfchecking tests and transaction database logging
Assertion-based Verification
Use automatic protocol assertion checking to ensure system compliance during every simulation
Coverage-based Verification
Formally measure functional test coverage vs. design architecture Check all properties & scenarios have been tested
36
High-level structured methods and psuedo-random generation generates comprehensive workloads to test design 3X more productivity Up 10X better code density Closed-loop method achieves 100% functional coverage Assertions provide always on validation of protocols
37
38
TestWizard Verification is
Transaction modeling using complex data structures Test generators generate random transaction sequences based on test parameters Transactors stream transactions in/out of DUT Result checking is dynamic self-checking using transaction database queries Assertion-based verification checks protocol compliance on the fly Closed-loop functional & assertions coverage analysis adapts traffic shape for more effective & efficient testing
39
40
Records
Hierarchical (i.e. Records of records) Record Pointers Dynamically allocated/deallocated Dynamic field sizing Unions support composite types Extend base records Randomization
Record streaming functions simplify BFM/transactors Transaction database logging and SQL-like queries to simplify transactionlevel verification Mutable and hierarchical provides unstructured data aggregation Supports string, numeric constants and numeric ranges
41
42
43
Boundary case pseudo-random Weighted Distributions Related case pseudo-random Cyclic pseudo-random Filter functions Enumerated value ranges
44
45
46
47
Does the router generate ICMP Type 11 (timeout messages) back to IP source address when TTL and HLIM reaches zero
Predictor stores IP pkt in Port #0 egress expect xlog db and starts timeout assertion thread Router egress emits ICMP Type 11 packet Dynamic checker examines actual egress pkt and compares to expected egress pkt; eventual match or timeout assertion violation
Is IPv6 QoS met for certain specified flow labels Does router perform correct label stack operations Does router correctly respond to LDP label map requests for good and bad labels
48
#5 Assertions
On the Fly protocol verification of System-level and Interface protocols Powerful & concise temporal property semantics to Verilog/VHDL
Highly reusable Comparable to Sugar (Accellera FPSL WG) Sugar-to-TestWizard translator available soon
49
#5 Assertions contd
roperty: For all read or write ansactions in the address range: hAA:-hBB, data transfers of 1-16 bytes re allowed. Data transfers can complete ut-of-order. On transaction completion TestWizard assertions support complex etection, byte counter must match Overlapping sequences using threads and nitial transaction header byte count
tags
50
#5 Assertions contd
Assertion:
Define requestor address range and byte count range lists Define assertion thread tags for requestor header address and byte count Define assertion tags for split completion address and byte count Define sequence: On new transaction request (start), save address and byte count headers to new assertion thread( h1); wait for end2 to signal split completion (tag3), then match on any threads using completion address; if match exists, on next clock cycle (h4), capture completion byte count (h2); compare completion byte count to requestor byte count
51
#6 Feature-Set Coverage
Answers the Qs
Am I done yet Have I tested everything Are these tests efficient Coverage tasks monitor system input and responses Coverage equations determine coverage level
Coverage models
# Verification Cycles
52
Set overall and incremental coverage criteria to DYNAMICALLY select random modes and control total run length
53
Coverage Example
Profile router workload by profiling output check coverage
// Setup transaction packet verifier profiler initial begin $tb_list_append_values(verify_packet_LH, "verify_ip_icmp_timeout_msg_latency", "verify_ip6_icmp_timeout_msg_latency", "verify_ip6_flow_latency", "verify_mpls_stack_forwarding", "verify_ldp_map_request"); $tb_list_append_values(verify_packet_H, `verify_ip_icmp, `verify_ip6_icmp, `verify_ip6_flow, `verify_mpls, `verify_ldp); $tb_profile_create(verify_packet_PH, verify_packet, verify_packet_H, verify_packet_LH); end /* Update transaction packet verifier profiler */ always @(update_profile) begin $tb_update_profile(top.mon0.verify_packet_PH, top.mon1.verify_packet_PH, top.mon2.verify_packet_PH, top.mon3.verify_packet_PH); test_coverage0 = 25 * $tb_profile_coverage(top.mon0.verify_packet_PH); test_coverage1 = 25 * $tb_profile_coverage(top.mon1.verify_packet_PH); test_coverage2 = 25 * $tb_profile_coverage(top.mon2.verify_packet_PH); test_coverage3 = 25 * $tb_profile_coverage(top.mon3.verify_packet_PH); test_coverage = test_coverage0 + test_coverage1 + test_coverage2 + test_coverage3; if (verbose) $display("test_coverage=%3.3f", test_coverage); end
54
Question: Have I tested every router workload scenario? Measured Answer: Egress packets on port 0 checked for 2 out of 5 workload scenarios
55
Simulator independent write code that runs on all simualtors regardless of (PLI/VPI, FLI/VHPI) Supports both client-server and PLI-based programming models
Leverage diagnostics programs and protocol stacks to generate test content VCI API features
Remote Verilog and C/Perl task calls Remote C/Perl routine call backs Synchronization primitives Verilog-C/Perl variable and record accesses Shared Verilog-C/Perl semaphores
56
Product Roadmap
Core Technology
Guided Search Insight - Automatic functional test generation Intelligent Random TestWizard - VHDL, Perl - Advanced Assertions - Transaction db - ModelSim TestWizard - Sugar
Directed
June01
02
03
57
% Effort
% Bugs Found
58
Uses the same random test generator to deterministically solve for test parameter values that satisfy coverage requirements using
Multi-variable filter constraints Enumerated test parameter value sets Temporal properties (assertions and assumptions)
Generates incremental feature-set coverage reports Outputs test/transaction sequencing to drive regression on all the simulation platforms
Fusion Simulation Technology
Verilog Logic Simulation Symbolic Simulation Temporal SAT Solver
59
Insight Example
Symbolic Simulation
SAT Target
60
Insight Example
Insight solves for missed coverage points Egress result checker (one per port) Ingress packet traffic monitor (one per port) For egress monitor (port 0), generates 3 directed tests using existing testbench and assertions (design properties) Outputs directed test sequencer for regression testing
top.mon0.verify_packet, top.mon0.verify_packet_PH ("verify_ip_icmp_timeout_msg_latency": 32'sh00000001) = 0 : 0.00 ("verify_ip6_icmp_timeout_msg_latency": 32'sh00000002) = 0 : 0.00 ("verify_ip6_flow_latency": 32'sh00000003) = 4 : 57.14 ("verify_mpls_stack_forwarding": 32'sh00000004) = 3 : 42.86 ("verify_ldp_map_request": 32'sh00000005) = 0 : 0.00 (default) = 13 : 185.71
61
Insight Example
Symbolic simulation
Seed environment
Symbolic simulation starts at time 113,000 ns About 2X packet time + interval time (i.e. 1000 clock cycles)
SAT
SAT targets
Egress ports result checker coverage monitors Ingress packet traffic monitors Generate packet from ingress port with route to egress port 0 with following parameters
Protocol: IPV4 TTL field = 1 or 0 Other random parameters are selected randomly
62
Conclusion
Models complex transactions Generates directed and random transaction-level tests Supports powerful transaction database for result validation of complex transaction processing Assertions provide powerful and reusable protocol verification Validates feature set coverage of system operation and port traffic
Insight using SAT solver is 1st coveragedriven, auto-directed functional test generator Delivers 3X better productivity with up to 10X less verification code Improves design integrity with advanced and measurable methods Builds on existing simulators, HDLs, and methods you already use today!
63