Protecting Your Investment:

Creating an Effective FAT/SAT/BAT

Procedure

October 21
st
, 2009
Presented in conjunction with the
Laboratory Robotics Interest Group
New England

Open Source Software Initiative
Agenda
Introduction
Types of Laboratory Automation Systems
Creating a Project Plan
Factory Acceptance Testing
Site Acceptance Testing
Biological / Chemical Acceptance Testing
Lessons Learned
Summary





Some Definitions
FAT - Factory Acceptance Testing

SAT - Site Acceptance Testing

BAT - Biological Acceptance Testing

CAT - Chemical Acceptance Testing

Introduction
Purchasing Laboratory Automation is an investment
What are you buying (workstation or integrated system?)
Adding a new capability or enabling technology?
Not always about efficiency

Properly plan for success
Define your project goals and criteria for success before
engaging any vendor
Design a Test Plan which uses the actual workflow(s)
Test the new capability successfully adds value



Getting Started
Map your process


Build a Project Charter
Why automate?
Added value?, Criteria for success?
What can kill the project?

Setup a Project Team and Determine Stakeholders
Include an Automation Lead/ Project Manager, Scientific Lead, &
Purchasing Representative
A Facility Lead and Software Lead may be necessary depending
on the complexity of the system
Add Reagent Incubate Add Reagent Read
Getting Started (II)
Capture User Requirements Specification (URS)

Use requirements to define an objective Request for
Proposal (RFP) process

Select a vendor which..
Is willing to work with you to properly test the system
Is willing to tie milestone payments to project phases
URS RFP
Vendor
Selected
Contract
Workstations
Workstations are standalone pieces of equipment
integrated with a small # of peripheral devices
Typically stacker based and designed for a specific
workflow

Integrated Systems
Integrated Systems vary in complexity and usually
include 1 or more robotic arms and a number of
peripheral devices
Typically support a number of different workflows and
are controlled with a scheduler package
Integrated Systems Continued
Integrated systems can be designed to serve a
number of processes

Historically integrated systems have been built
to support:
High Throughput Screening (HTS)
Ultra High Throughput Screening (uHTS)
Compound Storage and Replication
Combinatorial Chemistry
Cell Culture
Project Plan
In general, the project plan is similar in all cases
and should include:
project goals
the process map(s)
Schedule and milestones
Resource plan
Payment terms
Reference to test plan documents (ATPs)


Project Plan Continued
Consider breaking project into separate milestones with payment
percentages tied to the completion of these milestones

Ensure that there is a written and clear understanding with
vendor for what acceptance testing will be required

Recommend leaving a percentage of total project payment as a
final milestone payment

Identify a clear procedure for change management

Consider applying payment penalties for
Missing key deadlines
Not satisfying system / design requirements
Example Project Plan Timeline
PO
Accepted
Define
System
Specifications
Build and Test Ship and Install
Kickoff
Meeting
Agreement on
Acceptance
Testing
Define Functional
Design Spec
Define
Assay/Consumables/
Factory
Acceptance
Test
Site Acceptance
Test
Bio/Chem
Acceptance
Test
Acceptance Test Plan (ATP)
Draft a formal document (ATP) for testing procedures for
each test phase (FAT / SAT / BAT)

ATP should include language which
Outlines requirements for testing procedures
Outlines Testing Scripts
Defines criteria for success / failure
Defines when / what / if interventions are acceptable
Roles & responsibilities and more
Example Assay
Labware
Source Plate/Assay Plate

Stated Protocol
Dispense 10l BufferA
Dispense 10l BufferB
Incubate 60 min at Room Temperature
Wash
Read

Additional Assay Information
Unstated Protocol
Read Barcodes from Source and Assay Plate
Shake for 1min after 2
nd
dispense
Seal plate with clear heat seal before reading
Use Barcode for name of resulting reader file

Acceptance Test Plan (ATP) (II)
Any contract language regarding payment structure,
penalties etc. should be in the purchase agreement and
not in the ATP

The ATP can reference milestone payments

Between the Project Plan and/or Purchase Agreement,
all parties should have a clear understanding of financial
responsibilities

Request that the FAT and SAT be performed on a wet
system
Acceptance Test Plan (ATP) (III)
The ATP document should be finalized and agreed upon
well in advance of the actual FAT
allows the vendor to run through the FAT testing (pre-FAT)

Vendor is responsible for unit testing of Custom and
Standard devices prior to FAT

The Actual FAT should not occur until the vendor has
confirmed that they have gone through and tested the
FAT process (pre-FAT )

FAT: Workstations and Integrated Systems
FAT will test the purchased equipment using your
materials and protocols at the vendors site

FAT should use all of the equipment on the system
Should not be limited to 1 giant all encompassing test
should entail the primary task(s) for which the system was
purchased

FAT should not execute an experiment or protocol which
will not be used in actual lab

Workstation FAT
Not all vendors will agree to complete a FAT for a workstation
purchase
Insist upon executing a FAT for any system with customization
If the vendor is unwilling to include a successful FAT as criteria for
payment, offer to pay for application labor to execute the FAT

FAT should include:
module level testing
error recovery testing
testing of new interfaces and any data handling that the system will
generate
safety requirement testing
QC testing

Not all vendors will agree to QC testing, however most will provide
an exception in the event it does not work in the SAT or BAT
Integrated System FAT
All modules should be tested
If any of the modules will be run in standalone mode confirm that
they will operate in that mode
Module level and system level error recovery should be tested

QC and Safety testing are required

1 sample should be run from start to end so the entire
process can be monitored, before running multiple
samples

If appropriate, test throughput matches requirements

Integrated System FAT (II)
Run the most complex process using the maximum
expected run size (regardless of run duration)

Information System testing should be done as part of
FAT

Always check the error recovery option

Review System documentation
Confirm the most current layout is provide
Manuals should be provide for standard and custom equipment
Cable layout a bonus, in particular when labeled.
Navigating through FAT defects
System requirements will define when interventions are
acceptable. For example:
Day time run semi-automated system Interventions Acceptable
Fully automated system which runs overnight Interventions Not
Acceptable

When FAT runs require interventions
refer to the ATP
acceptable to proceed without starting over?

The # of successful continuous days of running should
be defined in advance and required for the system to
pass the FAT
Navigating through FAT defects (II)
Any exceptions / defects should be documented and
witnessed by both the vendor and customer

Only accept defects to be fixed later when you are
confident they will be resolved easily

It is typically riskier, more expensive and time consuming
to fix problems on the customers site compared to the
vendors site
Examples of Exception
A non-recoverable error

Communication Errors between hardware and
controlling software




SAT
SAT should replicate the FAT testing procedures

SAT confirms that the system was not impacted by transport and
installed successfully in the customer lab

Both customer and vendor representatives should be present for all
SAT testing

Repeat testing for any exceptions that occurred during the FAT
confirms defects have been corrected to the customers satisfaction

Document and witness any preexisting or new exceptions that occur
during the SAT

Only accept SAT when all critical system exceptions have been
rectified and re-tested

To BAT or not to BAT?
A Biological Acceptance Test or Chemical Acceptance Test is a final
testing phase which
Uses actual experiment materials
Confirms the actual experiment / process can be executed successfully
using the system

Vendors rarely agree to execute BAT tied to a payment milestone
Numerous variables outside of their control
External factors can impact experiment performance
Include as requirement within RFP

BAT should be applied when
Purchasing a large / complex integrated system
Experiment or process can be repeated manually with good precision


BAT
Set expectations for BAT before signing the purchase
agreement / contract

A successful BAT will show that the automation does not
negatively impact experiment / process results
Precision and accuracy of data should be equal to or better than
running the experiment manually

Apply financial recourse for un-successful BAT
Final milestone


Lessons Learned
Avoid scheduling major milestone at the end of a
quarter or end of the year.
Time constraints may influence the acceptance of the
FAT/SAT/BAT/CAT testing
Financial constraints from users companies

Negotiate service contract quote and associated
services (parts and labor) before contract signoff


Summary
Understand the process you wish to automate and the
instrumentation you decided to purchase to complete the
task

A FAT, SAT and BAT should be performed on any
integrated system. For workstation a SAT and BAT
should be performed

Work with your team to protect your investment. It may
not be your money but your name is associated with the
purchase

Acknowledgments
Ted Manley
William Goode
John Davies
Brian Duffy
Matt OBrien
Josh Martin
Agilent/Velocity 11

Backup Slides
Resources

Example Project Plan