Pilot UAT Evaluation Final Report

Results Summary
xCube version 0.5 holds much promise to improve our Loc UA handoff process. The
Pilot team is excited about the potential to exert greater control over our files and
schedule, and we feel confident that xCube will deliver a better solution for us. During
this development cycle, the Offglobe team has been responsive to our feedback in the
features area and have already resolved many outstanding issues identified earlier in the
cycle. However, some serious concerns remain regarding the current usability of the tool.
These concerns have been documented as responses to a post-pilot UAT questionnaire
addressing a number of critical areas.
Some areas were rated a Pass according to the established criteria; in these areas, the tool
passed with the minimum pass rating. In general, testers expressed frustration with the UI
design and the lack of intuitive process flow. Other areas were rated a failing score. The
xCube UAT version 2025.4 has therefore received an overall Fail rating from the UA
Pilot team in the usability area. This rating reflects the overall picture of workarounds
and shortcomings compared to pilot team expectations for the first release. The scoring is
based upon objective measurement criteria in a number of areas, described below. The
major issues were: 1) this version is missing critical features required to support pilot
tasks; 2) time to complete tasks as compared to the current process is unfavorable; and 3)
tool stability/performance under multi-user conditions is unfavorable. Evidence that the
areas of performance and stress were not evaluated by core test in the tool interaction
space, but only in the check-in task, is concerning in light of the error rate encountered in
the course of UAT.
The recommendation from the Pilot team is to delay entry into Production until the most
critical issues are addressed and the tool takes no more time to use for handoff than the
current system.

Pilot Team UAT Pass Overview

Description
The UA Content Pilot Team conducted an acceptance pass of the xCube tool, version 0.5,
between September 13-16, 2004. The test pass consisted of pilot team members following
a written series of process steps to accomplish essential pilot team tasks for their owned
content areas.
At the conclusion of the test pass, participants completed a questionnaire designed to
elicit quantitative usability feedback on the tool and their success in using it to perform
the pilot tasks.
Combined results are available on (link removed); the document title is Pilot UAT
Results.rtf.

Content Area Coverage

8 content areas were tested; questionnaire feedback was received for 5 out of 8 content
areas.
Task Coverage
The pilot team tasks evaluated were:

Check for new components in CS tree

Componentize new components as needed for HO to vendors
Confirm HO criteria for your areas
Evaluate new and updated assets (Create Test Packages)
Mark evaluated assets as Piloted
Make EOL recommendations
Create a test handoff package to verify piloted assets.

Test Results
1 to 5
The following areas were evaluated on a ranked 1 to 5 scale for each task, with a
minimum passing score of 3. Mean of scores across all tasks are reported below.

Tab UI 3.0
Online help 2.3
Server wait time 2.5
Pilot process 3.0
Time investment 2.8

Yes/No
The following areas were evaluated on a yes/no scale. The percentage summary is an
average of responses to each question over all tasks to provide an overall picture of tool
health; the second column indicates how many testers answered yes at least once to the
given question.

Feature set is missing critical features present in the old system* 42% 3 of 5
Feature does not work the way I expect* 36% 3 of 5
Memory error was encountered* 26% 3 of 5
Server error was encountered* 35% 4 of 5
Degraded performance in multiuser scenario 100% 4 of 4

* These are combined scores across a number of times the question was asked for
different tasks. Some tasks stood out in bringing the overall score down; creating a test
handoff is the most problematic task. See combined results report for details.
This question was asked in the affirmative. Inverse results are presented to align with
the other questions.
Action items
Pilot team to ensure all critical issues are filed as bugs and appropriately prioritized
by Friday, Sept. 24.
Offglobe to estimate timeline for fixing Pri 1 and Pri 2 issues identified.
Offglobe to produce new roadmap for entering Production phase. Timeline is
dependent upon time required to implement and regress must fixes from previous
bullet.
Offglobe to allocate extra support resources for the first handoff in Production.