Automated Low-Level Analysis and Description of

Diverse INtelligence (ALADDIN) Video

Intelligence Advanced Research Projects Activity (IARPA)

John Garofolo, ALADDIN Program Manager

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

The Challenge

Massive numbers of video clips are generated daily on many types of

consumer electronics.
Such clips can include any possible scene and events, and generally have
limited quality control.
The "unconstrained" video clips produced by anyone who has a digital
camera present a significant challenge for manual as well as automated
analysis.

Analysts who must work with such data are overwhelmed by its
volume and lack of tools to probe it.
No technology currently exists to provide an analyst with the information
necessary to fully understand the content or temporal processes within
these data.

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

The Goal
Develop advanced technology to empower analysts to
create sophisticated event-oriented queries against diverse
collections of open-source video clips
Video clip collections are processed once to create a general
representation of event-related content that is useful for many
potential queries.
Analysts create specific event search queries that are modeled after
observational reports of events.
Information in the content representation is quickly analyzed for
relevance to the queries and the most relevant clips are highlighted
and textually summarized in an analyst-friendly form.

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Event Queries

An event:
is a complex activity occurring at a specific place and time;
involves people interacting with other people and/or objects;
consists of a number of human actions, processes, and activities that are loosely or
tightly organized and that have significant temporal and semantic relationships to the
overarching activity;
is directly observable.

Is searched via an Event Query incorporating

a descriptive name of the event to be searched

textbook-style definition for the event,
explication of event domain-specific knowledge
a list of evidence that an analyst would use to identify the event
a small set of video clips containing examples of the event.

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Approach
Sorted
Sorted
Listing
Listing of
of
Clips
Clips with
with
Confidences
Confidences
and
and
Recountings
Recountings

Event
Event
Query
Query

Event
Event
Agent
Agent

Event
Event Agent
Agent
Generator
Generator

Open Source
Video Clip
Collection

Content
Content
Description
Description
Representation
Representation
Generator
Generator

CDR
CDR
Data
Data and
and
Knowledge
Knowledge
Sources
Sources
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Example ALADDIN Event Query

Courtesy of NIST MED 11 Evaluation
Event Name:
Working on a woodworking project

Exemplars:

Definition:
One or more people fashion an object out of wood.

Event Explication:
Woodworking is a popular hobby that involves crafting an object out of wood.
Typical woodworking projects may range from creating large pieces of furniture to
small decorative items or toys. The process for making objects out of wood can
include cutting wood into smaller pieces . (continues)

Evidential Description:
scene: Often indoors in a workshop, garage, artificial lighting. Occasionally
outdoors
objects/people: Woodworking tools (automatic or non-automatic saws, sander,
knife), paint, stains, sawhorses, toolbox, safety goggles
activities: Cutting and shaping wood, attaching pieces of wood together,
smoothing/sanding wood
sounds: sawing, chopping, planning, hammering,

Target User: An Internet information analyst or experienced Internet searcher with event-specialized knowledge.
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Event Search Agents

Generates
a confidence probability for each clip in the collection about its
relevance to the event query.
a hard decision about the relevance of each clip using the confidence
threshold and confidence probability (Multimedia Event Detection)
a detailed recounting (summary transcript) of the event-salient
information about the clips (Multimedia Event Recounting)
describing the spatial and temporal details of the occurrence
including the key observations regarding the scene, people, objects, sound,
speech, text, and activities pertaining to the event occurrence
that is useful for users to perform analyses of the detected evidence and
system decision
that is clear, concise, user-oriented, and is useful for downstream processing
that can be rendered at various levels of detail

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Example Recounting Fighting a Vehicle Fire

Fighting a vehicle fire (99%)
<fire> = fire (99%), explosion (98%), slow motion video of fire and explosion (85%)
<vehicle> = a big rig (90%), truck (97%), possible fuel truck (50%)
<fighting fire> = spraying water on fire (88%), fire truck (98%)
<geographic location> = LA Freeway (99%), Northbound on 101 near Sunset Blvd (1/4 mile),
Hollywood Blvd (3/4 mile), & Gower Street (1 mile) (92%)
<auxiliary> = helicopter (97%), newsworthy, important (98%), traffic stopped both ways (89%),
daytime (92%), small fire on hill nearby (90%), five people in line for picture (80%), parked
cars and people standing (89%)
One perspective:
1. Not a script listing most important items first
2. Hierarchical (only showing 2 levels)
3. Analyst can view several types of evidence,
e.g.,
corresponding audio/video clips
extracted components, including multiple instances
a timeline of events

4. Items grouped and summarized

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Program Phases and Metrics

Final
Goals
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Testing
Events

Training
Events

The TRECVID MED 11

Events

Process-Observed
Events

Life Events
Wedding ceremony

Attempting a board trick

Feeding an animal
Landing a fish
Working on a woodworking
project

Process-Observed
Events
Changing a vehicle tire
Getting a vehicle unstuck
Grooming an animal
Making a sandwich
Parkour
Repairing an appliance
Working on a sewing project

Life Events
Birthday party
Flash mob gathering
Parade

Program Manager Contact Info

John Garofolo
IARPA, Incisive Analysis Office
Office of the Director of National Intelligence
Intelligence Advanced Research Projects Activity
Washington, DC 20511
Fax: 301-851-7672
Phone: 301-851-7438
Electronic mail: john.s.garofolo@ugov.gov

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)