You are on page 1of 52

Multisensor Data Fusion

Andrs Navarro

Contents

Introduction

Definition
Sensors and levels

Fusion models
Fusion techniques Distributed Data Fusion Scenarios Conclusions

Introduction

Motivation

Master Thesis First approach to sensor fusion Overview of state-of-the-art

Guide for people who do not know about sensor fusion to get introduced to the issue

Introduction

Why is sensor fusion needed?

Research questions

What is multisensor data fusion? Is there an unanimous definition?

What models for multisensor data fusion exist in the literature? Do they have common descriptions? Do they contradict each other?
What techniques or methods can be used in multisensor data fusion? Centralized or decentralized data fusion? What scenarios can multisensor data fusion be applied?

Definition

JDL Data Fusion Lexicom


A process dealing with the association, correlation, and combination of data and information from single and multiple sources to achieve refined position and identity estimates, and complete and timely assessments of situations and threats, and their significance. The process is characterized by continuous refinements of its estimates and assessments, and the evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

A. Steinberg and C. Bowman.


Data fusion is the process of combining data or information to estimate or predict entity states.

Wald
Data fusion is a formal framework in which are expressed means and tools for the alliance of data originating from different sources. It aims at obtaining information of greater quality; the exact definition of greater quality will depend upon the application.

Sensors and levels

Commensurate multisensor data

Non commensurate multisensor data Feature-level Fusion Information extraction

Direct Data Fusion

High-level Fusion

Fusion models

JDL data fusion model Dasarathy's functional model Waterfall fusion process model

Boyd Loop
Thomopoulos' Fusion Model

Durrant-Whyte architecture
The Omnibus process model Endsley's Situation Awareness General Data Fusion Architecture

Fusion models

JDL data fusion model

Sources
Sources preprocessing

Level 1
Level 2 Level 3 Level 4 Database management system HCI

Fusion models

JDL model revisions

Drawbacks

Different revisions
New definitions

Fusion models

Dasarathy's functional model

Levels of abstraction
Data Feature Decision

Categorization of data fusion functions in terms of the type of data level at input/output.

Fusion models

Waterfall fusion process model

Fusion process in stages Omission of feedback data flow is the major limitation.

Fusion models

Boyd Loop

OODA cycle:

Fusion models

Thomopoulos' Fusion Model

Fusion models

Durrant-Whyte achitecture

Fusion models

The Omnibus process model

Fusion models

Endsley's Situation Awareness

Fusion models

General Data Fusion Architecture

Network based

Levels described as classes with attributes and functions.

Fusion models

Models classification

Elmenreich

Abstract Generic Rigid

Durrant-Whyte and Henderson


Architecture: Meta, Algorithmic, Conceptual, Logical and Execution Centralized - Decentralized Local Global Modular Monolithic Heterarchical - Hierarchical

Techniques

Classification

Overview

Kalman Filter Probabilistic Inference

Artificial Neural Networks


Fuzzy Logic Support Algorithms

Selection

Techniques

Classified by

JDL levels

Techniques

Classified by

JDL levels Type of method Fusion problems

Techniques

Classified by

JDL levels
Type of method

Techniques

Classified by

JDL levels Type of method Fusion problems

Techniques

Classified by

JDL levels
Type of method Fusion problems
Data Association Estimation Identity declaration Decision-level identity fusion

Techniques

Kalman Filter

Discrete Kalman Filter

Extended Kalman Filter

Assumed noise

Techniques

Kalman Filter with INS

Inertia System:
Good high frequency information Drift at a slow rate

Other Position System


Good data at low frequency, on the average High frequency noise

The Kalman Filter approach is instead to use the statistical characteristics of the errors in both the external information and the inertial components to determine this optimal combination of information. Actually, the filter statistically minimizes the errors in the estimates of the navigation parameters: on an ensemble average basis, no other means of combining the data will outperform it, assuming the internal model in the filter is adequate.

Techniques

Kalman Filter with INS

Direct implementation

Filter fails System Fails High sample rate CPU load

Indirect feedforward
Erros in the inertial must remain of small magnitude

Indirect feedback

Techniques

Probabilistic Inference

Bayesian Inference
It can be used to discriminate between conflicting hypotheses Initial beliefs are needed before any evidence is ever collected Sensor fusion:

Techniques

Probabilistic Inference
Bayesian Networks

Probabilistic graphical model that represents a set of variables and their probabilistic interdependencies
Algorithms to perform inference and learning

Dynamic Bayesian Networks

Extension of Bayesian networks that allows the representation of temporal information Signals

Hidden Markov models

Model for Markov process: a stochastic process in which the probability distribution of the current state is conditionally independent of the path of past states

Techniques

Probabilistic Inference

Dempster-Shafer Theory

Generalization of Bayesian Theory: Instead of requiring probabilities for each question, belief functions are used.

Two ideas:

Obtain degrees of belief for one question from subjective probabilities fora related question. Use Dempster's rule for combining these degrees of belief:
Sensor1 m1(u0) Sensor2 m2(u0)

m(u0)=m1(u0)m2(u0)

Techniques

Probabilistic Inference

Dempster-Shafer Theory
Generalization of Bayesian Theory: Instead of requiring probabilities for each question, belief functions are used. Two ideas: Obtain degrees of belief for one question from

subjective probabilities fora related question. Use Dempster's rule for combining these degrees of belief:
Sensor1 m1(u0) Sensor2 m2(u2)

m(u0)=m1(u0)m2(u2)

Techniques

Probabilistic Inference

Dempster-Shafer Theory
Generalization of Bayesian Theory: Instead of requiring probabilities for each question, belief functions are used. Two ideas: Obtain degrees of belief for one question from

subjective probabilities fora related question. Use Dempster's rule for combining these degrees of belief:
Sensor1 m1(u0) Sensor2 m2(u1)

Dempster's rule

Techniques

Probabilistic Inference

Generalized evidence processing theory

Unifies the Bayesian theory with the D-S, combining their advantages and avoiding their disadvantages Each sensor collects evidence and assigns the evidence via probability masses; unlike D-S, GEP assigns and combines probability masses based on the a priori conditional probability of the hypotheses.

Techniques

Artificial Neural Networks

Computational model of biological neural networks:

Densely interconnected set of artificial neurons: simple units as perceptron

Feed-forward / recurrent Non linear statistical data modelling:

They can learn a complex relationship between inputs and outputs, normally established by the unit weights

Learning:
Units' weights Backpropagation algorithm Network structure

Techniques

Artificial Neural Networks

Neurons can be trained to represent sensory information and, through associative recall, complex combinations of the neurons can be activated in response to different sensory stimuli.

The main advantage of neural networks for multisensor fusion is that there is no need of a model for the sensors or the uncertainties

Techniques

Fuzzy Logic

Fuzzy set theory:


Elements have degrees of membership to the different sets, differing from classical set theory, where elements belong or do not belong to a certain set. Rules: IF antecedent THEN consequence Operators: OR, AND and NOT Steps: Fuzzyfcation

Rule evaluation Aggregation Defuzzyfication

Techniques

Fuzzy Logic

Uncertainty in multisensor fusion can be directly represented in the inference process by allowing each proposition to be assigned a degree of truth.

Fuzzy Fusion Network:


Input

data Feature Extraction Feature Level Fusion Decision Level Fusion

Techniques

Support Algorithms

Required functions for the fusion system


Library of basic numerical methods Database management Man-Machine Interaction Sensor Management

Techniques

Techniques Selection

Goals:
Maximum effectiveness: Algorithms should make inferences with maximum specificity in the presence of uncertain and missing data, dealing with minimal or no available a priori information. Operational constraints and time constraints must be considered. Resource efficiency in CPU and communications load. Operational flexibility to account for operational needs with changing a priori data. Functional growth.

Techniques

Techniques Selection

Steps:
1.Identifying categories of data-processing techniques or algorithms. 2.Surveying existing prototype and fielded data fusion systems. 3.Analyzing system requirements. 4.Analyzing and defining operational concepts for manual and automatic processes. 5.Identifying preliminary algorithms 6.Performing trade-off analyses of algorithm effectiveness versus required system resources. 7.Preparing detailed designs and prototypes of selected algorithms. 8.Refining and tuning the algorithms.

Distributed Data Fusion

A collection of processing nodes, connected by communication links, in which none of the nodes has knowledge about the overall network topology.

Requirements

Dynamic Topology Management Information- Sharing Strategies Algorithms

Scenarios

Image Data Fusion

Advantages:

Reduction of overall uncertainty and increase of accuracy. New features in a scene can be perceived More timely information is available

Scenarios

Robot Navigation

Scenarios

Ambient Intelligence

Electronic environments that are sensitive and responsive to the presence of people. Characteristics:

Embedded

Context-aware
Personalized Adaptive Anticipatory

Scenarios

Augmented Reality

Integration of virtual content in a real environment in real time. Alignment between virtual content and real world
Accurate position

Fusion of tracking systems:


INS
GPS Ultrasound Vision-based

Scenarios

Augmented Reality

Interaction with other perceptual systems:

Orienting, Auditory, Haptic, Taste, Smell

Feature extraction Human Behaviour

Sensors: Position, orientation, body gestures, speech, vital signs, eyetracking...

Human behaviour and experience models and simulations

Conclusions

What is multisensor data fusion? Is there an unanimous definition?

Most of definitions are restrictive to a certain terminology and applications A broader definition is needed to cover such a wide diversity of sensor fusion applications Wald's definition is chosen Discussion will continue

Conclusions

What models for multisensor data fusion exist in the literature? Do they have common descriptions? Do they contradict each other?

A common point in most of them is the need of divide the fusion process in levels of data abstraction.

More disagreement is found in the idea of a cycling processing.


Relationship between specification and usability Combine the underlying ideas for the final design.

Conclusions

What techniques or methods can be used in multisensor data fusion?

Data fusion at different levels of abstraction implies the use of multiple techniques. A layout or scheme for the implementation of any kind of sensor fusion application is not feasible. The design of the fusion algorithms is a lengthy task where multiple fusion techniques can be combined.

Conclusions

Centralized or decentralized data fusion?

The decentralized fashion has some advantages and some disadvantages comparing to the centralized one. The implementation of Distributed Data Fusion requires:
The use of certain fusion models that allow decentralization. Specific algorithms

Conclusions

Final conclusion
Multisensor Data Fusion is a broad issue due to the wide range of scenarios that it can be applied to. Therefore, to find a definition, a model or an algorithm scheme that is explicit, meaning that it can be followed to implement a real system, and at the same time usable for any kind of application, is an unfeasible task. Hence, a view of the different approaches, theories and implementations in the issue of sensor fusion can be presented, intending to be useful as a collection of different ideas that should be combined in the implementation of a real fusion system.

Multisensor Data Fusion

Andrs Navarro