Speech & Audio Processing - Part–II

Digital Audio Signal Processing

Marc Moonen
Dept. E.E./ESAT-STADIUS, KU Leuven
marc.moonen@esat.kuleuven.be
homes.esat.kuleuven.be/~moonen/

Speech & Audio Processing

•  Part-I (H. Van hamme)
speech recognition
speech coding (+audio coding)
speech synthesis (TTS)

•  Part-II (M. Moonen): Digital Audio Signal Processing

microphone array processing
noise cancellation
acoustic echo cancellation
acoustic feedback- cancellation
active noise control
3D audio

PS: selection of topics

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 2

1
 Digital Audio Signal Processing

•  Aims/scope
•  Case study: Hearing instruments
•  Overview
•  Prerequisites
•  Lectures/course material/literature
•  Exercise sessions/project
•  Exam

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 3

Aims/Scope

Aim is 2-fold :

•  Speech & audio per se

S & A industry in Belgium/Europe/…
•  Basic signal processing theory/principles :
Optimal filters
Adaptive filter algorithms (APA, Filtered-X LMS,..)
Kalman filters (linear/nonlinear)
etc...

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 4

2
 Case Study: Hearing Instruments 1/14
Hearing
•  Outer ear/middle ear/inner ear
•  Tonotopy of inner ear: spatial arrangement of where sounds of
different frequency are processed

© www.cm.be
Low-freq
tone

= Cochlea

Neural activity
High-freq for low-freq tone
tone
Neural activitity
Digital Audio Signal Processing: Introduction Version 2014-2015 for high-freq
Lecture-1: Introduction tone p. 5

Case Study: Hearing Instruments 2/14

Hearing loss types:
•  conductive
•  sensorineural
•  mixed
One in six adults (Europe)
…and still increasing
Typical causes:
•  aging
•  exposure to loud sounds
•  …
[Source: Lapperre]

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 6

3
 Case Study: Hearing Instruments 3/14
Hearing impairment : Dynamic range & audibility

Normal hearing Hearing impaired

subjects subjects
Level

100dB

0dB

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 7

Case Study: Hearing Instruments 4/14

Hearing impairment : Dynamic range & audibility

Dynamic range compression (DRC) (…rather than `amplification’)

Level
100dB
Output Level (dB)

100dB

0dB

0dB 100dB
0dB Input Level (dB)

Design: multiband DRC, attack time, release time, …

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 8

4
 Case Study: Hearing Instruments 5/14
Hearing impairment : Audibility vs speech intelligibility
•  Audibility does not imply
intelligibility SNR
•  Hearing impaired subjects 20dB

need 5..10dB larger

signal-to-noise ratio (SNR)
for speech understanding in
0dB
noisy environments 30 50 70 90
•  Need for noise reduction Hearing loss (dB, 3-freq-average)

(=speech enhancement) algorithms:

•  State-of-the-art: monaural 2-microphone adaptive noise reduction
•  Near future: binaural noise reduction (see below)
•  Not-so-near future: multi-node noise reduction (see below)
Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 9

Case Study: Hearing Instruments 6/14

!Hearing Aids (HAs)
1921

•  Audio input/audio output

(`microphone-processing-loudspeaker’)
•  ‘Amplifier’, but so much more than an amplifier!!
•  History:
•  Horns/trumpets/…
2007 (Oticon)

•  `Desktop’ HAs (1900)

•  Wearable HAs (1930)
•  Digital HAs (1980)
•  State-of-the-art:
•  MHz’s clock speed
•  Millions of arithmetic operations/sec, …
•  Multiple microphones

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 10

5
 Case Study: Hearing Instruments 7/14

Alessandro Volta 1745-1827

!Cochlear Implants (CIs)

•  Audio input/electrode stimulation output

•  Stimulation strategy + preprocessing similar to HAs
•  History: Intra-cochlear
•  Volta’s experiment… electrode
•  First implants (1960)
•  Commercial CIs (1970-1980)

© Cochlear Ltd
•  Digital CIs (1980)
•  State-of-the-art:
•  MHz’s clock speed, Mops/sec, …
•  Multiple microphones
!Other: Bone anchored HAs, middle ear implants, …
Electrical stimulation Electrical stimulation
Digital Audio Signal Processing: Introduction for low
Version 2014-2015 frequency for high frequency
Lecture-1: Introduction p. 11

Case Study: Hearing Instruments 8/14

•  External Processor
© Cochlear Ltd

Digital/analog-conversion (cfr infra)

Digital processing & filterbank
Etc..
•  Coil
Inductive/magnetic coupling
•  Implant
Electrode array

PS: number of CI-implantees worldwide approx. 200.000

PS: 1 CI is approx. 25kEURO, plus surgery, revalidation,..
PS: 3 companies (Cochlear LtD, Med-El, Advanced Bionics)

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction 12/32
p. 12

6
 Case Study: Hearing Instruments 9/14

HA technology requirements
•  Small form factor (cfr. user acceptance)
•  Low power: 1…5mW (cfr. battery lifetime ≈ 1 week)
•  Low processing delay: 10msec (cfr. synchronization with lip reading)

DSP challenges in hearing instruments

•  Dynamic range compression (cfr supra)
•  Dereverberation: undo filtering (`echo-ing’) by room acoustics
•  Feedback cancellation
•  Noise reduction

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 13

Case Study: Hearing Instruments 10/14

DSP Challenges: Feedback Cancellation
•  Problem statement: Loudspeaker signal is fed back into microphone,
then amplified and played back again
•  Closed loop system may become unstable (howling)
•  Similar to feedback problem in public address systems (for the
musicians amongst you)

Similar to echo cancellation

Model in GSM handsets, Skype,…
F but more difficult due to
signal correlation
-

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 14

7
 Case Study: Hearing Instruments 11/14

DSP Challenges: Noise reduction

Multimicrophone ‘beamforming’, typically with 2
microphones, e.g. ‘directional’ front microphone and
‘omnidirectional’ back microphone

“filter-and-sum”
the
microphone signals

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 15

Case Study: Hearing Instruments 12/14

Binaural hearing: Binaural auditory cues
•  ITD (interaural time difference)
•  ILD (interaural level difference)
signal
ILD

ITD

•  Binaural cues (ITD: f < 1500Hz, ILD: f > 2000Hz) used for
•  Sound localization
•  Noise reduction
=`Binaural unmasking’ (‘cocktail party’ effect)
0-5dB

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 16

8
 Case Study: Hearing Instruments 13/14

Binaural hearing aids

•  Two hearing aids (L&R) with wireless link & cooperation
•  Opportunities:
•  More signals (e.g. 2*2 microphones)
•  Better sensor spacing (17cm i.o. 1cm)
•  Constraints: power/bandwith/delay of wireless link
•  ..10kBit/s: coordinate program settings, parameters,…
•  ..300kBits/s: exchange 1 or more (compressed) audio signals
•  Challenges:
•  Improved localization through cue preservation
•  Improved noise reduction + benefit from binaural unmasking
•  Signal selection/filtering, audio coding, synchronisation, …

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 17

Case Study: Hearing Instruments 14/14

Future: Multi-node noise reduction – sensor networks

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 18

9
 Overview
General speech communication set-up :
- background ‘noise’ → noise suppression, source separation
- far-end echoes → acoustic echo cancellation
- reverberation → de-reverberation/deconvolution

Applications :
•  teleconferencing/teleclassing
•  hands-free telephony
•  hearing aids, etc..
Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 19

Overview : Lecture-2&3

Microphone Array Processing

Spatial filtering - Beamforming
Fixed (Lecture 2) vs. adaptive (Lecture3)
Example filter-and-sum beamformer :

S (ω )
Y1 (ω ,θ )
F1 (ω )
Y2 (ω ,θ )
F2 (ω )
Z (ω ,θ ) d m Y (ω ,θ )
Y1 (ω Y,θm)(ω ,θ )
1
Σ
Fm (ω ) d m cosθ

θ
YM (ω ,θ )
FM (ω )

Application: hearing aids

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 20

10
 Overview : Lecture-4

Noise Reduction

`microphone_signal[k] = speech[k] + noise[k]’

•  Single-microphone noise reduction

–  Spectral Subtraction Methods (spectral filtering)
–  Iterative methods based on speech modeling
(Wiener & Kalman Filters)

•  Multi-microphone noise reduction

–  Beamforming revisited
–  Optimal filtering approach : spectral+spatial filtering

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 21

Overview : Lecture-5

Acoustic Echo Cancellation

Adaptive filtering problem:

•  non-stationary/wideband/… speech signals
•  non-stationary/long/… acoustic channels

Adaptive filtering algorithms

AEC Control
AEC Post-processing
Stereo AEC

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 22

11
 Overview : Lecture-6

Acoustic Feedback Cancellation

•  Ex: Hearing aids
•  Ex: PA systems
•  correlation between filter input (`x ’) and near-end signal (‘ n ’)
•  fixes : noise injection, pitch shifting, notch filtering, ...

amplifier

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 23

Overview : Lecture-7

Active Noise Control

•  Solution based on `filtered-X LMS’

•  Application : active headsets/ear defenders

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 24

12
 Overview : Lecture-7bis & 8

3D Audio & Loudspeaker Arrays

•  Binaural synthesis
…with headphones
head related transfer functions (HRTF)
…with 2+ loudspeakers (`sweet spot’)
crosstalk cancellation

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 25

Aims/Scope (revisited)

Aim is 2-fold :
•  Speech & audio per se
•  Basic signal processing theory/principles :
Optimal filtering / Kalman filters (linear/nonlinear)
here : speech enhancement
other : automatic control, spectral estimation, ...
Advanced adaptive filter algorithms
here : acoustic echo cancellation
other : digital communications, ...
Filtered-X LMS
here : 3D audio
other : active noise/vibration control
Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 26

13
 Lectures

Lectures: 7*2hrs (Lectures 1-7) + 1*1hr (Lectures 8)

–  PS: Time budget = (15hrs)*4 = 60 hrs

Course Material: Slides

–  Use version 2013-2014 !
–  Download from DASP webpage
http://homes.esat.kuleuven.be/~dspuser/dasp/

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 27

Prerequisites

•  H197 Signals & Systems (JVDW)

•  HJ09 Digital Signal Processing (I) (PW)

signal transforms, sampling, multi-rate, DFT, …

•  HC63 DSP-CIS (MM)

filter design, filter banks, optimal & adaptive filters

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 28

14
 Literature
Literature (General) (available in DSP-CIS library)
•  Simon Haykin
`Adaptive Filter Theory’ (Prentice Hall 1996)
•  P.P. Vaidyanathan
`Multirate Systems and Filter Banks’ (Prentice Hall 1993)

Literature (specialized) (some available in DSP-CIS library)

•  S.L. Gay & J. Benesty
`Acoustic Signal Processing for Telecommunication’ (Kluwer 2000)
•  M. Kahrs & K. Brandenburg (Eds)
`Applications of Digital Signal Processing to Audio and Acoustics’
(Kluwer1998)
•  B. Gold & N. Morgan
`Speech and Audio Signal Processing’ (Wiley 2000)

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 29

Exercise Sessions/Project

Direction-of-arrival θ

Acoustic source localization

–  Direction-of-arrival estimation
–  Noise reduction
–  Echo cancellation
–  Simulated set-up
Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 30

15
 Acoustic Source Localization Project

•  Runs over 4 weeks (non-consecutive) PS: groups of 2

•  Each week
–  1 PC/Matlab session (supervised, 2.5hrs)
–  2 ‘Homework’ sesions (unsupervised, 2*2.5hrs)
PS: Time budget = 4*(2.5hrs+5hrs) = 30 hrs

•  ‘Deliverables’ after week 2 & 4

•  Grading: based on deliverables, evaluated during sessions

•  TAs: guiliano.bernardi@esat (English+Italian)

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 31

Acoustic Source Localization Project

Work Plan

–  Week 1: Design Matlab simulation set-up

–  Week 2: Direction-of-arrival (DoA) estimation
*deliverable*

–  Week 3: DoA estimation + noise reduction

–  Week 4: DoA estimation + echo cancellation
*deliverable*

..be there !
Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 32

16
 Exam

•  Oral exam, with preparation time

•  Open book
•  Grading

7 for question-1
7 for question-2

+6 for project
___
= 20

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 33

September Retake Exam

•  Oral exam, with preparation time

•  Open book
•  Grading

7 for question-1
7 for question-2

+6 for question-3 (related to project work)

___
= 20

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 34

17
 Website
1)  TOLEDO

2)  http://homes.esat.kuleuven.be/~dspuser/dasp/

•  Contact: guiliano.bernardi@esat
•  Slides (use `version 2014-2015’ !!)
•  Schedule
•  DSP-library
•  FAQs (send questions to marc.moonen@esat)

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 35

Questions?

1)  Ask teaching assistant (during exercises sessions)

2)  E-mail questions to

teaching assistant
or marc.moonen@esat

3) Make appointment
marc.moonen@esat
ESAT Room 01.69

Digital Audio Signal Processing: Introduction Version 2014-2015 Lecture-1: Introduction p. 36

18