COSIN: Content-Based Retrieval System for Cover Songs ∗

Yi Yu J. Stephen Downie Fabian Moerchen

Department of Information and
Computer Sciences, Nara
Women’s University, Japan
yuyi@ics.nara-wu.ac.jp
Lei Chen
Department of Computer
Science,Hong Kong University
of Science and Technology
leichen@cs.ust.hk
Graduate School of Library
and Information Science
UIUC, USA
jdownie@uiuc.edu
Kazuki Joe
Department of Information and
Computer Sciences, Nara
Women’s University, Japan
joe@ics.nara-wu.ac.jp
Siemens Corporate Research,
755 College Road East
Princeton, USA
fabian.moerchen@siemens.com
Vincent Oria
Department of Computer
Science,New Jersey Institute
of Technology, USA
oria@njit.edu

ABSTRACT
We develop a content-based audio COver Song IdeNtifi-
cation (COSIN) system to detect/group cover songs.The
COSIN takes music audio content as input and performs
similarity searching to locate variants of the input (i.e., cover
versions). Identified cover songs are returned in the rank or-
der according to their similarity to the input.The COSIN
also incorporates a set of tools to evaluate retrieval perfor-
mance so researchers can explore different retrieval schemes
and parameters (e.g. recall, precision).The COSIN utilizes a
suite of techniques to detect cover songs including: Pitch +
Dynamic Programming (DP), Chroma + DP, and Seman-
tic Feature Summarization (SFS) + Hash-Based Approxi-
mate Matching (HBAM). Demonstration system shows that
COSIN is a very potential music content retrieval tool. Run-
ning some music retrieval schemes on COSIN platform, re-
cent experiments with SFS + LSH Variants demonstrate
a nicely balanced efficiency (search speed) v. performance
(search accuracy) tradeoff.
Categories and Subject Descriptors
H.3.3 [Information Systems]: Information Search and Re-
trieval; H.5.5 [Information Systems]: Sound and Music
Computing
General Terms
Algorithms, Performance, Experimentation
Keywords
Content-based audio retrieval, cover songs, musical audio
sequences summarization, hash-based indexing
∗This is a postgraduate project in NWU, second author was
supported by the Andrew W. Mellon Foundation, and sixth
author was partially supported by a grant from DoD-ARL
through the KIMCOE center of Excellence.
Copyright is held by the author/owner(s).
MM’08, October 26–31, 2008, Vancouver, British Columbia, Canada.
ACM 978-1-60558-303-7/08/10.
1. INTRODUCTION
Open Internet provides a powerful platform for on-line
amusement, where users upload their own music audio or live
performance and share them with the world. www.yyfc.com
and www.fancovers.com are such web sites that are very
attractive among youngsters.With large numbers of music
audio tracks emerging on such music social communities,
content-based cover song (http://en.wikipedia.org/wiki/Co
ver version) retrieval has become even more important. Sim-
ilar music audio tracks can be detected or grouped to im-
prove the searching experience.
Audio cover song detection as a new Music Information
Retrieval (MIR) task was first introduced and served as a
contest section in Music Information Retrieval Evaluation
eXchange (MIREX) 2006.We are in the process of designing
a content-based COver Song IdeNtification (COSIN) sys-
tem for visualizing and comparing performances of retrieval
schemes and evaluating the corresponding system parame-
ters.Currently some techniques related to cover songs de-
tection and retrieval have been implemented in our COSIN
system. These techniques include audio feature extraction
(Chroma[1], Pitch[2], Semantic Feature Summarization (SFS),
similarity searching methods (Dynamic Programming (DP)[1],
K-Nearest Neighbor (KNN), Hash-Based Approximate Match-
ing (HBAM, consisting of LSH[4] and LSH variants–SoftLSH
and Exact Locality Sensitive Mapping(ELSM)[6]) and sys-
tem measure metrics.Evaluation over a collection of real
cover songs indicates that the COSIN system is a suitable
tool for evaluating audio-content-based retrieval schemes.
COSIN demonstration video and some interesting exam-
ples of cover songs can be found at http://www.e.ics.nara-
wu.ac.jp/∼yuyi/.
2. RELATED TECHNIQUES
Now we review the typical techniques for content-based
cover songs detection implemented in COSIN system.The
task is to identify/group the cover version songs according
to their relevance. To guarantee the detection accuracy, the
popular detection methods use the DP [1][2][3] or its vari-
ants [5] to perform audio sequence comparison.However, it is
very time consuming since the feature (Pitch [2][3], Chroma
[1], MFCC [3]) sequence of musical audio is very high di-
mensional. To speed up the audio feature sequence compar-
ison, SFS is extracted from the audio sequences as a concise
 Similarity Searching
representation similar to [6]. With SFS as the fixed-length
summarization of audio documents, DP is no longer neces-
K-Nearest
sary. In addition, index-based approximate techniques (such Exhaustive
Semantic Neighbor
Matching
as [4]) can be applied to avoid exhaustive search. The main Feature (KNN)
Summarization Ranked list
idea of index-based approach is to extract some audio rep- Locality Sensitive Hash-Based
(SFS) Hash (LSH) or Approximate
resentation from the acoustic data, which can be indexed LSH variants
Audio Query Matching Results of
and retrieved efficiently. Experimental evaluation demon- System
strates that the approach of SFS + LSH variants, which Chroma Performance
Dynamic Audio
Pitch Sequence Measures
needs not perform the traditional exhaustive sequence com- Programming
Audio Feature (DP) Comparisons
parisons, exhibits a better tradeoff between efficiency and Extractor MFCC
effectiveness. Mel-Magnitude

3. COSIN SYSTEM
The objective of our COSIN system is to design a simi-
larity retrieval mechanism: taking a musical audio as input
and performing similarity searching and finally returning au-
dio tracks relevant to this audio query in a ranked list.The
performance of each cover song detection method can be
evaluated and the corresponding results can be listed based
on different evaluation metrics. COSIN consists of four main
components: audio feature extraction, similarity searching,
adjustable collection, and performance measurement. Fig-
ure 1. shows a flow diagram of audio cover songs detection
with COSIN.
Audio feature extraction. Each track is a 30-second-
long clip in single channel 16-bit wave format, 16-bit per
sample and re-sampled to the rate of 22.05kHz. The audio
data is normalized and then divided into overlapped frames.
Each frame contains 1024 samples (46.4ms) and the adjacent
frames have 50% overlap (the step length is 23.2ms). Each
frame is weighed by a hamming window and appended with
1024 zeros to fit the length of 2048 point FFT and then used
to calculate STFT. Chroma is calculated from the instanta-
neous frequencies of STFT. From the amplitude spectrum
Pitch, MFCC and Mel-magnitude are also computed. The
details about the feature extraction methods can be found in
[1][2][3]. Based on feature sets of Chroma, Pitch, MFCC and
Mel-magnitude, a combined SFS feature can be extracted.
Similarity searching. To facilitate the comparison among
the detection methods, three independent matching/searching
schemes are included in this part. DP is a typical method
to calculate the pair-wise sequence distance [2][3], which
can offer a high accuracy while taking long time for match-
ing. KNN is a non-sequence-comparison, exhaustive search-
ing method. Hash-based approximate techniques (LSH and
LSH variants)[4][6] maps audio features to integer values by
heuristics and reduces the exhaustive search by hashing.
Adjustable collection. Our cover song collection con-
sists of two non-overlapping datasets with 5275 music au-
dio tracks in total. Query collection (Covers79) includes 79
songs and in a total 1072 audio tracks (on average 13.5 cov-
ers per song). The database collection is composed of 4203
single-cover songs in addition to the 1072 queries. We may
adjust the collection size to test the cover songs detection. It
is also very easy to add/remove music audio tracks to/from
the collections.
Performance measurement. To evaluate and compare
detection methods under the fair environment, we use some
well-known evaluation metrics (such as recall and precision)
in this demonstration. It is convenient to judge the relevance
of music audio tracks in the ranked list.
System function summary. COSIN system presents
the following functions to assist researchers in assessing dif-
Adjustable
Collection
Figure 1: COSIN system architecture.
ferent retrieval schemes of the cover songs detection. 1)
Offer a GUI for content-based audio retrieval. 2) Support
flexible management of cover song collection. 3) Implement
both common schemes and some new methods. 4) Present
the retrieved results ranked by relevance. 5) Evaluate the
performances of retrieval schemes. 6) Display the retrieval
results of different detection methods.
4. FUTURE WORK
We are working for extending COSIN functionality to
broader applications. For example, many more potentially
useful audio features can be added as candidates for serving
the similarity searching to improve the retrieval effective-
ness. We can also add some machine learning methods to
train the semantic audio features. Moreover, index-based
approximate techniques can be added to further compare
with hash-based retrieval. We try to exploit the compatibil-
ity with other audio-based content retrieval system, such as
query-by-humming/singing (www.midomi.com) and query-
by-mood and so on.
5. REFERENCES
[1] D. Ellis and G. Poliner. Identifying cover songs with
chroma features and dynamic programming beat
tracking. IEEE ICASSP’07, Vol.4, pp.1429–1432, 2007.
[2] W. H. Tsai, H. M. Yu, and H. M. Wang, A
Query-by-Example Technique for Retrieving Cover
versions of Popular Songs with Similar Melodies,
ISMIR’05, pp.183–190, 2005.
[3] Y.Yu, J. S. Downie, and K. Joe, An Evaluation of
Feature Extraction for Query-by-Content Audio
Information Retrieval, IEEE ISMW’07, pp.297–302,
2007.
[4] P. Indyk and R. Motwani. Approximate Nearest
Neighbor: Towards Removing the Curse of
Dimensionality, ACM STOC’98, pp.604–613, 1998.
[5] Y. Yu, K. Joe, J. S. Downie, Efficient Query-by-Content
Audio Retrieval by Locality Sensitive Hashing and
Partial Sequence Comparison, IEICE Trans. Info. and
Sys., Vol.E91-D, No.6, pp1730–1739, 2008.
[6] Y. Yu, Scalable Content-Based Music Retrieval on
Acoustic Datasets via Hashing, PhD Forum, GHC’08,
2008.