phonological phenomena * multilingual text extraction * topic tracking * machine translation

digital libraries * text mining * tutorial dialogue systems * mulltimodal interfaces * speech synthesis * dialog systems * minority languages
computer-assisted language learning * question answering * information theory * technology supported education * language modeling

Language Technologies Institute

Carnegie Mellon

information retrieval * computational linguistics * machine learning * biosequence modeling

 ランゲージ テクノロジー Sprachtechnologie

Tecnologías del lenguage

Technologies de la langue

“Thus it may be true that the way to translate from Chinese to Arabic, or
from Russian to Portuguese, is not to attempt the direct route... Perhaps the
way is to descend, from each language, down to the common base of human
communication--the real but as yet undiscovered universal language--and
then re-emerge by whatever particular route is convenient.”
- Warren Weaver
2
 Contents
Overview 4

Ongoing Research 6

Academic Programs 18

LTI Courses and 20

Admissions

Faculty 22

Contents
3

Language Technologies Institute

© 2004 Carnegie Mellon University

Edited by C. Adèle Weitz
 Overview

Language Technologies Institute

The Language Technologies Institute (LTI) in the School of Computer Science at Carnegie Mellon University conducts
research and provides graduate education in all aspects of language technologies, including computational linguistics,
machine translation, speech recognition and synthesis, statistical language modeling, information retrieval and web search
engines, text mining, information management, digital libraries, intelligent tutoring, and more recently bio-sequence/bio-
language, structure and function analysis (genome, proteome). The LTI combines linguistic approaches with machine
learning and corpus-based methods, depending on the scientific questions investigated and project needs.

The LTI was established in 1996, combining the Center for Machine Translation (CMT), which was founded in 1986, and
other areas of computational language research. The LTI contains a unique mix of theoretical and systems-building re-
searchers specializing in various aspects of computer science, artificial intelligence, computational linguistics and machine
learning, and provides a rich and diverse environment for collaboration among faculty, graduate students, visiting scholars,
and research staff. As part of the School of Computer Science at CMU, LTI faculty and students collaborate closely with
members of the Computer Science Department, the Center for Automated Learning and Discovery, the Robotics Institute,
The Institute for Software Research International, and the Human-Computer Interaction Institute. Collaborative research
areas include mobile computing, computational biology, multi-agent systems, cognitive modeling, intelligent tutoring sys-
tems, multi-media interfaces, text and data mining, artificial intelligence systems, and machine learning theory and algo-
rithms.

The LTI offers both a Masters and PhD in Language Technologies.

The curricula of the graduate programs are based on a set of core courses
that include linguistic and statistical methods for language analysis,
fundamental computer science, and in-depth coverage of focus areas
in language technology such as machine translation, information re-
trieval, and speech recognition. Students benefit from a modular set of
laboratory courses, in which they learn the basics of natural language
technology through intensive hands-on practice. In addition, students
have the opportunity to expand their education with courses from the
other institutes in the School of Computer Science (listed above), in-
cluding courses in algorithms, artificial intelligence, computer systems,
machine learning, statistics, and computational biology.

In addition to fundamental and theoretical research, the LTI very much

focuses on large-scale challenges of consequence to industry, govern-
ment or society in the large, often spawning start-up companies, or
Overview

international-scale projects. The original LYCOS search engine, for

instance, was created at CMU, as was the original VIVISIMO meta-
search engine. The C-STAR international speech-translation consor-
tium, and the Universal Library project were also initiated at CMU –
the goal of the latter being the dissemination of the collected works of
humankind worldwide with free universal access.

The LTI offers and encourages collaboration with industry, national or

international, ranging from industrial affiliates and in-residence visit-
ing researchers, to targeted industrial education programs and longer-
term joint R&D projects. Such projects have produced practical re-
sults including high-accuracy machine translations for Caterpillar Inc,
and the Condor search engine used in Korea.
4
Carnegie Mellon University and Pittsburgh
Located in Pittsburgh, Pennsylvania, Carnegie Mellon Univer-
sity is well known for its interdisciplinary research centers and
institutes, such as the Pittsburgh Supercomputing Center, the Soft-
ware Engineering Institute, the Data Storage Systems Center, The
Information Networking Institute, the Engineering Design Re-
search Center, the Institute for Complex Engineered Systems,
the Robotics Institute, the Human-Computer Interaction Insti-
tute, the Language Technologies Institute, The Systems and Se-
curity Center and the Entertainment Technology Center. In these
organizations, researchers and faculty from diverse disciplines
collaborate on problems, benefiting from varied viewpoints and
scientific approaches.

Once the greatest steel-producing city in the United States, to-

day Pittsburgh is a medium-size modern city. With growing high-
technology industries such as computer science and biotechnol-
ogy, Pittsburgh’s energy reflects the vision of the future more
than the shadow of the past. Many high-technology industries
are locating research labs in Pittsburgh, including Intel, Seagate,
Rand, Hyundai, and others. Some might be surprised to learn
that Pittsburgh is economically diverse, visually exciting, archi-
tecturally active, ethnically rich and educationally innovative.

Carnegie Mellon is well integrated into its Pittsburgh surround-

ings. Ten minutes east of the downtown business district, the
103-acre campus is situated in Oakland, the educational and
medical mecca of the city. In addition to Carnegie Mellon, there
are four other institutions of higher education in this section of
the city, which provide a wide range of educational opportuni-
ties. Adjacent to campus is the 500-acre Schenley Park, com-
plete with public golf course, tennis courts, outdoor pool, ice-skating rink, and numerous jogging, mountain biking and
cross-country skiing trails.

Overview
5

Language Technologies Institute

 Ongoing Research
The modeling of human language lies at
Research Methodology
the confluence of linguistics, artificial in-
telligence, statistics, machine learning,
and cognitive science, and has been the
focus of intense research in the past few
decades.
Research efforts at the LTI pursue a vari-
ety of approaches, from linguistic and
knowledge-based methods, to supervised
and unsupervised statistical, corpus-
driven machine learning techniques.
In all of these approaches, domain
knowledge and theory (linguistic or sta-
tistical) inspire the general technical ap-
proach to solving a language technology
problem, and suggest the overall structure
of the model. Data is then used to esti-
mate model parameters, and to further
refine the model, suggest salient features,
optimize parameters, and ultimately, as-
sess the quality and viability of the ap-
proach. The general research paradigm
is much like that used in other areas of
science and engineering: a model is for-
mulated and used to make predictions, and
those predictions are then evaluated in
an effort to improve the model in an it-
erative process.
Ongoing Research
Data Knowledge
Model
Predictions
Fig 1:Research methology integrating
6
data, knowledge and model.
Research Methodology
A Statistical Model that can be refined
with expert knowledge
As one example, sequential models such
as hidden Markov models are of great
use in language technologies such as
speech recognition and information ex-
traction from text and web documents.
Although often effective, hidden Markov
models make strong independence as-
sumptions that can limit their predictive
performance. Research by LTI faculty
has led to alternative frameworks, such
as Conditional Random Fields, that make
fewer independence assumptions and
allow more expert knowledge to be in-
corporated as features into the model.
These models have been applied to in-
formation extraction, parsing, and even
image analysis, and have more recently
been adopted to biological sequence
analysis problems such as gene finding
and protein super-secondary structure
prediction.
Amino Protein
Acid Structural
Sequence Knowledge
Learned
Structural
Mapping
Model
Structural
Predictions
Fig 2: Biosequence to 3D structure pre-
dictions in Biolinguistics
A Linguistic Model that is refined with
Machine Learning
Another example is the framework devel-
oped within the Avenue project for infer-
ring transfer rules for Machine Transla-
tion from bilingual data. The learned rules
are mappings between syntactic structures
between the two languages, and are used
in a runtime Machine Translation system
to produce translations. Researchers at the
LTI have created a theoretical framework
that defines the kinds of models, i.e. gram-
mar rules, that can be learned from data.
A collection of specific instances of such
rules, constituting a “transfer grammar”
for a particular pair of languages, is then
inferred from a balanced word-aligned
small corpus of human translated sen-
tences. The framework addresses ques-
tions about how rules can combine to
cover more complex structures, what
types of linguistic features can be ex-
pressed in the rules and passed from one
rule to another, and so forth. The result-
ing “model” (i.e set of transfer rules) for
a particular language pair can be refined
in an active learning feedback loop where
users correct the output of the Machine
Translation system, and the system auto-
matically uses the corrections augmented
with active learning methods to refine the
underlying rule set.
Bilingual Theoretical
Data Learning
Framework
Learned
Translation
Rules
Machine
Translation
Output
Fig 3: Data, knowledge, model and pre-
dictions in AVENUE’s translation rule
learning system.

Translation of human language was one of
the very first tasks attempted by the devel-
opers of first digital computers in the
1950s. Over fifty years later, fully auto-
matic Machine Translation (MT) remains
one of the most difficult and challenging
topics of research within Artificial Intelli-
gence. With the emergence of universal
access to information enabled by today’s
internet, language has become a critical
barrier to global information access and
communication, and the need for MT is
greater than ever before. The LTI origi-
nated as the Center for Machine Transla-
tion in the mid-1980s, and MT continues
to be a prominent sub-discipline of research
with the LTI. The LTI is unique in the
breadth of MT problems and approaches
that are being investigated and pursued in
the context of a variety of research projects,
and in the number of faculty and research-
ers involved in MT research.
Part of the excitement of MT as a research
field lies in its wide range of challenges,
from cutting-edge applications that are
commercially feasible today, through tech-
niques that could have practical applica-
tion within a few years, to problems that
will not be fully solved until the advent of
true Artificial Intelligence. The ultimate
goal of this area can be characterized as
machine translation that is: (1) general pur-
pose (any topic or domain); (2) high qual-
ity (human quality or better); and (3) fully
automatic. Remarkably, our current MT
capabilities can reasonably satisfy any two
of these three criteria, but we cannot yet
meet all three at once. Our KANT project
produces fully automatic, high quality
translations for information dissemination
in well-defined technical domains such as
electric power utility management and
heavy equipment technical documentation
(as in the Catalyst application for Cater-
pillar). Our Example-based MT and Sta-
tistical MT systems can produce fully au-
tomatic translation in broad or unlimited
domains, but have not yet approached or
surpassed human quality levels. In other
Machine Translation
projects such as BABYLON and Diplomat/
Tongues, successful multi-lingual commu-
nication is achieved by augmenting this
limited-quality MT with human interaction
in order to help resolve translation errors.
There is currently active research being
conducted within the LTI on all of the ma-
jor approaches to MT. Each of these ap-
proaches has some unique strengths but
also inherent weaknesses and limitations.
Consequently, different approaches are
suitable for different scenarios. Our main
research thrusts are in machine learning
approaches to MT, including corpus-based
approaches such as Generalized Example-
Based MT and Statistical MT systems that
have focused primarily on Chinese-to-En-
glish and Arabic-to-English MT. Addition-
ally, we conduct ongoing work on Multi-
Engine MT (MEMT), combining the re-
sults of different MT techniques in order
to exploit each technique’s strong points.
Since traditional rule-based approaches to
MT require lengthy development cycles,
and corpus driven MT requires large
amounts of pre-translated parallel text for
training, the LTI is investigating alterna-
tive MT paradigms for minority languages,
such as Quechua and Mapudungún. The
goal of the AVENUE project is to produce
MT systems requiring neither extended
human development cycles (too costly) or
huge parallel corpora (not available). We
Types of Machine Translation
Interlingua
Semantic
Analysis
Transfer Rules
Syntactic
Parsing
Source
(Arabic) Direct: SMT, EBMT
Language Technologies Institute
are investigating methods such as unsu-
Machine Translation
pervised learning of complex morphol-
ogy, and transfer-rule induction from lim-
ited numbers of selected word-aligned
phrases and sentences, via machine learn-
ing methods such as seeded version
spaces. Although we have had initial suc-
cess, much of the challenge remains be-
fore us.
Another aspect in which MT systems dif-
fer is in their input/output modes: text ver-
sus speech. The JANUS project, for in-
stance, combines speech recognition with
language translation in the large. Other
projects, such as the Speechalator, pro-
duced limited-scope, speech-to-speech
MT on a hand-held device that also in-
cludes fluent speech synthesis. The area
of speech-to-speech MT is still young and
growing, with technical difficulties, such
as how to translate from a lattice of sen-
tence recognition hypotheses, produced
by the speech recognizer, rather than sim-
ply a single known sentence as input.
Human factors issues include clarifica-
tion dialogs when the recognition or the
translation is problematic, and how to
train users implicitly to use the system
for maximum effectiveness.
Ongoing Research
Finally, we have some non-traditional
projects, such as investigating Dolphin
language (we kid thee not), and whether
it can be interpreted.
Sentence
Planning
Text
Generation
Target
(English)
7
 Ongoing Research
In collaboration with the Human-Com-
puter Interaction Institute, the LTI fac-
ulty carries out research in intelligent tu-
toring, focusing on explorations of the
role of language in learning and learning
research, and specifically how language
Tutoring Systems
Tutoring Systems
Tutorial dialogue is a unique, intensely
dynamic form of instruction that allows for
a high degree of expressiveness and sensi-
tivity both in terms of the tutor’s adapta-
tion of material to the individual needs of
students as well as in the opportunity it cre-
In order for tutorial dialogue systems to
become widespread and have a real impact
on education, it is imperative that they
become easier to build. Another active area
of research at the LTI is building tools, such
as Carmel-Tools for authoring domain

technology can be used to support that
endeavor. A major thrust of this research
is to explore issues related to eliciting and
responding to productive student expla-
nation behavior. It involves many
broader issues such as influencing student
expectations, motivation, and learning
orientation. This interdisciplinary re-
search agenda involves five primary foci:
*Controlled experimentation and analy-
sis of student interactions with each other
as well as with human tutors and com-
puter tutors in order to explore the stimuli
that encourage productive student behav-
ior, appropriate learning orientation, and
ultimately effective learning
*Analysis of think aloud protocols in
learning scenarios in order to better un-
derstand the process of learning
* Basic research in language technology
to support the semi-automatic analysis of
language interactions in learning sce-
narios (text classification, automatic es-
say grading, etc.)
Ongoing Research
ates for students to make their thinking
transparent to the tutor. State-of-the-art
tutorial dialogue systems have focused on
leading students through directed lines of
reasoning to support conceptual under-
standing, clarifying procedures, or coach-
ing the generation of explanations for jus-
tifying solutions, problem solving steps,
predictions about complex systems, or un-
derstanding of computer architecture.
Evaluations of state-of-the-art tutorial dia-
logue systems provide a powerful proof-
of-concept, demonstrating conclusively
that the language technology exists for sup-
porting productive learning interactions in
natural language. Carnegie Mellon re-
searchers are at the forefront of this move-
ment, both in terms of producing landmark
systems and widely used
resources.
A current thrust of this
work at the LTI involves
pushing this technology
into new areas, such as
supporting design
activities in an exploratory
specific knowledge sources for robust
processing of student explanations, to
facilitate this process. Thus, beyond
producing technology to be used to address
our own theoretical questions about
learning interactions, we aim to produce
reusable technology that can facilitate the
work of other researchers pursuing their
own related questions. Our ultimate goal
is to produce resources that are simple
enough to be used by non-AI researchers
and practitioners, such as education
researchers, domain experts, and
instructors, and thus to put the power of
tutorial dialogue in the hands of those with
the pedagogical expertise to maximize its
effectiveness and meet the real needs of
students.

* Basic research in dialogue technology learning environment. An

to enable interaction in natural language important part of this work
in tutorial environments between humans involves the development
and computer tutors or to support inter- of the DReSDeN tutorial
actions in natural language between hu- dialogue planner to
man learners in collaborative settings (ro- manage a range of mixed-
bust language understanding, dialogue initiative tutorial dialogue
management, etc.) strategies including
negotiation dialogue for
* Development of easy-to-use tools for encouraging students to
building scalable language interaction develop the skills to ask
interfaces and tutorial environments more themselves important
generally (semantic knowledge source questions leading to a
authoring tools, etc.) thoughtful, reflective
decision making process.
Another current thrust of
this work involves the
optimization of time
management in tutorial
dialogue interactions.
8
 CALL - Computer Aided Language Learning
Learning a new language is a process of trial
and error where the student observes lan-
guage, tries to imitate it, and finds out how
good the imitation was. Until recently, the
best ways to learn a new language were ei-
ther by going to the country or by having a
personal tutor. First attempts to create lan-
guage learning software lacked feedback and
authenticity. This is where language tech-
nologies are starting to provide a potent al-
ternative. The use of speech recognition has
enabled students to speak to a system and
find out exactly what phonetic and prosodic
errors were made and how to correct them.
Using modeling of native and non-native
speech and knowledge of the native lan-
guage of the student, we have developed
The central goal of the Universal Library is
to digitize, index and make universally avail-
able all published works of humankind, in-
cluding books, periodicals, artwork and
music. A further goal is to provide value-
added information services such as auto-
mated summarization, reading assistants, full
content search and translation over the
internet in any language. Imagine the situa-
tion in which every researcher, every teacher,
every citizen and even every schoolchild
would have everything ever written at her
fingertips, regardless of what country she
lives in, her economic status, the school she
attended, or her native language. The am-
plification of human potential would be vast,
and would lead to the ultimate democrati-
zation of information and knowledge. Like
Rome, the Universal Library is not built in a
day, but rather it is a project for the ages,
requiring constant improvement and enrich-
ment.
We estimate that approximately 100 million
different books have been published in the
history of the world, but only a tiny fraction
are available digitally. The remainder must
first be scanned – a labor-intensive chore.
Toward that end, with the support of the
National Science Foundation, we have or-
ganized the Million Book Project, a joint
effort of CMU and the governments of In-
dia and China, with other countries such as
Egypt and Turkey joining the cause. Hun-
dreds of digital scanners have been distrib-
powerful pinpointing techniques to dis-
cover where errors lie in elicited speech.
The use of natural grammars allow us to
analyze student writing and suggest cor-
rections. Natural language grammars also
allow us to read along with a student and
give help in understanding a passage upon
request. By using information retrieval,
we can find appropriate texts for a
student’s level of reading and lexical and
grammar knowledge and can give other
researchers the tools to determine how
hard a new text can be and still be effec-
tive, for example, what percentage of new
words can be in a text which still allows
the student to generalize the meaning. For
learning research, members of the LTI
The Universal Library
uted to centers in those countries, where
personnel furnished by their governments
scan books for two shifts per day. As of
fall 2004 about 100,000 volumes have
been scanned. These are passed through
optical character recognition software,
indexed, and added to the Universal Li-
brary. Approximately half of the scanned
books are in English; the remainder are
in a wide range of other languages. The
first million books are expected to be
complete by the end of 2006, at which
point we will embark on the Ten Million
Book Project.
The Language Technologies Institute and
ISRI provide indexing software and stor-
age infrastructure for the Million Book
Project. The CMU Libraries furnish
metadata, archiving and copyright clear-
ance support. A number of research
projects are underway to explore appli-
cations and uses for the Universal Library.
One of them, the Universal Dictionary, is
an effort to build a database of every word
in every language. This will serve as a
basic resource for machine translation and
multilingual searching. We are also ex-
ploring new methods of navigating huge
text spaces. As the size of the collection
grows, the limitations of keyword search,
particularly for multilingual queries, be-
come severe. What is needed is a lan-
guage-independent search method that is
able to retrieve based on concepts rather
Language Technologies Institute
faculty have projectss with experts in
Intelligent Tutoring from the Human
Computer Interaction Institute as well as
with psychologists and language learn-
ing specialists at Carnegie Mellon and the
CALL
University of Pittsburgh.
Universal Library
than specific terms, which suggests a
multimodal rather than a pure text-based
interface.
Copyright is major barrier to free distri-
bution of content. The vast majority of
works ever published are still in copy-
right. Of these, more than 90% are out
of print, which means that they produce
no revenue either for the author or the
publisher. We are endeavoring to encour-
age publishers to allow the Universal Li-
brary to scan their out of print books and
permit them to be viewed on the Internet
and retrieved through search engines.
Publishers who do this often find an in-
Ongoing Research
creased demand for their books. We are
also working with the government of In-
dia to develop a new copyright statute that
would provide funds, analogous to the
UK Public Lending Right, to be distrib-
uted to copyright owners whose works
are accessed on the Internet, with micro-
payments to be provided through public
funds. Eventually the availability of such
payments could remove further obstacles
to offering copyrighted material.
The Universal Library enjoys coopera-
tive relationships with other institutions,
including the Internet Archive, the Digi-
tal Library Federation, and the National
Academy Press.
9
 Ongoing Research
Large amounts of genomic and protein
sequence data for homo sapiens and
other organisms have become available,
together with a growing body of corre-
lated protein structure and function data
creating an opportunity for addressing
the sequence mapping and structure
folding problems with increasingly so-
phisticated data-driven (statistical and
computational) methods to discover,
Computational Biology
characterize and model regularities and
outliers in the biological data. Machine
learning methods, with large amounts of
data, led to multiple breakthroughs in
language technologies such as automatic
speech recognition, document classifi-
cation, information extraction, statisti-
cal machine translation and other chal-
lenging natural language processing
tasks. Our research exploits the analogy
between mapping words to meaning via
syntax, in order to decipher the funda-
mental meaningful building blocks of
biological sequence language, via its
structure to its underlying function.
Ongoing Research
Figure: Crystal structure of pectate lyase
(pdb id 1EE6), a triple beta helix protein
(www.rcsb.org).
10
Computational Biology
The goal is to derive new hypotheses to
correlate these building blocks with struc-
tural, dynamic and functional “meaning”
for different living organisms in terms of
folding, activity, interactions, and path-
ways. For example, one of the important
challenges we try to tackle is the predic-
tion of super secondary structures such as
the beta helix (Figure). We work with su-
per secondary protein structure experimen-
talists so that the hypotheses generated
from this computational approach can be
tested by wet lab experiments.
In another attempt to relate protein primary
sequence to its structure and function, we
have engaged in a project to infer evolu-
tionary selectional pressure from residue
conservation in multiple sequence align-
ment of proteins families. Many measures
have been proposed for quantifying the
overall degree of sequence conservation in
a multiple sequence alignment of protein.
However, these measures fail to identify
which particular properties are conserved
at each position in the alignment. We de-
rived an algorithm for systematically iden-
tifying the conservation of specific physi-
cal-chemical properties in individual posi-
tions in a multiple sequence alignment. We
have applied our method to the diverse
GPCR family and demonstrate the compu-
tational significance of the properties we
have identified by successfully using them
to predict whether specific amino acids will
occur in particular positions in the align-
ment. We have also used our method to
annotate Rhodopsin, a well characterized
member of the GPCR family, with a
selectional pressure profile, which allowed
us to biologically interpret our findings.
We further applied our method to a mul-
tiple sequence alignment of an HIV-I pro-
tein, and are gearing to apply it to a large
set of protein families, including crystallins
and various globins. Looking ahead, we
plan to refine our method by incorporating
phylogenetic histories, and separating mu-
tation.
The above examples illustrate the power
of combining computational linguistics,
statistical machine learning and bio-se-
quence/structure/function discovery; we
expect to tackle other interesting problems
in this field, such as protein-protein inter-
action predictions and aspects of immune
system modeling at the molecular level.
 RADAR: Reflective Agent with Distributed Adaptive Reasoning
The RADAR project is an example of a
large-scale interdisciplinary research
project conducted at the LTI. RADAR is a
five-year research project that spans many
units in Carnegie Mellon’s School of Com-
puter Science, including many LTI re-
searchers. The overall goal is to develop a
software-based “cognitive personal assis-
tant” who could anticipate the needs of his
superiors. RADAR will help busy manag-
ers work more effectively, by coping
adaptively with various tasks ranging from
routine to complex problem solving. This
new technology should be equally valuable
to managers in industry, academia, and gov-
ernment. RADAR will help its human mas-
ter in many ways: scheduling meetings, al-
locating resources, maintaining a project
web site, producing coherent reports from
disorganized snippets of information, and
dealing with the constant flood of e-mail.
Additionally, all of the capabilities of RA-
DAR will be tested jointly in a crisis man-
agement task, such as re-organizing a con-
ference on very short notice after losing its
planned venue.
The key scientific challenge is to endow
RADAR with enough flexibility, learning
capabilities and general knowledge to
handle all these diverse tasks, including
requests and situations that were not an-
ticipated by RADAR’s designers. When
faced with a surprising new request, RA-
DAR might not know how to proceed, but
it should do something sensible. Perhaps
it can weave together fragments of old
Language Technologies Institute
plans to create a new one that fits the
current situation. Perhaps it can simply
ask for advice. But even turning free-
form advice into an executable plan of
action is a difficult research problem.
The technologies underlying RADAR
range from planning and problem solv-
ing (e.g. unifying hard and soft time and
space constraints), to natural language
processing (e.g. extracting useful infor-
mation from email streams), to dynami-
cally adaptable user interaction (e.g.
RADAR
when and how to ask for advice or offer
suggestions). All of these capabilities
depend on machine learning technology:
learning from human advice, learning by
observation, learning by active experi-
mentation, and transferring the results of
learning across problems and domains.
Ongoing Research
11
 Ongoing Research
Speech Processing
Speech is the most natural way for humans
to communicate and we find it so easy to
use, that most of us are surprised to learn
how complex the processing of spoken lan-
guage actually is. As one of our goals at
LTI is to make speech communication with
and through computers more useful, we
work on improving the fundamental tech-
Speech Processing
nologies of automatic speech processing,
i.e. speech recognition and speech synthe-
sis. Also, we develop new technologies
using those components, such as speech-
to-speech translation, spoken dialog sys-
tems, audio-based information extraction
and retrieval, and computer aided language
learning.
Speech Recognition
Automatic Speech Recognition is the pro-
cess of decoding a spoken speech signal
into a written form, that is, a sequence of
words. To do this, the analog speech sig-
nal needs to be digitized and then –for ef-
ficiency reasons-- reduced to its essential
relevant information, which is mainly done
by a form of frequency decomposition (the
picture below shows a spectrogram repre-
sentation of a speech waveform from con-
versationally spoken speech. The bright-
ness of the colors indicate the energy level
present at a given frequency. The final rep-
resentation of speech in the computer is a
stream of parameter vectors over time.
These vectors will be classified into pho-
nemes – the smallest linguistically distinct
sounds of a language. For this purpose pro-
Ongoing Research
totypes of these phonemes (so-called
acoustic models) will be trained before-
hand. With the help of the pronunciation
dictionary that relates each word to a con-
catenation of phonemes, the speech de-
coder can find possible word candidates
and in combination with the language
model the most likely sequence of words
is chosen to transcribe the spoken speech
signal.
Humans are able to understand equally well
the articulated read speech of a TV news
anchor speaker and our over-excited friend
12
calling from a loud party. What seems
so easy for us, is very difficult for ma-
chines: part of the difficulty lies in the
Speech Processing
fact that the speech signal can be heavily
affected by background noises, channel dis-
tortions, or cross-talk, but also that spoken
speech varies in speaking style, speed, and
content. More difficulties arise in speech
recognition because different words might
be pronounced the same (as in “two”, “to”,
and “too”), one word might be pronounced
differently (such as “the” in “the teen” vs
“the adult”) and also because speech is
spoken continuously, so it provides no natu-
ral segmentation. For instance, the same
phonetic sequence can be segmented into
two different word sequences: “This ma-
chine can recognize speech,” or “This ma-
chine can wreck a nice beach.” Which se-
quence will be picked depends on the ex-
pectation of the listener.
In order to learn the knowledge in the com-
ponents of the automatic speech recognizer,
namely the acoustic models, the pronun-
ciation dictionary, and the language model,
today’s speech recognition algorithms must
use data from which those models are
trained. Thus, the acoustic model learns the
most likely way people pronounce particu-
lar phonemes in particular contexts. The
pronunciation dictionary models the most
likely sequences of phonemes to build
words, and the language model learns the
most likely sequences of words to build
sentences. The language model is statisti-
cally trained and scores all of the possible
phrases that could have been spoken. It
differs from more traditional parsing tech-
niques, although they may overlap, since
speech is less likely to be in traditional lin-
guistic sentences.
Speech Synthesis
Speech synthesis is the process of generat-
ing natural sounding and appropriate
speech from text or other computer-read-
able formats. The task can be viewed in
three parts:
• Text Analysis: General text may contain
numbers, abbreviations, and other non-
standard words that require proper treat-
ment if they are to be pronounced intelli-
gibly. In English, the string of digits “1984”
has several pronunciations depending on
whether it is a year”nineteen eighty four,”
a quantity “one thousand nine hundred
(and) eighty four,” or a telephone number
which can be pronounced “one nine eight
four.”
• Linguistic analysis: Once given the
words, we still require the pronunciations.
This can be done by a pronunciation dic-
tionary. However, no matter how large the
dictionary is, we will still encounter words
outside of its vocabulary due to neologisms,
names, etc. Therefore, a letter-to-sound
rules system is also required. Prosody, in-
cluding tune, duration, and phrasing are the
components that make speech interesting.
There are many ways to pronounce words;
recreating the prosody and style (i.e. po-
lite or urgent) makes the speech more un-
derstandable and more acceptable to us-
ers.
•Waveform generation: Currently, the most
common form of constructing waveforms
from phonetic and prosodic descriptions is
by concatenating short pieces of pre-re-
corded natural speech and modifying their
prosody to match the desired form. Tradi-
tional approaches record all phone-phone
transitions in a language (called diphones).
Although this technique is robust, more
general unit selection synthesis, in which
the database contains more varied speech
with multiple examples of phones in vari-
ous contexts, and an appropriate selection
algorithm, seem to offer promise of much
higher quality speech.
Speech Processing Research Projects
Throughout the LTI’s speech processing
research there are always two directions
that influence the work: knowledge driven
and data driven. A substantial amount of
knowledge is required in order to build
such systems. Knowledge of acoustic pho-
netics, pronunciation, linguistics, signal
processing etc is needed to define the
framework within which we are working.
For example how to find the phoneme set
for languages that have not yet been stud-

ied, or how to find pronun-
ciation of words that are
not found in a dictionary,
how to include knowledge
of syntax and semantics to
aid speech processing.
Human language complex-
ity and variability is such that no hand-
written rules can cover all cases, thus our
knowledge based techniques are also
closely coupled with statistically based
methods. A common theme appearing
throughout all the LTI’s work is develop-
ing and applying machine learning tech-
niques to appropriately defined knowledge
driven frameworks to improve the useful-
ness of the work. These interdisciplinary
approaches encourage sharing of tech-
niques over different projects: language
modeling techniques may also be used in
text summarization; novel machine learn-
ing techniques may be applied to speech
problems. The LTI’s speech research al-
lows standard components to be used in
other larger projects thus making them
more useful, but also offering greater chal-
lenges in the application of speech and
language that lead to more fundamental
research. In the following we list a selec-
tion of projects and applications which
are currently under development in LTI:
• Speech-to-Speech Translation: The
Consortium for Speech Translation
Research (C-STAR) is a speech-to-
speech translation system developed
jointly between CMU and a interna-
tional partners from Japan, Korea,
Italy, France, and China. Here,
speech recognition must deal with
many languages, recognize and trans-
late in real-time, and handle many dif-
ferent users. Recent work investi-
gates deploying such systems on re-
source constrained mobile devices,
and improve the robustness and qual-
ity of domain dependent translation.
In the project STR-Dust (“Speech
Translation for Domain Unlimited
Spontaneous Communication Tasks”)
we push the limits of today’s speech
translation coverage to rather unlim-
ited domains, such as in meetings, lec-
tures, and news.
• Multilingual Speech Recognition:
Since speech recognition is the most
natural means to allow communication
across language and culture barriers,
speech recognizers in many different
languages are an essential prerequisite
for making speech-driven communi-
cation applications attractive and
available to the public. The project
GlobalPhone focuses on the rapid de-
ployment of speech recognizers in
many languages, i.e. by reducing the
required effort in terms of time and
cost to build such recognizers, thus en-
abling support for languages for which
few or no resources are available.
• Meeting Summarization: A micro-
phone records a multi-person meeting.
Off-line speech recognition technol-
ogy transcribes the meeting, including
the difficult task of separating the
voices and identifying the speaker.
Information retrieval technology is
then used to index the data so we can
answers queries such as “Find the part
where Bob and Jane talked about next
years’ budget.”
• Dialog systems: The CMU DARPA
Communicator project allows experi-
ments in mixed initiative dialog be-
tween humans and machines via the
telephone in the domain of flight, ho-
tel and car rental information. This
requires accurate, real-time recogni-
tion across the reduced bandwidth and
potentially noisy telephone, real-time
access to networked information, natu-
ral language generation, parsing, syn-
thesis, and a dialog manager.
• Computer aided language learning:
The Fluency project applies speech
recognition techniques to aid non-na-
tives in pronunciation.
Language Technologies Institute
•Language Modeling:
Classical language
modeling techniques
collect statistics on
word tri-grams (three
word sequences), but
because there are
many words in the lan-
guage and even more tri-grams, col-
Speech Processing
lecting enough data to find all ex-
amples is difficult. Thus, smoothing
and back-off techniques are often re-
quired. A number of new language
modeling techniques are being in-
vestigated within the LTI, including
class-based language models that
consider whole word classes, not
just individual words, for n-grams.
Additionally, new statistical model-
ing techniques such as maximum
entropy are being developed to give
better estimates of the probability
distribution of word sequences.
• Synthesis using Festival, FestVox,
and Flite: To ensure that our speech
synthesis work is available to the
widest range of users, we work with
the University of Edinburgh’s Fes-
tival Speech Synthesis System, a
free software synthesis toolkit and
engine. We have also produced the
FestVox tools for building new
voices in new languages allowing
the construction of both general
voices, and domain specific voices.
Also with the small footprint CMU
Ongoing Research
Flite system, synthesis can be used
on any platform.
• Robust speech recognition: current
automatic speech recognition sys-
tems are limited in their ability to
adapt to the effects of new speak-
ers, difficult acoustical environ-
ments, non-native accents, and
spontaneous speech production.
Researchers at the LTI are carrying
out a broad program of research to
improve the robustness of automatic
13
speech recognition using a variety
of techniques.
 Ongoing Research
Information Retrieval and Text Mining
Web search engines are a well-known type of Information Retrieval (IR) system that uses statistical inference to locate documents
that satisfy an information need. Greater search engine accuracy is always useful, but finding information is no longer the most
important IR problem. Today’s tools often deliver much more information than a person can read easily. IR and Text Mining
research at the LTI involves a wide range of issues related to finding, validating, organizing, summarizing, analyzing, and commu-
nicating large amounts of information. Our goal is to enable people to routinely base decisions on far more information than is
practical today.

Machines cannot understand the meaning of a multimedia document in the way that a human can, but many useful tasks can be
accomplished with limited forms of understanding. Statistical corpus analysis, probabilistic inference, and machine learning are the
tools of IR and Text Mining research. Research at the LTI is grounded in theory, and tested in large-scale applications. Conse-
quently, research projects focus on everything from basic theory to software engineering. Several representative examples are
Information Retrieval and Text Mining

described below.

Distributed IR (federated search) in the

real world: Peer to peer networks con-
tain searchable digital libraries (leaf
nodes), and directory services (hub
nodes) that route messages and merge
results from different sites.

Research on Advanced IR Architec-
tures develops systems that combine
standard search queries with detailed,
long-term user and task models and
highly structured documents. Document
structure may indicate how the document
is organized (e.g., XML), or it may be
provided by language analysis tools (e.g.,
named-entities, part-of-speech, syntactic
parsing). This research supports LTI
projects on open-corpus language tutor-
ing, such as the REAP reading compre-
hension project. Much of this research
is distributed via the open-source Lemur
Ongoing Research
Translingual Information Retrieval
uses queries in one language (e.g., En-
glish) to find documents in other lan-
guages (e.g., German, Chinese and Ara-
bic). Traditional machine translation
methods do not work well when queries
are short, out of context, and not sen-
tences. Our research focuses on corpus-
based translation of query terms by learn-
ing empirical associations among multi-
lingual lexicons from translation mates
(documents, paragraphs, passages or sen-
tences), and by mapping queries and
documents to conceptual ‘interlingua’ that
The Distributed Information Retrieval
project studies environments, such as the
Web, large corporate networks, and peer-
to-peer networks, in which thousands of
search engines are available. Cooperation
cannot be assumed, so robust techniques
are required for automatically character-
izing search engines, selecting among
them, searching them, and integrating re-
sults retrieved from different sources.
The Language Technologies Institute pio-
neered research in automated Text Sum-
marization with the Maximum Marginal

Toolkit . bridge the language barrier. Relevance Metric and its application to
user-profile-relevant document summari-
zation. Research also focuses on summa-
rizing dialog clusters of topically related
documents and automated generation of
briefings from corpora.
14

The REAP project uses word histo-

grams to model how children use lan-
guage at different ages. These allow
the Lemur search engine to select texts
that use vocabulary a particular child
is likely to understand.
The Lemur Toolkit for Language Model-
ing and Information Retrieval is an open-
source softwate toolkit developed at the
LTI and the University of Massachusetts.
The heart of Lemur is a set of indexing
methods that support a wide range of IR
capabilities. Lemur includes multilingual
search engines that are based on several
probabilistic and vector-space retrieval
models. Its powerful query language and
support for text annotations and document
structure make it particularly useful for
question answering, language tutoring and
other research at the LTI. It also includes
a variety of other IR capabilities, such as
federated search, text summarization and
document clustering. Lemur is used in
universities and research laboratories
around the world. For more information see
www.lemurproject.org
Text mining research at the LTI consists
of four major related categories, listed
below.
* Text categorization and filtering is a
form of supervised machine learning,
where texts are assigned categories – e.g.
emails to folders, web pages to taxonomic
classes, books to catalog codes – first by
training statistical classifiers from ex-
ample corpora, and classifying new texts
with the trained classifiers. Filtering is a
form of dynamic text categorization,
where the categories are defined implic-
itly and evolve over time.
* Information extraction from free text,
involves tasks such as detecting entities
(people, places, organizations, etc.), de-
tecting roles (e.g. Clinton as senator, or
Carter as peace envoy), and detecting re-
lations (who does what to whom). One
concrete accomplishment is a software
package called Minor Third for learning-
based information extraction (using
HMMs, CRFs and other techniques).
Information extraction is used in ques-
tion answering, in topic detection and
tracking, and in cognitive agents (e.g. in
the RADAR project). Information extrac-
tion may also be used to instantiate tables
(e.g. what product is sold by whom at
what price) for classical data mining.
* Topic detection and tracking (TDT)
utilizes supervised learning to track top-
ics or events defined by examples texts,
and unsupervised learning to detect the
emergence of new topics or events in
news streams. The latter entails detect-
ing novelty – i.e. discovering in which
ways to detect that a textual description
of an event indicates a novel one, even if
it may contain points in common with
earlier different events of like type. TDT
is inherently a dynamic time-series learn-
ing challenge, where topics may drift over
time, events initiate and fade from the
news, or morph into other events.
Language Technologies Institute
* Large public comment databases
are a feature of modern democratic so-
cieties. Email and the Web make it easy
for people to express their opinions
about proposed government policies
and regulations, consumer products,
and a wide range of other topics. Popu-
lar and controversial topics can quickly
cause hundreds of thousands of com-
ments. Text mining research at the LTI
Information Retrieval and Text Mining
includes interactive methods of orga-
nizing, summarizing, and exploring
large databases of unstructured text
comments. Tools of this type can in-
crease the responsiveness and transpar-
ency of democratic governments, and
allow companies to better track cus-
tomer opinions about products and ser-
vices.
Part of text mining relates strongly to
information retrieval, using notions such
as inverted indexing and text-to-text
similarity metrics: cosine similarity in
high-dimensional vector spaces, and
generative similarity in statistical lan-
guage modeling approaches. The LTI
is active in all aspects of text mining,
including computational methods that
scale to large real-world challenges, and
that apply to different languages.
15 Ongoing Research
 Ongoing Research
Knowledge-Based NLP and Question Answering
The LTI has a long history of research
in knowledge-based natural language
processing and computational linguis-
tics, dating back to Carbonell’s work
on knowledge-based interlingual ma-
chine translation and Tomita’s work on
efficient natural language parsing tech-
niques, when the precursor of the LTI
Knowledge-Based NLP
was Carnegie Mellon’s “Center for
Machine Translation.” Of particular
note are the KANT and KANTOO
systems developed by Nyberg,
Mitamura and Carbonell that brought
high-accuracy interlingua machine
translation to large-scale practical use
for translating technical literature at
Caterpillar Inc. to several languages.
This line of work is characterized by
careful linguistic analysis, large-scale
knowledge engineering, and solid sys-
tem building. More recently, knowl-
edge-based systems are combined with
machine learning, such as in the AV-
ENUE project where translation trans-
fer rules are learned from a minimal
number of word-aligned translation
pairs via new techniques such as
seeded-version-space learning. The
current “pure” knowledge-based
projects at LTI are:
Ongoing Research
New Bill of Rights
16
Knowledge Acquisition from Natural
Language Text
The knowledge acquisition bottleneck has
long been decried as one of the limiting
factors for applications of artificial intelli-
gence – how can we get all of the appro-
priate world knowledge into the computer
so that it can solve problems of practical
significance in a new domain? In our re-
search on knowledge acquisition from text,
we are working to define a formal map-
ping between specific structures in natural
language and corresponding meaning rep-
resentations in a formal representation (e.g.
frame logic). The goal of CMU’s contri-
bution to the HALO-II project is to reduce
the cost of encoding knowledge for a prob-
lem-solving system by making it possible
to acquire knowledge directly from an ex-
isting text (e.g., from a textbook). Current
work focuses on acquiring various types
of knowledge (ontologies, rules, processes,
etc.) from college textbooks in domains
such as Biology, Chemistry, and Physics.
Open-Domain Question Answering for
Multi-Lingual Text Collections
As the size of available on-line text collec-
tions grows ever larger, simple search en-
gines are becoming less and less effective
Get the right information
To the right people
At the right time
On the right medium
In the right language
With the right level of detail
at helping users to find answers in on-
line documents. More advanced ques-
tion-an-answering (QA) systems, such as
LTI’s JAVELIN project, use NLP tech-
niques (segmentation, stemming, pars-
ing, semantic interpretation, unification,
etc.) to a) understand the underlying
meaning of the questions they are posed,
and b) find the most likely answers in
the target collection(s). Information gath-
ering becomes a collaborative process,
where the system and user work together
in an ongoing dialog to refine the search
for ever-better answers. In addition to
research on basic parsing and interpre-
tation of unrestricted text, we are also
actively working on: a) information gath-
ering dialogs; b) sophisticated multi-
layer retrieval strategies; c) a range of
approaches to information extraction
(pattern-based, statistical, etc.); and d)
synthesis of answers from multiple an-
swer candidates. The team is also inves-
tigating the use of dynamic planning in
QA (e.g., to explore first the strategies
that are most likely to yield a good an-
swer when processing time is limited),
and use of reasoning and belief networks
to piece together individual pieces of in-
formation from several documents.

At the LTI we welcome industrial partici-
pation in our research, in the form of in-
dustrial affiliates, customized education
programs, guest researchers participating
in common projects, and funded R&D ef-
forts. The LTI collaborates with many in-
dustrial partners, national and international,
with several of these partnerships actively
ongoing and others having successfully
concluded. LTI industrial partners, past
and present, include: IBM, Intel, SRI,
Hitachi, Fujitsu, Siemens, Denso, Vulcan,
Ontoprise, Lycos, Northrup Grunman,
Hewlett Packard, Boeing, Justsystems,
CNRI, Dong-A-Seetech, Caterpillar,
Carnegie Group,Dynamix, General Elec-
Countries represented in the Language Technologies Institute (languages of study, faculty, staff,
students and/or visiting researchers)
Industrial Programs
tric, and ATR research. We engage in the
following types of relations:
Affiliateships – where industry participates
in the results of LTI research, including pub-
lications, data sets, software, and briefings.
In-residence fellows – where industrial re-
searchers or engineers join the LTI for a
specified duration to receive full-immersion
training in new technologies, to work on joint
projects, or both.
Language Technologies Institute
Customized intensive courses –
ranging from executive one- or two-
day sessions to much longer and
more technical targeted offerings on
selected topics in Language Tech-
nologies and its applications.
Sponsored R&D projects – where
LTI technologies are extended, in-
Industrial Programs
tegrated and/or applied to address
challenging problems in industry or
government.
Ongoing Research
17
 Academic Programs
The LTI currently offers two graduate degree programs: a PhD in Language Technologies and a Masters in Language
Technologies. We also participate in an undergraduate Linguistics minor (language technologies track). The LTI also offers
shorter certificate programs in Language Technologies as well; please contact us for further information on these.
Financial Support Student Evaluation
All PhD and most Masters students accepted into the Lan- Following the long-standing SCS tradition, the LTI does
guage Technologies Institute are awarded a Research Fel- not focus only on courses or exams, and does not have a
lowship for the academic year, covering full tuition and a fixed timeline for completion of the PhD degree, although
living allowance, usually renewable for the duration of the our target is five years. Instead, we carry out an individual-
program, as long as the student maintains good standing. ized student evaluation at the end of each semester based
on research performance, classes and other contributions.
Students are encouraged to apply for support from outside For each student, we write a letter indicating to them whether
Carnegie Mellon (fellowships, foreign government grants, they are making “satisfactory progress” towards complet-
etc.) As an incentive to seek funding from other sources, a ing their degree. Students are in good standing as long as
supplement is provided to the stipend of any student who
they are making satisfactory progress.
obtains outside support.

PhD in Language & Information Technologies

The PhD in Language and Information Technologies is a research-oriented degree program consisting of the following
components: successful completion of a set of courses, mastery of certain proficiencies, and a program of research, directed
by a faculty advisor, culminating in a PhD thesis.

Ph.D. Curriculum Presentation: Satisfied via a public presentation of good

A student working towards a PhD in Language and Infor-
mation Technologies must successfully complete at least
six courses in the LTI and two courses from any depart-
ment in the School of Computer Science (For more course
information, see “LTI Courses”).
Of these eight courses, the student must take at least one
Academic Programs
quality, such as an external conference presentation or an
internal seminar presentation reviewed by several faculty
members.
Programming: Satisfied by demonstrating competence in
computer programming of language technology; this is nor-
mally satisfied in the course of the student’s research, but
could also be satisfied via explicit apprenticeship if desired.

from each of the LTI Focus Areas (Linguistic, Computer

Science, Task Orientation, and Statistical/Learning) and Teaching: Satisfied by two successful Teaching Assistant-
must take at least two lab courses, which involve hands-on ships (TA), as determined by the faculty members for whom
work in one of four different areas (Speech, Machine Trans- the student serves as TA. Typical TA responsibilities in-
lation, Information Retrieval, and Natural Language Pro- clude planning a portion of the syllabus, developing exer-
cessing). The lab modules are self-paced, with Teaching cises, and delivering some lectures under faculty supervi-
Assistant and faculty guidance. Students are encouraged sion. Of the two TA-ships, typically one will be for an un-
to consider taking additional elective courses beyond the dergraduate class and one will be for a graduate class.
eight required. Students may select additional courses from
the LTI, from related courses in the Computer Science De-
partment, or from other related CMU or University of Pitts-
burgh departments. Areas of possible interest include
Speech, Linguistics, Statistics, and Human-Computer In-
teraction.

Proficiencies
The following skills must be demonstrated in the course of
graduate study, with flexibility in the form and timing of
their demonstration:
Writing: Satisfied by producing a peer-reviewed confer-
18

ence paper or a written report that at least two SCS faculty

certify as being of conference-paper quality. The topic of
the paper may be the student’s research results, a compre-
hensive survey of a research area, a linguistic analysis pa-
per, or any other pertinent topic.
Research and PhD Thesis
It is expected that all PhD students engage in active re-
search from their first semester. Moreover, advisor selec-
tion should occur within 1-2 months of entering the PhD
program, with the option to change at a later time. Roughly
half of a student’s time should be allocated to research and
half to courses until the coursework is completed.
Once the coursework is completed, the student should be-
gin to move towards a thesis topic, in consultation with the
student’s advisor. Once a suitable topic is defined, the stu-
dent prepares and presents a dissertation proposal, based
on their initial work on that topic. The dissertation pro-
posal is normally expected at the end of the third year, and
describes the general area of investigation and the specific
problem(s) to be addressed, a clear argument for the signifi-
cance of the problem, relevant past work, expected scientific
contributions of the proposed work, and a projected timeline
for completion. A dissertation committee consisting of the
advisor, at least two other CMU faculty in language technolo-
gies, and at least one external member should be formed prior
to the proposal. The dissertation itself, normally completed
during the fifth year, includes a detailed description of all the
work done, including its clear evaluation and the final scien-
tific contributions. The thesis is then defended in a public
oral presentation. A successful defense results in the award-
ing of the PhD degree.

Master of Language Technologies

The Master of Language Technologies (MLT) is a professional degree that is normally completed in two years. Students
choose an individualized curriculum from a flexible set of courses and self-paced laboratory modules that cover linguistic
and statistical approaches and basic computer science. The curriculum is usually tailored to emphasize a specialty in one of
three language technology areas: Machine Translation, Information Retrieval, or Speech Technology. Directed research is
an integral part of the MLT program; each MLT student carries out research under the guidance of a faculty advisor.
With some modifications and enhancements, the MLT curriculum also forms the course-based component of the PhD Pro-
gram. The more research-oriented MLT students are encouraged to apply for continuing studies in the PhD program, with
most of their MLT courses and hands-on work being credited towards the PhD.

Master of LT Curriculum

Academic Programs
The curriculum for the MLT consists of a minimum of
120 course units at a senior or graduate level. From these
120 units, six courses must be LTI courses and two other
courses must be SCS courses. There are additional con-
straints on course selection, required in order to meet SCS-
wide Masters requirements. A concentrated form of this
degree may be completed in one year without the research
component.

Master of LT Thesis Option

A Masters Thesis Option is available for students who
wish to demonstrate independent research ability during
their enrollment in the LTI Masters program. Students
who choose the Masters Thesis Option will be expected
to follow thesis guidelines that are similar in character to
19

those for the LTI PhD. The Masters thesis requirements

are less rigorous, however, since the Masters disserta-
tion is expected to be defined, completed, and publicly
defended in less than one year.

Language Technologies Institute

 Courses and Admissions

LTI Courses

Sample Course Descriptions

We briefly describe here the main focus of a sample of our courses; we list these in numerical order. To see a complete list
of current courses and course descriptions, please see: http://www.lti.cs.cmu.edu/Courses/

11-731 Machine Translation: A graduate-level course

surveying the history, techniques, and research topics in
the field of Machine Translation.

11-741 Information Retrieval: This course studies the

theory, design, and implementation of text-based informa-
tion systems. The IR core components of the course in-
clude important retrieval models (Boolean, vector space,
probabilistic, inference net, language modeling), cluster-
ing algorithms, automatic text categorization, and experi-
mental evaluation. A variety of current research topics are
also covered, including cross-lingual retrieval, document
summarization, machine learning and topic detection and
tracking.

11-743 Advanced IR Seminar/Lab: This is a seminar

that focuses on current research in Information Retrieval.
The seminar covers recent research on subjects such as
11-682 Human Language Technologies (“Words for
retrieval models, text classification, information gathering,
Nerds”): During the last decade computers have begun
fact extraction, information visualization, summarization,
to understand human languages. Web search engines, lan-
text data-mining, information filtering, collaborative fil-
guage analysis programs, machine translation systems,
tering, question answering systems, and portable informa-
speech recognition, and speech synthesis are used every
tion systems.
day by tens of millions of people in a wide range of situa-
LTI Courses and Admission

tions and applications. This course covers the fundamen-

11-751 Speech Recognition: This course provides an in-
tal statistical and symbolic algorithms that enable com-
troduction to the theoretical foundations, essential algo-
puters to work with human language, from text process-
rithms, major approaches, experimental strategies and cur-
ing to understanding speech and language.
rent state-of-the-art systems in speech recognition.
11-711 Algorithms for NLP: A graduate-level course on
11-752 11-752: Phonetics, Prosody, Perception and
the computational properties of natural languages and the
Synthesis: This course offers insight into how human
fundamental algorithms for the symbolic processing of
perception of speech relates to the physical properties of
natural languages.
the signals. It covers practical aspects of speech includ-
ing projects. The second half of this course concentrates
11-717 LT for Computer-Aided Language Learning:
on speech synthesis and building of synthetic voices.
This course studies the design and implementation of
CALL systems that use Language Technologies such as
11-761 Language and Statistics: This course covers some
Speech Synthesis and Recognition, Machine Translation,
of the central themes and techniques that have emerged in
and Information Retrieval.
statistical methods for language technologies and natural
language processing.
11-721 Grammar and Lexicon: A graduate-level course
on linguistic data analysis and theory, focusing on meth-
11-791/792 Software Engineering for Language Tech-
odologies that are suitable for computational implemen-
nologies I/II: This two-course sequence combines class-
tations. The course covers major syntactic and morpho-
room material and assignments in the fundamentals of soft-
logical phenomena in a variety of languages. The empha-
ware engineering (11-791) with a self-paced, faculty-su-
sis is on examining both the diversity of linguistic struc-
pervised directed project (11-792). The two courses cover
tures and the constraints on variation across languages.
all elements of project design, implementation, evaluation
and documentation.
20

11-723 Formal Semantics: A graduate-level course on

formal linguistic semantics: Given a syntactic analysis of
a natural language utterance, how can one assign the cor-
rect meaning representation to it, using a formal logical
system?
Example Course Sequences
For purposes of illustration, the following tables give possible curriculums for a PhD student specializing in Machine
Translation, a MLT student following the two-year track, and an MLT student following the condensed one-year track. For
other PhD students, specializations in Speech, Information Retrieval, and Multimedia Systems might be similar in structure,
with appropriate course substitutions. The two-year MLT track allows a student to work on a research project, to work as a
Teaching Assistant, or to engage in other compensated work. The one-year MLT track requires full concentration and does
not usually allow the student to engage in any other work. Note that these are only examples and are not meant to apply to
every case, or even most cases.

Year 1 Year 2

Grammar & Lexicon Software Engineering for LT 1

Algorithms for NLP Elective
First
Directed Study Principles of Translation
Semester Semester 1 Semester 2 Semester 3
Laboratory Directed Study (Spring) (Summer)
(Fall)
Laboratory
Linguistic Basis of NLP Speech Understanding Software Engineering for
Speech Recognition Information Retrieval IT (2)
Artificial Intelligence Machine translation
Machine Translation Elective Elective (Directed Study)
S eco n d Statistical Methods in Elective
Directed Study Software Engineering for LT 2 NLP Laboratory (2 modules)
Semester Information Retrieval
Laboratory Directed Study Software Engineering for Laboratory (1 module)
Laboratory IT (1)
Laboratory (1 module)
MLT Two-Year Track
MLT Concentrated 12-Month Track

Year 1 Year 2 Year 3 Year 4 Year 5

Grammar and Software Teaching (TA) Elective or Seminar Research

Lexicon Engineering for LT 1 Research Research
First Algorithms for NLP Language and
Semester Statistics
Self-paced Lab
Research Research PhD Sample Track for MT
Concentration

LTI Courses and Admission

Machine Translation Software Thesis Proposal Elective or Seminar Thesis Defense
Artificial Intelligence Engineering for LT 2 Research Research
S eco n d Principles of
Self-paced Lab
Semester Translation
Research
Research

Admissions

The ideal applicant to either the Masters or PhD

program would have a strong background in com-
puter science and either linguistics or statistics,
depending on the specific area of interest. Many
successful students are of course stronger in one
area than another, but admission to our programs
is highly competitive, so any lack of background
in one area must be compensated for by excel-
lence in other areas. A clearly articulated inter-
est in some specific aspect of language technolo-
21

gies is also important. For detailed information

on application requirements and procedures, visit:
http://www.lti.cs.cmu.edu/About/how-to-
apply.html

Language Technologies Institute

 Faculty
Alan W Black
Associate Research
Professor
BS Computer Science
Coventry Polytechnic 1984
MS Knowledge Based
Systems
University of Edinburgh
1986
PhD Artificial Intelligence
University of Edinburgh
1993
Ralf Brown
Senior Systems
Scientist
BS Computer Science
Towson University 1986
PhD Computer Science
Carnegie Mellon
University 1993
Faculty
Jamie Callan
Associate Professor
22
BA Applications of
Computer Science
Univ. of Connecticut 1984
MS Computer & Information
Science
Univ. of Massachusetts
1987
PhD Computer Science
Univ. of Massachusetts
1993
Alan Black has created practical implementations of computational theories of speech and language.
After a wide background in morphology, language modeling in speech recognition, and computation
semantics, he now works in all aspects of speech generation. As an author of the free software Festival
Speech Synthesis System, he researched text analysis, prosodic modeling, waveform generation, and ar-
chitectural issues in synthesis systems. His work targets data-driven computational models that allow
synthesizers to capture speaker style. Specifically, he studies data-driven prosodic models, automatic
building of voices in English and other languages. To allow spoken output anywhere, he also deploys this
work on handheld computers, specifically addressing rapid development of voices in new languages,
modeling of speaker individuality, and evaluation of voice quality.
Professor Black teaching is very practical, thus his courses involve significant exercises that allow stu-
dents to gain experience in building synthetic voices, statistically trained models, etc. After some practical
experience it is easier to understand the underlying theoretical issues and their relative importance.
www.cs.cmu.edu/~awb
speech synthesis speech to speech translation spoken dialog systems
Ralf Brown's research interests cover several areas of language technology, such as reference resolution,
disambiguation, corpus-based machine translation, cross-language information retrieval, and topic track-
ing in news. His recent research has focused on Example-Based Machine Translation and its applications,
particularly in the context of multi-engine translation systems, and on topic tracking in news. He also
works with machine-learning techniques for extracting patterns from parallel text in order to build trans-
lation systems with less training material.
Current and recent projects include RADD (Rapidly-Adaptable Data-Driven Machine Translation), AV-
ENUE (machine translation for languages with few resources), Topical Novelty Detection in the TDT
(Topic Detection and Tracking) program for detecting new events in the news and tracking their evolu-
tion, TONGUES (rapid development of bi-directional speech-to-speech translation systems), and
MUCHMORE (cross-language information retrieval in the medical domain).
www.cs.cmu.edu/~ralf
machine translation cross-language information retrieval topic tracking
Jamie Callan is interested in a wide range of information retrieval and text mining topics. In recent years
his research has focused on four problems listed below.
• Federated Search (“Distributed IR”): Provide access to many search engines through a single
search interface; includes peer-to-peer search. Research topics include learning what each engine
contains, selecting which to search, searching them, and integrating results from different sources.
• Adaptive Document Filtering: Monitor information streams to find documents that satisfy an infor-
mation need. The system should learn a person’s information needs, rapidly identify desired docu-
ments, and distinguish between novel and redundant information.
• Large-Scale Text Analysis: Develop tools for rapidly analyzing large text datasets. For example,
when a government agency receives 100,000 comments about a new regulation, it needs to know
which groups commented, what topics were discussed, and what supporting evidence was cited.
• IR for Language Applications: Search engines are increasingly used in question answering and
language tutoring systems. Such applications require rich text annotation (e.g., syntax, named en-
tity), complex queries, and retrieval models that combine varied forms of evidence.
His students initially work closely with him to study specific ideas while learning research skills and IR.
As students gain expertise, they develop their own interests and have more freedom in exploring them.
www.cs.cmu.edu/~callan
information retrieval adaptive information filtering text data mining

Jaime Carbonell
Allen Newell Professor
Director, LTI
BS Physics and Mathematics
MIT 1975
MS Computer Science
Yale University 1976
PhD Computer Science
Yale University 1978
Maxine Eskenazi
Associate Teaching
Professor
BA Modern Languages
Carnegie Mellon University
1973
DEA Linguistics
University of Paris VII 1981
Doctorat de Troisieme Cycle
Computer Science
University of Paris XI 1984
Scott E. Fahlman
Research Professor
BS Electrical Engineering
and Computer Science
MIT 1973
MS Electrical Engineering
and Computer Science
MIT 1973
PhD Artificial Intelligence
MIT 1977
Jaime Carbonell is the Director of the Language Technologies Institute and Allen Newell Professor of
Computer Science at Carnegie Mellon University. He received his BS degrees in Physics and Math-
ematics from MIT, and his MS and PhD degrees in Computer Science from Yale University. His
current research interests span several areas of artificial intelligence and language technologies, in-
cluding: machine learning, data and text mining, natural language processing, machine translation,
and automated summarization (where he invented MMR search-diversity technology). Professor
Carbonell's most recent research directions are structural computational biology, using machine learning
and language technologies to predict structure and function from protein sequences and biophysical
knowledge, and autonomous assistant agents that learn via observation, experience, and NL commu-
nication.
Professor Carbonell is a key participant in the RADAR project and serves in key advisory positions
external to CMU.
www.cs.cmu.edu/~jgc
artificial intelligence natural language processing machine learning/translation
Maxine Eskenazi's research interests lie in the variability of the speech signal whether it be to aid non-
native speakers to learn a language, to enable systems to dialogue with the elderly, or to process the of
speech of any other group of speakers whose production differs greatly from the average. At the LTI,
she has created the Fluency project, which develops basic algorithms and systems for language learn-
ing. The systems are used for foreign language learning as well as for learning American English
dialects. They are also used to test pedagogical theories about language learning. Work on this project
has spun off a company, Carnegie SpeechTM, which has created products based on Fluency algo-
rithms. She also works on the use of authentic materials for language learning. Here we characterize
the language learners knowledge and the knowledge to be acquired (curriculum) and then determine
which texts, from a very large database of texts taken off the Web, should be shown to the learner next.
Teacher-created curriculum and learners interests add to the power of adaptation to the individuals
needs.
Dr. Eskenazi views teaching as a constant dialogue. It is an occasion for all individuals concerned to
come together and learn something, question something, change something.
www.lti.cs.cmu.edu/~max
computer-aided language learning speech processing speech recognition
Scott Fahlman is responsible for the knowledge-representation research effort on the RADAR Project,
a large DARPA-funded research effort whose goal is to build an automated cognitive assistant for
busy managers, making extensive use of AI and machine-learning techniques.
As a researcher, he is primarily interested in Artificial Intelligence and its applications. Currently, he Faculty
is working on SCONE, a practical system that can represent a large body of real-world knowledge
and that can efficiently perform the kinds of search and inference that seem so effortless for us hu-
mans. He believes that such "knowledge base" systems will be important tools in the future, perhaps
used in even more ways than database systems are used today.
With respect to natural language understanding, the field has made considerable progress focusing on
superficial aspects of language, but Professor Fahlman believes that future progress depends on our
23
ability to extract and represent the actual meaning of a piece of text, and to use large amounts of
background knowledge in understanding the text,using powerful new tools for knowledge represen-
tation and inference.
www.cs.cmu.edu/~sef
artificial intelligence knowledge representation machine learning
 Faculty
Eugene Fink's research interests are in various aspects of artificial intelligence, including machine
learning, planning, problem solving, automated problem reformulation, e-commerce applications,
medical applications, and theoretical foundations of artificial intelligence. His interests also include
computational geometry and algorithm theory.

He is currently working on an intelligent system for automated allocation of offices and related re-
sources, in both crisis and routine situations. This work is part of the RADAR project, aimed at
creating a general-purpose assistant for office managers. He is also working on techniques for identi-
fication of both known and surprising patterns in large-scale databases, and applying these tech-
niques to homeland security. This work is part of the ARGUS project, which is a joint research project
Eugene Fink involving Carnegie Mellon and Dynamix Technologies.
Systems Scientist
BS Mount Allison University www.cs.cmu.edu/~eugene
1991
MS University of Waterloo
1992
PhD Carnegie Mellon
University
1999
artificial intellegence machine learning computational geometry

Robert
Frederking
Senior Systems Scientist,
Director LTI Graduate
Programs
BS Computer Engineering
Case Western Reserve
University 1977
PhD Computer Science, AI
Carnegie Mellon University
1986
Bob Frederking's primary research area has been machine translation applications that do not currently
permit the use of purely knowledge-based techniques. This includes rapidly developing Machine Trans-
lation (MT) for new languages and translating text and speech that are not limited to a narrow, well-
defined domain. Our main technical approach in this area is Multi-Engine MT (MEMT). MEMT applies
several different MT techniques to the same text, and then attempts to select the best results from each
technique. He developed and implemented the initial chart-based dynamic-programming technique for
merging the results from the different engines and our current merging technique, which uses statistical
language modeling to select among the different technique outputs. He has also been involved in LTI
projects in Cross-Language Information Retrieval, Question Answering, and Information Extraction
from email, among other things.
Professor Frederking believes that successful advising and teaching hinge largely on successful commu-
nication: presenting advice (or a lecture), understanding what (if anything) the student is having trouble
with, and then providing the information or guidance that he or she needs to resolve any difficulties. As
the Chair of the LTI's graduate programs, he is the default advisor for students who are not project-
supported.
www.cs.cmu.edu/~ref/
speech-to-speech MT rapid-development wide-coverage MT question answering

Alex Hauptman's research aims to design and build intelligent programs that process data from large
volumes of multimedia data, including text, image, video, and audio and make the data useful for other
applications, so as to improve speech recognition, image understanding, NLP, machine learning, ques-
tion answering and IR. The challenge is to find the right data, to process it into a suitable form for
Faculty

training, learning, or re-use, and to build mechanisms that can successfully utilize this data.

This work takes part in the context of the Informedia digital video project, which aims to achieve ma-
chine understanding of video and film media, including all aspects of search, retrieval, visualization and
Alex Hauptmann summarization in both current and archival content collections. The base technology developed under
Informedia combines speech, image and natural language understanding to automatically transcribe,
Senior Systems Scientist
segment and index linear video for intelligent search and image retrieval.
24

BA Psychology
Johns Hopkins University
1982
MA Psychology
Johns Hopkins University www.cs.cmu.edu/~alex
1982
Diplom Computer Science
Technische Universität Berlin
1984
PhD Computer Science
Carnegie Mellon University
1991
multimedia analysis multimedia interfaces Informedia digital video library

Judith Klein-
Seetharaman
Assistant Professor, Depart.
of Pharmacology, Univ. of
Pittsburgh School of Med.
Research Scientist, LTI
Diplom in Biology, Univ. of
Cologne, Germany 1995
Diplom in Chemistry, Univ.
of Cologne, Germany 1996
PhD Biological Chemistry,
MIT 2000
John Lafferty
Professor (CSD, LTI)
BA
Middlebury College 1982
MS
Princeton University 1984
PhD Mathematics
Princeton University 1986
Alon Lavie
Associate Research
Professor
BA Computer Science
Israel Institute of
Technology 1987
MS Computer Science
Carnegie Mellon University
1993
PhD Computer Science
Carnegie Mellon University
1996
How does sequence map to structure and function of proteins in different organisms? Dr. Klein-
Seetharaman takes a linguistically inspired view of this question in analogy to “How do words map to
meaning in natural languages?” using stochastic language modeling technologies. Computational models
are validated experimentally by interdisciplinary (biochemical and biophysical, in particular NMR
spectroscopic) studies of purified proteins and model peptide sequences. The emphasis lies on testing
predicted sequence dependence on structural and dynamic aspects of folding/misfolding and func-
tional properties of proteins. Specific proteins that are expressed, purified and studied experimentally
in Dr. Klein-Seetharaman’s laboratory include the G-protein coupled receptor rhodopsin, the glutamate
receptors and the epidermal growth factor receptor. These systems function in diverse signal transduc-
tion pathways, but resemble each other in their mechanism of action. Each receptor undergoes sub-
stantial conformational changes during the signaling process and the investigation of the precise mo-
lecular details of these changes is instrumental to elucidating the molecular mechanism of signaling by
these molecules.
www.cs.cmu.edu/~judithks
Computational biology/bioinformatics biochemistry/biophysics structural biology
The central focus of John Lafferty's research is machine learning, including algorithms, theory, and
statistical methods for learning from data. The motivating applications for this work most often comes
from text and natural language processing, information retrieval, and other areas of language tech-
nologies. For example, in recent work with his colleagues he has studied approximate inference
algorithms for a family of mixture models appropriate for document collections, and applied the
algorithms to automatically extract the subtopic structure of scientific articles. Over several years
Professor Lafferty has been involved in the development of a language modeling approach to infor-
mation retrieval, including a general approach to IR based on decision theory. In other work he is
researching learning algorithms for sequential and graph-structured data, using a framework called
conditional random fields for combining the strengths of graphical models with discriminative classi-
fication methods such as support vector machines and logistic regression.
www.cs.cmu.edu/~lafferty
natural language processing machine learning information theory
Alon Lavie's main areas of research are Machine Translation (MT) of both text and speech, and
Spoken Language Understanding (SLU). His current most active research is on the design and devel-
opment of new approaches to Machine Translation, for languages with limited amounts of data re-
sources. He has also worked extensively on the design and development of Speech-to-Speech Ma-
chine Translation systems and on robust parsing algorithms for analysis of spoken language.
Faculty
Faculty
Professor Lavie is co-PI of the AVENUE project (funded by NSF/ITR), where we are developing a
general framework for building prototype MT systems for languages for which only scarce amounts
of data and linguistic resources are available. He also works on parsing algorithms for spoken lan-
guage analysis of databases of transcribed spoken language (such as CHILDES). He was co-PI of the
Nespole! and C-STAR speech translation projects and of the LingWear and Babylon mobile speech
translation projects, where he directed the design and development of the analysis and translation
25
components.
He is the principal instructor of the graduate-level course on "Algorithms for NLP". He also teaches
the section on "Natural Language Processing" for the "Introduction to Human Language Technolo-
gies" course, and supervise the Lab in NLP (11-712) lab course at the LTI.
www.cs.cmu.edu/~alavie
machine translation spoken language understanding machine learning
 Faculty
Lori Levin
Associate Research
Professor
BA Linguistics
University of Pennsylvania
1979
PhD Linguistics
MIT 1986
Teruko Mitamura •
Associate Research
Professor
LTI Finance Director
MA Linguistics
University of Pittsburgh
1985
PhD Linguistics
University of Pittsburgh
1989
Faculty
Eric Nyberg
Associate Professor
26
BA Computer Science
Boston University 1983
PhD Computational
Linguistics
Carnegie Mellon
University 1992
Lori Levin works on linguistic issues in machine translation of spoken and written language. Her
career-long research interest is the design of multi-lingual systems that accommodate typologically
diverse languages. LTI's AVENUE project focuses on translation of languages with scarce data re-
sources. Developing a machine translation system typically requires a level of economic and human
resources that may not be available for all languages. Research on MT for minor languages combines
linguistic typology and machine learning to automate the production of machine translation systems
for new languages.
She is also part of a consortium for designing semantic interlingual representations of text meaning.
We are using multi-parallel corpora (multiple versions of the same text) to center in on what is com-
mon among sentences that are supposed to convey the same meaning. In addition to the interlingua
design, the consortium is producing annotated multi-parallel corpora, tools for annotation, and evalu-
ation metrics.
Her other interests include computer-assisted language learning, especially tools to assist second
language readers with comprehension of authentic texts.
www.cs.cmu.edu/~lsl
minority languages machine translation interlingua representations lexicons
Professor Mitamura’s research focuses on the following projects:
• JAVELIN-II (open-domain, multilingual question answering): A system which combines NLP,
planning, IR and MT to answer natural language questions and refine the search strategy in consul-
tation with the user.
• CAMMIA (Conversational Agent for Multilingual Mobile Information Access): A system which
extends VoiceXML with NLP and dialog management to support dynamic multi-task dialogs in
Japanese and English.
KANT (Knowledge-based Accurate Natural Language Translation): A project founded in 1991 for
the research and development of large-scale, practical translation systems for technical documenta-
tion. KANT uses a controlled vocabulary and grammar for each source language, and explicit, yet
focused semantic models for each technical domain to achieve very high accuracy in translation.
Teruko Mitamura teaches the courses Machine Translation, Grammars and Lexicons, and LT for CALL.
www.cs.cmu.edu/~teruko
knowledge-based MT question answering Japanese NLP and dialog systems
Eric Nyberg's research at LTI is currently focused on three main areas:
• Open-Domain Question Answering. The JAVELIN project combines natural language dialog, in-
formation retrieval, text understanding, fact extraction, and probabilistic reasoning to answer com-
plex questions about entities, relationships and events expressed in unstructured text.
• Conversational Agents for Mobile Multilingual Information Access. The CAMMIA project is cre-
ating speech dialog systems for robust, multi-task dialogs in mobile environments such as car
navigation systems.
• Knowledge-Based Machine Translation. Since the late 1980’s he has worked on controlled lan-
guage, document checking and machine translation for technical documentation; the current sys-
tem, KANTOO, is now in use at Caterpillar, Inc.
Professor Nyberg also teaches a two-course series on software engineering and information technol-
ogy, where students learn about software analysis, design, and construction in the context of real-world
team projects.
www.cs.cmu.edu/~ehn
machine translation integrated information management software engineering

Carolyn Penstein
Rose
Research Scientist
BS Computer Science
University of California at
Irvine, 1992
MS Computational
Linguistics Carnegie Mellon
University 1994
PhD Language and
Information Technologies
Carnegie Mellon
University 1997
Roni Rosenfeld
Professor
BS Mathematics and Physics
Tel-Aviv University 1985
MS Computer Science
Carnegie Mellon University
1991
PhD Computer Science
Carnegie Mellon University
1994
Alex Rudnicky
Principal Systems
Scientist
BS Psychology
McGill University 1975
MS Psychology
Carnegie Mellon University
1976
PhD Psychology
Carnegie Mellon University
1980
Carolyn Penstein Rose's primary research objective is to develop and apply language technology
(i.e., robust language understanding technology and dialogue management technology) to enable
effective computer based and computer supported instruction. The important role of students making
their thinking explicit through verbal explanation is well established. Thus, a major thrust of her
current research is to explore issues related to eliciting and responding to student explanation behav-
ior. However, many of the underlying issues, such as influencing student expectations, motivation,
and learning orientation, transcend the specific input modality. She is the PI for two tutorial dialogue
projects, namely CycleTalk for thermodynamics tutoring, and Calculategy for calculus tutoring. She
is also co-PI for a physics tutoring project headed up by Kurt VanLehn at the University of Pittsburgh.
She has served as a co-instructor for Grammar Formalisms and the Master's of HCI Project Course.
Professor Penstein Rose is also the primary instructor of the Conversational Interfaces course, which
is jointly listed in LTI and HCI.
www.cs.cmu.edu/~cprose/
robust language understanding technology supported education
Professor Rosenfeld’s research spans two key areas:
• Computational Molecular Biology and more specifically Computational Biolinguistics. Many
of the problems in this area involve statistical modeling of long sequences of building blocks
(nucleotides or amino acids) and their relationship to proteins and their function. This is very
similar to the problem of modeling natural language: long sequences of letters or words, and
their relationship to the deep structure and meaning of sentences. He is currently working to
detect and characterize specific selectional pressure in proteins.
• Speech interaction with PDAs, web portals, and robots is now feasible. But what is the ideal
style for human-machine speech communication? Natural language interfaces are easy for people,
yet they are brittle, difficult to develop, and they strain recognition technology. Furthermore, by
trying to emulate people, they fail to communicate the functional limitations of the machine. Are
there better alternatives? The Speech Graffiti (aka USI) project is designing and evaluating new
speech-based interaction paradigms.
www.cs.cmu.edu/~roni
statistical language modeling speech recognition/interfaces machine learning
Alex Rudnicky's research centers on interactive systems that use speech. He is interested in the following
problems:
• Speech systems that learn: his research attempts to develop a process that, given an abstract
specification of capabilities, supports the automatic configuration of a speech system for an inter-
active task, and then supports incremental learning over the life of the application.
Faculty
Faculty
• Automatic detection and recovery from error: Automatic systems cannot easily detect and recover
from communication breakdowns. We can, however, use features of recognition, understanding,
and dialog to predict the likelihood of misunderstanding at a given instance, and then apply heu-
ristic strategies for guiding the conversation back onto track.
• A theory of language design for speech-based interactive systems: Speech-mode communication
predisposes the user to choose certain words and grammatical preferences. Understanding the
27
underlying principles of these preferences(and how these are influenced by the system's language)
leads to better language design for interactive systems.
• The role of speech in the computer interface: We can analyze an interface in terms of its intended
task(s), costs of interactions, and the perceived user value. We've studied models based on time,
system error, and task structure, which are useful for simple systems and appear to be extensible to
more complex systems.
www.cs.cmu.edu/~rudnicky
spoken language interaction speech recognition interface design
 Faculty
Tanja Schultz believes that automatic speech recognition systems are the most natural front-end for
applications which allow human communication across language and culture barriers. Therefore, her
research focuses on revealing techniques and algorithms to construct human-human communication
and human-machine interaction applications that can robustly function in multilingual environments.
Furthermore, her work involves the rapid deployment of speech recognizers in new tasks and lan-
guages. The massive reduction of effort in terms of time and costs is necessary to speed up the develop-
ment of recognizers in new tasks and languages. It is her belief that this is an essential prerequisite in
order to make speech-driven applications attractive and available to the public and also to include
speakers of languages in which only few or no resources are available.

In her teaching, Tanja combines an introduction to theoretical foundations, essential algorithms, and
Tanja Schultz state-of-the-art systems strategies with experimental practice. Combining these two facets, students
Research Scientist have the opportunity to gain a deep understanding of the theory as well as hands-on expertise to develop
MS Mathematics and
Physical Education
speech recognition and understanding systems. Her goal is to provide a foundation and help them to
University of Heidelberg explore their own ideas.
MS Computer Science
University of Karlsruhe www.cs.cmu.edu/~tanja
1995
PhD Computer Science
University of Karlsruhe speech recognition human-human communication human-machine communication
2000

Michael Shamos' research interests include digital libraries, language identification, electronic voting,
electronic negotiation, Internet law and policy, and experimental mathematics. As Co-Director of the
Institute for eCommerce, he runs the technology side of the Masters of Science in Electronic Commerce
program. Additionally, he teaches the courses Ecommerce Technology, Electronic Payment Systems,
and Ecommerce Law and Regulation. He is also the Director of the Universal Library project.

Professor Shamos' business and consulting experience includes serving as an expert witness in com-
puter software and electronic voting cases, as an examiner of electronic voting systems, as a consultant
Michael Shamos on electronic voting, and as an arbitrator in computer-related disputes for the American Arbitration
Distinguished Career Profes-
Association. Additionally, he was a Supervisory Programmer with the National Cancer Institute from
sor, Co-Director, Institute for 1970 to 1972 while a commissioned officer in the United States Public Health.
eCommerce, Director, Univer-
sal Library www.ecom.cmu.edu/shamos.html
AB Physics
Princeton University 1968
MA Physics
Vassar College 1970
PhD Computer Science
Yale University 1978
JD Duquense University digital libraries language identification Internet policy electronic negotiation
1981

Rich Stern's research group develops techniques that improve the accuracy of speech recognition sys-
tems in difficult acoustical environments. They deal with problems in recognition accuracy resulting
from additive noise sources, background music, competing talkers, room reverberation, and other sources
of degradation such as non-native accents or spontaneous speech production. His group has been
Faculty

developing creative solutions to these problems using classical statistical compensation techniques,
microphone arrays, and signal processing based on auditory physiology and perception. He has also
worked in the areas of language modeling, the integration of phonetic, syntactic, and semantic informa-
tion, and multimodal fusion of information. In addition to his speech recognition work, he has also
maintained an active research program in psychoacoustics, where he is best known for theoretical work
Richard Stern in binaural perception.
Professor of Electrical and
Computer Engineering
28

BS Electrical Engineering Professor Stern has taught the ECE courses in digital signal processing and signals and systems for
MIT 1970 many years, along with other courses in the general areas of communication theory and acoustics. He
MS Electrical Engineering and frequently lectures on speech recognition, signal processing, speech perception, and speech production
Computer Sciences for various LTI courses.
University of California,
Berkeley 1972 www.ece.cmu.edu/~rms
Ph.D Electrical Engineering
and Computer Science
MIT 1977

robust automatic speech recognition auditory perception signal processing


Alex Waibel
Associate Director, LTI
Director, Interactive
Systems Lab, Professor
BS Electrical Engineering
MIT 1979
MS Computer Science
Carnegie Mellon University
1980
PhD Computer Science
Carnegie Mellon University
1986
Eric Xing
Assitant Professor
MS Computer Science,
Rutgers University,1998
Ph.D Molecular Biology and
Biochemistry,
Rutgers University, 1999
Ph.D Computer Science,
University of California,
Berkeley, 2004
Yiming Yang
Professor
MS Computer Science
Kyoto University 1982
PhD Computer Science
Kyoto University 1986
text categorization
Alex Waibel holds joint appointments in LTI, the Human-Computer Interaction Institute (HCII), the
Computer Science Department and the Robotics Institute. He also directs the Interactive Systems
Laboratories (ISL), both here at Carnegie Mellon University and at Karlsruhe University in Germany.
In the ISL, they aim at building more flexible and natural Multimodal Human-Computer Interfaces
and Computer Mediated Human-Human Interfaces. His speech related research interests include
language processing, language translation and machine learning. His efforts led to the creation of
JANUS, one of the first and most advanced speech-to-speech translation programs, C-STAR (Consor-
tium of Speech Advanced Research). His former multimodal and speech projects included the Meet-
ing Room, the Genoa Meeting recognizer, the Meeting Browser, LingWear, a wearable tourist assis-
tant and medical translation system, and NESPOLE!, a multi-language E-commerce application. Some
of his latest related research projects are TC-Star and Str-Dust (unlimited domain speech translation)
and CHIL (Computers in the Human Interaction Loop)
www-2.cs.cmu.edu/~ahw/
perceptual user interfaces speech recognition multimodal interaction
The major theme of Professor Xing's research is understanding and modeling how living systems
function and evolve based on mathematical principles, and developing probabilistic inference and
learning algorithms for computational biology and for generic intelligent systems of a wide range of
applications such as vision, IR and NLP. Currently, his projects largely fall into two categories:
• Computational Biology, with an emphasis on developing formal models and algorithms that
address problems of practical biological and medical concerns, such as, 1) modeling genome-
microenvironment interactions in cancer development and embryogenesis via joint analysis of
genomic, proteomic, cytogenetic and pathway signaling data; 2) statistical inference of haplotype,
linkage and pedigree for genetic, clinical and forensic applications; and 3) modeling substitution,
recombination, selection and genome rearrangement for comparative genomic analysis.
• Statistical Machine Learning, emphasizing theory and algorithms for learning complex probabi-
listic models, learning with prior knowledge, and reasoning under uncertainty. We focus on, 1)
variational inference/learning theory and algorithms; 2) algorithms and applications of Bayesian
nonparametrics and hierarchical Bayesian models in data mining; and 3) probabilistic and optimi-
zation-theoretic methods for semi-unsupervised learning and kernel machines.
www.cs.cmu.edu/~epxing
computational biology machine learning computational statistics
Yiming Yang is a Professor of the Language Technology Institute and the Computer Science Depart-
ment at Carnegie Mellon University. Professor Yang received her B.S. degree in Electrical Engineer-
ing, and her Ph.D. degree in Computer Science (Kyoto University, Japan). Her research has been
centered on statistical classification methods and their applications to a variety of challenging prob- Faculty
Faculty
lems in the real world, including automated text classification, novelty detection and event tracking,
protein sequence analysis, cross-language information retrieval, web-mining for multimedia question
answering, and intelligent email filtering and organization.
Professor Yang teaches Information Retrieval and Advanced Statistical Learning courses.
29
www.cs.cmu.edu/~yiming
translingual information retrieval event detection and tracking
 Adjunct Faculty

David Evans
founder Clairvoyance
former Director of the Laboratory for Computational Linguistics, CMU
former Director of the Academic Computational Linguistics Program, CMU

Phil Hayes
co-founder Carnegie Software Partners
former faculty member Computer Science Department, CMU

Michael “Fuzzy” Mauldin

creator of LYCOS
Chairman of the Board of Conversive, Inc.
former faculty member LTI, CMU

Vibhu O. Mittal
Senior Scientist, Google, Inc.

Raúl E. Valdés-Pérez
co-founder and President of Vivisimo Inc
former faculty member Computer Science Department, CMU
30
31