See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/325203163

NL2API: A Framework for Bootstrapping Service Recommendation using Natural

Language Queries

Conference Paper · May 2018

CITATIONS READS

0 21

5 authors, including:

Anup Kalia Maja Vukovic

IBM IBM
24 PUBLICATIONS   83 CITATIONS    84 PUBLICATIONS   895 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Web Service Composition View project

Trust and Commitments View project

All content following this page was uploaded by Anup Kalia on 17 May 2018.

The user has requested enhancement of the downloaded file.

 NL2API: A Framework for Bootstrapping Service Recommendation using Natural
Language Queries

Chen Lin∗ , Anup K. Kalia† , Jin Xiao† , Maja Vukovic† and Nikos Anerousis†
∗ Department of Computer Science
North Carolina State University, Raleigh, NC
Email: clin12@ncsu.edu
† IBM T. J. Watson Research Center, Yorktown Heights, NY 10598
{anup.kalia, jinoaix, maja, nikos}@us.ibm.com

Abstract—Existing approaches to recommend services using
natural language queries are supervised or unsupervised.
Supervised approaches rely on a dataset with natural language
queries annotated with categorizing labels. As the annotation
process is manual and requires deep domain knowledge,
these approaches are not readily applicable on new datasets.
On the other hand, unsupervised approaches overcome the
limitation. To date, unsupervised approaches are primarily
based on matching keywords, entity relationships, topics and
clusters. Keywords and entity relationships ignore the semantic
similarity between a query and services. Topics and clusters
capture the semantic similarity, but rely on mashups that
explicitly capture relationships between services. Again, for new
services, the information are not readily available. We propose
NL2API, a framework that relies solely on service descriptions
for recommending services. NL2API has the benefit of being
immediately applicable as a bootstrap recommender for new
datasets. To capture relationships among services, NL2API
provides different approaches to construct communities where
a community represents an abstraction over a group of services.
Based on the communities and users’ queries, NL2API applies
a query matching approach to recommend top-k services. We
evaluate NL2API on datasets collected from Programmable
Web and API Harmony. Our evaluation shows that for sizable
datasets such as Programmable Web NL2API outperforms
baseline approaches.
Keywords-service discovery, topic modeling, deep learning,
community detection, clustering, web services
I. I NTRODUCTION
Web services help users to develop applications by offer-
ing them a wide range of functionalities in the domains of
IT, e-commerce, marketing, location, communications, and
so on. With the increasing number of services in different
domains, the discovery process has become challenging.
Existing discovery techniques rely on two broad ap-
proaches, supervised and unsupervised. In supervised ap-
proaches a training dataset for a domain is prepared where
natural language based queries are annotated with specific
services by domain experts [9] or from crowd [14]. There are
a few challenges associated with supervised approach. First,
the annotation process is time consuming and laborious.
Second, annotations can be misleading and biased. Unsu-
pervised approaches overcome the limitations by considering
techniques such as keyword matching, entity relationship ex-
traction, and extraction of topics and clusters. The keyword
matching approach [11] and the entity relationship extrac-
tion [17] technique match users’ natural language queries
syntactically with service descriptions, thereby, ignoring the
semantic similarity and may lead to false positives. For
example, consider a user wants to find services related to
bus. The response returns a service Message Bus that has
no relation to the user’s actual intent, i.e., transport services.
For another instance, approaches based on the extraction of
topics and clusters [13], [6], [5], [16] capture the semantic
similarity between a user query and service descriptions.
However, these approaches rely on mashups that explicitly
capture the relationships between services – mashups may
not be available for all services.
In the paper, we propose a novel unsupervised framework
natural language to service APIs (NL2API) that recommends
services based on users’ queries and service descriptions.
The approach does not rely on mashups since they are not
available for new services especially if they have not been
used in any application. Thus, NL2API acts as a bootstrap
service recommender for new services. NL2API identifies
relationships between services by extracting communities
based on their semantic similarities of intents. Each com-
munity is represented as a tree like structure where the
the internal nodes represent hierarchical intents and the leaf
nodes represent topics. A topic refers to a common theme
across a group of services, whereas an intent refers to an
abstraction over topics, or other intents. NL2API identifies
the communities which are most closely related to the
users’ queries and then recommends top-k services with a
confidence score from these communities using the latent
semantic index (LSI) technique.
The proposed framework was evaluated on two datasets:
Programmable Web and API Harmony [8]. The results show
that the proposed framework can retrieve relevant results
without external information such as mashups. The results
also show that the proposed framework outperforms baseline
approaches especially when it was applied to large dataset
such as Programmable Web.
 II. R ELATED W ORK
The existing approach for service recommendation is
based on supervised approaches or unsupervised approaches.
A. Supervised Approaches
Su et al. [14] propose a framework for web service discov-
ery. Since the framework relies on a supervised approach,
Su et al. creates a training data based on inputs from crowd.
The training data consists of mappings between informal
commands based on natural language and service calls. Su
et al. provide an intermediate representation or a canonical
command for each service in terms of HTTP verb, resource,
return type, required parameters, and optional parameters.
Then, the crowd is employed to paraphrase canonical com-
mands. They use LSTM for training and evaluation of
their annotated dataset. Their approach has limitations. One,
collecting informal commands for a large repository of
services can be challenging. Two, with inclusion of new
domains, the intermediate representation has to be refined.
Kalia et al. [9] provide service discovery in a service
catalog based on natural language requests from users.
Their approach relies on multiple labels associated with
each service. For example, a service can be associated with
a category, task, and an action. To prepare the training
data Kalia et al. consider IT change requests and annotate
them with multiple labels based on a service catalog. For
building their model they use the classifier chain multilabel
classification and improve the classification accuracy by
providing a feedback based on extracted parameters from
users’ requests. Their approach is limited to a specific
catalog of services. For new catalogs, the annotation process
can be cumbersome and needs extensive domain knowledge
for annotation.
B. Unsupervised Approaches
Xiong et al. [17] provides a framework that considers
users’ personalized requirements and semantic graphs to
generate recommendations for users. The semantic graph is
constructed based on natural language API descriptions. A
semantic representation is captured as triples in the form
of (X, α, Y) where X and Y are entities and α is the
string of words that intervenes between X and Y. Relation-
ship are constructed by extracting typed dependency trees
from descriptions. Using the relationships, they recommend
top-k recommendations. Their approach does not discover
the semantic relationships between services based on their
descriptions. Thus, the approach may fail to recognize the
users’ intent from their queries.
Rahman et al. [11] provide an approach that uses crowd-
sourced knowledge for recommendation of APIs. Their
approach exploits the relationships of keywords extracted
from Stack Overflow questions and answers with APIs
based on their descriptions. One important limitation of their
approach is that the approach rely on external dataset such as
Stack Overflow. We assume that Stack Overflow might not
provide data for a new domain. Another limitation of their
approach is that they consider keyword based matching that
ignores the semantic similarity of users’ queries with service
descriptions and relationship between services.
Xie et al. [16] provide an approach that takes input as
mashups (related services) and clusters them into groups
based on their textual descriptions. For the clustering they
used the K-medoids algorithm. From the clusters, services
are recommended based on user’s requirements. The ap-
proach has several limitations. One, the approach relies on
mashups which may not be present for a new domain. Two,
the approach clusters mashups rather than services. Thus, the
identified clusters may not correctly depict the relationship
between services.
Gao and Wu [5] provide a framework to recommend
a set of services based for mashup composition based on
users’ text inputs. In addition to matching services with
users’ requirements they emphasize on the quality of service
recommended. For recommending top-k services, rather than
using traditional topic modeling techniques, they create a
mapping between service descriptions and mashup compo-
sition. The supervise technique first learns the relevance
between service descriptions and mashups. Then, they apply
clustering using K-means to recommend top-k services.
They used the learned model from the supervised approach
to improve clustering. Clearly the approach rely on mashup
composition to cluster services that may not be present for
new domains.
Samanta and Liu [13] provide an approach for service
recommendation that applies Hierarchical Dirichlet process
on services to generate topics. The approach considers
mashups as its input to generate topics. A cosine similarity
between the topics generated from services and mashups is
computed to generate a candidate list of services. Further,
the usage history of services in mashups is used to produce
top k-services from the candidate list. Hao et al. [6] propose
an approach to refine service descriptions for a specific
query. The approach relies on mashup descriptions that
provide application scenarios for services and such descrip-
tions provide additional information for services that can
be used to reconstruct the original service descriptions. For
reconstruction of service descriptions, Hao et al. consider
Latent Dirichlet Allocation to extract topics from mashup de-
scriptions and the query given by the user. Both approaches
rely on mashups that may not be present for all kinds of
domains.
III. T HE NL2API F RAMEWORK
In the section, we present an overview of our proposed
framework NL2API. Our framework has three components:
a preprocessing unit, a community extractor, and a query
matcher. In the following sections, we present each of the
components in greater detail.
A. The Preprocessing Unit
The preprocessing unit takes natural language service
descriptions as input and preprocesses them using a pipeline
that removes punctuations, applies part-of-speech (POS)
tagger, removes frequent, infrequent, and corpus specific
stop words. In our framework, we removed infrequent words
that were in three or less number of service descriptions and
frequent words that appear in more than 10% of the service
descriptions. For corpus-specific stop words, we heuristically
select the low-quality topics and remove the most frequent
terms in the topics. Table I shows examples of such words.
Table I
E XAMPLES OF MOST FREQUENT, INFREQUENT, AND CORPUS - SPECIFIC
STOP WORDS EXTRACTED FROM SERVICE DESCRIPTIONS IN
P ROGRAMMABLE W EB .
Type Words
Frequent service, time, method, platform, developer, api, web,
words system, site, respone, functionality
Infrequent phyloinformatic, carregistrationapi, myhurricane,
words atomicreach, hotukdeal, polypeptide, cooladata,
eventcategorie, changeavatar, zubhium, americascup
Corpus- management, feature, message, solution, integration,
specific app, tool, function, client, request, xml, way, provider
stopword
B. The Community Extractor
A community represents a hierarchical structure that rep-
resents a set of topics related to a common intent. Within
each community (Figure 1), the structure can be constructed
as a tree where the internal nodes represent intents and leaf
nodes represent topics. Each topic is inferred from a list of
service descriptions. Note that the depth of the hierarchical
tree indicates intents at different levels of abstraction. Intents
closer to the root node represent abstract and general intents
(e.g., investment) whereas intents closer to topics represent
detailed and specific intents (e.g., stock exchange). Topics
that share common ancestor have higher degree of relevance.
For example, in Figure 1, the topics “music”, “movie”,
and “video” are closely related. Similarly, topics “location”,
“route” and “map” are related.
To extract communities, we utilize representative topic
modeling and clustering techniques, and devise solutions that
integrates them into our desired tree structure. Accordingly,
we propose three solutions. One, the baseline approach
that uses a topic modeling technique where each learned
topic forms a single community. Note that the baseline
approach does not construct a hierarchical intent tree. Two,
the bottom up approach that first infers topics, then applies
the community detection method to identify communities.
Three, the top down approach that learns a latent high level,
low dimensional vector representation for each service and
then groups similar vectors into community using k-means
Figure 1. Examples of communities constructed from service descriptions
cluster approach. Finally, a topic modeling method is applied
to services within each cluster. A detailed descriptions are
provided in the following paragraphs.
1) Topic Modeling (TNMF): Existing topic modeling
techniques consider the well-known algorithms such as La-
tent Dirichlet Allocation (LDA) [1] and non-negative matrix
factorization (NMF) [3]. However, empirical studies [12],
[19] have shown that their effectiveness greatly reduces
when the length of documents decreases. Considering most
service descriptions are short sentences, we consider the
non-negative matrix factorization based on term correlation
matrix (TNMF) [18] as the method has been proven to out-
perform LDA, NMF, probabilistic latent semantic analysis
(PLSA), graph regularized NMF [3], and symmetric NMF
[10] for multiple datasets containing short texts.
Table II
E XAMPLES OF TOPIC AND TOPIC WORDS EXTRACTED FROM SERVICE
DESCRIPTIONS IN P ROGRAMMABLE W EB .
Topic Topic Words
Topic 1 train, bus, commuter, timetable, transport, de-
parture, transit, journey
Topic 2 wind, humidity, weather, precipitation, baromet-
ric, temperature, irradiance, forecast
Topic 3 student, class, course, education, school, college,
instructor, assignment
Topic 4 election, committee, voter, vote, ballot, voting,
candidate, privilege
TNMF [18] is a method designed exclusively for discover-
ing topics especially from short texts. Traditional topic mod-
eling NMF decomposes the term-document matrix, which
indicates the term occurrence in a given document. However,
for short texts, the term-document matrix can be extremely
sparse, which prevents the model to learn reliable topics.
TNMF tackles the problem by making an assumption that
terms co-occurring frequently are most likely to be relevant
to a common topic. For example, if the terms address
and zipcode co-occur in several documents at the same
time, they are more likely to talk about a common topic
location. Thus, TNMF learns reliable topics by decomposing
term-correlation matrix [18] instead – the matrix does not
suffer from the scarcity problem, in fact, the size of terms
remain stable even when the number of document grows.
An example of topic and topic words extracted from service
descriptions in Programmable Web is shown in Table II.
2) Topic Modeling (TNMF) + Louvain’s Community De-
tection (LCD): The second approach first learns topics using
TNMF, then uses Louvain’s Community Detection (LCD) Figure 2. The network is created using topics and services. Communities
[4] to extract communities using a constructed network that are derived by applying the LCD approach.
models pairwise association between topics and services.
Specifically, the network is constructed based on a learned
topic-document matrix V. In the network graph, nodes change in the modularity larger than a predefined threshold
represent services (or documents) and topics. An weighted is possible. For a more detailed description of LCD, please
edge is formed between the topic and services if the cor- refer to [2]. In table III we show examples of communities
responding entry in V is non-zero; the weight is the value extracted from service descriptions in Programmable Web
in that entry. Figure 2 shows an example of such network. using TNMF and LCD.
After the network is completed, LCD is applied to extract
Table III
communities from it. E XAMPLES OF COMMUNITIES , TOPICS , AND TOPIC WORDS EXTRACTED
LCD is a greedy optimization algorithm where the value FROM SERVICE DESCRIPTIONS IN P ROGRAMMABLE W EB USING TNMF
AND LCD.
to be optimized by LCD is a scale value called modularity
(ranging between -1 to 1). Modularity measures the density
Communities Topics Topic Words
of edges inside communities to edges outside communities.
Optimizing modularity would lead to the best possible Community 1 Topic 1 mining, block, bitcoin, currency
Topic 2 equity, market, stock, exchange
partition of a community. Iterating through all possible
Topic 3 sell, audit, buy, priority
partitions is highly impractical. Thus, based on a heuristic
approach with the following two phases repeated iteratively Community 2 Topic 4 gene, research, metabolic
Topic 5 sequence, protein, amino, acid
until convergence:
In the first phase, all nodes are assigned to their own
communities. The change of modularity is computed for a 3) LSTM based Autoencoder + K-means clustering: The
moving node i from its own community to its neighboring third approach explores a language generation task: training
community j (only contains node j at the beginning), using a Long short-term memory (LSTM) based autoencoder to
the following heuristic function build an embedding for the service descriptions and then,
h P +k  P +k 2 i decodes the embedding to reconstruct the original sentence.
in i,in tot i The embedding can be seen as a high-level low-dimensional
∆Q = − −
2m 2m (1) representation of original service descriptions. The K-means
hP  P 2 ki 2 i
in
− tot
−( ) clustering method is used to partition the service embedding
2m 2m 2m into different clusters (or community).
P
where in is the sum of all the weights P of edges inside LSTM [7] has been frequently used on language genera-
the community that i is moving into. tot is the sum tion tasks like machine translation or parsing. The power of
of all weights of the edges in the community. ki is the LSTM comes from its ability to capture local dependencies
weighted degree of i, ki,in is the sum of weights of the between words: neighboring words are combined to express
links between i and other nodes in the community. Once the a particular meaning. It is a special type of Recurrent Neural
value is computed for all communities node i is connected Network (RNN) that can avoid the vanishing (and exploding)
to, node i can be combined into the community that resulted gradient problem. An LSTM unit contains three major com-
in the greatest ∆Q. the process is applied sequentially to all ponents Forget, Input and Output. The components interact
communities in the network. with each other to control how information flows. Forget
In the second phase, the nodes in the same community determines what information from previous memory cell
are grouped together – the entire community is treated as is expired and should be thrown away. Input component
a node, thus a new network is constructed. Then, the first determines what information is new and requires updating.
phase is reapplied. the procedure will be repeated until no Output component is an activation function that filters the
value from a memory cell.
The LSTM based autoencoder shown in Figure 3 creates
a compact representation of service descriptions. The au-
toencoder is a neural model that consists of two LSTMs.
One LSTM encodes a sequence of words into a fixed-length
vector representation. The output of the last LSTM unit
generates an embedding of service descriptions. The second
LSTM decodes the representation into another sequence of
symbols. The encoder and decoder of the proposed model
are jointly trained to maximize the conditional probability
of a target sequence given a source sequence. In the ex-
ample shown in Figure 3, the target sequence and source
sequence are the same since autoencoder aims at obtaining
a representation that fully captures the content of its input
– a good representation is to retain a significant amount of
information about the input.
Figure 3. LSTM based autoencoder
After the low dimensional representation is obtained,
we apply the K-means clustering method to partition the
services into K specified clusters (or community). Finally,
TNMF is applied to services within each community to
extract topic per community. An example is demonstrated
in Table IV.
Table IV
E XAMPLES OF COMMUNITIES , TOPICS , AND TOPIC WORDS EXTRACTED
FROM SERVICE DESCRIPTIONS IN P ROGRAMMABLE W EB USING LSTM
AUTOENCODE , K-M EANS , AND TNMF.
Communities Topics Topic Words
Community 1 Topic 1 currency, trading, market
Topic 2 product, sale, order, price
Topic 3 advertiser, publisher, revenue
Community 2 Topic 4 sequence, protein, structure
Topic 5 analysis, search, dataset, result
Topic 6 database, gene, record, protein
C. The Query Matcher
The query matcher recommends a list of services based on
the query from a user. Based on the given query, we check
communities and underlying topics the query is related to.
To check the relatedness, we compute a matching score
M(query, topic) between a query and a topic, i.e., the sum
of the similarity scores between each word in a query
and each top keyword in a given topic. We compute the
word similarity using well-known Wu-Palmer score [15] that
calculates the relatedness of two word sense by computing
the path length from the least common subsumer of the two
word sense to the root node that captures the most specific
concept they share.
The matching query is given as follows.
X X
M (query, topic) = Similarity(u, v) (2)
u∈query w∈topic
Similarity(u, v) = max(W u − P almer(ui , vj )) (3)
where u are words in query and w are topic terms.
ui and vj are word senses. For a given query, we select
topics with the highest M(query, topic) and intents that
are the ancestors of the identified topics. If topics under
consideration belong to different communities and do not
share a common ancestor, then the parent nodes of the topics
are identified. The services under the identified intents are
returned as candidates.
Given a set of services as candidates, Latent Semantic
Index (LSI) is used to calculate a matching score between
the candidate services and that query. Services with highest
matching score is recommended to the user.
IV. E VALUATION
We evaluate NL2API on two datasets: Programmable Web
and API Harmony.
Programmable Web is one of the largest repository
to discover and search application programming interface
(APIs) or services to be used for web or mobile applications.
Everyday new services are being added to Programmable
Web to make them available to developers. In addition to
service, Programmable Web provides mashups data, source
code to use APIs, a list of frameworks, libraries, and SDK
each of which is connected to API-driven development.
API Harmony provides a catalog of web services. API
Harmony helps a developer to better understand the APIs
through executable examples and by providing the historical
usage of a specific service. API Harmony has three features.
One, developers can use API Harmony to discover services.
It uses past search information to recommend services for a
new search query. Two, developers can learn about services
based on their documentations, community discussions, and
sample codes. Three, API Harmony provides developers
with examples and historical usages on how to integrate a
specific service into their applications.
Table V shows the count of services we consider for
each dataset. For Programmable Web, we develop a crawler
to mine services along with their names, descriptions and
categories. For API Harmony, we collect the services directly
from its authors. Each service has a name, description, url
paths and parameters.
Table V
C OUNTS OF SERVICES IN EACH DATASET.
Dataset # of Services
Programmable Web 12372
API Harmony 1180
We evaluate the NL2API framework in terms of coverage.
Coverage measures the number of relevant recommendations
returned by a specific approach i.e., (1) TNMF, (2) TNMF
and LCD, and (3) LSTM and K-means. A recommendation
is relevant if it is an element of the ground truth.
A. Experiment Design
To evaluate NL2API, we need ground truth. As all of
the proposed approaches are unsupervised, determining the
ground truth is challenging. Traditional approaches rely on
mashups where we can get the information about relation-
ship of one service to another. However, NL2API is agnostic
of mashups.
One way to collect the ground truth is to expose NL2API
to crowd and ask them to provide queries and recommen-
dations. However, the approach is quite limiting as crowd
workers may not have the expertise to provide relevant
queries and interpret recommendations. To overcome the
limitation Su et al. [14] provide an intermediate represen-
tation for a service for crowd to generate queries, however,
scaling the approach for multiple datasets with large number
of services is infeasible. Thus, we provide an experimental
agnostic to crowd and can be applied to any unsupervised
approach as follows:
Step 1: Generate a set of intents. For each dataset, we
randomly select N descriptions. We hire two investigators
with sufficient and similar technical expertise to read N
descriptions and produce N intents they both agree on. An
intent here refers to a description that concisely specifies
the type of service a user is looking for. Both investigators
must agree on the semantic interpretation of all of the intents
produced.
Step 2. Determine ground truths for each intent. For each
intent in N , a ground truth result set is constructed. Ideally,
the ground truth set is formed by manually evaluating every
services in a dataset and selecting all services that match
the intent. However, the is infeasible due to the large size of
the dataset, especially Programmable Web. The investigators
hence approximate the exhaustive search by conducting a
series of keyword based searches on the data set: first, the
investigators independently formulate likely keyword queries
based on an intent, and collect the results that matches the
intent. Then, they further generate new keywords based on
the description of services identified. The process is then
iterated until there are no new service matching the intent
and all search queries have been exhausted. Finally for each
intent, the investigators merge their independent set of valid
services to produce a final ground truth set for each intent.
Step 3. Generate queries for each intent. We hire a third
investigator to produce queries for each intent identified in
Step1 . Queries generated can be of two types, one based on
a set of keywords and another based on semantics or well
formed natural language phrases. For example, to search
speech to text services, one can search using keywords
such as speech to text and tools for speech to text or
using semantics such as get speech to text services and find
platforms that convert speech to text.
Step 4. Compute coverage. We ask the third investigator to
provide the queries consisting of keywords and semantics for
each intent. We run the queries utilizing a search approach
in NL2API to obtain a result set under that specific method.
The intersection of the result set with the ground truth set of
that intent yields a valid recommendation set. And the ratio
of the valid recommendation set over the ground truth set of
the intent yields the coverage measurement for the specific
search approach for that intent.
Coverage is therefore a normalized ratio capturing how
well a search method worked for a given intent. We
multiplying the ratio by 100 to obtain a percentage score.
• Keywords Coverage. The coverage measures
performance of keyword queries.
• Semantics Coverage. The coverage measures
performance of semantic queries.
• Total Coverage. The coverage measures performance
of overall recommendation from both keyword and
semantic queries.
V. R ESULTS
We apply the steps given in Section IV-A to the datasets
from Programmable Web and API Harmony. For each
dataset we randomly generate five (N =5) service descrip-
tions. For each description, the two investigators come
up with five intents. For example, for a description such
as Picwing provides easy photo sharing, bridging online
sharing and offline sharing . . ., the agreed intent could be
put photo online for sharing. We provide the intents to
the third investigator who generates queries and search for
valid recommendations using all of the three approaches. For
example, for an intent such as put photo online for sharing,
the third investigator can suggest a query based on keywords
such as image sharing and semantics such as share photos
with my friends.
For each approach and each query in an intent, we ob-
tain the recommendations corresponding that approach and
that intent, and compute three coverage scores: keywords,
 LSTM+K-Means
TNMF+LCD
TNMF
(a) total coverage (PW).
LSTM+K-Means
TNMF+LCD
TNMF
0 10 20 30 40 50 −5
(d) total coverage (AH).
Figure 4. Boxplots comparing different approaches applied on datasets from Programmable Web (PW) and API Harmony (AH).
semantics, and total coverage. Figure 4 shows the boxplots
of coverage performance for each approach by averaging the
coverage scores across 5 intents. the exercise is conducted
for Programmable Web (PW) and for API Harmony (AH).
Total Coverage. For the Programmable Web dataset we
find that the median of the total coverage for the approaches
TNMF + LCD (10.41%) and LSTM + K-means (9.12%) out-
perform the baseline approach TNMF (9.02%). However, for
the API Harmony dataset we find that the median of the total
coverage for the approach TNMF (28.35%) outperforms the
approaches TNMF + LCD (10.75%) and LSTM + K-means
(8.48%). The observations show that for a large dataset
such as Programmable Web extracting communities could
be beneficial in discovering closely related services. For a
small dataset such as API Harmony, a baseline approach
is sufficient to discover relevant services. the suggest that
with increasing size of datasets, our approach is beneficial
to developers.
Keyword Coverage. For the Programmable Web dataset,
we find that the median of the keyword coverage for the
approaches TNMF (8.33%) and TNMF + LCD (8.33%)
outperforms the LSTM + K-means (3.33%) approach. For
the API Harmony dataset, we find that the median of
the keyword coverage for the approach TNMF (30.79%)
outperforms the approaches TNMF + LCD (6.32%) and
LSTM + K-means (5.09%). From the observations we find
that for searching services based on keyword queries topic
modeling approaches based on TNMF can provide relevant
recommendations. Additionally, for the Programmable Web
dataset we find that the standard deviations (std) and vari-
ances (var) for the TNMF + LCD (std = 8.18%, var =
(b) keywords coverage (PW). (c) semantics coverage (PW).
10 20 30 40 50 −5 15 30 45 60
(e) keywords coverage (AH). (f) semantics coverage (AH).
66.93%) and LSTM + K-means (std = 7.5%, var = 56.56%)
approaches are significantly larger than the TNMF approach
(std = 0.83%, var = 0.69%). The observation suggests that
output based on communities constructed could vary largely
from one query to another i.e., for some queries we can get
a higher coverage than others. However, the may not hold
for the TNMF approach. For the API Harmony dataset, we
find the standard deviation (std) and variance (var) for the
TNMF approach is larger than approaches TNMF + LCD
and LSTM + K-means. The observation suggests that a
larger dataset can be crucial to improve the performance
of the recommendations.
Semantic Coverage. For the Programmable Web dataset,
we find that the median of the semantic coverage for
the approach TNMF + LCD (12.5%) outperforms the ap-
proaches TNMF (9.7%) and LSTM + K-means (9.7%).
For the API Harmony dataset, we find that the median of
the semantic coverage for the TNMF approach (23.22%)
outperforms the TNMF + LCD (13.02%) and LSTM + K-
means (7.97%) approaches. From the observations we find
that for a larger dataset such as Programmable Web the com-
munities extracted using LCD could be useful in discovering
relevant services based on semantic queries. In addition, we
observe that for the Programmable Web dataset, the standard
deviations (std) and variances (var) for the TNMF + LCD
(std = 13.2%, var = 174.7%) and LSTM + K-means (std
= 5.8%, var = 33.7%) approaches are significantly larger
than the TNMF approach (std = 4.49%, var = 20.17%). The
observation suggests that based on communities constructed
we can get higher coverage for certain queries. It may not
hold for the TNMF approach. For the API Harmony dataset,
 we find the standard deviation (std) and variance (var) for
the TNMF + LCD approach is larger than the TNMF and
LSTM + K-means approaches. The observation suggests that
for semantic queries, generating communities from topics
are beneficial for better recommendations irrespective of the
data size.
VI. D ISCUSSION
We present NL2API a framework for recommending top-k
services based on three different approaches. The approaches
are agnostic of mashups or pre-specified mappings from
natural language queries to services. Thus, NL2API can act
as a quick bootstrapping recommender for datatsets that do
not have mashups or prior history of services’ usage. We
evaluate NL2API on two different datasets, Programmable
Web and API Harmony. Our findings show that for larger
datasets such as Programmable Web the TNMF + LCD and
LSTM + K-means approaches provide significantly higher
coverage than the baseline TNMF approach. However, for
smaller dataset such API Harmony, the baseline approach
can produce better results than the other approaches.
In the future, we plan to further expand our investigation.
One, we plan to create a larger repository of services. Two,
we plan to provide a more efficient query matching tech-
nique such as word vector to discover relevant communities
generated by the TNMF + LCD and the LSTM + K-means
approaches. Three, we plan to perform an extensive user
study to further evaluate the effectiveness of our approaches.
R EFERENCES
[1] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet alloca-
tion. Journal of machine Learning research, 3(Jan):993–1022,
2003.
[2] V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefeb-
vre. Fast unfolding of communities in large networks.
Journal of statistical mechanics: theory and experiment,
2008(10):P10008, 2008.
[3] D. Cai, X. He, J. Han, and T. S. Huang. Graph regularized
nonnegative matrix factorization for data representation. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
33(8):1548–1560, 2011.
[4] A. Clauset, M. E. Newman, and C. Moore. Finding com-
munity structure in very large networks. Physical review E,
70(6):066111, 2004.
[5] W. Gao and J. Wu. A novel framework for service set
recommendation in mashup creation. In Proceedings of
IEEE International Conference on Web Services, pages 65–
72, Honolulu, jun 2017. IEEE.
[6] Y. Hao, Y. Fan, W. Tan, and J. Zhang. Service recommenda-
tion based on targeted reconstruction of service descriptions.
In Proceedings of IEEE International Conference on Web
Services, pages 285–292, Honolulu, jun 2017. IEEE.
[7] S. Hochreiter and J. Schmidhuber. Long short-term memory.
Neural computation, 9(8):1735–1780, 1997.
View publication stats
[8] IBM. Api harmony, 2017.
[9] A. K. Kalia, J. Xiao, M. F. Bulut, N. Anerousis, and
M. Vukovic. Cataloger: Catalog recommendation service
for it change requests. In Proceedings of International
Conference on Service Computing, pages 545–560, Malaga,
nov 2017. Springer.
[10] D. Kuang, C. Ding, and H. Park. Symmetric nonnegative
matrix factorization for graph clustering. In Proceedings
of the 2012 SIAM international conference on data mining,
pages 106–117. SIAM, 2012.
[11] M. M. Rahma, C. K. Roy, and D. Lo. RACK: Auto-
matic api recommendation using crowdsourced knowledge.
In Proceedings of 23rd IEEE International Conference on
Software Analysis, Evolution, and Reengineering, pages 349–
359, Klagenfurt, 2016. IEEE.
[12] M. Rosen-Zvi, T. Griffiths, M. Steyvers, and P. Smyth. The
author-topic model for authors and documents. In Pro-
ceedings of the 20th conference on Uncertainty in artificial
intelligence, pages 487–494. AUAI Press, 2004.
[13] P. Samanta and X. Liu. Recommending services for new
mashups through service factors and top-k neighbors. In Pro-
ceedings of IEEE International Conference on Web Services,
pages 381–388, Honolulu, jun 2017. IEEE.
[14] Y. Su, A. H. Awadallah, M. Khabsa, P. Pantel, and M. Ga-
mon. Building natural language interfaces to web apis.
In Proceedings of the 26th ACM International Conference
on Information and Knowledge Management, pages 1–10,
Singapore, nov 2017. ACM.
[15] Z. Wu and M. Palmer. Verbs semantics and lexical selection.
In Proceedings of the 32nd annual meeting on Association
for Computational Linguistics, pages 133–138. Association
for Computational Linguistics, 1994.
[16] F. Xie, J. Liu, M. Tang, D. Zhou, B. Cao, and M. Shi.
Multi-relation based manifold ranking algorithm for api rec-
ommendation. In G. Wang, Y. Han, and G. M. Pérez,
editors, Proceedings of 10th Asia-Pacific Services Computing
Conference, pages 15–32, Zhangjiajie, 2016. Springer.
[17] W. Xiong, Z. Wu, B. Li, Q. Gu, L. Yuan, and B. Hang.
Inferring service recommendation from natural language api
descriptions. In Proceedings of IEEE International Confer-
ence on Web Services, pages 316–323, San Francisco, jun
2016. IEEE.
[18] X. Yan, J. Guo, S. Liu, X. Cheng, and Y. Wang. Learning
topics in short texts by non-negative matrix factorization on
term correlation matrix. In Proceedings of the 2013 SIAM
International Conference on Data Mining, pages 749–757,
San Diego, 2013. SIAM.
[19] W. X. Zhao, J. Jiang, J. Weng, J. He, E.-P. Lim, H. Yan, and
X. Li. Comparing twitter and traditional media using topic
models. In European Conference on Information Retrieval,
pages 338–349. Springer, 2011.