Evaluation of

Recommendation System
Group 2
Prasad Hegde
Kameswaran Rengarajan
Venkata Gangala
Shu Wang
Zi Yang

Executive summary
Peoples life is tightly related to Internet. They get used to listening to the music online, purchasing online,
and also reserve their travel services online. Because of these trends, E-commerce is becoming more and
more attractive and prevalent. Assume that recommendation systems are able to predict customers
potential needs in high accuracy, companies will increase customer satisfactions and make more revenues
through the assistance of recommendation system. In this project, we studied the mechanism and
algorithm of recommendation system of jd.com, an E-business platform in China, and the effects on
companys sales, satisfactory and profit.

Project motivation
According to SEC Filling Form of Pandora Media Inc., as of June 2015, Pandora had 79.4 million active
usersi (P-PANDORA MEDIA INC, FORM 10-Q, 2015). For Amazon, the CEO Jeff Bezos said that the
company has more than 244 million active users. And nations No.1 online travel businessExpedia Inc.
increased its sales/revenue from 3.3 billion in 2010 to 5.76 billion in 2014 ii (Annual Financials for
Expedia Inc., 2015). By considering the reason of increased popularity and profitability for online
business, the evaluation of recommendation systems should be taken into account since a substantial
number of successful online services attached importance on developing and implementing this system.
For instance, when you searched Taylor Swift in Pandora Radio, it would like to list recommend artists
such as Katy Perry and Selena Gomez to you as well. When you purchased a pillow through Amazon, it
will also list recommended products such as sheet and pillowcase for you. And when you booked flight
tickets to Orlando, Expedia would like to recommend rental car, hotel, and theme park ticket to you as
well.

Introduction
By definition, recommendation system or recommender system are a subclass of information filtering
system that seek to predict the rating or preference that a user would give to an item iii (Francesco
Ricci, Lior Rokach, Barcha Shapira, 2011). Since lots of companies using recommendation system for
their E-commerce, its necessary to analyze how the system works and judge which kind of
recommendation systems will bring real benefits to companies. Evaluating recommendation systems
requires a definition of what constitutes a good recommender system, and how this should be measured. In
reality, developing a recommendation system needs investment of money, technology and human capital.
However, not all recommendation systems could become the beneficial tool for companies to make more
profits.
In our project, our team will describe operation principle of recommender system. We will discuss two
algorithmic techniquescontent-based filtering and collaborative filtering for computing
recommendations. Also, this project will mention a hybrid approach, which has been developed to avoid
certain limitations related to content-based and collaborative filtering. And we will discuss methods to
improve the effectiveness of recommendation system for online business.

Description
The emergence and popularity of the Internet gives customers a lot of information, but with the rapid
development of the network bring substantial growth, users can not absorb really useful information faced
with large bundle of it. The efficient use of information reduces, which is the so-called information
overload issues.

Personalized recommendation system is a measure to solve with a lot potential. It recommend to the user
information and commodities based on the characteristics of the their interest and purchasing behavior. A
good recommendation system will not only provide users with personalized services, but also build close
relationship with users and make them dependent.
With the continuous expansion of the scale of e-commerce, trade and the number of fast-growing species,
customers take a lot of time to find what they really want. This process of searching bunches of
information and products will undoubtedly result into losing customers. To solve these problems,
Jingdong commodity recommendation system came online. Personalized recommendation system is an
advanced business intelligence platform built on a large scale data mining, collaborative filtering
algorithm, and analysis of user behavior, to help Jingdong provide personalized decision support and
information services to its customers. The system has been online for half year, already contributing 10%
increase in order at both PC terminal and mobile terminal.

BI Techniques
A complete recommendation system should consist of the following three parts: recording part to record
customer purchase behavior, model analysis part to study user preference, and algorithm recommendation
part based on algorithm efficiency. Recommendation system or recommender system could be divided
into several types regarding the algorithm it uses: collaborative filtering system, content-based system,
network-based system, and hybrid system. Jd.com selects algorithm depending on which algorithm is
more efficient in a specific event.
Recall Models
Recall model, namely to obtain candidate data sets, is generally realized from three dimensions: historical
behavior-based recall, user preference-based recall, and geographical-based recall.
Behavior-based recall recommends related or similar goods according to user purchase. Lots of systems of
this kind recommend similar items based on a user's browsing history, but Jingdong takes a further step,
setting up a dividing line of buying behavior. When the user has purchased a commodity, Jingdong will
recommend related or similar goods of types and uses selected, instead of repeating recommending the
same product. For example, the system will recommend Kindle protective cover for the Kindle buyers, not
another model of Kindle. Meanwhile, system will recommend soap, shampoo and items in this kind in a
purchasing period.
User preference-based recall includes two important elements: user portrait and multi-screen connection.
Combining brands, target crowd, commodity price and user behavior such as clicking, purchasing, and
marking, Jingdong portraits or classifies users, to determine the long-term recommendation category.
Shopping for a new era of mobile habits, Jingdong also pays attention to different terminal types based on
the user ID and MEI data to a more portrait of the user.
Geographical-based recall divides the entire map into multi-grid and uses statistical data to give results.
Take Beijing as an example, goods purchased in SanLiTun area are more cards and water, while in
primary and secondary schools, data concentrate in socks, racks and so on.
Model Efficiency Analysis
Three models mentioned above contain multiple sub-models, such as online-related, online-similar,
offline-related, and offline-similar, recent best sale, etc. For a specific user, which model to use becomes
an issue. The system will assess the number of hits and forwarding rate.

The Search Department Director of JD.com Shangkun Liu gave an important conclusion: model based on
real-time user behavior has a higher efficiency, such as online-related products and online-similar
recommendation. In order to achieve better results, Jingdong uses algorithm that melts several models,
rather than using a single model to do it.
Efficiency increases mainly from two models recently being used: Recently Click and Recent Concerns
model, and Add-to-Cart model. The former will show recently viewed items on user home page, and the
latter is to recommend goods already in shopping cart but have not been purchased.
Content-based Filtering
Content-based filtering is one of the most basic approach taken by recommendation system. This approach
constructs a recommendation on the basis of users behavior or preference. Problem of information
overload exists for a long time. And researchers are working with technology that allows automatic
categorization and recommendation of information to a user based on the users personal preferences iv
(Herlocker, 2000). Content-based filtering approach would like to track users historical searching records.
For instance, when you search Wildest Dream or leave comments on MV about Taylor Swift under
YouTube usually, content-based filtering can use the historical information to identify and provide similar
content (interviews of Taylor Swift and Taylors Live Show). The key for this methods is to recommend
items that are similar to those that a user liked in the past.
Collaborative Filtering
Collaborative Filtering is another basic approach taken by recommendation system. In E-commerce, this
is the method and process used to match data of one user with data for similar users, based on purchase
and browsing patternsv (collaborative filtering, 2015). For example, Facebook will recommend some new
friends to you if these people have common friends with you. And LinkedIn would like to recommend
people with similar background and experience to you. For another example, when you select a book in
Amazon, the web also gives you some recommendation and says: people who purchased this book also
viewed The key for this method is to categorize similar people into groups and assume that people in a
same group are likely to obtain similar preferences or interests.
Hybrid Approach
Hybrid approach, we can simply understand by its name. Its the method that combine content-based
filtering and collaborative filtering approaches. Hybrid approach can implemented into four ways
combing separate recommender systems, adding content-based characteristics to the collaborative
approach, adding collaborative characteristics to the content-based approach and developing a single
unifying recommendation approach. For instance, Netflix uses hybrid approach. It recommends movies by
analyzing customers historical preference and the rating they gave (content-based filtering) as well as by
comparing the watching and searching habits of similar users (collaborative filtering). vi

Advantages of Recommendation systems and approaches

Prediction using user activity data
Recommendation system uses the past activity of users, browsing history to predict the future and
recommend apt product to customers. Since it is using past activities, the ability to understand customers
activity and trends are more accurate and relevant. All the data collection is done by using real time
software therefore the changing customer shopping habits can also be easily identified.
Improve traffic on website and engage customers

Customers get more attracted by the personalized web content, product recommendations. This helps in
improving the customer traffic and also improves the chances of customers adding more products to the
cart. At the same time it helps the website owners improve their profit, add new products to the existing ecommerce site and also satisfy customer needs, delve more deeply into the product line without having to
perform search after search.
Reduces Workload and Overload:
Using the business intelligence tools and techniques the built recommendation software is helpful in
reducing the time to manually gather huge amount of data and process it. Recommendation systems
automats the existing data and helps overload in IT staff and reduces time for analyzing data.
Provides reports and advices to improve:
Identification of current market trends is a key for success of a business. In recommendation systems
generating reports on the trends and future is an integral part of a personalization system. It gives the
clients daily and accurate reports which helps the client to make a decision or helps in understanding the
present needs to change.

Disadvantages of Recommendation systems and approaches:

Limited Content Analysis:
In order to effectively use recommendation systems we need to have large junks of data to understand and
create trends. For example: Amazon, Google, Wal-Mart are the major customers which can efficiently use
recommendation systems. New customer or small-scale businesses cannot make use of recommendation
systems efficiently.
Difficulty to setup
Recommendation systems are very intense. Data driven and requires high technical capabilities to set up
and use it effectively. It is difficult to set up and use it without having the specific skill set required for the
recommendation system.
Wrong Interpretation
Recommendation systems sometimes may be wrong and create a disatisfaction among customers because
of the misinterpretation, proposing a wrong product to the customer. In extreme cases
recommendations can be offensive.

Concern other than accuracy (Privacy, Diversity)

The online recommendation system has become an immense success that many people around the world
are using to buy their products such as electronic goods, cosmetic items, etc. The simple idea of buying
the products sitting at the home and ordering at leisures pace just by the click of the mouse has captured
the imagination of the customers of the world. But, this facility also has its own disadvantages like any
other product or offering. We will look into the details of what are the limitations these recommender
systems have.

One of the major concerns of recommendation systems is the privacy of the customers. Tailoring of
individual needs of the customers and recommending products is the main purpose of these systems. Some
users may not like the benefits given by these personalization techniques. This is due to the fact that this
personalization may come by collecting data which may has done involuntarily and also the
recommendation systems may have shared these data with some unknown third party vendors. People will
be more concerned about the recommender system accessing their personal information such as phone
numbers and email ids. Similarly the less important information such as the age, gender, marital status
will also be accessed by the recommender system. Customers will be seriously concerned about these
crucial information being shared by the recommender system.
The recommender system also collects demographic data which can be merged with individual identities
such as email addresses and social security numbers and also the preferences of customers such as
purchase history, product ratings. In a similar way, data which are left by the users online without their
knowledge. These data are accumulated by the algorithms automatically to identify the user preferences
for future recommendations. On the other hand, users give direct feedback to the recommender systems
that states their preferences such as product ratings and movie reviews. These two different methods have
been utilized frequently for better prediction accuracy and efficiency in recommender systems.
The direct feedback given by the users to the recommender systems have been preferably used in showing
items to users based on location. This approach gave users more flexibility, control and advantages and
people were more willing to employ it in their searches. Whereas, the information left by the people online
without their knowledge were looked upon as looking into their data without their approval which could
severely impact the utility of the recommendation systems. This could give the recommender system a
negative review and impact the sales of the company. There are ways in which the recommender system
could improve their efficiency by not compromising on the privacy of their customers. One such method
is that gives the users the freedom to give the recommender system their personal information which can
be read by the service providers. Thus in this method the privacy of the customers is not compromised as
they willingly give their information to the recommender system.
In another method, a data hiding mechanism can be developed in which the users have the ability to hide
their true information and identities and use another name to conceal their true identity such that the
recommender system does not get access to their personal information and their privacy can be kept intact.
In yet another way, the assurances can be given to the users by the recommender system that their personal
information can be controlled. These are the ways in which the privacy can be maintained by the
recommender system so that the personal information such their identities, address, email ids are not
accessed by the recommender system.
The diversity of the recommender systems in many ways helps in the wide variety of results and its
ambiguity. In the recommender system, new users join all the time and in many cases old and frequent
users move out. Similarly, new data and products get added and also old ones which are not in use get
deleted frequently. Due to this the content of the recommender system also changes very rapidly. For
example in Amazon, new products, offers and discounts get updated by the companies, due to which the
customer base of the company changes which leads to a very diverse field of customers. Similarly, Netflix
also updates new movies which get released every week. This also makes the diversity of company to
increase as new users may join to get the access for the updated movies.
Previously, the recommender systems used ratings for the products it showed to the customers but for
better utilization, the recommender system should take into consideration the preferences of the users and
give the best results to the customers. Hence, by diversifying the results given to the customers by the

recommender systems we can reduce the risk of promoting unwanted search results and products.
Similarly, the recommender system should provide a wide variety of results to the users that change with
time and also which suits the preferences of the users.

Improving Recommendation System

The process of searching a product or a much-needed content online with plethora of product and content
available is a tedious task; with so many options it is very difficult to keep the customer interested. This is
where recommendation system comes in. The main purpose of this system is to provide users with valid
content that interests the users. Although the existing systems are pretty well equipped to a certain extent
to provide the users with the content of their interest there is still a lot of room for improvement. One of
the methods that can bring about the improvement is Conversational Recommendation System.
Conversational Recommendation System provides assistance to the users in their decision-making by an
interactive process; these systems are designed to mimic the human interaction where preferences are
extracted out of having a conversation rather than providing them up front. CRS basically assists users in
their content or product seeking by explicitly or implicitly acquiring the user preference (explicitly
through querying and implicitly through users activity logs) to present users with different choices to
make their decision. Although there are different ways this can be achieved Conversational
Recommendation System typically follows a rigid approach by reformulating the interactive query. There
can be variations to CRS i.e. some strategies would require user inputs and once relevant information is
obtained the system recommends content to the users. On the other hand some strategy will make some
suggestions to the users upfront and at a later point in time will query for the user preference and
personalize any recommendations to the users in the future.
Although CRS provides a substantial improvement to the online recommendation system this too has
some shortcomings. In the strategies mentioned above the scenarios are typically hard coded i.e. same sets
of predefined actions will be executed despite of the possibilities of other actions.

Summary
Evaluating recommendation systems is critical in business since companies profits, customers shopping
experience and satisfaction are highly depend on recommender system. Effective recommendation system
will bring benefits to the company. However, useless recommendation system would waste both financial
and human capital. In an attempt to keep competitive advantage in E-commerce, its essential for
companies to evaluate the effectiveness and investment return about the developing or existed
recommendation systems.

Reference

i P-PANDORA MEDIA INC, FORM 10-Q, http://services.corporate-ir.net/SEC.Enhanced/SecCapsule.aspx?

c=227956&fid=10233283
ii Market Watch--Annual Financials for Expedia Inc.,
http://www.marketwatch.com/investing/stock/EXPE/financials
iii Francesco R, Lior R, Barcha S: Introduction to Recommender Systems Handbook
iv J.L.Herlocker, Understanding and Improving Automated Collaborative Filtering Systems
v http://www.webopedia.com/TERM/C/collaborative_filtering.html
vi http://www.csdn.net/article/2015-04-15/2824476,
https://www.uie.com/articles/recommendation_systems/http://www.researchgate.net/publication/221267362_Imp
roving_recommender_systems_with_adaptive_conversational_strategies