Evaluating SuperMusic: streaming context-aware mobile music service

Arto Lehtiniemi
Nokia Research Center Visiokatu 1 FIN-33720 Tampere +358 50 4835451

arto.lehtiniemi@nokia.com

ABSTRACT
The digital music offering and consumption is rapidly increasing and more and more portable devices are offering music listening capabilities. Mobile network bandwidth is also increasing but streaming the music isnt yet common. Most of the mobile music players still rely on locally stored music. They also lack possibilities to request proper music recommendations from online music catalog. This paper presents an evaluation of SuperMusic: prototype streaming context-aware mobile music service. SuperMusic is bringing a large 200 000 track test music catalog available everywhere and including two different music recommendation methods in addition to social interaction of the users. A five week user trial was conducted with 42 users to find out the feasibility of the SuperMusic concept, to find out critical design issues and to evaluate the performance of the music recommendation methods compared to random recommendations. SuperMusic concept satisfied 97% of the users and it was seen as a potential killer application in the music domain with some modifications. The evaluation brings up many suggested design improvements including rebuilding the user interface, improving the streaming of music and the recommendation algorithms, in addition to further enhancing social interaction between the users.

1. INTRODUCTION
In everyday life, people consume increasing amounts of music with mobile devices. There is a big effort to transfer and maintain a music collection between different devices. In many cases the suitable music is not available at the moment when it would be needed. Also, the increasing amount of online music brings up a challenge of finding new music that pleases the listener and adapts to his/her musical taste. SuperMusic is a prototype streaming context-aware mobile music service. The goal of the evaluation of the SuperMusic player was to find out the feasibility of the SuperMusic concept and see in practice how well the current infrastructure supports streaming high quality music with SuperMusic in feasible way to offer instant access to a large online music catalog. The quality assessments of our two music recommendation methods were also being investigated. SuperMusic player includes an improved version of our content based music recommendation algorithm and a early prototype version of a context based music recommendation algorithm. These algorithms try to find new music for the users out of the online music catalog with different criteria: similarity and the context of the user. The evaluation also addressed aspects like user acceptance, goodness of different SuperMusic player features and user interface. A great number of improvement suggestions for the future development were collected and analyzed.

Categories and Subject Descriptors

H.5.5 [Information interfaces and presentation (e.g., HCI)]: Sound and Music Computing

General Terms
Human factors

2. RELATED RESEARCH
Common technologies behind current music recommendation systems can be categorized as collaborative filtering systems, content-based recommender systems, or hybrid recommender systems. Content-based recommenders can be further divided into music signal analysis and manual content classification systems. Hybrid recommendation systems combine some features from these different recommendation technologies [15], see [1] for a general overview of these different systems. Collaborative filtering systems are designed to collect specific usage and rating information, in order to provide recommendations to users. In commercial implementations, mainstream music is often widely represented. Manual content classification systems such as Pandora can be highly precise and provide novel new recommendations, but their drawback is the huge effort required

Keywords
Usability, HCI, evaluation, music, playlist, recommendation, player, streaming, context, service

314

to describe the songs one by one. [3][8] Automatic content-based recommender systems have varying performance and precision, as documented in the literature, but their benefits are clear: automatic recommenders are cheap, they are able to recommend more obscure artists, and they do not necessarily need a usage database to support the recommendations. Some of the newer recommendation designs include different levels of context sensitivity to enhance the recommendations [9]. Most of the context sensitive solutions require user input to determine the context e.g. mood. In [10] the importance of contextual information is highlighted: Environment related features offer great potential as meta-data and can be a valuable resource in automatic generation of music playlists. Mobile context can be challenging to the context aware services, because the tasks and goals can be quite different than static indoor position in addition to the amount of other unpredictable external factors that can be encountered. The social acceptability of new context-aware technologies are dependent on how well they fit into the daily routines of everyday life [14]. In [4] it was shown that the attitudes of the users towards location-aware services were quite positive. There are not many examples available of streaming mobile music services including recommendations and social features. MyStrands social player [7] is a music player also available in the mobile domain including music recommendation and social networking. Social features of the MyStrands music player include discovering new friends who have played the same songs as you and finding out the musical affinity against another members of the system. The users music consumption is limited to local music files. Push!Music is a prototype wireless peer-to-peer mobile music player with sharing capabilities [2]. This player enables users to push songs to each others mobile terminals. In their user study ([2]) it was shown that the participants had two main reasons for sharing the songs: dissemination where the sender liked the songs and wanted others to hear it as well and recommendation when the sender sent a song that he believed the receiver would appreciate. In this study the participants knew each other well and there were no results what would have happened between users that didnt know each other beforehand. Interaction design of a recommendation systems has been studied in [13] pointing out that the users do not mind giving a little more input to the system in order to receive more accurate recommendations. Also the perceived usefulness and ease of use of the recommender system correlated positively with the presence of longer descriptions of individual items. The transparency of the recommendation system increased when the recommended items had descriptions attached indicating why they were given as a recommendation. General design principles derived in [3] indicate that minimizing user effort such as time to register, download and get recommendations in addition to the high quality of the recommendations are the key element in designing a successful system. On the other hand regarding the user interface, a bad user interface will decrease the overall rating of the recommendation system [12][13].

3. PROTOTYPE STREAMING MOBILE MUSIC SERVICE (SUPERMUSIC)

SuperMusic is a streaming context-aware mobile music service bringing a 200 000 track music database available to a mobile phone by streaming the music over the air through cellular network. SuperMusic is a Symbian client that is build on top of the S60 music player (Figure 1). The client interacts with a dedicated server build with Python. The communication between the server and the client is done using SOAP protocol [11].

Figure 1. Main player view of the SuperMusic client In addition to ordinary mobile music player functionality SuperMusic creates a personalized and context sensitive music listening experience by offering two different music recommendation methods: similar sounding recommendation and context based recommendation combined with collaborative filtering technologies. The latter is our first attempt to recommend suitable music for the users current situation by detecting the location (GPS and cell-id) and time. The users have a chance to vote the recommended music and let the system learn their musical preference and opinion concerning song similarity. SuperMusic includes different music searching methods in addition to the music recommendations. The users are able to search music with the current artist and album name. There is also a free text search available. These music recommendations, voting and searching features are easily accessible from the options menu of the main player view in addition to finding out why a certain song was recommended. By selecting the recommend to this situation or one of the different search options the server returns a playlist of music that automatically starts playing. The My Music button on the main screen opens music library, which contains the rest of the most important SuperMusic player functions (Figure 2). Networking with friends is also possible with SuperMusic. Songs can be sent to friends as recommendations from the options menu of the main view. The listening history and favorite songs from friends can be viewed and listened with one click (Figure 2). All listened songs can be also sent to the users Last.FM profile [5].

315

test users had in average 440 albums worth music in their music collection. Almost half of the users had experience with other music recommendation systems. Half of the users were placed into a control group and they received and voted on random recommendations during the whole trial without knowing about it. This was done to benchmark the performance of the two different recommendation systems.

4.3 Field trial structure

All the users took part in the 5 week field trial where they had the chance to use SuperMusic player as their main music listening device. After the trial period a 1-2 hour interview session was arranged with each user. Most of the interviews were held face to face but few had to be performed over the phone because of the different physical location of the users. Figure 2. Music library view of the SuperMusic client The music library also enables the user to see automatically detected situations, music recommendations from friends, personal information e.g. song count and access the locally stored music with basic S60 music player categorization. The interview was semi-structured and covered most of the SuperMusic functions in detail. Also a questionnaire about SuperMusic functionality was given to the users to obtain more quantitative data to support the answers in the interview. In the end, test subjects were rewarded for their effort of using the system and participating the interview session with movie tickets. In addition, gift certificates were given for the three most active listeners. A leader board of top listeners was published every week during the trial period to motivate the users to be active.

4. FIELD TRIAL 4.1 Test arrangement

There were two different pilot stages before starting the user trials. In the first stage developers were using the SuperMusic player actively and reporting encountered problems. Most critical flaws were fixed immediately before the user trial. In second stage of piloting the SuperMusic player was installed for a group of 20 users and they had the chance to try out the system and give feedback. Once the trials started the software release was frozen to provide a consistent test platform. A trial web page was created where the users could download the SuperMusic client to their mobile phone and view a getting started guide for basic guidance. The getting started guide consisted of detailed installation instructions and basic usage of the software illustrated with screenshots. A background questionnaire was sent to trial participants via email before the trial period to gather more detailed information of the participants and their music consumption.

4.4 Data analyzing methods

The user trial produced both qualitative and quantitative data. Qualitative data was collected with the interviews and quantitative data was obtained from the SuperMusic server in addition to a short questionnaire. A qualitative content analysis method was used to process the interview data. This was done in several different phases: converting the data, browsing through the data, categorizing the data and interpreting the data.

5. RESULTS
The field trial provided great amount of feedback and ideas for further improvement. The results are divided to different categories covering the most important aspects of the SuperMusic player. Most of the results can quite easily be generalized to be applied to other similar mobile music players in terms of user acceptance, features, user needs and design related issues.

5.1 User interface issues

5.1.1 Ease of use
SuperMusic player is easy to use according to 73% of the users. It must be taken into account that the users were Nokia employees with prior experience on mobile phone music players and above average capabilities in using new mobile applications. Still, there were some parts of the UI that were not automatically clear. Many of the users had difficulties in understanding the situations concept: What is my situation? How can I determine my situation?. The my situations -menu had just numbers, a set of automatically detected situations that were unclear to the users. Also the terminology regarding situations was considered hard to understand. This was one of the few features of SuperMusic

4.2 Participants
There were 42 users selected for the 5 week trial. When recruiting users there were no restrictions concerning age or education level. The main attribute was music consumption: active music listeners interested in using SuperMusic as their primary music device for the trial period. All of the users are working for Nokia and they used N95 mobile phones for the trial. The average age of participants was 30 years. The selected test user group consisted of 26 men and 16 women with 47% playing an instrument as a hobby. Most of the test users were very active music listeners with 88% listening to music every day including in average 32% of all music listened with a mobile device. The

316

player that test subjects sought help from the Getting Started manual and from online help. About 50% of the test subjects had some or severe problems in finding the received recommendations from friends. It was said that there should be an indicator on the main view for new recommendations such as there is an icon of received SMS in the phone. Currently there is a pop-up dialog when a new recommendation has been received which can be unintentionally clicked away quite easily. Another approach would be that the New recommendation received dialog could have a button mapped to it enabling direct access to the recommendations folder. Searching functions were inconsistent in the UI: some started playing the music automatically after the search and the free text search displayed a search dialog. Many users though that with e.g. Artist search they could type in a search string also. Current implementation searched for similar artists to what was currently playing. Because of the automatic playing some had difficulties in understanding that the system searched a full playlist as a result instead of the one song that started playing. The separation of own music and streamed music is not clear in the UI. Sometimes the users had difficulties understanding which functions of the player apply to the online music and which are for local music files. This problem is due the fact that the player contains all the features of the old S60 player and the new online music features have been added on top of it without redesigning the UI completely.

Many test subjects had several classes from current categorization that they never use. There was a lot of interest in browsing through the online catalog since it is a natural part for many in selecting the music. Very common scenario is that the user does now know what to listen and wants to browse though the music collection to get an inspiration. There were several development ideas how to enable browsing the online catalog. One convenient approach is to browse the online catalog with a home computer/web browser and gather the playlists there. Created playlists could be sent to mobile phone with just one click. To reduce the amount of content when browsing with a mobile device, genre and similar artist information could be used. Artist, album and alphabetical categorization could be done under a certain genre to keep the amount of content small enough. Also if there are some artists that the user likes, there could be a possibility to browse though similar artists to keep the amount of results feasible in mobile browsing.

5.1.4 Help & guidance

The getting started manual was checked or read by 83% of the users. In general the Getting Started manual was considered to be a good idea because of its short length and descriptive screenshots. This was the main source of help and explanation about the features for most of the users. Help from the SuperMusic player was used only by 8% of the users to find answers to problems: 78% of the users never looked the help from the SuperMusic player. 14% of the test subjects just tried though the different options and also opened help but were not seeking anything special from there. Within this test group it was considered that the help should contain more advanced information instead of basic player guidance. Tooltips were thought to be the most convenient way of offering instant help.

5.1.2 UI improvement
It was said that the music player should look more similar to the newer Nokia N-series music player: basic features for playing should be mapped to the cursor keys like in the new platform music player to enable easier access to main functions. This would also leave more space for additional information on the screen. Another benefit from this approach is that the player could be used without looking to the screen. It was brought up by many users that the amount of visualization in the UI should be increased to make the player more appealing. Many users also wished for configurable themes from the player in addition to displaying the album art for the music. There should be a progress bar indicating the status of song buffering for online music. This would give better feedback to the user about what is going on and makes the wait to feel shorter. Some thought that as the recommendations are among the key features in this player that dedicated buttons should be brought to the main screen instead of options menu. Same applies to voting of the recommendations.

5.2 Streaming the music

Successful streaming is the key element for SuperMusic player. Test subjects were asked if they encountered problems with streaming music and Figure 3 shows the results for the 5 week trial period.

5.1.3 Music categorization and browsing

There was some users that made contact during the trial asking whether is would be possible to browse through the online catalog somehow without using the free text search. This inspired to get in more details finding out how the music should be categorized, and are there any convenient ways of browsing a large online catalog. It seems that many users have different patterns how to browse though their music. Most common within this test group are browsing the artist names and artists based on genre classification.

Figure 3. Problems streaming the music during the trial. Most of the problems in streaming were related to the network connection. Typical cases were network switching from 3G to 2G and vice versa that caused pauses in the playback. SuperMusic is

317

not able to stream music properly without a 3G connection or WLAN although sometimes Edge connection with full reception is enough for seamless playback. Users that are in the Most of the time ok group (45%) had only little troubles with the streaming and could use the system without worrying too much about the technical issues concerning streaming. Users in this group had typically small pauses in the playback every now and then, especially when they were on the move. Users inside the Ofter severe problems group (37%) said that their music listening was reduced because of the network related problems. Some had to change back to the platform player or iPod when they were e.g. driving car, jogging and travelling in a bus. Adaptive buffering was seen as a solution to help avoiding pauses in the playback. It was suggested that when there is good network coverage when starting to play a song the player could buffer the whole song (or even next songs from the playlist) in advance to offer more seamless experience. Also the device could have a cache to store some of the successfully streamed songs in case there are network problems. This approach would guarantee access to some music in case of losing the network coverage and not having any local songs stores in the device. The online music was streamed in 96kbps Advanced Audio Coding (AAC). All of the test subjects were happy with the sound quality of the stream using the N95 default headset. Some of the users connected the SuperMusic player to a home stereo system while when listening at home. It was suggested that when there is enough network capacity also music videos could be streamed in addition to the song to enhance the listening experience. In total, test subjects listened 15 627 songs (tracks listened more than 50% of total track length). Also 9857 tracks were listened less than 50%. In total the test participants were dealing with 25 484 tracks during the 5 week trial period. Six of the most active users were listening more than 1000 songs each during the trial. Some of the test subjects streamed over 4GB of data. Users that encountered problems with streaming were not so active towards the end of the trial. Also Christmas deadlines were closing and some were very busy during the last couple of weeks of the trial.

There wasnt much difference between random recommendations and context-aware recommendations. There are several reasons affecting to this. The system should learn the user preferences quicker. SuperMusic had very little user data before this trial. The system needed input from the users in terms of voting to learn about the context and was only able to give context dependent recommendations during the last week of the trial. Before the last week of the trial the Recommend to the situation feature relied mainly to simple collaborative filtering due to the lack of user data resulting worse performance as expected. This is an important design issue for further development how to teach the system to learn users context much faster so that the user does not get bored with the recommendation quality and stop using it before the system performs in optimum way. Some test subjects also had difficulties in understanding the concept of situation. Here are two user comments to clarify this: If I'm being honest, I didn't understand the concept of situation... I don't know if there is music for my situation at all? and I have no idea how this system should have adapted to my situations but apparently it didnt. Most common situations to use this feature were identified in the interviews as: home, work, bus, car, jogging, evening and gym. There were practically no others. Some of the users would have wanted to guide the system e.g. defining a mood that could have helped the system to perform better. Also it was said that the user could have given some guidance of his musical taste upon first registration in order to speed up the learning process. Although the performance of the context-aware recommender wasnt very good, 78% of the test subjects considered it as a good and interesting concept and would be interested using it if the performance could be enhanced: If it would work it could be quite good. But in the beginning I didn't understand what was it about and I lost interest of that.

5.4 Recommendations based on song similarity

The Search similar function enables the users to search similar sounding songs as the currently playing song using automatic music signal analysis methods. The quality of the Search similar algorithm was compared to the random music given as recommendations for the control group based on the users voting. Users were able to vote after each recommendation whether the recommended song is similar compared to the seed song (Figure 5).

5.3 Recommendations to the situation

The quality of the context-aware recommendations were compared to the random music given as recommendations for the control group based on the users voting. Users were able to vote after each recommendation whether the song is suitable for the current situation (Figure 4).

Figure 4. Results of situation recommendations compared to random music recommendations of the control group.

Figure 5. Results of similar sounding recommendations

318

5.6 Recommendation criteria

The difference between good recommendations in test group compared to the random group is 0,478. The observed z-test value for this difference is 21,914, which is significant with a (onetailed) p-value < 0,0001. The Search similar algorithm provided 63% of similar sounding recommendations according to the users votes. The performance could still be improved by applying a genre filter to the results. It was discovered that most of the test subjects tend to evaluate the similarity of a song based on the genre in addition to what it sounds like and the language of the vocals. One alternative would be to let the user define which genres should not be included in the recommendations at all to leave out the musical styles that the user does not like at all. Also restricting the results according on the release year of the song might benefit the recommendation quality. The users expressed several comments concerning the different eras of the recommended songs when the release years were very far apart. In [6], there is a more in-depth analysis of the previous version of the Search similar algorithm. The importance of finding similar songs in this kind of music application was appreciated: 75% of the users considered Similar recommendations to be very important feature as it is, some describing it to be the most important feature of the SuperMusic player. Recommendation criteria are displayed in the voting popup dialog after requesting a recommendation. Seeing the recommendation criteria was seen as important by 63% of the test subjects. Many thought that it is nice to get an explanation of why a song is being recommended, especially if the song is very different from the seed. Current reasons included e.g. if a friend has recently listened to the song or that the recommended song is very similar to the seed song. The explanations should be more descriptive and detailed to give a better understanding of the functionality of the recommendation algorithms. Users had many good ideas concerning what should be displayed as recommendation criteria. Also some of the test subjects that were not interested about the feature contributed some nice feedback and improvement ideas for the recommendation criteria that might make it more appealing (Table 1) Table 1. Recommendation criteria alternatives Recommendation criteria alternatives Because you like Metallica you might also like this Because you have listened this many times you might also consider this You looked so beautiful this morning that I will play you this song Since its raining outside this is the choice for you New coming artist, This artist is a new arrival to the charts This is modern Motown (if the user has liked Motown) This is Norways Metallica This song has a good guitar track and was chosen for you because of that This song is currently on the Radio Energy Top10 list This is same genre and era than the song you were just listening to 70% of people how listen mostly same music as you have liked this song This song has the same singer/guitarist/producer as some songs you like This is an original/cover version of a song you like This is the most popular artist in SuperMusic within this genre

5.5 Voting the recommendations

When listening to a recommended song, after 5 seconds there will be a voting popup encouraging the user to give a vote for the song. The voting dialog disappears automatically after 10 seconds if there was no reply. Voting the recommendations was found easy by 76% of the users. The voting was clearly visible but a deeper analysis is needed to find out why the percentage isnt higher. Some users thought that the voting dialog came too early and disappeared too soon to tell the opinion about the song even it is possible to get the voting dialog again from the options menu. Also some users would have wanted to skip the dialog but there was no button marked for that. The test subjects didnt realize that the select button will skip the dialog there should have been a label on top of it. Again there was some variation in this question: some were happy with the current implementation and some even said it is the only good way to get people to vote for the recommendations. A couple of test subjects were against the pop-up dialog. There were two options for voting, good and bad, satisfying 60% of the users. It was said that having more options would make the voting more complicated and the user to think too much about it which could eventually affect to the voting frequency. Second most popular option was to have third voting option. It was said that the voting status for the current song should be visible in the UI e.g. Not yet voted. It was pointed out that the recommended playlist should change automatically based on the user voting of the playlist items e.g. if the user votes bad for two first tracks, the contents of the playlist could automatically change to adapt the user preference.

5.7 Friends Social networking

5.7.1 Finding friends
One third of the users had friends in the SuperMusic system. Most of them had only one friend. Finding friends was considered to be difficult if you didnt know someone beforehand. At least for Finnish people it is not an easy task to add an unknown person as your friend without knowing anything about that person. In the cases where people knew someone beforehand, finding friend from the service was considered to be quite easy with the implemented search functions. The users usually found out about other people from the trial email address list. This was the most common case.

319

Those who had friends shared some recommendations but very often didnt notice that there was also option to see and listen what music the friends had listened to. This was thought to be a good feature but a bit hidden in the system.

a better understanding why the song was recommended to him. It was also wished by some that it would be possible to send recommendations to several friends at once.

5.7.2 Promoting social interaction

The users were asked how the system could promote social interaction and what kind of information they would like to see relating to other users. In general most of the test subjects (90%) thought that some kind of social interaction within a music player would be beneficial. The player should search known friends automatically using contact data from phonebook and various other services e.g. Facebook and Messenger. One suggestion was to bring My neighborhood option to SuperMusic player where the users could see people that listen to similar music as neighbors and interact with them. The system could also use neighbors music consumption as bases for some recommendations. There could be different communities and channels for certain kind of interests e.g. I just love the 80s or Just progressive. Members of that community could interact and send recommendations to the community channel in a similar way as sending messages to a chat room. Currently users are able to listen to songs that their friends have listened recently. It was suggested that it would be possible to listen what other people are currently listening e.g. tuning in to your friends channel and sharing the music listening experience. There should also be more information available about the users of the system including e.g. profile, picture, musical taste, musical profile analyzed by the system and how active they are with SuperMusic player. It was mentioned that it would be also nice to see your own listening history in similar manner as you can view your friends recently listened tracks. Also adding Search by profile to searching options when finding friends and it would display users with most similar music profiles. Compatibility with other social networking sites e.g. Facebook, Last.FM should be taken into account in future versions in terms of sending messages from SuperMusic to people that are using other services. In addition sending recommendations to people that are not using SuperMusic player was seen as a nice feature. Those people could listen to the song with their browser.

5.8 Searching for music

Searching was used very much during the trial and it was seen as one of the most important features in SuperMusic player. The user was able to search more songs from the same artist or album for the currently playing song in addition to a free text search that browsed through the artist, album and song name tags of the online music. Users pointed out that it would be a good idea to be able to search music also with genre. There could be more advanced search with configurable options e.g. genre, release year. Also it could be beneficial to be able to limit the free text search to only one search field. The amount of results was also seen too small. The current search returned a playlist of 20 results and users thought that the number of results should be increased to be able to browse more results. It was also mentioned that being such an important function the free text search is too far away in the application menus from the main view: Options->Search->Free text search. Some statistics of most listened songs and artist were generated from the SuperMusic server and send as email to the trial users. The users really liked the idea and were hoping that Supermusic could provide different statistics with links to listen to the songs e.g. most popular rock songs, most listened this week, todays specials, this months top 10 artists, highly rated etc. Some of the users had an account in Last.FM service and highly appreciated the possibility to send information of the listened tracks there.

5.9 Other technical issues

The prototype system implementation wasnt fully stable and caused occasional crashes and reboots of the phone. Almost everyone had some pauses during the playback every now and then depending of the network coverage. The battery consumption was considered as the biggest showstopper with SuperMusic player. Some even said that the fully charged battery didnt last more than 30 minutes in worst case. For many, the battery lasted between around 2 hours of continuous usage. Battery consumption and pauses during the playback could also be helped with adaptive buffering. When bigger partition of the song would be buffered in advance the actual streaming buffer could be smaller and have great impact on battery consumption. Browsing through the SuperMusic UI caused small interruptions to the playback which was seen as annoying. Also the current implementation of SuperMusic didnt support fast forwarding the streamed media. It was said that that functionality would be important to have in future versions.

5.7.3 Sending and receiving recommendations

Sending recommendations was considered to be easy. The same didnt apply on receiving recommendations: 21% of the test subjects said they never received any recommendations. A couple of recommendations were sent to all of the users so this indicated that 21% of the users never found the recommendations from the player. Also, 36% of the users said that had some difficulties in finding the friends recommendations from the system. Currently when receiving new recommendations from friends, senders name and timestamp is displayed followed by a link to the song name that can be listened There is a clear need to attach a short message or a pre-defined tag/symbol to describe the recommendation in more detail. This was brought up by 77% of the users. Some added the possibility to reply to the message while only 16% of the users were satisfied with the current approach having only a link to the song. By adding a message to the recommendation would give the recipient

5.10 Users evaluation of the system features

The users were asked to rate different features and aspects of SuperMusic with numbers from 1 to 10. The highest grade indicates that the SuperMusic player is excellent in this area and the lowest grade indicates that the SuperMusic player performance is very poor in this area (Table 2).

320

7. REFERENCES
Table 2. Users evaluation of different SuperMusic aspects Metric Ease of use Features Look and feel Novelty Lasting appeal Functionality/Stability Average grade (Scale 1-10) 7,3 7.8 6,6 7,8 7,6 6,5 [1] Adomavicius, G., Tuzhilin, A. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Trans. Knowledge and Data Engineering, 17(6), 2005. [2] Jacobsson, M., Rost, M., Hkansson, M., Holmquist, L. E. Push!Music: Intelligent Music Sharing on Mobile Devices. Adjunct Proc. of UbiComp 2005, Tokyo, Japan, 2005 [3] Jones, N., Pu, P. User Technology Adoption Issues in Recommender Systems. In Proceedings of Networking and Electronic Commerce Research Conference (NAEC2007), Lake Garda, Italy, October 18-21, 2007 [4] Kaasinen, E. 2003. User needs for location-aware mobile services. Personal Ubiquitous Comput. 7, 1 (May. 2003), 7079. DOI= http://dx.doi.org/10.1007/s00779-002-0214-7 [5] Last.fm, http://www.last.fm [6] Lehtiniemi, A., Seppnen, J. Evaluation of automatic mobile playlist generator. In Proceedings of Mobility Conference 2007., pages 460-468, Singapore Polytechnic, Singapore, September 10-12, 2007 [7] MyStrands, http://www.mystrands.com [8] Pandora, http://www.pandora.com [9] Park, H-S., Yoo, J-O., Cho, S-B. A Context-Aware Music Recommendation System Using Fuzzy Bayesian Networks with Utility Theory. In Proceedings of Fuzzy Systems and Knowledge Discovery (FSKD2006), Xian, China, September 24-28, 2006 [10] Reynolds, G., Barry, D., Coyle, E. Towards a Personal Automatic Music Playlist Generation Algorithm: The Need for Contextual Information. In Proceedings of 2nd Audio Mostly Conference: Interaction with Sound., pages: 84 - 89, Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany, 2007 [11] Simple Object Access Protocol (SOAP), http://www.w3.org/TR/2000/NOTE-SOAP-20000508/ [12] Swearingen, K., Sinha, R. Beyond algorithms: An HCI perspective on recommender systems. In Proceedings of ACM SIGIR Workshop on Recommender Systems, New Orleans, LA, USA, 2001 [13] Swearingen, K. and Sinha, R. Interaction Design for Recommender Systems. In Proceedings of the Conference on Designing Interactive Systems (DIS'02). London, England, 2002 [14] Tamminen, S., Oulasvirta, A., Toiskallio, K., and Kankainen, A. 2004. Understanding mobile contexts. Personal Ubiquitous Comput. 8, 2 (May. 2004), 135-143. DOI= http://dx.doi.org/10.1007/s00779-004-0263-1 [15] Yoshii, K., Goto, M.., Komatani, K., Ogata, T., Okuno, H.G. Hybrid collaborative and content-based music recommendation using probabilistic model with latent user preferences. In Proceedings of Int. Conf. Music Information Retrieval (ISMIR), Victoria, Canada, 2006.

Users self evaluation shows that features and novelty are the strongest parts in SuperMusic player whereas look and feel & functionality/stability are the weakest links. Although the scores are quite close to each other, they support well the textual findings from this trial. The overall feeling of the SuperMusic performance during the trial period was positive with 89% of the normal trial participants and 58% of the participants belonging to random group. After the interview session the random group users also got the chance to try the SuperMusic with the real recommendations. The feedback was very positive. Despite of the problems with SuperMusic the user satisfaction is great for the whole concept: 97% of the users liked to use the system and 95% would like to continue using the system.

6. CONCLUSIONS
SuperMusic player was seen as a great concept that should be further developed. Almost all (97%) of the users were satisfied to the SuperMusic as a concept and 95% said they would like to continue using SuperMusic player. There were some technical problems that still need to be solved. Most important issues concern battery consumption and buffering the playback in advance when there is good network coverage so that there would be fewer pauses when streaming online music. Recommendations should be further developed, especially context-aware recommendations. Search similar performance was satisfying with 63% of good recommendations. To further enhance the performance, recommend similar should incorporate genre and release year information. Currently the performance of recommendations to situation was poor and need to be improved to the next version including faster learning of the user preferences. Recommend to the situation was seen as potential concept by 78% of the users. The UI would also need many small improvements to offer better user experience. When working with a large online music catalog, easy browsing of the catalog and efficient search functions are among the key issues of designing an appealing music service. Users also highlighted the importance of social interaction within the music player.

321