eChallenges e-2014 Conference Proceedings

Paul Cunningham and Miriam Cunningham (Eds)

IIMC International Information Management Corporation, 2014
ISBN: 978-1-905824-46-5

Best Practices for Validating Research

Software Prototypes - MARKOS Case Study
Alicja LASKOWSKA1, Jesus GORROOGOITIA CRUZ2, Pawe KDZIORA1,
Ilaria LENER3, Bartosz LEWANDOWSKI1,
Cezary MAZUREK1, Massimiliano DI PENTA4
1
Pozna Supercomputing and Networking Center,
ul. Noskowskiego 12/14, 61-704 Pozna, Poland
Tel: +48 61 8582001, Fax: +48 61 8585954,
Email: {alicja, kedziora, bartekel, mazurek}@man.poznan.pl
2
ATOS Spain SA, Calle de Albarracn, 25 28037 Madrid
Spain, Email: jesus.gorronogoitia@atos.net
3
T6 Ecosystems srl, Via Genova 30, Rome, 00184 Italy
Tel: +39 06 47823286, Fax +39 0647882798, Email: ilener@t-6.it
4
University of Sannio, Piazza Guerrazzi, 1, 82100 Benevento, Italy,
Email: dipenta@unisannio.it
Abstract: A common approach in research projects is to plan the evaluation and in
particular the validation phase In the last months of the project. When done late in
the project, validation cannot provide valuable inputs to the development of the
research idea, because there is no space to bring about changes, and very often ends
up as confirmatory tests. Software validation can and should occur since creating the
first lines of code until the completion of the system in test. Thus, we propose the
approach where research project can adopt the agile paradigm used by startups
release early, release often, not only to development, but also to validation:
validate early, validate often, to improve the quality of projects results on one hand
and to be in line with potential users on the other.

1. Introduction
The term of validation refers to the evaluation of the prototype from users perspective, and
aims at answering the question: Does the system meets users needs? The validation
process can provide valuable feedback in research projects when working with users on an
ongoing basis. There is no sense in building a technically correct system that is not useful
for the intended users.
The increasing importance of user collaboration and involvement in driving innovation
is recognized. Moreover, validation of research results becomes extremely important in the
context of Horizon 2020. The work programme explicitly states that focus areas should
support activities that meet the demands from consumers, policy makers and other users of
the results [1].
However, a common approach in research projects is to perform the evaluation and in
particular the validation phase during last months of the project. In such a case, very often,
the validation takes a very soft form of acceptance tests, focusing on single aspects of the
prototype, and makes the purpose of the activity ceremonial [2]. There are many reasons to
postpone, limit or even omit validation. Ambitious plans and limited resources for testing in
research projects result in an anxiety about showing immature system and expecting a lot of
criticism. When done late in the project, validation cannot provide valuable inputs to the
development of the research idea, because there is no space to bring about changes.
Copyright 2014 The Authors www.eChallenges.org Page 1 of 9
Therefore software evaluation, and in particular validation, can and should occur since the
early development stages until the completion of the system in testing phase.
This paper describes the validation approach defined and used in the MARKOS (the
MARKet for Open Source) European Project [3], a FP7-ICT-2011-8 STREP project, which
aims to realize a service and an interactive application, which provides an integrated view
on the open source projects available on the web. The project focuses on functional,
structural and license aspects of a software source code [4]. Many services available on the
Web often provide textual-based code search. MARKOS differs from such services in the
sense that it provides the possibility to browse the code structure at a high level of
abstraction, what facilitates the understanding of the software from a technical point of
view. It also shows relationships between software components released by different
projects, giving an integrated view of the available open source software at a global scale.
Last, but not least, MARKOS is able to track potential legal issues resulting from license
incompatibilities, and provides argumentations for these issues to support developers in the
search for alternative solutions to their problems. Along with research objectives, one of the
MARKOS aims is to involve end-users in order to allow to practice its results in scenarios
coming from industrial and open source communities.

2. Objectives
The objective of this paper is to briefly present an approach for validating research software
prototypes. This paper outlines the approach, highlighting its advantages and lessons learnt
based on MARKOS experience. Holistic testing and validation requires a lot of effort. This
is a very wide area where different approaches and standards are provided. However, such
comprehensive and complex methods, are very time and effort consuming, and therefore
are not feasible to be applied in a research project with restricted resources. The overall
goal of the paper is to convince the reader that the presented approach is lightweight
enough to be used in research projects, following agile development methodologies.

3. Background and related work

The approach used in MARKOS project and presented in the paper was not defined entirely
from the scratch. Instead, the authors considered and reused parts of existing research and
industry methodologies, methods, practices and standards related to requirements validation
[5], requirements reviews, prototyping, acceptance testing, beta testing [6], or user
perceived service quality [7] and usability [8], considering their applicability for MARKOS
software prototype validation purposes. The final approach was highly inspired by rapid
prototyping ideas typical to design-thinking [9], with its human-centered ethos, as well as
agile developments methods.
The quality attributes were adopted from standard ISO 25010 [10] and methods for
validating web portals, further tailored to the research nature of the project. Finally, some
shared characteristic of participatory design and validation observed in the living lab
methods [11] were derived, in particular the shared characteristic to involve users early in
the process of validation in real-life environments and incorporating the validation results
into products development and services. Particularly interesting is the FormIT method [12],
evolving in spiral throughout three stages of product/service development: the design of
concepts, the design of prototypes, and the design of the final system, where real-life
environment validation is maintained through the process as much as possible.

4. Approach description
Below the authors highlights best practices being the key characteristic of the approach
adopted in MARKOS project for validating research software prototypes. The validation

Copyright 2014 The Authors www.eChallenges.org Page 2 of 9

process is meant to be lightweight and seamlessly integrated within the development life
cycle. In particular it tries to adopt agile and start-up practices to research projects. It
consists of several cycles and combines different validation activities throughout the
development process (see Figure 1).

Figure 1: An overview of validation approach

The first and most important best practice of this approach is to start the validation very
early and treat validation as a continuous process throughout the whole development life
cycle. First validation activities should be started in the project conceptual phase when
initial requirements are specified for most important actors.

4.1 Practice 1: Start Very Early and Follow an Iterative Process

Validation should start even before writing first line of code, and shall be drawn from the
requirement elicitation and specification phase. When the validation starts before
development, validation can evaluate whether or not there is a need for a particular feature.
Once an implementation of the feature has been accomplished, validation can evaluate the
implementation, but starting from the assumption that initial feature expectations were
already satisfied. This approach is quite convenient, because it minimizes the risk of
implementing features that do not satisfy end-users' expectations, reducing the risk of
wasting resources on developing useless features.
Funded research projects usually have defined the main concept of the solution at the
proposal writing stage, which is further elaborated in the beginning of project, therefore,
users can be involved very early in the validation of that concept, providing real-life context
and helping in the specification of their needs and their prioritization. At this stage the
proposed activities are a focus group workshop and a field survey. A focus group is highly
efficient way for capturing background information from users and capturing user needs. In
turn, a survey can be used as a mean to gather priorities on planned functionalities.
Moreover, the validation needs to be done throughout the development life cycle of the
system, because the system changes as new user needs arise. Gradually, the requirements
give way to working prototypes. Following the agile paradigm, validation activity is meant
to verify the implemented sets of requirements, materialized in the form of a series of
prototype releases, and feed any findings back into the development process as issues to be
fixed in the next cycles. The proposed approach foresees validation of several major

Copyright 2014 The Authors www.eChallenges.org Page 3 of 9

releases of the prototype during the project lifetime, every few months. The exact timing
needs to be aligned with the project dynamics and resources.

4.2 Practice 2: Focus on Key Aspects

The overall goal of validation is to make the system usable in a broad sense i.e. the user
can do what he wants to do the way he expects to be able to do it, without hindrance,
hesitation, or questions [12]. However, a broad range of characteristics can influence the
quality of system, as defined in industry standards (e.g. ISO 25010). From the broad range
into account the purpose of usage of the software product, a narrow set of aspects has been
chosen to drive the evaluation process.
For a prototype of a service atop an information resource, or simply a software
prototype delivered in SaaS model, such as MARKOS, the approach allows to cover, at
least partially, the following aspects of the validated system:
benefits - to assess the perceived value of a system and benefits it provides to users,
functionality - to assess if the capabilities of prototype meet users needs,
information - to assess if the content/information resource is of the kind the users look
for,
UI usability - to assess if the prototype meets expectations on the easiness and
intuitiveness to access and use MARKOS features,
performance - to ascertain the prototype meets basic performance requirements.
Although all aspects should be kept in mind over the project lifetime, an attention
should be given to different aspects at various project phases. For example, performance
cannot be validated at the early project stages without a prototype.

4.3 Practice 3: Use Different Tools in a Flexible Way

As the aspects are quite different in nature several different tools need to be used along the
validation process to check the system from different perspectives. In MARKOS case we
decided to use five main types of validation activities: surveys, interviews (both individual
and group discussions), usability workshops, field trials and comparative experiments
(experiments comparing task execution with and without the validated system) as ways for
collecting feedback. These tools are used in turns, on different project stages according to
project needs e.g. interview and discussions are good at early stages while comparative
experiments are more useful when prototypes are getting more stable, because in early
stages it can struggle with many performance and usability issues, which should be solved
in first place.

4.4 Practice 4: Conduct Validation in Different Environments

Evaluation carried out in different environments shall include:

x a lab/controlled assessment, where users are invited to laboratories or workshops
organized under controlled conditions;
x a field environment assessment implying a real world context assessment where end-
users use the prototype by themselves.
Lab/controlled and field assessments are complementary. While field trials are more
open and less formal, there is more space for completely new insights from external users.
People using the system in their own environment behave differently compared to when
assisted and under surveillance. In turn, lab/controlled workshops allow to perform more
careful investigations on specific issues, aspects or components.

Copyright 2014 The Authors www.eChallenges.org Page 4 of 9

 One of the main challenges in validation is to identify and engage the relevant group of
users, to provide feedback channels and to keep in contact with them.

4.5 Practice 5: Involve Small Groups of Representative Users

Proactive and pragmatic approaches for involving representatives from different user
communities are also important. Validation should involve the main target user groups. The
proposed practice is to consider small groups of users, but conduct several validation
cycles. Such groups are easier to organize and their involvement is less expensive for a
project budget. In a small group, it is also easier to keep good contact with all participants
and therefore the feedback is more complete. Another challenge is the access to the
relevant communities. The best approach is to have several partners having access to
potential users of developed system. As all MARKOS partners employ developers, the
project took the dogfooding approach involving all partners in validation. Thanks to this,
validation is done in both industrial and academic communities and in several different
cultural contexts.

4.6 Practice 6: Interact Closely with Dissemination

Dissemination and validation, in fact, interact with each other on the reciprocal exchange
basis (see Figure 2).

Figure2: Interaction of validation and dissemination activities

The dissemination supports the full involvement of different stakeholders in the external
validation process, addressing tailor-made messages. On the other hand, inputs collected
from external validation help to refine the message of dissemination campaign. In this
framework, the participation in dissemination events is an opportunity for both disseminate
system value proposition and achievements and to gather useful feedbacks from external
stakeholders.

4.7 Practice 7: Building and Validating the System Should Go Hand-in-Hand

The seamless integration within the development life cycle requires the exchange of
validation findings in each cycle. Findings should be fed back into project immediately
after the validation activity. Each finding reported to the development team should include

Copyright 2014 The Authors www.eChallenges.org Page 5 of 9

a good reasoning summarizing the comments from users and help understood their needs. It
is also useful to identify and assign the role in a project that can implement the finding. It
should be mentioned that the synthesizing of needs requires combining ideas from many
points of view and a proper balancing between the original vision and the received
feedback. Leveraging user feedback to validate the ideas, design and implementation does
not mean that all comments raised by the users will be applied. The feedback needs to
complement the scope of the project, adding value to an original vision of what the product
should look like. Also the way to implement some findings is still under analyses.

5. Case Description, Initial Results and Lessons Learnt

The approach has been defined and used in the MARKOS project. At the point of writing
the paper, MARKOS project crossed the halfway mark and finished first validation phase,
encompassing two cycles: concept validation and first prototype. The activities conducted
so far included: a survey on the initial usage scenario, a focus group workshop on the
conceptual stage, a pilot assessment and a usability workshop of the first MARKOS
software prototype.
Altogether almost 50 people, mostly with a technical background, have provided direct
feedback to the MARKOS requirements and the first prototype. The group consisted of
people from the MARKOS Partners organizations, but not personally involved in the
MARKOS project, people from organizations in close relationship with MARKOS
Partners, or completely external users involved thanks to dissemination activities. On the
basis of the users comments a set of over forty actionable findings have been defined,
influencing the development process of the MARKOS prototype. Most of the comments
considered the functionality and usability of the MARKOS system.

5.1 Conceptual Phase Validating the Demand and the Problem

The first validation activity was a survey based on four usage scenarios, namely Software
Search, Code reuse, License verification and User comments. For each scenario,
there were a few specific questions, asking technical details (different for each scenario)
and eight general questions (common for all scenarios) related to the overall potential
usefulness of such functionality. The specific questions were used to directly get users
insights on particular design decisions. For example we asked users, which types of
software entity they would like to search. The feedback was that all types are expected, but
APIs, packages, interfaces, classes, and programs/applications ranked highest. The aim of
general questions was to compare scenarios, their perceived usefulness to users and impact
on the working routine and suggested improvements. The general questions included for
example: How long does it take to perform the described activity without MARKOS?,
Please estimate how many times a year you would use a system providing the type of
activity described in the scenario., Which of the scenarios steps are the most important
for you? or Is there any scenario step that you would like to remove or change?.
Answers to these questions helped the MARKOS team to identify the most innovative
features and specify requirements priorities for implemented. Rankings of features included
in each usage scenario have been created. Features ranked highest were assigned highest
priority in the development plan. In particular, features which were initially scheduled to be
released in the second version of the MARKOS system (e.g. Restrict the search to
implementations that adopt non-reciprocal licenses and Find all implementations) have
been rescheduled to be implemented in the first year prototype. Users also provided some
ideas for specific new features e.g.: a way to add some metrics or information related to the
quality of the software implementation, or possibility to link some external documentation
to the source code elements and aggregate information from other sources.

Copyright 2014 The Authors www.eChallenges.org Page 6 of 9

 Another validation activity at the conceptual phase was a focus group workshop. To
complement individual responses collected in the survey, the idea here was to get feedback
in an interactive group setting where participants are free to talk with other. The workshop
involved five developers external to project and consisted of two parts. The warm-up
questions focused on current experiences of users on the area of searching and analysing
open source software which MARKOS tries to consolidate and improve In the second part
the MARKOS system main features and early mock-ups were presented to developers.
Apart from confirming the need for an easy to use tool for developers for checking
license compliance, the discussion resulted in two remarkable findings great importance of
social aspects in analysing and selecting open source components for reuse and the need
for an aggregator of OSS related information and extensive authoring and dependency
information in search/browsing. As an outcome of the analysis of the focus group
workshop, the MARKOS team has been considering and analysing the possibility to extend
the functionalities related to the management of user content. Although the latter was
considered outside the scope of the project, to introduce some social aspect to the system it
was decided that MARKOS will build its own social asset i.e. an annotation framework
in including comments database.
In summary, users feedback at the concept validation phase is crucial to help validate
the demand and the problem i.e. to identify what is the core functionality and make the
important decisions on what to implement first. Moreover the findings help in requirements
elaboration and design and implementation decisions on particular requirements. User can
also provide many interesting ideas for new features. However, due to limited project
resources, only crucial new features could be included in the development plan. To keep the
proper balance between the original plan and the received feedback, in MARKOS, each
new idea was discussed and analysed on project forum at technical meetings. In particular,
general mechanism to provide user comments was considered sufficient for enabling user
generated content and feedback on the software entities in MARKOS.

5.2 (First) Prototype Validation Validating the Product

The first MARKOS prototype validation included a pilot field trial and a usability
workshop. The pilot field trial was the first validation activity based on a running software
prototype, involving eleven users with technical, research and business background from
partner organisations. Users were asked to try the first prototype and provide feedback
using the feedback channels created for the usual users, i.e. by email, an online survey or
face-to-face.
The most important finding obtained from this pilot field trial of the first MARKOS
prototype was the need to improve and extend its content i.e. the knowledge base, by
increasing the number of analysed projects added to the repository. It turned out that the
very limited knowledge base did not allowed to recognize the potential of such service.
The improvements also shall consider features of the user interface, especially, adding a
help section and redesigning the prototype homepage. Moreover, the improvement of the
integration of the license analysis tool and other MARKOS components was prioritized.
Although users found the MARKOS license analysis tools to be one of its key features,
majority of developers assessing those tools claimed they were too complex for regular use.
The field trial outcomes updated the system features and development plan in the way that
some of more advanced envisioned features of the license analysis tools were postponed or
dropped. Instead, the remaining efforts of the project were decided to be put on simplifying
and improving both the usability and look and feel of the License Analyser core features.
Specific, design suggestions came from a usability workshop, which was carried out a
few weeks after the pilot field trial with a goal to elaborate more on the usability issues the

Copyright 2014 The Authors www.eChallenges.org Page 7 of 9

users experience when using MARKOS. The suggested adjustments of MARKOS
prototype portal included mainly a unification of the portal layout, adding short 3-step
introduction, improving visualisation of the properties of software entities and unification
of the look and feel of the license analyzer component. Changes have been accepted and
added to the plan for the next development cycle. Also a major redesign of the License
Analyser component was planned and usability testing workshops have been scheduled for
the next validation cycle.
As mentioned earlier in this paper, a validation is closely related to dissemination
activities. During the pilot field trial, MARKOS attended two important events, which were
beneficial for validation process in terms of involving new external users and collecting
their feedback. At the Open World Forum 2013 [13], the project discussed possible
community collaboration models and established initial contacts within OSS communities,
such as OW2 community and Xen (Linux Foundation), as well as with other competitors on
the market. MARKOS first prototype was also presented at the ICT 2013 Conference [14],
where MARKOS met potential users from different areas such as SMEs, Industry,
innovative organizations and developers, resulted in collecting good feedback on
MARKOS features (e.g. GANT [15] project expressed an interest in MARKOS and
License Analyser in particular).
To summarize, prototype validation brings valuable and concrete insights for improving
the solution. At this stage field trials and technical lab/controlled experiments (such as
usability workshops) should go in turns in several cycles, and, building and validating the
product should go hand-in-hand and be an iterative process, with some amount of validated
learning [16] along the way.

5.3 Final Prototype Validation

Next prototype validation cycles will include more usability workshops focused on specific
MARKOS components e.g. the License Analyser. While getting closer to the final
validation, it is foreseen to use comparative experiments to evaluate the actual quality of
use of MARKOS system. Such an experiment will compare results of different groups of
people conducting the same task using MARKOS and other systems. It is also planned to
have a case study of MARKOS usage in GANT community.

6. Conclusion
This paper presents a pragmatic approach to software prototypes validation. The work
presented in this paper was performed in the context of the MARKOS European research
project aiming to support open source developers in community-based software
development. The described approach was used to validate the prototype service of the
MARKOS project, which provides an integrated view on the Open Source projects
available the on web, focusing on functional, structural and license related aspects of
software code. The presented approach can be used in research projects aiming at delivering
prototype services and applications.
The approach identified five important quality aspects for validating research software
prototypes. It also highlights the need to start early and continue validation as a way to
obtaining valuable findings for research, adopting the agile development philosophy to
release early, release often to release and validate early and often. Early assessments
combined with agile software development improve the delivery of software products that
fulfil end-users expectations and reduce the risk of failure when achieving the user
satisfaction. Even if these approaches are getting popular in the industrial software
development, especially in start-up communities, they have not yet been widely adopted in
research and innovation projects, as explained in the introduction.

Copyright 2014 The Authors www.eChallenges.org Page 8 of 9

 The MARKOS example shows that adopting this philosophy allows to drive the
development in the way to fulfil the real needs of users and enhance the overall quality of
the system. In particular, early concept validation allows to set up the priorities for the
development and extend or adjust the planned software features before the start of the
actual development. Then validation activities help to set the priorities for the next
development cycles. Several validation cycles targeted towards different groups of users
also improve diversification of feedback sources and support community building process,
which is an another important aspect of sustainability of the project results.

References
[1] Horizon 2020 Work Programme 2014 2015, European Commission Decision C (2013) 8631 of 10
December 2013
[2] M. Bolton, User Acceptance Testing A Context-Driven Perspective, DevelopSense, 2007
[3] The MARKet for Open Source - An Intelligent Virtual Open Source Marketplace. MARKOS project
2012-2015, http://www.markosproject.eu/
[4] G. Bavota et all, The MARKet for Open Source: An Intelligent Virtual Open Source Marketplace
A. Abran, et all, Guide to the Software Engineering, Body of Knowledge, ISBN:0769510000, IEEE Press
Piscataway, NJ, USA, SWEBOK, 2004 Version
[5] M. R. Fine, Beta Testing for Better Software, ISBN 0-471-25037-6, Wiley Computer Publishing, 2002
[6] Z. Yanga, S. Cai, Z. Zhou, N. Zhou, Development and validation of an instrument to measure user
perceived service quality of information presenting Web portals , Information and Management,
Elsevier, 2004
[7] J. Rubin, D. Chisnell, Handbook of Usability Testing - How to Plan, Design, and Conduct Effective
Tests, ISBN: 978-0-470-18548-3, Wiley Publishing, 2008
[8] T. Brown, Design Thinking, Harvard Business Review, 2008
[9] ISO/IEC 25010:2010(E), Systems and Software Engineering - Systems and Software Product Quality
Requirements and Evaluation (SQuaRE) - System and Software Quality Models, International
Organization for Standardization, Geneva, 2010.
[10] E. Almirall, M. Lee, J. Wareham, Mapping Living Labs in the Landscape of Innovation
Methodologies, Technology Innovation Management Review, 2012
[11] A.Sthlbrst, B. Bergvall-Kreborn, FormIT an approach to user involvement. In European Living
Labs - A new approach for human centric regional innovation, edited by Schumacher, J. and Niitamo,
V.-P. Berlin: Wissenschaftlicher Verlag, 2008
[12] The Open World Forum 2013, http://openworldforum.org
[13] ICT 2013, http://ec.europa.eu/digital-agenda/en/ict-2013
[14] http://www.geant.net/Pages/default.aspx
[15] http://theleanstartup.com/principles

Copyright 2014 The Authors www.eChallenges.org Page 9 of 9