3

Volume

TECHNOLOGY MANAGEMENT 204

Dr. Rolando Dayco
1st Semester 2010-
2011

Management of
Research and
Development
 A N E V A L U AT I O N R E P O R T O N T H E

Philippine Research, Education and

Government Institution NETwork
(PREGINET)

Submitted By:

Alvarez, Mary. Joan

Cuevas, Juan Ceasar M.
Derecho, Charisse C.
October 2010
 1
Table of Contents
Topics Chapte Page
r No.
I. Objectives of the Evaluation 1 1 to 2
II. Background of the PREGINET Project 2 3
III. Evaluation of the PREGINET Project 3 4
A. Problem Identification and Proposal
4 5
Stage
B. Implementation Stage 5 6 to 8
C. Post-Evaluation Stage 6 9
IV. Conclusion 7 10 to 12
V. List of ReferencesAppendices 8

Objectives of the Evaluation

T
his evaluation report aims to assess the Philippine Research, Education and
Government Institution NETwork (PREGINET) project as a research and
development (R&D) project focusing only on the aspects that has direct
impact to research and developmentR&D.

Specifically, the objectives are to:

To Specific, Measurable, Attainable, Relevant and Time-bound

• Ddetermine the concrete research and development (R&D)R&D output/s of

the project;

• To Ddetermine how effective the Assess the PREGINET project, PREGINET, in

terms of project phases: Planning, Implementation and Post Implementation
 was managed from the planning and the proposal stage, to the
2
implementation and post-evaluation stages and how each of these stages
contributed to the success or failure of the project, and in terms of R&D

• To Assess the impact of PREGINET in terms of its R&D objectives and output

• Assess the impact of PREGINET in terms of its contribution to the

advancement of network technologiesPprovide recommendations effectively
managefor the management of future DOST-funded R&D projects.

Background of the PREGINET Project

I
n 1996-1997, the strategy of the RPWEB RPWeb initiative was for government
agencies to connect through commercial ISPs. Unfortunately, they experienced
varying levels of service and there was a a lack of sufficient local connectivity
between commercial ISPs.

During those times, There became a trend that industrially-advanceially

advanced countries have embracinged a knowledge economy and realizing
realized the value of setting-up Research and Education Networks (RENs) to
foster innovation. ASTI found this e relevant to the Philippines and thus identified
the “need” for the Philippines country to have a network that is contiguous and
wide in coverage, with available bandwidth on which next-generation network
technology and services can be testinged and deploymentdeployed, and where
applications development, experimentation, and demonstration can be
conducted.

The PREGINET project was conceptualized with all these in mind. Thehe project,
based on its documented research proposal, seekssought to accomplish the
following objectives:

ii
• foster partnerships among government, academe, industry, and the
research community all over the country;
• facilitate the introduction of next-generation network technology and
services by providing better network infrastructure and new services (e.g.
security, quality of service, etc.) for research, education, and key
government institutions, and
• said network will be used in the development and demonstration of novel
applications for science, education, health, and business

iii
P R E G I N E T E V A L U A T I O N R E P O R T

3
The PREGINET project details are summarized in TableFigure 1 below:

Comprehensive Program to Enhance Technology

Program Enterprises
(DOST-COMPETE)
Virtual Centre for Information Technology Innovation in
Information Technology (VCTI-IT): Development of a High
Project Performance Research and Education Network: The Philippine
Research, Education, and Government Information Network
(PREGINET)
June 1, 2000 – May 31 2005
1st Extension: June 1, 2005 – May 31, 2006
Duration
2nd Extension: June 1, 2006 – February 28, 2007
3rd Extension: March 1, 2007 – December 31, 2007
Total Project
Php78,887,793.10
Cost
Implementing Advanced Science and Technology Institute – Department of
Agency Science and Technology (ASTI-DOST)
Monitoring Philippine Council for Advanced Science and Technology
Agency Research and Development (PCASTRD)
Cooperating
DOST Agencies and Regional Offices, PREGINET Partners
Agencies
Figure TableFigure 1 PREGINET Project Information

Evaluation of the PREGINET Project

TI
P R E G I N E T E V A L U A T I O N R E P O R T

n order to determine how effective the Phe projectREGINET project was managed,
A
it will be evaluated in terms of thein the context of each of the three project
management stages in project management:

A. Problem IdentificationConcept DevelopmentConcept

Development aAnd Proposal Stage;
- Problem Identification and Solution Proposal
- Project Objectives and Expected Output

B. Implementation Sstage, and

- Research Outputs
- Use of Funding
- Staffing
- Risk Management
- Product Development

C. Post-iEmplementation evaluation Evaluation stagStagee

- Evaluation Team
- Evaluation Objectives
- Evaluation Methods
- DAP Evaluation ResultsFindings/ Results of DAP Evaluation
- DAP Recommendations.

Problem IdentificationConcept Development and

Proposal Stage
I
n this section, we view the project in terms of its strategic justification 1, which
requires that the objectives of the project are: (1) are consistent with the
agency’s strategic objectives; 2) are are consistent with general Government
policy; 3) can be justified on the grounds the market is failing to efficiently deliver

1
Department of Treasury and Finance Government of Western Australia, Project Evaluation Guidelines,
(January 2002)
P R E G I N E T E V A L U A T I O N R E P O R T

the proposed project outputs; and 4) take advantage of and has consideration for
current and expected future economic conditions.

The project implementerproponent, ASTI, essentially identified the following

problems and/or needs that arewere to be solved addressed by the PREGINET
project:

• Interconnectivity of government networks;

• Advancement of ICT Information and Communication Technologies (ICT) in
the Philippines, and
• Network dedicated to research.

The problem identification process and the actual identification of the problems
identified that are to be addressed by ASTI is are the basic initial weaknesses ofin
the project. It is a case where the implementer proponent attempts to do too
muchalready has pre-conceived solution, which led to the flawed problem
identification process and in with inaadequate screening and evaluation. A of
alternative solutions to the problem were not identified and there was no an
assessment of whether the proposed solution is the best option to address the
identified problems was not conducted.
.

The first problem, interconnectivity of government networks, is out of scope of

the institute’s ASTI’s mandate, which is to “Conduct scientific research and
development in the advanced fields of Information and Communications
Technology and Microelectronics.” Interconnectivity on a nationwide scope is the
DOTC’sDepartment of Transportation and Communications’ roleand is essentially
DOTC’s burden to bear. Given the project’s initial timeframe of five (5) years, the
capabilities of commercial internet service providers (ISPs) are bound to catch up
and will be sufficient to meet the needs of non-research government firms. This
can be done without DOST-ASTI spending their R&D budget to subsidize these
government agencies’ internet connection.

The second problem , unlike the first one, is in line with ASTI’s mandate; however,
the approach to the solution is again weak and should ha’ve been re-evaluated.
The implementer’s proponent’s rationale goes: in order for us to research and
develop advanced networking technologies we need to first spend our budget on
a network. If they had a clear R&D strategy with a project portfolio and strategy
for product development of advanced, undiscovered and cutting edge
technologies then that will warrant the need for a high-bandwidth network.
Unfortunately, the R&D strategy is also weak as will be described in later
sections.
P R E G I N E T E V A L U A T I O N R E P O R T

Suggested: The second and third problems are in line with ASTI’s mandate;
however, the approach to the identification of the solution is again weak and
should have been re-evaluated. The proponent’s rationale is:: in order for us to
promote the advancement of ICT in the Philippines, we need to first to set-up a
dedicated network infrastructure for local research institutions to have access to
international research papers,; facilitate knowledge exchanges and
facilitateenable the development, testing and demonstration of advanced
networking technologies.

ASTI could have focused on determining ways to advance ICT in the Philippines
by creating a well-balanced project portfolio with clear R&D strategies for the
development of advanced, undiscovered and cutting-edge technologies. Having a
high-bandwidth network is neither a necessary pre-requisite nor the only means
to achieving the desired objectives and virtual research communities can be
organized even without setting-up the said network infrastructure. Unfortunately,
the R&D strategy is also weak as will be described in later sections.

What could have been morewas more feasible and efficient to advance ICT in the
Philippines is towas to focus their efforts on purely R&Ddeveloping new network
technologies projects and taking steps to ensure their commercialization.; Aan
example can be made of the Pawikan Project.2 Pawikan is a network monitoring
software that was developed under PREGINET and was ready to be sold to a a
local telecomstelecommunications company. However, Ddue to lack of proper
documentation and turnover of the source code, the knowledge left when the
developer resigned. What was left was a source code that no one in the
organization i was willing to study and the thus, the project was
abandonedfizzled.

To address the need for a research networkMoreoverFurthermore, a research

network run solely by a a private telecommunications company company would
hav’ve been cheaper as market competition between internet service
providersavailable ISPs lowered the ratesdrives the rates down. Needless to say,
and it would also serves the same purpose as one that is administered by
ASTIPREGINET. In fact, PREGINET services are not as competitivecannot compete
in terms of ratesbecause they acquire the internet connection also through these
ISPs and even the network reliability of the networkis reportedly problematic. As
for researches, with the materials that ASTI can access via their partnership with
Asian Internet Interconnection Initiatives (AI3)AI3 and Asia-Pacific Advanced
Network (APAN)APAN, they could have provided in a web portal where local
research agencies institutions can visit to gain access and thereby encourage
more participation and sharing of research outputs.

2
Pawikan Network Monitor, http://pawikan.sourceforge.net/
P R E G I N E T E V A L U A T I O N R E P O R T

Objectives and Expected Output

In this section, the objectives and output are assessed. We look at how clearly
they were defined and how success and failure of these objectives will be useful
for the proper evaluation of the project.

The project proposal states five main objectives and one generic statement for
the expected output (Figure 2).

Figure 2 PREGINET Project Objectives and Expected Output

The project proposal states five main objectives and 1 statement of expected
output (see figure 2).

On the surfaceIn the literal sense, the objectives may sound are clear and
plausible. However, they are simply answerable with Yes/No and are not
measurable or tangible by any means. Thus it is possible that practically anything
can happen in the project and the objectives will still be met.

On the surface, the objectives may sound clear and plausible. However, they are
simply answerable with Yes/No and are neither measurable nor tangible. Thus, it
is possible that anything can happen in the project and the objectives will still be
met.

To say that Tthe expected outputs are vague would be an understatement. The
expected outputs and cannot even be seen or measured. Questions like How,
P R E G I N E T E V A L U A T I O N R E P O R T

B
What, How many or How much should be answered by the expected outputs. The
following are a few examples:

• Attain X atotaln X amount of savings due to usage of PREGINET services

such as video conferencing and voice over internet protocol (VOIPVoIP) in
government agencies, instead of normal long-distance calls and travel;
• Reach an a X number of partners who’ve published research papers in
referreed journals;
• Commercialize at least one (1) new networking technology per year, and
• Achieve an X percentage measurable increase in researches published in
scientific papers, and , patents granted, etc.
, implementationsed and use of tilized network technologies in 5 years of
operation

The objectives could have been patterned after “SMART” objectives, a mnemonic
used in project management during the project objective- setting stage.
“SMART”SMART stands for Specific, Measurable, Attainable, Relevant and Time-
bound.

In view of the foregoing, Wwe can say that It is hard to say if this is intentional,
but at this point in the planning process, the project should have been rejected or
have been returned for revision and further re-evaluationed or even rejected.

Implementation Stage

T
he project implementation framework is summed up by the diagram below
(Figure 43)(Appendix 1). Given that DOST mandated the thrust towards
Science and Technology (S&T) development/advancement (?) and wants
sought to address the ICT needs of the Philippines, the projectthe intention is to
aimed to utilize ICT technologies, Network network services and R&D on next
-generation products to producein a high a high -performance Research and
Education networkREN, which is the (Philippine Research, Education, and
Government Information NETwork (PREGINET)PREGINET. The overall goal of
P R E G I N E T E V A L U A T I O N R E P O R T

PREGINET the project is to establish a nationwide broadband network to

interconnect academic and research institutions as well as government agencies
in order to strengthen the country's science and technologyS&T capability and
further to enable e-governance.

Figure 3 PREGINET Project Conceptual Framework

Its The project’s direct beneficiaries are the following:

1. Information Technology IT Industry

2. User Communities for Agriculture
3. User Communities for Distance Education
4. User Communities for Networking Technologies
5. User Communities for Disaster Mitigation
6. User Communities for Bioinformatics and Telemedicine

Based on the approved project proposal, Aat the end of thise project, these were
the expected deliverables are:

1. Operational exchange point for research, education, and government

network interconnection;.
2. Network testbed for research in and testing of next generation
network technologies, services, and equipment, and for the development,
P R E G I N E T E V A L U A T I O N R E P O R T

experimentation, and demonstration of novel applications in the areas of e-

commerce, telemedicine, distance education, among others;.
3. Competency in network equipment testing and integration services;
4. . Competency in network technology integration and network
services provisioning;.
5. Developed and demonstrated novel applications for e-commerce,
telemedicine, distance education, and other areas, and.
6. Connectivity to research networks outside the country.

Analysis on Implementation Stage

The analysis of the project’s iImplementation stage focuses on major areas that
are crucial to the success or failure of PREGINET as an R&D Research and
Education networkDevelopment organizationproject. The areas for analysis are
the following:

• Research Outputs
• Return of Investment
• Staffing
• Risk Management
• Product Development
P R E G I N E T E V A L U A T I O N R E P O R T

Research Outputs

Included in the intended deliverables of this project are the establishment of a: N

network testbed for research in and testing of next -generation network
technologies, services, and equipment, and for the development,
experimentation, and demonstration of novel applications in the areas of e-
commerce, telemedicine, distance education, among others.

However, at the assessment of the final status of the project, it was discovered
that no new generation technologies wereas developed, but instead, it was
merely adaptation adoption of matured technologies made developed
abroadreadily available in the market.

In addition, Aamong the functions for the network testbed would be to alloware to
enable experimentation and testing of new technologies and equipment (e.g.
ATM, wireless broadband technologies, IP over SONET/SDH, DSL technologies,
etc). However, since these technologies can also be tested using
commericialcommercial ISPs, it would have been a more cost-efficient strategy if
the project did not spend for the setting-up ofoffset-up the network infrastructure
and instead also, the project should have been directed at testing these
technoloused the services of cheaper and more reliable services of gies in
commercial ISPs. Furthermore, manpower cost can also be reduced as
deployment and network support will not be required rather than establish a high
speed network where they can test it. It could have been more cost effective
andand the research staff can focus more important endeavors such as on thethe
development of new network technologies a better focus of their skills.

Some areas of research done during the project are: Network Administration
(DNS, Mail, Web, FTP, ProxyProxy); Multimedia over IP (Videoconference,
Videostreaming); Virtual Classroom System; Digital Content System (Library
Management System, Digital Library); Voice over IP; Network Management; AP
BioBox; and IPv6 Implementation. (See Appendix 1). TThese are matured
technologies already being used widely abroad and the findings show that no no
new radical or incremental research was includedinnovations were produced by
the project.

Moreover, it was also discovered that they the project hashad no established
mechanism way of concretely concretely monitoring or documenting what new
-generation network technologies technology researcheshthe R&D were
conducted and facilitated done through the network by their partners (See
Appendix 2).
P R E G I N E T E V A L U A T I O N R E P O R T

If one of PREGINET’s main thrust was increaseincreased R&D research onof new
network technologies or other advanced system applications or tools, their
quantitative measures of success could have been the following:

• Number of refereed papers published

• , Number of patents generated
• , Number of innovations developed
• , Number of innovations commercialized
• , Amount of external research grants obtained
• and Return on R&D investment.

Return on Investment

Overall, the approved budget for the PREGINET project was P79M (Figure 4). Due
to the unavailability of information on actual breakdown or components of the
project budget, we cannot adequately assess whether the budget spending was
cost-efficient.

Nevertheless, as we have earlier identified the project’s deficiency in problem

identification and solutions-matching, we have established the fact that the
project budget could have been spent more efficiently and effectively if it was
used for the development of new products that can be commercialized instead of
funding the infrastructure set-up, internet connection and network support to
partners.
P R E G I N E T E V A L U A T I O N R E P O R T

Figure 4 Project Budget – Approved, Released, Spent

Overall, the funding for the PREGINET project was a total of P79,010,719.60.
Although the main advantage of having a dedicated network should be that it has
a high-speed connectivity (more faster than that of commercial ISPsnormal
internet service) in order connect Philippines universities and other research
institutions to the global research and education networks. Consequently, this
network connection should allow the creation of emerging network technologies
(that need higher bandwidth for testing).
Moreover, the fact that the project’s original design did not include strategies for
sustainability, it is evident that the proponent did not intend to generate
revenues or recover the cost incurred in the project implementation. A Cost
Benefit Analysis (CBA) should have been conducted to determine whether the
project benefits adequately justifies the cost and the Return on Investment (ROI)
should have been determined to assess whether it is a sound business proposal
for the government to invest in.

Obviously,

However, in terms of Return on Investment, Tthe PREGINET has not been

successful in proving that this project was unable to come up with anything that
would constitute agenerate revenues for the government except the minimal
service charges they recently introduced. Furthermore, these charges cannot
adequately cover its operational expenses since at present; PREGINET has been
subsidized and included as part of ASTI’s annual budget. Return of Investment.If
the focus was more on the development of new network technologies and
products that can be commercialized, it could have been the source of revenues
or returns on the government’s almost P79M investment. In fact, at presently
DOST-ASTI is spending its yearly budget just to keep the network operational.new
network technologies that can justify the cost. A Cost Benefit analysis between
developing a dedicated network and commericialcommercial ISP shcould have
been done. An ROI study should have been done to see the returns of this project
to its intended reciepientsrecipients. During the 5 year project, the budget
covered the following areas: Personnel, Maintenance and Operation Expenses
and Capital Outlay.
P R E G I N E T E V A L U A T I O N R E P O R T

Staffing

With its original 11-man project team (Figure 5), PREGINET experienced high staff
turn-over rate during the implementation. As a result, some activities were either
discontinued or had to be redone/redesigned (due to lack of proper technical
documentation). One example of the activity that was discontinued was the
deployment of the VClass System. Initially, a number of partners became
interested in the online learning management system. The project was able to
carry out a few trainings for institutions that expressed their interest to use it
such as the Philippine Rice Research Institute (PhilRice) and the Benguet State
University (BSU). However, when the whole development team resigned, there
was nobody left to continue with its development and deployment. The same
case happened with the development and deployment of the Greenstone Digital
Library, and IPv6.

Total No. of Members

Figure 5 Project Team Composition

The project experienced high staff turn-over for the duration of its
implementation. As a result, some activities were either discontinued or had to be
redone/redesigned (due to lack of proper technical documentation). One example
of the activity that was discontinued was the deployment of the VClass System.
P R E G I N E T E V A L U A T I O N R E P O R T

Initially, a number of partners became interested in the online learning

management system. The project was able to carry out a few trainings for
institutions that expressed their interest to use it such as the Philippine Rice
Research Institute (PhilRice) and the Benguet State University (BSU). However,
when the whole development team resigned, there was nobody left to continue
with its development and deployment. The same case happened with the
development and deployment of the Greenstone Digital Library, and IPv6.

One factor that can be attributed to this was the unattractive compensation and
benefits package offered to the researchers, as DOST, a line agency is covered by
the Salary Standardization Law (SSL)low salary coverage of ASTI for its
researchers. In addition, it is a fact that competitive packages are available from
the private sector, which realizes the real value of these skilled human resources.
In this case, high staff turn-over rateTheir transfer to higher salary work became
is inevitable; possible due to the Salary Standardization Law which could hinder
ASTI from addressing this.

Although there are also other motivating factors that ons keep researchers loyal
to their employeran organization like: challenging, interesting & important project
assignments;, adequate research facilities and , personnel, equipment & time,
and , status that maintain pPraise and recognition from peers in the scientific
community, the attractiveness and value of these may vary from employee to
employee. One factor that this project was not able to address sufficiently was
the responsive and equitable reward & recognition system which can also cover
salary, bonuses, monetary benefits or non monetary benefits like peer
recognition,

In any case, a systematic and establishedized documentation process could have

solvepossibly reduce the impact of d the problem disrupted and discontinued of
abandoning or recreating programsprograms caused by and the likbecause of
staff resignations. Efficient Sdocumentation processes and knowledge transfer
mechanisms and protocols trict documentation should be properly in place.and
knowledge transfer policies within the project should have been part of the
Standard Operation Process. e.

Risk Management

As the number of connected partners increased, so did the need for a higher
bandwidth as well as a more reliable network. Initially, the regional access points
were mainly the DOTC Telecommunications Office (Telof) facilities. However,
during network problems, the Telof’s response time to address the problem was
limited by factors such as: lack of capacity to properly address/fix the problem;
and government standard operating procedures. For example, if the technical
problem involved an equipment that needed to be fixed/replaced/procured, the
P R E G I N E T E V A L U A T I O N R E P O R T

request had to go through the usual government procedures that caused major
delays in the operation of the network.

These kinds of Risk should have been included in making contingency plans in
the beginning of the project. As a result, slow network connection, major delays
in operation and unsatisfactory service affected PREGINET’s partners and client.

Product Development

Successful R&D projects, whether incremental or radical ones,ones improves

products & services thus contributesservices thus contribute to higher
productivity, lower costs & increased profits. It should provide customers (in the
case of PREGINET, its beneficiaries in government, academe and private
industries) with varied choices, better choices product performance & lower
prices.

The ultimate objective of the project should have been Advanced network
services may beare necessary to createthe development of highly advanced, first
-in -the -market, cutting- edge, support next- generation applications that is
competitive in performance and price. (especially those that need high speed
internet) which is why PREGINET was created in the first place.These In turn, new
products can beould have been commercialized and could possibly advance ICT
development in theor make the Philippines a leader in Network Technology
development. However, As mentioned earlier, researchpresent endeavors of
areas covered by PREGINET, which in These new research areas iincludeclude
among others, d IpPv6, Multimedia over IP, Network Measurements, Content and
Applications development, hosting and distribution , which are hardly cutting
edge already mature technologies and readily available in the first in the market.

However, tThe fact that the project did not result in a service or portfolio of
products that are commercialized or new and even proven to be effective
alternatives to existing technologies makes it apparent that the project failed in
the area of product development, which should have been the foremost indicator
of the project’s performance., reveals a weakness in implementation or
technology diffusion since it should have resulted in the use of advance network
technologies by the Government, Education and Private firms in this country..
P R E G I N E T E V A L U A T I O N R E P O R T

C
Post-Implementation Evaluation Stage
C
“TA
n impact assessment3 is a process aimed at structuring and
supporting the development of policies. It identifies and
assesses the problem at stake and the objectives pursued. It
identifies the main options for achieving the objective and analyses their likely
impacts in the economic, environmental and social fields. It outlines
advantages and disadvantages of each option and examines possible
synergies and trade-offs."

The Philippine Council for Advanced Science and Technology Research and
Development (PCASTRD) under DOST was tasked to monitor the project
accomplishments for the most part of project implementation based on the
reports submitted by the project team in compliance with DOST policies and
procedures. On the other hand, the Development Academy of the Philippines’
Center for Knowledge Management (DAP-CKM) was commissioned by ASTI to
conduct the impact assessment of the PREGINET Project. Thus, in this section,
we will evaluate the post-evaluation stage of the project in view of the impact
assessment conducted by DAP and the analysis will be focused on the following
aspects:

• Evaluation Team
• Evaluation Objectives
• Evaluation Methods
• Findings/ Results of DAP Evaluation
• DAP Recommendations

Evaluation Team

The DAP project team consisted of Elena Avedillo-Cruz, Managing Director,

Cristina Bobis, Project Officer, Sheryl dela Cruz, Technical and Administrative
P R E G I N E T E V A L U A T I O N R E P O R T

Staff, Caridad Aspiras, Supervising Fellow, and Segundo E. Romero, Technical

Consultant.

Impact Assessment, http://en.wikipedia.org/wiki/Impact_assessment

The result of the impact evaluation could have been considered objective
because it was conducted by a third party organization. However, in the set-up,
ASTI was the contracting party, project proponent and implementer. In order to
ensure the impartiality of the project evaluation, the contracting party to the
agreement with the evaluating team should be independent from that of the
project proponent or implementing unit. As the funding and monitoring
agencies for the project, DOST or PCASTRD, respectively could have been the
contracting party for the evaluation.

Evaluation Objectives

In terms of impact, ASTI agreed to the following DAP evaluation objectives:

1. Determine whether the stated project goals and objectives were

achieved;
2. Ascertain the contribution of the project to the overall program
objectives;
3. Assess the benefits of the projects;
4. Determine conditions under which the projects are most effective;
5. Identify the factors that contributed to the projects’ success or failure;
6. Recommend possible projects that can be pursued in this field and
suggestions to improve performance of such projects included.

Just like the objectives set by the PREGINET project during the proposal stage,
the above objectives are vague and should have also been patterned after the
“SMART” objectives. Moreover, assessing the impact of the project in terms of
R&D can be most appropriately expressed through quantifiable indicators such
as the number of new network technologies developed.

Evaluation Methods

There were two major methods used in evaluating the projects’ impact – the
desk review and the field assessment. For the desk review, relevant and
existing project documents were analyzed and the preliminary evaluation
which resulted from the desk review was then validated using three sets of
evaluation tools – the survey questionnaire, the key informant interviews (KII)
and the case studies.
P R E G I N E T E V A L U A T I O N R E P O R T

However, it was noted that only 29 (37%) of the 78 target respondents

accomplished and returned the forms to DAP for data processing, which were
used as basis for the evaluation (Figure 6). This shows that the basis for the
evaluation is not representative of the total population rendering the results to
be dubious and questionable.

Figure 6 Retrieval Rate of Survey Questionnaire

DAP Findings

DAP evaluated the project in terms of four major areas – project design,
delivery, impact, and sustainability. According to DAP:

• The project objectives were attained because it was able

to:

a.
advance the development of next generation technologies,
applications and services and made these accessible for
user communities;
b.
provide a platform that enabled the introduction,
development, testing and deployment of new network
technologies; and
c.
build up its partner institutions from the academe,
government, and research institutions with around 169
partnerships established as of August 2008

• The project remained focused on the objectives it set to

achieve. Thus, it was able to establish a national REN,
develop new technologies, and build the capability of human
resources to support the achievement of these objectives.

The result that the “project objectives were attained” is not adequately
justified given the fact that the accomplishments were not measurable. In
P R E G I N E T E V A L U A T I O N R E P O R T

addition, if assessed in the light of advancing the development of next-

generation technologies, only Pawikan was developed but was not successful
as it was not fully commercialized.

• The overall assessment for PREGINET’s project design is

summarized below:

a.
The intent for the PREGINET project was clear from the
start, except for the quantifiable and measureable targets
that should have been set at the start. This was confirmed
by the documents reviewed as well as the results of the
field validation where the key informants consulted cited
the following as bases for identifying the services, products
and technologies promoted by ASTI:

i. Anticipated and visible need of the country in network

technologies, which is usually not pushed or probed by
the private sector
ii. Need for the country to have an R&E network (REN) to
be able to
iii. Participate or have access with the research activities
of other RENs around the world
iv. Needs and requirements of the user communities of
PREGINET in line with the thematic areas being
focused by ASTI, namely environment, education,
government, and enterprise development
v. Focus of PREGINET is the development and adoption
of advanced networking technologies that can benefit
PREGINET partners.
vi. Services provided are identified based on the
technologies and applications used to promote the
development of advanced networks.

DAP’s overall assessment that “the project was successful” is contradicted by

their own finding that “there were no quantifiable and measureable targets set
at the start of the project.” It will not be possible to fully assess and effectively
evaluate a project’s impact without benchmarking it with “SMART” objectives.
Therefore, PREGINET’s success as declared by DAP may be disputed.

b.
Guided by the strategic objective “to establish a
nationwide broadband network that will link research and
education institutions for the development, testing, and
deployment of next generation network technologies,
P R E G I N E T E V A L U A T I O N R E P O R T

services, and applications intended for distance education,

agriculture, disaster mitigation, telemedicine, and
bioinformatics”, the PREGINET project employed strategies
that were found to be responsive to identified needs.
These areas are as follows: a) acceleration of government
connectivity, b) development of IT manpower base, c)
promotion of technological innovation and
experimentation, and d) creation of strategic partnerships
among local institutions. The project, however, could have
been optimized had it been promoted extensively. It
seemed that the infrastructure was already provided by
the project but content development was inadequate. In
addition, the project’s original design did not include
strategies for sustainability. It was during the latter part of
the project that sustainability measures and mechanisms
were incorporated into the design.

Another DAP finding is that “the PREGINET project employed strategies that
were found to be responsive to identified needs.” This is also contrary to fact
that there was no concrete proof that the project led to the “acceleration of
government connectivity, development of IT manpower base, promotion of
technological innovation and experimentation, and creation of strategic
partnerships among local institutions.”

DAP Recommendations

Based on the results of the evaluation, DAP recommended the following:

a. Strengthen advocacy activities to promote the benefits of

the PREGINET. In spite of the attainment of project objectives
there is still a lingering perception that the partners have not
fully realized or appreciated the benefits of the network.
b. Actively solicit content from the partners. While there is an
existing infrastructure, the lack of content to offer to partners
should be addressed
c. Seek the involvement of other stakeholders for advocacy
such as the executive and legislative branches of the
government. The PREGINET project has been funded from the
national government budget so it should be supported by the
rest of the government sector to optimize its usefulness.
d. PREGINET should speak for itself. It should publicize its
accomplishments in the form of stories, case studies, and
best practices.
e. There is no substitute for better service. Partners who were
highly satisfied with PREGINET's technologies, applications
P R E G I N E T E V A L U A T I O N R E P O R T

and services could give testimonials to inform and invite

others to the network.

If DAP concentrated on assessing the project in terms of measurable

objectives, specifically the development of new technologies or products, it is
apparent that the project was not successful. In view of this, we recommend
the following for future DOST projects:

a. Objectives should be specific, measurable, attainable, relevant and time-

bound (“SMART”).
b. Tools for financial analysis like CBA and financial indicators such as ROI,
NPV should be used in project prioritization and selection. However, this
does not mean that financial tools and indicators are the only criteria for
project selection because a well-balanced project portfolio requires
inclusion of a few radical projects that would facilitate the long-term
future viability of the organization.
c. Quantitative measures should be set to determine the project’s success,
such as:
- Number of refereed papers published
- Number of patents generated
- Number of innovations developed
- Number of innovations commercialized
- Amount of external research grants obtained
- Return on R&D investment
d. If competitive compensation and benefits package cannot be granted to
skilled research staff, provide challenging, interesting & important
project assignments; adequate research facilities and equipment, among
others and put in place efficient documentation processes and
knowledge transfer mechanisms and protocols to reduce the project
discontinuity caused by staff turn-over.
e. Maintain the objectivity of the project evaluation by ensuring that the
contracting party to the agreement with the evaluating team is
independent from that of the project proponent or implementing unit.
f. Monitoring processes, tools and mechanisms should be established and
project evaluation should not be done only after project completion.
Instead, for long-term projects, annual project audit and monitoring of
accomplishments should be conducted to ensure that for each project
phase, the project milestones and targets are achieved.

he Advanced Science and Technology Institute (ASTI) of the Department of

Science and Technology (DOST) commissioned the Development Academy of
the Philippines’ Center for Knowledge Management (DAP-CKM) to conduct an
impact evaluation of the VCTI-IT: Development of a High Performance
Research and Education Network: The Philippine Research, Education and
P R E G I N E T E V A L U A T I O N R E P O R T

Government Information NETwork (PREGINET),which is a component of the

COMPETE program funded under DOST’s Grants-In-Aid (GIA).

DAP Evaluation Team and Time Frame

The DAP project team consisted of Elena Avedillo-Cruz, Managing Director,

Cristina Bobis, Project Officer, Sheryl dela Cruz, Technical and Administrative
Staff, Caridad Aspiras, Supervising Fellow, and Segundo E. Romero, Technical
Consultant.

The Philippine Council for Advanced Science and Technology Research and
Development (PCASTRD) handled the monitoring of PREGINET and VCTI-RF
projects since information technology and microelectronics were the priority
areas of PCASTRD. However, PCASTRD only took actual charge of the project
after the first implementation year.

The result of the impact evaluation could have been considered objective
because it was conducted by a third party organization. However, in the set-up,
ASTI was the contracting party and at the same time the implementing agency
of PREGINET. As the funding and monitoring agencies respectively, DOST or
PCASTRD should have been the contracting party to ensure the impartiality of
the assessment.

Evaluation Objectives

In terms of assessing impact, the evaluation aimed to:

1. Determine whether the stated project goals and objectives were achieved;
2. Ascertain the contribution of the projects to the overall program objectives;
3. Assess the benefits of the projects;
4. Determine conditions under which the projects are most effective;
5. Identify the factors that contributed to the projects’ success or failure;
6. Recommend possible projects that can be pursued in this field and
suggestions to improve performance of such projects included.

Evaluation Methods

There were two major methods used in evaluating the projects’ impact – the desk
review and the field assessment. For the desk review, relevant and existing project
and project-related documents were analyzed mainly to reconstruct the project
design and to track the project results and accomplishments. The preliminary
evaluation which resulted from the desk review was then validated

DAP Findings
P R E G I N E T E V A L U A T I O N R E P O R T

DAP evaluated the project in terms of four major areas – project design, delivery,
impact, and sustainability. According to DAP:

• The project objectives were attained because it was able to:

d. advance the development of next generation technologies, applications
and services and made these accessible for user communities;
e. provide a platform that enabled the introduction, development, testing and
deployment of new network technologies; and
f. build up its partner institutions from the academe, government, and
research institutions with around 169 partnerships established as of
August 2008.

•It remained focused on the objectives it set to achieve. Thus, it was able to
establish a national REN, develop new technologies, and build the capability
of human resources to support the achievement of these objectives.

• Some of the factors considered helpful to the project included:

a. International recognition and support that enabled the country to become
integrated with the global research community
b. Strengthened partnership with international research and education
networks that enhanced connectivity to the research and education
community
c. Continuing R&D activities to keep abreast with current trends in advanced
network technologies
d. Dynamic and young staff who were flexible, open to learning, and willing to
spend extra hours to deliver the project
e. Support of DOST
f. Advocacy on the benefits of the project

However, certain conditions should be improved such as optimizing some

internal systems, e.g. after-sales service, billing system, documentation of
processes, and policies for the use of PREGINET services.

• The overall assessment for PREGINET’s project design is summarized

below.
a. The intent for the PREGINET project was clear from the start, except for the
quantifiable and measureable targets that should have been set at the
start. This was confirmed by the documents reviewed as well as the results
of the field validation where the key informants consulted cited the
following as bases for identifying the services, products and technologies
promoted by ASTI:
vii. Anticipated and visible need of the country in network technologies,
which is usually not pushed or probed by the private sector
viii. Need for the country to have an R&E network (REN) to be able to
ix. Participate or have access with the research activities of other RENs
around the world
P R E G I N E T E V A L U A T I O N R E P O R T

x. Needs and requirements of the user communities of PREGINET in line

with the thematic areas being focused by ASTI, namely environment,
education, government, and enterprise development
xi. Focus of PREGINET is the development and adoption of advanced
networking technologies that can benefit PREGINET partners.
xii. Services provided are identified based on the technologies and
applications used to promote the development of advanced networks.

b. Guided by the strategic objective “to establish a nationwide broadband

network that will link research and education institutions for the
development, testing, and deployment of next generation network
technologies, services, and applications intended for distance
education, agriculture, disaster mitigation, telemedicine, and
bioinformatics”, the PREGINET project employed strategies that were
found to be responsive to identified needs. These areas are as follows:
a) acceleration of government connectivity, b) development of IT
manpower base, c) promotion of technological innovation and
experimentation, and d) creation of strategic partnerships among local
institutions. The project, however, could have been optimized had it
been promoted extensively. It seemed that the infrastructure was
already provided by the project but content development was
inadequate. In addition, the project’s original design did not include
strategies for sustainability. It was during the latter part of the project
that sustainability measures and mechanisms were incorporated into
the design.

DAP Recommendations

f. Strengthen advocacy activities to promote the benefits of the

PREGINET. In spite of the attainment of project objectives there is still a
lingering perception that the partners have not fully realized or
appreciated the benefits of the network.
g. Actively solicit content from the partners. While there is an existing
infrastructure, the lack of content to offer to partners should be
addressed
h. Seek the involvement of other stakeholders for advocacy such as
the executive and legislative branches of the government. The PREGINET
project has been funded from the national government budget so it
should be supported by the rest of the government sector to optimize its
usefulness.
i. PREGINET should speak for itself. It should publicize its
accomplishments in the form of stories, case studies, and best practices.
j. There is no substitute for better service. Partners who were highly
satisfied with PREGINET's technologies, applications and services could
give testimonials to inform and invite others to the network.
P R E G I N E T E V A L U A T I O N R E P O R T

ANALYSIS

How is the performance of the network?

Did DAP conduct an interview with the users of the REN?
How was the service offered to its partners? Was it marketed aggressively?
Are there LGUs using the network?
What is the criteria for becoming a partner?
What were the actual applications developed?
But public appreciation for the PREGINET is still somewhat low. Thus,
there is really a need to promote the benefits of the network. What are they
doing about this?
Further, the technical competency of the staff involved in the implementation
of PREGINET is being continuously upgraded through exposure and
participation in gatherings where application of the ICT technologies are tested
and demonstrated. There is an expressed need, however, during the data
Final Report
26
gathering phase to further strengthen the R&D group of PREGINET particularly
on identifying services and technologies that would be beneficial. What are
they doing about this?

Is there a target no. of institutions per year that will become a partner?
No measurable objective
What new technologies were developed?

However, certain conditions should be improved such as optimizing some

internal
systems, e.g. after-sales service, billing system, documentation of processes,
and
policies for the use of PREGINET services.
The project has definitely opened opportunities for ICT development and
advancement in the country. Through the development and testing of
applications
in new areas, several network and information and communication
technologies have
been enhanced with PREGINET.
PREGINET has made its mark with the TELCOs. The growing appreciation of the
TELCOs on the importance of PREGINET role is itself recognition of the benefits
the project has brought about in helping the advancement of network
technology
P R E G I N E T E V A L U A T I O N R E P O R T

and strengthening the network infrastructure. What does this mean?

4
Conclusion
-B

CONCLUSIONased on our analysis of the PREGINET Project, we arrived at the

conclusion that the project was unsuccessful and did not have a significant
impact on R&D. This is based on the
fact that if we were to evaluate the project and focus on the R&D aspect, the
following will be the results:

• Only one network technology, a monitoring tool called

Pawikan was developed but it was not commercialized rendering
the project failed in terms of R&D output.
• In terms of cost-efficiency, the project failed as evident in the fact that
during the 7-year project duration, there were no successful new
technologies developed and the research activities were mere adoption
of available and matured technologies that does not justify the P79M
project cost.
• The project lacks effective monitoring tools and mechanisms to
determine the R&D output of partner-beneficiaries such as research
papers published and patents granted, if there were any.
• The project did not generate enough revenues to maintain its operations
or at least recover the government’s investment.
• PREGINET services have no competitive advantage over other
commercially-available services, as it does not provide unique features,
competitive rate or better performance.


 Research done over the project duration are not classified as
advanced network technologies, but rather adpatation of foreign
technologies. It cannot be considered a success in advancing new, next
generation network technologies for the Philippine Government,
P R E G I N E T E V A L U A T I O N R E P O R T

5
Academe and Private firms. The lack of measures and Key performance
indicators for R&D, both for PREGINET and its PARTNERS, led to non
measurement of R&D output, papers, research, patents. Morever, it
cannot justify DOST’s return of Investment on the project in this 7 year
project.
 In relation to this, there should have been new products and
services that resulted from R&D that can benefit society or prove to be a
better alternative for Filipino users. But seeing how there was a lack of
technology diffusion, lack of commercialization and there is still
preference for private network providers, PREGINET was not successful in
this aspect.
 Low monetary benefits affect the work motivations of Researchers
and thus, led to increase turn-over of staff. An offshoot of high turnover
and lack of SOP regarding documentation and knowledge transfer is the
non-deployment of some projects when researchers resign.

List of References

Appendix

Appendix 1. List of Research Areas duingduring PREGINET project

• IPv6 - the standard for the next generation Internet, and which addresses the
limitations of the current Internet protocol, IPv4;
• Multimedia over IP - which will deploy H.323 and multimedia (voice and video)
streaming as standard network services for the research and academic
community to provide distance learning, telemedicine, and other uses of
videoconferencing and multimedia streaming;
P R E G I N E T E V A L U A T I O N R E P O R T

• Network Measurements - which involve the use of network monitoring tools

and systems to measure the performance of various services and applications
under test as well as to monitor network health;
• Network Infrastructure - which deals with the establishment of the PREGINET
nationwide network and the study of different wire and wireless broadband
network technologies; and
• Content and applications development, hosting and distribution – which
includes the provision of high-availability servers and computing systems for
scientific applications that shall be made available for use by the partner
institutions for their projects over PREGINET.

Appendix 2. PREGINET Partners