Susanna Hall IST611, Professor Pam Berger Assignment 2, Task #1 Comprehensive Evaluation Report April 1, 2011

Title of website: URL: Evaluation Consultant: Contact:

Rye Middle/High School Media Center http://media-center.ryecityschools.schoolfusion.us Susanna Hall, Syracuse University MSLIS student susanna.me.hall@gmail.com

Goals/Learning Objectives Demonstrate the ability to evaluate a technology used in an educational environment. Report evaluation results and offer constructive suggestions for improvement.

General Description of the Website The Rye Middle/High School Media Center Web page is housed within the official website of the Rye City School District in Rye, NY (http://ryeschools.org/). Its purpose is to inform the students and staff of Rye Middle School and Rye High School about the resources available at their shared library. It is a single, static page without access to menu bars or sidebars for navigation. A sans-serif font is used consistently, with some color variation, on a white background.

The content on this page, from top to bottom, includes: A small panoramic color photo of the library facility Address, phone, and hours information A welcome paragraph with basic collection and facility information

 An announcement about the AccessMyLibrary iPhone app (in bold orange font with a small linked icon through which users can download the app) A Subscription Databases text box with Internal Access and External Access links in red A Library Links text box with headings in red text: four expandable headings and five nonexpandable links. The five latter links include one-sentence annotations. To the right of these two text boxes, in a narrow right column, are two final sections: Library Files contains folders titled High School Research Resources and High School Summer Reading with Microsoft Word documents inside. Department Contacts includes three staff names with email & phone contact info, but no titles A number of hits counter

Unique features of this page: This Web page is a sub-page (called a Classroom Page or FusionPage) of the Rye City School District site, which uses a a template created by SchoolFusion, an educational website content management company. The sites header, main menu, sidebar, and footer remain static when Classroom/Fusion pages are being viewed on the site. Many overall design and functionality elements of the site (colors, logos, sidebar location, navigation, search, help, site map, download speeds, and translation options) are not within the control of the library staff, so any weaknesses in these areas would best be attended to by the districts website administrator or SchoolFusion itself.

Brief Description of the Evaluation The Rye Middle/High School Media Center Web page is being assessed using two evaluation instruments--The WebMAC Professional (version 2.0) (2004) and Content Validity Scale (1999)--designed by Dr. Ruth V. Small and Dr. Marilyn P. Arnone, professors at Syracuse Universitys School of Information Studies and Directors at the iSchools Center for Digital Literacy. Small and Arnones instruments are based on expectancy-value theory and have been tested and validated with hundreds of users (2008). The authors state, The Website Motivational Analysis Checklist (WebMAC) instruments evaluate themotivational quality of
2

websites. A website with high motivational quality contains those features that motivate customers to visit, explore, and return to a website. These instruments are intended for use by educators to evaluate the sites they use in their teaching or for student learning. The criteria measured by the thirty-two items on the WebMAC are unique in that they assess web content and design elements that are motivational along a value dimension of stimulating and meaningful content, as well as elements that determine expectation for success by being organized and easy to use. Other web evaluation instruments do not directly attend to the motivational quality of educational websites (for instance, the A&TRT Website Scorecard designed by Neal and Herzig, 2007). Assessing motivation is essential because educational websites should be viewed by student and staff users as stimulating and not simply functional. The WebMAC asserts that if website design and content are not attractive, eye-catching, pleasing, unique, interesting, and interactive--thus stimulating users to be curious and explore the site--a website cannot be considered awesome. The ten items on the Content Validity Scale do not directly measure the motivational aspects of a website. As an earlier iteration of the WebMAC instrument, eight out of the ten items on the Content Validity Scale clearly overlap with items on the WebMAC. Therefore, this report will attend only to the items that are unique to this instrument, namely: #4: If the website presents concepts or principles in its domain (e.g. science, art), they are appropriately presented without confusing or missing information. #6: The content is appropriate for the intended audience.

Evaluators and Methods The three evaluators were Susanna Hall, MSLIS student, teacher, Boston Public Schools; Consultant A, College Counseling Coordinator, Boston Public Schools; and Consultant B, Director of Communications at a Boston nonprofit organization. The evaluators independently assessed this Web page using the two instruments. The steps taken to complete the WebMAC instrument are as follows: First, for each of the thirty-two items listed, the evaluator chooses a number to rate that item in relation to the website. A 3 means they strongly agree, a 2 means they somewhat agree, a 1 means they
3

somewhat disagree, a 0 means they strongly disagree, and NA means that the item is not applicable to the website. If there are NA items, the evaluator is asked to go back afterwards and mark each one using a 0-2 number scale that rates whether or not the website could benefit by including that item. Once the items all have a score, the scores are transferred evenly into four columns and totaled to get the SMOE scores--the totals that indicate how stimulating, meaningful, organized, and easy-to-use the site is. These four numbers are then transformed onto a SMOE graph [my title] that allows the evaluator to see the overall motivational pattern of the website--is it highly motivating, needs some improvement, or needs much improvement, and in which areas? The scores are then combined and plotted on a X/Y grid in order to get an overall motivational quality score for the site. Finally, there are two yes or no questions and two open-ended questions about the strengths and areas for improvements of the site. The Content Validity Scale is a much simpler instrument, with no advanced scoring or graphing functions. It is a simple list of ten items that are to be rated on a scale from 1-5, in a range of strongly disagree (1) to strongly agree (5). Evaluator results were sent to Susanna Hall and were added and plotted according to the directives of the WebMAC instrument. Brief phone conversations between Susanna Hall and consultants A and B allowed for a discussion of evaluators responses to the yes or no questions and thoughts about the websites greatest strengths and areas for improvement. The two unique items on the Content Validity Scale were also discussed.

Synthesis of Overall Findings The three evaluators of the Rye Middle/High School Media Center Web page reported remarkably consistent results on both instruments. Before reporting on the item results of the WebMAC and Content Validity Scale, here are the overall findings from the four questions posed at the end of the WebMAC instrument:

There is enough at this website to warrant a return visit. YES. The evaluators imagined they were staff and student users of the Web page, and agreed that these users would be likely to return to the page because it is the repository of the online resources they need access to on a regular basis for academic research: the library catalog, subscription databases, and other online resources. They may also return to get contact information for the library and its staff.

I would recommend this website to my colleagues. NO. The Web page, beyond providing needed content for its intended users, does not stand out in terms of stimulating or interactive content or design.

What is the greatest strength of this website? This is a utilitarian Web page with a very clear and consistent informational purpose and no redundant or unnecessary content. The content and links posted on the page have been thoughtfully chosen and annotated. Students and staff can easily access their library catalog, nearly sixty subscription databases, four local library sites, thirteen useful internet sites, and an online dictionary and Purdue OWL writing resource links. They can easily find contact information for their library and link to the AccessMyLibrary School Edition iPhone App.

What needs improvement at this website? This Web page is not very stimulating, particularly for student users. It contains only one photograph, does not contain any interactive elements, and the annotations may not be simply or engagingly written enough for middle school users. The resources effectively address a wide variety of academic research needs, particularly for older students, but there is room to grow. For example, there are no resources related to independent reading or writing (i.e. popular reading titles or creative writing links), technology or new media tools (i.e. Photoshop or social networking tutorials), or copyright. Though middle school appropriate resources are included (click on Subscription Databases > Internal Access > Encyclopedias to find leveled encyclopedia links), many resources are likely to be too advanced for middle school users (i.e. the Purdue Writing Lab).
5

WebMAC Professional (version 2.0) Results The WebMAC instrument evenly divides its thirty-two items into four categories, labelled S for how stimulating the website is, M for how meaningful the website is, O for how organized the website is, and E for how easy to use the website is. The numbered scores for each item are plotted onto a line graph. The results for the Rye Middle/High School Media Center Web page are graphed below:

These results show that the Rye Middle/High School Media Center Web page is highly motivating in the measures of Meaningfulness and Organization and needs some improvement in the measures of Stimulation and Ease of Use.

Notes on Meaningfulness and Organization: The carefully chosen and annotated research resources are credible and have the potential to become meaningful and valuable for high school and adult users. They are current and unbiased. The Web page designers have a solid command of how to organize content effectively within the link and files boxes provided in the SchoolFusion page template.
6

Notes on Stimulation and Ease of Use: The layout of the site could be more stimulating, as has already been mentioned. In terms of ease of use, this page received some low scores because it does not have a help button or a search function. (This is the fault of the SchoolFusion template--a help element is not included, and the search function does not search group, club, or classroom pages on the site.) Because many of the ease of use elements of this page are out of the control of the pages creators, the E score here is unfairly skewed. For this reason, the recommendations and suggestions for improvement below will mainly focus on the stimulation category.

The WebMAC instrument also requires evaluators to plot their SMOE number scores on a scatter plot with X and Y axes in order to get an overall motivational quality score. Here, the S and M numbers are added to create a Value dimension score (31 total) and the O and E numbers are added to create an Expectation of Success dimension score (36 total). The averaged results for the Rye Middle/High School Media Center Web page are plotted below (only the upper right quadrant of the plot is shown):

These results show that the Rye Middle/High School Media Center Web page is an Awesome Website in terms of its overall motivational quality. When added and plotted, the pages high scores in the meaningfulness and organization measures somewhat mitigate its lower scores on the stimulation and ease of use measures.

Content Validity Scale Results (for two unique items only) #4: If the website presents concepts or principles in its domain (e.g. science, art), they are appropriately presented without confusing or missing information. On average, the evaluators gave this item a neutral score of 3, since this Web page is not subject-specific in the way that a content teachers page would be, but it does provide subscription database links that are helpfully organized into content areas. #6: The content is appropriate for the intended audience. On average, the evaluators gave this item a 2, which amounts to a somewhat disagree score. The majority of the content seems to have been chosen to meet the research needs of high school students and adults. Also, the annotations are accessible to skilled readers, but will likely be difficult to understand for lower skilled readers and middle school students.

Recommendations/Suggestions for Improvement Although it is challenging to design a stimulating and comprehensive library website within the confines of one static Web page, it can be done, and the Rye Middle/High School Media Center can get there by getting creative and using the additional FusionPages functionalities that are available.

Design suggestions to improve stimulation and curiosity: Currently this Web page uses the links and files functionalities, but it can also use announcements, discussion topics, a library-specific calendar, blog, wikis, podcasts, slideshows, streaming video, and CIPA compliant messaging (CIPA is the Childrens Internet Protection Act). User interactivity and engagement can improve by adding just one of the following tools:
8

discussions, a blog (with post comments enabled), a wiki, and/or the ability to instantly message or chat with a librarian. Visual stimulation and curiosity can be piqued by using a slideshow, a podcast, or a video link. Perhaps a video tutorial on how to navigate a commonly used database could be made using free Jing screencasting software. Additionally, the library photo should be made larger if possible to create a more engaging header for the page. This is a gorgeous panoramic shot of the library, and it displays a welcoming and well-organized facility, but it is currently quite small and its details are hard to discern. Finally, adding small photos of the library staff, with names and job titles in captions, would add to a welcoming atmosphere for the page.

Content suggestions to improve stimulation and curiosity: Add fun resources that meet additional information needs of student users. Independent reading links could lead users to book lists with popular titles and genres (Amazon or Goodreads lists, for example). Students who write poetry or fan-fiction could be directed to sites through which they can publish their work, and students who create images or music could be directed to free, web-based image and audio editing tools. Students could learn about respecting copyright notices on others work and applying Creative Commons licensing to their own creative work.

Suggestions to improve appropriateness of content for middle school users: Page designers might consider creating a new link box with resources that specifically target middle school student users; the links could be made more engaging by inserting eyecatching thumbnail images. More appropriate academic writing links could be added to this box as well, since the Purdue OWL site is not aimed at a middle school audience. The language and length of the resource annotations should also be simplified and made more engaging for middle school users.

Two overall design suggestions: All of the links on this page are located below the fold of the browser (see Figure 1 on page 11), which means that users are required to scroll down the page before they can access a resource. Effective Web pages should include the most important and/or frequently used resources above the fold so that users can see and access them right away. To this end, this page should be redesigned so that the library catalog and database links are prominently
9

displayed at the top of the page. The welcome paragraph could be shortened or moved lower on the page. It takes up a lot of real estate above the fold, and will mainly be of interest to adult visitors to the site. Lastly, in the August-September, 2009 edition of the ALA-published journal Library Technology Reports, David Lee King, a public library branch manager from Kansas, contributed a seven chapter report filled with excellent recommendations for library website designers, titled Building the Digital Branch: Guidelines for Transforming Your Library Website (accessible via the Gale General OneFile database). This report argues that a library website should be a vital, functional resource for patrons, and its emphasis is on creating a user-friendly website that includes some of the interactive opportunities provided by Web 2.0 tools. The chapters begin with the initial needs assessment and design stages and then move on to the creation, enhancement, and assessment of a library website. The three chapters that are most appropriate for the Rye Middle/High School Media Center Web page at this time are Chapter 4: Building the branch, Chapter 5: What to do once your site is built, and Chapter 6: Creating community at your digital branch. Mr. King is also presenting a online workshop on this topic through ALA Techsource on June 8, 2011 ($50 to register).

Reflection on the Experience Evaluating library websites allows us to plan for, reflect upon, and continually create a robust digital experience for our online users, and the WebMAC instrument is a valuable tool for this job. Of course, there are other criteria for designing effective and engaging websites, but the WebMAC gets at some of the most important criteria related to user-friendliness and motivation. Neal and Herzig (2007) also suggest measuring ADA compliance, the good first impression a website can create, elements that demonstrate community orientation, and search engine optimization. As websites move from their initial versions to their revised and enhanced versions, usability testing is another area that should be researched so that librarians can observe how users are actually navigating through and using their site.

10

FIGURE 1

Above the fold content as of 3/13/11

Above the fold content as of 3/13/11 Above the fold content as of 3/13/11

Below the fold content as of 3/13/11

11

References and Resources

King, David Lee. "Building the branch." Library Technology Reports 45.6 (2009): 16+. General OneFile. Web. 13 Mar. 2011. Retrieved from http://find.galegroup.com/gtx/infomark.do? &contentSet=IACDocuments&type=retrieve&tabID=T002&prodId=ITOF&docId=A207461275&source=g ale&srcprod=ITOF&userGroupName=massb&version=1.0 King, David Lee. "Creating community at the digital branch." Library Technology Reports 45.6 (2009): 30+. General OneFile. Web. 13 Mar. 2011. Retrieved from http:// find.galegroup.com/gtx/infomark.do?&contentSet=IACDocuments&type=retrieve&tabID=T002&prodId=ITOF&docId=A207461277&source=g ale&srcprod=ITOF&userGroupName=massb&version=1.0 King, David Lee. "What to do once your site is built." Library Technology Reports 45.6 (2009): 23+. General OneFile. Web. 13 Mar. 2011. Retrieved from http://find.galegroup.com/gtx/ infomark.do?&contentSet=IACDocuments&type=retrieve&tabID=T002&prodId=ITOF&docId=A207461276&source=g ale&srcprod=ITOF&userGroupName=massb&version=1.0 Neal, D., & Herzig, C. (2007). A&TRT's website scorecard: Evaluate (& improve) your library's website. Texas Library Journal, 83(2), 56, 58, 60-1. Retrieved from Library Lit & Inf Full Text database Small, R. & Arnone, M. (1999) Content Validity Scale. Retrieved from http:// informationliteracy.org/users_data/8442/IST61111.ContentValidityScale.Fall2006.doc. Small, R. & Arnone, M. (2008). Website Motivational Analysis Checklist (WebMAC) and Content Validity Scale. Retrieved from http://digital-literacy.syr.edu/projects/view/59. Small, R. & Arnone, M. (1999, revised 2004). Website motivational analysis checklist (WebMAC) Professional (v. 2.0). Retrieved from mauer.wikispaces.com/file/view/WebMAC+Pro+2.0+with+scoring.doc

12