Web Literacy Workshop

Arnold Gamboa Tona Gardner Jo Ann Kobuke Tonya Schauwecker

Purdue University EDCI 57200 Group Design Portfolio Project December 2, 2012

Contents
Introduction ............................................................................................................................................ 1 Front-end Analysis: .............................................................................................................................. 1 Goal Statement ....................................................................................................................................... 1 Goal Analysis ........................................................................................................................................... 1 Subordinate and Entry Skills .................................................................................................................... 2 Identification of Level of Entry Skills ........................................................................................................ 5 Learner Survey Results and Summary ...................................................................................................... 6 Learner Survey Context ....................................................................................................................... 6 Summary of Learner Survey Results ..................................................................................................... 7 Learner Characteristics Chart ............................................................................................................... 8 Performance Context Chart ................................................................................................................... 10 Performance Context......................................................................................................................... 11 Learning Site Constraints ................................................................................................................... 11 Learner Context Chart ....................................................................................................................... 11 Design Evaluation Chart for Subordinate Skills, Performance Objectives, and Parallel Test Items ........... 12 Instructional Goal in Context ............................................................................................................. 12 Terminal Objective ............................................................................................................................ 12 Parallel Test Item ............................................................................................................................... 12 Design Analysis Charts ....................................................................................................................... 13 Instructional Design Strategy ................................................................................................................. 20 Objective Sequence and Clusters ....................................................................................................... 20 Pre-instructional Activities ............................................................................................................. 20 Content Presentation and Learner Participation ............................................................................ 21 Assessment.................................................................................................................................... 22 Follow-Through Activities .............................................................................................................. 23 One-on-One Formative Evaluation Process and Analysis ................................................................ 29 Small-Group Formative Evaluation Process and Analysis ................................................................ 31 Assessing Learner Attitudes about Instruction ............................................................................... 31 Observations and Proposed Revisions ................................................................................................... 33 Appendices............................................................................................................................................ 35 Appendix A Online Learner Survey .................................................................................................. 36 1

Appendix B Pretest and Results....................................................................................................... 37 Appendix C Investigating Sources Worksheet (Blank) ...................................................................... 38 Appendix D Investigating Sources Rubric......................................................................................... 39 Appendix F Results of Sample Practice Test ..................................................................................... 41 Appendix G - Post-Workshop Evaluation ............................................................................................ 43

Introduction
A local school library needs to present a workshop to middle school students on how to determine the quality of information available on the Internet.

Front-end Analysis:
A required component of advancing students into twenty-first century skills according to the College and Career Readiness (CORE) Anchor Standards for Writing is that students must be able to demonstrate a variety of technological skills, including determining the quality of Internet information. Currently, middle school students do not conduct research using reliable Internet sources, often citing biased, nonacademic websites and thus need training in proper Internet research techniques. The front-end analysis identified a performance gap. Students did not have the requisite analytical skills to identify reliable Internet search results. The designers task was to construct a one-hour workshop to close this performance gap.

Goal Statement
Middle school students will be able to use specific criteria to determine the quality of information found on the Internet.

Goal Analysis
Middle school students will be able to use specific criteria to determine the quality of information found on the Internet.

Analyze Results of Internet Boolean Search 1

Use or Discard Results Based on Analysis 2/3

Subordinate and Entry Skills

The following diagram outlines the subordinate skills addressed by the instructional goal, including entry skills that students are assumed to possess. The decision tree is demonstrated at A in the second diagram and B in the third diagram, both showing the analysis flow students will be taught.

Create and Launch Boolean Search 1

Analyze Results

Entry Skills Below the Line

Touch Typing Proficiency

Ability to Use a Web Browser

Ability to Use a Search Engine

Ability to Create a Boolean Search

Know What a Browser Is

Know What a Search Engine Is

Know What a Boolean Search Is

Subordinate and Entry Skills (Continued)

Discard Results
5

Yes

Are Results Obviously Unsuitable ?

No

List of Reasons Results may be Unsuitable Paid Sponsors, School Children as Authors, etc.

Discard Results
5

No

No

No

No

Yes

No

WHO Is there an author and what are his credentials?

WHY WHEN Is the information current? Is author writing article? Is there bias?

Yes

Yes

Yes

Yes

No

WHAT Does article add to research?

WHERE Is there authority for information?

HOW Is the information supported with references?

List of Reasons Results may be Unsuitable Paid Sponsors, School Children as Authors, etc.

Use Results
6

Yes

Identification of Level of Entry Skills

Four areas of entry skills were identified. They were typing skills, use of web browsers and search engine, and knowledge of Boolean search expressions. Typing Skills: Students have adequate typing skills. These skills were taught in the sixth grade with

85% of the student achieving the districts goal for corrected typing speed.
Proficiency with web browsers: Students have adequate knowledge of web browsers and their operation. They are familiar with the popular browsers such as Internet Explorer, Google Chrome and Firefox. The school district has Internet Explorer installed on all district computers. All sixth grade students are taught how to use web browsers in their computer lab classes. Proficiency with search engines: Students have adequate knowledge of search engines. An initial

study conducted in English classrooms found that students have been taught to use the major web search engines, such as Google, Yahoo, and Bing. All sixth grade students are taught how to
use search engines in their computer lab classes. Proficiency with Boolean search expressions: Students were trained in Boolean search techniques at the same time as instruction in search engines. Proficiency level is not an obstacle to instruction. After discussion with administrators and teachers, and conducting student surveys, designers concluded that the target student population had sufficient entry skills and thus a formal assessment of entry skills was not needed.

Learner Survey Results and Summary

Learner Survey Context
Students in grades 6, 7, and 8 at Horizon Science Academy Toledo were given an online survey of five questions to determine their previous knowledge of Internet quality, their desire to learn about Internet quality, and their preferred mode of instruction. A subgroup of 54 students, 18 from each grade level, completed the survey online as part of their bell work in three different computer classes. Surveys were completed online through Kwiksurveys.com, link available here and in Appendix A: http://kwiksurveys.com/s.asp?sid=xcyh3zwe5nqpaij57670#.ULlg6-A3v5o.gmail Learner Survey Results: 1. How often do you search the Internet for information?
Daily 44 Weekly 5 Monthly 2 Rarely 2 Never - 1

2. How often do you determine the quality of Internet sites before using the information? Never 17 Rarely 8 Often 6 When its for school 17 Always 6

3. How often have you learned about how to determine the quality of an Internet site? Yearly 6 All the time 3 Whenever a teacher gives a research assignment 18 remember 3 Never 24 I dont

4. When learning new information, how do you prefer the teacher presents the information?
Strongly Disagree Disagree Neutral Agree Strongly Agree

I prefer to read a handout or worksheet I prefer to listen to my teacher explain the information I prefer to read notes on the board I prefer to watch a video or PowerPoint I prefer to use the computer to do hands-on activities I prefer to read a book about the subject I prefer to have someone show me directly

22 times

20 times

1 time

6 times

5 times

15 times 52 times 2 times

13 times 1 time 6 times

1 time 1 time 1 time

9 times 0 times 26 times

16 times 0 times 19 Times

1 times

2 times

1 time

2 times

48 times

38 times

10 times

2 times

2 times

2 times

1 times

2 times

1 time

14 times

36 times

5. Would you like to learn how to determine if an Internet site is reliable? Strongly Disagree 10 Disagree 27 Neutral 10 Agree 4 Strongly Agree - 3

Summary of Learner Survey Results

The initial student survey concluded that students are not eager to learn proper search techniques or to differentiate between reliable or quality Internet sources and unreliable or poor quality Internet sources. Students feel that they will be able to master quickly the techniques they learn in the workshop and will be able to apply the new knowledge in future research projects.

Learner Characteristics Chart

Information Categories I. Abilities A. Entry Skills

Data Sources Needs assessment data from teachers Interviews with teachers Survey of students Interviews with teachers Survey of students

Learner Characteristics Students have adequate entry skills to participate in workshop instruction.

B. Prior knowledge of topic area

Students have little previous instruction determining the quality of Internet sources. Students in the group are average and below average in their reading Exile scores. Teachers indicate that students are computer literate. Students are engaged in activities that involve the use of computers. Students prefer to be active during the learning process as opposed to just watching or just listening. Students stated they are not interested in learning how to determine the quality of Internet sources but they enjoyed completing assignments with computers. Students indicated they would likely be interested in learning about quality Internet sources if they were actively using computers at the time of instruction. Students indicated they would enjoy working in the media center on computers.

C. Educational and ability levels

School records

Interviews with teachers

D. General learning preferences

Interviews with teachers and students

II. Attitudes A. Attitudes toward content

Survey of students

B. Attitudes toward potential delivery system

Survey of students

C. Motivation for instruction

Interviews with teachers and students

Teachers indicated students enjoy working with computers but that some students will not enjoy the program even in the media center. D. Attitudes toward training organization Interviews with students Students generally expressed neutral feelings toward teachers and media specialist. Some students had high regards while some had low. Group is generally not motivated to work beyond simple and quick operations. Group will do well with research subject matter that is interesting to their age group and preferences. While students did not acknowledge the benefits of the workshop at the time of the survey, they are capable and motivated to complete the task with engaging materials and resources.

III. General group characteristics

Overall impressions

Performance Context Chart

Information Categories 1. Managerial/Supervisory Support

Data Sources Interviews with: teachers administrators district media specialists

Learner Characteristics Teachers are enthusiastic about technology integration. Administrators are willing to support the workshop with class time allotment and resource availability. District media specialists are willing to help with the training and preparing computers for use during the workshop. District media center has 25 computer terminal stations and a working computer projector.

2. Physical aspects of site

On-site visit Students will work independently throughout the program using the computers in the media center. Discussion and interpretation of the presented example websites will allow social interaction.

3. Social and learning aspects of site

Interviews with teachers On-site visit

Computer, worksheets, pens or pencils, computer projection machine 4. Learning tools Common CORE requirements indicate that students must be computer literate by 2014, which includes being able to differentiate between reliable and unreliable internet sources.

5. Relevancy

Interview with teachers On-site visit

10

Performance Context
Administrators, teachers, and media specialists are eager to begin the workshop, as the new CORE standards will be fully enforced in 2014. Administrators have instructed the district media specialist to ensure all computers in the media center are fully functional and that a projector is available during the days of the workshop. Teachers and media specialists will work together to coordinate workshop scheduling to avoid conflicts and prepare worksheets for students to complete during and after the workshop.

Learning Site Constraints

Due to the number of students in the middle school and the number of available computer terminals in the media center, students will have to attend the workshop in groups of no more than 25. There are 18 classes of 25 students, which will cause scheduling conflicts with other classes needing to use the media center. The workshop will have to run six periods per day for three days.

Learner Context Chart

Information Categories I. Nature of site A. Learner setting

Data Sources On-site visit

Learner Characteristics District media center with 25 computer terminals. Teachers will provide students with rubrics for determining the quality of a website, a PowerPoint presentation of examples and a decision making process, and a follow-up rubric for independent research. Computers for individual students Computer projection machine Constraints - time

B. Resources

C. Compatibility 1. With learner needs

Interviews with teachers

Site is compatible with learner needs as students will conduct most post-workshop assignments in the media center or at home Teachers will conduct the workshop, ensuring important and relevant information is conveyed to students

2. With instructional needs

Interviews with teachers

11

Design Evaluation Chart for Subordinate Skills, Performance Objectives, and Parallel Test Items
Instructional Goal in Context
While using any computer (CN), middle school students will be able to use specific criteria (B) in order to determine the quality of information found on the Internet (CR). Type of Learning: Intellectual skill

Terminal Objective
Upon completion of a one-hour workshop conducted in the Horizon Science Academy Toledo media center (CN), students will conduct research on Narcissa Whitman (B) to determine the quality of Internet sources and their ability to apply new knowledge gained through the workshop (CR).

Parallel Test Item

Using the Investigating Sources rubric handout, conduct research on Narcissa Whitman and determine if four Internet sources available through an Internet search engine provide quality information. Remember to discard all sources from the following sites. Wikipedia Blogs Social media sites Question and answer forums Advertisements Student and class websites

12

Design Analysis Charts

Main Instructional Goal

The instructional goal of this workshop is to teach middle school students how to determine the reliability of data obtained from an internet search whenever performed regardless of location.

Terminal Objective
Middle school students will accurately determine the reliability of data obtained from an internet search whenever performed regardless of location.

Parallel Test Items

Students will be given a research topic and required to complete a rubric analysis for a web site using the Investigating Sources rubric. After completion, students will be able to state whether the web site contained reliable data for use in writing an informational paper on the assigned topic. Use Yes/No Checklist for the following questions. 1. Was a rubric for a web site submitted? 2. Did students provide evidence for all six elements? 3. Were evidence comments on point and relevant to section topic? 4. Did the students evidence support their conclusion about the reliability of the web site? Scale would range from 1 to 5 with 5 indicating mastery.

13

Main Step in Instructional Goal

Performance Objective for Main Step

Perform an internet search using Boolean operators. Obtain relevant search results. Choose a web site. Determine the reliability of the data obtained and the feasibility of its use as a resource.

Parallel Test Items

1. Analyze Results of Internet Boolean Search

A. Perform an internet search on Narcissa Whitman to identify a web site to analyze. B. Read the information on the web site. C. Complete the Big 6 Essential Questions rubric for the web site. D. Circle your conclusion based on your analysis.

14

Subordinate Skills
1.1 Determine what types of web sites are obviously unreliable for research use.

Performance Objectives
1.1.1 Identify obviously unreliable web sites.

Parallel Test Items

A. Students will orally provide examples of obviously unreliable web sites.

1.2 WHO: Determine whether author can be identified and credentials verified.

1.2.1

Identify the web site author. Determine if the author has the education, credentials, and/or experience to be an expert or authority.

1.2.2

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to WHO B. Authors name and professional accomplishments. If not available, write, none found. C. Educational institution, government agency, or business associated with the web site. If not available, write, none found. D. If information in WHO section is reliable, write Yes in first column. If not reliable write No.

15

Subordinate Skills
1.3 WHERE: If no author, determine whether sponsoring organization has authority for information on web site.

Performance Objectives
1.3.1 If there is no author, determine whether there is a sponsoring organization instead? Is it well known and respected?

Parallel Test Items

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to WHERE B. Is there a sponsoring organization? C. Is the organization well known and respected? D. If the organization is reliable, write Yes in the first column. If not, write No in the first column.

1.4 WHEN: Determine whether the information presented is up-to-date.

1.4.1

When was the site content created or last updated? Are the facts and figures up-to-date? Is the information recent enough to make it relevant/accurate?

1.4.2

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to WHEN B. When was the site created or last updated? C. Are the facts and figures up-to-date? D. If the information is upto-date, write Yes in the first column. If not, write No in the first column.

1.4.3

16

Subordinate Skills
1.5 WHY: Determine whether the writer has a bias (motivation).

Performance Objectives
1.5.1 Do the writer and/or the sponsoring organization have a bias (motivation) that may interfere with the accuracy of the information? Does the content fairly handle all sides (perspectives) related to the topic? Does writer present the information without exaggeration or misrepresentation?

Parallel Test Items

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to WHY B. Do the writer and/or the sponsoring organization have a bias (motivation) that may interfere with the accuracy of the information? C. Does the content fairly handle all sides (perspectives) related to the topic?

1.5.2

1.5.3

D. Does the writer present the information without exaggeration or misrepresentation? E. If the article is not biased, and therefore reliable, write Yes in the first column. If not, write No in the first column.

17

Subordinate Skills
1.6 WHAT: Determine what information this article adds to your research

Performance Objectives
1.6.1 Is the content useful for your writing purpose? Is the web site language understandable and appropriate for your writing purpose?

Parallel Test Items

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to WHAT B. Is the content useful for your writing purpose? C. Is the language understandable and appropriate? D. If the web is useful for your topic, write Yes in first column. Otherwise, write No.

1.6.2

18

Subordinate Skills
1.7 HOW: Determine how the author or organization supports their information.

Performance Objectives
1.7.1 Has the author or sponsoring organization supported the web content with references or links to other reliable sources? Has the author or sponsoring organization used supported evidence instead of emotional appeals?

Parallel Test Items

A. Go to the web site shown in the PowerPoint. Using the Investigating Sources worksheet, find the following and write in evidence next to HOW B. Do the writer and/or the sponsoring organization provide references or working links to other reliable sources? C. Do the references support or conflict with the web site information? D. Is the information supported by rational evidence instead of emotional appeals? E. If the article is supported, write Yes in the first column. If not, write No in the first column.

1.7.2

19

Instructional Design Strategy

Objective Sequence and Clusters
The instruction will occur in a one-hour session, with objectives assigned to seven clusters or lessons, in addition to the pre-instructional activities, the assessment, and follow-through activities.

Cluster Chart
Clusters 1-7 Instructional Goal Step: Main Step: Analyze Results of Internet Boolean Search Cluster 1 Objectives 1.1.1 Cluster 2 Objectives 1.2.1 1.2.2 Cluster 3 Objectives 1.3.1 Cluster 4 Objectives 1.4.1 1.4.2 1.4.3 Cluster 5 Objectives 1.5.1 1.5.2 1.5.3 Cluster 6 Objectives 1.6.1 1.6.2 Cluster 7 Objectives 1.7.1 1.7.2

Pre-instructional Activities Five minutes were allotted at the beginning of the class to introduce the subject of the workshop. The initial screen of the PowerPoint was on the front whiteboard. Students were instructed to read the information while the teacher distributed the handout that would be completed during the workshop. The teacher then explained the focus of the workshop, discussing the need to determine reliability of web site and content for use in preparing research papers. The teacher also pointed out the need for these skills in general life situations, for example performing research in advance of a major purchase such as a car or electronic gear. Motivation Student attention is captured by presentation of the initial PowerPoint screen . Interest is maintained by teachers interaction with students explaining the value of mastering the concepts being presented emphasizing how students will benefit, both in schoolwork at the high school and college level, and in life. Objective Students are told the instructional goal is to provide them with a strategy for analyzing internet research. The handout provided will walk them through the analysis steps and provide a basis for their decision to use or not use their search results. Entry Skills This student group is taught on a regular basis by the teacher, and thus she has firsthand knowledge of the students level of computer proficiency and general internet search skills. She deems them to be adequately prepared for the workshop. In addition, designers had previously determined that students had sufficient prior training related to required entry skills. Thus, no entry skill assessment was deemed necessary. 20

The teacher briefly reviewed how to enter the research question into the web browser to initiate an internet search to stimulate students to recall their prior training. There were no questions from students about initiating an internet search. Student Grouping - One of the designers teaches at a local middle school similar to that of the client. The designer/teacher agreed to use her classroom to conduct the formative evaluation of the workshop. Her class contains a target population similar in demographics to the clients urban middle school of 6 th 7th and 8th grade students. The two populations have similar ethnic and demographic characteristics. The evaluation class totaled 14 students. Media Selection- The workshop was presented in a computer lab at the designer/teachers school. The resources in the lab dictated the media selection. The computer lab used in the evaluation had similar resources to the clients computer labs, including a 1:1 ratio of computers to students, projector, and whiteboard. The teacher presented the instruction using a PowerPoint and a web site constructed so students could easily access the sites used during the presentation for evaluations (this eliminated search errors, conserving instruction time). Paper handout were used which were also posted online for later student access. Content Presentation and Learner Participation Cluster 1: Subordinate skill 1.1 & Performance Objective 1.1.1 = 5 minutes The teacher presents the information on types of web sites that are obviously unreliable, using PowerPoint slide shown at front of classroom. The teacher leads a short class discussion about need for reliable knowledge. The teacher hands out materials The teacher explains Investigating Sources worksheet Students orally provide examples of obviously unreliable web sites.

Cluster 2: Subordinate skill 1.2 & Performance Objective 1.2.1 1.2.2 = 5 minutes The teacher introduces the concept of the Six Big Questions to explain the analysis process. The teacher then presents information on the first question: WHO After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

Cluster 3: Subordinate skill 1.3 & Performance Objective 1.3.1 1.3.3 = 5 minutes The teacher then presents information on the second question: WHERE After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

21

Cluster 4: Subordinate skill 1.4 & Performance Objective 1.4.1 1.4.4 = 5 minutes The teacher then presents information on the third question: WHEN After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

Cluster 5: Subordinate skill 1.5 & Performance Objective 1.5.1 1.5.2 = 5 minutes The teacher then presents information on the fourth question: WHY After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

Cluster 6: Subordinate skill 1.6 & Performance Objective 1.6.1 1.6.3 = 5 minutes The teacher then presents information on the fifth question: WHAT After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

Cluster 7: Subordinate skill 1.7 & Performance Objective 1.7.1 1.7.2 = 5 minutes The teacher then presents information on the sixth question: WHY After listening to the teachers presentation and viewing related PowerPoint slides, students visit a preselected web site to practice analysis described. The appropriate site has been selected to demonstrate Lesson objective. Students complete the relevant section of the Investigating Sources worksheet.

A fifteen minute wrap-up will be given by the teacher to iterate the big six essential questions. The independent research assignment will be given and student questions will be solicited. Students then have until the end of class to complete the assignment. They will be instructed to do the assignment as homework if they are not finished by the end of the period. Assessment Pretest At the beginning of the workshop, students will be assigned a pre-test in the form of a Bell Work Worksheet to complete before the workshop instruction began (Appendix B). Embedded Practice Tests - Students will use the Investigating Sources Worksheet (Appendix C) to answer questions of reliability as they view the Investigating Sources PowerPoint (Appendix H). Embedded test questions will be used to determine if students need further instruction to determine if a 22

website is reliable. In the workshop, students will be informed that the terms quality and reliable will be used interchangeably in order to eliminate confusion. Posttest At the end of the workshop, students will be given a homework assignment to perform an internet search on a specified topic. They will receive an Investigating Sources Rubric (Appendix D) to remind them of the analytical procedures to employ. Students will analyze one web site that results from the search. Students will complete an Investigating Sources worksheet to evidence their work. Student Group and Media Selection Students will follow along with the PowerPoint. The teacher will pause periodically so students can use their computers to access the web site being discussed. Students will work independently to review the web site and complete the relevant section of the Investigating Sources worksheet using a pen or pencil. The teacher will answer any questions and continue with the next segment of the workshop. Follow-Through Activities Memory Aids -Students will complete an Investigating Sources worksheet, which includes an abbreviated version of the Rubric. Students will then be given a more detailed Investigating Sources Rubric, which will act as a memory aid to reinforce information provided in the workshop. In addition the handouts and PowerPoint will be posted on the schools student resources page. Performance Transfer The process students use to complete the Investigating Sources worksheet during the workshop mimics the process they will use to work on their own. The learning context mirrors the performance context. The students are working on computers, performing internet research and web site analysis can apply these skills in any location with a computer and internet access. Students will eventually be able to conduct research using quality Internet sources throughout their education without the use of the Investigating Sources worksheet, as the information will become embedded in their long-term memory stores. Media Selection Students will be provided a paper Rubric as a reference. In addition, the rubric, Investigating Sources worksheet and the workshop PowerPoint will be available on the schools student resource site.

23

Content

Content Presentation Examples and Nonexamples

Student Participation Practice Items Feedback

Objective 1.1.1 Identify obviously unreliable web sites. Automatically unreliable Internet sources are those that are written by any person of the greater community with no credibility. WHO: Objective 1.2.1 Determine the author of a website Objective 1.2.2 Determine if the author has the education, credentials, and/or experience to be an expert or authority. Websites without authors cannot be determined to be reliable. Keep looking for more information. WHERE: Objective 1.3.1 If there is no author, determine whether there is a sponsoring organization instead? Is it well known and respected? Websites that do not explain where their information came from or their authority on the subject should be rejected. See PowerPoint slide 6 for examples Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. Ask students for examples of from the recent election that would tell them the source is unreliable. See PowerPoint slide 7 for unreliable source. See PowerPoint slide 9 for reliable source. Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. When students correctly identify each slide as reliable or unreliable use encouraging words. Wikipedia, blogs, question and answer forums, class and student web pages, social media sites, Ask students to identify websites they know to be unreliable. Remind students that anyone can create a website or contribute to a forum; therefore, knowing where to start will be helpful

24

Content

Content Presentation Examples and Nonexamples

Student Participation Practice Items Feedback

WHEN: Objective 1.4.1 When was the site content created or last updated? Objective 1.4.2 Are the facts and figures up-todate? Objective 1.4.3 Is the information recent enough to make it relevant/accurate? The date of publication is important depending on the topic of the research assignment. See PowerPoint slide 12 for example that would not be reliable due to the information being outdated, though previous applications of the rubric would make it reliable. Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. Explain to students that historical research topics could use older information, but research topics that have time-sensitive issues, should be current. Ask for topics that my need time-sensitive research. When proper examples are given, reply with encouraging words.

25

Content

Content Presentation Examples and Nonexamples

Student Participation Practice Items Feedback

WHY Objective 1.5.1 Does the writer and/or the sponsoring organizations have a bias (motivation) that may interfere with the accuracy of the information? Objective 1.5.2 Does the content fairly handle all sides (perspectives) related to the topic? Objective 1.5.3 Does writer present the information without exaggeration or misrepresentation? Websites that only give an opinion of their side of an issue or that have ulterior motives can create misinformation. Websites such as these should be rejected. See PowerPoint slide 14 for unreliable source. Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. Ask students if they know of another source that might talk about farming in an unbiased way. If students cannot think of one, ask them what site extension they should look for. When students answer correctly, give encouraging praise.

26

Content

Content Presentation Examples and Nonexamples

Student Participation Practice Items Feedback

WHAT Objective 1.6.1 Is the content useful for your writing purpose? Objective 1.6.2 Is the web site language understandable and appropriate for your writing purpose? Some information on websites, even if they are respectable websites, is not appropriate for all grades, ages, or assignments. See PowerPoint slide 17 for example that would not be useable. Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. Explain to students that complicated language increases the risk that information will be misunderstood and possible used incorrectly.

27

Content

Content Presentation Examples and Nonexamples

Student Participation Practice Items Feedback

HOW Objective 1.7.1 Has the author or sponsoring organization supported the web content with references or links to other reliable sources? Objective 1.7.2 Has the author or sponsoring organization used supported evidence instead of emotional appeals?

Websites that do not link to other credible sources should be rejected.

See PowerPoint slide 20 for reliable source.

Direct students to the Investigating Sources Worksheet to fill in the appropriate blank. Explain to students that teachers want them to cite their sources for the same reasons we want websites to site their sources. It helps us to determine if the site is reliable.

Ask students if a source can still be credible if they do not cite sources. Use thinking strategies and previous questions from the rubric to help determine if the site should be used.

28

One-on-One Formative Evaluation Process and Analysis A one-on-one evaluation session was performed with a student volunteer. The purpose was to obtain a first impression from the student about the workshop and quickly flush out any significant areas of weakness. Materials for the one-to-one evaluation - A PowerPoint presentation was used to walk student through concepts. Slides projected major concepts and provided web site address for sites used in the presentation. Screen shots of web sites were included as a focal point for students as the teacher demonstrated the analytical process. Handouts consisted of a worksheet to be completed during the presentation, and a rubric for future use. Participant and Instruction - The student volunteer for this one-on-one phase was a female twelfth grade student. Since she was already a student of the teacher, she was comfortable and at ease during the presentation. While she is older than the target population, this was not considered a hindrance, as she was able to provide more detailed and articulate feedback that aided in evaluation. The teacher did take into account this age difference when evaluating her responses. The reasons for the evaluation were explained to the student, who understood her feedback was being requested to assist in determining workshop quality and viability. Due to time constraints only a single one-on-one session was performed. Pre-instructional Materials and Content Presentation - The teacher presented the PowerPoint and the student was asked to complete the worksheet during the presentation. At the conclusion, she was asked to respond to questions such effectiveness of timing, pace of instruction, clarity of concepts, among other things. Student Participation - Student participation took the form of a review of the web sites presented in the PowerPoint and completion of the analysis worksheet. The teacher reviewed the worksheet to identify any errors that might indicate a weak area in the instructional materials. The student made no errors on the worksheet. This was not completely indicative of a lack of weakness in the instructional materials, as the student was older than the target population, and thus more likely to better comprehend instruction. It is anticipated that the younger target population will provide a better measure of this element. Assessment At the conclusion of the workshop, the student indicated that the PowerPoint was too wordy or boring and that the concepts were not that hard to get. The teacher attributes this in part to the students age. She anticipates that it will not be so easy for the target population; however, she did note the boring and too wordy comments. The teacher found during the presentation that combining the Who and Where question on the same PowerPoint slide would be confusing to the target population and noted it for revision. Due to time constraints change to the PowerPoint were not possible before the small group evaluation. However, the teacher made notes about revisions she would implement during the small group evaluation.

29

1.

Although the PowerPoint slide could not be changed teacher would note that the two questions of Who and Where were separate questions focusing on different elements.

2. The teacher, noting the students comments about long and boring, and too wordy, identified PowerPoint slides that could be skipped to improve pacing. The teacher tightened up her verbal presentation to be more succinct and less wordy. 3. Rather than reading the initial descriptive first screen, which set the tone for the workshop, the teacher opted to display the initial screen and instruct the students to read it while the students were arriving and handouts were distributed. 4. Students were instructed to visit the web sites being discussed in each PowerPoint slide at the time of the discussion, rather than at the end of the workshop as was done in the one-on-one evaluation. 5. The teacher determined that the web address for the web site, which contained the PowerPoint web site address, was too long and cumbersome. The student made typing errors when trying to access the site. The teacher created a Google short URL for students to use in the small group evaluation.

30

Small-Group Formative Evaluation Process and Analysis Materials used in Small-Group Evaluation - Students completed the bell work pretest activity, viewed the Investigating Sources PowerPoint and simultaneously completed the Investigating Sources Worksheet. The students completed the session with an independent research assignment using the Investigating Sources Rubric to complete an Investigating Sources worksheet. Participants and Instruction Students in the test group are tenth grade English II students with limited reading comprehension, analytical, and writing skills. The test group and the target group have equal Reading Lexile score ranges (5.3-9.0), comprehension, analytical, and writing skills. Small group testing on this audience is therefore acceptable. Pre-instructional Materials and Content Presentation Students were asked to complete a bell work pre-test worksheet to assess preliminary extent of student knowledge in subject being taught. Results are included in Appendix B. In addition, students read the first screen of the PowerPoint to set the stage as handouts were distributed. The teacher introduced the topic and explained the small-group evaluation approach informing the students that their feedback would assist in workshop analysis and potential revisions. Assessment During the workshop students completed an Investigating Sources worksheet with information from two web sites show in the PowerPoint. The results of those practice tests are included in Appendix E. The workshop was designed to provide 15 minutes of time at the end for students to perform an independent research assignment. Due to additional time being spent in classroom discussion, the students did not have time to begin the assignment before the end of class. The assignment was given as homework instead. Students were to return the completed homework assignment the next day. At the end of the workshop, students were to complete a post-workshop evaluation available at https://docs.google.com/spreadsheet/viewform?formkey=dEFKMmlrd2hZX1l0b0huVVJmaTZxYnc6MQ (Appendix G) Assessing Learner Attitudes about Instruction

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

I gained a good understanding of the subject matter Handouts were clear and useful Overall, the session was informative and valuable

0 times 0 times 0 times

0 times 0 times 0 times

1 time 0 time 3 time

3 times 8 times 8 times

9 times 5 times 2 times

31

Short answer How could this workshop be improved? Ten students indicated that the PowerPoint was long and there was too much text on the slides. Five students indicated being confused by the Who and Where combination slide. Most students did not indicate a need for improvement beyond the PowerPoint issues. One student was not able to complete the workshop evaluation so the above totals represent the thirteen remaining students.

32

Observations and Proposed Revisions

Instruction Element
Introduction

Strengths
Teacher introduction showed benefits to students and motivated them to participate Bell Work Worksheet served well as a pre-test to level of student pre-knowledge PowerPoint presentation held student interest

Weaknesses
None

Proposed Revisions
None

Pretest

None

None

Presentation

Some PowerPoint slides were not as useful as anticipated

Review and eliminate non-essential slides. Consolidate where feasible. Incorporate work flow changes in use of Weebly web site with students visiting sample sites as teacher is discussing them, rather than visiting them on their own at end of presentation. Revise PowerPoint and separate Who and Where into separate slides.

Weebly web site where URLs for PowerPoint presentation web sites were stored allowed student to click on URL and access PowerPoint web site example. Eliminated wasted time and web site access errors

As a result of one-onone evaluation teacher new use of Weebly web site needed changing. She made revisions during small group presentation take improved student work flow. Combined Who and Where questions in the PowerPoint was confusing. They are separate questions and should be presented as separate slides. None

Examples

Examples clearly demonstrated concepts and were understood by students

None

Pace and Timing

Teacher felt she had to rush to complete PowerPoint. Workshop ran over allotted time Not enough time to 33

complete independent research project or take evaluation survey

Instruction Element
Practice Activities

Strengths
Students were able to follow along with PowerPoint, accessing sample web sites as teacher described how to analyze site.

Weaknesses
URL for web site being analyzed is missing on Investigating Sources worksheet. Difficult to remember which site is being analyzed. Some students left evidence sections blank when no information found. This made it appear student had not completed entire analysis. Teacher would have preferred to do assignment in the classroom, giving students to ask questions. While not a problem for this older group of students, the assignment might pose problems for younger target group.

Proposed Revisions
Revise Investigating Sources worksheet to include URL of site being analyzed.

Investigative Sources worksheet contained abbreviated analysis rubric and provided guidance and a place to record analysis results.

Instruct students to write No evidence instead of leaving blanks in order to document complete decision-making process See Pace and Timing Proposed Revisions

Posttest

Posttest was assigned as homework. Results were returned the next day. All results showed students successful at finding reliable internet search results for research question.

34

Appendices
Appendix A - Online Learner Survey Form (Blank) Appendix B - Pretest and results Appendix C - Investigating Sources Worksheet (Blank) Appendix D - Investigating Sources Rubric Appendix E - Completed Student Practice Test for two web sites (Investigating Sources Worksheet) Appendix F - Results of sample practice test Appendix G - Post-Workshop Evaluation Appendix H - PowerPoint Investigating Sources Presentation With Presenter Notes

35

Appendix A Online Learner Survey

36

Appendix B Pretest and Results

Bell Work Worksheet Name ____________________________________ Period _________________________ Yes 1. Is Wikipedia a good source to use for school research? 2. If you cant find the author of an article, or if youve never heard of the company, should you use the source? 3. Does the date of the information on a website matter? 4. If a website is completely for or against an issue, should you gather information from that site? 5. It is better to use websites with big words and a lot of technical jargon because that means the sites are important. 6. If a website doesnt list their sources, should you look at other sources instead? Results 1. - Yes (6) No (5) Unsure (2) 2. - Yes (1) No (9) Unsure (3) 3. - Yes (4) No (7) Unsure (2) 4. - Yes (3) No (7) Unsure (3) 5. - Yes (4) No (8) Unsure (1) 6. - Yes (5) No (6) Unsure (2) No Unsure

37

Appendix C Investigating Sources Worksheet (Blank)

Investigating Sources Worksheet

Source (Web Address): _________________________________________________________________________

Meets Standard (Y/N)

Criteria

Evidence

Who: Who is the writer? Does the author have the background (education/credentials) or personal experience to be an authority? Where: If no author, does the sponsoring organization have authority for this information? When: When was the site content created/updated? Are the facts and figures within the text up-to-date?

Why: Does the writer have a bias (motivation) that may interfere with the accuracy of the information? Does the content fairly handle all sides (perspectives) related to the topic?

What: Does the information relate to your topic? Is the content included on the website about a topic useful for your writing purpose?

How: Has the author or sponsoring organization supported the web content with references or links to other reliable sources? Has the author or sponsoring organization used supported evidence instead of emotional appeals?

38

Appendix D Investigating Sources Rubric

Investigating Sources Rubric Use to Evaluate Reliability of Web Site

Analysis Criteria
Who: Who is the author? Does the author have the background (education/credentials) or personal experience to be an authority? Where: Where does the author or sponsoring organization get their authority for the information presented? When evaluating the sponsoring organization review organizations domain extensions. When: When was the site content created or last updated? Are the facts and figures presented up-to-date? Is the information recent enough to make it relevant/accurate? Why: Does the author or sponsoring organization have a bias (motivation) that may interfere with the accuracy of the information? Does the content fairly handle all sides (perspectives) related to the topic? What: Is the content useful for your writing purpose? Is the web site language understandable and appropriate for your writing purpose? How: How has the author or sponsoring organization supported the web content with references or links to other reliable sources? How has the author or sponsoring organization used supported evidence instead of emotional appeals?

39

Appendix E Sample of Completed Student Practice Test (Investigating Sources Worksheet)

40

Appendix F Results of Sample Practice Test

Responses on Frontier Trails Web Site (Unreliable SIte) Identi fy Autho Author's Publication Determine Whether r Affiliation Date Biased WHO WHERE WHEN 1 0 1 0 1 0 1 1 1 0 1 0 1 0 1 0 1 0 1 0 1 1 1 0 1 0 100% 15% 0 0 0 0 0 0 0 0 0 0 1 0 0 WHY WHAT HOW 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0% 0% 0% 16

Student Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 Student 7 Student 8 Student 9 Student 10 Student 11 Student 12 Student 13 Student 14*

Point Total 1 1 1 2 1 1 1 1 1 1 3 1 1 -

8%

Conclusion: Student group was able to find greater positive analytical criteria on reliable site, than unreliable site. *Note: one of the fourteen students did not return their research sheet, thus no data is available for that student.

41

Appendix F Continued Results of Sample Practice Test

Student

Responses on National Park Service Site (Reliable Site) Identify Author's Publication Determine Whether Author Affiliation Date Biased WHO WHERE WHEN WHY WHAT HOW 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 0 1 1 1 1 0 1 1 1 100% 85% 0 62% 0 54% 0 31% 0 54% 50 1 Point Total 6 2 5 4 2 1 6 2 6 4 5 5 2

Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 Student 7 Student 8 Student 9 Student 10 Student 11 Student 12 Student 13 Student 14*

*Note: one of the fourteen students did not return their research sheet, thus no data is available for that student.

42

Appendix G - Post-Workshop Evaluation

43

Appendix H - PowerPoint Investigating Sources Presentation With Presenter Notes

Slide 1

Horizon Daily News

www.horizondailynews.com

The Technology Innovators of Ohio

- Since 1879

Critical Investigation Announced

Monday, Nov. 26th - An investigation has been launched at the Horizon Science Academy in Downtown Toledo. Authorities say some questionable sources have been discovered there. Lead investigator Tonya Schauwecker has been asked to put together a team of trained investigators to consider suspects and rid the campus of these questionable characters. Schauwecker stated, I have a lot of confidence in my team. They are intelligent young men and women and I know with the right tools, they will able to solve this problem. A training session is planned this afternoon. Schauwecker has put together a list of the usual suspects to initiate the investigation. Investigators will follow up with the familiar six essential questions. By the end of the week, Schauwecker expects to be able to release the results of the teams investigation.

This slide can be displayed while the student handouts are distributed.

Slide 2

Research Investigator Training

Using the Six Essential Big Questions

44

Slide 3

Eliminate the Usual Suspects

Wikipedia Blogs Advertisements Social Media Sites Student or Class Web Sites Question and Answer Forums

This slide may require some explanation, but do not allow students to dwell. If students need more clarification, explain that the remainder of the workshop information will help them understand why these sources are rarely reliable.

Slide 4

Six Big Questions

Who/Where? When? Why?

What?

Credible Source
How?

This slide shows that all of the criteria of the Six Big Questions comes together to contribute to the credibility of a source.

45

Slide 5

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Consider the Evidence:

Can we identify the writer?

Does the writer have the education, credentials, and/or experience to be an expert or authority? If no author, does the sponsoring organization have authority for this information?

Note: If not easily located, author and/or sponsoring organization information can also be found in FAQs, About Us, and Author Bio sections.

Slide 6

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Reminder:

When evaluating the sponsoring organization you may begin with domain extensions:
.com and .net (commercial business related) .org (organization typically nonprofit but may have bias)

.gov (government agency)

.edu (educational but verify writers authority)

We are assuming that students are familiar with these domain extensions. This is only meant to be a reminder.

46

Slide 7

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Example:

http://www.frontiertrails.com/oldwest/oregontrail.htm

Click on picture of website to link to website. This is an example of a site that lack credibility. Although the name of the author is included, we are given no other information about the author.

Slide 8

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Consider the Evidence:

The link to Contact Us is broken. What other concern may we have from the information from submission page on this site?

The site developers take submissions from people without asking about or verifying their education or credentials or background.

47

Slide 9

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Example:

http://www.nps.gov/oreg/historyculture/index.htm

Click on picture of website to link to website. This is an example of a credible website.

Slide 10

Who/Where?

When?

Why?

What?

How?

Big Questions: Who/Where?

Consider the Evidence:

The link to Contact Us section includes a long list of people within government offices with specific e-mails. There are also policies, publications and budgets included in the About Us section. What does this tell us about the people and organization responsible for the information on this site?

This is an example of a credible site.

Slide 11
48

Who/Where?

When?

Why?

What?

How?

Big Question: When?

Consider the Evidence: When was the site content created/updated? Are the facts and figures within the text up-to-date? Is the information recent enough to make it relevant/accurate?

Note: Some topics are more time sensitive than others.

Historical topics (including biographies) or literary critiques may not be as time -sensitive but many topics such as medical treatments, scientific advancements, legal requirements, and many more can change a lot in a short amount of time so it is important to use current sources.

Slide 12

Who/Where?

When?

Why?

What?

How?

Big Question: When?

Sample:

http://www.manatee.k12.fl.us/sites/elementary/samoset/psgk3ex.htm# Why is this document (found through a Google search) not credible for a research paper a student is currently writing?

Since Pluto is no longer classified as a planet, this information is outdated. This is an example of why it is important to double check your information to make sure the facts are still accurate.

Slide 13

49

Who/Where?

When?

Why?

What?

How?

Big Question: Why?

Consider the Evidence: Does the writer and/or the sponsoring organization have a bias (motivation) that may interfere with the accuracy of the information? Does the content fairly handle all sides (perspectives) related to the topic? Does writer present the information without exaggeration or misrepresentation?

Slide 14

Who/Where?

When?

Why?

What?

How?

Big Question: Why?

Sample:

http://www.humanesociety.org/issues/confinement_farm/facts/gestation_crates.html

This is an example of a biased site.

50

Slide 15

Who/Where?

When?

Why?

What?

How?

Big Question: Why?

Consider the Evidence:

How might the goal of the organization bias the information they provide?

Slide 16

page 01 Who/Where?

page 02 When?

page 03 Why?

What?

How?

Big Question: What?

Consider the Evidence:
Is the content of the website useful for your writing purpose? Is the language used on the web site understandable and appropriate for your writing purpose?

Writing purpose may include the following: to inform, to persuade, to compare and or contrast, etc.

51

Slide 17

page 01 Who/Where?

page 02 When?

page 03 Why?

What?

How?

Big Question: What?

Example:

http://pediatrics.aappublications.org/content/120/5/1183.full

This sample serves an illustration of a site that is not appropriate for a middle school students writing purpose. This article is written with pediatricians as the intended audience with likely language and terminology that will be well above that of a middle school student.

Slide 18

page 01 Who/Where?

page 02 When?

page 03 Why?

What?

How?

Big Question: What?

Consider the Evidence:

Despite a reputable source, why might this source not be useful for a middle school student writing a paper on autism?

This sample serves an illustration of a site that is not appropriate for a middle school students writing purpose. This article is written with pediatricians as the intended audience with likely language and terminology that will be well above that of a middle school student.

52

Slide 19

Who/Where?

When?

Why?

What?

How?

Big Question: How?

Consider the Evidence: Has the author or sponsoring organization supported the web content with references or links to other reliable sources? Has the author or sponsoring organization used supported evidence instead of emotional appeals?

Slide 20

Who/Where?

When?

Why?

What?

How?

Big Question: How?

Example:

http://www.cdc.gov/tobacco/youth/index.htm

This is an example of a credible source.

53

Slide 21

Who/Where?

When?

Why?

What?

How?

Big Question: How?

Consider the Evidence:

What does the collection of reports and surveys included on this site tell us about its credibility?

This is an example of a credible site.

Slide 22

Six Big Questions

Who/Where? When? Why?

What?

Credible Source

How?

All the Big Question elements come together to make a credible source.

This slide reminds students that all of the Big 6 Questions criteria contributes to the credibility of the source.

Slide 23
54

Using the Investigating Sources Worksheet

Read criteria in middle column. Write very brief notes of evidence on right. Write yes or no in left column evaluating each criteria based on evidence.

Some templates used in presentation from www.presentationmagazine.com

This slide briefly explains the tool students will be using to evaluate sources. This rubric includes all of the Big Six Questions criteria.

55