Assessing Student Information Literacy Competencies

At Austin Peay State University (APSU)

IMLS/ACRL Assessment Project Report #2

 
Context

During the July, 2001, IMLS/ACRL Project meeting in San Francisco, Nana Lowell and Laurie Collins spoke to participants about how class-level summative assessment can lead to program-level formative assessment.  The assessment plan described here specifically targets Heritage (Humanities) 1010-1020, which is a freshman-level Writing, Speaking and Researching Across the Curriculum alternative core course sequence.  In addition to providing baseline data on the information literacy learning outcomes of Heritage students in support of the IMLS/ACRL Assessment Project, a transferable instructional planning process is being learned by all APSU Librarians who teach.  This instructional planning process is a building block, which, along with

·         incorporation of the new Information Literacy (IL) Standards,

·         articulation of IL skills taught freshmen through graduate years,

·         integration of IL concepts within the appropriate disciplines, and

·         inclusion of assessment for information literacy learning outcomes,

will ultimately lead to the migration from a more traditional Library User Education Program to the formation of an Information Literacy Partnership on the APSU campus.  Participating in the IMLS/ACRL Assessment Project is a key part of this process because it builds on Information Literacy Immersion Institute training.  It also provides the structure and support necessary to effect changes in the current program. 

The selected instructional planning process utilizes a framework (model) taught by Deb Gilchrist during Information Literacy Immersion Institutes and in various other workshops.  Learning outcomes, curriculum, pedagogy, assessment and criteria are addressed in this model.  The chart found later in this report outlines the plan, which uses this model to structure assessment of student information literacy learning outcomes within the two Writing, Speaking and Researching (HUM 1010-1020) courses targeted for the IMLS/ACRL Assessment Project.

Purpose

The purpose of this assessment plan is to improve student learning of information retrieval techniques and information evaluation in three sections of Heritage (Humanities) 1010-1020 courses taught over the course of a full year.  Assessment will be tied directly to instruction delivered during three Fall 2001 class meetings and two Spring 2002 class meetings.

Using multiple assessment methods, which Nana Lowell and Laurie Collins labeled ‘triangulation’ during the IMLS/ACRL Project meeting, is considered good practice by others as well.  Sally Brown and Peter Knight advocate this approach in Assessing Learners in Higher Education (Philadelphia: Kogan Page, 1994, pp. 23-25).  They also promote the assessment of more than one activity and the assessment of the same competences several times.  The American Association for Higher Education (AAHE) http://www.aahe.org/assessment/principl.htm echoes these assessment principles.  Additionally, AAHE advocates

Therefore, multiple assessment methods, including

·         pretest/posttest questions,

·         two assignments with checklists/grading criteria (scoring rubrics),

·         self & peer assessments, and

·         research logs

will be used.  Past Heritage Research assignments are being modified and used as assessment instruments in this plan.  Pretest and posttest questions, as well as self & peer assessments will track students’ knowledge and perceptions of their knowledge about the information literacy concepts they are taught.  Inter-rater reliability in the grading of a website evaluation assignment will be gauged. 

Objectives
The information literacy learning objectives addressed in this plan, are taken either word-for-word, or are adapted, from the ACRL/IS Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians <http://www.ala.org/acrl/guides/objinfolit.html>.  See specific objective numbers in the chart, which follows.  Targeted learning objectives deal primarily with two skill areas, namely, search protocols and evaluation criteria.
  Although additional learning objectives dealing with these two skill areas exist, attention is focused on those objectives deemed most appropriate to students taking a 1000 level course.  

The Research Log will be tracked in Blackboard.  It will be used later in the first semester and possibly during the second semester, to track further development of the skills in the two skill areas.  The Research Log explores ACRL Information Literacy Competency Standard 4, Indicator 2, Outcomes 1 & 2, which state that the information literate student “maintains a journal or log of activities related to the information seeking, evaluating, and communicating process” and “reflects on past successes, failures, and alternative strategies.”


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page one

 

Note:  Deb Gilchrist’s framework uses “learning outcomes” as the starting point for planning assessment.  Therefore, I am referring to the targeted objectives from the ACRL/IS Objectives document as learning outcomes.  The planning chart details the outcomes (objectives), along with the associated curriculum, pedagogy, assessment and criteria undertaken in this plan.  Numbers listed in the Learning Outcome column correspond to the Information Literacy Standard. Performance Indicator. Learning Outcome. Instruction Section Objective (e.g. 1.1.c.1).

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

N/A  (occurs before instruction)

 

N/A (occurs before instruction)

 

Preliminary Questions about Evaluating Information (pretest to determine knowledge base prior to instruction)

 

 

 

 

 

Note: Completion of the questionnaires is required, but does not impact grades.

 

Students almost always or always use criteria. 

 

They can list the five common evaluation criteria, which include accuracy, authority, currency, coverage and objectivity. 

 

They use additional sources to verify information.

 

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page two

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Demonstrate the ability to use search protocols appropriate to the retrieval system (finding tool) 2.1.c.7, including how to

 

- broaden/narrow topic 1.1.d.3

 

- subject search & synonyms

   2.2.b.3

 

-keyword search  2.2.d.6

 

-truncation 2.2.d.7

 

-Boolean operators 2.2.d.2

 

-limit by format or type 2.2.e.3

 

-assigned subjects 2.2.c.4

 

-system help 2.1.c.2

 

 

Information containers (e.g. books, articles) and finding tools (e.g. catalogs, databases) in context

 

 

 

 

 

 

 

The eight search protocols

 

 

 

How to access Felix Online Catalog and Expanded Academic (EA) through the APSU Library website

 

 

How to use Felix and EA

 

 

 

Have students do a short brainstorming session (10-15 minutes) about information containers and finding tools

 

Present information about the cost of information and an overview of the nature of electronic information databases and catalogs

 

Provide information about search protocols which students will use in the first assignment

 

Introduce students to Felix and EA

 

 

 

 

Have students complete Expanded Academic tutorial

& explore Felix outside of class

 

Heritage Research Assignment (Using Search Protocols in Felix Online Catalog and Expanded Academic)

 

Assignment has students conduct searches on their Heritage paper/speech topics using the eight search protocols covered in class. 

 

Students conduct their searches in Felix Online Catalog and in Expanded Academic.

 

Students write notes on Felix and EA printouts in which they demonstrate their understanding of the search protocols.

 

Protocol Checklist & Grading Criteria (scoring rubric)

 

 

 

Library instructors assess student learning by marking which search protocols are used on the protocol checklist

 

The Checklist helps determine the number of protocols used, as well as which protocols are used the most/least among all students

 

The Grading Criteria (scoring rubric) reflects

-whether students used both Felix and EA,

-how many protocols students tried in Felix and in EA, and

-how well their notes reflect an understanding of the protocols

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU - page three

 

 

Learning Outcomes

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Determine the relevance of information sources to a given information need 2.4.a.4

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

 

Apply the five established evaluative criteria

--accuracy, authority, objectivity, currency and coverage -- to information sources 3.2.c.1 – 4

 

 

 

 

 

Information need & relevance

 

 

 

 

How to verify information

 

 

 

 

 

 

The five criteria and specific questions which address each criterion for evaluating information

 

 

 

 

 

 

 

Have students do a short brainstorming session (10-15 minutes) about information relevance and verification

 

Cover how other sources can be used to verify information

 

 

 

 

 

Present the five criteria with specific questions to student

 

Have students assist the instructor in using the criteria and associated questions to evaluate a website during class

 

 

 

 

Website Evaluation Assignment

 

 

 

Each student will complete a short self-assessment questionnaire on their completed website evaluation assignment.

 

Each student completes a peer assessment questionnaire on one of their classmates’ completed website evaluation assignment to provide information about student perceptions. 

 

The results of the two questionnaires will be compared.

 

Completion of the assessment questionnaires is required, but does not impact grades.

 

 

Criteria Checklist & Grading Criteria (scoring rubric)

 

 

 Student perceptions determined through the questionnaires will be compared with instructor-generated scores on the assignment.

 

Inter-rater reliability will be gauged by comparing additional librarians’ grading of a sample of the completed evaluation assignments.

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page four

 

 

Learning Outcomes

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Using Blackboard, keep a research log demonstrating that they are able to

 

 

 

Identify and use search language and protocols appropriate to the retrieval system 2.1.c.7; 2.2.e.1; 2.2.e.3; 2.2.e.4; 2.3.a.5

 

Evaluate the quality of the information retrieved using criteria such as authorship, point of view/bias, date written, citations, etc.  2.4.a.2; 3.4.g.2; 3.4.g.3

 

How to use Blackboard

 

 

 

 

 

Search protocols (covered during the 1st class session)

 

 

 

 

Evaluation criteria (covered during the 2nd class session)

 

 

Show students how to use Blackboard to record their research logs

 

Divide students into groups and

 

1) Have each group use search protocols with a web search engine to find information on a Social Science topic

 

 

2) Have each group evaluate one of the websites they identify using the evaluation criteria

 

 

 

Research Log (under development)

 

Students reflect on the process they use to gather information for their Social Science papers. 

 

 

Student’s reflections about this process will demonstrate the degree to which they are able to apply the search protocols and evaluation criteria they learned earlier this semester.

 

 

 

Analysis of research log to see the degree to which students are able to accomplish learning outcomes

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart- APSU – page five

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

The five criteria and specific questions which address each criterion for evaluating information

 

No additional activities; the follow-up questions may be viewed as a learning activity

 

Follow-up Questions about Evaluating Information (posttest to determine knowledge base after instruction)

 

 

 

Note: Completion of the questionnaires is required, but does not impact grades.

 

 

 

 

 

Students almost always or always use criteria. 

 

They can list the five common evaluation criteria, including accuracy, authority, currency coverage and objectivity.

 

They use additional sources to verify information.

 

 

 

 



Heritage Student Information Literacy Learning Outcomes Assessment Administration Timeline

 

Month

Day

 

Assessment Administered

 

 

 

August, 2001

24

 

Pretest - Preliminary Questions about Evaluating Information

 

31

 

Research Assignment: Using Search Protocols

 

October, 2001

10

 

Website Evaluation Assignment

 

 

 

 

19

 

Self Assessment Questionnaire

 

 

 

 

 

 

Peer Assessment Questionnaire

 

 

November, 2001

7

 

Research Log

 

 

December, 2001

11

 

Posttest - Follow-up Questions about Evaluating Information



Preliminary Questions About Evaluating Information

 

1)      I use criteria to evaluate information.  Circle one.

 

 

Never --- Almost never --- Sometimes --- Almost always --- Always

 

 

2)      List five criteria for evaluating information.

 

 

1.

 

2.

 

3.

 

4.

 

5.

 

 

3)      Mark the steps listed below which apply to the following statement. 

 

I verify information by

 

___        communicating with the author

 

___        consulting another source

 

___        asking a friend

 

___        I don’t; instead, I assume the information is correct

 

____          Write any other steps you take below

 


 

Heritage Research Assignment#1:

Using Search Protocols

DUE: September 10, 2001 in class

 

The three resources you need for this assignment are all available on the opening APSU Library webpage at http://library.apsu.edu.  They are

 

 

First, work your way through the Expanded Academic tutorial.

 

Second, refer to the Information Retrieval Concepts outline sheet, which we covered in class on August 31, 2001, as you conduct searches about your upcoming paper/speech topic in both

 

 

As you conduct your searches in these two resources, use the “Search Protocols” at the bottom of the Information Retrieval Concepts outline sheet.  Be creative and show to the best of your ability that you understand how to conduct searches in BOTH resources using ALL EIGHT protocols.

 

You will need to conduct several searches in each source, print at least five records or screens from each resource, and write notes on each sheet of paper you print.  PRINT YOUR NAME on each sheet and STAPLE them together.  You will turn in these sheets as part of the assignment. 

 

In your notes on each sheet, explain the protocol or protocols you tried out which resulted in the record or screen that you printed.  You can use more than one protocol per search if you like.  Make sure you include enough details in your notes so that we can follow what you were trying to accomplish in each of your searches. 

 

The point is for you to experience each resource and show us that you understand and can apply the protocols.  You will be graded (see Grading Criteria sheet) according to

 

 

You may have questions.  If you do, please contact your library instructor.  It is your responsibility to turn in your work on time.  If you are not in class on the due date, turn in your assignment to the Library Information Desk.  Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).


Search Protocol Checklist

 

 

 

Search Protocols Used in FELIX

 

q       Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

 

q       Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

 

q       Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

 

q       Truncate your search term (e.g. famil* picks up family, families, familial)

 

q       Use the Boolean operators AND, OR, NOT to search several concepts at once

 

q       Use subjects that have been assigned to the source to find other sources

 

q       Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

 

q       Use help screens within database or catalog

 

 

 

 

 

Search Protocols Used in EXPANDED ACADEMIC

 

q       Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

 

q       Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

 

q       Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

 

q       Truncate your search term (e.g. famil* picks up family, families, familial)

 

q       Use the Boolean operators AND, OR, NOT to search several concepts at once

 

q       Use subjects that have been assigned to the source to find other sources

 

q       Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

 

q       Use help screens within database or catalog


Heritage Research Assignment

Information Retrieval Concepts (Search Protocols)

Grading Criteria

 

 

Score

Levels

Number

of

sources

covered

Number of

protocols tried

in Expanded

Academic

Number

of protocols

tried

in Felix

Content

of

notes

A

Used both

sources

Tried 7-8

protocols

Tried 7-8

protocols

Detailed, in-depth,

clear descriptions of

appropriately applied

protocols

B

N/A

Tried 5-6

protocols

Tried 5-6

protocols

 

Understandable and

clear, but insufficient

detail and depth

C

Used only

one

source

Tried 3-4

protocols

Tried 3-4

protocols

Understandable but abbreviated and not entirely clear

D

N/A

Tried 1-2

Protocols

Tried 1-2

Protocols

Single words which

do not appear

relevant

 

 

Grade for number of sources:

   _____

Grade for

number of protocols #1:

    _____

Grade for number of protocols #2:

      ______

Grade for content of notes:

 

              ________

 

 

Late penalty (if any):  ______

 

Overall Grade on this assignment: ________


Website Evaluation Assignment

Due in Class on ____________________

 

Search and select a high-quality website about your topic, which will help you prepare your paper or speech, and examine it closely.  Use the five established evaluative criteria and associated questions, which we covered in class and which were provided to you in written form, as a resource for completing this assignment.   Additional information may be found by conducting a search for the following words in Google (http://www.google.com): criteria evaluation information.

 

Write a thorough typed double-spaced paper (at least 500 words) covering the website you identified.  Give specific examples from your website.  Your grade on this assignment will be based on how well you incorporate your responses to all of the questions below, as well as how well you address the criteria and questions covered in class.

 

     1)  What is your topic for the History block?

 

      2)  What is the complete/full URL of your website?  Print out and attach the first page of your website to your paper.

 

3)  How useful will it be in your paper and speech? (Be specific.)

 

4)  How accurate is this website? Are there spelling or grammatical errors? Does the website contain outrageous or misleading information? (Use the accuracy questions.)

 

5)  Who wrote or sponsored this website?  Is it an educational, commercial, governmental or organizational website?  What audience was the website written for?  Be sure to look in the website’s header, footer and any links for clues. (Use the authority questions.)

 

6)  How objective is the website?  Is a particular viewpoint presented in the website?  If so, what is the viewpoint?  If not, give evidence of how the website covers more than one viewpoint.  Be sure to state what viewpoints are covered.

      Does the website present

 

·         Facts

·         Opinions

·         Both facts and opinions

 

Give at least one specific example of a fact and/or an opinion covered by the website.

 

 

Over

 

7)  How current is this website?  When was the information first posted?  Has it been updated within the last year? If present, how current are the sources cited or links contained in the website? (Use the currency questions.)

 

8)  What are the breadth and depth of the website’s coverage?  What is its purpose?  Does it cover a highly specialized topic area in great detail or does it merely provide a general overview of a broad topic?  (Use the coverage questions.)

 

9)  What are some steps you can take to verify the accuracy of the information contained in this website?  (Be specific.)

 

 

Please contact your library instructor if you have questions.  It is your responsibility to turn in your work on time.  If you are not in class on the due date, turn in your assignment to the Library Information Desk.  Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).

 

 

This is the grading criteria:

 

Score Levels

Number of

Criteria Covered

Information Verification

Content of Essay

A

Attempted all

5 evaluative criteria (accuracy, authority, coverage, currency and objectivity)

Stated 2-3 possible steps to take

in verifying information on website

Detailed, in-depth,

clear descriptions

of appropriately

applied criteria

B

Attempted 

4 of 5 evaluative criteria listed above

Stated 1-2 possible steps to take in verifying information on website

Understandable and clear, but insufficient

detail and depth

C

Attempted

3 of 5 evaluative criteria listed above

Stated 1 possible step to take in verifying information on website

Understandable but abbreviated and

not entirely clear or simply copying some class materials

D

Attempted

2 of 5 evaluative criteria listed above

Stated no verification steps

Vague or nonsensical sentences, or copies class materials with no original work

 


Self-Assessment of Information Evaluative Skills

 

Circle the ONE answer out of the available choices that comes closest to what you perceive your abilities to be.  Please be honest.  Completing this questionnaire is required, but your specific answers to the statements will NOT impact your grade.

 

 

1)    Overall, based on the established evaluative criteria which were covered in class, I think the website I selected was high-quality.       

 

Yes                     No

 

 

2)    As it relates directly to the topic about which I am writing and speaking, the website I selected is

 

of no use      of very limited use      of some use    useful       very useful

 

 

3)    I successfully applied the following number of established evaluative criteria in my Website Evaluation Assignment which I completed in this class.

 

one       two           three        four                     five

 

 

4)    Before taking this class, I applied evaluative criteria to information and its sources.

 

never   almost never              sometimes      almost always              always

 

 

5)    At the present moment, I can list from memory the following number of established evaluative criteria that we covered in class.

          

one       two                 three              four                         five

 

 

6)    Before taking this class, I was in the habit of checking other sources to either confirm or question the point of view or bias of an information source.

 

never       almost never       sometimes            almost always   always


Peer Assessment of Information Evaluative Skills

 

Circle the ONE answer out of the available choices that comes closest to what you perceive your classmate’s abilities to be,  based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website your classmate evaluated before you read your classmate’s work and answer the questions.  

 

 

Please be honest.  Taking the time to complete this questionnaire in a serious fashion is required, but your specific answers to the statements below in no way impact your grade or your classmate’s grade.  Your classmate will NOT see the results of your assessment.

 

 

 

 

1)     Overall, based on the established evaluative criteria which we covered in class, the website my peer selected (see attached completed Website Evaluation Assignment) is high-quality.

 

Yes                    No

 

 

 

 

 

2)     In relation to the paper and speech topic my peer has selected (see attached completed assignment), the selected website  is

 

of no use     of very limited use     of some use     useful     very useful

 

 

 

 

 

 

3)     In my opinion, my peer successfully applied the following number of established evaluative criteria which were discussed in class.

 

one               two                   three              four                     five


Name: ___________________________

 

Heritage Research Assignment #2

Website Evaluation

Grading Criteria

 

Paper Topic: ______________________________________________

 

Is the website relevant to the student’s topic? (circle one)     yes        no

 

 

Evaluation Criteria Checklist                               Verification Methods

 

q       Accuracy                      o  communicate with author

q       Authority                      o  consult another source

q       Coverage                         o  ask a friend

q       Currency                      o  assume information is correct

q       Objectivity                         o  other steps  _____________

_________________________

 

 

Score Levels

Number of

Criteria Covered

Information Verification

Content of Essay

A

Attempted all

5 evaluative criteria, including accuracy, authority, coverage, currency and objectivity

Stated 2-3 possible steps to take

in verifying information on website

Detailed, in-depth,

clear descriptions

of appropriately

applied criteria

B

Attempted 

4 of 5 evaluative criteria listed above

Stated 1-2 possible steps to take in verifying information on website

Understandable and clear, but insufficient

detail and depth

C

Attempted

3 of 5 evaluative criteria listed above

Stated 1 possible step to take in verifying information on website

Understandable but abbreviated and

not entirely clear or simply copying some class materials

D

Attempted

2 of 5 evaluative criteria listed above

Stated no verification steps

Vague or nonsensical sentences, or copies class materials with no original work

 


Follow-up Questions About Evaluating Information

 

 

4)      I use criteria to evaluate information.  Circle one.

 

 

Never --- Almost never --- Sometimes --- Almost always --- Always

 

 

5)      List five criteria for evaluating information.

 

 

1.

 

2.

 

3.

 

4.

 

5.

 

 

6)      Mark the steps listed below which apply to the following statement. 

 

I verify information by

 

___        communicating with the author

 

___        consulting another source

 

___        asking a friend

 

___        I don’t; instead, I assume the information is correct

 

___          Write any other steps you take below

 

Assessing Student Information Literacy Competencies

At Austin Peay State University (APSU)

IMLS/ACRL Assessment Project Report #2

 

 

Context

During the July, 2001, IMLS/ACRL Project meeting in San Francisco, Nana Lowell and Laurie Collins spoke to participants about how class-level summative assessment can lead to program-level formative assessment.  The assessment plan described here specifically targets Heritage (Humanities) 1010-1020, which is a freshman-level Writing, Speaking and Researching Across the Curriculum alternative core course sequence.  In addition to providing baseline data on the information literacy learning outcomes of Heritage students in support of the IMLS/ACRL Assessment Project, a transferable instructional planning process is being learned by all APSU Librarians who teach.  This instructional planning process is a building block, which, along with

 

·         incorporation of the new Information Literacy (IL) Standards,

·         articulation of IL skills taught freshmen through graduate years,

·         integration of IL concepts within the appropriate disciplines, and

·         inclusion of assessment for information literacy learning outcomes,

 

will ultimately lead to the migration from a more traditional Library User Education Program to the formation of an Information Literacy Partnership on the APSU campus.  Participating in the IMLS/ACRL Assessment Project is a key part of this process because it builds on Information Literacy Immersion Institute training.  It also provides the structure and support necessary to effect changes in the current program. 

 

The selected instructional planning process utilizes a framework (model) taught by Deb Gilchrist during Information Literacy Immersion Institutes and in various other workshops.  Learning outcomes, curriculum, pedagogy, assessment and criteria are addressed in this model.  The chart found later in this report outlines the plan, which uses this model to structure assessment of student information literacy learning outcomes within the two Writing, Speaking and Researching (HUM 1010-1020) courses targeted for the IMLS/ACRL Assessment Project.

 

Purpose

The purpose of this assessment plan is to improve student learning of information retrieval techniques and information evaluation in three sections of Heritage (Humanities) 1010-1020 courses taught over the course of a full year.  Assessment will be tied directly to instruction delivered during three Fall 2001 class meetings and two Spring 2002 class meetings.


Using multiple assessment methods, which Nana Lowell and Laurie Collins labeled ‘triangulation’ during the IMLS/ACRL Project meeting, is considered good practice by others as well.  Sally Brown and Peter Knight advocate this approach in Assessing Learners in Higher Education (Philadelphia: Kogan Page, 1994, pp. 23-25).  They also promote the assessment of more than one activity and the assessment of the same competences several times.  The American Association for Higher Education (AAHE) http://www.aahe.org/assessment/principl.htm echoes these assessment principles.  Additionally, AAHE advocates

 

 

Therefore, multiple assessment methods, including

 

·         pretest/posttest questions,

·         two assignments with checklists/grading criteria (scoring rubrics),

·         self & peer assessments, and

·         research logs

 

will be used.  Past Heritage Research assignments are being modified and used as assessment instruments in this plan.  Pretest and posttest questions, as well as self & peer assessments will track students’ knowledge and perceptions of their knowledge about the information literacy concepts they are taught.  Inter-rater reliability in the grading of a website evaluation assignment will be gauged. 

 

Objectives
The information literacy learning objectives addressed in this plan, are taken either word-for-word, or are adapted, from the ACRL/IS Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians <http://www.ala.org/acrl/guides/objinfolit.html>.  See specific objective numbers in the chart, which follows.  Targeted learning objectives deal primarily with two skill areas, namely, search protocols and evaluation criteria.
  Although additional learning objectives dealing with these two skill areas exist, attention is focused on those objectives deemed most appropriate to students taking a 1000 level course.

 

The Research Log will be tracked in Blackboard.  It will be used later in the first semester and possibly during the second semester, to track further development of the skills in the two skill areas.  The Research Log explores ACRL Information Literacy Competency Standard 4, Indicator 2, Outcomes 1 & 2, which state that the information literate student “maintains a journal or log of activities related to the information seeking, evaluating, and communicating process” and “reflects on past successes, failures, and alternative strategies.”


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page one

 

Note:  Deb Gilchrist’s framework uses “learning outcomes” as the starting point for planning assessment.  Therefore, I am referring to the targeted objectives from the ACRL/IS Objectives document as learning outcomes.  The planning chart details the outcomes (objectives), along with the associated curriculum, pedagogy, assessment and criteria undertaken in this plan.  Numbers listed in the Learning Outcome column correspond to the Information Literacy Standard. Performance Indicator. Learning Outcome. Instruction Section Objective (e.g. 1.1.c.1).

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

N/A  (occurs before instruction)

 

N/A (occurs before instruction)

 

Preliminary Questions about Evaluating Information (pretest to determine knowledge base prior to instruction)

 

 

 

 

 

Note: Completion of the questionnaires is required, but does not impact grades.

 

Students almost always or always use criteria. 

 

They can list the five common evaluation criteria, which include accuracy, authority, currency, coverage and objectivity. 

 

They use additional sources to verify information.

 

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page two

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Demonstrate the ability to use search protocols appropriate to the retrieval system (finding tool) 2.1.c.7, including how to

 

- broaden/narrow topic 1.1.d.3

 

- subject search & synonyms

   2.2.b.3

 

-keyword search  2.2.d.6

 

-truncation 2.2.d.7

 

-Boolean operators 2.2.d.2

 

-limit by format or type 2.2.e.3

 

-assigned subjects 2.2.c.4

 

-system help 2.1.c.2

 

 

Information containers (e.g. books, articles) and finding tools (e.g. catalogs, databases) in context

 

 

 

 

 

 

 

The eight search protocols

 

 

 

How to access Felix Online Catalog and Expanded Academic (EA) through the APSU Library website

 

 

How to use Felix and EA

 

 

 

Have students do a short brainstorming session (10-15 minutes) about information containers and finding tools

 

Present information about the cost of information and an overview of the nature of electronic information databases and catalogs

 

Provide information about search protocols which students will use in the first assignment

 

Introduce students to Felix and EA

 

 

 

 

Have students complete Expanded Academic tutorial

& explore Felix outside of class

 

Heritage Research Assignment (Using Search Protocols in Felix Online Catalog and Expanded Academic)

 

Assignment has students conduct searches on their Heritage paper/speech topics using the eight search protocols covered in class. 

 

Students conduct their searches in Felix Online Catalog and in Expanded Academic.

 

Students write notes on Felix and EA printouts in which they demonstrate their understanding of the search protocols.

 

Protocol Checklist & Grading Criteria (scoring rubric)

 

 

 

Library instructors assess student learning by marking which search protocols are used on the protocol checklist

 

The Checklist helps determine the number of protocols used, as well as which protocols are used the most/least among all students

 

The Grading Criteria (scoring rubric) reflects

-whether students used both Felix and EA,

-how many protocols students tried in Felix and in EA, and

-how well their notes reflect an understanding of the protocols

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU - page three

 

 

Learning Outcomes

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Determine the relevance of information sources to a given information need 2.4.a.4

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

 

Apply the five established evaluative criteria

--accuracy, authority, objectivity, currency and coverage -- to information sources 3.2.c.1 – 4

 

 

 

 

 

Information need & relevance

 

 

 

 

How to verify information

 

 

 

 

 

 

The five criteria and specific questions which address each criterion for evaluating information

 

 

 

 

 

 

 

Have students do a short brainstorming session (10-15 minutes) about information relevance and verification

 

Cover how other sources can be used to verify information

 

 

 

 

 

Present the five criteria with specific questions to student

 

Have students assist the instructor in using the criteria and associated questions to evaluate a website during class

 

 

 

 

Website Evaluation Assignment

 

 

 

Each student will complete a short self-assessment questionnaire on their completed website evaluation assignment.

 

Each student completes a peer assessment questionnaire on one of their classmates’ completed website evaluation assignment to provide information about student perceptions. 

 

The results of the two questionnaires will be compared.

 

Completion of the assessment questionnaires is required, but does not impact grades.

 

 

Criteria Checklist & Grading Criteria (scoring rubric)

 

 

 Student perceptions determined through the questionnaires will be compared with instructor-generated scores on the assignment.

 

Inter-rater reliability will be gauged by comparing additional librarians’ grading of a sample of the completed evaluation assignments.

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page four

 

 

Learning Outcomes

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

Using Blackboard, keep a research log demonstrating that they are able to

 

 

 

Identify and use search language and protocols appropriate to the retrieval system 2.1.c.7; 2.2.e.1; 2.2.e.3; 2.2.e.4; 2.3.a.5

 

Evaluate the quality of the information retrieved using criteria such as authorship, point of view/bias, date written, citations, etc.  2.4.a.2; 3.4.g.2; 3.4.g.3

 

How to use Blackboard

 

 

 

 

 

Search protocols (covered during the 1st class session)

 

 

 

 

Evaluation criteria (covered during the 2nd class session)

 

 

Show students how to use Blackboard to record their research logs

 

Divide students into groups and

 

1) Have each group use search protocols with a web search engine to find information on a Social Science topic

 

 

2) Have each group evaluate one of the websites they identify using the evaluation criteria

 

 

 

Research Log (under development)

 

Students reflect on the process they use to gather information for their Social Science papers. 

 

 

Student’s reflections about this process will demonstrate the degree to which they are able to apply the search protocols and evaluation criteria they learned earlier this semester.

 

 

 

Analysis of research log to see the degree to which students are able to accomplish learning outcomes

 


Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart- APSU – page five

 

 

Learning Outcome

What do we want students to be able to do as a result of this instruction?

 

 

Curriculum

What does the student need to know in order to do this well?

 

Pedagogy

What activity will facilitate the learning?

 

Assessment

How will the student demonstrate the learning?

 

Criteria

How will I know that the student has done this well?

 

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

 

 

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

 

The five criteria and specific questions which address each criterion for evaluating information

 

No additional activities; the follow-up questions may be viewed as a learning activity

 

Follow-up Questions about Evaluating Information (posttest to determine knowledge base after instruction)

 

 

 

Note: Completion of the questionnaires is required, but does not impact grades.

 

 

 

 

 

Students almost always or always use criteria. 

 

They can list the five common evaluation criteria, including accuracy, authority, currency coverage and objectivity.

 

They use additional sources to verify information.

 

 

 

 


Heritage Student Information Literacy Learning Outcomes Assessment Administration Timeline

 

Month

Day

 

Assessment Administered

 

 

 

August, 2001

24

 

Pretest - Preliminary Questions about Evaluating Information

 

31

 

Research Assignment: Using Search Protocols

 

October, 2001

10

 

Website Evaluation Assignment

 

 

 

 

19

 

Self Assessment Questionnaire

 

 

 

 

 

 

Peer Assessment Questionnaire

 

 

November, 2001

7

 

Research Log

 

 

December, 2001

11

 

Posttest - Follow-up Questions about Evaluating Information


 


Preliminary Questions About Evaluating Information

 

1)      I use criteria to evaluate information.  Circle one.

Never --- Almost never --- Sometimes --- Almost always --- Always

2)      List five criteria for evaluating information.

   

1.  

2.

3.

4.

5.

 

3)      Mark the steps listed below which apply to the following statement. 

I verify information by

___        communicating with the author

___        consulting another source

___        asking a friend

___        I don’t; instead, I assume the information is correct

____          Write any other steps you take below

 


Heritage Research Assignment#1:

Using Search Protocols

DUE: September 10, 2001 in class

 

The three resources you need for this assignment are all available on the opening APSU Library webpage at http://library.apsu.edu.  They are

 

 

First, work your way through the Expanded Academic tutorial.

 

Second, refer to the Information Retrieval Concepts outline sheet, which we covered in class on August 31, 2001, as you conduct searches about your upcoming paper/speech topic in both

 

 

As you conduct your searches in these two resources, use the “Search Protocols” at the bottom of the Information Retrieval Concepts outline sheet.  Be creative and show to the best of your ability that you understand how to conduct searches in BOTH resources using ALL EIGHT protocols.

 

You will need to conduct several searches in each source, print at least five records or screens from each resource, and write notes on each sheet of paper you print.  PRINT YOUR NAME on each sheet and STAPLE them together.  You will turn in these sheets as part of the assignment. 

 

In your notes on each sheet, explain the protocol or protocols you tried out which resulted in the record or screen that you printed.  You can use more than one protocol per search if you like.  Make sure you include enough details in your notes so that we can follow what you were trying to accomplish in each of your searches. 

 

The point is for you to experience each resource and show us that you understand and can apply the protocols.  You will be graded (see Grading Criteria sheet) according to

 

 

You may have questions.  If you do, please contact your library instructor.  It is your responsibility to turn in your work on time.  If you are not in class on the due date, turn in your assignment to the Library Information Desk.  Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).



Search Protocol Checklist
 

Search Protocols Used in FELIX

 

q       Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

 

q       Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

 

q       Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

 

q       Truncate your search term (e.g. famil* picks up family, families, familial)

 

q       Use the Boolean operators AND, OR, NOT to search several concepts at once

 

q       Use subjects that have been assigned to the source to find other sources

 

q       Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

 

q       Use help screens within database or catalog

   

Search Protocols Used in EXPANDED ACADEMIC

 

q       Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

 

q       Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

 

q       Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

 

q       Truncate your search term (e.g. famil* picks up family, families, familial)

 

q       Use the Boolean operators AND, OR, NOT to search several concepts at once

 

q       Use subjects that have been assigned to the source to find other sources

 

q       Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

 

q       Use help screens within database or catalog



Heritage Research Assignment

Information Retrieval Concepts (Search Protocols)

Grading Criteria

 

 

Score

Levels

Number

of

sources

covered

Number of

protocols tried

in Expanded

Academic

Number

of protocols

tried

in Felix

Content

of

notes

A

Used both

sources

Tried 7-8

protocols

Tried 7-8

protocols

Detailed, in-depth,

clear descriptions of

appropriately applied

protocols

B

N/A

Tried 5-6

protocols

Tried 5-6

protocols

 

Understandable and

clear, but insufficient

detail and depth

C

Used only

one

source

Tried 3-4

protocols

Tried 3-4

protocols

Understandable but abbreviated and not entirely clear

D

N/A

Tried 1-2

Protocols

Tried 1-2

Protocols

Single words which

do not appear

relevant

 

 

Grade for number of sources:

   _____

Grade for

number of protocols #1:

    _____

Grade for number of protocols #2:

      ______

Grade for content of notes:

 

              ________

 

 

Late penalty (if any):  ______

 

Overall Grade on this assignment: ________



Website Evaluation Assignment

Due in Class on ____________________

 

Search and select a high-quality website about your topic, which will help you prepare your paper or speech, and examine it closely.  Use the five established evaluative criteria and associated questions, which we covered in class and which were provided to you in written form, as a resource for completing this assignment.   Additional information may be found by conducting a search for the following words in Google (http://www.google.com): criteria evaluation information.

 

Write a thorough typed double-spaced paper (at least 500 words) covering the website you identified.  Give specific examples from your website.  Your grade on this assignment will be based on how well you incorporate your responses to all of the questions below, as well as how well you address the criteria and questions covered in class.

 

     1)  What is your topic for the History block?

 

      2)  What is the complete/full URL of your website?  Print out and attach the first page of your website to your paper.

 

3)  How useful will it be in your paper and speech? (Be specific.)

 

4)  How accurate is this website? Are there spelling or grammatical errors? Does the website contain outrageous or misleading information? (Use the accuracy questions.)

 

5)  Who wrote or sponsored this website?  Is it an educational, commercial, governmental or organizational website?  What audience was the website written for?  Be sure to look in the website’s header, footer and any links for clues. (Use the authority questions.)

 

6)  How objective is the website?  Is a particular viewpoint presented in the website?  If so, what is the viewpoint?  If not, give evidence of how the website covers more than one viewpoint.  Be sure to state what viewpoints are covered.

      Does the website present

 

·         Facts

·         Opinions

·         Both facts and opinions

 

Give at least one specific example of a fact and/or an opinion covered by the website.

 

 

Over

 

7)  How current is this website?  When was the information first posted?  Has it been updated within the last year? If present, how current are the sources cited or links contained in the website? (Use the currency questions.)

 

8)  What are the breadth and depth of the website’s coverage?  What is its purpose?  Does it cover a highly specialized topic area in great detail or does it merely provide a general overview of a broad topic?  (Use the coverage questions.)

 

9)  What are some steps you can take to verify the accuracy of the information contained in this website?  (Be specific.)

 

 

Please contact your library instructor if you have questions.  It is your responsibility to turn in your work on time.  If you are not in class on the due date, turn in your assignment to the Library Information Desk.  Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).

 

 

This is the grading criteria:

 

Score Levels

Number of

Criteria Covered

Information Verification

Content of Essay

A

Attempted all

5 evaluative criteria (accuracy, authority, coverage, currency and objectivity)

Stated 2-3 possible steps to take

in verifying information on website

Detailed, in-depth,

clear descriptions

of appropriately

applied criteria

B

Attempted 

4 of 5 evaluative criteria listed above

Stated 1-2 possible steps to take in verifying information on website

Understandable and clear, but insufficient

detail and depth

C

Attempted

3 of 5 evaluative criteria listed above

Stated 1 possible step to take in verifying information on website

Understandable but abbreviated and

not entirely clear or simply copying some class materials

D

Attempted

2 of 5 evaluative criteria listed above

Stated no verification steps

Vague or nonsensical sentences, or copies class materials with no original work

 



Self-Assessment of Information Evaluative Skills

Circle the ONE answer out of the available choices that comes closest to what you perceive your abilities to be.  Please be honest.  Completing this questionnaire is required, but your specific answers to the statements will NOT impact your grade.

1)    Overall, based on the established evaluative criteria which were covered in class, I think the website I selected was high-quality.      

Yes                     No

 
2)    As it relates directly to the topic about which I am writing and speaking, the website I selected is

of no use      of very limited use      of some use    useful       very useful

 

3)    I successfully applied the following number of established evaluative criteria in my Website Evaluation Assignment which I completed in this class.

one       two           three        four                     five

 

4)    Before taking this class, I applied evaluative criteria to information and its sources.

never   almost never              sometimes      almost always              always

5)    At the present moment, I can list from memory the following number of established evaluative criteria that we covered in class.          

one       two                 three              four                         five

 

6)    Before taking this class, I was in the habit of checking other sources to either confirm or question the point of view or bias of an information source.

never       almost never       sometimes            almost always   always


Peer Assessment of Information Evaluative Skills

Circle the ONE answer out of the available choices that comes closest to what you perceive your classmate’s abilities to be,  based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website your classmate evaluated before you read your classmate’s work and answer the questions.  

Please be honest.  Taking the time to complete this questionnaire in a serious fashion is required, but your specific answers to the statements below in no way impact your grade or your classmate’s grade.  Your classmate will NOT see the results of your assessment.

 

1)     Overall, based on the established evaluative criteria which we covered in class, the website my peer selected (see attached completed Website Evaluation Assignment) is high-quality.

Yes                    No

 

2)     In relation to the paper and speech topic my peer has selected (see attached completed assignment), the selected website  is  

of no use     of very limited use     of some use     useful     very useful

 

3)     In my opinion, my peer successfully applied the following number of established evaluative criteria which were discussed in class.

one               two                   three              four                     five


 

Name: ___________________________

 

Heritage Research Assignment #2

Website Evaluation

Grading Criteria

 

Paper Topic: ______________________________________________

 

Is the website relevant to the student’s topic? (circle one)     yes        no

Evaluation Criteria Checklist                               Verification Methods  

q       Accuracy                                             o  communicate with author

q       Authority                                              o  consult another source

q       Coverage                                             o  ask a friend

q       Currency                                             o  assume information is correct

q       Objectivity                                           o  other steps  _____________

_________________________  

Score Levels

Number of

Criteria Covered

Information Verification

Content of Essay

A

Attempted all

5 evaluative criteria, including accuracy, authority, coverage, currency and objectivity

Stated 2-3 possible steps to take

in verifying information on website

Detailed, in-depth,

clear descriptions

of appropriately

applied criteria

B

Attempted 

4 of 5 evaluative criteria listed above

Stated 1-2 possible steps to take in verifying information on website

Understandable and clear, but insufficient

detail and depth

C

Attempted

3 of 5 evaluative criteria listed above

Stated 1 possible step to take in verifying information on website

Understandable but abbreviated and

not entirely clear or simply copying some class materials

D

Attempted

2 of 5 evaluative criteria listed above

Stated no verification steps

Vague or nonsensical sentences, or copies class materials with no original work

 


Follow-up Questions About Evaluating Information

 

1)      I use criteria to evaluate information.  Circle one.

Never --- Almost never --- Sometimes --- Almost always --- Always

 

2)      List five criteria for evaluating information.

 

1.  

2.

3.

4.

5.

 

3)      Mark the steps listed below which apply to the following statement. 

I verify information by 

___        communicating with the author

___        consulting another source

___        asking a friend

___        I don’t; instead, I assume the information is correc

___          Write any other steps you take below

 


END