Assessing Student Information Literacy Competencies

At Austin Peay State University (APSU)

IMLS/ACRL Assessment Project Report #2

Context

During the July, 2001, IMLS/ACRL Project meeting in San Francisco, Nana Lowell and Laurie Collins spoke to participants about how class-level summative assessment can lead to program-level formative assessment. The assessment plan described here specifically targets Heritage (Humanities) 1010-1020, which is a freshman-level Writing, Speaking and Researching Across the Curriculum alternative core course sequence. In addition to providing baseline data on the information literacy learning outcomes of Heritage students in support of the IMLS/ACRL Assessment Project, a transferable instructional planning process is being learned by all APSU Librarians who teach. This instructional planning process is a building block, which, along with

· incorporation of the new Information Literacy (IL) Standards,

· articulation of IL skills taught freshmen through graduate years,

· integration of IL concepts within the appropriate disciplines, and

· inclusion of assessment for information literacy learning outcomes,

will ultimately lead to the migration from a more traditional Library User Education Program to the formation of an Information Literacy Partnership on the APSU campus. Participating in the IMLS/ACRL Assessment Project is a key part of this process because it builds on Information Literacy Immersion Institute training. It also provides the structure and support necessary to effect changes in the current program.

The selected instructional planning process utilizes a framework (model) taught by Deb Gilchrist during Information Literacy Immersion Institutes and in various other workshops. Learning outcomes, curriculum, pedagogy, assessment and criteria are addressed in this model. The chart found later in this report outlines the plan, which uses this model to structure assessment of student information literacy learning outcomes within the two Writing, Speaking and Researching (HUM 1010-1020) courses targeted for the IMLS/ACRL Assessment Project.

Purpose

The purpose of this assessment plan is to improve student learning of information retrieval techniques and information evaluation in three sections of Heritage (Humanities) 1010-1020 courses taught over the course of a full year. Assessment will be tied directly to instruction delivered during three Fall 2001 class meetings and two Spring 2002 class meetings.

Using multiple assessment methods, which Nana Lowell and Laurie Collins labeled ‘triangulation’ during the IMLS/ACRL Project meeting, is considered good practice by others as well. Sally Brown and Peter Knight advocate this approach in Assessing Learners in Higher Education (Philadelphia: Kogan Page, 1994, pp. 23-25). They also promote the assessment of more than one activity and the assessment of the same competences several times. The American Association for Higher Education (AAHE) http://www.aahe.org/assessment/principl.htm echoes these assessment principles. Additionally, AAHE advocates

involvement of faculty, librarians, students and other campus constituencies in assessment efforts, and
assessment that looks at the learning process in which students engage as they work toward certain outcomes.

Therefore, multiple assessment methods, including

· pretest/posttest questions,

· two assignments with checklists/grading criteria (scoring rubrics),

· self & peer assessments, and

· research logs

will be used. Past Heritage Research assignments are being modified and used as assessment instruments in this plan. Pretest and posttest questions, as well as self & peer assessments will track students’ knowledge and perceptions of their knowledge about the information literacy concepts they are taught. Inter-rater reliability in the grading of a website evaluation assignment will be gauged.

Objectives
The information literacy learning objectives addressed in this plan, are taken either word-for-word, or are adapted, from the ACRL/IS Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians <http://www.ala.org/acrl/guides/objinfolit.html>. See specific objective numbers in the chart, which follows. Targeted learning objectives deal primarily with two skill areas, namely, search protocols and evaluation criteria. Although additional learning objectives dealing with these two skill areas exist, attention is focused on those objectives deemed most appropriate to students taking a 1000 level course.

The Research Log will be tracked in Blackboard. It will be used later in the first semester and possibly during the second semester, to track further development of the skills in the two skill areas. The Research Log explores ACRL Information Literacy Competency Standard 4, Indicator 2, Outcomes 1 & 2, which state that the information literate student “maintains a journal or log of activities related to the information seeking, evaluating, and communicating process” and “reflects on past successes, failures, and alternative strategies.”

Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page one

Note: Deb Gilchrist’s framework uses “learning outcomes” as the starting point for planning assessment. Therefore, I am referring to the targeted objectives from the ACRL/IS Objectives document as learning outcomes. The planning chart details the outcomes (objectives), along with the associated curriculum, pedagogy, assessment and criteria undertaken in this plan. Numbers listed in the Learning Outcome column correspond to the Information Literacy Standard. Performance Indicator. Learning Outcome. Instruction Section Objective (e.g. 1.1.c.1).

Learning Outcome –

What do we want students to be able to do as a result of this instruction?

Curriculum –

What does the student need to know in order to do this well?

Pedagogy –

What activity will facilitate the learning?

Assessment –

How will the student demonstrate the learning?

Criteria –

How will I know that the student has done this well?

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

N/A (occurs before instruction)

Preliminary Questions about Evaluating Information (pretest to determine knowledge base prior to instruction)

Note: Completion of the questionnaires is required, but does not impact grades.

Students almost always or always use criteria.

They can list the five common evaluation criteria, which include accuracy, authority, currency, coverage and objectivity.

They use additional sources to verify information.

Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page two

Learning Outcome –

What do we want students to be able to do as a result of this instruction?

Curriculum –

What does the student need to know in order to do this well?

Pedagogy –

What activity will facilitate the learning?

Assessment –

How will the student demonstrate the learning?

Criteria –

How will I know that the student has done this well?

Demonstrate the ability to use search protocols appropriate to the retrieval system (finding tool) 2.1.c.7, including how to

- broaden/narrow topic 1.1.d.3

- subject search & synonyms

2.2.b.3

-keyword search 2.2.d.6

-truncation 2.2.d.7

-Boolean operators 2.2.d.2

-limit by format or type 2.2.e.3

-assigned subjects 2.2.c.4

-system help 2.1.c.2

Information containers (e.g. books, articles) and finding tools (e.g. catalogs, databases) in context

The eight search protocols

How to access Felix Online Catalog and Expanded Academic (EA) through the APSU Library website

How to use Felix and EA

Have students do a short brainstorming session (10-15 minutes) about information containers and finding tools

Present information about the cost of information and an overview of the nature of electronic information databases and catalogs

Provide information about search protocols which students will use in the first assignment

Introduce students to Felix and EA

Have students complete Expanded Academic tutorial

& explore Felix outside of class

Heritage Research Assignment (Using Search Protocols in Felix Online Catalog and Expanded Academic)

Assignment has students conduct searches on their Heritage paper/speech topics using the eight search protocols covered in class.

Students conduct their searches in Felix Online Catalog and in Expanded Academic.

Students write notes on Felix and EA printouts in which they demonstrate their understanding of the search protocols.

Protocol Checklist & Grading Criteria (scoring rubric)

Library instructors assess student learning by marking which search protocols are used on the protocol checklist

The Checklist helps determine the number of protocols used, as well as which protocols are used the most/least among all students

The Grading Criteria (scoring rubric) reflects

-whether students used both Felix and EA,

-how many protocols students tried in Felix and in EA, and

-how well their notes reflect an understanding of the protocols

Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU - page three

Learning Outcomes –

What do we want students to be able to do as a result of this instruction?

Curriculum –

What does the student need to know in order to do this well?

Pedagogy –

What activity will facilitate the learning?

Assessment –

How will the student demonstrate the learning?

Criteria –

How will I know that the student has done this well?

Determine the relevance of information sources to a given information need 2.4.a.4

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

Apply the five established evaluative criteria

--accuracy, authority, objectivity, currency and coverage -- to information sources 3.2.c.1 – 4

Information need & relevance

How to verify information

The five criteria and specific questions which address each criterion for evaluating information

Have students do a short brainstorming session (10-15 minutes) about information relevance and verification

Cover how other sources can be used to verify information

Present the five criteria with specific questions to student

Have students assist the instructor in using the criteria and associated questions to evaluate a website during class

Website Evaluation Assignment

Each student will complete a short self-assessment questionnaire on their completed website evaluation assignment.

Each student completes a peer assessment questionnaire on one of their classmates’ completed website evaluation assignment to provide information about student perceptions.

The results of the two questionnaires will be compared.

Completion of the assessment questionnaires is required, but does not impact grades.

Criteria Checklist & Grading Criteria (scoring rubric)

Student perceptions determined through the questionnaires will be compared with instructor-generated scores on the assignment.

Inter-rater reliability will be gauged by comparing additional librarians’ grading of a sample of the completed evaluation assignments.

Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart – APSU – page four

Learning Outcomes –

What do we want students to be able to do as a result of this instruction?

Curriculum –

What does the student need to know in order to do this well?

Pedagogy –

What activity will facilitate the learning?

Assessment –

How will the student demonstrate the learning?

Criteria –

How will I know that the student has done this well?

Using Blackboard, keep a research log demonstrating that they are able to

Identify and use search language and protocols appropriate to the retrieval system 2.1.c.7; 2.2.e.1; 2.2.e.3; 2.2.e.4; 2.3.a.5

Evaluate the quality of the information retrieved using criteria such as authorship, point of view/bias, date written, citations, etc. 2.4.a.2; 3.4.g.2; 3.4.g.3

How to use Blackboard

Search protocols (covered during the 1st class session)

Evaluation criteria (covered during the 2nd class session)

Show students how to use Blackboard to record their research logs

Divide students into groups and

1) Have each group use search protocols with a web search engine to find information on a Social Science topic

2) Have each group evaluate one of the websites they identify using the evaluation criteria

Research Log (under development)

Students reflect on the process they use to gather information for their Social Science papers.

Student’s reflections about this process will demonstrate the degree to which they are able to apply the search protocols and evaluation criteria they learned earlier this semester.

Analysis of research log to see the degree to which students are able to accomplish learning outcomes

Heritage Student Information Literacy Learning Outcomes Assessment Planning Chart- APSU – page five

Learning Outcome –

What do we want students to be able to do as a result of this instruction?

Curriculum –

What does the student need to know in order to do this well?

Pedagogy –

What activity will facilitate the learning?

Assessment –

How will the student demonstrate the learning?

Criteria –

How will I know that the student has done this well?

List the five common criteria (accuracy, authority, currency, coverage and objectivity) which can be used to evaluate information 1.4.b.3

Demonstrate an understanding that other sources may provide additional information to use in verifying or corroborating the accuracy and completeness of information 3.2.c.5

The five criteria and specific questions which address each criterion for evaluating information

No additional activities; the follow-up questions may be viewed as a learning activity

Follow-up Questions about Evaluating Information (posttest to determine knowledge base after instruction)

Note: Completion of the questionnaires is required, but does not impact grades.

Students almost always or always use criteria.

They can list the five common evaluation criteria, including accuracy, authority, currency coverage and objectivity.

They use additional sources to verify information.

The three resources you need for this assignment are all available on the opening APSU Library webpage at http://library.apsu.edu. They are

Second, refer to the Information Retrieval Concepts outline sheet, which we covered in class on August 31, 2001, as you conduct searches about your upcoming paper/speech topic in both

As you conduct your searches in these two resources, use the “Search Protocols” at the bottom of the Information Retrieval Concepts outline sheet. Be creative and show to the best of your ability that you understand how to conduct searches in BOTH resources using ALL EIGHT protocols.

You will need to conduct several searches in each source, print at least five records or screens from each resource, and write notes on each sheet of paper you print. PRINT YOUR NAME on each sheet and STAPLE them together. You will turn in these sheets as part of the assignment.

In your notes on each sheet, explain the protocol or protocols you tried out which resulted in the record or screen that you printed. You can use more than one protocol per search if you like. Make sure you include enough details in your notes so that we can follow what you were trying to accomplish in each of your searches.

The point is for you to experience each resource and show us that you understand and can apply the protocols. You will be graded (see Grading Criteria sheet) according to

You may have questions. If you do, please contact your library instructor. It is your responsibility to turn in your work on time. If you are not in class on the due date, turn in your assignment to the Library Information Desk. Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).

Search Protocols Used in FELIX

q Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

q Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

q Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

q Truncate your search term (e.g. famil* picks up family, families, familial)

q Use the Boolean operators AND, OR, NOT to search several concepts at once

q Use subjects that have been assigned to the source to find other sources

q Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

q Use help screens within database or catalog

Search Protocols Used in EXPANDED ACADEMIC

q Start with your specific topic (e.g. stem cell research); broaden (e.g. genetics) or narrow (e.g. parkinson’s disease and stem cell research) your search as needed

q Try searching by subject (e.g. death penalty) ; use different words to describe your topic (e.g. capital punishment) because sometimes the finding tool uses a different subject than you chose

q Try searching by keyword, which picks up words in the following fields in the record: subject, title, abstract or text, if available

q Truncate your search term (e.g. famil* picks up family, families, familial)

q Use the Boolean operators AND, OR, NOT to search several concepts at once

q Use subjects that have been assigned to the source to find other sources

q Limit sources by collection or format (e.g. videos) or by type (e.g. refereed/scholarly journals)

q Use help screens within database or catalog

Grading Criteria

Search and select a high-quality website about your topic, which will help you prepare your paper or speech, and examine it closely. Use the five established evaluative criteria and associated questions, which we covered in class and which were provided to you in written form, as a resource for completing this assignment. Additional information may be found by conducting a search for the following words in Google (http://www.google.com): criteria evaluation information.

Write a thorough typed double-spaced paper (at least 500 words) covering the website you identified. Give specific examples from your website. Your grade on this assignment will be based on how well you incorporate your responses to all of the questions below, as well as how well you address the criteria and questions covered in class.

2) What is the complete/full URL of your website? Print out and attach the first page of your website to your paper.

4) How accurate is this website? Are there spelling or grammatical errors? Does the website contain outrageous or misleading information? (Use the accuracy questions.)

5) Who wrote or sponsored this website? Is it an educational, commercial, governmental or organizational website? What audience was the website written for? Be sure to look in the website’s header, footer and any links for clues. (Use the authority questions.)

6) How objective is the website? Is a particular viewpoint presented in the website? If so, what is the viewpoint? If not, give evidence of how the website covers more than one viewpoint. Be sure to state what viewpoints are covered.

Give at least one specific example of a fact and/or an opinion covered by the website.

7) How current is this website? When was the information first posted? Has it been updated within the last year? If present, how current are the sources cited or links contained in the website? (Use the currency questions.)

8) What are the breadth and depth of the website’s coverage? What is its purpose? Does it cover a highly specialized topic area in great detail or does it merely provide a general overview of a broad topic? (Use the coverage questions.)

9) What are some steps you can take to verify the accuracy of the information contained in this website? (Be specific.)

Please contact your library instructor if you have questions. It is your responsibility to turn in your work on time. If you are not in class on the due date, turn in your assignment to the Library Information Desk. Your grade will be lowered by one letter grade for each calendar day that it is late (unless absence is excused).

Circle the ONE answer out of the available choices that comes closest to what you perceive your abilities to be. Please be honest. Completing this questionnaire is required, but your specific answers to the statements will NOT impact your grade.

1) Overall, based on the established evaluative criteria which were covered in class, I think the website I selected was high-quality.

2) As it relates directly to the topic about which I am writing and speaking, the website I selected is

3) I successfully applied the following number of established evaluative criteria in my Website Evaluation Assignment which I completed in this class.

4) Before taking this class, I applied evaluative criteria to information and its sources.

5) At the present moment, I can list from memory the following number of established evaluative criteria that we covered in class.

6) Before taking this class, I was in the habit of checking other sources to either confirm or question the point of view or bias of an information source.

Circle the ONE answer out of the available choices that comes closest to what you perceive your classmate’s abilities to be, based on your examination of her/his completed Website Evaluation Assignment. It will be helpful to review the assignment (attached) and the website your classmate evaluated before you read your classmate’s work and answer the questions.

Please be honest. Taking the time to complete this questionnaire in a serious fashion is required, but your specific answers to the statements below in no way impact your grade or your classmate’s grade. Your classmate will NOT see the results of your assessment.

1) Overall, based on the established evaluative criteria which we covered in class, the website my peer selected (see attached completed Website Evaluation Assignment) is high-quality.

2) In relation to the paper and speech topic my peer has selected (see attached completed assignment), the selected website is

3) In my opinion, my peer successfully applied the following number of established evaluative criteria which were discussed in class.

Paper Topic: ______________________________________________

Assessing Student Information Literacy Competencies

At Austin Peay State University (APSU)

IMLS/ACRL Assessment Project Report #2

Context

· incorporation of the new Information Literacy (IL) Standards,

· articulation of IL skills taught freshmen through graduate years,

· integration of IL concepts within the appropriate disciplines, and

· inclusion of assessment for information literacy learning outcomes,

Purpose

involvement of faculty, librarians, students and other campus constituencies in assessment efforts, and
assessment that looks at the learning process in which students engage as they work toward certain outcomes.