Institution description

Austin Peay State University (APSU), Tennessee’s designated public comprehensive liberal arts university is located in urban Clarksville (pop. 103,000), 45 miles northwest of the state capital, Nashville.  Of the 7,200 students enrolled at APSU, 63% are full-time students, 80% commute and 35% are non-traditional (22 or older).  Nearly 1,750 students attend the nearby Fort Campbell (Army) Center.  Over 1,500 students are enrolled in distance learning courses.  APSU’s educational emphasis is the liberal arts and professional programs; the Nursing and Education Programs are considered among the best in the State.
            The APSU administration’s commitment to information literacy instruction and assessment began in 1986 when the first User Education Librarian was hired.  An instructional facility equipped with 24 computers and instructional equipment, including a video networking system and LCD projector, was built in 1994; a third equipment replacement cycle just occurred.  In 2000, the User Education Librarian’s participation in the ACRL Information Literacy ’00 Immersion Institute received full financial support from the University.  Most recently, APSU matched the ACRL national assessment training project’s funding and approved leave for the User Education Librarian to attend training sessions during ALA Midwinter and Annual Conferences.  APSU faculty and librarians have a long history of collaboration, including work in the Heritage Program (see below).  Most recently, library faculty partnered with a communications professor to develop and teach a new online graduate Multimedia Literacy course in Fall 2001.  For more information about information literacy instruction and assessment at APSU, see http://library.apsu.edu/library/3_9.htm.

Project participants

A pilot project involving five library faculty members working with five English and communication faculty members occurred during Fall 2001.  This project built on a 13-year relationship among English, communication and library faculty members who collaborated together to create two courses (HUM 1010-1020) that are part of an alternative liberal arts core.  The assessment instruments used with HUM 1010 students were then modified and used during Spring 2002.
            The User Education Librarian worked with four English instructors in six English 1010 class sections during Spring 2002.  Instructors selected for the project had involved their students in library instruction during previous semesters and the User Education Librarian had worked with each of them before.  Assessment conducted with these six English 1010 composition classes is the subject of this project.  Demographics of the students involved are as follows:

• Gender: 87 women, 50 men
• Ethnicity: 90 Caucasian, 31 African-Americans, 9 Hispanic and seven Other
• Age range: 17 to 50; mean age = 22
• Student Status: 83 First-time, 31 Continuing, 12 Re-entry and 11 Transfer
• Class Level: 115 Freshmen, 16 Sophomores, 4 Juniors, and one Senior
• Residence: 100 Commute and 37 Live on-campus
• Instruction: 106 had no prior college library instruction, 31 had prior instruction

Project description

            Assessment at APSU prior to involvement in this national project was limited primarily to student perception survey questionnaires and essays, as well as instructor feedback and a review/revision process conducted by librarians each semester.  In this project, APSU librarians focused on learning outcomes related to evaluative criteria.  By the end of the project, librarians had gained assessment experience with nine classes involving over 150 students.  The following information relates directly to the six English 1010 class sections with which the User Education Librarian worked during Spring 2002.
            The selected instructional planning process utilized a framework (model) taught by Deb Gilchrist (Pierce College Library Director) during the Information Literacy Immersion Institute and in various other workshop settings.  Learning outcomes, curriculum, pedagogy, assessment and criteria, including questions that address each of these instructional design elements, are the focus of this model.  A chart was used to record these elements as they were planned.  First, specific learning outcomes were identified.  This was accomplished by examining both the ACRL/IS Objectives for Information Literacy Instruction (see http://www.ala.org/acrl/guides/objinfolit.html) and the ACRL Information Literacy Competency Standards for Higher Education (see http://www.ala.org/acrl/ilcomstan.html).  The instruction and assessment targeted one manageable skill area (e.g., evaluative criteria) that was not overly ambitious and that was achievable by the level of students who were being taught.  Next, the assessment instruments were developed.  The learning outcomes which students were expected to achieve were carefully considered in the design of the instruments.  Once assessment instruments were developed, the criteria describing the expected level of student performance were written.  Having the learning outcomes, assessment and criteria in mind was crucial when it came time to consider curriculum, which would prepare students to perform well in the assessment stage.  Finally, employing good pedagogy (learning activities) ensured that students met the learning outcomes.
            Six assessment instruments for gathering information about student learning outcomes related to ACRL IL Competency Standard Three were developed by the APSU User Education Librarian in consultation with other librarians teaching in the Heritage Program.   The instruments include 1) a preliminary questionnaire (pretest) and 2) a followup questionnaire (posttest) designed to measure student knowledge of the five established evaluative criteria (authority, accuracy, coverage, currency and objectivity).  These instruments also gathered information concerning the degree to which students use evaluative criteria and verification methods.  The pretest and posttest asked identical questions.  (3) A Web site evaluation assignment was designed to measure student application of the five criteria listed above, as well as application of information verification methods.  A grading criteria (rubric) sheet with which to score the Web site evaluations was incorporated into the Web site evaluation assignment so that students were aware of how the assignment was to be graded.  Two questionnaires were designed to assess student perceptions of (4) their own abilities to successfully apply the established evaluative criteria, as well as (5) their peers’ abilities.  Finally, a questionnaire was created to assess the (6) library instructor’s perceptions of students’ abilities to successfully apply the established evaluative criteria.  Copies of these instruments follow this description.
            The study employed a quasi-experimental design using primarily a pretest/posttest analysis, which utilized both parametric and non-parametric techniques.  All data were matched (i.e. pretest, posttest) within three different instructional conditions (i.e. none, active learning, lecture).  Data was entered into a spreadsheet-type statistical program, SYSTAT (Version 10, ©SPSS Inc., 2000).  Following the study’s research questions, response frequencies were calculated and the comparisons were made using statistical tests such as Chi Square analyses and Analyses of variance (ANOVA).
           Six classes (141 students) were divided into three groups as follows:

• A control group of 48 students participated in the same activities (see below) as the other groups with one exception, that is, they did not receive instruction from the librarian until after all assessment was completed.
• An active learning experimental group of 47 students completed the same activities and participated in an active learning instructional session on evaluative criteria.
• A lecture experimental group of 46 students completed the same activities and participated in a lecture-formatted instruction session on evaluative criteria.

Activities completed by all students included

• a preliminary questionnaire (pretest) prior to receiving instruction;
• an instructional session (except for the control group);
• a Web site evaluation assignment;
• a self-assessment questionnaire;
• a peer assessment questionnaire; and
• a followup questionnaire (posttest).

Out of 141 students, 112 completed both the pretest and the posttest. The Web site evaluation assignment was completed by 109 students.
            Challenges encountered in this project were overcome through persistence and hard work.  First, the three HUM 1010 sections assessed during Fall 2001 contained too few students to produce significant results.  Therefore, a new assessment study was developed to use with the six English 1010 classes in Spring 2002.  A second challenge was coordinating the timetable for gathering data; with six instruments to administer in six classes, careful planning and time management were vital.  The third and final challenge involved obtaining a good assessment return rate.  A thorough analysis of the assessments received and persistent followup with students who had not turned in all their work paid off in a reasonably high return rate (approximately 79%).
            Assessment is a logical extension of the groundwork laid in establishing information literacy partnerships at APSU.  The training and experience gained through participation in this project will allow APSU to continue assessing information literacy instruction.  Specific knowledge gained in this project addresses the need to employ active learning in instruction to achieve the best outcomes.  Ongoing assessment activities will enable APSU librarians to shape instruction to best meet the needs of a changing student population.

Project Results

Student knowledge, application and perceptions relating to evaluative criteria are the focus of this project.  For more details about this project, see http://library.apsu.edu/library/3_9assess.htm.

Knowledge - As a whole, students listed an average of one criteria on the pretest, while averaging three criteria on the posttest.  No significant differences appeared among groups on the pretest; however, posttest results among the groups were significantly different.  While only four control students listed three to five criteria, 32 active learning and 10 lecture students successfully listed three to five criteria, F (2, 109) = 60.47, p = .001.  Notably, 19 of the 41 active learning students listed all five criteria on the posttest, compared to two lecture and no control students. 
            Student knowledge as a whole increased between the pretest and the posttest with respect to individual criteria.  As evidenced in the table below, the number of students who identified specific criteria is listed for both the pretest and the posttest.

            Significant differences among groups emerged as a function of instruction.  The table below illustrates the differences in students’ knowledge of the evaluative criteria.

Application - When students completed the Web site evaluation assignment, 49% were able to successfully apply all five criteria.  Students appeared to have the easiest time applying the coverage and currency criteria, while having more trouble applying authority, accuracy and objectivity.  All 41 active learning students successfully applied both coverage and currency; all 31 of the control students successfully applied coverage.  Four (lecture) students were unsuccessful with coverage; ten (five control and five lecture students) were unsuccessful with currency.  The accuracy criterion was successfully applied by all but five active learning students; nine control students and sixteen lecture students had trouble applying accuracy.

Perceptions - Students’ perceptions of the number of criteria that they could list increased over the course of the study.  Initially, they were able to list only one criterion.  Immediately following the Web site assignment they believed they were able to list four.  However, in reality, they were only able to accurately recall three criteria, F (2, 196) = 193.38, p = .001.  Of the students completing self-assessments, 88 of the 105 believed they could list three to five of the criteria at that point in time; on the posttest, 46 of the 112 students actually did list three to five criteria.  Finally, when students were asked to self-assess the number of criteria they applied in the Web site evaluation assignment, their assessments were significantly correlated to that of the instructor, r (98) = .32, p= .003.  However, peer assessments were not correlated to either the instructor or the self.

------------------------------------------------------------------------------------------------------------------------------------------------

Preliminary Questions About Evaluating Information

      Your Name  _______________________________

1)        I use criteria to evaluate information.  Circle one.

Never --- Almost never --- Sometimes --- Almost always --- Always

2)        List five criteria for evaluating information.  

1.

2.

3.

4.

5.

3)        Mark the steps listed below which apply to the following statement.   

I verify information by

___            communicating with the author

___            consulting another source

___            asking a friend

___            I don’t; instead, I assume the information is correct

___            Write any other steps you take below

Follow-up Questions About Evaluating Information

                 Your Name  ______________________________

1)        I use criteria to evaluate information.  Circle one.

Never --- Almost never --- Sometimes --- Almost always --- Always

2)        List five criteria for evaluating information.

1.

2.

3.

4.  

5.

3)        Mark the steps listed below which apply to the following statement. 

I verify information by

___            communicating with the author

___            consulting another source

___            asking a friend

___            I don’t; instead, I assume the information is correct

___            Write any other steps you take below

Website Evaluation Assignment

Due in Class on ____________________

Your Name  _______________________________________________

Search and select a high-quality website about a paper topic you plan to write this semester.  Examine the website closely.  Use the five established evaluative criteria (accuracy, authority, currency, coverage and objectivity) and associated questions below, as a resource for completing this assignment.

Write a thorough typed double-spaced paper (at least 500 words) covering the website you identified.  Give specific examples from your website.  Your grade on this assignment will be based on how well you incorporate your responses to all of the questions below, as well as how well you address the criteria (accuracy, authority, currency, coverage and objectivity).

1)      What is your topic?

2)      What is the complete/full URL of your website?  Print out and attach the first page only of your website to your paper.

3)      How well does this website cover your specific topic or information need?  How useful will it be in your paper? (Be specific.)

4)      How accurate is this website? Are there spelling or grammatical errors? Does the website contain outrageous or misleading information?

5)      Who wrote or sponsored this website?  Is it an educational, commercial, governmental or organizational website?  What audience was the website written for?  Be sure to look in the website’s header, footer and any links for clues.

6)      How objective is the website?  Is a particular viewpoint presented in the website?  If so, what is the viewpoint?  If not, give evidence of how the website covers more than one viewpoint.  Be sure to state what viewpoints are covered.

Does the website present

·        Facts

·        Opinions

·        Both facts and opinions

Give at least one specific example of a fact and/or an opinion covered by the website.

Over

  ___________________________________________________________________________________________________

7)      How current is this website?  When was the information first posted?  Has it been updated within the last year? If present, how current are the sources cited or links contained in the website?

8)      What is the breadth and depth of the website’s coverage?  What is its purpose?  Does it cover a highly specialized topic area in great detail or does it merely provide a general overview of a broad topic? 

9)      What are some steps you can take to verify the accuracy of the information contained in this website?  (Be specific.)

This is the grading criteria:

  _______________________________________________________________________________________________

Self-Assessment of Information Evaluative Skills

Your Name  _______________________________________

Circle the ONE answer out of the available choices that comes closest to what you perceive your abilities to be.  Please be honest.  Completing this questionnaire is required, but your specific answers to the statements will NOT impact your grade.

1)     Overall, based on the established evaluative criteria, I think the website I selected was high-quality.

Yes                     No

2)     As it relates directly to the topic about which I plan to write a paper, the website I selected is

of no use      of very limited use      of some use    useful      very useful

3)     I successfully applied the following number of the established evaluative criteria in my Website Evaluation Assignment.

one       two                three                    four                     five

4)     Before taking this class, I applied evaluative criteria to information and its sources.

never   almost never              sometimes     almost always      always

5)     At the present moment, I can list from memory the following number of established evaluative criteria covered in the assignment.          

one       two                three                    four                   five

6)     Before taking this class, I was in the habit of checking other sources to either confirm or question the point of view or bias of an information source.

never       almost never       sometimes           almost always   always

  ________________________________________________________________________________________________________

Peer Assessment of Information Evaluative Skills

Your Name  ________________________________________

 Circle the ONE answer out of the available choices that comes closest to what you perceive your classmate’s abilities to be,  based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website your classmate evaluated before you read your classmate’s work and answer the questions.  

Please be honest.  Taking the time to complete this questionnaire in a serious fashion is required, but your specific answers to the statements below in no way impact your grade or your classmate’s grade.  Your classmate will NOT see the results of your assessment.

1)      Overall, based on the established evaluative criteria, my peer selected (see attached completed Website Evaluation Assignment) is high-quality.

Yes                        No

2)      In relation to the paper topic my peer has selected (see attached completed assignment), the selected website is  

of no use of very limited use      of some use    useful      very useful

3)      In my opinion, my peer successfully applied the following number of established evaluative criteria that were either discussed in class and/or found within the assignment.        

one                   two                      three              four                         five

Instructor Preliminary Assessment of Information Evaluative Skills

Student Name  _______________________________  

Circle the ONE answer out of the available choices that comes closest to what you perceive the student’s abilities to be, based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website the student evaluated before you read the student’s work and answer the questions.  

The student will NOT see the results of this particular assessment.  Please forward this sheet to the librarian as soon as you complete the preliminary assessment.

1)      Overall, based on the established evaluative criteria, the student selected (see attached completed Website Evaluation Assignment) is high-quality.

Yes                        No

2)      In relation to the paper topic the student has selected (see attached completed assignment), the selected website is

of no use of very limited use      of some use    useful      very useful

3)      In my opinion, the student successfully applied the following number of established evaluative criteria.            

one                   two                      three              four                         five