A SECOND YEAR OF WRITING ASSESSMENT AT HOPE COLLEGE by Dr. Richard Ray Spring 2001

Last Fall I reported to you that the Assessment Committee had completed a first round of writing assessment (see here). The purpose of this report is to let you know what we found when we conducted the second round of this project. The second round of the writing assessment project was conducted in exactly the same manner as the first, with one exception that I'll describe below. Students in six senior seminar sections were asked to write a 2-3 page "Life View Paper Proposal." They were provided with written instructions for what to include in the proposal. Students submitted two copies: an anonymous copy that was given to the Assessment Committee for analysis and a copy with the student's name on it that was turned in to the professor to be used in any way he or she deemed appropriate. One hundred and six usable papers were submitted for analysis. Students also provided the following information on a form that was readable by the Frost Center's survey scanner:

• Major
• Courses involving writing
• Types of writing they have done at Hope College
• Perception of their writing ability compared to other Hope College students
• Percentage of professors who took and active role to improve their writing
• GPA
• Grade in English 113

Five of the six faculty readers had also served as readers for the first round of writing assessment in December 1999. Faculty readers rated papers on a four-point scale on the following elements: focus, content development, organization, style, grammar and mechanics, and overall fulfillment of the assignment. During the first round of writing assessment a three-point scale was used. Since we had a more experienced group of trained readers for the second round, we thought we would be able to discriminate more finely than during the first round, so we used a four-point scale rather than a three-point scale. The following definitions were used to rate the various elements of each paper:

1 - This feature of competent writing, in this context, is clearly absent.

2 - This feature of competent writing, in this context, is often absent.

3 - This feature of competent writing, in this context, is often the norm.

4 - This feature of competent writing, in this context, is clearly the norm.

A mean score was calculated for each paper from the subscale scores. Two readers scored each paper independently. A third rater read papers whose mean scores from the first two raters differed by more than one point. There was a high degree of inter-rater reliability. Only three papers (2.9%) required a third reader.

Results

The results of the 2000 writing assessment project are similar to those obtained in 1999. Like 1999, it was difficult to identify a writing subscale that is clearly weakest. "Organization" was still the strongest aspect of these papers. "Overall fulfillment of the assignment" ranked near the bottom of the 2000 sample, and was the weakest area in 1999.

Two areas present consistent findings from the two samples. First, the students with the highest self-reported GPAs

were generally the strongest writers. Also, students who report more experience in journal and short response kinds of writing tended to score higher on the life view paper proposal. Neither of these findings is particularly surprising. It will be interesting to see what trends become evident after a third (and we think, final) round of writing assessment next fall.

Students participating in the 2000 writing assessment were more consistent than those in 1999 in terms of the correlations between the various writing subscales. In 2000, those students who demonstrated strength in any given writing subscale tended to be strong in the other elements of writing as well. The same can be said of students who demonstrated weak writing on any particular subscale.

In 1999 we noticed that students in the social sciences lagged a bit behind students in the arts, humanities, and natural sciences in terms of their scores on this writing assignment. The data for the 2000 sample show only minor differences between the academic divisions. Students who double majored across academic divisions (Biology-Religion, for example) scored higher on this assessment than other students, but the reasons for this are unclear.

In 1999 we noticed that a higher than expected percentage of students who scored in the lower third of the sample had self-reported English 113 scores of A or A-. This seemed unusual, and we wondered if this represented a deterioration of students' writing skills between their first and fourth year of college. The 2000 sample, however, is distributed in a manner that is more reassuring. Only 36% of the students who scored in the lower third of the sample

reported earning a grade of A or A- in English 113, while 70% of the students in the top third earned similar grades. This finding should be interpreted carefully. Students' grades in English 113, like many other courses, are often based on improvement over time. The writing in that course is often graded only after several drafts and revisions, usually with feedback from a professor. The papers written for this assignment may not (and in fact, probably were not) similarly edited or otherwise revised. Furthermore, students' motivation for this task may be lower than average, since professors typically did not tie course grades to this assignment. In other words, the kind of writing in English 113 may likely be quite different from this writing assignment in several aspects.

Like the 1999 sample, female students in the 2000 sample tended to be better writers than their male peers. More than twice as many female students scored in the top third on the writing assessment (33.9% of females vs. 14.3% of males).

Discussion and Future Directions

The writing assessment project has given us an interesting perspective on the strengths and weaknesses of our students' writing. As a result of the data we have collected to this point, we can say that we know more about the quality of our students' writing than at any time in the past. Still, it is prudent that we acknowledge the limitations of our approach as we think about how to use the information. Although the method we have used the past two years seems to be reliable, it has only been tested using one kind of writing - a short, persuasive proposal. We should be careful about making assumptions about how well our students write when faced with other kinds of writing challenges like technical reports or more creative works. Also, we know that the best writing is usually the product of editing and revision. It is unclear to us how much editing has gone into the writing samples collected over the past two years. We intend to ask the students in the 2001 sample questions designed to discern the degree of effort students put into this paper.

Our plan is to collect one more round of writing samples using this method during the fall of 2001. We will then have three years of data with which to work. We hope to be able to analyze the data and present it to the Academic Affairs Board during the Spring, 2002 semester. It may be appropriate at that time to convene a campus-wide conversation on the implications of the findings, perhaps in the form of a faculty task force on writing improvement.

If you have any comments regarding the writing assessment project, please contact Richard Ray (ray@hope.edu or 7708). The Assessment Committee extends its thanks to the senior seminar faculty and the faculty readers who have helped make this important project possible.