WRITING ASSESSMENT AT HOPE COLLEGE by Dr. Richard Ray Fall 2000

Last year the Assessment Committee initiated an effort to collect direct measures of student abilities for two of the most critical aspects of our general education program: writing and quantitative literacy. Although the data for the quantitative literacy pilot project will not be available until Christmas, the results of the first round of writing assessment have been analyzed and are presented here for your information. Keep in mind that these data are based on only 86 papers from senior seminar students in a single semester. Before drawing firm conclusions about our students' writing, we will have to collect more data over a longer period.

Methods

Students were asked to write a 2-3 page "Life View Paper Proposal" in their senior seminar course during the Fall '99 semester. Students were given specific instructions on how to prepare the paper. Students submitted two copies of the papers. One anonymous copy was used by the Assessment Committee while the second copy with the student's name was used by the professor to provide feedback to the students on their plans for their life view papers. Eighty-six usable papers were obtained. Students also completed a scantron form with the following self-reported measures:

* Major

* Courses involving writing

* Types of writing they have done at Hope College

* Perception of their writing ability compared to other Hope College students

* Percentage of professors who took and active role to improve their writing

* GPA

* Grade in English 113

Six faculty members were trained to rate the papers on a three-point scale on the following elements: focus, content development, organization, style, grammar/mechanics, and overall fulfillment of the assignment. Raters used the following definitions when scoring the subscales:

1 - Never. This feature of competent writing in this context is virtually nonexistent.

2 - Often. This feature of competent writing in this context generally outweighs its absence.

3 - Always. This feature of competent writing in this context virtually never falters.

A mean score was calculated for each of the papers from the subscale scores. Each paper was scored independently by two readers. Papers whose mean scores differed by more than one point were read by a third rater. There was a high degree of inter-rater reliability. Only six papers (6.9%) required a third reader.

Results

Although "overall fulfillment of the assignment" was the weakest area of writing in this sample, there was a high degree of consistency between all the writing subscales. Students in this sample scored the highest on "organization."

The best writers in this sample were those students who received the highest grades in English 113, have the highest GPA, and who took courses that emphasize short essay writing assignments. Another interesting finding is that the best writers seem to recognize their abilities in this area and consistently rank themselves near the top compared with their peers. There were no strong correlations between students' writing ability as measured on this test and their experience in courses that required them to write longer essays, research papers, or lab reports.

Students' writing abilities for a particular writing subscale were generally correlatedto their abilities in each of the other five subscales. In other words, if a student was well organized in their writing, chances are good that they were also well focussed. This relationship broke down in only in the area of content development, with relatively low correlations between this subscale and the style and grammar/mechanics subscales.

A higher percentage of students in the arts, humanities, and natural sciences had top scores on the assignment than did students in the social sciences. This finding should be interpreted with caution, since none of the senior seminar sections normally taken by education students participated in the study. Had these students been part of the sample, the social science aggregate data may have looked much different.

An examination of the distribution of writing assessment scores grouped by self-reported English 113 grades reveals two interesting findings. We divided the sample into three groups - those that scored in the top one-third on the writing assessment exercise, those that scored in the middle one-third, and those that scored in the lowest one-third. As might be expected, the group with the highest scores had the greatest percentage of students who earned either and A or A- in English 113. The other finding of note is that over 50% of the students who scored in the lowest one-third also received a grade of A or A- in English 113. What accounts for this deterioration in writing ability?

The female students in this sample tended to be the better writers. Seventy percent of the female students scored in the top one-third while 68% of the male students scored in the lowest one-third.

Discussion and Future Directions

Assessment of students' writing abilities is a complex process. This task is made even more difficult when attempted over an entire student body and across academic majors that often try to develop very different sets of communication skills in their students. The kind of writing assignment that was used in this assessment was chosen because the Committee felt that it represented a type of writing that most of our students will have to do at some point in their lives - a short, persuasive proposal. The generalizability of these data to other kinds of writing is unknown. Even though these data were drawn from a relatively small cross-section of the student body and the exercise focussed on only one kind of writing, this project does give us the first glimpse of our students' writing ability that we've ever had. Although we should avoid the temptation to make sweeping curricular changes based on these data, the Assessment Committee feels we should continue to build the database by repeating the project over the next few years. By doing so we will eventually have enough data upon which we can make recommendations for this important part of our general education program.

There are presently seven senior seminar sections involved in this year's data collection. We plan to rate the papers and analyze the data over the Christmas break and report back to the faculty during the second semester. One change we are discussing involves moving from a three-point to a four-point rating scale in an effort to discriminate more finely between papers of differing quality. If any of you have suggestions for how our writing assessment project can be improved, please contact Rich Ray at ray@hope.edu or ext. 7708.