Some Types of Assessment to Consider
There is no completely satisfactory way to assess educational outcomes.
From this there are two possible conclusions:
1. If you can't do it right, just don't do it at all.
This is a dodge. You can use this argument to avoid doing anything you
don't want to do. There are very few things that can be done so well that
no one can find fault with the way they were done.
2. While the assessments may not be real good, they can provide useful
feedback for improvement of the quality of the educational process.
Since all assessments are fallible, you must interpret any results you get intelligently, with a thorough understanding of the context within which the assessment was done, and with awareness of the limitations of the assessment procedure.
Educational outcomes are complex, multifaceted things. The desired outcomes occur on many levels. Some of the higher level goals of the educational process are extremely difficult to assess ("Make them better citizens"). Some objectives are relatively easy to assess ("A college educated person should be able to locate Egypt on a world map").
No one assessment procedure can capture the "outcome" of a unit, course, program, or educational process. You have to use multiple convergent measures to attempt to bracket the outcomes.
Multiple convergent measures are complex, and difficult to interpret. Frequently the different measures send conflicting, even contradictory, messages. We are never surprised when "concrete" persons can't handle this. What is surprising is how often college professors and university administrators can't deal with it.
The Idealist Trap
The goal of student outcome assessment is to improve the quality of
instruction. And while even the most complex assessment process will fail
as the "ideal" outcome measure, relatively simple assessment
procedures can provide useful feedback that can help improve the quality
of instruction. It is better to know what percentage of the students can't
find Egypt on a map (partly because it is relatively easy to do something
about such a deficit) than to remain ignorant of outcomes because you can't
figure out a way to assess the higher level outcomes of the program in
some ideal sense.
The Generalist Trap
It is best to put your energy and resources into collecting information
that you can use.
Assessments need to be designed to have diagnostic specificity.
An assessment score that keeps coming out low, should provide direction
concerning what to do to correct the problem or at least suggest where
to look for the problem.
An example might help here:
The information that EKU Psychology majors score low on the GRE Advanced
test in Psychology provides us no information concerning what the weaknesses
in our program is. The GRE Advanced Test is a general test. It has no diagnostic
specificity. If however, we give a nationally normed subject matter test
in psychology which provides scores and percentiles for our students in
a number of areas, and we find that our students are scoring low on say
the "Abnormal Psychology" section of the test, we know we need
to beef up the content of that course (or at least find out what content
areas the national test is tapping that our students don't know).
The Complexity Trap
While it is costly and difficult (if it is possible at all) to assess some
of the higher level goals of a program, it is relatively simple to do assessments
at other levels. It is a mistake not to collect and use for decision making,
the data that is easy to collect. It is easy to get bogged down in the
development of complex assessment procedures and miss out on the assessment
value of information that is easy to collect, or the assessment uses of
data that we are already collecting (like final exams).
The Operationist Trap
Avoid the operationist trap. The trap here is that the data collection
process, no matter how obviously simplistic, tends to become the definition
of the outcome. We end up searching under the lamp post because the light
is better. One must continually guard against putting too much stock in
the assessment procedures which are adopted.
When we begin to use a test to assess the outcome of a course or program,
the test can soon "become" the outcome of the course. We teach
for the test and loose sight of the higher level goals toward which we
originally directed the course or program. We should not give up the higher
level goals, simply because they are difficult or impossible to assess.
This would be a very bad side effect of the student outcome assessment
movement.
1. Use multiple convergent measures.
2. Design all measures to have diagnostic specificity.
3. Keep it simple.
4. Use data that is already being collected when possible.
5. Don't give up your higher level goals just because you can't figure
out how to assess them.
6. Assessment won't happen unless you assign somebody to do it. (Usually
that means they need to prepare a report and submit it to the faculty).
7. It may be a while before the data you are collecting is useful for assessment.
In order to know if more of our students are voting in national elections
now than they did 10 or 15 years ago, we need to have been collecting data
for 10 or 15 years.
8. It will take several revisions and several years until a reasonably
satisfactory process is developed. The first version of your assessment
process or test is likely to be very bad. The second version may be better.
In three or four years (if you keep at it), it could be pretty good.
9. Assessment should take place within the greater context of planning
for department and curriculum change. It should be part of an organized,
ongoing effort to improve what you do.
1. Subject matter tests.
A comprehensive final, with a couple of integrative essay questions, may
be the best outcome assessment you have. You've been using the final to
assess the students. You can turn that around and use the final to assess
the course. If the students aren't able to integrate the material, what
are you doing wrong in the course? Are there more effective methods?
A departmental final, particularly in a multiple section course, can be valuable in assessing coverage of the course, providing uniformity in coverage, and in identifying areas that are being neglected.
A nationally normed subject matter test, which produces scores on topic areas which are of diagnostic significance, (if one exists in your area), can be a useful tool. The problem is that nationally normed tests cost money.
2. Pretest-Post test.
The emphasis here is on the gains made by students. This is sometimes called "value added" assessment. The focus should be on "talent development."
One of the easiest ways to assess progress if to give a version of the comprehensive final at the beginning of the course and compare scores to the final given at the end of the course. Of course such a test is designed primarily to assign grades to students and as such may not be completely appropriate for assessment of educational outcomes.
Alternatively, a specially designed pretest and post test can be used. The advantage is that such a test can include elements designed to have specific diagnostic value that may not be useful in assigning grades to students.
If your courses are more or less sequenced, you can give a pretest to all students in one of the beginning courses and give a post test in one of the last courses and compare the performance of students who take both tests. The advantage of this method is that some of the higher level goals can be assessed.
3. Student Portfolios.
Each student keeps a portfolio with samples of research and writing, journals, demonstrations, projects, performances, presentations, etc. The portfolio is started when the student becomes a major and is continued through the senior year with periodic evaluations to help the student grow and strengthen the portfolio. A sample of these is submitted to the outcome assessment committee. The committee reviews them to identify patterns of strength and weakness among the students and to make recommendations for curriculum and program changes based on these strengths and weaknesses.
4. Senior Project.
Students are required to do a senior project which requires integration of much of what they have learned in the program. The students do the project for a grade. A sample of these is submitted to the outcome assessment committee which prepares a narrative report detailing general strengths and weaknesses identified in these papers and recommendations for curriculum and program changes which arise from this evaluation.
5. Mastery Tests.
One way to assess student outcomes is to require students to demonstrate skills in certain areas. Alverno College requires all students to demonstrate mastery of eight abilities: critical thinking, problem-solving, communication, social interaction, valuing, responsibility to the global environment, effective citizenship, and aesthetic response.
6. Assessment of Attitudes and Beliefs and Values.
Most programs have goals beyond the subject matter or content related goals. Sometimes, attitude shift data can be useful in assessing these outcomes although it has relatively little diagnostic specificity. If the attitudes of the seniors aren't different, what do you do about it?
7. Behavioral Changes.
Some educational goals are changes in behavior. College graduates should be more likely to vote, more likely to read a newspaper or a book, more likely to watch a news documentary, less likely to watch "All my Children", etc. Americans are so used to polls, that most of them will be glad to tell you what they do and how often they do it. Unfortunately, this type of outcome often has less diagnostic specificity.
8. Exit Interviews.
Interviewing students who are leaving your program, (graduates, persons transferring out, persons changing their major to another major, etc.), can provide useful information on what you are doing right and what you are doing wrong. This can be very time consuming. It must be carefully structured to have diagnostic specificity.
9. Alumni Surveys.
The educational process should impact the students life after they get out of school. An alumni follow up survey can provide useful assessment information. In what areas did they feel ill prepared when they took their first job? In what areas were they better prepared than the people from other universities? What subject matter from the program do they use regularly, and what have they not thought about since they got out of the course? Are the salary ranges of graduates better now than they used to be? Are they staying in the field? Did they vote in the last election?
10. Other Measures.
A variety of other assessment procedures exist. They include: How many books in your area were checked out of the library? How many were checked out by majors? How many courses does the department list in the catalog? How often are these courses offered? Does the course content match the catalog description? Which courses are most attractive to students? Which courses are least attractive? Why? It should be possible to get attrition data on your students. How many of the graduates get into graduate programs. Some of this data should be available and can be included in an outcome assessment report.
Establish a outcome assessment committee charged with preparing an annual assessment report.
I'd like to meet with them or with faculty.
Start doing something.
Survey group to find out what is being done in various departments. Sharing information is important.
Course listings
Course offerings
Enrollment patterns (comparisons over the last few years)
Number of Majors, Number of people in each program, number of graduates, number of persons in service courses, distributions of majors of persons in service courses, number of FTE's generated, on campus, off campus, goals for expansion and changes
Assessment methods used
Participation (This semester only 4 of our 12 sections used the comprehensive department final, we hope to expand this somewhat next semester with an eventual goal of 100% participation)
Revisions to Assessment process made. (The subject matter pre-post tests were revised this semester by a committee of faculty who teach the course, of the 200 multiple choice items 78 were deleted 24 were re-written . . .)
Report on progress toward development of assessment procedures (A faculty committee has been working on an alumni follow up questionnaire. A database of alumni addresses is being developed by . . . )
Report of recommended changes based on assessments. (The assessment committee read and evaluated a sample of the integrative essay question for this course and found integrative skills of the students to be generally low but not completely inadequate. The committee has recommended the use of . . . )
Narrative evaluation based on interviews with students and faculty.
Summarize their comments.
Copyright © 1996 Steve Falkenberg