Department of Foreign Languages & Literatures
Outcomes Assessment Report
April 1, 2002
Prepared
by Terry L. Papillon,
Assistant
Chair of the Department and Associate Professor of Classics
Department
Chair: Judith L. Shrum
Part I. Introduction
The Department of Foreign Languages &
Literatures works under the following mission statement, and the assessment of
its activities is carried out with a view to its priorities:
The Department of
Foreign Languages and Literatures provides students the opportunity to gain a
knowledge of the culture, literature, structure, and vocabulary of the diverse
languages offered. The department seeks to graduate majors and minors with an
advanced command of the written and spoken language and with the general
analytical, critical and communicative skills and global outlook essential to
an educated member of society. The department fulfills a service mission
through offering foreign languages for students who must meet the foreign
language requirement, for students interested in language study for
intellectual and cultural enhancement, and for students and faculty in other
disciplines for whom a knowledge of foreign languages and cultures is a
valuable professional tool. In all cases, the department strives to broaden
students' awareness of the nature of language and of cultural and national
diversities and similarities. It also strives to break down the barriers of
ethnocentrism and xenophobia, thus increasing communication and understanding
on an international scale. The department strongly promotes the scholarly and
pedagogical activities of its faculty, who advance, preserve and transmit the
collective knowledge of the discipline through research, publication, teaching,
and participation in professional life.
The categories for evaluation rose from the
work of an ad hoc committee during the spring semester 2001 (see Appendix 2 for
the report of that committee) and the actual assessment occurred during the
academic year 2001-2002. The categories of assessment include:
Preparation
for graduate school
Oral
and Written proficiency in the target language
Preparation
for global/interdisciplinary careers
Technological
literacy
Critical
thinking skills
Ability
to use cultural knowledge
Service
to non-major students through the core
Service
to non-major students outside core courses
Quality
of teaching
Instruments used to collect data include:
Senior
oral exit interviews, conducted during May 2001 by the Department Chair
Alumni
surveys, conduced in fall 2001. 26 surveys from graduates from 1998-2001.
Oral
proficiency interviews of ten graduating seniors done by nationally certified
ACTFL examiners in phone interviews, conducted during spring 2002.
Phone
interviews with four employers conducted by the Assistant Chair of the
Department about recent graduates who had given permission for such interviews.
SPOT
form data[1]
taken from a random sample of 12 courses from 1998 and 2001.
Paper
survey of degree candidates during fall 2001,
prepared by the Academic Assessment Program and conducted by Department
faculty.
See Appendix 2 for how each instrument related to
each category. In some cases, more instruments were used than were called for
in the preliminary report (Appendix 2), simply because they were available. The
Senior Exit Interviews and employer phone interviews, for example, offered
information on most of the topics, and so they are included in each section. In
other instances, not all instruments suggested were available and others had to
be substituted with other appropriate measures.
In addition to the above measures, it should also
be pointed out that the Department’s requirement of a minimum grade point in
the major courses ensures a basic level of competency. The Department also requires that every student in the major
demonstrate an oral proficiency level as set by the American Council on the
Teaching of Foreign Languages (ACTFL). This verification traditionally happens
at the point of transition from lower division to upper division coursework.
Meeting the level of “advanced low” ensures that the students have the basic
language skills in speaking and listening to continue with the coursework for
the major. This verification happens in two ways: a student can pass the Oral
Proficiency course (French/German/Spanish 3126) which includes an interview
with a faculty member trained in giving the Oral Proficiency Interview or be
exempted from this course based on an oral interview with a faculty member.
Part II. The Assessment
According to the Alumni survey, 29% thought their
preparation for graduate school was EXCELLENT, 33% said it was SATISFACTORY, 4%
said NOT SATISFACTORY, and 35% said it was NOT APPLICABLE to them. Thus, 94% of
those who answered the question thought that their preparation was satisfactory
or excellent.
Question 8 in the Academic Assessment Survey, which
surveys the topic “Prepare for graduate school,” had the following responses:
8% said "did not achieve," 42% said "achieved to a small
degree," 17% said "achieved to a moderate degree," 33% said
"achieved to a large degree," and 0% said "definitely
achieved." Of the 12 respondents
on this survey, only one said they were planning to attend graduate school in
the fall, though 6 said they would attend within the next five years.
In the Senior Exit Interview Report (Appendix
3), the Department Chair noted the following: “They felt adequately prepared in
writing but not in speaking and wished for more emphasis on speaking at the
4000-level.”
According to the Alumni survey, 40% thought their
proficiency was EXCELLENT, 56% said it was SATISFACTORY, 4% said NOT
SATISFACTORY, and 0% said it was NOT APPLICABLE to them.
The Assistant Chair of
the Department selected at random 10 seniors graduating from December 2001 to
August 2002; of these, 8 were majors in FL and 2 were not. These 10 students
were given an Oral Proficiency Interview
(OPI) via telephone by an external examiner who is certified by ACTFL. On the
ACTFL scale from novice low to superior, students need to be at the 'advanced
low' level (7 on a 10 point scale as reported here) to pass their oral
proficiency level at the beginning of upper division work (see comments on oral
proficiency requirement above). At the time of graduation, the 10 students
surveyed had an average score of 7.5, or half way between 'advanced low' and
'advanced mid' when they graduated. The 8 majors, however, averaged 8.1, or
slightly above 'advanced mid.' In the 1997 Outcomes Report we only reported
outcomes for professor-examined students in the teaching option. This year we examined majors and non-majors
using ACTFL certified external examiners.
While direct comparisons are not possible, we have a more reliable
measure of performance showing that the outcome performance is within our
expected range.
In phone
interviews with employers of
recent graduates, on the question of oral proficiency, not all graduates are
involved in occupations that use their languages, but for those who were the
responses were that employees were 'above average' or 'far above average' in
their abilities.
In
the Senior Exit Interview Report
(Appendix 3), students asked for more upper-division classes in which speaking
could be developed even further, either a 4000-level oral proficiency class, or
reorganization of upper-division courses to allow for less teacher-centered
lecture formats and more interactive student participatory activities. Some
sections and classes have already begun to move in this direction.
Of
the 26 graduates who returned alumni
surveys, 24 said that study abroad should be required. Of the 2 who said it
should not, one had studied abroad and one did not indicate whether she/he had
studied abroad. Students talked about the importance of this for proficiency
and cultural knowledge.
Preparation for global/interdisciplinary careers
According to the Alumni survey, 27% thought their preparation for global or
interdisciplinary careers was EXCELLENT, 62% said it was SATISFACTORY, 12% said
NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.
In phone
interviews with employers, on
the question of global or interdisciplinary preparation, the responses were
that employees were usually 'above average' in their abilities. One new
employee is being nominated for the 'best first year teacher' award because of
his work with other faculty and resource people in the school district.
In the Senior
Exit Interview Report (Appendix 3), the Department Chair offered the
following comments: Students felt well prepared in all areas except speaking,
and here they noted a stark difference between what they study and what their
careers will expect of them. Students asked for more courses in technical
writing related to their field, a class on current issues in the target country
in general as well as courses with targeted information related to
agricultural, business, and engineering fields. Students were well aware of the
difference between learning language in instructed settings and using it in
real settings. They felt unprepared for
the real settings and suggested more such opportunities, accompanied by
contextualized courses to reinforce their understanding of language structure,
either linguistic or grammatical.
Technological literacy
According to the Alumni survey, 27% thought their technological literacy was
EXCELLENT, 62% said it was SATISFACTORY, 12% said NOT SATISFACTORY, and 0% said
it was NOT APPLICABLE to them.
In the Senior
Exit Interview Report (Appendix 3), the Department Chair offered the
following comments: Students felt that their some of their classes in this
department were more high-tech than some of their engineering classes. They enjoyed faculty use of web chats,
international videoconferencing, grades on line, on-line courses, research
links on line, use of videodisc programs, writing forums, and electronic
portfolios. They appreciated the distance learning courses but found it
difficult to engage in peer editing using that format. They liked having
insights into the research conducted by faculty, citing the case of one
professor who put a catalogue of language change on line as a result of his
research. They said that the student computer lab available in the department
removed a barrier for them, making it easier to complete technologically
related assignments right here where the faculty members are also available for
help if needed.
According
to the Alumni survey, 46% thought
their critical thinking and problem solving skills were EXCELLENT, 50% said it
was SATISFACTORY, 0% said NOT SATISFACTORY, and 4% said it was NOT APPLICABLE
to them.
In phone
interviews with employers, on
the question of critical thinking skills, the responses were that employees
were consistently 'above average' or 'far above average' in their abilities.
One employer commented about how the employee 'always digs, going to other
people in the county as resources for situations that need to be treated.'
Another employee was meant to be an intern for two years, but her advisor left
for another job, and so she has had to cope with learning her job herself and
was commended for being "a fighter" and finding out what she needed.
A third, new teacher, finds herself preparing for a move from one building to a
split school in two buildings; the principal praised her work dealing with such
a shocking change in just her first year.
In
the Senior Exit Interview Report
(Appendix 3), the Department Chair offered the following summary: Students felt
that 4000-level literature classes were helpful in showing them a step-by-step process
to look deeply into a literary work and analyze it. In addition, writing
intensive classes were helpful in showing them how to construct and edit
quality arguments in a comprehensive essay. German students pointed out that
they were still operating at the level of comprehension, unable to move toward
analysis, and that they relied on the professor more than they wished,
resulting in more lecture format than their preferred discussion format. French
students asked that the selection of literary topics be modernized a bit,
focusing on political systems, culture, popular press, the target country
today, current issues. Spanish students liked the recombination of
Civilization/Culture/Literature courses at the 3000-level. Spanish students
also asked for more discussion format in some classes that are largely
teacher-centered.
In the Academic Assessment Survey, question 2,
which surveys the topic “Improve Intellectual Skills (critical thinking,
logical reasoning, problem-solving ability)” had the following responses: 0%
said "did not achieve," 0% said "achieved to a small
degree," 25% said "achieved to a moderate degree," 33% said
"achieved to a large degree," and 42% said "definitely
achieved."
According
to information acquired from SPOT forms
(Appendix 1) student who evaluate their courses at the end of the semester also
had favorable views of their gains in critical thinking skills. All twelve
courses analyzed had a score of 85% or better where those who responded ranked
their gains in this area as average or more than average. Performance was about
the same from 1998 to 2001.
According
to the Alumni survey, 61.5% thought
their ability to use cultural knowledge was EXCELLENT, 38.5% said it was
SATISFACTORY, 0% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to
them.
In phone
interviews with employers, on
the question of cultural knowledge, the responses were that employees were
consistently 'above average' or 'far above average' in their abilities. One
principal was particularly impressed with the teacher's energy and ability to
get her students to do research on modern Hispanic persons for research papers
and reports.
In the Senior
Exit Interview Report (Appendix 3), the Department Chair offered the
following comments: Students said that they are able to use cultural knowledge
especially in the ways they relate to people from the target country and other
international folks. Their self-awareness is also heightened, and they feel
confident that they have the basis to act internationally with high levels of
understanding. French students asked
for more Francophone studies; German students felt especially confident about
their ability to function in German business; Spanish students asked for more
variety in the survey courses, especially more about the Southern Cone
countries and Central America, with a little less focus in these courses on
Mexico and Cuba. Students found the influence of graduate students very
beneficial in the courses where Area Studies Master’s students were involved as
well as in the Intensive Second Language Institute in the summer.
Of the 26 graduates who returned the Alumni survey, 24 said that study
abroad should be required. Of the 2 who said it should not, one had studied
abroad and one did not indicate whether she/he had studied abroad. Students
talked about the importance of this for proficiency and cultural knowledge.
Service through the core
For students who had majors in the Department:
According to the Alumni survey, 52%
thought their experience of the core in our Department was EXCELLENT, 29% said
it was SATISFACTORY, 0% said NOT SATISFACTORY, and 19% said it was NOT
APPLICABLE to them. Apparently, this question was taken by most students
completing the survey as asking them about THEIR experience of core courses in
the Department.
For students who had majors outside the Department,
information was more difficult to gather: According to information acquired
from SPOT forms (Appendix 1)
students who evaluate their courses at the end of the semester had favorable
views of their gains in core skills. All twelve courses analyzed had a score of
85% or better where those who responded ranked their gains in this area as
average or more than average. Performance was about the same from 1998 to 2001.
Service to non-major students outside core
courses
According to the Oral Proficiency Interviews (OPI), the 10 students examined had an
average score of 7.5, or between 'advanced low' and 'advanced mid' when they
graduated. The two students who were not majors had an average score of 5.0 or
'intermediate mid' level. The two non-majors did not advance nearly as far as
those who were majors. One obvious observation is that non-majors have fewer
courses than majors. From the statistics gathered, we are able to draw only
limited information, since so few non-majors agreed to be tested. But those who
were tested show a level of proficiency lower than the majors, and given the
more limited course work, this is logical.
Quality of teaching
According to the Alumni survey, 83% thought the Department’s quality of teaching was
EXCELLENT, 17% said it was SATISFACTORY, 0% said NOT SATISFACTORY, and 0% said
it was NOT APPLICABLE to them.
Six courses – 2 each of French, German, and
Spanish; with statistics from 1998 and 2001, for a total of 12 specimens – were
analyzed for documentation from the SPOT
forms. For question 7, “overall rating of the professor” all 12 sections
rated professors highly. In 10 of the 12 sections all respondents said that the
teaching was either good or excellent. For the other 2 sections, those rating
the course good or excellent were 92% and 81%.
In the Senior Exit Interview Report (Appendix
3), the Department Chair offered the following summary: Students praised many
faculty members and criticized the performance of others. Sometimes praise and criticism were
contained in their comments about a single professor. This section of the report will discuss what faculty behaviors
students found helpful and what they found not to be helpful. Students appreciated faculty members who
were knowledgeable, organized, and consistent.
They praised faculty members who taught courses according to the course
or catalog description sheet, whose syllabus clearly outlined expectations, and
who stayed on schedule. They
appreciated multiple drafts for essays, opportunities to redress errors, and
assignments that were evenly spread across the semester. Students found that
faculty generally stepped out to help students in advising as well as course
work, though some did not know that they had an advisor. The caring and
compassionate attitude of the faculty members was much appreciated. Students
found that it was not helpful if
faculty members asked them to perform busy work, selected materials that were
below the students’ level, challenged them beyond their level without support,
were overly critical without explanation of what the student had done wrong,
and were unavailable for consultation.
Students pointed out a disconnect between the emphasis on communication
of an idea as stated in a syllabus and papers that were handed back to them
with grammatical errors as the only focus. They especially found it annoying to
get a paper back with an 86 grade and no comments on how to improve. In terms
of structuring of the curriculum, students felt that 30 authors in a survey
course was too many and that selection of highlighted authors may be better.
They also suggested that revisiting works in the light of future works would be
helpful as well.
Part
III. Conclusions
The Department is fulfilling its obligations to its
students. The student perceptions of success are high, the external indicators
are strong, and employers are pleased with our 'product.' Specifically,
technology literacy seems to be strong, critical thinking and problem solving
skills are strong, though students are less convinced because they focus on
theory and literary analysis as a major component of this rubric, while alumni
and employers tend to have a more pragmatic emphasis on getting jobs done and
problems solved. Oral proficiency is well within the expected range, and we
have continued our emphasis on ACTFL guidelines, study abroad, and practical
training.
Faculty, students, and alumni all express some
interest in further development of more pragmatic and professional training at
the upper level. These are issues that are already in process or under
consideration for future development (cf. vision statement, Title VI grant for
professional training in French, plans for new MA programs in French and
Spanish) and so we can be optimistic about these desires. There are also
observations about a need for better preparation for those going on to graduate
school. This has been a topic of discussion among faculty and will also be
helped by the presence of our own graduate degrees, which will make the
Department more aware of the needs of undergraduates as they prepare for
further work.
The
Department has done a good job of considering its plans from the 1997 Outcomes
Report and acting on them. There is more emphasis on functional and
professional tracts as well as on contemporary culture. The Department has
tried to get students involved in the research component of the unit (e.g.
undergraduate research, honors theses, federally funded study abroad programs,
honors scholarship finalists, and use of technology to increase student contact
with faculty research). The students also are actively involved in the general
life of the Department and in their own FLL student community with the informal
gatherings in the student lounge and with social gatherings between students
and faculty (e.g. group attendance at international films or lectures on other
campuses, Classics Table, faculty/student dinners).
Appendix 1:
Statistical samples taken from SPOT
forms for French, German, and Spanish
Prepared by the Department Executive
Committee, February 2002
Course # FR3205
|
1998 |
2001 |
|
Less than
average |
7 |
5 |
|
Average |
71 |
55 |
|
More than
average |
7 |
10 |
Course
# FR3305
|
1998 |
2001 |
|
Less than
average |
0 |
10 |
|
Average |
58 |
57 |
|
More than
average |
17 |
14 |
Responses to SPOT question 7: Overall rating of this instructor
Course #FR3205
|
1998 |
2001 |
|
Poor |
0 |
0 |
|
Fair |
0 |
0 |
|
Good |
7 |
5 |
|
Excellent |
93 |
95 |
Course
#FR3305
|
1998 |
2001 |
|
Poor |
0 |
5 |
|
Fair |
8 |
14 |
|
Good |
50 |
52 |
|
Excellent |
42 |
29 |
Course # GR3306
|
F 1998 |
F 2001 |
|
Less than
average |
11 |
0 |
|
Average |
44 |
56 |
|
More than
average |
22 |
44 |
Course
# GR3125
|
F 1998 |
F 2001 |
|
Less than
average |
0 |
0 |
|
Average |
67 |
60 |
|
More than
average |
17 |
20 |
Responses to SPOT question
7: Overall rating of this instructor
Course # GR3306
|
F 1998 |
F 2001 |
|
Poor |
0 |
0 |
|
Fair |
0 |
0 |
|
Good |
17 |
10 |
|
Excellent |
67 |
90 |
Course
# GR3125
|
F 1998 |
F 2001 |
|
Poor |
0 |
0 |
|
Fair |
0 |
0 |
|
Good |
17 |
10 |
|
Excellent |
83 |
90 |
Course # SPAN 3324
|
1998 |
2001 |
|
Less than
average |
|
|
|
Average |
41 |
92 |
|
More than
average |
22 |
0 |
Course
# SPAN 3204
|
1998 |
2001 |
|
Less than
average |
5 |
13 |
|
Average |
73 |
44 |
|
More than
average |
9 |
31 |
Responses to SPOT question
7: Overall rating of this instructor
Course # SPAN 3324
|
1998 |
2001 |
|
Poor |
|
|
|
Fair |
|
|
|
Good |
4 |
23 |
|
Excellent |
96 |
77 |
Course
# SPAN 3204/4984 (pilot # for new 3204)
|
1998 |
2001 |
|
Poor |
|
|
|
Fair |
|