Department of Foreign Languages & Literatures

Outcomes Assessment Report

April 1, 2002

 

Prepared by Terry L. Papillon,

Assistant Chair of the Department and Associate Professor of Classics

Department Chair:  Judith L. Shrum

 

Part I. Introduction

 

The Department of Foreign Languages & Literatures works under the following mission statement, and the assessment of its activities is carried out with a view to its priorities:

 

The Department of Foreign Languages and Literatures provides students the opportunity to gain a knowledge of the culture, literature, structure, and vocabulary of the diverse languages offered. The department seeks to graduate majors and minors with an advanced command of the written and spoken language and with the general analytical, critical and communicative skills and global outlook essential to an educated member of society. The department fulfills a service mission through offering foreign languages for students who must meet the foreign language requirement, for students interested in language study for intellectual and cultural enhancement, and for students and faculty in other disciplines for whom a knowledge of foreign languages and cultures is a valuable professional tool. In all cases, the department strives to broaden students' awareness of the nature of language and of cultural and national diversities and similarities. It also strives to break down the barriers of ethnocentrism and xenophobia, thus increasing communication and understanding on an international scale. The department strongly promotes the scholarly and pedagogical activities of its faculty, who advance, preserve and transmit the collective knowledge of the discipline through research, publication, teaching, and participation in professional life.

 

The categories for evaluation rose from the work of an ad hoc committee during the spring semester 2001 (see Appendix 2 for the report of that committee) and the actual assessment occurred during the academic year 2001-2002. The categories of assessment include:

 

Preparation for graduate school

Oral and Written proficiency in the target language

Preparation for global/interdisciplinary careers

Technological literacy

Critical thinking skills

Ability to use cultural knowledge

Service to non-major students through the core

Service to non-major students outside core courses

Quality of teaching

 

Instruments used to collect data include:

 

Senior oral exit interviews, conducted during May 2001 by the Department Chair

Alumni surveys, conduced in fall 2001. 26 surveys from graduates from 1998-2001.

Oral proficiency interviews of ten graduating seniors done by nationally certified ACTFL examiners in phone interviews, conducted during spring 2002.

Phone interviews with four employers conducted by the Assistant Chair of the Department about recent graduates who had given permission for such interviews.

SPOT form data[1] taken from a random sample of 12 courses from 1998 and 2001.

Paper survey of degree candidates during fall 2001,  prepared by the Academic Assessment Program and conducted by Department faculty.

 

See Appendix 2 for how each instrument related to each category. In some cases, more instruments were used than were called for in the preliminary report (Appendix 2), simply because they were available. The Senior Exit Interviews and employer phone interviews, for example, offered information on most of the topics, and so they are included in each section. In other instances, not all instruments suggested were available and others had to be substituted with other appropriate measures.

 

In addition to the above measures, it should also be pointed out that the Department’s requirement of a minimum grade point in the major courses ensures a basic level of competency. The Department also requires that every student in the major demonstrate an oral proficiency level as set by the American Council on the Teaching of Foreign Languages (ACTFL). This verification traditionally happens at the point of transition from lower division to upper division coursework. Meeting the level of “advanced low” ensures that the students have the basic language skills in speaking and listening to continue with the coursework for the major. This verification happens in two ways: a student can pass the Oral Proficiency course (French/German/Spanish 3126) which includes an interview with a faculty member trained in giving the Oral Proficiency Interview or be exempted from this course based on an oral interview with a faculty member.

 

Part II. The Assessment

 

Preparation for graduate school

 

According to the Alumni survey, 29% thought their preparation for graduate school was EXCELLENT, 33% said it was SATISFACTORY, 4% said NOT SATISFACTORY, and 35% said it was NOT APPLICABLE to them. Thus, 94% of those who answered the question thought that their preparation was satisfactory or excellent.

Question 8 in the Academic Assessment Survey, which surveys the topic “Prepare for graduate school,” had the following responses: 8% said "did not achieve," 42% said "achieved to a small degree," 17% said "achieved to a moderate degree," 33% said "achieved to a large degree," and 0% said "definitely achieved."  Of the 12 respondents on this survey, only one said they were planning to attend graduate school in the fall, though 6 said they would attend within the next five years.

In the Senior Exit Interview Report (Appendix 3), the Department Chair noted the following: “They felt adequately prepared in writing but not in speaking and wished for more emphasis on speaking at the 4000-level.”

 

Oral and Written Proficiency in the Target Language

 

According to the Alumni survey, 40% thought their proficiency was EXCELLENT, 56% said it was SATISFACTORY, 4% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.

The Assistant Chair of the Department selected at random 10 seniors graduating from December 2001 to August 2002; of these, 8 were majors in FL and 2 were not. These 10 students were given an Oral Proficiency Interview (OPI) via telephone by an external examiner who is certified by ACTFL. On the ACTFL scale from novice low to superior, students need to be at the 'advanced low' level (7 on a 10 point scale as reported here) to pass their oral proficiency level at the beginning of upper division work (see comments on oral proficiency requirement above). At the time of graduation, the 10 students surveyed had an average score of 7.5, or half way between 'advanced low' and 'advanced mid' when they graduated. The 8 majors, however, averaged 8.1, or slightly above 'advanced mid.' In the 1997 Outcomes Report we only reported outcomes for professor-examined students in the teaching option.  This year we examined majors and non-majors using ACTFL certified external examiners.  While direct comparisons are not possible, we have a more reliable measure of performance showing that the outcome performance is within our expected range.

In phone interviews with employers of recent graduates, on the question of oral proficiency, not all graduates are involved in occupations that use their languages, but for those who were the responses were that employees were 'above average' or 'far above average' in their abilities.

In the Senior Exit Interview Report (Appendix 3), students asked for more upper-division classes in which speaking could be developed even further, either a 4000-level oral proficiency class, or reorganization of upper-division courses to allow for less teacher-centered lecture formats and more interactive student participatory activities. Some sections and classes have already begun to move in this direction.

Of the 26 graduates who returned alumni surveys, 24 said that study abroad should be required. Of the 2 who said it should not, one had studied abroad and one did not indicate whether she/he had studied abroad. Students talked about the importance of this for proficiency and cultural knowledge.

 

Preparation for global/interdisciplinary careers

 

According to the Alumni survey, 27% thought their preparation for global or interdisciplinary careers was EXCELLENT, 62% said it was SATISFACTORY, 12% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.

In phone interviews with employers, on the question of global or interdisciplinary preparation, the responses were that employees were usually 'above average' in their abilities. One new employee is being nominated for the 'best first year teacher' award because of his work with other faculty and resource people in the school district.

In the Senior Exit Interview Report (Appendix 3), the Department Chair offered the following comments: Students felt well prepared in all areas except speaking, and here they noted a stark difference between what they study and what their careers will expect of them. Students asked for more courses in technical writing related to their field, a class on current issues in the target country in general as well as courses with targeted information related to agricultural, business, and engineering fields. Students were well aware of the difference between learning language in instructed settings and using it in real settings.  They felt unprepared for the real settings and suggested more such opportunities, accompanied by contextualized courses to reinforce their understanding of language structure, either linguistic or grammatical.

 

Technological literacy

 

According to the Alumni survey, 27% thought their technological literacy was EXCELLENT, 62% said it was SATISFACTORY, 12% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.

In the Senior Exit Interview Report (Appendix 3), the Department Chair offered the following comments: Students felt that their some of their classes in this department were more high-tech than some of their engineering classes.  They enjoyed faculty use of web chats, international videoconferencing, grades on line, on-line courses, research links on line, use of videodisc programs, writing forums, and electronic portfolios. They appreciated the distance learning courses but found it difficult to engage in peer editing using that format. They liked having insights into the research conducted by faculty, citing the case of one professor who put a catalogue of language change on line as a result of his research. They said that the student computer lab available in the department removed a barrier for them, making it easier to complete technologically related assignments right here where the faculty members are also available for help if needed.

 

Critical thinking skills

 

According to the Alumni survey, 46% thought their critical thinking and problem solving skills were EXCELLENT, 50% said it was SATISFACTORY, 0% said NOT SATISFACTORY, and 4% said it was NOT APPLICABLE to them.

In phone interviews with employers, on the question of critical thinking skills, the responses were that employees were consistently 'above average' or 'far above average' in their abilities. One employer commented about how the employee 'always digs, going to other people in the county as resources for situations that need to be treated.' Another employee was meant to be an intern for two years, but her advisor left for another job, and so she has had to cope with learning her job herself and was commended for being "a fighter" and finding out what she needed. A third, new teacher, finds herself preparing for a move from one building to a split school in two buildings; the principal praised her work dealing with such a shocking change in just her first year.

In the Senior Exit Interview Report (Appendix 3), the Department Chair offered the following summary: Students felt that 4000-level literature classes were helpful in showing them a step-by-step process to look deeply into a literary work and analyze it. In addition, writing intensive classes were helpful in showing them how to construct and edit quality arguments in a comprehensive essay. German students pointed out that they were still operating at the level of comprehension, unable to move toward analysis, and that they relied on the professor more than they wished, resulting in more lecture format than their preferred discussion format. French students asked that the selection of literary topics be modernized a bit, focusing on political systems, culture, popular press, the target country today, current issues. Spanish students liked the recombination of Civilization/Culture/Literature courses at the 3000-level. Spanish students also asked for more discussion format in some classes that are largely teacher-centered.

In the Academic Assessment Survey, question 2, which surveys the topic “Improve Intellectual Skills (critical thinking, logical reasoning, problem-solving ability)” had the following responses: 0% said "did not achieve," 0% said "achieved to a small degree," 25% said "achieved to a moderate degree," 33% said "achieved to a large degree," and 42% said "definitely achieved."

According to information acquired from SPOT forms (Appendix 1) student who evaluate their courses at the end of the semester also had favorable views of their gains in critical thinking skills. All twelve courses analyzed had a score of 85% or better where those who responded ranked their gains in this area as average or more than average. Performance was about the same from 1998 to 2001.

 

Ability to use cultural knowledge

 

According to the Alumni survey, 61.5% thought their ability to use cultural knowledge was EXCELLENT, 38.5% said it was SATISFACTORY, 0% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.

In phone interviews with employers, on the question of cultural knowledge, the responses were that employees were consistently 'above average' or 'far above average' in their abilities. One principal was particularly impressed with the teacher's energy and ability to get her students to do research on modern Hispanic persons for research papers and reports.

In the Senior Exit Interview Report (Appendix 3), the Department Chair offered the following comments: Students said that they are able to use cultural knowledge especially in the ways they relate to people from the target country and other international folks. Their self-awareness is also heightened, and they feel confident that they have the basis to act internationally with high levels of understanding.  French students asked for more Francophone studies; German students felt especially confident about their ability to function in German business; Spanish students asked for more variety in the survey courses, especially more about the Southern Cone countries and Central America, with a little less focus in these courses on Mexico and Cuba. Students found the influence of graduate students very beneficial in the courses where Area Studies Master’s students were involved as well as in the Intensive Second Language Institute in the summer.

Of the 26 graduates who returned the Alumni survey, 24 said that study abroad should be required. Of the 2 who said it should not, one had studied abroad and one did not indicate whether she/he had studied abroad. Students talked about the importance of this for proficiency and cultural knowledge.

 

Service through the core

 

For students who had majors in the Department: According to the Alumni survey, 52% thought their experience of the core in our Department was EXCELLENT, 29% said it was SATISFACTORY, 0% said NOT SATISFACTORY, and 19% said it was NOT APPLICABLE to them. Apparently, this question was taken by most students completing the survey as asking them about THEIR experience of core courses in the Department.

For students who had majors outside the Department, information was more difficult to gather: According to information acquired from SPOT forms (Appendix 1) students who evaluate their courses at the end of the semester had favorable views of their gains in core skills. All twelve courses analyzed had a score of 85% or better where those who responded ranked their gains in this area as average or more than average. Performance was about the same from 1998 to 2001.

 

Service to non-major students outside core courses

 

According to the Oral Proficiency Interviews (OPI), the 10 students examined had an average score of 7.5, or between 'advanced low' and 'advanced mid' when they graduated. The two students who were not majors had an average score of 5.0 or 'intermediate mid' level. The two non-majors did not advance nearly as far as those who were majors. One obvious observation is that non-majors have fewer courses than majors. From the statistics gathered, we are able to draw only limited information, since so few non-majors agreed to be tested. But those who were tested show a level of proficiency lower than the majors, and given the more limited course work, this is logical.

 

Quality of teaching

 

According to the Alumni survey, 83% thought the Department’s quality of teaching was EXCELLENT, 17% said it was SATISFACTORY, 0% said NOT SATISFACTORY, and 0% said it was NOT APPLICABLE to them.

Six courses – 2 each of French, German, and Spanish; with statistics from 1998 and 2001, for a total of 12 specimens – were analyzed for documentation from the SPOT forms. For question 7, “overall rating of the professor” all 12 sections rated professors highly. In 10 of the 12 sections all respondents said that the teaching was either good or excellent. For the other 2 sections, those rating the course good or excellent were 92% and 81%.

 In the Senior Exit Interview Report (Appendix 3), the Department Chair offered the following summary: Students praised many faculty members and criticized the performance of others.  Sometimes praise and criticism were contained in their comments about a single professor.  This section of the report will discuss what faculty behaviors students found helpful and what they found not to be helpful.  Students appreciated faculty members who were knowledgeable, organized, and consistent.  They praised faculty members who taught courses according to the course or catalog description sheet, whose syllabus clearly outlined expectations, and who stayed on schedule.  They appreciated multiple drafts for essays, opportunities to redress errors, and assignments that were evenly spread across the semester. Students found that faculty generally stepped out to help students in advising as well as course work, though some did not know that they had an advisor. The caring and compassionate attitude of the faculty members was much appreciated. Students found that it was not helpful if faculty members asked them to perform busy work, selected materials that were below the students’ level, challenged them beyond their level without support, were overly critical without explanation of what the student had done wrong, and were unavailable for consultation.  Students pointed out a disconnect between the emphasis on communication of an idea as stated in a syllabus and papers that were handed back to them with grammatical errors as the only focus. They especially found it annoying to get a paper back with an 86 grade and no comments on how to improve. In terms of structuring of the curriculum, students felt that 30 authors in a survey course was too many and that selection of highlighted authors may be better. They also suggested that revisiting works in the light of future works would be helpful as well.

 

Part III. Conclusions

 

The Department is fulfilling its obligations to its students. The student perceptions of success are high, the external indicators are strong, and employers are pleased with our 'product.' Specifically, technology literacy seems to be strong, critical thinking and problem solving skills are strong, though students are less convinced because they focus on theory and literary analysis as a major component of this rubric, while alumni and employers tend to have a more pragmatic emphasis on getting jobs done and problems solved. Oral proficiency is well within the expected range, and we have continued our emphasis on ACTFL guidelines, study abroad, and practical training.

Faculty, students, and alumni all express some interest in further development of more pragmatic and professional training at the upper level. These are issues that are already in process or under consideration for future development (cf. vision statement, Title VI grant for professional training in French, plans for new MA programs in French and Spanish) and so we can be optimistic about these desires. There are also observations about a need for better preparation for those going on to graduate school. This has been a topic of discussion among faculty and will also be helped by the presence of our own graduate degrees, which will make the Department more aware of the needs of undergraduates as they prepare for further work.

The Department has done a good job of considering its plans from the 1997 Outcomes Report and acting on them. There is more emphasis on functional and professional tracts as well as on contemporary culture. The Department has tried to get students involved in the research component of the unit (e.g. undergraduate research, honors theses, federally funded study abroad programs, honors scholarship finalists, and use of technology to increase student contact with faculty research). The students also are actively involved in the general life of the Department and in their own FLL student community with the informal gatherings in the student lounge and with social gatherings between students and faculty (e.g. group attendance at international films or lectures on other campuses, Classics Table, faculty/student dinners).

 


Appendix 1:

 

Statistical samples taken from SPOT forms for French, German, and Spanish

 

Prepared by the Department Executive Committee, February 2002

Language: FRENCH

 

Analysis of Critical Thinking Skills: Responses to SPOT question 14b: I would rate my gains in this course compared with similar courses as follows: Logical thinking and problem solving ability:

 

Course # FR3205

1998

2001

Less than average

7

5

Average

71

55

More than average

7

10

Course # FR3305

1998

2001

Less than average

0

10

Average

58

57

More than average

17

14

 

Responses to SPOT question 7: Overall rating of this instructor

 

Course #FR3205

1998

2001

Poor

0

0

Fair

0

0

Good

7

5

Excellent

93

95

Course #FR3305

1998

2001

Poor

0

5

Fair

8

14

Good

50

52

Excellent

42

29

 

Language: GERMAN

 
Analysis of Critical Thinking Skills: Responses to SPOT question 14b: I would rate my gains in this course compared with similar courses as follows: Logical thinking and problem solving ability:

 

Course # GR3306

F 1998

F 2001

Less than average

                     11

                     0

Average

                     44

                     56

More than average

                     22

                     44

Course # GR3125

F 1998

F 2001

Less than average

                     0

                     0

Average

                     67

                     60

More than average

                     17

                     20

 

 

Responses to SPOT question 7: Overall rating of this instructor

 

Course # GR3306

F 1998

F 2001

Poor

                     0

                     0

Fair

                     0

                     0

Good

                     17

                     10

Excellent

                     67

                     90

Course # GR3125

F 1998

F 2001

Poor

                     0

                     0

Fair

                     0

                     0

Good

                     17

                     10

Excellent

                     83

                     90

 

Language: SPANISH

 

Analysis of Critical Thinking Skills: Responses to SPOT question 14b: I would rate my gains in this course compared with similar courses as follows: Logical thinking and problem solving ability:

 

Course # SPAN 3324

1998

2001

Less than average

 

 

Average

41

92

More than average

22

0

Course # SPAN 3204

1998

2001

Less than average

5

13

Average

73

44

More than average

9

31

 

 

Responses to SPOT question 7: Overall rating of this instructor

 

Course # SPAN 3324

1998

2001

Poor

 

 

Fair

 

 

Good

4

23

Excellent

96

77

Course # SPAN 3204/4984 (pilot # for new 3204)

1998

2001

Poor

 

 

Fair

 

 

Good

41

6

Excellent

59

94

 


Language: FRENCH

 
Area 2 Success

 

Course#FR3305

F2001

Course#HUM2714

F2000

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

10

 

Average

71

 

More than average

19

 

 

b. Logical thinking and problem solving ability.

 

Less than average

10

 

Average

57

 

More than average

14

 

c. Appreciation of the subject matter and discipline field.

 

Less than average

10

 

Average

52

 

More than average

38

 

 

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

0

 

Average

57

 

More than average

36

 

 

b. Logical thinking and problem solving ability.

 

Less than average

14

 

Average

71

 

More than average

7

 

 

c. Appreciation of the subject matter and discipline field.

 

Less than average

0

 

Average

43

 

More than average

57

 

 

 


Language: GERMAN

 

Area 2 Success

 

Course # GR3305

S 2000

Course # GR3306

F 2001

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

0

 

 

Average

67

 

 

More than average

33

 

 

 

b. Logical thinking and problem solving ability.

 

Less than average

11

 

                    

Average

56

                  

More than average

11

 

        

c. Appreciation of the subject matter and discipline field.

 

Less than average

0

 

Average

78

 

More than average

22

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

0

 

                     

Average

44

 

                     

More than average

56

 

                 

 

b. Logical thinking and problem solving ability.

 

Less than average

0

 

 

Average

56

                 

More than average

44

 

  

c. Appreciation of the subject matter and discipline field.

 

Less than average

0

 

Average

56

 

More than average

44

 

 

Language: SPANISH

 

Area 2 Success

 

Course # SPAN 4984 (pilot # of new 3204) F 2001

Course # SPAN 2744

S 2001

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

13

 

Average

31

 

More than average

44

 

 

b. Logical thinking and problem solving ability.

 

Less than average

13

 

Average

44

 

More than average

31

 

c. Appreciation of the subject matter and discipline field.

 

Less than average

6

 

Average

19

 

More than average

75

 

 

 

a. Knowledge of principles, theories, techniques, etc.

 

Less than average

0

 

Average

42

 

More than average

50

 

 

b. Logical thinking and problem solving ability.

 

Less than average

7

 

Average

42

 

More than average

21

 

c. Appreciation of the subject matter and discipline field.

 

Less than average

0

 

Average

21

 

More than average

79

 

 

 


Appendix 2:

 

Memo to John Muffo outlining plans for Assessment

 

 

MEMORANDUM

Revised on July 5, 2001 (changes in blue)

 

To:       John Muffo, Director

            Academic Assessment

 

From:  Judith L. Shrum, Chair

Department of Foreign Languages and Literatures

 

Re:            Outcomes Assessment to be conducted in 2001-2002 for the FLL Dept.

 

Date:    March 27, 2001

 

We determined this plan through several meetings of an adhoc committee consisting of eleven faculty members representing full-time, part-time, tenured and tenure-track faculty from all languages we teach.  We consulted our departmental plan for 1997-2001, our vision statement approved in1997, and strategic planning documents from the College of Arts and Sciences and the University.  We believe that these assessment strategies will provide information needed by your office as well as information we need to make future decisions.

 

We are requesting assistance from your office for some of the assessments, a total request of $2100.

 

 

 

OBJECTIVE

STUDENT GROUP

MEANS OF ASSESSMENT

FUNDING NEEDED

Prepare students for grad school

Majors, Double majors, minors.

Alumni survey fall 2001

 

Dept. will fund paper survey mailing

Develop oral and written proficiency

Graduating senior majors randomly selected from among majors and double majors.

Arrange for ACTFL Oral Proficiency Interviews in February 2002. Compare results with proficiency scores when these students completed 3126 oral proficiency course.  Augment with question from exit interview about their perceptions of the OP classes; triangulate with study abroad data. 

$140 per student x 10 students = $1400

Prepare students for global/inter-disciplinary career contexts

Majors, Double majors, minors.

Alumni survey fall 2001, perhaps edited for specific issues; employer feedback through phone interviews with 5 employers.

Dept. will fund paper survey mailing and phone calls

Citizens who are technologically literate

Majors, Double majors, minors.

Alumni in fall 2001 and exit survey  February 2002 edited to determine student perceptions of how we use technology to help them learn

Dept. will fund paper survey mailing

Critical and analytical thinking

Majors, Double majors, minors.

SPOT form, item 14, fall 2001

Dept. will fund copying costs

Able to use cultural knowledge

Randomly selected group of 25 students from majors, double majors, and minors

Sections from a standardized test, perhaps the Praxis II Subject Area Assessments

Dept. will fund purchase of tests

Non-major students in language  classes other than Core

Students in Chinese, Hebrew, Italian, Japanese classes (many will not be graduating seniors)

Paper survey done in class; triangulate with SPOT responses, fall 2001

Dept. will fund copying costs

Non-major, non-minor students in upper division classes

5 randomly selected non-major, non-minor students

Oral Proficiency Interviews, February 2002

$140 x 5 = $700

Core Area 2 courses

All students in Core Area 2 courses

Paper survey to be administered to randomly selected students in Core classes fall 2001; paper survey will be mailed to randomly selected students who completed Core classes in fall 2000 and spring 2001; triangulate with responses on  SPOT form,  item 14.

Dept. will fund paper survey mailing

Quality of teaching

All students enrolled in classes in spring 2001 and fall 2002

SPOT forms

Dept. will fund copying costs

 


 Appendix 3:

 

Department Chair’s Report of Senior Exit Interviews, May 2001

 

Department of Foreign Languages & Literatures

Written by J. Shrum, Department Chair

 

 

 

Of the 24 students who graduated from the department of Foreign Languages and Literatures in May 2001, 10 participated in oral exit interviews conducted on May 7 and 8, 2001.  One additional student participated via email.  The students were a mix of French, Spanish, and German majors.  Our usual format for the exit interview is to encourage students to tell us in an open-ended discussion what worked and what didn’t work for them as they pursued their majors/minors/double majors in our department.  This year we added an additional dimension to the questions, structuring them according to the goals outlined in our Outcomes Assessment Report, which also form the organizational structure for this report: 

 

III.               Prepare students for graduate school

IV.              Develop oral and written proficiency

V.                 Prepare students for global/interdisciplinary career contexts

VI.              Prepare citizens who are technologically literate

VII.            Engage students in critical and analytical thinking

VIII.         Enable students to use cultural knowledge

IX.              Provide quality teaching

 

Prepare students for graduate school

            Among these students, two were planning on graduate school at VT (Comm. Studies and Education), one was going to law school, one to medical school, two planned to enter the Peace Corps, one was seeking employment with the CIA or FBI, one was working at VT admissions, and one was to become a commissioned military officer. They spoke highly of the organizational skills they learned in their writing intensive courses; writing papers for their upper division classes and reading difficult prose enabled them to think about sophisticated concepts.  They felt adequately prepared in writing but not in speaking and wished for more emphasis on speaking at the 4000-level.

 

Develop oral and written proficiency

The American Council on the Teaching of Foreign Languages (ACTFL) sets guidelines for performance in listening, speaking, reading, and writing and provides oral interviews with certified examiners for a fee of $130. The department’s first-and second-year courses have proficiency objectives, but no examinations are given at that level.   The department teaches two courses, at the junior level, designed to develop oral proficiency, 3125 and 3126 in French, German, and Spanish. Only 3126 is required, and students must demonstrate certain levels of proficiency upon entry and exit from these courses.  The final exit level is expected to be “advanced” as determined by informal interviews by our own faculty members who have had training in administering ACTFL’s Oral Proficiency Interview. Professors Fernández and Witthoeft are certified examiners and Dr. Fernández is now a trainer of examiners. Results of the official oral interviews conducted by ACTFL certified examiners are featured as part of the Outcomes Assessment Report.

In the exit interviews for graduating seniors, students asked for more upper-division classes in which speaking could be developed, either a 4000-level oral proficiency class, or reorganization of upper-division courses to allow for less teacher-centered lecture formats and more interactive student participatory activities. They suggested debates, opportunities for informal conversation, web chats that are then brought into class and continued. Students in French OPI classes liked the idea of using a textbook because it adds structure.  Spanish students said they liked the informal interaction provided by some OPI instructors, but they also thought a textbook would add organizational benefits and enable them to give form to the vocabulary and grammar they are practicing.  

 

Prepare students for global/interdisciplinary career contexts

            Students felt well prepared in all areas except speaking, and here they noted a stark difference between what they study and what their careers will expect of them.  As one French/agriculture student put it, “we know all about the bald opera singer but can’t say ‘milking parlor.’” Students asked for more courses in technical writing related to their field, a class on current issues in the target country in general as well as courses with targeted information related to agricultural, business, and engineering fields. Students supported the study abroad programs and suggested a requirement of some form of immersion, either study abroad or work with local target language communities. Students were well aware of the difference between learning language in instructed settings and using it in real settings.  They felt unprepared for the real settings and suggested more such opportunities, accompanied by contextualized courses to reinforce their understanding of language structure, either linguistic or grammatical.

 

Prepare citizens who are technologically literate

            Students felt that their some of their classes in this department were more high-tech than some of their engineering classes.  They enjoyed faculty use of web chats, international videoconferencing, grades on line, on-line courses, research links on line, use of videodisc programs, writing forums, and electronic portfolios. They appreciated the distance learning courses but found it difficult to engage in peer editing using that format. They liked having insights into the research conducted by faculty, citing the case of one professor who put a catalogue of language change on line as a result of his research.  They said that the student computer lab available in the department removed a barrier for them, making it easier to complete technologically related assignments right here where the faculty members are also available for help if needed.

 

Engage students in critical and analytical thinking

            Students felt that 4000-level literature classes were helpful in showing them a step-by-step process to look deeply into a literary work and analyze it. In addition, writing intensive classes were helpful in showing them how to construct and edit the quality arguments in a comprehensive essay. German students pointed out that they were still operating at the level of comprehension, unable to move toward analysis, and that they relied on the professor more than they wished, resulting in more lecture format than their preferred discussion format. French students asked that the selection of literary topics be modernized a bit, focusing on political systems, culture, popular press, the target country today, current issues.  Spanish students liked the recombination of Civilization/Culture/Literature courses at the 3000-level. Spanish students also asked for more discussion format in some classes that are largely teacher-centered.

 

Enable students to use cultural knowledge

            Students said that they are able to use cultural knowledge especially in the ways they relate to people from the target country and other international folks. Their self-awareness is also heightened, and they feel confident that they have the basis to act internationally with high levels of understanding.  French students asked for more Francophone studies; German students felt especially confident about their ability to function in German business; Spanish students asked for more variety in the survey courses, especially more about the Southern Cone countries and Central America, with a little less focus in these courses on Mexico and Cuba. Students found the influence of graduate students very beneficial in the courses where Area Studies Master’s students were involved as well as in the Intensive Second Language Institute in the summer. The diversity of thought at perspective brought by these students enable undergraduates to widen their cultural horizons.

 

Provide quality teaching

            Students praised many faculty members and criticized the performance of others.  Sometimes praise and criticism were contained in their comments about a single professor.  This section of the report will discuss what faculty behaviors students found helpful and what they found not to be helpful.  Students appreciated faculty members who were knowledgeable, organized, and consistent.  They praised faculty members who taught courses according to the course or catalog description sheet, whose syllabus clearly outlined expectations, and who stayed on schedule.  They appreciated multiple drafts for essays, opportunities to redress errors, and assignments that were evenly spread across the semester. Students found that faculty generally stepped out to help students in advising as well as course work, though some did not know that they had an advisor. The caring and compassionate attitude of the faculty members was much appreciated. Students found that it was not helpful if faculty members asked them to perform busy work, selected materials that were below the students’ level, challenged them beyond their level without support, were overly critical without explanation of what the student had done wrong, and were unavailable for consultation.  Students pointed out a disconnect between the emphasis on communication of an idea as stated in a syllabus and papers that were handed back to them with grammatical errors as the only focus. They especially found it annoying to get a paper back with an 86 grade and no comments on how to improve. In terms of structuring of the curriculum, students felt that 30 authors in a survey course was too many and that selection of highlighted authors may be better. They also suggested that revisiting works in the light of future works would be helpful as well.

           


Appendix 4:

 

Outcomes Assessment Report, 1997

Department of Foreign Languages & Literatures

Written by J. C. Eustis

 

            The Department of Foreign Languages and Literatures’ assessment of its programs is based on the following statement of goals and objectives for its graduating students:

The Department of Foreign Languages and Literatures seeks to graduate majors and minors with an advanced command of the spoken and written language and a thorough knowledge of the culture of a particular country or area of the world.  In addition, the department seeks to provide its students with the analytical, critical, and communicative skills essential to an educated member of society.  Likewise, the department has the objective of developing in its students a global outlook, broadening their awareness of cultural and national diversities and commonalities.  The department seeks as well to give its students a strong foundation for careers in foreign language education, in domestic fields requiring foreign language and culture skills, and in the international sphere, and also for graduate and professional education.

            To assess its success in achieving these goals and objectives, the department uses the following measures:

            ž QCA in the major

            ž Oral proficiency interviews

            ž course grade averages

            ž in-major “quality-control” graduation requirements

            ž Student Perception of Teaching (SPOT) scores and written comments

            ž senior exit interviews

            ž alumni survey

Subsumed within the discussion in the report are the data from unsolicited letters of praise and from other student and parent statements

QCA in the major

            The department policy whereby all majors must earn a grade of “C” or better in all required courses in the major insures that all of the department’s graduates have achieved what the department deems to be at least a minimally acceptable level of skill or knowledge in the linguistic, literary, and cultural areas of study comprising the major in a foreign language.  The required curriculum for foreign language majors includes coursework that gives all majors ample exposure to the material that must be learned to meet the objectives of the foreign language major.

Oral Proficiency Interview

            The department’s Oral Proficiency requirement inaugurated in 1988, provides a means of accurately evaluating our students’ achievement of competence in oral communication.  To fulfill the requirement, students must take and/or test out of the second semester of the Oral Proficiency course sequence 3125-3126.

            Admission to each semester is by oral examination, and to pass this course a student must achieve a proficiency rating of “Intermediate-High” in 3125, and of “Advanced” in 3126.  The course itself and the rating scale used in evaluating students are based on national standards established by the American Council on the Teaching of Foreign Languages Oral Proficiency Interview (ACTFL-OPI) Guidelines.  It is the department’s goal that students who seek to obtain elementary- and secondary-school foreign language teaching certification through

                                                                                                            Outcomes 97, p. 2

the Education Option program achieve a rating of “Advanced” on the ACTFL Oral Proficiency Interview scale.  In fact, about 20% of the students accomplish this goal, while the remainder achieve the level of “Intermediate-Mid” or “Intermediate-High.”  These students also are required to take the Praxis or the National Teachers Examination, which cover not only language skills, but also literature and culture.  All students passed the relevant sections of the Praxis or the NTE on their first or second effort. 

 

Course grade averages

 

These data are forthcoming.

 

 

 

In-major “quality-control” graduation requirements

            Since its first Outcomes Assessment report in 1992, the department has taken a number of steps to strengthen its programs based on previous assessment results.  In particular, efforts to introduce greater variety into the upper division curricula are reflected in the revision of the 4000-level course offerings in French to include more cultural and historical topics, in the expansion of the linguistics component in the Spanish major, and in the revision of the Spanish 4000-level requirement to allow for more non-literary coursework.  The French Section also has pioneered the incorporation of interactive computer technology into its culture and civilization courses and has created special non-contact technology-based 1C courses to supplement the junior-level language skills sequence.  These changes afford students more exposure and practice with the language, especially in native-speaking contexts.  The Spanish Section has introduced a new third-year bridge course that gives students a stronger grammar foundation, has offered advanced courses in film and theatre, and also has begun to make use of instructional technology in its courses at all levels.  Department-sponsored study abroad opportunities have been expanded through the initiation of a summer program in France and the establishment of a semester or academic-year student exchange program with a university in Ecuador.  Further initiatives that were spurred by the last outcomes assessment exercise, such as the development of professional language tracks and the development of additional long-term study abroad programs, currently are underway.  The department also is working on a Writing Across the Major plan that it will submit for approval during the next academic year.  It is expected that the broad writing strategy together with the addition of advanced listening comprehension and speaking skills classes will give our programs even greater coherence and reinforcement of the progressive development of students’ foreign language proficiency. 

Student Perception of Teaching (SPOT) scores and written comments

            Further evidence of the department’s success in achieving its goals and objectives is provided by the average departmental score of 3.67 for the past ten semesters on Student Perception of Teaching course evaluation forms.  Written student comments are correspondingly laudatory.  The department’s effectiveness in teaching also is reflected in the five Certificates of Teaching Excellence, two Alumni Awards, two Diggs Teaching Scholar Awards, and one Wine Award won by Foreign Language faculty members since 1992.  Since1994, an average of three FLL majors per year have been initiated into Phi Beta Kappa, an average of seven students per year have graduated with honors, one student won a Fulbright Grant to France, and four                                                                                                                  Outcomes 97, p. 3

students have won highly competitive summer study abroad scholarships offered by the Spanish National Honor Society, Sigma Delta Pi.  Since 1992, an average of ten FLL graduates per year have been accepted into graduate school.

Senior exit interviews

            Student feedback for the present report from exit interviews with graduating FL majors and double majors is overwhelmingly positive with regard to the faculty and staff in the department.  Students repeatedly praise the dedication, concern, knowledgeability, enthusiasm, and teaching ability of their instructors and point to the faculty as the departments’ most salient strength.  So too the small class sizes, the close, personalized interaction between professors and students, and the caring responsiveness of the staff are singled out for praise and appreciation.  At the same time, a number of desiderata that consistently are mentioned include:  1) the need for less emphasis on literature and more variety of advanced courses, especially courses that focus on contemporary life and current events, oral skills, grammar, and practical applications of foreign languages for professional purposes. Students in German and in Spanish would like to see the civilization and culture courses in those languages extended beyond a single semester.  Students in Spanish emphasized the desirability of the study abroad program in Spain.  In all three majors, students generally agreed that greater flexibility in the advanced course requirements would be advantageous, particularly so as to allow a degree of concentration on individual student career directions.  Interest also was expressed in finding ways to ensure that FLL graduates have a stronger general knowledge of the world areas represented by the language(s) studied.  Similarly, the students surveyed stressed a need for more attention to career counseling.  A number of majors wished for more participation by faculty members in language-related extracurricular activities,  and it was suggested that the department should seek to inform and involve its majors more in the research activities of the faculty. 

            For the most part, the graduating seniors surveyed were highly complimentary of the department and positive about their experience as foreign language majors.  Many of the improvements they would like to see already are on the department’s agenda, while others will be targeted as priority items beginning in the immediate future.

Alumni survey

            An alumni survey sent in spring 1997 to FLL graduates from 1992-1996 (approximately 100 persons), yielded 46 responses: 16 in French, 9 in German, and 21 in Spanish.  Slightly under one-half of the respondents have enrolled or currently are enrolled in a graduate program.  Of the 26 who indicated that they are or were employed in a language-related vocation (including teaching as a graduate assistant) since graduation, nearly all felt that the major had prepared them either “very well” (12) or “well” (13) for such employment.  The one remaining responded that their preparation had been “average.”  Similarly, 43 respondents felt that the major had “broadened [their] global outlook and/or increased [their] awareness of cultural and national diversity,” particularly as a result of study abroad in many cases.  Nearly all of the respondents rated the advising they received while at Tech (their relationship with their advisor) as either “excellent” (33) or “good” (11).  One graduate described the relationship as non-existent, and one considered the relationship to have been unsatisfactory.  The survey also indicated that while only one-half of the respondents had undertaken study abroad, 87% (40) felt that the department should require a study abroad experience of all majors.  However, the support of such a requirement by several respondents was with the caveat that scholarship aid would be indispensable for some students and that there should be the possibility of exceptions and/or alternatives to the rule in special cases.

            With regard to a question (#10) asking about the courses that best prepared the graduates for their post-graduate job(s), 30% (14) of the respondents gave no response or indicated that this question was not applicable.                                                                                                                    Outcomes 97, p. 4

Otherwise there was no consensus among the German graduates, while in French and in Spanish the vast majority indicated that the most useful courses were those that stressed grammar and composition together with the Oral Proficiency courses, i.e., courses that focus on communicative language skills per se.

            The alumni responses to questions relating to curricular deficiencies and areas of our programs that could use improvement (#s11,13,18) mirror the perceptions of the graduating majors: more functional courses such as in foreign languages for professional purposes, more emphasis on practical communicative skills, more grammar, more variety of advanced courses generally, more concentration on present-day life and culture, and more attention to career counseling.  As was noted above, the alumni respondents also emphatically endorsed the idea of a study abroad requirement for foreign language majors.

            To date, the department has not sought to measure job preparedness through employer satisfaction phone surveys.  Only about one-half of the approximately 50% of foreign language graduates from the past five years who responded to a recent survey are or have been employed in jobs relating to their foreign language skills and specialized academic preparation.  In addition, many of the graduates who responded to the alumni survey either were not willing to have the department contact their employer or indicated that this category was not applicable.  Of the alumni who did indicate that their employer could be contacted, most are in public school foreign language teaching where the department has reliable feedback through its Teacher Certification Program director, Dr. Judith Shrum, who is in regular contact with supervising teachers and public school foreign language administrators.  Verbal reports consistently indicate that Virginia Tech Foreign Language Teaching Certification graduates generally are considered to be extremely well prepared and that they do an outstanding job in the classroom.

            Like the exiting seniors whom we interviewed, the alumni indicated a very high overall level of satisfaction with the department’s major programs.  They too consider the faculty generally to be outstanding and also cite as distinct strengths of the department its small class sizes, the individual attention given to students by their professors, the balanced curriculum that addresses thoroughly the development of proficiency in all of the principal language skills (reading, writing, listening comprehension, and speaking), in culture, and in literature.  In many cases alumni also expressed appreciation for the opportunities for additional language practice and cultural enhancement provided by the department’s extracurricular activities.

Summary

            All in all,  the picture of the FLL major programs that emerges from this review is a highly positive one that indicates that the department is fulfilling its educational mission very effectively.  At the same time, the assessment process also has been of great usefulness in reinforcing the department’s sense of some of its relative deficiencies and in bringing to its attention several less evident additional needs.  Already the department has developed an internal resource reallocation plan for the future that promises to allow for the implementation of such things as  more functional language courses or even tracks in some or all of the majors.  Likewise, curricular revision and changes in in-major requirements can allow for a greater variety of advanced courses designed to address the need for more conversational practice and more emphasis on contemporary life and issues.  More specialized attention and minor resources for space and materials development can bring about an improvement in career counseling.  Greater integration of our majors into the intellectual life of the department and of the profession itself can be achieved by inviting students to attend departmental research and other discussion series and by encouraging students to do research and teaching “internships” under the supervision of foreign language faculty members.  Finally, the inclusion of students in the life of the department also can be heightened through the creation of a more comfortable and inviting student lounge space within the department that can serve                                                                                                                      Outcomes 97, p. 5

as well as a study abroad and career information center.  Basically, nearly all of the ways to improve the department indicated in the senior exit interviews and in the alumni survey are useful guidelines for directions the department already has begun to pursue or will take in the near future.  We look forward to reaping the fruits of our efforts and seeing the benefits of this exercise reflected in the next Outcomes Assessment report. 

 

June 4, 1997


Appendix 4a (appended to 1997 Outcomes report)

 

Student Comments during Senior Interview

April 29 and 30, 1997

Compiled by J. Shrum

 

            Professors Eustis, Head, and Shrum, Chair-Elect, met with 14 students on April 29 and 30, 1997 to conduct senior exit interviews.  The format of the interviews was open, with students given the opportunity to speak their minds as their thoughts occurred to them.  There was no structured set of interview questions.  Students were simply asked to comment on what they found good and useful about their study in the Department and what they would like to see changed. 

 

            The tabulation of student comments follows in the Appendix, but their comments can be summarized in these ways.  Students were pleased with the academic and personal quality of the faculty and staff with whom they worked.  They liked having professors more than once, and their enjoyed working with Ph.D. faculty.  Given their high regard for the faculty, students asked for extended opportunities to work as undergraduate TAs or to share in research projects with faculty.  Clearly, faculty have been successful role models for students who eagerly seek additional opportunities to understand and share in the professional life of faculty members.

 

            Student comments about courses, curriculum, and the major generally indicate a need for more applied courses, more utilitarian uses of language in varied content areas, and less literature.  Students enjoy literature, however, and suggested ways to combine it with other aspects of language, e.g. grammar through literature, or the study of cultural geography through literature.  Students feel that their knowledge of contemporary issues related to the language they study in minimal. Students also wanted more demanding grammar and oral proficiency courses at the upper levels of study.

 

            This group of students wants to be more challenged, and made a number of suggestions for courses, e.g. phonetics and morphology in French, that fall along traditional lines as well as suggestions for more unique courses, e.g. medical geography in the target language, industrial use of language, etc.  Students were mixed in their view of the use of technology for language learning; some suggested more and others thought it unproductive use of class time.   

 

            In terms of pedagogy, students would like to see an oral component in every class. They would like more participatory and less didactic courses, and they appreciated opportunities for creative writing and use of analytical thinking.  They suggested reading shorter and fewer selections, but in greater depth with more discussion.

 

            The students urged the faculty to re-consider the ordering of pre-requisites and to be more clear in communicating them.  They suggested a careful look at courses to avoid incongruous sequences, repetition of some material to the exclusion of other contemporary material. 

 

            In terms of their own physical space in the Department, students would like to see the lounge more useful, including information on careers, professions and study abroad.  They also urged more incorporation of the student clubs into the life of the Department, perhaps through revitalization of the student lounge or inclusion in the research and professional teaching and research lives of the faculty.



[1] Percentages on SPOT form reporting do not always add up to 100% because of various accounting procedures. All reports given here represent numbers as presented on the SPOT forms.