Paul D. Allison, Chair
Jacob Cytryn
Irma Elo
Robert Perelman
D. Kent Peterman
Ingrid Waldron

Comments should be directed to Paul Allison, allison@soc.upenn.edu

1. Analysis of Student Records
2. Comparison of General Requirement Courses
3. Scientific and Quantitative Literacy, and Interest in Science
4. Self-Reported Educational Experience
5. Faculty Evaluations

1. Analysis of Student Records

One of the rationales for the Pilot Curriculum was that a reduced number of general requirement courses would make it possible for students to create a curriculum tailored to their specific interests with the expanded number of elective credits available to them. In this section, we will examine what students have actually done with their course choices, at least through the end of their third year at Penn. Using archival records for students who entered as freshmen in the fall of 2000, we compare the course taking patterns of 198 students enrolled in the Pilot Curriculum with those of 70 students who applied for the Pilot Curriculum but were not chosen (“applicants”) and 1,275 students who chose not to apply (“non-applicants”). The analyses covers courses taken in the first through third years of residence at Penn.

Table 3 compares students on several dimensions related to course-taking patterns. Because Pilot students generally have more free electives to work with, it is plausible that they would be more likely to double or triple major, pursue dual degrees, submatriculate, or study abroad. The first four rows of Table 3 show no such tendencies. The last column in the table is a p-value for testing the null hypothesis that there are no real differences among the three groups for each outcome. A p-value less than .05 is usually taken as statistically significant evidence that there are real differences.

A major concern about the Pilot Curriculum has been that students in the program would take fewer natural science courses, given that they are only required to take one general requirement course in Category 3 “Earth, Space and Life” and another in Category 2 “Science, Culture and Society.” To evaluate this possibility, we classified courses taken by students as “science” if the course was taught in one of the following departments: astronomy, biochemistry, bioengineering, biological basis of behavior, biology, biomedical studies, cell and molecular biology, chemistry, cognitive science, computer science and engineering, electrical science, engineering and applied science, environmental studies, genomics and computational biology, geology, materials science and engineering, mathematics, mechanical engineering and applied mechanics, physics, psychology, or statistics. Majors were classified as science majors if they were in any of these departments. For Pilot students, we also treated any course labeled COLL003 (the earth, life and space category) as a science course. However, we did not include COLL002 courses (the science, culture and society category) because we suspected that some faculty might question whether these courses are sufficiently rigorous or have enough science content to be considered science courses.

Under this definition of science, we see that the percentage of students who majored in science is about the same for all three groups. With regard to courses, Pilot students took an average of 6.2 science courses compared to 6.8 taken by applicants and non-applicants, a difference that was far from statistically significant. Of course science majors are required to take multiple science courses, so the real issue is how the non-science majors compare across the three groups. Among the non-science majors, Pilot students took an average of 4.1 science courses, compared with 4.4 for applicants and 4.2 for non-applicants, another non-significant difference.
Although the average number of courses is quite comparable across the three groups, there are notable differences at the upper and lower end of the distribution. Twenty-eight percent of the Pilot students who are not science majors took only one science course (presumably COLL03), compared with only 6 or 7 percent of the non-pilot students, a difference that is highly significant. This differenced is counterbalanced at the other end of the distribution by a tendency for some Pilot students to take many science courses.

The disparity is even more marked for courses in mathematics or statistics for which Pilot students have no requirement. As the last row in the table shows, among non-science majors, fully 53 percent of Pilot students took no math/stat courses, compared with around 20 percent of non-pilot students (who either satisfied their requirement by AP credit or postponed it until their last semester). Again, this difference is highly significant.

Table 2. Comparisons between Pilot Students, Applicants and Non-Applicants.

2. Comparison of General Requirement Courses

The most visible component of the Pilot Curriculum is the set of special courses designed for Pilot students to satisfy the General Requirement. Several of these courses have been offered each semester since the fall of 2000. In this section, we compare student evaluations of Pilot General Requirement courses with those for regular General Requirement courses. Specifically, we compare all 15 Pilot General Requirement courses taught in the spring or fall of 2002, with a set of 28 courses chosen from the regular General Requirement taught in the Spring of 2002.

The 28 regular curriculum courses were selected with representation from each of the seven sectors. We tried to choose courses that served as introductions to particular fields and the courses that students are most likely to take in a given sector. But we did not want to include only very large lectures, for that would have introduced a systematic bias in comparison to the Pilot classes, which have fallen in the range of small to medium. Some of the courses chosen (Devil’s Pact, for instance) are interdisciplinary in much the same way that Pilot courses are. In fact, two of the courses (Devil’s Pact and What Every Lawyer, Businessman and Citizen Needs to Know about Molecular Biology) have subsequently been approved as Pilot courses.

We first compared the courses using data from standard student evaluation forms distributed in all courses. As shown in Table 3, the regular curriculum courses tend to be somewhat larger with a median of 69 students, compared with a median enrollment of 47 for the Pilot courses. This difference is not quite statistically significant, however. Courses from the two curricula are nearly identical in mean rating of the courses, at a little over 2.5. For the instructor ratings, some of the team-taught Pilot courses had the students evaluate each instructor separately. In those cases, we simply used the mean of the multiple ratings. Again, there is little if any difference in mean ratings of the Pilot courses and the regular courses.

We also requested that instructors distribute supplemental evaluation forms that were designed to focus on questions that were particularly relevant to the success of the Pilot courses. We got responses from 14 of the 15 Pilot course instructors, but only 12 of the 28 regular course instructors. With the exception of Psychology 1, we did not obtain the cooperation of faculty teaching the very largest classes--Political Science 1, History 2, Art History 102--which were also courses with aims most in contrast with the Pilot classes. So the factors that differentiate the Pilot courses and the regular curriculum courses ended up being less clear than we had wished.

To test whether the 12 respondents were a representative subsample of the 28 courses, we compared respondents and non-respondents on available measures from the standard evaluation form. For all measures, the differences were small and not statistically significant.

For five of the six questions that were identical across the forms completed by regular and Pilot course instructors, the Pilot courses were evaluated less favorably by substantial margins. For example, an average of 60 percent of students in the regular curriculum courses said the course was somewhat better or much better than other courses they were taking in the semester, while the average for Pilot students was only 47 percent. In Pilot courses, an average of 57 percent of the students said the pace was about right, but in regular curriculum courses the average was 69 percent. None of these differences achieved statistical significance, although the p-values for the two questions just mentioned came close.

In addition to comparing Pilot and regular curriculum courses, we also looked at changes over time in student evaluations of the Pilot courses. Figure 1 shows the average evaluations of courses and instructors over the first five semesters of Pilot courses. As with student evaluations generally, ratings for instructors are always higher than ratings for courses. But there is little trend over time, and nothing that is statistically significant.

Table 3. Comparisons of Student Evaluations in Pilot and Regular General Requirement Courses

Figure 1.  Mean Student Evaluations of Pilot Courses and Instructors, By Semester

3. Scientific and Quantitative Literacy, and Interest in Science

To evaluate the contributions of the Pilot curriculum and the regular curriculum to scientific and quantitative literacy, we developed and validated a Science Survey. This survey includes multiple choice items that assess scientific reasoning, and knowledge of important concepts, facts and terminology, primarily in biology, chemistry, physics, environmental sciences and math. A 24-item paper and pencil version of the Science Survey was administered to the entering freshman class during Freshman Orientation in September, 2001. Early in the fall semester, 2003, a 22-item Web-based version of the survey was administered to a sample of seniors (students who entered Penn in fall, 2000 and had at least 20 CU of course credit). The survey for seniors included additional items to assess interest in science (self-rated interest in science and how often the student gets science information from various non-course sources).

We received usable surveys from approximately 83% of the entering freshmen (89% of Pilot students vs. 82% of non-Pilot students). The response rate for the senior sample was 68% and did not differ significantly for students in the Pilot Curriculum (124 respondents), students who applied for the Pilot Curriculum but were not chosen (32), and our sample of non-applicant non-Pilot seniors (132). For the seniors, we were able to compare respondents with non-respondents and found several significant differences. Respondents were more likely to be science majors, had earned more CU's, had more total CU's, and had higher GPAs (with an almost significant difference in the same direction in SAT math scores). Thus, our data almost certainly overestimate the scientific and quantitative literacy of seniors in the College.

For comparability we give the proportion of correct answers for the 22 scientific and quantitative literacy items which were included in the Science Survey for both freshmen and seniors. On average, entering freshmen gave correct answers for 56% of these items, whereas seniors gave correct answers for 61% of these items. For seniors, the proportion of correct items was significantly higher for science majors (67%) than for non-science majors (58%). These percentages suggest the possibility that a College education contributes to improved scientific and quantitative literacy for science majors, but not for non-science majors. We hope to gain further insight concerning this issue in fall, 2004, when we obtain follow-up information from the freshmen we surveyed in fall, 2001.

For the freshmen, Pilot students answered more of the scientific and quantitative literacy items, resulting in more correct answers, but this may have been due to testing in a smaller Dean's session which was just for Pilot students. Results from our sample of seniors suggest that scientific and quantitative literacy may have been similar for incoming Pilot applicants and non-applicants. Specifically, seniors in the Pilot Curriculum, seniors who had applied for the Pilot Curriculum but were not chosen, and non-applicant non-Pilot seniors did not differ significantly in SAT math and verbal scores or number of AP or college-level science and math courses taken before coming to Penn.

Among seniors, the proportion correct for the scientific and quantitative literacy items did not differ significantly for those in the Pilot Curriculum, those who applied for the Pilot Curriculum but were not chosen, and non-applicant non-Pilot seniors. In addition, there was no significant difference in the proportion of students with low scientific and quantitative literacy scores. These three groups also did not differ significantly in terms of the Science Interest scale, average number of science and math courses they reported taking at Penn, or whether or not they had a science or math major (defined as Biology, Biochemistry, Biological Basis of Behavior, Biophysics, Chemistry, Cognitive Science, Environmental Studies, Geology, Mathematics, Physics and Astronomy, Psychology). In conclusion, our results indicate that seniors in the Pilot Curriculum did not differ from other seniors in terms of scientific and quantitative literacy or interest in science.

4. Self-Reported Educational Experience

The seniors who participated in the Science Survey also completed an Undergraduate Education Survey concerning their educational experience at Penn. This survey included ratings of various aspects of the seniors’ educational experience (Table 4) and a request for “comments or suggestions concerning your Penn Education” (comments summarized in Table 5). The overall response rate was 66%, which included 123 Pilot students, 31 Applicant Non-Pilot students (Pilot applicants who were not selected for the Pilot program), and 129 Non-Applicant Non-Pilot students.

The students’ ratings (Table 4) and open-ended comments (Table 5) both indicated considerable satisfaction with their Penn education, but dissatisfaction with advising during their first two years. These ratings did not differ significantly by Pilot status. The specific criticisms of advising included complaints about poorly informed advisors (especially faculty advisors), advisors who did not provide sufficient direction, and turnover of advisors (e.g., due to leaves). Among the many additional comments and suggestions, the largest categories were requests for fewer requirements (especially requests for fewer distribution requirements from non-science majors), complaints about faculty who were not interested in teaching or accessible to students, and dislike of science courses (a category which may have been increased by the preceding Science Survey) including complaints that courses were too big and professors were uninterested, and objections to being in science classes with premeds.

Students gave relatively high ratings for the contribution of their Penn education to a variety of general intellectual abilities, with lower ratings for the contribution of their Penn education to more specific types of learning, especially for science and the arts (Table 4). For these ten items, only one showed significant variation by Pilot status; this difference suggests that Pilot students may have gained more in terms of “ability to learn on your own, pursue ideas, and find information you need”. Self-reports of participation in thirteen categories of learning experiences also showed few differences by Pilot status. More of the Pilot students reported a “research-oriented independent study or other research experience”, which is not surprising since the Pilot curriculum requires a research experience. In addition, more Pilot students reported a “culminating senior experience (comprehensive exam, capstone course, thesis, etc.)”. This probably relates to the research experience requirement, since research experience correlates strongly with participating in a culminating senior experience.

In summary, these self-reports are in agreement with other results suggesting relatively little effect of the Pilot curriculum on the undergraduate experience, although the requirement for a research experience has the expected effect of increasing participation in research.

Table 4. Student Evaluations of their Undergraduate Educationa

How would you evaluate your entire educational experience at Penn?
Mean rating: 3.4 (between 3 = good and 4 = excellent)

Overall, how would you evaluate the quality of academic advising you received in your first two years at Penn? Mean rating: 2.2 (between 2 = fair and 3 = good)

To what extent has your experience at Penn contributed to your knowledge, skills, and personal development in the following areas? (1= Very little, 2= Some, 3= Quite a bit, 4= Very much)

Which of the following have you done or do you plan to do before you graduate from Penn?
% Answering Yes (with undecided counted as half each)

Table 5. Summary of Comments Concerning Penn Educationa

5. Faculty Evaluations

Interviews with first semester faculty

In spring of 2001, following the first semester of the Pilot, committee members conducted telephone or face to face interviews with each faculty member teaching a Pilot course. The committee thought it important to interview individuals rather than teams to ensure that issues of coordination and integration could be discussed candidly. Interviewers worked from the same set of questions, but allowed discussions to flow naturally. The questions were as follows:

1. Overall, what were the successes and problems you encountered in teaching your Pilot curriculum course?
2. How did you feel about the team teaching in this course? What were the advantages and disadvantages?
3. In what way did you collaborate with your colleagues, as opposed to simply dividing the course into independent units?
4. What were the advantages and disadvantages of the interdisciplinary approach for this course?
5. How did the amount of preparation time for this course compare with other first-time courses you have taught?
6. Did the experience of teaching this course differ from what you expected? How?
7. Do you think that the students in your Pilot course were different from other students you've taught at this level? If so, how?
8. Overall, how effective do you think the course was in achieving your goals?
9. Do you plan to teach this course again? If not, why not?
10. What advice would you give to someone who is planning to teach a Pilot course?
Optional:
1. What was it that attracted you to teach a Pilot curriculum course?
2. Did you use Blackboard or a listserv? If yes, was it useful?

Faculty consistently reported enthusiasm for team teaching (all but one of the courses in the first semester was team-taught), and a number of them commented that it was the most satisfying intellectual experience related to undergraduate education that they had ever had. Several observed that team teaching far from reducing the amount of time they spent preparing for class increased it significantly. Most of the teams reported spending time each week coordinating class presentations and discussions in addition to the efforts they had made during the previous summer to prepare a syllabus that integrated their various perspectives.

Forums with faculty

Following these interviews, the committee organized a forum attended by many of the faculty who had taught in the first academic year (fall and spring), faculty who were preparing new courses for the second year of the Pilot, and members of the evaluation committee. The session was taped. Discussion centered on the challenges of team teaching and the value of an interdisciplinary approach in courses intended to serve as vehicles for general education for first and second year students. Faculty who had taught or who were teaching these courses generally expressed high satisfaction with the experience. They stressed the importance of good regular communication with one another and a genuine interest in one another’s material. The team teaching and the interdisciplinarity usually meant that faculty had to back away from the thorough coverage of content to which they were accustomed in their other teaching. They endeavored instead to bring students to understand various approaches that may be taken to a question and how they relate to each other.

Feedback from a number of students indicated that they did not always understand what the courses expected of them, and faculty at the forum suggested that they may need to make a special effort at various steps along the way to explicate just what they are doing with the material and why they are doing it. One committee member, noting that team teaching in the long run could be sustained only if enrollments became much larger than they were in the first year of the Pilot, wondered which model of interdisciplinary general education was preferable: large classes with team teaching or smaller classes taught by individuals. The response from faculty experienced teaching Pilot courses was that they wanted to have it both ways: smaller, team-taught classes with, however, two-person teams rather than three-person teams. In sum, the model employed in the Pilot classes so far was highly regarded by the faculty teaching them, even if they recognized that something would have to give if the general requirement of all students in the College were to be built around this model.

A similar forum held in the spring of the second year yielded a markedly different tone. The conversation focused more on some of the feedback from students that by now had been circulating, to the effect that they found integration of the multiple perspectives offered through team teaching difficult and frustrating. These reports of student sentiment elicited some bemoaning of the lack of maturity of some first and second year students, of the common fixation on grades, and of the reluctance of some to allow themselves to be engaged by the open-ended flow of ideas that many of these courses aimed to inspire and their preference to be told exactly what specific material the professors wanted them to learn.

Debriefings with faculty

In the second year of the Pilot, administrative directors from the College began holding regular post-mortem meetings with faculty teaching Pilot courses (collectively in the case of team-taught courses) to review syllabi, learn how satisfied faculty were about how the course and the various assignments went, learn how satisfied they were with the quality of students’ work in the class (homework assignments, class discussions, exams or term papers), and to ask if there were changes they would like to make when offering the course again, and to read together the students evaluations. The purpose of these meetings was to help faculty take stock of the course and to offer resources for ongoing improvement.

Through the exercise of reading the evaluations, they discovered in perhaps every case a complexity in students’ responses to these courses that is not reflected in the standard course evaluations. Although the standard evaluations of overall course and instructor quality were in many instances lower than the school averages and were often lower than the evaluations these individual faculty themselves were used to seeing for their other courses—this was especially so in the case of team-teaching—the comments from the supplemental evaluation forms told a more interesting story. The supplemental forms elicited more comments, and more thoughtful comments, than is typical on the standard evaluations. In every course, there were a good number of students who were very enthusiastic about the course and the quality of the instructor(s) and also, in most courses, a number who were critical. For what it is worth, the positive comments usually outnumbered the negative ones, even when the total responses on the Likert scale associated with that question were evenly split between positive and negative. For whatever reason, enthusiastic students were more willing to express their enthusiasm with commentary than were students who were critical.

We have not been able to see whether a similar pattern is evident in the supplemental evaluations of a set of courses from the regular General Requirement, because the comments have not been collated.

6. Student Focus Groups

The Pilot Evaluation Committee has held 10 focus groups to gain insights into experiences of students who are enrolled in the Pilot Curriculum and in the regular curriculum. Three focus groups were held in 2000-2001, four in 2001-2002, and three in 2002-2003, with each group consisting of 5 to 12 students. All focus groups were videotaped and a graduate student observer prepared written summaries of the discussions.

The focus groups of Pilot students in the first two years solicited students’ views on features of the Pilot Curriculum that had led them to apply for the program, quality of Pilot Curriculum courses, use of electives, academic plan, research requirement, and advising. The focus groups in the third year emphasized students’ experiences with advising and the research requirement. The focus groups with non-Pilot students similarly sought information about students’ views on quality and choice of courses, an institution of an academic plan, research requirement, and advising.

The differences in the two curricula were mainly reflected in the students’ views of the choice of courses that they were required to take. Pilot students complained about the limited number of available required Pilot courses in their first two years at Penn and regular curriculum students expressed some frustration with required, mainly science, regular curriculum courses they had to take. In general, Pilot students appreciated the smaller number of requirements and regular curriculum students, particularly those who had applied for the Pilot Curriculum but were not chosen and those with a double major, expressed frustration with the number of regular curriculum requirements. Both Pilot Curriculum and regular curriculum students told us that required courses they had taken influenced their subsequent choice of courses. Experiences with advising ranged from very helpful to not helpful at all among both Pilot Curriculum and regular curriculum students, although overall Pilot students appeared to have been somewhat more satisfied with their advisors. Pilot students expressed frustration about the lack of information concerning the research requirement and, when asked, regular curriculum students were apprehensive about instituting a research requirement for all Penn students.

Pilot Students (6 focus groups)

The principal reasons for applying for the Pilot Curriculum were that it had fewer requirements and thus provided greater freedom to choose electives than the regular curriculum. These aspects seemed particularly attractive to students interested in a double major. Some students were also attracted to the Pilot Curriculum by the interdisciplinary nature of Pilot courses.

Most students were content in having chosen the Pilot Curriculum and many, but not all, would recommend it to others. At the same time, students were critical of the limited choice of Pilot Curriculum courses in their first two years at Penn and the quality of many of the courses offered. Courses students had taken shaped their views about course quality and not all views were negative. Most frequent complaints centered on too much material being covered in a single semester and a lack of coherence in courses with multiple instructors. Courses that students perceived to be better integrated and in which team teaching worked smoothly received more favorable reviews. In some cases Pilot Curriculum courses had influenced students’ subsequent choice of courses by encouraging further study in a given area or by discouraging students from taking additional courses in a subject matter. Recommendations for changes included increased variety in course offerings, improvement in the quality of instruction (if courses were to be team taught, two rather than three professors was preferable), and smaller classes.

In the first two years, students expressed frustration about the lack of information regarding the research requirement. By the spring of their junior year, none of the Pilot students had engaged in research experiences for the goal of fulfilling the Pilot Curriculum research requirement. However, many reported having conducted research in a variety of other settings: summer internships, an optional research paper for a class, an independent study, etc. A Vagelos scholar had conducted research in a lab, and others noted that in their view a research requirement for a major would overlap with the Pilot Curriculum research requirement.

Students gave mixed opinions about the required academic plan. Because many students must prepare a plan for a major, some students found the requirement redundant while others noted that the exercise was beneficial in helping them think through what they wanted to do and what was feasible. None of the students reported much give and take with their advisors in writing their academic plans. Nevertheless, many students who participated in a focus group in their junior year believed that writing the plan had been helpful, and most students reported that they were following their plans.

The reports on advising ranged from very helpful to not helpful at all and reflected the students’ experience with his/her advisor. Most students reported that they had sought information on the following topics: course selection, clarification of requirements, identification of faculty in various departments, exploring and selecting a major, and learning about academic options outside the major. Pilot students also reported that their advisors were unable to answer questions about the criteria or expectations for the research requirement.

Pilot Curriculum applicants currently enrolled in the regular curriculum (2 groups)

During the freshman year, students in this group felt that they had more choice in course offerings than the Pilot students, who had to choose from a limited number of available Pilot Curriculum courses. By the sophomore year, however, these views had changed. The regular curriculum students now felt that their freedom to choose was limited, they could not find interesting courses in all required areas, and some students complained about the quality of science courses for non-majors. These students felt that these science courses were of a lesser quality than required social science courses, which are taken by both majors and non-majors.

As was the case with Pilot students, required courses students had taken had influenced their subsequent choice of courses by both encouraging and discouraging further study in a given area.

Several students stated that they were planning to conduct research within their majors as a senior thesis, but all felt that a research requirement for all Penn students would not be a good idea. They stated that students who have no interest in engaging in research should not be required to do so. Students recommended some changes in the regular curriculum, such as allowing upper level courses to count for regular curriculum requirements.

Non-applicants enrolled in the regular curriculum (2 groups)

These focus groups were held with sophomores and juniors in 2001-2002 and 2002-2003. Students felt that they had adequate choice of classes, although those who were double majoring expressed some frustration in their ability to take electives. Students also expressed the view that the regular curriculum had provided a beneficial structure during their freshman year and that required courses gave them an opportunity to explore multiple fields. Students also noted that regular curriculum required courses had influenced their subsequent choice of courses, including a choice of a minor. These students expressed mixed views about instituting a research requirement for all Penn students. Students recommended several changes in the regular curriculum, including lowering the number of required courses, providing more flexibility in choosing courses among sectors, and allowing advanced placement courses to count towards the General Requirement.

As was the case with the Pilot students, students enrolled in the regular curriculum had mixed views of the advising process and these experiences were similar to those of the Pilot students. On the whole, however, a larger proportion of Pilot students were somewhat more positive about the advising process than non-Pilot students. At the same time, there was not much difference in the number of times Pilot Curriculum and regular curriculum students met with their advisors, or topics that were discussed, except that none of the regular curriculum students said that they had discussed academic options outside their major with their advisors.

7. Academic Advising in the Pilot Curriculum

If advising of Pilot freshmen differs in effectiveness from advising freshmen in the regular curriculum, it may be because of the different courses the Pilot students are taking or because of the way the additional electives available to Pilot students changes their outlook on their individual academic programs. These same factors affect advising of Pilot students in the sophomore year, along with the additional feature of the required academic plan. While sophomores in the regular curriculum continue with their freshman advisors, they are not required to meet with them. Conversations that do occur are not structured as they are for Pilot students, for whom the academic plan dictates a certain purpose and importance to their meetings. After the sophomore year, there are no systematic differences in the provisions made for their academic advising of Pilot versus non-Pilot students.

All students in the first Pilot class did in fact engage in a planning process with their advisors, and all of them submitted their academic plans in writing by the end of the second semester of the sophomore year. Advisors were asked to forward those plans to the College Office so they could be made available for analytical purposes and to transmit as appropriate to students’ major advisors. Except for a few that were lost by the student or the advisor after the advisor had signed off on them, the College Office received plans for each student.

Evaluation Committee’s Meetings with Pilot Curriculum Advisors

During the spring of the first two years of the Pilot (2001 and 2002), the evaluation committee met with those who advised students in the Pilot. For the first meeting, the committee formulated four questions in advance:

Did the advisors experience anything different about advising students in the Pilot curriculum compared with previous experiences advising students in the regular curriculum?

To what extend did they find students in the Pilot thinking ahead about their academic programs?

Did students in the Pilot seem more adventuresome in their thinking about their academic programs?

Were students in the Pilot curriculum asking different kinds of questions from those asked by students in the regular curriculum?

The general consensus among both faculty and assistant deans serving as academic advisors was that advising students in the Pilot was more satisfying but also more challenging than advising students in the regular curriculum. Questions students asked tended to be more about how they might follow up on interests raised in their Pilot classes than about what courses to take to satisfy a certain sector requirement. In general, discussions with advisees were more focused on students’ academic interests and goals and less on structural matters pertaining to requirements.

Advisors thus found themselves engaged in more interesting conversations with advisees about academic content, but they often felt unprepared to respond to questions about academic interests with helpful practical recommendations. One strategy some adopted was to refer students to faculty whom they knew who worked in the areas of their advisees’ interests or, failing that, to the undergraduate chair in the relevant department. Although it was somewhat disconcerting, especially for faculty serving as advisors, to not themselves be able to provide helpful advice in every instance, they felt that making referrals in this way after a conversation about students’ interests was preferable to what they remembered from advising sessions with students in the regular curriculum. Those conversations typically revolved around helping students decide which courses to take to fulfill various sector requirements.

In addition, advisors passed on to the committee some of the same concerns from students that the committee has heard directly from students themselves, namely that there were not many courses from which to choose and that students often had a difficult time integrating material and perspectives presented in team-taught courses, especially those involving three-person teams.

The second meeting with Pilot advisors was held in the spring of 2002 after Pilot students in the first class were expected to have written and submitted their academic plans. Several of the eleven advisors present required advisees to rewrite their plans after a first conversation with students about them. A common observation advisors made about the plans was that students tended to devote more attention to the Pilot courses they had taken than to what they would do with the elective portion of their studies. (It was observed that this tendency may have been created unwittingly by some of the sample plans that were made available to students.) Despite this, advisors found the exercise to be constructive. Several were adamant that the plans not be filed away but that they be made available to major advisors, not as firm commitments to which students would be held but rather as starting points for further conversations in which students’ plans for the major, and specifically for research, would become more refined.

8. Assessment of Academic Plans

Students in the Pilot Curriculum are required to write a comprehensive academic plan and discuss it with their advisor by the end of their sophomore year. According to the current guideline, the academic plan should include the following elements: brief personal narrative, overview of one’s program, choice of major, research project, and free electives.

To evaluate this requirement, the evaluation committee examined academic plans submitted by Pilot students in the spring of 2002. Each committee member read ten randomly chosen plans from among those submitted. These were discussed at a committee meeting.

Most of the academic plans were about one page long, single spaced. Not surprisingly, the plans varied widely in tone and content. The consensus of the committee was that while most of the plans were quite thoughtful, they generally did not say much about what had been regarded by the faculty as the most important element, namely, the intentions and rationale for taking elective courses. Furthermore, they tended to be more retrospective than prospective. A considerable amount of space was devoted to what general requirement courses they had taken, why they had chosen those courses, and what they had learned from them. Secondarily, they often talked at some length about the intellectual journey that led to their choice of a major, and how that choice related to their career plans, or lack thereof. Some students had a clear idea of how they intended to satisfy the research requirement, while others were quite vague or uncertain.

9. Research Requirement

The Pilot Curriculum is expected to offer students significant opportunities for individual research, scholarship and/or creative projects, and all Pilot students must engage in some form of research prior to graduation. Research is currently an option for students enrolled in the regular curriculum, although it is required in some majors. Pilot students have been informed that they can fulfill the research requirement in a number of ways – by taking an upper level research seminar or an independent study, writing a senior thesis, or conducting research in a laboratory. Students have been advised to consult with their major advisor to discuss ideas, available research opportunities, and to identify faculty who can offer guidance in the specific area of research. The college expects that most Pilot students will fulfill the research requirement in the context of their major during the students’ senior year. Research projects will be evaluated and approved by the student’s major department unless the student’s research project is in an area unrelated to the major, in which case the department related to the area of research will evaluate and approve the student’s work.

In spring 2002, the Committee on Undergraduate Education (CUE) solicited information about the nature of research experiences that are or could be offered for undergraduate students in the college. Not surprisingly, responses to this inquiry suggest that there is considerable variation in departmental research requirements and available research opportunities that are related to disciplinary considerations and the number of majors. All departments provide some opportunities for independent research, but all departments do not require that students engage in independent research as a condition for graduation. Most departments in humanities and social sciences expect students to take upper level research seminars, independent studies with a research component, or a capstone course, or students have the option of writing a senior thesis (in some departments this option is available to all students and in others students must qualify for departmental honors). The rigor of these requirements varies by department. In the science departments, most students gain research experience by working in a lab as a research assistant, a work study student, an independent study student, or a volunteer. In addition, many students gain experience in designing experiments, evaluating research findings, or conducting lab experiments as a part of an upper-level course within the major. Many departments, especially those with a large number of majors and science departments with considerable research related expenses associated with lab experiments, voiced concern about substantial changes in existing research requirements.

In the fall of 2003, the college surveyed Pilot Curriculum seniors via an email questionnaire to determine whether students were making progress in fulfilling the research requirement. Students were asked to give a brief description of their project, indicate the type of research they were engaged in (lab and other experiments, library research, secondary data analyses, survey study, creative project, other), whether the student planned to work on this project for one semester or longer, and the name of the student’s faculty advisor. Of the 175 Pilot students surveyed, 115 had responded by mid-November. Table 6 provides summary results from this survey.

Overall 30 students were engaged in some form of research in a laboratory that in many cases also involved library and/or other related research. Forty-five students reported that they were engaged in secondary or survey data analyses, including ethnographic field work, interviews and observations, also often in combination with library and other research. Five students indicated that their research involved a “creative project”, and 27 students identified library research either alone or in combination with unspecified research methods as their primary research methodology. Finally, eight students mentioned that their research experience consisted of an internship or a senior thesis without specifying the type of analyses they were undertaking. It is quite possible that other students were also writing a senior thesis without explicitly mentioning that their research was leading to a senior thesis.

In response to the question – “Do you plan to work on this project one semester or more than one semester?” – about half of the respondents reported that they planned to work on their projects more than one semester.

It is also noteworthy that only 66% of the Pilot students surveyed had responded to the email questionnaire by mid-November. It is possible, although not known, that non-response is associated with a lack of research experience on the part of the non-respondents. Further follow-up will be needed to determine whether this is in fact the case.

Table 6. Responses to the College Survey of Pilot Students’ Research Experience