We have developed a brief, multiple-choice Science Survey which evaluates knowledge and understanding of important scientific concepts and facts, as well as scientific reasoning ability. This survey was designed for the purpose of evaluating science learning resulting from a liberal arts education at a selective university. The following paragraphs provide a brief description, followed by information about how to obtain the Science Survey for use in evaluating scientific literacy at the university level.

Undergraduate Chairpersons and other science faculty developed items to assess knowledge and understanding of biology, chemistry, environmental science, physics, psychology, mathematics and statistics, as well as scientific reasoning. (We are particularly grateful to Drs. Jon Baron, Don Berry, and Larry Gladney for their very helpful contributions.) These items were evaluated and revised by science faculty and compiled for pilot testing. Based on feedback from undergraduates who participated in the pilot tests, as well as analysis of the patterns of response, we selected 24 items for the Science Survey.

This Science Survey was administered to entering Freshmen during Orientation in September, 2001. We received 1265 usable surveys (a response rate of 83%). In this setting, the rate of missing values was relatively high for some items, particularly those near the end of the Science Survey. Students who skipped items or failed to finish the Science Survey had fewer correct answers for the items they completed, so we have treated missing values as wrong responses in our analyses. For each item, we evaluated correlations between whether the item was answered correctly and (1) the percent correct for the whole Science Survey and (2) the number of AP and college-level science and math courses taken before entering the university. We evaluated one item as unsatisfactory due to very low correlations, as well as high missing values, and this item was omitted from the analyses described below. The other 23 items were considered satisfactory and have been included in our analyses.

The Score for the Science Survey was calculated as the sum of the values for the individual items. For each of the 20 single-component items, a value of 1 was assigned if the answer was correct and 0 otherwise. Three items had four components each, and for these items the value was the sum of 0.25 for each correct component. The mean Score was 12.6, indicating an overall average of 55% correct. The Score on the Science Survey correlated (r = 0.27) with the number of AP and college-level science and math courses taken before entering the university, providing some support for the validity of the Science Survey Score as a measure of scientific literacy.

The Cronbach Alpha for the Score for the Science Survey was 0.61. Factor analysis (using principal components analysis with varimax rotation) yielded five factors which accounted for 5-7% of the variance each. One of these factors evaluated science knowledge, and another evaluated primarily quantitative skills. The other factors were harder to interpret. The Cronbach Alpha and the factor analysis indicate that this Science Survey evaluates a broad range of abilities related to scientific literacy and not a single unitary concept.

The Science Survey is available in two forms to those who are interested in evaluating scientific literacy at the university level. The first form has the 24 items used in the survey of entering Freshmen described above. The second form includes 23 items, several of which have been revised to improve clarity and precision and one of which is a new proposed replacement item. We plan to pilot test this revised version in the fall, together with additional items to evaluate attitudes toward science and understanding of the nature of science.

Copies of the Science Survey are available by contacting Dr. Kent Peterman, College of Arts and Sciences, 120 Logan Hall, University of Pennsylvania, Philadelphia, PA 19104-6304 (peterman@sas.upenn.edu) (215-898 -7867).

By Ingrid Waldron, Kent Peterman, and Paul Allison
(from the Department of Biology, College of Arts and Sciences Office, and Department of Sociology, respectively)
June, 2002