Essay

The Philadelphia Social History Project (PSHP) was a large and ambitious interdisciplinary project that played a central role in transforming the study of urban and social history. From its inception in 1969 until the project’s closure in 1985, the PSHP employed dozens of research associates and computer programmers, as well as hundreds of undergraduate students, to create a computerized database to study the effect of industrialization and urbanization on Philadelphia’s minority populations between 1838 and 1880.

Under the direction of Theodore Hershberg (b. 1941) at the University of Pennsylvania, the PSHP received $78,000 from the Center for Studies of Metropolitan Problems of the National Institute of Mental Health in 1969 to create machine-readable data and develop automated processes to study rates of social mobility of the African American, Irish, and German populations in Philadelphia. Building on research models developed by scholars such as Stephan Thernstrom, and responding in part to increased public awareness of civil rights issues in the United States, Hershberg and PSHP researchers sought to determine, among other topics, whether the African American socioeconomic experience was unique or whether it shared characteristics with Philadelphia’s other large minority ethnicities.

Beginning with data from the Pennsylvania Abolition Society and the Society of Friends manuscript censuses from 1838, 1847, and 1856, and supplementing this with data from the U.S. population censuses of 1850, 1860, 1870, and 1880, Hershberg and his associates developed ways to convert tens of thousands of records into machine-readable form and input these into a large database. Computer programmers, many from the Graduate Group in Computer and Information Science at the Moore School at the University of Pennsylvania, wrote sophisticated programs so that the data could be manipulated and analyzed by PSHP research associates. With additional funding eventually totaling more than $3 million from NIMH and other agencies, the PSHP database was expanded to include census information about native-born white Americans, as well as data from U.S. Census manufacturing schedules, mortality registers, city business directories, and lists of institutions and voluntary associations.

New Strides in Linking Data

Beyond the creation of a large, comprehensive database, PSHP’s greatest accomplishments may have been the processes its researchers and programmers developed to assign geographic coordinates to data and to link disparate records within the database. By placing every person and institution within a specific map “grid square” (a fixed area, usually 1 x 1¼ city blocks), Hershberg and his associates could study the city and its population in ways, and at scales, that previously had been impracticable. For instance, by associating data about transportation networks with grid coordinates, researchers could compare “journeys-to-work” of different communities and better understand how transportation affected the social and spatial organization of the city.   Sophisticated computer programs added breadth and depth to PSHP studies because linkages could be made to match individuals, industries, or associations from multiple data sources. PSHP researchers could, with reasonable certainty, match individuals and businesses, even those whose names or addresses had changed, across data sets from different years.

PSHP was one of the earliest, and largest, projects in the humanities to employ a computerized database to conduct its research. It drew researchers from multiple disciplines including history, demography, ethnic studies, urban history, economics, and sociology and operated on an unprecedented scale of staffing, funding, and ambition. It produced massive amounts of statistical information and developed innovative and sophisticated programs to manipulate and analyze data. But while the project was frequently hailed as transformative because of its interdisciplinary focus and because of the multiple uses to which its database could be put, it also was criticized for overemphasizing quantitative, rather than qualitative, data analysis. Critics also called attention to the project’s comparatively small output of published scholarship (relative to its considerable funding) and inaccuracies in the database (in particular, categorization of occupational and industrial data from the 1900 and 1930 censuses).

Hershberg left the Philadelphia Social History Project in 1981 and later founded the Center for Greater Philadelphia at the University of Pennsylvania. The PSHP continued to operate until its funding ended in 1985. Many of the electronic data sets were retained by PSHP research associates. The paper and microfilm archive, containing published and unpublished reports, system manuals, program and tape documentation, administrative records, maps, printouts of tabulated information from PSHP data sets, and photocopies of census schedules, was transferred to the University of Pennsylvania Libraries.

Regan Kladstrup is the Assistant Director of the Special Collections Processing Center at the Kislak Center for Special Collections, Rare Books and Manuscripts, University of Pennsylvania Libraries.

Copyright 2015, Rutgers University