Where Do We Want to Go Now? Information Technology, On-Line Experimentation, and the Social Sciences
Kent E. Portney
kent.portney@tufts.edu
Information technology has taken us to the door of a future that promises amazing new opportunities for teaching and research. This may have been said many times before, but it is a statement that holds as true today as ever before. This brief essay will focus on one type of opportunity, one that is emerging broadly in the social sciences, and one that Political Science, as a discipline, has tended to be slow to embrace. This one type of opportunity emerges from the development of on-line research and teaching collaboratories – on-line places where teachers and scholars can collaborate on their work and benefit from a shared experiences and results without each having to rely on the beneficence of their own institutions or extramural funding agencies to support and pay for it. In this essay, I shall focus on one type of emerging collaboratory that represents the works of a variety of social and behavioral scientists – web sites that provide access to on-line experiments.
In 1989, William Wulf (then at the National Science Foundation and subsequently president of the National Academy of Engineering) coined the term "collaboratory" to describe the concept of using information technologies to make geographically separate research units function as a single laboratory. Wulf defined a "collaboratory" as a "...‘center without walls’ in which the nation’s researchers can perform their research without regard to geographical location—interacting with colleagues, accessing instrumentation, sharing data and computational resources, and accessing information in digital libraries." [1] Since that time, the collaboratory idea has been expanded to advance opportunities for integrating teaching and research, and for promoting cross-disciplinary research – i.e. making it possible for researchers to collaborate and share resources not only across geographic areas but also across disciplines. Although the vast majority of progress toward creating actual working collaboratories has taken place in the sciences and engineering, in recent years the idea has started to take hold in the social sciences.
In October of 1997, the "NetLab Workshop" was held at the National Science Foundation to discuss the state of knowledge networking research in the social and behavioral sciences. The workshop brought together scientists from a number of disciplines with a goal of identifying the potential for furthering our understanding of social interactions via knowledge networking research, with a focus on the development of a new medium for research on social interactions: large-scale web-based experimentation. The result of the Workshop was a document providing, in part, guidance as to where social science experimentation collaboratories might be expected to be most fruitful, and advocating greater NSF support for building such initiatives.[2]
Partly as a result of subsequent NSF initiatives, and partly as a result of independent work of scholars around the country and the world, experimentation collaboratories are beginning to emerge. Although participation of teachers and researchers has fallen far short of the high expectations, opportunities for participation are likely to become easier and more plentiful in the years to come. There are already many dozens of web sites that provide some form of access to on-line experiments in the social sciences, especially in economics and sociology,[3]but very few of them have been conceived as collaboratories. Rather, they tend to be stand-along operations, each having it’s own (sometimes cryptic) pedagogical or research foundations, jargon, and methods of use. The advent of the collaboratory promises to provide some degree of standardization of content, access, and use.
What these experiment collaboratories have to offer is quite simple. If you anticipate using the Internet to conduct on-line research experiments in the future, the chances are that you will do so in the context of one or another collaboratory. Instead of designing and executing the experiment yourself, you may avail yourself of the experiments that are already contained and standardized in a collaboratory, or you will participate by making your experiments accessible to researchers and research subjects through a collaboratory. If you wish to have your students learning about some substantive experiment research results (results of experiments that someone else has already conducted and published), you can have your students read the research study AND engage them in their own experiment. To do this in the future, you are very likely to have your students access existing on-line experiments made available by a collaboratory. Moreover, regardless of whether the purpose is research or instruction, the resulting data become part of a large and ever-growing underlying database to be analyzed as needed. The product of this, of course, will be large, multi-investigator, studies.
In this essay, I would like to point toward two projects that serve as emerging models for what experimentation collaboratories promise and provide. One of these is an NSF funded project centered at the University of South Carolina, and the second is located at the University of Mississippi. What distinguishes these projects from the dozens of other on-line experimentation initiatives is that they have been conceived from the outset as collaboratories, designed to be multi-disciplinary, multi-investigator, and multi-institutional operations.
The Web-Lab Project at the University of South Carolina represents a significant advance in providing on line experiments in sociology and economics.[4] It initially integrates two pre-existing projects at that university, the “Business, Economics, Accounting, and Marketing” Laboratory (BEAM) project and the ExNet III project. The BEAM project, with economist Lisa Rutström and others as principals, provides access for teachers and researchers to a number of experiments, including versions of Charles Holt’s “Trading in a Pit Market,”[5] Susan Laury and Charles Holt’s “Multi-Market Equilibrium, Trade, and the Law of One Price,”[6] Holt and Lisa Anderson’s public choice experiment “Agendas and Strategic Voting,”[7] and Holt and Laury’s “Voluntary Provision of a Public Good” experiments,[8] among others.[9] The ExNet III project, developed by Sociology Professor David Willer, focuses on providing access to network exchange experiments.[10]
The intent of Web-Lab is to use these two projects as the foundation for expanding into a variety of other areas of investigation, and to create collaborations with other institutions. To date, Web-Lab has collaborated with a number of other institutions, including Georgia State University’s Experimental Economics Laboratory, and and sociologists at the University of Iowa, with additional testing of the Web-Lab architecture by sociologists at the University of New Mexico, the University of Mississippi, Southeast Missouri State University, and Western Illinois University.
The PsychExperiments Project at the University of Mississippi, headed by Psychology Professor Ken McGraw, provides access to on-line experiments created and used largely by experimental psychologists,[11] although in principle the model is applicable across the social and behavioral sciences. PsychExperiments was launched in Fall 1998 with funding from the U.S. Deptartment of Education’s Fund For the Improvement of Post Secondary Education (FIPSE) and from the University of Mississippi. It serves as an on-line cognitive and social psychology laboratory that takes experimentation on the Web beyond forms-based data collection to true interaction. It offers convenient data retrieval and analysis, a feature not yet available at other sites offering one or more interactive experiments (Francis, http://www.psych.purdue.edu/~coglab ).
In a nutshell, PsychExperiments allows prospective instructors to create and use experiments standardized in Authorware. Once created, these experiments are posted on the web site, and any instructor in the world can visit this site, select from the list of experiments, administer them to a class, and download the results. In order to participate in the development of these experiments, users must have a copy of Authorware, which is provided free to everyone who participates in the prospective training workshops.[12]
To integrate the diverse efforts of those who wish to contribute experiments to the current site, PsychExperiments has adopted a model or template Authorware program into which developers paste new experiments. This template allows developers to produce novel experiments that look and function like the others at the site. To do so, authors open up color-coded icons to make changes in variable values and names. Then they insert a single map icon that contains the flowline for the new experiment. This portion of the piece, called "New experiment," presents the screens unique to the new experiment and measures the dependent variable. Using this template, novel experiments can be prepared and ready for class use in a day or two by anyone who has reasonable facility with Authorware.
Frequency of Data File and Source File Downloads Over First Five Months at PsychExperiments
Experiment
Data File Downloads
Source File Downloads
Facial Recognition
126
113
Mental Rotation
64
64
Mueller-Lyer
50
49
Pitch Memory
28
21
Poggendorff Illusion
52
36
Reaction Time
46
37
Stroop
81
61
Temporal Judgment
12
9
Word Recognition
55
46
Totals
514
436
The statistics of site use indicate the degree of acceptance PsychExperiments has gained. These statistics are based on usage of the nine experiments then available (details about each of these experiments may be found on the PsychExperiments web site). Since going on-line in September 1998, 16,675 hits to the home page have been recorded and hits continue at the rate of about 200 per day. User information has been submitted by 733 users from 246 different universities, high schools, and other institutions in the United States and from Germany, Poland, United Kingdom, Lebanon, Canada, Portugal, Netherlands, Hong Kong, Belgium, Spain, Korea, Japan, and Finland. As shown in Table 1, data from the experiments have been downloaded 514 times, and Authorware programs have been downloaded 436 times.
Excel macros for data analysis have been taken by 62 users. In Fall, 1998, and Spring, 1999, excluding use by classes at the University of Mississippi, 30 college and university classes had registered to use this resource, as well as one junior college class and two high school classes that participate in the International Baccalaureate program. Undergraduate institutions range from the University of Alaska--Anchorage to St. Anselm College in New Hampshire.
To date, a graduate student at the University of Indiana has developed her own experiment to be posted at the site, a California 4th grade student collected data for her science fair project, and one undergraduate has completed a research project requiring remote site data collection from Prader-Willi clients. These are meager achievements in an absolute sense, yet they illustrate the potential of a Web-based collaboratory to meet the needs of a wide variety of potential users.
In summary, these two collaboratories provide a glimpse into future directions for on-line research and teaching in the social sciences. Increasingly, such collaborative initiatives will serve as the foundation for substantial amounts of the work that we do. Of course, one might ask whether such fundamental changes in the nature of research and teaching will produce better learning or higher quality research, but that is an issue for another day.
[1] Computer Science and Telecommunications Board (CSTB). 1993. National Collaboratories: Applying Information Technology for Scientific Research. Washington, DC: National Academy Press.
[2] See http://www.uiowa.edu/~grpproc/netlab.htm for the on-line version of the report.[3] For some examples from economics, see Cal Tech’s Laboratory for Experimental Economics and Political Science at http://eeps.caltech.edu/ , the Virtual Laboratory at Berkeley at http://elsa.berkeley.edu/vlab/vlab.html , and Charles Holt’s Classroom Experiments site at http://www.people.Virginia.EDU/~cah2k/teaching.html , to name a few.
[4] http://econ.badm.sc.edu/beam/ . For a description of the Web-Lab Project in Acrobat pdf format, see http://econ.badm.sc.edu/beam/web-lab_project.pdf . [5] Charles A. Holt, Journal of Economic Perspectives 10(1), 1996: 193-203.
[6] Susan K. Laury and Charles A. Holt, Southern Economic Journal, 65(3) 1998, 611-621.
[7] Charles A. Holt and Lisa R. Anderson, Southern Economic Journal, 65(3) 1998, 622-629.
[8] Charles A. Holt and Susan K. Laury, Journal of Economic Perspectives 11(4), 1997: 209-215.
[9] More information about each of these experiments is available from the BEAM web site at http://theweb.badm.sc.edu/laury/games.html . [10] See Dudley Girard and David Willer "ExNet III: A Web-Based Resource for Experimental Research" Social Psychology, Spring 1999.[11] http://www.olemiss.edu/PsychExps/
[12] The experiments currently at PsychExps were developed using Authorware 4.0 but Authorware 5.0 was released Fall '98. Unlike 4.0, which came in both Mac and PC versions, 5.0 authoring requires a PC. The completed programs can be run on either the Mac or PC platform, as before.
last modified on
01/11/2002