Internal and External Validity:

Bivariate Analysis.

Multiple Regression

Tables.

Internal and External Validity:

Links:

• GUIDE 4:(scroll down a bit)
• Psych 404 (very good, check items a,b,c,d,e,f,g,h,i)
• Psych 202 (briefer) (.pdf file)
• Educational Psychology Interactive

• Campbell & Stanley: Threats to Internal and External Validity

Expecially study the examples of internal validity in the Psych 404 page.:

Scenarios:

Scenario 1:

After years of grading his Introductory American Government exams, Professor Casper Curmudgeon discovered a pattern: Students who received low grades (C or less) on his midterm exams consistently improved their scores – by an average of half a letter grade – on their final exams.  At first he thought this was due to sample mortality (weaker students dropping out of the course), but the effect remained when he calculated the averages for only for those students who took both tests.  He also discovered that students who received good grades (B+ or better) on the midterm had their grades drop half a letter grade on the final.

"Grades," Curmudgeon concluded, "are powerful motivating force."  Students who received good grades on the midterm, he reasoned, slack off in their studies while those who receive low grades try harder.  From then on he never gave a grade higher than a B on his midterms.

Scenario 2:

In 1980 the National Highway Transportation Safety Administration conducted an experiment to evaluate the effectiveness of a new high-mounted rear window brake light on passenger cars.  Working with a national rental car agency, they randomly installed rear brake lights on half the agency's fleet of cars.   At the end of a two-year trial it was discovered that the cars with the new lights experienced 35% fewer rear-end crashes and 25% fewer fatal accidents than the cars without the lights.   If installed on all automobiles, they reasoned, thousands of lives would be saved.  Subsequently, all automobile manufacturers were required to install the lights. 

But after 5 years, when 90% of all automobiles had the lights, the traffic fatality and rear –end collision rates had dropped only 2% (per million miles traveled).  What went wrong?

Scenario 3:

The Normal Police department instituted an intersection safety program.   At the beginning of the month, the department identifies the intersections with the most traffic accidents and then implements an intensive police patrol at those intersections.   The new patrol has been an enormous success, said the police chief, and has resulted in an average reduction of 35% in accidents at those intersections.
Does the program really work?

Scenario 4:

 Cutting taxes is not the way to economic prosperity.  The 1980 tax reductions instituted under President Reagan led to an immediate recession, rising crime and poverty rates and huge budget deficits.  The 1992 tax increases passed under the Clinton administration were followed by eight years of economic growth, steady reductions in crime and poverty, and the first budget surpluses in decades. 

Scenario 5:
In 1998 states that had instituted and used the death penalty had an average murder rate of 6.6%, while in the states that did not employ the death penalty, the murder rate was only 3.7%.

Clearly this shows that the death penalty, rather than deterring crime actually causes more of it.

Bivariate Analysis.

Multiple Regression:

Multiple Regression Terms.
(Dependent Variable: NAEP Reading Scores)

N = 41 states; ^ap<.01; ^bp<.05

NOTE: RC = scores on NAEP Reading Comprehension Test, fourth-graders;
SL: Books = school library, number of books per child;
SL: Software = school library, software available;
SL Service = school library; services available;
PL = public library, annual circulation per capita;
Exp = expenditures per child.

Constant (or intercept) The estimated value of the dependent variable when all the independent variable equal zero. Generally, this is not a meaningful number.  Example: not shown in this table.

b    The unstandardized regression coefficient (what the slope would be if all the other variables in the equation were held constant).  The change in the dependent variable for each one-unit increase in the independent variable.  Example: If the number of school library books per child were to increase by 10 (say from 50 to 60 books per child), we would expect the reading scores to go up 4.46 points, all other things being equal.

Standard error:  The confidence interval for the unstandardized coefficient.  Multiply this by 2 to get the 95% interval.  If the b is negative and is twice as large as its standard error, we can be 95% sure that the actual effect is indeed negative.  Example:  The effect of SL: Software is .087 plus or minus .108 (2 times .054).  Because the confidence interval ranges from a -.02 to +.195 we cannot be sure that effect of software is positive.  The coefficient (.087) is thus not significant)

t (or t-statistic or t-ratio).  The unstandardized coefficient divided by the standard error.  The unstandardized coefficient is regarded as significant if it is twice the standard error, if the t is larger than 2.

Example: SL books; SL service; and PL all have t statistics greater than 2.

p  The probability that the coefficient (either a standardized or unstandardized coefficient or a correlation coefficient) is due to the small number of cases.  A p value less than .05 is generally regarded as significant.  SL: Software and school expenditures have no significant effect on reading scores.

beta (standardardized regression coefficient).  What the correlation coefficient would be if all the variables in the equation were held constant. Compare this with the correlation coefficient to see if the original relationship was spurious.  Example: SL Servicee has the strongest independent effect on readings scores; the effect is negative; all other things being equal, states with more library services have lower reading scores.

R-square.   A measure of the accuracy of the whole equation.  The percent of the variation in the dependent variable that is explained by the independent variables.  Example:  The five variables in the equation explain 60% of the variation in readings scores.

Tables.

*50 state data

N = 41 states; ^ap<.01; ^bp<.05

NOTE: RC = scores on NAEP Reading Comprehension Test, fourth-graders;
SL: Books = school library, number of books per child;
SL: Software = school library, software available;
SL Service = school library; services available;
PL = public library, annual circulation per capita;
Exp = expenditures per child.

Stephen D. Krashen, "School Libraries, Public Libraries, and the NAEP Reading Scores" SLMQ Volume 23, Number 4, Summer 1995  (article on-line)
 

N= 245 cases (convicted defendants)
Dependent variable = Sentence Severity {ranges from 0 (for a deferred sentence) to 30 (15 or more years}.
(from Krashen, op.cit.):

Table 3. Multiple-Regression Analysis
(Dependent Variable: NAEP Reading Scores)