+ - 0:00:00
Notes for current slide
Notes for next slide

Causality and Validity

EDP 618 Week 3

Dr. Abhik Roy

1 / 39
2 / 39

Experiments and Causation

3 / 39

Cause

4 / 39

Cause

  • Variable that produces an effect or result
4 / 39

Cause

  • Variable that produces an effect or result

  • Most causes are inus -

A cause is an insufficient (i)

but non-redundant (n)

part of an unnecessary (u) but

sufficient condition (s)

4 / 39

Cause

  • Variable that produces an effect or result

  • Most causes are inus -

A cause is an insufficient (i)

but non-redundant (n)

part of an unnecessary (u) but

sufficient condition (s)

  • A given event may have many different causes
4 / 39

Cause

  • Variable that produces an effect or result

  • Most causes are inus -

A cause is an insufficient (i)

but non-redundant (n)

part of an unnecessary (u) but

sufficient condition (s)

  • A given event may have many different causes

  • Many factors are required for an effect to occur, but they can rarely be fully known and how they relate to one another

4 / 39

Effect

5 / 39

Effect

  • Difference between what did happen and what would have happened
5 / 39

Effect

  • Difference between what did happen and what would have happened

  • This reasoning generally requires a counterfactual

5 / 39

Counterfactual

6 / 39

Counterfactual

  • Knowledge of what would have happened in the absence of a suspected causal agent
6 / 39

Counterfactual

  • Knowledge of what would have happened in the absence of a suspected causal agent

    • Physically impossible
6 / 39

Counterfactual

  • Knowledge of what would have happened in the absence of a suspected causal agent

    • Physically impossible

    • Impossible to simultaneously receive and not receive a treatment

6 / 39

Counterfactual

  • Knowledge of what would have happened in the absence of a suspected causal agent

    • Physically impossible

    • Impossible to simultaneously receive and not receive a treatment

    • Therefore, the central task of all cause-probing research is to approximate the physically impossible counterfactual

6 / 39

Causal Relationships

7 / 39

Causal Relationships

A causal relationship requires three conditions

7 / 39

Causal Relationships

A causal relationship requires three conditions

  1. Cause preceded effect (temporal precedence)
7 / 39

Causal Relationships

A causal relationship requires three conditions

  1. Cause preceded effect (temporal precedence)

  2. Cause and effect covary

7 / 39

Causal Relationships

A causal relationship requires three conditions

  1. Cause preceded effect (temporal precedence)

  2. Cause and effect covary

  3. No other plausible alternative explanations can account for a causal relationship

7 / 39

Cause, Effect, and Causal Relationships

8 / 39

Cause, Effect, and Causal Relationships

  • In experiments
8 / 39

Cause, Effect, and Causal Relationships

  • In experiments

    • Presumed causes are manipulated to observe their effect
8 / 39

Cause, Effect, and Causal Relationships

  • In experiments

    • Presumed causes are manipulated to observe their effect

    • Variability in cause related to variation in an effect

8 / 39

Cause, Effect, and Causal Relationships

  • In experiments

    • Presumed causes are manipulated to observe their effect

    • Variability in cause related to variation in an effect

    • Elements of design and extra-study knowledge are used to account for and reduce the plausibility of alternative explanations

8 / 39

Causation, Correlation, and Confounds

9 / 39

Causation, Correlation, and Confounds

  • Correlation does not prove causation
9 / 39

Causation, Correlation, and Confounds

  • Correlation does not prove causation

  • Correlations do not meet the first premise of causal logic (temporal precedence)

9 / 39

Causation, Correlation, and Confounds

  • Correlation does not prove causation

  • Correlations do not meet the first premise of causal logic (temporal precedence)

  • Such relationships are often due to a third variable (i.e., a confound)

9 / 39

Manipulable and Nonmanipulable Causes

10 / 39

Manipulable and Nonmanipulable Causes

  • Experiments involve causal agents that can be manipulated
10 / 39

Manipulable and Nonmanipulable Causes

  • Experiments involve causal agents that can be manipulated

  • Nonmanipulable causes (e.g., ethnicity, gender) cannot be causes in experiments because they cannot be deliberately varied

10 / 39

Causal Description and Causal Explanation

11 / 39

Causal Description and Causal Explanation



Causal description

identifying that a causal relationship exists between A and B

11 / 39

Causal Description and Causal Explanation



Causal description

identifying that a causal relationship exists between A and B

Molar causation

the overall relationship between a treatment package and its effects

11 / 39

Causal Description and Causal Explanation



Causal description

identifying that a causal relationship exists between A and B

Molar causation

the overall relationship between a treatment package and its effects

Causal explanation

explaining how A causes B

11 / 39

Causal Description and Causal Explanation



Causal description

identifying that a causal relationship exists between A and B

Molar causation

the overall relationship between a treatment package and its effects

Causal explanation

explaining how A causes B

Molecular causation

knowing which parts of a treatment are responsible for which parts of an effect

11 / 39

Causal Models

causality
12 / 39

Causal Models

moderator-mediator

moderator-mediator

13 / 39

Causal Models

moderator-mediator

moderator-mediator



moderator-mediator

moderator-mediator

13 / 39

Modern Descriptions of Experiments

14 / 39

Randomized Experiment

15 / 39

Randomized Experiment

  • Units are assigned to conditions randomly
15 / 39

Randomized Experiment

  • Units are assigned to conditions randomly

  • Randomly assigned units are probabilistically equivalent based on expectancy (if certain conditions are met)

15 / 39

Randomized Experiment

  • Units are assigned to conditions randomly

  • Randomly assigned units are probabilistically equivalent based on expectancy (if certain conditions are met)

  • Under the appropriate conditions, randomized experiments provide unbiased estimates of an effect

15 / 39

Quasi-Experiment

16 / 39

Quasi-Experiment

  • Shares all features of randomized experiments except assignment
16 / 39

Quasi-Experiment

  • Shares all features of randomized experiments except assignment

  • Assignment to conditions occurs by self-selection

16 / 39

Quasi-Experiment

  • Shares all features of randomized experiments except assignment

  • Assignment to conditions occurs by self-selection

  • Greater emphasis on enumerating and ruling out alternative explanations

16 / 39

Quasi-Experiment

  • Shares all features of randomized experiments except assignment

  • Assignment to conditions occurs by self-selection

  • Greater emphasis on enumerating and ruling out alternative explanations

    • ... through logic and reasoning, design, and measurement
16 / 39

Natural Experiment

  • Naturally-occurring contrast between a treatment and comparison condition
17 / 39

Natural Experiment

  • Naturally-occurring contrast between a treatment and comparison condition

  • Typically concern nonmanipulable causes

17 / 39

Natural Experiment

  • Naturally-occurring contrast between a treatment and comparison condition

  • Typically concern nonmanipulable causes

  • Requires constructing a counterfactual rather than manipulating one

17 / 39

Nonexperimental Designs

  • Often called correlational or passive designs (i.e., cross-sectional)
18 / 39

Nonexperimental Designs

  • Often called correlational or passive designs (i.e., cross-sectional)

  • Statistical controls often used in place of structural design elements

18 / 39

Nonexperimental Designs

  • Often called correlational or passive designs (i.e., cross-sectional)

  • Statistical controls often used in place of structural design elements

  • Generally do not support strong causal inferences

18 / 39

Experiments and the Generalization of Causal Connections

19 / 39

Most Experiments are Local but have General Aspirations

20 / 39

Most Experiments are Local but have General Aspirations

  • Most experiments are localized
20 / 39

Most Experiments are Local but have General Aspirations

  • Most experiments are localized

  • Limited samples of utos

units (u)

treatments (t)

observations (o)

settings (s)

20 / 39

Most Experiments are Local but have General Aspirations

  • Most experiments are localized

  • Limited samples of utos

units (u)

treatments (t)

observations (o)

settings (s)

  • What Campbell labeled local molar causal validity
20 / 39

Construct Validity: Causal Generalization as Representation

21 / 39

Construct Validity: Causal Generalization as Representation

  • Premised on generalizing from particular sampled instances of units, treatments, observations, and settings to the abstract, higher order constructs that sampled instances represent
21 / 39

External Validity: Causal Generalization as Extrapolation

22 / 39

External Validity: Causal Generalization as Extrapolation

  • Inferring a causal relationship to unsampled units, treatments, observations, and settings from sampled instances
22 / 39

External Validity: Causal Generalization as Extrapolation

  • Inferring a causal relationship to unsampled units, treatments, observations, and settings from sampled instances

  • Enhanced when probability sampling methods are used

22 / 39

External Validity: Causal Generalization as Extrapolation

  • Inferring a causal relationship to unsampled units, treatments, observations, and settings from sampled instances

  • Enhanced when probability sampling methods are used

    • Broad to narrow
22 / 39

External Validity: Causal Generalization as Extrapolation

  • Inferring a causal relationship to unsampled units, treatments, observations, and settings from sampled instances

  • Enhanced when probability sampling methods are used

    • Broad to narrow

    • Narrow to broad

22 / 39

Approaches to Making Causal Generalizations

23 / 39

Approaches to Making Causal Generalizations

  • Sampling
23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

  • Surface similarity

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

  • Surface similarity

  • Ruling out irrelevancies

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

  • Surface similarity

  • Ruling out irrelevancies

  • Making discrimination

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

  • Surface similarity

  • Ruling out irrelevancies

  • Making discrimination

  • Interpolation and extrapolation

23 / 39

Approaches to Making Causal Generalizations

  • Sampling

  • Probabilistic

  • Heterogeneous instances

  • Purposive

  • Grounded theory

  • Surface similarity

  • Ruling out irrelevancies

  • Making discrimination

  • Interpolation and extrapolation

  • Casual explanation

23 / 39

Statistical Conclusion Validity and Internal Validity

24 / 39

Validity

25 / 39

Validity

  • Approximate truthfulness of correctness of an inference
25 / 39

Validity

  • Approximate truthfulness of correctness of an inference

  • Not an all or none, either or, condition, rather a matter of degree

25 / 39

Validity

  • Approximate truthfulness of correctness of an inference

  • Not an all or none, either or, condition, rather a matter of degree

  • Efforts to increase one type of validity often reduce others

25 / 39

Statistical Conclusion Validity

26 / 39

Statistical Conclusion Validity

Validity of inferences about the covariation between treatment (cause) and outcome (effect)

26 / 39

Internal Validity

27 / 39

Internal Validity

Validity of inferences about whether observed covariation between A (treatment/cause) and B (outcome/effect) reflects a causal relationship from \(A\) to \(B\) as those variables were manipulated or measured*

27 / 39

Construct Validity

28 / 39

Construct Validity

Validity of inferences about the higher order constructs that represent sampling particulars

28 / 39

External Validity

29 / 39

External Validity

Validity of inferences about whether a cause-effect relationship holds over variations in units, treatments, observations, and settings

29 / 39

Threats to Validity

30 / 39

Threats to Validity

  • Reasons why an inference may be partly or wholly incorrect
30 / 39

Threats to Validity

  • Reasons why an inference may be partly or wholly incorrect

  • Design controls can be used to reduce many validity threats, but not in all instances

30 / 39

Threats to Validity

  • Reasons why an inference may be partly or wholly incorrect

  • Design controls can be used to reduce many validity threats, but not in all instances

  • Threats to validity are generally context-dependent

30 / 39

Internal Validity

31 / 39

Internal Validity

  • Inferences about whether the observed covariation between \(A\) and \(B\) reflects a causal relationship from \(A\) to \(B\) in the form in which the variables were manipulated or measured
31 / 39

Internal Validity

  • Inferences about whether the observed covariation between \(A\) and \(B\) reflects a causal relationship from \(A\) to \(B\) in the form in which the variables were manipulated or measured

  • In most cause-probing studies, internal validity is the primary focus

31 / 39

Threats to Internal Validity: Single-group Studies (1/2)

32 / 39

Threats to Internal Validity: Single-group Studies (1/2)

A research team wants to study whether having indoor plants on office desks boosts the productivity of IT employees from a company. The researchers give each of the participating IT employees a plant to place by their desktop for the month-long study. All participants complete a timed productivity task before (pre-test) and after the study (post-test)

32 / 39

Threats to Internal Validity: Single-group Studies (2/2)

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position

Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position

Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect

Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position

Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect

Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long

Testing. Exposure to a test can affect test scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect

33 / 39

Threats to Internal Validity: Single-group Studies (2/2)

History. Events occurring concurrently with treatment that could cause the observed effect

Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer

Maturation. Naturally occurring changes over time that could be confused with a treatment effect

Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position

Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect

Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long

Testing. Exposure to a test can affect test scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect

Example. Participants showed higher productivity at the end of the study because the same test was administered. Due to familiarity, or awareness of the study’s purpose, many participants achieved high results

33 / 39

Threats to Internal Validity: Multi-group Studies (1/2)

34 / 39

Threats to Internal Validity: Multi-group Studies (1/2)

A researcher wants to compare whether a phone-based app or traditional flashcards are better for learning vocabulary for the SAT. They divide 11th graders from one school into three groups based on baseline (pre-test) scores on vocabulary. For 15 minutes a day, Group A uses the phone-based app, Group B uses flashcards, while Group C spends the time reading as a control. Three months later, post-test measures of vocabulary are taken

34 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group

Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group

Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect

Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group

Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect

Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms

Selection. Systematic differences over conditions in respondent characteristics that could also cause the observed effect

35 / 39

Threats to Internal Validity: Multi-group Studies (2/2)

Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat

Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly

Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions

Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group

Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect

Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms

Selection. Systematic differences over conditions in respondent characteristics that could also cause the observed effect

Example. Low-scorers were placed in Group A, while high-scorers were placed in Group B. Because there are already systematic differences between the groups at the baseline, any improvements in group scores may be due to reasons other than the treatment

35 / 39

Estimating Internal Validity in Experiments

36 / 39

Estimating Internal Validity in Experiments

  • By definition randomized experiments eliminate selection through random assignment to conditions
36 / 39

Estimating Internal Validity in Experiments

  • By definition randomized experiments eliminate selection through random assignment to conditions

  • Most other threats are (should be) probabilistically distributed as well

36 / 39

Estimating Internal Validity in Experiments

37 / 39

Estimating Internal Validity in Experiments

  • Only two likely validity threats (typically) arise from experiments
37 / 39

Estimating Internal Validity in Experiments

  • Only two likely validity threats (typically) arise from experiments

    • Attrition
37 / 39

Estimating Internal Validity in Experiments

  • Only two likely validity threats (typically) arise from experiments

    • Attrition

    • Testing

37 / 39

Estimating Internal Validity in Quasi-Experiments

38 / 39

Estimating Internal Validity in Quasi-Experiments

  • Differences between groups tend to be more systematic than random
38 / 39

Estimating Internal Validity in Quasi-Experiments

  • Differences between groups tend to be more systematic than random

  • All threats should be made explicit and then ruled out one by one

38 / 39

Estimating Internal Validity in Quasi-Experiments

  • Differences between groups tend to be more systematic than random

  • All threats should be made explicit and then ruled out one by one

  • Once identified, threats can be systematically examined

38 / 39

That’s It!

Any questions?

39 / 39

That’s It!

Any questions?













Creative Commons License

This work is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
39 / 39
2 / 39
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
oTile View: Overview of Slides
Esc Back to slideshow