Causality and Validity
EDP 618 Week 3
Dr. Abhik Roy
Cause
Variable that produces an effect or result
Most causes are inus -
A cause is an insufficient (i)
but non-redundant (n)
part of an unnecessary (u) but
sufficient condition (s)
A given event may have many different causes
Many factors are required for an effect to occur, but they can rarely be fully known and how they relate to one another
Counterfactual
Knowledge of what would have happened in the absence of a suspected causal agent
Physically impossible
Impossible to simultaneously receive and not receive a treatment
Therefore, the central task of all cause-probing research is to approximate the physically impossible counterfactual
Causal Description and Causal Explanation
Causal description
identifying that a causal relationship exists between A and B
Molar causation
the overall relationship between a treatment package and its effects
Causal explanation
explaining how A causes B
Molecular causation
knowing which parts of a treatment are responsible for which parts of an effect
Threats to Internal Validity: Single-group Studies (1/2)
A research team wants to study whether having indoor plants on office desks boosts the productivity of IT employees from a company. The researchers give each of the participating IT employees a plant to place by their desktop for the month-long study. All participants complete a timed productivity task before (pre-test) and after the study (post-test)
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position
Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position
Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect
Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position
Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect
Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long
Testing. Exposure to a test can affect test scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect
Threats to Internal Validity: Single-group Studies (2/2)
History. Events occurring concurrently with treatment that could cause the observed effect
Example. A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer
Maturation. Naturally occurring changes over time that could be confused with a treatment effect
Example. Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position
Instrumentation. The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect
Example. In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long
Testing. Exposure to a test can affect test scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect
Example. Participants showed higher productivity at the end of the study because the same test was administered. Due to familiarity, or awareness of the study’s purpose, many participants achieved high results
Threats to Internal Validity: Multi-group Studies (1/2)
A researcher wants to compare whether a phone-based app or traditional flashcards are better for learning vocabulary for the SAT. They divide 11th graders from one school into three groups based on baseline (pre-test) scores on vocabulary. For 15 minutes a day, Group A uses the phone-based app, Group B uses flashcards, while Group C spends the time reading as a control. Three months later, post-test measures of vocabulary are taken
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group
Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group
Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect
Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group
Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect
Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms
Selection. Systematic differences over conditions in respondent characteristics that could also cause the observed effect
Threats to Internal Validity: Multi-group Studies (2/2)
Additive and interactive threats. The impact of a threat can be added to that of another threat or may depend on the level of another threat
Example. Groups B and C may resent Group A because of the access to a phone during class. As such, they could be demoralized and perform poorly
Attrition. Loss of respondents to treatment or measurement can produce counterfactual effects if that loss is systematically correlated with conditions
Example. 20% of participants provided unusable data. Almost all of them were from Group C. As a result, it’s hard to compare the two treatment groups to a control group
Regression. When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with a treatment effect
Example. Because participants are placed into groups based on their initial scores, it’s hard to say whether the outcomes would be due to the treatment or statistical norms
Selection. Systematic differences over conditions in respondent characteristics that could also cause the observed effect
Example. Low-scorers were placed in Group A, while high-scorers were placed in Group B. Because there are already systematic differences between the groups at the baseline, any improvements in group scores may be due to reasons other than the treatment
That’s It!
Any questions?
This work is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License