Cohen’s Kappa

class: center, middle, inverse, title-slide

.title[
# Cohen’s Kappa
]
.subtitle[
## EDP 618 Week 10
]
.author[
### Dr. Abhik Roy
]

---

section {
    height: 600px;
    width: 60%;
    margin: auto;
    border-radius: 21px;
    background-color: #212121;
}

.remark-slide-container {
background: #212121;
}

.hljs-github .hljs {
    background: transparent;
    color: #b2dfdb;
}

.hljs-github .hljs-keyword {
    color: #64b5f6;
}

.hljs-github .hljs-literal {
    color: #64b5f6;
}

.hljs-github .hljs-number {
    color: #64b5f6;
}

.hljs-github .hljs-string {
    color: #b7b3ef;
}

.hljs-github .hljs {
    background: transparent;
    color: #b2dfdb;
}

.hljs-github .hljs-keyword {
    color: #64b5f6;
}

.hljs-github .hljs-literal {
    color: #64b5f6;
}

.hljs-github .hljs-number {
    color: #64b5f6;
}

.hljs-github .hljs-string {
    color: #b7b3ef;
}

section p {
    text-align: center;
    font-size: 30px;
    background-color: #212121;
    border-radius: 21px;
    font-family: Roboto Condensed;
    font-style: bold;
    padding: 12px;
    color: #bff4ee;
    margin: auto;
}

#center {
text-align: center;
}

#right {
  text-align: right;
}

.center p {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.tab {
    display: inline-block;
    margin-left: 40px;
}

.obr
{
    display:block;
    margin-top:-15px;
}

.container {
  display: flex;
}

.container > div {
  flex: 1; /*grow*/
  margin-right: 40px;
}

td, th, tr, table {
    border: 0 !important;
    border-spacing:0 !important;
    overflow-x: hidden;
    overflow-y: hidden;
    background-color: unset !important;
    color: unset !important;
  }

tbody > td > tr:hover {
 background-color: unset !important;
 color: unset !important;
 }
</style>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(/Users/skynet/Documents/WVU/Teaching/GitHub.nosync/edp618/static/img/course_hex.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
 let tries = 0
 function addLogo () {
 if (typeof slideshow === 'undefined') {
 tries += 1
 if (tries < 10) {
 setTimeout(addLogo, 100)
 }
 } else {
 document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
 .forEach(function (slide) {
 const logo = document.createElement('a')
 logo.classList = 'xaringan-extra-logo'
 logo.href = 'https://edp618.asocialdatascientist.com'
 slide.appendChild(logo)
 })
 }
 }
 document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(/Users/skynet/Documents/WVU/Teaching/GitHub.nosync/edp618/static/img/course_hex.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
 let tries = 0
 function addLogo () {
 if (typeof slideshow === 'undefined') {
 tries += 1
 if (tries < 10) {
 setTimeout(addLogo, 100)
 }
 } else {
 document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
 .forEach(function (slide) {
 const logo = document.createElement('a')
 logo.classList = 'xaringan-extra-logo'
 logo.href = 'https://edp612.asocialdatascientist.com'
 slide.appendChild(logo)
 })
 }
 }
 document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---

class: highlight-last-item
layout: true

---

# Reliability and Validity 
--

.pull-left[
<center>
Reliability being consistent
</center>
]

.pull-right[
<center>
Validity on target
</center>
]

---

# Recall Scenarios

.pull-left[
<center>
Not Reliable and not Valid
 
<img src="img/image10.png" alt="norelnoval" style="width: 150px;"/>
</center>
]

.pull-right[
<center>
Reliable but not Valid
 
<img src="img/image12.png" alt="relnoval" style="width: 150px;"/>
</center>
]

.pull-left[
<center>
Not Reliable but Valid
 
<img src="img/image13.png" alt="norelval" style="width: 150px;"/>
</center>
]

.pull-right[
<center>
Reliable and Valid
 
<img src="img/image14.png" alt="relval" style="width: 150px;"/>
</center>
]

---

# Basic Tenant

.pull-left[
A test can be Reliable without being Valid
]

.pull-right[
A test cannot be Valid unless it is Reliable
]

---

# Reliability

.pull-left[
<center>
Intra-rater Reliability the degree of agreement between different measurements done by the same person
</center>
]

.pull-right[
<center>
Inter-rater Reliability degree of agreement between between different measurements done by multiple people
</center>
]

---

# Cohen's Kappa `$\kappa$`

+ *Officially*. Measure of the agreement between two raters who each classify `$N$` items into `$C$` mutually exclusive categories

+ *Basic idea*.

+ Quantitative measure of reliability for two raters that are rating the same thing

+ With a correction for how often that the raters may agree by chance
	  
--

+ *Lay terms*. Measure of how well do different people agree

---

## Evaluating

.pull-left[
<center>
 `\kappa < 0` 
random agreement among raters
</center>
]

.pull-right[
<center>
 `\kappa = 1` 
complete agreement among raters
</center>
]

---

## Decision Matrix

.pull-left[
<center>
 `A` 
Agreement 
The total number of instances that both Raters said were correct
</center>
]

.pull-right[
<center>
 `B` 
Disagreement 
The total number of instances that Rater 2 said was incorrect, but Rater 1 said were correct
</center>
]

.pull-left[
<center>
 `C` 
Disagreement 
The total number of instances that Rater 1 said was incorrect, but Rater 2 said were correct
</center>
]

.pull-right[
<center>
 `D` 
Agreement 
The total number of instances that both Raters said were incorrect
</center>
]

---

## Calculations

---

## Probability of Agreement `$P_0$`

<center>
Number in Full Agreement / Total
</center>
.center2[
`$$\dfrac{A+D}{A+B+C+D}$$`
]

---

## Probability of Correct Random Agreement `$P_{correct}$`

<center>
Number Correct in Full or Partial Agreement / Total
</center>
.center2[
`$$\dfrac{A+B}{A+B+C+D}\cdot\dfrac{A+C}{A+B+C+D}$$`
]

---

## Probability of Incorrect Random Agreement `$P_{incorrect}$`

<center>
Number Incorrect in Full or Partial Agreement / Total
</center>
.center2[
`$$\dfrac{C+D}{A+B+C+D}\cdot\dfrac{B+D}{A+B+C+D}$$`
]

---

## Probability of Random Agreement

.center2[
`$$P_e = P_{correct} + P_{incorrect}$$`
]

---

## Kappa `$\kappa$`

.center2[
`$$\kappa = \dfrac{P_0 - P_e}{1-P_e}$$` 
]

---

## Interpretation

.center2[
<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
 <tr>
 <th style="text-align:center;color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Value of `kappa` </th>
 <th style="text-align:left;color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Strength of Agreement </th>
 </tr>
 </thead>
<tbody>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `leq 0.20` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Poor </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.21-0.40` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Fair </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.41-0.60` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Moderate </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.61-0.80` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Good </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `geq 0.81` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Very Good </td>
 </tr>
</tbody>
</table>
]

---

#  That’s It!

Any questions?

<center>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>
</center>