Introduction

• We have discussed how to do many analyses with continuous outcomes
• Also discussed regression with categorical outcomes
• What about analyses with all categorical outcomes?

Contingency Tables

• Def
  • Table containing multivariate frequency distributions of variables
  • Can be 2+ dimensions
  • Ex.

Research Questions

• Examples of questions one can ask with such data
  1. Are two or more categorical variables independent?
  2. Are two or raters similar for a given measure (i.e. inter-rater reliability)

Testing for independence

• Def
  • Let \(f_x\) denote the distribution of \(X\), \(f_y\) distribution of \(Y\)
  • Let \(f_{xy}\) denote the joint distribution of \(X, Y\)
  • Random variables \(X\) and \(Y\) are independent \(\leftrightarrow\) \(f_{xy}=f_x*f_y\)
  • This is equivalent to \(f_{y|x}=f_y\) and \(f_{x|y}=f_x\)

##          
##           Female Male Sum
##   NO_ASD     214  275 489
##   YES_ASD     21   77  98
##   Sum        235  352 587

##          
##               Female       Male        Sum
##   NO_ASD  0.36456559 0.46848382 0.83304940
##   YES_ASD 0.03577513 0.13117547 0.16695060
##   Sum     0.40034072 0.59965928 1.00000000

Testing for independence

• Null: \(X\) and \(Y\) are independent
• Alt.: \(X\) and \(Y\) are dependent
• Methods
  1. Chi-square test
  2. Fisher’s Exact Test

Chi-square test

• Test statistic: \(\chi^2=\sum_{i} (O_i-E_i)^2/E_i\)
  • \(O_i=\) observed count in cell \(i\)
  • \(E_i=\) expected count in cell \(i\) under null/independence
  • Large deviation \(\rightarrow\) large test statistic \(\rightarrow\) low p-value
  • Under null:
    • As \(n \rightarrow \infty\), \(\chi^2 \rightarrow\) chi-square distribution
    • Degrees of freedom = \((R-1)*(C-1)\) (\(R=\#\) of rows, \(R=\#\) of columns)

## Call: xtabs(formula = ~SSM_ASD_v24 + Gender, data = ibis_data)
## Number of cases in table: 587 
## Number of factors: 2 
## Test for independence of all factors:
##  Chisq = 16.963, df = 1, p-value = 3.811e-05

Fisher’s Exact Test

• What if \(n\) is small?
  • \(\rightarrow\) chi-square distribution approx is poor
  • Need alternative test
• Exact test has no ‘’test statistic’’, only a p-value/confidence interval
• P-value is accurate no matter sample size
  • However, more computationally intensive
  • Use chi-square test if sample size is large

## 
##  Fisher's Exact Test for Count Data
## 
## data:  table_xtabs
## p-value = 2.614e-05
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  1.675426 5.024610
## sample estimates:
## odds ratio 
##   2.848825

More then two variables

• What if we have more then 2 count variables?
  • Ex. may want to assess association between 2 count variables, controlling for third
  • I.e., want to stratify by 3rd variable, see if you can reject null

Mantel-Haenszel Test

• Null: Odds Ratio(\(X,Y\))=1 for all values of \(Z=z\)
• Alt.: At least one \(\neq 1\)
• Test statistic: \(\chi^2_{MH}\)
  • Under null:
    • As \(n \rightarrow \infty\), \(\chi^2 \rightarrow\) chi-square distribution
    • Degrees of freedom = 1
    • Valid for series of 2 by 2 tables

## , , Study_Site = PHI
## 
##            Gender
## SSM_ASD_v24 Female Male
##     NO_ASD      52   70
##     YES_ASD      3   24
## 
## , , Study_Site = SEA
## 
##            Gender
## SSM_ASD_v24 Female Male
##     NO_ASD      57   65
##     YES_ASD     10   20
## 
## , , Study_Site = STL
## 
##            Gender
## SSM_ASD_v24 Female Male
##     NO_ASD      55   71
##     YES_ASD      5   14
## 
## , , Study_Site = UNC
## 
##            Gender
## SSM_ASD_v24 Female Male
##     NO_ASD      50   69
##     YES_ASD      3   19

## Call: xtabs(formula = ~SSM_ASD_v24 + Gender + Study_Site, data = ibis_data)
## Number of cases in table: 587 
## Number of factors: 3 
## Test for independence of all factors:
##  Chisq = 24.145, df = 10, p-value = 0.007225

Ordinal categories

• Above all assumed categories had no explicit ordering
• Def: an ordinal variable is a categorical variable where categories have inherent ordering
• Can inform analyses, improve statistical power
• Ex.
  1. ‘’linear-by-linear’’ test
  2. Cochran-Armitage test (one ordinal, one nominal)

Ordinal regression

• Can also use regression models to analyze an ordinal outcome variable
• Recall: Logistic Regression

\[ \begin{align} &\text{logit}(\frac{p_x}{1-p_x})=\beta_0+\beta_1X_1+\ldots+\beta_pX_p \\ & p_x=\Pr(Y=1|X_1=x_1, \ldots, X_p=x_p) \end{align} \]

• Ordinal Logistic Regression

\[ \begin{align} &\text{logit}\bigg(\frac{p_x(j)}{1-p_x(j)}\bigg)=\beta_{0j}+\beta_{1j}X_1+\ldots+\beta_{pj}X_p \\ & p_x(j)=\Pr(Y\leq j|X_1=x_1, \ldots, X_p=x_p) \end{align} \]

Ordinal regression

• Predictors can be of any form (continuous, categorical, etc.)
• Many contingency table tests for association are equivalent to corresponding regreesion model
• Issue: Number of parameters in model can be large (as \(\beta\) function of \(j\))
  • Solution: Proportional odds model

\[ \begin{align} &\text{logit}\bigg(\frac{p_x(j)}{1-p_x(j)}\bigg)=\beta_{0j}+\beta_{1}X_1+\ldots+\beta_{p}X_p \\ & p_x(j)=\Pr(Y\leq j|X_1=x_1, \ldots, X_p=x_p) \end{align} \]

• Can test for deviation from proportion odds in R (and other software packages)

Inter-rater reliability

• Suppose you have a measurement, but want to validate it
• One concern: do two people grade the same data similarly using your measurement
• Can visualize the raters’ data using a contingency table

Quantifying Aggreement

• Common metric: Cohen’s \(\kappa\)
  • \(P_0\) = proportion of chance agreement
  • \(P_E\) = percentage of agreement expected by random chance
  • Ex. Suppose measurement can be yes or no
    • Say rater one says “yes” \(50\%\) of the time, rate two \(25\%\) of the time
    • Say rater one says “no” \(25\%\) of the time, rate two \(50\%\) of the time
    • Then under independence, rater would agree \(0.5*0.25+0.25*0.5=25\%\) of the time
    • so \(P_E=0.25\)
• \(\kappa=\frac{P_0-P_E}{1-P_0}\)

##                       Doctor2
## Doctor1                Depression Personality Disorder Schizophrenia Neurosis
##   Depression                    7                    1             2        3
##   Personality Disorder          0                    8             1        1
##   Schizophrenia                 0                    0             2        0
##   Neurosis                      0                    0             0        1
##   Other                         0                    0             0        0
##                       Doctor2
## Doctor1                Other
##   Depression               0
##   Personality Disorder     0
##   Schizophrenia            0
##   Neurosis                 0
##   Other                    4

##             value     ASE     z  Pr(>|z|)
## Unweighted 0.6512 0.09968 6.532 6.474e-11
## Weighted   0.6331 0.11939 5.303 1.140e-07

##             
## Kappa              lwr       upr
##   Unweighted 0.4557884 0.8465372
##   Weighted   0.3991025 0.8670846

Quantifying Aggreement

• Extensions
  1. What if more then 2 raters? Light’s \(\kappa\) (averages all pairwise agreement)
  2. What if variables are ordinal/levels no arbitrary? Weighted \(\kappa\)
• Note: Null is rater agree simply by random chance (or equivalent)
• Other common measure: intraclass correlation (ICC)
  • icc function in psy or irr packages

##                     rater1                  rater2                  rater3
## 1              4. Neurosis             4. Neurosis             4. Neurosis
## 2  2. Personality Disorder 2. Personality Disorder 2. Personality Disorder
## 3  2. Personality Disorder        3. Schizophrenia        3. Schizophrenia
## 4                 5. Other                5. Other                5. Other
## 5  2. Personality Disorder 2. Personality Disorder 2. Personality Disorder
## 6            1. Depression           1. Depression        3. Schizophrenia
## 7         3. Schizophrenia        3. Schizophrenia        3. Schizophrenia
## 8            1. Depression           1. Depression        3. Schizophrenia
## 9            1. Depression           1. Depression             4. Neurosis
## 10                5. Other                5. Other                5. Other
## 11           1. Depression             4. Neurosis             4. Neurosis
## 12           1. Depression 2. Personality Disorder             4. Neurosis
## 13 2. Personality Disorder 2. Personality Disorder 2. Personality Disorder
## 14           1. Depression             4. Neurosis             4. Neurosis
## 15 2. Personality Disorder 2. Personality Disorder             4. Neurosis
## 16        3. Schizophrenia        3. Schizophrenia        3. Schizophrenia
## 17           1. Depression           1. Depression           1. Depression
## 18           1. Depression           1. Depression           1. Depression
## 19 2. Personality Disorder 2. Personality Disorder             4. Neurosis
## 20           1. Depression        3. Schizophrenia        3. Schizophrenia
## 21                5. Other                5. Other                5. Other
## 22 2. Personality Disorder             4. Neurosis             4. Neurosis
## 23 2. Personality Disorder 2. Personality Disorder             4. Neurosis
## 24           1. Depression           1. Depression             4. Neurosis
## 25           1. Depression             4. Neurosis             4. Neurosis
## 26 2. Personality Disorder 2. Personality Disorder 2. Personality Disorder
## 27           1. Depression           1. Depression           1. Depression
## 28 2. Personality Disorder 2. Personality Disorder             4. Neurosis
## 29           1. Depression        3. Schizophrenia        3. Schizophrenia
## 30                5. Other                5. Other                5. Other
##                     rater4                  rater5                  rater6
## 1              4. Neurosis             4. Neurosis             4. Neurosis
## 2                 5. Other                5. Other                5. Other
## 3         3. Schizophrenia        3. Schizophrenia                5. Other
## 4                 5. Other                5. Other                5. Other
## 5              4. Neurosis             4. Neurosis             4. Neurosis
## 6         3. Schizophrenia        3. Schizophrenia        3. Schizophrenia
## 7         3. Schizophrenia                5. Other                5. Other
## 8         3. Schizophrenia        3. Schizophrenia             4. Neurosis
## 9              4. Neurosis             4. Neurosis             4. Neurosis
## 10                5. Other                5. Other                5. Other
## 11             4. Neurosis             4. Neurosis             4. Neurosis
## 12             4. Neurosis             4. Neurosis             4. Neurosis
## 13        3. Schizophrenia        3. Schizophrenia        3. Schizophrenia
## 14             4. Neurosis             4. Neurosis             4. Neurosis
## 15             4. Neurosis             4. Neurosis                5. Other
## 16        3. Schizophrenia        3. Schizophrenia                5. Other
## 17             4. Neurosis                5. Other                5. Other
## 18           1. Depression           1. Depression 2. Personality Disorder
## 19             4. Neurosis             4. Neurosis             4. Neurosis
## 20                5. Other                5. Other                5. Other
## 21                5. Other                5. Other                5. Other
## 22             4. Neurosis             4. Neurosis             4. Neurosis
## 23                5. Other                5. Other                5. Other
## 24             4. Neurosis             4. Neurosis             4. Neurosis
## 25             4. Neurosis             4. Neurosis                5. Other
## 26 2. Personality Disorder 2. Personality Disorder             4. Neurosis
## 27           1. Depression                5. Other                5. Other
## 28             4. Neurosis             4. Neurosis             4. Neurosis
## 29        3. Schizophrenia        3. Schizophrenia        3. Schizophrenia
## 30                5. Other                5. Other                5. Other

##  Light's Kappa for m Raters
## 
##  Subjects = 30 
##    Raters = 6 
##     Kappa = 0.459 
## 
##         z = 2.31 
##   p-value = 0.0211