Search
Create
Log in
Sign up
Log in
Sign up
Get ahead with a $300 test prep scholarship
| Enter to win by Tuesday 9/24
Learn more
Lecture 12: Reliability and Validity
STUDY
Flashcards
Learn
Write
Spell
Test
PLAY
Match
Gravity
Terms in this set (44)
Outline
1. quality control and quality assurance
2. Indices of reliability and validity
3. reliability
a. test-retest reliability
b. inter-rater reliability
c. internal consistency
4. Validity
a. content, criterion, construct validity
b. internal and external validity
5. Quantitative Bias analysis
6. missing data
Quality assurance (QA)
activities to assure quality of data that take place prior to data collection (protocol and manuals)
quality control (QC)
efforts during study to monitor quality of data (collection, processing)
Reliability
1. quality of measurement (consistency)
a. the more error, less reliable
Reliability studies
1. assess extent to which results agree, different or same observers/instruments
Indices of validity and reliability
Cat
1. sens/spec (only validity)
2. Youden's J (only val)
3. percent agreement
4. kappa
Continuous
1. scatter plot (cor)
2. pearson / spearman's cor
3. intracalss cor coef
4. coeffficient of variation
5. bland altman plot
Sens and Spec
1. indices of validity
2. E and O are categorical
Youden's J statistic
1. combines sens and spec; assumes equal weight
2 . J = sens+spec - 1
J = 1; perfect agreement
J= 0; test performs no better than chance
Percent agreement
1. True P and N / N * 100%
percent positive agreement
1. a / (b+c+d) * 100%
Cohen's kappa
1. proportion observed agreement - prop. of chance agreement / 1 - Prop chance agree
2. dependent on prevalence (higher prev, higher kappa)
kappa statistic
steps
1. test 1
chance agree (a+c * a+b)/N
chance not agree
2. sum of test 1 kappas / N
3. percent agree - step 2 . (1-step2)
indices, continuous, linear correlation
1. correlation plot
2. correlation coef
3. linear regression
indices, continuous, pair-wise comparison
1. mean difference
2. coefficient of variation
3. bland altman
correlation/scatterplot
1. data points on the line; both points in agreement
linear correlation coefficient
1. equally high when both observers read same values (doesnt have to be on the line)
2. very sensitive to outliers
limitations of pearsons r (spearman = nonpara)
1. linear assoc., not necessairly agreement
2. r is sensitive to range of values/outliers
3. Ho: r = 0
linear regression
1. detect systematic differences (slope and intercepts)
mean difference
1. differences in mean between groups and can do a t test
coefficient of variablity
1. lower coefficient of variability, less variation between replicate measurement
bland altman
1. difference between two groups plotted against the mean of two groups
2. middle line is mean +/- 95% CI
intraclass correlation coefficient
1. variance between individuals / total variance
2. if = 1 = high agreement between measurements
cronbach's alpha
1. average inter-correlation among items
2. increase # of items, increase alpha
asessing reliability
1. test-retest
2. inter-rater
3. internal consistency
test-retest reliability
1. consistency of participant's response over time
a. repeat measurements
2. estimate by correlation
inter rater reliability
1. agreement b/t different interviewers on same subjects/responses
2. estimation based on correlation scores
3. measured through
a. cohen's kappa
b. Intraclass correlation (ICC)
cohen's kappa for interrater reliabiltiy
1. for 2 raters
intraclass correlation (ICC)
1. two or more raters
2. ratio between groups variance and total variance
internal consistency reliability
1. ability to produce similar results using different samples
2. uses cronbach's
Validity
1. degree to which variability in measure reflects true measure
2. a measure cannot be valid unless reliable, but reliable might not be valid
3. types
a. content validity
b. criterion validity
c. construct validity
content validity
1. whether measurements chosen are repr.
criterion validity
1. success of measures used for estimation
a. concurrent: degree to which correlates w/ other measures of same parameter
b. predictive validity: degree to which predicts with other measures of parameter in future
construct validity
1. degree to which inferences can be made
a. whether measure relates to parameter being measured
2. types
a. convergent validity: measure correlates with other measures it is predicted to correlate with
b. discriminant validity: degree to which not similar to what it should not be similar with
internal validity
1. causal link b/t concepts and established study?
external validity
1. generalizability to populations
limitation of validity studies
1. gold standard might not be 100% valid
2. bias from collecting data
quantitative bias analysis (approaches to address)
1. quantify error
2. adjust for
a. selection bias
b. unmeasured confounders
c. information bias
3. conduct sensitivity analysis
missing data
1. types
a. MCAR
b. MAR
c. MNAR
2. traditional approaches
a. deletion
b.substitution
3. modern approaches
a. imputation
b. maximum likelihood and bayes
MCAR
1. missing completely at random
2. prob. xi is missing doesnt depend on value or value of other variables
3. dont have to address
MAR
1. missing at random
2. doesnt depend on xi after controlling for other variable
MNAR
missing not at random... BAD
deletion of records (traditional approach)
1. list-wise (complete case set)
a. if MCAR, then unbiased, less power
2. pair-wise
a. leads to different sample sizes for different parts of analysis
Single imputation (traditional approach)
1. mean substitution
2. regression substitution
multiple methods (modern appraoch)
1. multiple datasets created, differ yb imputed values
2. datesets anlysis combined and pool estimates
3. can be done with markov monte carlo
;