Assessment Test 1 - Part 1
Terms in this set (16)
Represent the magnitudeof a treatment effect
-reminds us not to confuse stat sig with importance
- standardized mean difference for group comparisons (Cohen's d)
- variance accounted for in correlational studies (R^2)
- odds of a binary outcome in a different group (odds ration)
**when comparing groups that received different treatments - Cohen's d is good
Cohen's d (standard mean difference)
- comparable to z-score from NR test; this expresses person's score in SD units
- Cohen's expresses size of difference btwn mean scores of diff groups in SD units
- subtract the mean scored of 2 groups
- divide result by the average of their SDs
d = (Mg1 - Mg2) / ([SDg1 + SDg2]/2)
-small = 0.2
-medium = 0.5
-large = 0.8 (almost 1 SD diff)
Precision of a finding - degree of error in the measurement value
- larger the CI the less precise
The CI is the range of values around a point estimate within which the true value would be expected to fall a specified percentage of the time (usually 95%), if the study could be repeated 100 times
When measurement error is low and samples are large, the CI is narrow and the finding is relatively precise
small sample size = wide CI
Critical Appraisal of a group comparison Treatment ( CATE)
- evaluator, date, evidence source
- foreground question (PICO format)
- specific appraisal points to evaluate
-overall judgements about validity and importance of the evidence
- Clinocal bottom line (Is the evidence so valid and so important that a change to current clinical practice should be considered? )
CATE Appraisal points
1. plausible rationale for treatment?
2. Was the study experimental?
3. Was there a control group? or control condition?
4. Was randomization used to create the contrasting group?
5. was the study prospective (not retro)?
6. Were the participants representative and/or recognizable?
7. were the treatments described clearly and implemented as intended?
8. were treatment effects measured with measures that were valid and reliable?
- need evidence of face validity adn inter-examiner reliability
- NR arent usually appropriate
9. were treatment outcomes measured with blinding? - Were measurements made by examiners who did NOT know which group the participant had been in?
10. were there any nuisance variables?
11. was there a statistically significant difference between groups (p </= 0.05)
12. if they did not differ significantly, was there enough stat power (>/= 0.8) to enable confidence in conclusion of no difference - if no power reported, having large sample size can increase confidence that no "Real difference" has been missed
13. was the finding (treatment effect) important?
-What was the effect size?
-Was there evidence of social validity, i.e., would naïve, non-experts notice the treatment effect?
-Was there evidence that the treatment effect was maintained over time, i.e., was it durable?
14. was the finding precise?
- narrow CI
15. was there a substantial cost-benefit advantage for the Tx?
- benefits exceeded harms?
- benefits exceeded costs?
based on 15 points - validity and importance
1. Compelling - confident that unbiased expert would agree that evidence is valid/important
2. suggestive - room for debate; but unbiased experts would prob agree that the evidence is valid / important
3. equivocal - evidence from the study can be questioned in so many ways that its validity/importance are suspect
Validity of Evidence - Compelling
- strong research design
- representative participants
- reliable measurement tools
- controls for subjective bias
-no nuisance variables
- low p-value (stat sig)
- sufficient power
*chance of inferential mistake is very low
Validity of importance - Suggestive
- some validity features are strong, but flaws that cast doubts on findings
- experts could disagree depending on how important they think the features are
Validity of Importance - Equivocal
- study has so many flaws that its impossible to trust
- Experts could reach opposite conclusions based on their own subjective biases
Importance of Evidence - COmpelling
- large effect size
- narrow CI
- evidence that effect is socially valid and durable over time
Importance of Evidence - Suggestive
- medium effect size
- broader CI
- evidence concerning social validity and/or maintenance is not provided
Importance of Evidence - Eqivocal
- trivial effect size
- CI is so wide that result is not useful
Clinical bottom line
- do i change my treatment (with input from E2 and E3)
Quality Cues that do not apply to N-of-1
representative sample (well described clients)
Stat Sig Diff (stable baselines, replicated effects)
adequate stat power (N/A)
Precision (narrow CIs) - N/A but replication over clients
Percentage of Non-Overlapping Data (PND)
ES metric for N-of-1
- simpler; less reliable than d
1. Find the highest point in the baseline phase
2. Find all of the points in the treatment phase that are higher than this (i.e., do not overlap with it).
3. Divide the number of non-overlapping points in the treatment phase by the total number of points in the treatment phase
>/= 90% -Treatment highly effective
70-89% - Treatment moderately effective
50-69% - Treatment minimally effective
< 50% - Treatment ineffective
Replication increases external validity of E2
An N-of-1trial involving only one client can provide strong E2 about him or her, but we can't assume that the results have external validity (i.e., would generalize to clients other than the one studied)
Doing similar trials on other clients, and finding similar effects, can increase confidence in this evidence
- at least 4 different clients = minimum for boosting the confidence of N-of-1