1
2	Medical Studies as Diagnostic Tests Clinical testing of a new treatment or preventive agent is analogous to using laboratory or clinical tests to diagnose a disease Goal is to find a procedure that identifies truly beneficial interventions Not surprisingly, the issues that arise when screening for disease apply to clinical trials Predictive value of a positive test is best when prevalence is high Use screening trials to increase prevalence of beneficial treatments
3	Medical Studies as Diagnostic Tests Statistical hypothesis testing as a diagnostic test P value: Probability of observing positive (statistically significant) test in absence of true treatment effect Level of significance is 1 - specificity Common choice of a=.05 means specificity is 95%
4	Medical Studies as Diagnostic Tests Statistical hypothesis testing as a diagnostic test (cont.) Statistical power: Probability of observing positive test in presence of true treatment effect Power is sensitivity Common choice of 80% sensitivity (not usually recommended by me)
5	Medical Studies as Diagnostic Tests Statistical hypothesis testing as a diagnostic test (cont.) Prevalence is the percentage of effective treatments among all tested treatments Positive predictive value is the probability that a statistically significant trial indicates a truly useful treatment
6	Preliminary Studies in Screening In cancer less than 5% of treatments studied in clinical trials are adopted NCI drug development program 1970 - 1985 350,000 unique chemical structures studied 83 pass preclinical and phase I testing 24 pass phase II tests for biological activity
7	Preliminary Studies in Screening Two possible approaches to studying new treatments Study every treatment in a large definitive experiment Perform small screening trials, with confirmatory trials of promising treatments passing early tests We can explore our ability to identify beneficial treatments with limited resources
8	Preliminary Studies in Screening Scenario 1: Only large trials 10% of drugs being investigated truly work Level of significance .05 1000 subjects provide 97.5% power to detect clinically important treatment effect 1,000,000 subjects available for clinical trials Study 1,000 new treatments 100 effective treatments, 900 ineffective treatments
9	Preliminary Studies in Screening Scenario 1: Only large trials (cont.) Statistically significant results: 143 significant trials 97.5% of effective treatments: 98 studies significant 5% of ineffective treatments: 45 studies significant Predictive value of a positive: 68% Only 68% of the 143 treatments identified truly work
10	Preliminary Studies in Screening Scenario 2: Use of pilot studies 10% of drugs being investigated truly work Level of significance .05 500 subjects provide 80% power to detect clinically important treatment effect 50 subjects provide 15% power to detect clinically important treatment effect
11	Preliminary Studies in Screening Scenario 2: Use of pilot studies (cont.) 1,000,000 subjects available for clinical trials 625,000 subjects in pilot studies of 12,500 new treatments 374,500 subjects in confirmatory trials of 749 new treatments
12	Preliminary Studies in Screening Scenario 2: Use of pilot studies (cont.) Pilot Studies Investigate 12,500 new treatments in pilot studies (625,000 subjects) 1,250 effective treatments, 11,250 ineffective treatments
13	Preliminary Studies in Screening Scenario 2: Use of pilot studies (cont.) Statistically significant results: 749 significant pilot studies 15% of effective treatments: 187 studies significant 5% of ineffective treatments: 562 studies significant Predictive value of a positive: 25% 25% of treatments in significant pilot studies truly work
14	Preliminary Studies in Screening Scenario 2: Use of pilot studies (cont.)} Confirmatory Trials Investigate 749 new treatments (374,500 subjects) 187 effective treatments, 562 ineffective treatments
15	Preliminary Studies in Screening Scenario 2: Use of pilot studies (cont.)} Statistically significant results: 178 significant pilot studies 80% of effective treatments: 150 studies significant 5% of ineffective treatments: 28 studies significant Predictive value of a positive: 84% 84% of the 178 identified treatments truly work
16	Preliminary Studies in Screening Comparison of scenarios Scenario 1: Only large trials Use 1,000,000 subjects Screen 1,000 new treatments Adopt 98 effective treatments Adopt 45 ineffective treatments Scenario 2: Use of pilot studies Use 999,500 subjects Screen 12,500 new treatments Adopt 150 effective treatments Adopt 28 ineffective treatments
17	Preliminary Studies in Screening Bottom line Pilot studies increase the predictive value of a positive study while using the same number of subjects. A greater number of effective treatments are identified due in part to the greater number of treatments screened. Phases of clinical trials (Different choices for statistical power in screening and confirmatory trials can be used to optimize strategy for a particular setting)