EPPP Research Methods and Statistics: Errors, Power, Effect Size and Validity Threats in the 7% Area

What the EPPP research methods area covers

The smallest content area on EPPP Part 1 is also one of the hardest. ASPPB's current content areas give Research methods and statistics 7%, through eight knowledge statements:

In ASPPB's 2026 doctoral program report, accredited-program candidates averaged 65.96% correct here in 2021–2025, just above lifespan development and second-lowest of the eight areas. Because the area is small, about 12 scored items, the average candidate loses only about 4 of them, as the EPPP difficulty guide works out. In Passy's view, the concepts are tightly defined, so focused work pays off quickly here.

Type I errors, Type II errors and power

A 2009 article in the Industrial Psychiatry Journal defines the two errors against the null hypothesis:

The points to know:

• Alpha is set in advance. A study designed with alpha = .05 accepts at most a 5% chance of incorrectly rejecting the null.
• Power is 1 − beta. It's the probability of detecting an effect of a specified size or greater if it exists in the population.
• Sample size reduces both errors. The article says the likelihood of both errors can be reduced by increasing sample size.
• Effect size matters. The ability to detect an association depends on how large it actually is.
• One-tailed tests need smaller samples, because they specify the direction of the effect in advance. Switching to a one-tailed test after seeing the data, to shrink the p value, is described in the same article as a lack of scientific integrity.

A quick way to keep them straight: a Type I error "sees" an effect that isn't there, and a Type II error misses one that is. Low power raises the risk of the second.

Multiple comparisons and the Bonferroni correction

Alpha applies to a single comparison. Run many, and the chance of at least one false positive climbs. A 2016 article in Perspectives in Clinical Research gives the numbers: comparing two groups 5 times at the 5% level makes the probability of a false positive as high as 23%, and 20 comparisons raise it to 64%. The arithmetic behind that is 1 − (1 − .05) raised to the number of comparisons: 1 − .95⁵ ≈ .23.

The simplest fix, the Bonferroni correction, divides the overall alpha by the number of comparisons: with 20 comparisons, each is tested at .05 ÷ 20 = .0025. The article lists other approaches too, such as Tukey, Hochberg and Holm's step-down methods. Practice computing the corrected alpha and spotting a study whose many unplanned tests call for one. Full EPPP prep planned for launch has explanations planned to work familywise-error items step by step, the same calculation as above.

Effect size and statistical vs clinical significance

KN56 pairs power with effect size and with clinical vs statistical significance. A 2013 primer in Frontiers in Psychology explains why: standardized effect sizes communicate the practical significance of results, instead of only their statistical significance.

• Cohen's d expresses a difference between means in standard deviation units. The commonly used interpretation, from benchmarks Cohen suggested in 1988, is small at 0.2, medium at 0.5 and large at 0.8.
• Use benchmarks with care. The primer calls the values arbitrary and not to be read rigidly, notes Cohen's view that the only reason to use them is a finding so novel it can't be compared with related research, and points out that small effects can have large consequences, such as a reliable drop in suicide rates at d = 0.1.

In practice: because larger samples detect smaller effects, a very large sample can make a small effect statistically significant, so a significant p value doesn't by itself tell you whether a change matters clinically. The effect size, and the size of change that matters in that setting, answer that question. The 1,000-person tables and SEM bands in the assessment and diagnosis guide are the measurement side of the same idea.

Internal, external and ecological validity

A 2018 article in the Indian Journal of Psychological Medicine separates three questions:

• Internal validity: do the design, conduct and analysis answer the research question without bias? Improper randomization, unblinding and missing data can compromise it.
• External validity: can the findings be generalized to other contexts, such as other patients or longer follow-up?
• Ecological validity: a subtype of external validity, asking whether findings generalize to real-life settings.

The classic internal-validity threats come from the Campbell tradition of research design. A 2020 catalog in Epidemiology lists nine: ambiguous temporal precedence, selection, history, maturation, regression to the mean, testing, instrumentation, attrition, and additive and interactive effects of these. It notes that history, maturation, regression, testing and instrumentation are particularly relevant to pre–post designs with no comparison group. Two examples from the catalog: maturation is change from the natural course of the outcome over time, and testing is when the act of measuring affects the outcome, as when being weighed motivates weight loss.

Random sampling vs random assignment

These two are easy to swap under time pressure:

• Random sampling is about who gets into the study. The 2018 article says that if sampling is random, the sample represents the population, so results can be generalized to that population. It serves external validity.
• Random assignment is about who gets which condition. It's the randomization in a randomized controlled trial, and the same article lists improper randomization among threats to internal validity.

A study can have one without the other. Without random assignment, groups can differ before the study starts, which is the selection threat; the Epidemiology catalog calls selection a core threat. Practice telling these apart with 95 research methods questions planned for EPPP prep, which full prep will include once it launches.

Single-case experimental designs

KN54 lists case designs among the study types. A 2024 review in the European Journal of Physical and Rehabilitation Medicine describes the basic single-case experimental designs:

• AB: a baseline phase (A) and an intervention phase (B). It's non-experimental on its own.
• Reversal or withdrawal (ABAB): baseline, intervention, withdrawal of the intervention, and reintroduction. The review notes these designs require outcomes that can reverse.
• Multiple baseline: the intervention is introduced in a time-lagged way across participants, or across settings or behaviors if they're independent of one another. The review's example uses three participants.
• Alternating treatments: rapid alternation between two or more interventions, comparing their effects on the same outcome.

In Passy's reading, that reversibility requirement is a useful key to design questions: when the outcome is a skill that won't return to baseline once learned, a multiple-baseline design fits better than ABAB.

Two Passy-written research methods items

These items were written by Passy for practice. They aren't ASPPB items.

Item 1. A researcher plans 10 independent t-tests and wants to keep the overall Type I error rate at .05 using the Bonferroni correction. What alpha should each test use?

• A. .005
• B. .05
• C. .50
• D. .0005

Keyed answer: A. The Bonferroni correction divides the overall alpha by the number of comparisons: .05 ÷ 10 = .005. B leaves each test at .05, so the chance of at least one false positive across 10 tests rises to about 40% (1 − .95¹⁰). C multiplies instead of dividing, and D divides by 100.

Item 2. A clinic measures anxiety before and after a 10-week group program, with no comparison group, and scores improve. Which alternative explanation does this design leave most open?

• A. Change over time that would have happened anyway
• B. Pre-existing differences between treatment groups
• C. Too many statistical tests inflating alpha
• D. Poor generalizability of the test's norms

Keyed answer: A. Without a comparison group, history, maturation and regression to the mean can't be separated from the program's effect, which is why the Epidemiology catalog flags these threats for pre–post designs with no comparison. B describes selection between groups, but there's only one group. C needs many tests, and D concerns external validity of the measure, not the cause of the change.

Studying EPPP research methods in a week

• Day 1: errors, power and the decision table until you can draw it from memory.
• Day 2: familywise error and Bonferroni, with five quick calculations.
• Day 3: effect sizes, and why significance isn't importance.
• Day 4: the nine internal-validity threats, each paired with a one-line example.
• Day 5: sampling vs assignment, designs and single-case variations.
• Days 6 and 7: mixed questions and review of every miss.

That fits the EPPP study plan, which gives this area about 9% of content time, a little above its 7% weight. Check your starting point with the 20 free EPPP practice questions, and for the definitions you'll want to recall in seconds, 41 research methods flashcards planned for full prep are sized for short daily review; checkout isn't open yet.