Health research often measures one thing two ways: two sets of diagnostic criteria for one disease, an observational study and a Mendelian randomization study of one drug target, or pooled hospital summaries and individual patient records for one risk factor. The answers often disagree, and the disagreement is usually reported as a single statistic, such as Cohen’s kappa, an odds ratio, or a pooled estimate, that does not show which patients the answers differ on, or whether they estimate the same quantity.
History
In measurement theory, a construct is measured validly when independent methods aimed at it converge. Campbell and Fiske (1959) made this testable with the multitrait–multimethod matrix: measures of one trait obtained by different methods should agree, which is convergent validity, and measures of different traits should not, which is discriminant validity. Webb and colleagues (1966) named the practice of measuring one thing by several independent methods triangulation.
Philosophers of science treat the same idea as robustness. Wimsatt (1981) argued that a result reached by several independent means is more reliable than one reached by any single means, and Hacking (1981) argued that we trust what a microscope shows because microscopes built on different physical principles show the same structures.
Epidemiology adopted the term for study designs. Lawlor, Tilling and Davey Smith (2016) set out triangulation as the comparison of study designs with different and unrelated sources of bias, and Munafò and Davey Smith (2018) argued for it as general practice in empirical research.
Contributions
Standard triangulation assumes the methods target the same quantity, so disagreement means one of them is biased. This program shows that disagreement often means the methods target different quantities, and that neither need be wrong.
In epidemiology, triangulation compares study designs that have different sources of bias. This program applies the same test to diagnostic criteria.
Agreement Methods. Cohen’s kappa alone does not show whether switching criteria only adds patients or also drops some, and the 2×2 table behind it can be recovered from the figures a study already prints.
Diagnostic Boundaries. The same question applied to single diseases: multiple sclerosis, where revisions of the McDonald criteria move patients in and out of the diagnosis; Alzheimer’s disease, where the two 2024 criteria place the same person on opposite sides of the boundary; and psychiatric classification, where four programs disagree about what makes two presentations the same disorder.
Cross-Design Evidence. For drug targets, observational and genetic evidence set against Phase III outcomes: a preregistered rule classifies 24 of 32 mechanisms correctly, and the observational evidence adds nothing beyond the genetic evidence.
Ecological Inference. For pooled data, an estimate built from hospital summaries against the patient-level effect in the same 42 million COVID-19 records: the two are different quantities.
Sources
- Campbell DT, Fiske DW (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin 56(2):81–105. DOI 10.1037/h0046016. APA PsycNet
- Webb EJ, Campbell DT, Schwartz RD, Sechrest L (1966). Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally. Revised edition, SAGE (2000). DOI 10.4135/9781452243443. SAGE
- Wimsatt WC (1981). Robustness, reliability, and overdetermination. In Brewer MB, Collins BE (eds), Scientific Inquiry and the Social Sciences. San Francisco: Jossey-Bass. Reprinted in Characterizing the Robustness of Science (2012), 61–87. DOI 10.1007/978-94-007-2759-5_2. Springer
- Hacking I (1981). Do we see through a microscope? Pacific Philosophical Quarterly 62(4):305–322. DOI 10.1111/j.1468-0114.1981.tb00070.x. Wiley
- Lawlor DA, Tilling K, Davey Smith G (2016). Triangulation in aetiological epidemiology. International Journal of Epidemiology 45(6):1866–1886. DOI 10.1093/ije/dyw314. PMC
- Munafò MR, Davey Smith G (2018). Robust research needs many lines of evidence. Nature 553:399–401. DOI 10.1038/d41586-018-01023-3. Nature