Virtual StatsPD@Waite meeting

Every month, the professional development meetings of statisticians and data scientists at Waite, known as StatsPD@Waite, bring together specialists in various aspects of data sciences in agriculture from Waite, Roseworthy and Adelaide.

Please join us for the next StatsPD@Waite seminar where Max Moldovan from the Biometry Hub, University of Adelaide will present on an imitated p-hacking with TMLE and the Super Learner: the methods fusion in question.

Also please note that the StatsPD@Waite meetings are recorded. If you have a question to the speaker but had rather not be recorded, please send your question via chat during the meeting and it will be asked on your behalf.

Please email Paul Eckermann with questions or for details of the Zoom meeting.

Title: An imitated p-hacking with TMLE and the Super Learner: the methods fusion in question

Presenter: Max Moldovan, Biometry Hub, University of Adelaide

Causal inference is in the centre of many applied studies. Causal inference from observational studies is of a particular challenge due to the potential of the suspected but largely unknown confounding mechanism. Targeted maximum likelihood estimation (TMLE) is one of the most prominent and widely-applied methods for estimating a causal effect from observational data (van der Laan and Rose, 2011).

TMLE is a semiparametric method that provides an optimal trade-off between bias and variance in the estimation of the target parameter, and is both double-robust (i.e., an estimate is consistent as long as either an outcome or treatment model is correctly specified) and data-adaptive (i.e., incorporating various statistical and machine learning methods), accounting for a non-random assignment of a treatment in observational studies. The data-adaptive property of the method is largely obtained from the combination of TMLE with an algorithm known as a Super Learner (SL) – a cross-validation based ensemble learning approach for parameter estimation. In practical sense, the TMLE method is a fusion of TMLE and the SL methods.

We have designed and implemented a numerical study, initially aiming to evaluate a variability of causal parameter estimates due to: 1) the choice of a seed passed to a pseudo-random number generator within the SL cross-validation routine; and 2) a composition of base learners within a SL ensembler. The results were obtained from the TMLE application using the popular R tmle package (Gruber and van der Laan, 2012). We demonstrate that pursuing the data-adaptive property through combination of TMLE and the SL can lead to a (partially) systematic but yet unexplained bias in a causal effect size estimation. Extending the initial numerical study, we demonstrate that the identified never-documented artefact of the TMLE application results on the gross violation of the probability coverage property underlying statistical inference.