Fall semester 2025 / Michael Nussbaum: Evaluation of Workflows for Interpretable Machine Learning
In psychological research, machine learning models are increasingly used to predict complex outcomes but their results are not directly interpretable. Explainable AI (XAI) methods such as Shapley values address this limitation by quantifying how predictors contribute to model predictions. Applying machine learning and XAI requires pipelines consisting of sequential steps: data splitting, model learning, predictions and the application of XAI methods. While consensus exists for the first three steps, a literature review of 53 recent empirical studies revealed substantial heterogenity in the final step, in particular regarding which dataset (training, tests, entire, or external) is used to compute the Shapley values.
This master's thesis investigated whether the choice of pipeline and dataset affects Shapley value estimation and the substantive conclusions drawn about feature importance. In a Monte Carlo simulation study conducted in R, five commonly used pipelines from psychological research were compared under controlled conditions with known ground truth. Data were generated from a linear model with 20 predictors, and within each of 250 replications a conditional random forest and a single-hidden-layer neural network were tuned via 5-fold cross- validation and fitted, with hyperparameter grids informed by pilot studies. The simulation systematically varied sample size (N= 200 vs. N=500) and evaluated how accurately the Shapley-based feature importance rankings, assessed via Kendall's rank correlations and absolute ranking differences, recovered the true importance structure of the data-generating process.
Results showed that dataset choice did not affect the resulting Shapley values, whereas model fit did. Pipelines that computed the Shapley values on the training, test, or entire dataset produced similar results. In contrast, pipelines without initial splitting achieved better model fit and more accurate Shapley values than pipelines that split initially. However, this effect was small in magnitude. For practical applications, researcher can therefore use any dataset to compute the Shapley value.
Spring semester 2026 / Leeroy Koller: Statistical Inference for SHAP Values: A Simulation Study on the Detection of Informative Predictors
In psychological research, machine learning models are increasingly used to predict complex outcomes from large and high-dimensional datasets. While these models often achieve high predictive accuracy, they are frequently criticized for lacking interpretability. Shapley value-based explanations (SHAP) have become one of the most widely used approaches for explaining how predictors contribute to model predictions. However, most applications treat SHAP values descriptively and rarely examine whether SHAP-based importance measures can reliably identify truly informative predictors.
This master’s thesis investigated the statistical properties of SHAP-based variable selection procedures using a comprehensive Monte Carlo simulation study. Building on recent developments in explainable artificial intelligence and statistical learning, different inferential approaches for SHAP importance were compared under controlled conditions with known ground truth. The simulation design systematically varied sparse and dense predictor structures and evaluated the methods with respect to false alarm rates, detection rates, and computational efficiency.
The results showed that mean absolute SHAP values reliably reflected the underlying importance structure of the data-generating process. However, the evaluated SHAP-based testing procedures exhibited substantial inferential weaknesses, including inflated false alarm rates and strong dependence on the underlying signal structure. Overall, the findings suggest that SHAP explanations are particularly valuable as descriptive interpretation tools, whereas statistical inference based directly on SHAP importance remains challenging and requires further methodological research.
Fall semester 2024 / Luana Brunelli: Reliability measures in intensive longitudinal data -- Understanding generalizability and multilevel theory with interactive tools
In psychological research, the use of intensive longitudinal data (ILD) is becoming increasingly common to capture within-person dynamics over time. However, this growing interest in ILD highlights the need for reliable measurement across repeated time points. If reliability is low, conclusions drawn from ILD can be misleading or unstable.
This project explores advanced approaches to estimating reliability in ILD settings, drawing on both Generalizability Theory and Multilevel Modeling. These frameworks take into account the nested nature of ILD—observations nested within individuals—and provide more accurate estimates of measurement consistency over time.
Through a comprehensive simulation study, we examined how key design choices—such as the number of participants, measurement occasions, and measurement error—influence the stability and accuracy of different reliability estimates. We also investigated how interactions between persons and items can systematically impact the interpretation of reliability across methods.
To support applied researchers, we developed a fully functional Shiny application that allows users to interactively explore how various design parameters affect reliability outcomes. This tool enables researchers to simulate scenarios, visualize outcomes, and make informed decisions about study design and reliability assessment. The simulation-based findings are illustrated in an interactive Shiny app, which can be accessed here: https://sdsunibas.shinyapps.io/luana/
Spring semester 2024 / Jan Radek: Partial Credit trees meet the partial gamma coefficient
In psychology, tests such as personality or language assessments undergo evaluation before being released to the public. A common step in this evaluation process is to determine whether all items are fair across different groups of individuals. If this is not the case for an item in a test, it is referred to as differential item functioning (DIF) or differential step functioning (DSF).
In this project, we focus on Partial Credit Trees (PCtree), a method that combines the partial credit measurement model from the item response theory with decision trees from machine learning. We extended PCtrees by incorporating an effect size measure for DIF/DSF in polytomous items, namely the partial gamma coefficient from psychometrics. Now researchers can see which items are affected by DIF/DSF and assess the meaningfulness the DIF/DSF effect size.
During his internship, Jan showed in a series of simulation studies that the partial gamma coefficient aids researchers in evaluating whether splits in the tree are meaningful, identifying DIF and DSF items, and preventing unnecessary tree growth when effect sizes are negligible. Additionally, we also addressed the issue of item-wise testing corrections, especially in longer assessments and illustrated the new method using data from the LISS panel.