Measuring the impact of generative AI in higher education

From hype to evidence-based learning

Generative AI is already firmly embedded in students' daily lives — for research, summarizing materials, and writing. Universities are now taking the next step: integrating these tools directly into teaching. The AIffectiveness project, led by the Bavarian Research Institute for Digital Transformation (bidt), looks beyond speculation to examine the actual impact of these tools.

A multi-dimensional research approach

Using OneTutor, an AI assistant developed at the Technical University of Munich (TUM), as a concrete case study, we accompany the rollout and use of generative AI across a wide range of courses. Our goal is to generate evidence-based insights into where AI tutors meaningfully improve the learning experience, and where they still need to evolve.

We don't believe in a simple "yes or no" when it comes to AI effectiveness. Instead, we analyze the specific context: group size, subject area, and the balance between factual knowledge and critical reasoning.

• Real-world deployment: Analyzing usage across 10 universities, from large lectures to small seminars.
• Quantitative surveys: Collecting data on student usage habits and faculty feedback at regular intervals.
• Usage Analytics: Incorporating direct data from the "OneTutor" system to correlate engagement with learning outcomes.
• Trust & Co-Creation: Investigating how trust in AI systems evolves over time among students and faculty.

„I hope that by the end we will understand exactly where these systems are useful and where they are not. It's not just about whether it works, but in what context — large lectures versus small seminars, or fact-based disciplines versus argumentative ones.“
Prof. Dr. Alexander Pretschner - Ex-Chair of the bidt Board of Directors & Professor of Software & Systems Engineering, Technical University of Munich

Partner institutions

• Technische Universität München
• Ludwig-Maximilians-Universität München
• Virtuelle Hochschule Bayern (vhb)
• Technische Universität Nürnberg
• Technische Hochschule Rosenheim
• Universität Augsburg
• Ostbayerische Technische Hochschule Amberg-Weiden
• Universität Bayreuth
• Hochschule Weihenstephan-Triesdorf
• Technische Hochschule Aschaffenburg

‍