Krüger, J., Çalıklı, G., Bershadskyy, D. et al. Guidelines for using financial incentives in software-engineering experimentation. Empir Software Eng 29, 135 (2024). https://doi.org/10.1007/s10664-024-10517-w
Bershadskyy et al. (2025) - coming soon
In the first article we develop guidelines for using financial incentives in empirical software engineering (SE) to address issues such as participant motivation and selection bias, thereby improving study validity. Drawing from a systematic literature review of 105 SE studies and insights from economics and psychology, we highlight the underuse of advanced incentive mechanisms, like performance-based pay, in SE experimentation.
Most SE experiments use simple incentives like completion fees or course credits, but few utilize tailored payoff functions.
Financial incentives can enhance participation rates, motivation, and the realism of studies, improving both internal and external validity.
Challenges include determining appropriate incentive amounts, managing costs, and avoiding motivational "crowding out" where external rewards diminish intrinsic motivation.
We provide a decision-making framework for researchers to assess the need for financial incentives and recommend 11 strategies for designing effective payoff functions. These include aligning incentives with task difficulty, participant characteristics, and real-world scenarios. They advocate for better documentation and reporting of incentives to enhance transparency and replicability. The guidelines aim to fill gaps in SE research practices and help standardize incentivization strategies for more robust experimentation outcomes.