AI & ML
Adapt the Object to the Subject: Representative Study Material for Empirical Software Engineering
arXiv cs.SE0 views
arXiv:2611.07772v1 Announce Type: new Abstract: External validity is a central concern in empirical software engineering, yet the debate has largely focused on whether study participants are representative. How well the study objects they reason about - code, models, or requirements - are representative of each participant's own work is often overlooked. Practitioners' own artifacts would be ideal, but are often out of reach, so researchers sample from public datasets or construct objects artificially. Whether this lack of subject-object fit affects participants' ability to act on the material remains unknown. We take a first step toward answering this question for requirements. We propose an approach that uses an LLM to generate context-adapted synthetic requirements just-in-time during an online survey from a short, participant-supplied context description, optionally injecting requirements smells as controlled stimuli. To validate the approach, we conducted a controlled experiment with 31 industry practitioners in a 2x2 design that crosses stimulus generation (static vs. synthetic) with quality (clean vs. smelly). Each participant rated the perceived representativeness and assessability of 20 requirements. Using Bayesian data analysis within a statistical causal-inference framework, we find that perceived representativeness has a strong positive effect on assessability. Synthetic requirements are rated as more representative and, primarily due to this gain, more assessable than static ones sourced from public datasets; smell injection leaves both perceptions unchanged. Our results support treating a lack of study-object representativeness as a first-class validity threat and position context-adapted synthetic study objects as a practical addition to the empirical methodology toolbox.
Source:https://arxiv.org/abs/2611.07772PreviousFast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical StudyNextWhich Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness
Related posts
The Step-by-Step Guide to Shutting Off Apple Intelligence Features in iOS 27
0CNET News
Will Democrats Rein in Big Tech? It Starts With Privacy and Data Centers
0CNET News

“Most apps never make it off localhost”: Cursor takes on Lovable and Replit with fully managed app deployment
0The New Stack
Inspired by honeycombs, a low-cost cooling structure lasts over 100 million cycles
0Phys.org Tech

Chainalysis admits all users affected in major data breach — names, usernames, password hashes, and live tokens all exposed
0TechRadar
iOS 27.2 adds Apple Photos invisible watermark feature, here’s how it works
09to5Mac