AI & ML
Characterizing Robustness in Semantic Code Clone Detection
arXiv cs.SE0 views
arXiv:2606.25272v2 Announce Type: replace Abstract: Semantic code clone detection seeks to recognize shared functionality across different implementations, making robustness to behavior-preserving changes central to its effectiveness. Prior studies have exposed failures under such changes, motivating a systematic examination of how sensitivity varies with the rewrites used for evaluation. We present Systematic Code Transformation (SCT), a framework organizing 51 operators by code level, target, and edit category, and evaluate 14 token, tree, graph, and prompt-based detectors on BigCloneBench, Google Code Jam, and OJClone. By comparing transformation groups, repeated application, and alternative constructions, we connect overall performance changes with specific rewriting conditions. Of 25 token, tree, and graph detector and dataset combinations starting above 90% F1, 21 lose at least 10 percentage points under the composition of all operators. These losses vary with transformation type and application intensity, but are not determined by edit counts or code growth alone. Even at matched locations and edit counts, alternative constructions can expose different errors despite similar aggregate losses. Further analysis distinguishes reduced score separability from changed fixed-threshold decisions with retained ranking ability, revealing different scope for threshold recovery. Overall, these findings show why benchmark effectiveness needs to extend to stable discrimination across equivalent implementations and how systematic transformation testing can assess that capability.
Source:https://arxiv.org/abs/2606.25272PreviousAI Slop and the Software CommonsNextLarge Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study
Related posts
Stop running everything through LLMs: How I fixed my research agent
0Hacker News New
Lessons from Building a Deep Research System
0Hacker News New
Decomp of David Horner's Drain Storm (1997)
0Hacker News New
How Sustainable Is a Solar Powered Website? (2020)
0Hacker News New
Now indexed by Vuink,500K+ words. 742 turns.135 Knowledge domains – UKA-135
0Hacker News New
IDE extension that keeps me in control of AI-written code
0Hacker News New