AI & ML
PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
arXiv cs.CR0 views
arXiv:2611.18293v1 Announce Type: new Abstract: Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024. Each PR includes its commits, diffs, review discussion, and LLM-generated labels for privacy relevance, fix type, and GDPR principles. PrivacyPR provides 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, together with Verified, a manually checked split of 250 privacy fixes and 250 non-fix PRs, and Small, a 100 + 100 subset for evaluation. On Verified, both annotators endorsed 72% of the LLM's GDPR-principle assignments. Beyond supporting empirical study of how privacy defects arise and are fixed, PrivacyPR provides a benchmark for privacy-aware code review on real, developer-accepted fixes. Its evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detect only 42-51% of defects under a generic review prompt, but 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs. These results show that privacy defects can be detected by coding agents, but that detection depends strongly on review intent and comes with false alarms. Data, code, reviews, judge verdicts, and an extensible evaluation harness are publicly available at https://doi.org/10.5281/zenodo.23744765.
Source:https://arxiv.org/abs/2611.18293PreviousFrom Bytes to Behavior: Request-Level Attribution from Live GPU Hidden StatesNextOn the Same Track: Privacy Leaks in Electric Rail Transport via Magnetic Fields
Related posts
An LLM confidence score isn’t correctness: the certainty trap
0dev.to
What a host-side layer for LLM agents is, in plain words (and what it is not)
0dev.to
OpenAI's Agents API Lets Any App Rent Codex, Here's How It Works
0dev.to

94% of Singapore Small Business Owners Surveyed Say More AI Guidance Is Needed
0Meta Newsroom
Nashville, Tennessee Real Estate Dataset: 7,436 Listings, Excel File (Nov 10, 2026)
0dev.to Data
A Hundred and Eighty Six Thousand Pounds of Stock Had Left One Branch and Never Arrived at Another
0dev.to Data