AI & ML
Skill Roulette: Measuring Skill Invocation Rate Across LLM Coding Agents
arXiv cs.SE0 views
arXiv:2611.23266v1 Announce Type: new Abstract: Agent skills package procedures for large language model (LLM) coding agents. An agent selects a skill from its name and description before reading the procedure. It may complete the task without invoking a relevant skill. We investigate how reliably coding agents invoke relevant skills and which factors affect that decision. We estimate invocation rates by repeating trials with identical inputs; invocation rate is the proportion of trials in which the agent invokes the relevant skill. We test 9 non-directional hypotheses about skill metadata, competing skills, contextual instructions, and task wording and composition. Eight concern invocation; the ninth tests whether making a skill available changes the occurrence of an action that the task does not request. We use tasks from 4 software engineering categories in a synthetic repository, running each of 4 coding-agent harnesses with 2 models. The planned 20,400 trials include 60 repetitions per Baseline category and configuration, 60 more per category in Claude Code with Sonnet 5.5 that admit each category's task and skill, and 10 per experimental condition. Comparisons are specified before collection and evaluated within each model and harness configuration using Boschloo's exact test.
Source:https://arxiv.org/abs/2611.23266PreviousEscaping the Anthropomorphic Trap: Resolving the Cognitive Mismatch in Future AI Software DevelopmentNextAn Industry Perspective on Software Vulnerability Management: A Systematic Review of Approaches, Tools, Practices, and Challenges
Related posts
Im looking fr any free ai models / tools are there?, leaVE UR Lights💡
0dev.to
I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones.
0dev.to

How much GPU memory does an LLM need? The whole sum, worked out on real cards
0dev.to

Routing Upscaling Jobs by Output Size, Not Input Size
0dev.to
A ten-second Flutter pause with a visible Skip button
0dev.to UX

Thống kê lịch sử đối đầu Aston Villa - Sunderland: Đại diện Midlands áp đảo toàn diện
0dev.to Data