AI & ML
What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays
arXiv cs.SE0 views
arXiv:2611.07650v1 Announce Type: new Abstract: Third-party application programming interface (API) relays offer access to large language models (LLMs), often at lower prices. Users depend on these services for model quality, instruction compliance, and accurate billing. However, existing studies leave unclear whether relays meet these expectations together. We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses. In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules, and six under matched correction families. Because answer comparisons cannot reveal every delivery problem, we examine returned content and tool permissions. No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round. Finally, to connect reported usage to actual costs, we compare input-token counts with official references and check balances. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices, including extra reported input tokens at one service. Our study shows that advertised models and prices are insufficient to judge relay reliability. It identifies unexplained billed token quantities and tool-permission violations as risks and shows why service evaluations need repeated measurements.
Source:https://arxiv.org/abs/2611.07650PreviousAiXapply: Fast and Reliable Full-File Code Integration with Specialized Small Models for IDE WorkflowsNextFix One, Miss Another: Leveraging Historical Bug Fixes to Uncover Residual Bugs in Database Systems
Related posts
After blocking Muse, Amazon now lets users text Alexa to help them shop
0TechCrunch
AI for scientific discovery: How researchers use Claude
0Hacker News
FAA Begins Retiring Decades Old Floppy Disks from Air Traffic Control
0Hacker News
WARN Act and Child Support — What Happens to Support Payments After a Layoff Notice
0dev.to Data
1,084 Mercado Libre Cards Told Me How Many Were Sold. Between Them They Used Ten Numbers
0dev.to Data
Three traps when an AI assistant reads insider trades from Form 4s
0dev.to Data