AI & ML
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
arXiv cs.MA0 views
arXiv:2605.23238v2 Announce Type: replace-cross Abstract: Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
Source:https://arxiv.org/abs/2605.23238PreviousSTRIATUM-CTF: A Protocol-Driven Agentic Framework for Autonomous CTF SolvingNextDenoising Multirobot Trajectories
Related posts
A Cracked Mirror: How People Perceive Gender Performance by Large Language Models
0arXiv cs.HC
BobFlow: Towards Human-Agent Collaborative Data Science Using Dataflows
0arXiv cs.HC
The Male Abstraction Bias in Internet Culture and Large Language Models
0arXiv cs.HC
TACT: A Pragmatic Approach to Qualitative Analysis with an AI Council
0arXiv cs.HC
How Do Artificial Agents Make Decisions in Light of Visual Data?
0arXiv cs.HC
Contextual Pictograms for Repairing Multilingual ASR Errors
0arXiv cs.HC