AI & ML
Evaluating Feedback-Driven Black-Box Processes via Stochastic Modeling and Simulation
arXiv cs.SE0 views
arXiv:2611.18477v1 Announce Type: new Abstract: We study feedback-driven black-box processes across software engineering and machine learning. These processes are inherently sequential: policies generate inputs, observe outputs/feedback from systems, and use the feedback to guide the design of subsequent inputs. While many policies are developed under structural assumptions on the underlying systems, many real-world systems operate without such knowledge, necessitating black-box policies that lack a unified evaluation framework. We introduce a stochastic framework that models and evaluates such policies using Markov chains and simulation, thereby connecting empirical practice with theory. We instantiate the framework in mutation-based fuzzing (widely used for discovering software bugs via sequential input mutation), by a novel Markov chain model of fuzzing whose transitions are governed by the mutation strategy and whose feedback is given by program execution. We develop a rigorous evaluation paradigm with new performance measures, sound estimators, and evaluation procedures with provable guarantees. Guided by the theory, we evaluate 9 fuzzers on 18 widely used fuzzing benchmarks and conduct case studies in 4 non-fuzzing domains. These results show that the framework provides a principled basis for evaluation while connecting to practice, with implications beyond fuzzing.
Source:https://arxiv.org/abs/2611.18477PreviousHerAgent: A Block-Level Checkpointing Harness for Coding-Agent Environment SetupNextMeta Build Systems as Logic Programs
Related posts
Coordinate Frame Recovery, Per Class NMS and Null Return Modelling for Sim2Real V2X LiDAR Detection
0arXiv cs.RO
Geometric Refinement and Distributional Alignment for Sim2Real Roadside LiDAR Detection
0arXiv cs.RO
D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models
0arXiv cs.RO
Beyond Trajectory Similarity: Evaluating Surgical Planning by Protocol Adherence
0arXiv cs.RO
HEIR: Harness Egocentric Intent for Human-Robot Interactions
0arXiv cs.RO
EmbodiExp: Compositional Experience Transfer for Embodied Agents
0arXiv cs.RO