AI & ML
INTENT: Out-of-Sample, Out-of-Time, and Cold-Start Evaluation on User Event Sequences
arXiv cs.IR0 views
arXiv:2611.13069v1 Announce Type: cross Abstract: Banks now train foundation models on ordered sequences of customer events such as transactions, app sessions, and authorizations. Deployed, such a model predicts forward in time, often for customers and merchants absent from training, yet each is evaluated on private data, so no two can be compared. Public benchmarks for such sequences rarely test a period the model has not seen; recommendation studies usually hold out each user's last events instead. We present INTENT (INTEraction, Novelty, and Transfer evaluation for next-action prediction), a public benchmark that scores a model in seven cells defined by whether the user, the period, and the target item appeared in training. An audit of 59 public dataset families retains five datasets; the most common defect is the lack of a calendar timestamp that orders events. No date lies within the time span of all five datasets, so each is split at its own cutoff, chosen by one rule fixed before any model is evaluated. Summed over the datasets, every cell holds at least 111,769 users. In reference results the cell determines the best model: a transformer leads on seen items after the cutoff but scores zero on new items, a variant that embeds item metadata helps mainly on MicroLens, and repeating the user's last item leads on new items in four of five datasets. This parameter-free control also shows that on three datasets most of the score lost after the cutoff reflects changed behavior rather than the model. The pipeline, the cutoff rule, and the per-cell user counts are publicly available.
Source:https://arxiv.org/abs/2611.13069PreviousIntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior PredictionNextActivation-Guided Graph Retrieval for Cognitive Memory Reconstruction in Language Model Systems
Related posts

Five Details That Make a Bedroom Feel French, From Someone Who Sells French Bedding
0dev.to Design
User-Friendly Mobile App UX Design – Best Practice & Process
0dev.to Design
The empty states in my app blamed the user, so I wrote a linter for that
0dev.to Design
Generic CRMs made me hate CRM, so I built one with only my trade's stages
0dev.to Productivity
Think Before You Solve: The Importance of Planning in Software Engineering
0dev.to Productivity
Attention Fusion for Bridge Deck Delamination Detection
0arXiv eess.IV