Computer use agents learn from screen recordings. Who pays?

Computer use agents are AI models that operate software the way a person does: they read the screen and send clicks, keystrokes and scrolls. They learn from screen recordings paired with a time-aligned log of every input, ideally with narration and the task's correct outcome. As of November 2026 the programs that publish rates mostly pay individuals, from $0.80 per accepted phone recording to $50 to $60 an hour for narrated desktop work, while a company can be paid only through its own consented, scoped program.

This post covers what these agents do, what a training record looks like, who pays for recordings, and why most programs turn away employer data.

What is a computer use agent?

A model whose observation is the screen and whose actions are mouse and keyboard events. Some agents also read the accessibility tree, but the common denominator is a screenshot in and an input event out.

The OSWorld benchmark shows how hard this still is. It set 369 tasks in real web and desktop apps across Ubuntu, Windows and macOS. In the original 2024 paper, humans completed over 72.36% of the tasks and the best model only 12.24%, with most failures in GUI grounding and operational knowledge (OSWorld). Grounding means finding the right pixel to click. Operational knowledge means knowing which menu holds the setting in the first place.

OSWorld's own action space (computer_13 in its actions.py) is a useful reference for what an agent can emit:

Every recorded human session has to be mapped into a vocabulary like this before a model can learn from it.

What does a screen-recording training record contain?

Video plus an input log, aligned on one clock. Sellers describe their records this way:

• Kled sells phone recordings as MP4 or MOV, each with a "time-aligned tap trace" captured in the app. Its 10,000+ recordings are listed as raw, with no labels (Kled dataset page).
• p(doom)'s crowd-cast recorder captures paired screencast video and keyboard and mouse input for the apps a contributor chooses.
• Markov says it holds over 500,000 hours of real work "with every action labelled and narrated," plus CAD environments with 10,000+ tasks, each with a brief, input files, gold output and rubrics (Markov).

Put together, a useful record looks something like this illustrative JSONL input log, written next to session.mp4:

{"t_ms": 0, "type": "TASK", "text": "Post vendor invoice INV-2207 and match it to PO 5531"} {"t_ms": 2140, "type": "CLICK", "x": 312, "y": 88, "button": "left", "app": "erp"} {"t_ms": 4020, "type": "TYPING", "text": "INV-2207", "app": "erp"} {"t_ms": 4810, "type": "HOTKEY", "keys": ["ctrl", "s"], "app": "erp"} {"t_ms": 5200, "type": "NARRATION", "text": "Quantity differs from the PO, so I hold it for approval"} {"t_ms": 9900, "type": "DONE", "outcome": "held_for_approval"} 

The narration and the outcome are what most raw recordings lack. A click tells a model where the user clicked. The narration tells it why, and the outcome lets someone grade an agent's attempt at the same task.

To train on this, you turn the log into pairs of "screen the user saw" and "action they took." A minimal version, also illustrative:

import json ACTIONS = {"CLICK", "TYPING", "PRESS", "HOTKEY", "SCROLL", "DRAG_TO"} def to_pairs(log_path: str, fps: int = 10) -> list[tuple[int, dict]]: """Pair each input event with the last video frame shown before it.""" pairs = [] with open(log_path) as f: for line in f: event = json.loads(line) if event["type"] in ACTIONS: frame = max(int(event["t_ms"] * fps / 1000) - 1, 0) pairs.append((frame, event)) return pairs 

Clock alignment matters more than it looks. If the video and the input log drift by half a second, a click can land on the wrong button in the training pair.

Who pays for screen recordings?

As published by each program and checked in November 2026:

Markov says it has sold more than 15,000 hours of data to frontier AI labs, according to its Y Combinator launch. Labs also record their own staff: Meta put a tool on US employees' computers that records mouse activity, keystrokes and screenshots for AI training, The Verge reported in April 2026.

Pay follows skill. An everyday phone task earns cents. Expert work in specialist software earns tens of dollars an hour, because that is the data labs cannot collect from the general public.

Why do most programs turn away employer data?

Because the work on an employee's screen usually belongs to the employer, and often shows client data. Mercor's listing says the work will not involve confidential or proprietary information from any employer, client or institution. p(doom) says confidential employer or client material does not qualify. Kled rejects recordings of someone else's account.

So an engineer cannot sign up alone and record the company ERP. The company can, if it runs the program itself. The pieces a buyer will expect are familiar to anyone who has built internal tooling:

1. An app allowlist. crowd-cast works this way: when a non-selected app is in front, the video goes black and input logging stops (p(doom) privacy docs).
2. Redaction or sandbox data, so customer names and account numbers never reach the recording.
3. Opt-in staff with written notice, and recordings kept out of performance reviews. Monitoring-notice laws vary by state and country.
4. Narration and recorded outcomes for each task.
5. A sample set first. Markov's custom collection starts with a sample before collecting at scale.

The guide to getting paid for screen recordings covers the legal side in more detail. Many companies can skip new recording entirely: the tickets, logs and process records they already keep describe the same workflows, and the workflow data guide shows how to describe them for a buyer.

Common questions

Is screen recording data the same as RL environment data?
No, but they are related. Recordings show a person doing a task once. An RL environment is a working copy of the software where an agent attempts the task many times and a grader scores it. Recordings with outcomes help builders write those graders. See RL environments and company data.

Why do phone recordings pay so little?
Everyday phone tasks are easy to collect from many people, so supply is high. Kled's screen task pays $0.80 per accepted recording, and its App Store reviews include complaints about low payouts. Desktop work in specialist software is scarcer, and published rates for it are far higher.

Can a developer record their own side projects for pay?
Often, yes, if the work is theirs to share. p(doom) asks contributors to record only work they have the right to share. Code under an employer's IP agreement, or client repositories, would fall outside that.

If your team spends its day in ERP, CAD, claims or accounting software, our free estimate asks six questions about the company and needs no recordings.

Origin Data Partners is an independent data partner that helps companies get their data bought. We are independent of the companies named here, and a buyer may pay Origin when a deal closes.