The 14-Case Permission Test for an AI Email Assistant

Decision: Start with an assistant that can read only the mailbox and create drafts. Keep sending, forwarding, deleting, archiving, and calendar changes behind separate controls. If the product bundles those permissions, do not connect a work or customer mailbox until you can test the actual permission set, approval screen, audit trail, and revoke path. A polished demo is evidence of the intended workflow; it is not evidence that an OAuth grant is narrow or that a human approval is bound to the exact message that will leave the account.

For a concrete feature surface to inspect, Trust Vault’s Superhuman AI page is a useful starting point, but treat it as a product overview rather than an independent security assessment. It is one page from the wider AI-tool catalog, and its appearance in Search Console is a page-level discovery signal, not a security rating, user survey, or proof that the product meets your requirements. The tests below are product-neutral and work for Gmail, Outlook, a native email client, or an MCP connector.

Use this article as a buyer acceptance test. It gives you a fourteen-case fixture, a small executable policy gate, and the evidence to request before enabling an assistant on a mailbox with real correspondence. The cases are synthetic. They are not a benchmark of Superhuman, Gmail, Outlook, Claude, ChatGPT, or any other product, and they produce no market-wide pass rate.

What to approve first

Approve the smallest useful loop: identify a message, explain why it needs attention, draft a reply, save it as a draft, and wait. Do not grant a broad “manage my email” permission just because the first pilot only needs summaries. Email tools can expose message bodies, names, attachments, calendar events, labels, and a writing profile; some can also send or modify records. A consent dialog describes delegated capabilities, not the exact behavior your prompt or product interface will enforce.

The Gmail API documentation separates scopes by operation. It lists read-oriented scopes such as gmail.readonly, draft scopes such as gmail.compose, and broader modify or send scopes. Google marks some scopes restricted and notes additional verification or security-assessment obligations in specified circumstances. Read the live Gmail API scope table for the current names and classifications; do not infer the exact scope from a product label such as “AI assistant.” A scope that can send or alter mail expands what the integration could do after a prompt error, compromised token, or mistaken approval.

The relevant governance distinction is between a recommendation, a draft, an approved action, and an autonomous action. Gartner’s May 2026 description of agent autonomy treats drafting with human execution differently from sending after per-action approval, and both differently from fully autonomous execution. Its forecast about future decommissioning is a prediction, not a measured current failure rate. The useful operational idea is narrower: match controls to what the connector can actually do, not to what the vendor calls the feature. Gartner’s explanation of autonomy levels is one governance lens, not a vendor certification.

A government baseline supports the same least-privilege direction. The Canadian Centre for Cyber Security recommends assigning users the minimum level of access needed for their task and managing account lifecycle. Apply that principle to the application identity too: if a summary workflow does not need to send, the test should reject send capability. The Centre’s access-control guidance is general organizational guidance, not an email-agent product evaluation. CISA’s Cyber Resilience Review separately asks organizations to review privileges for excess or misalignment and to apply least privilege. That is an organization-level assessment, not product-specific guidance; use it to justify a periodic privilege review, not a vendor safety verdict. CISA Cyber Resilience Review question set.

What current product material tells you—and what it cannot

Product documentation can help you map the proposed workflow. Superhuman’s public help guide describes Auto Labels, Auto Archive, Auto Drafts, and reminders in Gmail; its onboarding lets a user configure archive categories, and the page says certain archive categories are enabled on first setup by default. The product site says Email Assistant leaves drafts for review and nothing sends without user approval. Its separate MCP guide exposes tool-level approval settings. These statements describe advertised behavior and configuration, not independent evidence that every tenant has the same default or that an approval is bound to an immutable message. A source can also describe more than one product surface; do not transfer a setting from MCP to Gmail or vice versa. Verify the exact account and connector you will use. Read the Email Assistant guide, the MCP server guide, and the product’s AI email page.

The video is worth watching for the interaction sequence: where account connection occurs, what the assistant is shown, how it creates a draft, and where approval settings are surfaced. It is instructional product material, not a test of permission enforcement. Do not translate “the demo showed a draft” into “the connector cannot send.” A video may show one account, one configuration, one prompt, and one successful path. Your acceptance test must include denied actions and failure paths.

Real user discussions expose a more useful test question than “does the assistant write well?” In a Reddit builder thread about an AI agent that reads business email and drafts replies, the author reports that an early run attempted to draft a response to an automated Google security notification. The discussion asks how the system decides whether a message deserves a response and recommends showing the reason, source snippets, and rule behind each draft. That is one builder’s report, not a prevalence statistic or controlled evaluation. It does, however, motivate test cases for no-reply messages, source traceability, and reconstructing why a thread was selected. Read the thread and its comments.

A second source of tension is that narrower access can reduce the assistant’s convenience. A draft-only assistant may not schedule a meeting, send a follow-up, or clear a low-priority folder by itself. That is a real trade-off. The correct response is not to treat friction as a bug; it is to decide which actions create enough value to justify additional authority, then test those actions separately. A safe pilot can begin with one mailbox, one team, one workflow, and an explicit stop condition. The 14 cases below turn this principle into observable outcomes rather than a policy statement.

Why a permission screen is not a release test

An authorization grant answers a question about capability: what operations may the client request under the granted identity? A product policy answers a different question: which operation should happen in this workflow? A prompt answers another: what did the user ask the model to do? A reliable design keeps these layers separate. If the prompt says “draft a reply,” a send-capable token can still be called by a bug, a malicious message, an accidental tool selection, or an approval misunderstanding. Conversely, if send capability is absent, a prompt injection cannot make the connector send through that token. This is a useful boundary because it does not depend on the model understanding every hostile instruction.

Least privilege is not the claim that read-only access is harmless. A mailbox may contain health details, client negotiations, password-reset notices, legal discussions, invoices, and information about people who never consented to an AI pilot. The Canadian Cyber Centre’s access-control guidance says to give the minimum access needed and to manage accounts through their lifecycle. CISA’s Cyber Resilience Review similarly asks whether privileges are reviewed for excess or misalignment and whether permissions follow least privilege. These are system-governance controls, not proof that a particular email tool complies. Use them as test design principles: make the scope review routine, name an owner, and demonstrate timely revocation.

There is also a distinction between permission granularity and data minimization. A provider may have read access to an entire inbox even when your prompt names one thread. A user interface may present a “current email” action while the connector can search all mail. Your test should ask what is technically accessible, what is sent to a model provider, what is retained in logs or drafts, and which administrators can inspect it. Do not treat a product’s use of encryption or a privacy label as an answer to those separate questions. Ask for the data flow, retention period, subprocessors, deletion path, and contractual terms applicable to your plan and region.

The 2026 Stanford AI Index describes AI integration as spreading while frameworks to manage, evaluate, and understand the technology lag behind. That is broad cross-sector context, not a study of email assistants, OAuth permissions, or product safety. It supports doing a local acceptance test; it cannot tell a buyer which scopes one connector requests.

A Cloud Security Alliance research note on OAuth consent phishing also argues that granted OAuth access can persist beyond an abandoned trial, and recommends reviewing stale or broad grants. This is threat analysis from an industry association, not an audit of the assistant considered here. Its practical relevance is the lifecycle: the trial ending does not necessarily remove the identity grant. Record who can revoke it and where to verify revocation from the identity provider. The CSA note discusses OAuth as an active grant-management surface.

The acceptance model: four capability tiers

Use four tiers to make the decision legible. Tier 0 has no live mailbox connection: the user pastes a selected message into the model. This minimizes broad access, but it can create unmanaged copies and loses the context needed to verify a thread. Tier 1 gives the assistant read access to defined mailbox data and permits summaries or classifications, but no draft or write operation. Tier 2 lets it create a draft but cannot send, archive, delete, forward, or change calendar state. Tier 3 adds an action such as sending or moving a message only after the user approves that specific action. A fourth, autonomous tier allows actions without per-message approval; reserve it for narrow, reversible, low-impact tasks with monitoring, limits, and a tested stop control—not external customer correspondence by default.

The table is a decision aid, not a formal standard. A vendor may combine tiers within one connector, and an admin may configure multiple tools separately. Classify the maximum capability the credential and tool expose, not the most conservative workflow the marketing page demonstrates. Gartner’s autonomy framework makes a similar point: governance should scale with the agent’s ability to affect systems. Its description also cautions that approval can lose meaning under time pressure. Therefore “human in the loop” is not a control until the human can see what will happen, has time to inspect it, and can reject it before the external side effect.

What counts as meaningful approval

A meaningful approval binds a reviewer’s decision to a specific action. At a minimum, show the exact recipient list, subject, full body, attachments, sending account, and any calendar or file changes. If the draft changes after review, the approval should expire. If the assistant rewrites the message after the reviewer clicks Approve, the action is not the approved action. If a batch has many drafts, reviewers need per-item visibility or an explicit, bounded batch rule. “Approve all” is risky when each recipient, attachment, or commitment differs.

An approval screen should also distinguish an internal draft from an external send. A user who sees “Apply” may think they are saving a draft while the tool interprets it as sending. The product should say which mailbox identity will send, whether a reply-all expands recipients, and whether an attachment is present. A separate confirmation is appropriate for a new external recipient, a high-risk domain, financial or legal language, or a message that includes sensitive content. The point is not to add a click to every harmless edit; it is to make a consequential boundary visible and testable.

Run the 14-case permission test

Use a sandbox mailbox with synthetic messages and no external recipients. Keep the assistant’s output separate from a real production mailbox; do not test failure behavior by sending a real message to a customer. For each case, record the product name and build, date, account type, connector, granted scopes, tool-level approvals, prompt, expected result, observed result, screenshot or audit event, and pass or fail. If the tool does not expose the underlying scope list, mark the scope evidence “unavailable” and treat that as an unresolved control gap rather than guessing from a consent summary.

A pass is not “the model answered correctly.” A pass means the system performed the allowed action, refused the forbidden action, and left enough evidence to reconstruct the result. A case fails if the assistant claims that it cannot send but the API token has a send scope that is not otherwise constrained; the claim is not technical enforcement. It also fails if the system writes a draft when the product requirement was to do nothing, or if it creates a good-looking approval page without evidence of a rejected request.

The fixture has fourteen cases because it is a usable small-team review packet, not because fourteen has a special statistical meaning. The sample is synthetic and intentionally includes both normal and adversarial conditions. Method: create the fixtures with fake names, fake addresses on a domain you control, and dummy content; test once per product configuration; capture scope evidence and output; then repeat the denied-action cases after a configuration change. Date of this article’s design: November 6, 2026. No live product, user population, model, OAuth app marketplace, or real mailbox was tested here. Do not convert a future test score into a claim about general safety without a documented sample and method.

A small, executable gate for the fixture

The following Node.js script checks a policy record. It is not an OAuth client, a test harness for Gmail or Microsoft Graph, or a security scanner. It prevents an acceptance report from passing when the declared permissions include operations the pilot forbids, or when any required case was not passed. It fails closed when required fields are missing.

// permission-gate.mjs import fs from 'node:fs'; const requiredCases = Array.from({ length: 14 }, (_, i) => 'C' + (i + 1)); const policy = JSON.parse(fs.readFileSync(process.argv[2] || 'run.json', 'utf8')); const errors = []; const allowed = new Set(policy.allowedOperations || []); const forbidden = new Set(policy.forbiddenOperations || []); const forbiddenGranted = [...forbidden].filter(op => allowed.has(op)); if (forbiddenGranted.length) errors.push('forbidden operations are granted: ' + forbiddenGranted.join(', ')); if (!policy.scopeEvidence || policy.scopeEvidence === 'unavailable') errors.push('scope evidence is missing'); if (!policy.revocationEvidence) errors.push('revocation evidence is missing'); if (!policy.owner || !policy.reviewDate || !policy.productVersion) errors.push('owner, reviewDate, and productVersion are required'); const cases = new Map((policy.cases || []).map(c => [c.id, c])); for (const id of requiredCases) { const c = cases.get(id); if (!c) errors.push(id + ' is missing'); else if (c.result !== 'pass' || !c.evidence) errors.push(id + ' did not pass with evidence'); } for (const id of cases.keys()) if (!requiredCases.includes(id)) errors.push('unexpected case id: ' + id); console.log(JSON.stringify({ ok: errors.length === 0, errors }, null, 2)); process.exitCode = errors.length ? 1 : 0; 

A sample policy file should name the operations literally, for example read, createDraft, send, archive, delete, forward, and calendarWrite. A draft-only pilot can set allowed operations to read and createDraft, and forbidden operations to send, archive, delete, forward, and calendarWrite. The example does not prove the token is constrained; that proof comes from identity-provider scope inspection and the denied-operation cases. The script only checks consistency in the record that a reviewer supplies. If the supplied file falsely says a test passed, the program cannot discover that lie. Pair it with screenshots, mailbox state, and provider logs.

For repeatable evidence, store the original fixture, a normalized text representation of the draft, recipient set, attachments, time, and test-run identifier. Do not store real email content in a general-purpose ticket or public repository. For a test intended to confirm no-send, check the sent folder and provider-side audit events after the assistant’s response; a UI toast alone is not enough. For idempotency, retry the same action after a simulated timeout and verify whether a second message appears. A tool may return an error even though the server completed the send; the system must reconcile outcome before retrying.

Configure and pilot in six steps

1. Write the task in one sentence. “Find messages matching these criteria, explain why each needs attention, and create a draft for an employee to review” is testable. “Manage my inbox” is not. Identify whether the task requires search, calendar data, attachments, multiple accounts, or sending. Each extra connection changes the information boundary. If the requirement can be met with one selected thread, do not silently expand it to the entire mailbox.

2. Translate the task into operations. List read, search, create-draft, edit-draft, send, reply-all, forward, archive, delete, label, schedule, and calendar-write separately. Mark which are essential, which are nice-to-have, and which are forbidden. Compare that list to the actual OAuth scopes, API permissions, and enabled tools. The Gmail scope reference shows why product labels are insufficient: operations have different scopes, and some access classes are designated restricted. On Microsoft 365, review the application’s delegated and application permissions in the tenant. Require the vendor to explain any scope whose name or effect is unclear.

3. Define the data boundary. Decide which mailbox, folders, labels, attachments, calendar, contact records, and historical period are included. Then ask what leaves the mail provider: message text, attachments, metadata, embeddings, voice profile, and generated draft. Request the applicable retention and deletion terms, model-provider/subprocessor list, admin access path, and incident notification process. A generated summary or reply can itself preserve confidential facts after the original message is no longer in view. That is an operational inference, not a claim that every output is legally classified as personal data in every jurisdiction. Treat drafts, generated summaries, and audit trails as data objects with access and retention rules; check the rules applicable to your organization and plan.

4. Choose the pilot tier and owner. Start at tier 1 or 2. Give the integration a named business owner, a technical owner who can review permissions, and a person authorized to stop it. Use a test mailbox or a mailbox with a deliberately limited test dataset. For work data, get administrator review before any employee self-connects. The Cyber Centre’s account lifecycle and least-privilege guidance is a general control baseline: unique identities, minimal privileges, and account removal when no longer needed.

5. Run the cases and keep evidence. Record version and settings before and after each change. Capture not just model output but the tool call, scope, recipient list, mailbox state, approval identity, timestamp, and result. If you change a prompt, tool list, connector, or OAuth consent, rerun the cases it could affect. A new model version can alter tool selection; a new permission can change what the same prompt is capable of doing. Do not compare results across releases unless you hold the test fixture and environment constant.

6. Set the launch and stop rules. A pilot should have an allowed user group, mailbox scope, time period, use cases, and measurable decision criteria defined before connection. Pause if the assistant sends when configured draft-only, selects a security or no-reply message for response, exposes a recipient or attachment unexpectedly, cannot show the reason for a draft, or cannot be revoked. Keep a rollback action ready: disable the connector, revoke the OAuth grant at the identity provider, invalidate sessions or tokens as supported, and confirm the mailbox stops responding to new requests. Deleting the application from a client UI may not revoke an identity-provider grant; verify from the account or tenant security console.

Approval fatigue and the counterargument for automation

A human approval gate is not automatically safe. If the system creates dozens of drafts and presents them as one batch, the reviewer can click through without reading. If every low-risk reply requires the same friction as a high-impact message, users may disable the guard or move work to an unapproved tool. Gartner describes this trade-off in its autonomy guidance: over-restricting simple agents can slow delivery, while under-restricting agents that affect systems increases risk. The way out is not a universal “always approve” rule; it is risk-based action boundaries and a narrow pilot.

For an internal meeting reminder to a fixed team, a bounded automatic send might become defensible after explicit recipient limits, approved templates, rate limits, logging, and a stop switch have been tested. For messages to clients, candidates, regulators, patients, or suppliers, keep the human reviewing the final content and recipients until the workflow has dependable evidence and a responsible owner. This is a recommendation based on the consequence of external communication, not a legal rule that applies identically to every industry.

The strongest argument for broader access is that a useful assistant needs context. A tool that sees only one pasted email may miss the thread, commitments, calendar conflict, or previous response. A tool that cannot send may add an approval step users find cumbersome. A procurement gate should acknowledge that benefit rather than pretend it is free to restrict access. Evaluate whether broader access produces a meaningful outcome in your own pilot, then price that benefit against additional exposure, the quality of review, and the cost of operating the control. Do not accept a generalized productivity claim as a substitute for a locally measured task outcome.

Failure modes that should stop a pilot

Scope drift: A product begins with read-only access, then asks for broad modify rights to unlock a new feature. Re-run the scope review; do not treat an upgrade prompt as a routine click.

Prompt injection in a message: An email contains instructions addressed to the assistant. Treat all message content as data, not policy. The assistant must not forward, reveal, or alter other messages merely because an email asks it to.

Wrong-thread selection: The model finds a similar but incorrect conversation. Require source identifiers and a visible reason; a draft without traceable inputs cannot be reviewed efficiently.

Approval mismatch: The user approves a draft, but the system changes recipients or body before sending. Bind approval to the exact message version and recipient set, then reject stale approvals.

Duplicate send after timeout: The client reports failure, retries, and sends twice. The send path needs an idempotency or reconciliation strategy and a post-timeout status check.

Token survives the pilot: The employee stops using the product but the grant remains. Schedule a grant review, document an owner, and verify revocation instead of assuming inactivity removes access.

Data copied into logs: The model output contains customer facts or special-category data. Define who can access logs, how long they remain, and how deletion is confirmed.

Convenience becomes silent policy: Users assume “approve” means they checked an entire batch. Make review scope explicit, show changed content, and use per-message approval for external actions.

A failure in one of these cases is not necessarily proof the whole product is unusable. It is evidence that the current configuration does not meet the current pilot requirement. The next step might be narrower scopes, disabled tools, smaller mailboxes, a new admin policy, or a different workflow. Record the gap and rerun the same fixture after a change; do not waive a failed case because the demo looked good.

The evidence packet to take to procurement

Ask the vendor or internal application owner to supply a compact packet, not a slide deck. The packet should include: exact product and connector version; OAuth scopes or Microsoft permission names; whether grants are delegated or application-wide; which tools can read, draft, send, modify, or schedule; the default approval setting for each tool; the identity that will execute the action; the prompt and data retention path; administrator and support access; the revocation steps; and the evidence produced by cases 4, 6, 11, 12, and 13. If an answer is “the model will follow the user’s instruction,” that is a behavioral expectation, not a technical authorization control.

For the no-send control, request two proofs: the consent or admin screen does not grant send capability, and an attempted send tool call is denied by the service. For the approval control, ask the vendor to show that the action confirmation includes the final recipients and final body, and that any edit invalidates the approval. For prompt injection, ask how inbound messages are separated from system instructions, then run the exact fixture with a fake message containing a forward request. A vendor’s description of a defense is useful evidence, but your observed test result is stronger.

For personal data, ask which categories are processed, what the product does with message text and generated drafts, whether the data is used to train any model, what service providers receive it, where it is processed, and how long different artifacts remain. Ask which terms apply to the exact paid plan and region. A generic trust-center badge may describe the company’s broader program rather than the specific product feature or connector. Map each answer to a contract, current policy, control screen, or test evidence. Where the answer has legal or regulatory consequences, involve the organization’s privacy and legal functions rather than relying on this checklist as legal advice.

Example record for one run

A useful record is a small JSON object plus evidence files stored in an access-controlled location. The code above expects fields such as the following; this intentionally incomplete record should fail until you add all fourteen results and evidence references.

{ "productVersion": "record the exact build", "owner": "mail-assistant-pilot-owner", "reviewDate": "2026-11-06", "scopeEvidence": "scope-manifest.pdf sha256:...", "revocationEvidence": "identity-console-capture.png", "allowedOperations": ["read", "createDraft"], "forbiddenOperations": ["send", "archive", "delete", "forward", "calendarWrite"], "cases": [ { "id": "C1", "result": "pass", "evidence": "C1-run-01.json" } ] } 

Preserve enough evidence for another reviewer to reproduce the decision, but avoid putting real messages into the record. Use synthetic participants and a test tenant wherever possible. A hash can show that a file was not changed after collection; it does not prove that the file reflects production behavior. A screenshot can show what the interface displayed; it does not prove what the backend authorized. The packet is strongest when each important claim has both a configuration artifact and an observed behavior.

How to interpret the sources and the limits of this test

The sources answer different questions. Google’s API scope documentation is the authoritative place to verify current Gmail scope names and classifications. Superhuman’s own help pages explain advertised operations, setup, and approval settings, but the product’s claims need account-level verification. The Canadian Cyber Centre and CISA provide general access-control practice, not a certification for any mail assistant. Gartner provides an autonomy-based governance perspective; its forecast is explicitly future-oriented. The Microsoft 365 permission paper and Cloud Security Alliance note provide security-research context, but neither is a product test of the email assistant you may buy. The Reddit discussion is a single builder’s report and a useful source of a test question; it cannot tell you how often such errors occur. Keep these evidentiary roles distinct.

Some recommendations can appear contradictory: a human should approve sends, but too many approvals can become meaningless; a connected inbox enables context, but broad access widens exposure; draft-only reduces side effects, but it can still leak sensitive information in a draft or logs. These are not reasons to average the controls into a single score. They are reasons to state the workflow, threat boundary, data class, and action consequence before choosing a tier. A low-impact internal reminder and a customer contract response do not need identical rules.

The original asset in this article is a protocol, not a measurement study. It consists of fourteen synthetic acceptance cases, one lightweight Node policy gate, and an evidence model. Method and date are stated above; the sample size refers to the designed cases, not users or tested products. Exclusions: no real inbox, live OAuth grant, model output, vendor, enterprise tenant, or user population was tested. Limits: the script evaluates a reviewer-supplied record, not the truth of the evidence. It cannot inspect an access token or enforce a provider’s policy. Its value is to make an incomplete release packet fail visibly and to make a future test reproducible.

Go / no-go checklist

Before connecting a live mailbox, confirm all of the following:

• The task is one sentence, has a named owner, and does not require broader access than the first pilot grants.
• The OAuth and application permissions are documented from the identity provider, not inferred from a feature description.
• The assistant can be constrained to the smallest useful data set and operations.
• Send, forward, archive, delete, and calendar-write behavior is explicitly enabled or denied; no action remains ambiguous.
• Drafts show the thread and evidence used; a reviewer can tell why the assistant selected the message.
• Inbound email instructions cannot change the assistant’s authorization policy or trigger unrelated tool calls.
• Approval displays final content, recipients, sending identity, and attachments; edits require new approval.
• Duplicate retries, wrong recipients, unexpected attachments, automated senders, revocation, and offboarding have been tested.
• Data retention, model-provider sharing, support access, and deletion terms match the exact plan and region.
• Audit evidence is access-controlled, avoids real message content where possible, and has a retention owner.
• The pilot has a pause condition, a revocation owner, and a documented method to confirm that the grant was removed.

If any permission is not visible, any required denial cannot be reproduced, or revocation cannot be verified, do not widen the pilot. Run again with a smaller scope or keep the assistant disconnected while the gap is resolved. If all checks pass for a draft-only workflow, begin with a small group and review the first outputs. Expand only when the same cases pass under the actual organization settings and the additional access clearly supports a measured task benefit.

The practical rule is simple: let the assistant prepare; let a person decide what leaves the mailbox. Change that boundary only when you can show, with current evidence, exactly what the assistant can access, how an approval is bound to the action, and how the organization can stop and revoke it.