TL;DR

  • I let Claude auto-reply to engineers' questions for a month: 18 replies went out, and none of the main answers was overturned later
  • The line is not confidence, it is type: questions an AI can settle from evidence vs. questions someone has to decide
  • Evidence accumulates: every decision written into the project docs became grounds for the next answer

The setup

I am the only PM on a mobile app for beauty salons, working with four offshore engineers, one of them a tech lead. Questions arrive as tickets in Backlog, a project management tool widely used in Japan.

Questions came in at roughly nine per week. Each reply was small, but the context switching was not: stop what I was doing, re-read the ticket and the specs, answer, then get back. I estimated two to three hours a week went into replies.

Many of those questions did not need a human. I kept giving the same answers, and I wanted only real decisions to reach me.

How it worked

On weekdays, a job collected Backlog updates six times a day. From October 15 to November 13, it ran 120 times.

Questions whose answer is settled by evidence got an AI reply. Everything else was held back and sent to me the next morning as one summary of pending decisions. When the recommendation was split, the summary showed the split instead of forcing one answer.

The line

I first tried to draw the line by confidence: reply when the AI is confident, escalate when it is not. After a month, I counted the reasons for escalation. Not one was low confidence. Every single one was about the type of question.

The AI replies to questions that the specs, past decisions, or the code can settle on their own.

The AI does not reply when the question changes cost, involves personal data, legal, or security matters, defines new user-visible behavior, or is conditioned on PM approval in the ticket.

Of the escalations, user-visible behavior was 6, cost 3, personal data and legal 3, PM approval 3, and other reasons 3.

Evidence is the product

Claude answered from three sources: the project documents, past tickets and comments, and the code.

The documents mattered most. We kept adding shared rules and settled questions and answers to them. Every decision a human wrote down became grounds for a future answer.

One example: on October 16, a line went into the docs saying that a bug already in production gets its own ticket. After that, this type of question arrived four times, and the AI answered all four from that one line.

Two replies, concretely. On a framework upgrade ticket, asked about the store build environment and pre-merge checks, the AI replied: build once in the store environment before merging, and do not merge before the rehearsal. The engineer followed it and the run succeeded. On CI required checks, choosing between two ways to narrow them, the AI picked narrowing by job, and the engineer opened a PR with that approach.

Results

18 replies went out. None of the main answers was overturned later. Two were supplemented.

My own checking dropped to about twice a day. The engineers had no complaints; replies arrive about once an hour, and they did not seem to notice an AI was answering.

What went wrong

Judgment wobbles. On one estimate question, the recommendation changed run by run: 2 hours, 2.5 hours, 3 hours. Part of the cause was which code got read, a working tree or main. Split recommendations now stay split in the morning summary, and I decide them in one pass.

Once, the line broke. A question that required PM approval was escalated six runs in a row, then on the seventh run the AI approved it. The answer was not wrong and nothing bad happened, but a line written as instructions can break without any rule change.

The fixes: every miss becomes a new written rule, and approval-pending questions will get a blocker marker so the AI cannot touch them until the blocker clears. That second one is mechanical, not a judgment call, and it is next.

The rules, if you want to copy them

  • Reply only when documents, past tickets, or code settle the answer
  • Never reply when the question changes cost, touches personal data, legal, or security, defines new user-visible behavior, or needs PM approval
  • Attach a recommendation and a next action to everything escalated; if recommendations split, show the split and lower the confidence
  • Write every settled question and answer into the docs in one line; it becomes the next answer's evidence
  • Batch pending decisions into one morning summary instead of pinging all day