My team builds a form backend, and the most common message we get is some version of "my form stopped working". Usually the form never worked. Someone generated the site, clicked Send once, saw a green "Message sent!" and moved on. In early October we got tired of guessing how often that happens and decided to count it on sites built with Lovable and Bolt.

On paper the plan was short. Open a live site, find the contact form, fill it in with test values nobody would type for real, press Send and watch what leaves the page. If a request goes out with our text in it, the form sends. If nothing goes out and the page still celebrates, the form is fake. Every request after the click was blocked in the browser, so no site owner got a test message from us.

We wrote the rules down before the first site was tested. The test went through 793 sites and found 119 contact forms. It took six runs to get to the final answer, which was 27 fakes out of the 105 forms with a clear result.

Six runs, because the test itself kept getting things wrong. These are the five times it fooled us, more or less in the order we noticed.

1. Analytics looked like sending

Version one counted any request after the click as "the form sent something". If you've ever opened the network tab on a modern landing page, you can guess what happened. Pressing Send also pings Google Analytics, Tag Manager, Clarity, Hotjar, PostHog or Vercel's insights, and Lovable sites call their own analytics endpoint too. A fake form with three trackers on the page passed as a working one.

So "sends" got a stricter meaning. A request only counts if it carries what we typed. We picked test values that stand out, an example.com address and a phone number starting with 555, which made them easy to find in a URL or a request body. Known analytics hosts were ignored on top of that.

2. The answer was in an alert box

During the pilot, one site cleared its fields after Send and that was it. No toast, no text, no request. It looked like a form that simply does nothing. It turned out the site had shown an alert(), and the browser automation had dismissed it before we could read it.

After that, whatever a dialog said was treated as part of the page.

3. Some forms lie for two seconds

After the full run, we went through the fakes by hand. For each one we opened the site's JavaScript and read the submit handler. Most were exactly what we expected, and the pattern became familiar quickly: wait a second, show a green toast, reset the fields. No fetch, no email service, no database.

But two forms we had confirmed as fake by reading their code were classed as "nothing happened". They showed "Message sent!" for two or three seconds and then hid it. Our test read the page once, five seconds after the click, by which point the toast was gone.

The fix was to sample the page every 250 milliseconds for five seconds and keep every new line of text we saw. In the same pass we found a form that sent people to mailto: by setting location.href. Navigations like that never show up as network requests, so we hooked them separately.

4. Our test was the bot the honeypot was waiting for

This one is my favourite. A site returned a "queued" message, but its code clearly sent requests. On a closer look, the handler had a honeypot, a hidden field that only bots fill in. If the field has a value, the form quietly shows a fake success and drops the message. Our test filled every text field it could find, including that hidden one, so the site correctly treated us as a bot.

So the test stopped filling fields that a person can't see or reach. That created the next problem.

5. The wrong guess, then the real cause

After that change, three sites that had been sending started refusing our test values. We logged a diagnosis: the honeypot rule must be skipping ordinary fields named website or url. We narrowed the rule. Nothing changed.

The real cause took measuring rather than guessing. Those sites used shadcn/Radix select menus. A Radix select renders a hidden native <select>, one pixel by one pixel, marked aria-hidden, and the form validates against it. Our "never fill hidden fields" rule now skipped it, the required select stayed empty, and the form refused to submit. Honeypots are text boxes, so the final rule only skips hidden text fields, and hidden selects and checkboxes get filled.

We're including the wrong guess on purpose. It went into the log as a diagnosis, and it was only a guess.

What we checked so we could trust the result

Every time the test changed, it had to pass a set of local pages where we knew the right answer: a fake toast, a real fetch to an API, a native POST, a mailto: form, an analytics-only page, an AI chat box that isn't a contact form, and so on. By the end there were 23 of them, and the test classified all 23 correctly. We also ran a deliberately broken version of the classifier against the same pages. It got 10 of 23 right, which told us the pages could catch a broken test.

Of the 27 forms the final run called fake, we confirmed 21 by reading their submit handler. For the other 6 we couldn't find the handler in the minified bundle, so we report them as flagged by the test, not as proven.

What the numbers say

Of the 105 forms with a clear result, 54 sent the message. 27 showed a success message and sent nothing. 12 did nothing a visitor could see. 7 opened the visitor's email app, 3 moved to another page without the message, and 2 just reloaded the page.

The obvious objection is that these are demos. We had the same thought, so before counting we sorted the sites by fixed rules. Demos, with hackathon entries and placeholder phone numbers, faked their forms in 9 of 15 cases. Among the 73 sites with no signs of being a demo, 16 faked it, and so did 5 of the 30 on their own domains.

The sample has limits. It came from public listings, which lean towards developer projects. Since we blocked every request, the test shows whether a form tried to send, not whether an email arrived. And it's one moment in time.

What I took away from it

The AI tools build what a working form looks like from the outside, and a fake one looks exactly the same. You only see the difference in the network tab, or when a customer tells you they never heard back.

If you have a site with a form, open it in Chrome, press F12, go to Network, and send yourself a message. A new request with your text in it means the form works. Nothing new means it only says it does.

Have you ever written a test that turned out to be the bug? I'd like to hear your version in the comments.