I have been scraping catalogs, prices, product cards and listings for years. All that time a small homegrown HTML parser did the work: millions of pages, no complaints. It never went public because its tests covered only my own sites and there were no docs.

Last week I cleaned it up and released it as z3l.FastHtmlParser. The first version answered "how do I find a tag fast". Since then it has gone through five releases in one week, driven by feedback: XPath 1.0, CSS selectors, a review against hostile markup, and parsing straight from a char[] buffer. This post covers the idea, the numbers and the trade-offs.

dotnet add package z3l.FastHtmlParser 

A DOM is more than a scraper needs

In a high-volume scraping service the parser is the hottest loop, and your server bill grows with it.

HtmlAgilityPack and AngleSharp are excellent libraries, but they are built to be browsers. They create an object for every node, synthesize the implicit <html>, <head> and <tbody>, and keep a mutable tree so you can edit it and walk it with selectors.

A scraper does far less: find an element, read its text or an attribute, move on. So the idea is simple: don't build what you won't use. The document is the source string plus a flat list of tags, each one a set of integer offsets into that string.

The numbers

BenchmarkDotNet 0.15.8, .NET 10, Intel Xeon E-2236, Windows Server 2025. Pages are real saved pages of books.toscrape.com, quotes.toscrape.com and webscraper.io test sites. AngleSharp 1.10.0, HtmlAgilityPack 1.13.1.

Parse a page, time (lower is better):

Page FastHtmlParser HtmlAgilityPack AngleSharp
quotes, 11 KB 17 μs 155 μs 229 μs
laptops, 38 KB 53 μs 544 μs 959 μs
books, 51 KB 59 μs 610 μs 958 μs
books, no end tags* 59 μs 671 μs 949 μs
books ×10, 510 KB 0.61 ms 19.6 ms 12.2 ms

Parse a page, memory allocated:

Page FastHtmlParser HtmlAgilityPack AngleSharp
quotes, 11 KB 11 KB 243 KB 189 KB
laptops, 38 KB 31 KB 713 KB 608 KB
books, 51 KB 39 KB 812 KB 663 KB
books, no end tags* 39 KB 756 KB 644 KB
books ×10, 510 KB 391 KB 8.1 MB 6.3 MB

* The books page with all 75 </li> and 60 </p> removed, as on sloppy sites.

Parse and extract 20 books (title, price, rating). All libraries return exactly the same records; the setup checks it.

Library Time Allocated
z3l.FastHtmlParser 69 μs 45 KB
… with CSS selectors 75 μs 45 KB
… with XPath 109 μs 173 KB
HtmlAgilityPack (XPath) 760 μs 1,038 KB
AngleSharp (CSS selectors) 1,043 μs 707 KB

The interesting row is the CSS one: the same selectors AngleSharp runs are about 14x faster here, with 16x less memory. You don't have to learn a new API to get most of the speed.

On a real 1 MB Amazon search results page (100+ inline scripts, ~4,000 tags) parsing takes 0.65 ms against 11.3 ms (HtmlAgilityPack) and 15.1 ms (AngleSharp), and extracting 27 products takes 0.87 ms against 16.3 ms and 18.1 ms, with 23–29x less memory. That page isn't in the repo for copyright reasons; save your own as benchmarks/local/amazon*.html and run LocalScrapeBenchmarks.

What the API looks like

Three ways to query, pick whichever you like. All of them work on the document or inside a tag.

using z3l.FastHtmlParser; var doc = new HtmlParser(html, "https://books.toscrape.com/"); // 1. Find / FindAll: enums for tag and attribute names, no allocations while matching foreach (var book in doc.FindAll(new TagQuery(HtmlTag.Article) { Class = "product_pod" })) { var title = book.Find(HtmlTag.H3).Find(HtmlTag.A).AttributeByName(HtmlAttribute.Title); var price = book.Find(HtmlTag.P, cls: "price_color").Text(); var link = book.Find(HtmlTag.A).AttributeHref(); // absolute URL, resolved against the page } // 2. CSS selectors, as querySelectorAll / querySelector in a browser doc.Css("article.product_pod h3 > a"); doc.Css("div.quote:has(a.tag[href*=love]) span.text"); doc.CssFirst("ul#menu > li:last-child a")?.AttributeHref(); // 3. XPath 1.0 doc.XPathValues("//a[contains(@href, '/product/')]/@href"); doc.XPathValue("//dt[.='Price']/following-sibling::dd[1]"); // compile once, reuse on any number of pages and threads static readonly CssQuery Prices = CssQuery.Compile("div.card span.price"); page.Css(Prices); 

Chains are null-safe: every method accepts null and returns null, so a missing element never throws in the middle of a chain.

var phone = doc.Find(HtmlTag.Div, id: "contacts") .Find(HtmlTag.A, cls: "phone") .AttributeHrefTelephone(); // null if any step is missing 

How it works

1. A tag list instead of a tree

Each tag stores only integers: where it starts and ends, where its inner content starts and ends, its depth and its parent. Text, inner HTML and attributes are computed on demand as slices of the source. Descendants of a tag are simply the tags that follow it in the list until one starts past its end, so "search inside a tag" is a plain indexed for loop over a range.

2. Memory allocated once, at the exact size

'<' can't appear fewer times than there are tags, so a vectorized Count('<') gives an upper bound. Working arrays are rented from ArrayPool<T>.Shared, the final list is created once with CollectionsMarshal.SetCount, and the rented arrays are cleared before they go back to the pool, so they don't keep the page alive. No List<T> doubling, no copies.

3. Vectorized scanning

Outside a tag only '<' matters, found with vectorized IndexOf. Inside a tag only '<', '=' and '>' matter, found with IndexOfAny. Quoted attribute values are skipped as a block, so data-x="<p>...</p>" doesn't break anything. script, style, textarea, title and the other raw-text elements are scanned straight to their end tag, as HTML5 says.

4. Tag names without strings

Names are ReadOnlySpan<char> looked up through Dictionary<string, HtmlTag>.AlternateLookup<ReadOnlySpan<char>> (new in .NET 9) and turned into an enum. I tried FrozenDictionary and dropped it: about 16 ms of JIT warm-up on first use against ~2 ms for a plain Dictionary, and slower lookups on this workload. A custom comparer hashes the length plus three case-folded characters. The same trick drives the table of all 2,125 HTML5 named entities.

5. Queries without IEnumerable

Find, Parent, Next and NextSibling return a single tag through an indexed loop; TagQuery is a struct passed by in. A chain like doc.Find(...).Find(...).NextSibling(...) allocates nothing on the heap; the only allocation is the final Text() string.

Broken HTML

Clean test pages are easy. Real pages leave out </li>, </p>, </td> and </option>, and a naive parser then nests every <li> inside the previous one.

I tried three fixes:

  1. Full HTML5 implicit-close rules in the main loop. Correct, but 4–9% slower.
  2. Fast pass, then a local re-parse on trouble. Fell apart on the books page without end tags: all 20 books sit in one <ol>, so most of the page was rescanned. 61 μs became 119 μs.
  3. Check the top of the open-tag stack when a tag opens. If a new <li> opens while an unclosed <li> is on top, they're siblings: close the previous one. One integer comparison. This is the one that shipped.

The result is in the table above: the books page with all 75 </li> and 60 </p> removed parses in the same 59 μs as the original.

Then came a review against hostile and random markup (version 1.3.0), and it found real bugs:

  • <a title="see id=5" id="real"> reported the id as 5: attribute lookup now never looks inside another attribute's quoted value.
  • <br/> without a space was named br/.
  • Thousands of unclosed tags followed by stray end tags took seconds. Parsing is now linear on any markup.
  • In 1.4.1, XPath like //div//span//a on deeply nested or unclosed markup was quadratic: 20,000 nested tags took 16 s and gigabytes of memory. Now 3 ms.

The test suite runs every saved page and a set of broken-markup snippets through AngleSharp too, and checks that both find the same elements, texts, attribute values and parents, CSS selector results included. The only differences are the trees a browser builds implicitly, like an inserted <tbody>.

Parsing from a buffer

If your HTTP client already reads into a pooled char[], you don't have to make a string out of it:

var owner = MemoryPool<char>.Shared.Rent(maxChars); int length = reader.Read(owner.Memory.Span); using var page = HtmlParser.Parse(owner, length, url); // Dispose returns the buffer to the pool 

When to use something else

  • You need to modify the document. It's read-only (which is also why it's safe to query from many threads).
  • You need the browser's implicit tree. No synthesized <html>, <head> or <tbody>; table > tr matches as written in the source.
  • You're on .NET 8 or older. The design depends on AlternateLookup, so it targets .NET 9 and .NET 10 only.
  • You need JavaScript-rendered content. It's a parser, not a browser.

A week ago CSS selectors and XPath were on this list. They aren't anymore.

License

Free for individuals and for companies with under $1M annual revenue and fewer than 100 employees, commercial use included. Larger companies pay once: $1,000 per organization, perpetual, all versions, and it covers every z3l library. The source is on GitHub.

Links

I'd like to hear where it breaks: send a page it parses differently from your browser, and I'll add it to the tests.