Who's Hiring for This Job Title, and What Do They Pay? Searching 37,000 Company Career Sites With Python

The earlier posts in this series start from a list of companies. Pulling job postings straight from Greenhouse, Lever, Ashby and Workday shows the raw ATS endpoints, and getting a Slack alert when a company posts a new job turns that into a daily alert for the companies you're watching.

This post covers the opposite question: you know the job title, and you want to find out which companies are hiring for it and what they pay. You can't loop over a company list for that, because you don't know the companies yet. You have to search by title across every career site you can reach.

Why search the employer's career site at all

Job boards repost listings late, show the same role five times and keep jobs up after the employer has closed them. The employer's own applicant tracking system (ATS) is the source of truth: the job shows up there the day it opens and disappears the day it closes. Since many US states now require pay ranges in postings, the ATS is also where the salary data is.

The catch is that "every career site" means tens of thousands of separate boards on a dozen ATS platforms, each with its own field names.

Part 1: a title matcher that isn't too literal

Searching by title only works if the matcher handles real titles. "engineer" should match Engineers and Engineering Manager; "data engineer" should match Senior Data Engineer, Platform; and you need a way to say "exactly this phrase". This version matches a term when the title contains all of its words, on word starts, so plurals and "-ing" forms count. Quoted terms are exact phrases.

import re def compile_term(term): term = term.strip() if term.startswith('"') and term.endswith('"'): return [re.compile(rf"(?<!\w){re.escape(term[1:-1])}(?!\w)", re.I)] return [re.compile(rf"(?<!\w){re.escape(w)}", re.I) for w in term.split()] def matches(title, terms, exclude=()): if any(re.search(rf"(?<!\w){re.escape(x)}", title, re.I) for x in exclude): return False return any(all(p.search(title) for p in compile_term(t)) for t in terms) print(matches("Senior Data Engineer, Platform", ["data engineer"])) # True print(matches("Engineering Manager", ["engineer"])) # True print(matches("Data Engineering Intern", ["data engineer"], ["intern"])) # False print(matches("Analytics Engineer II", ['"analytics engineer"'])) # True 

Part 2: count roles, not postings

If you're counting who is hiring, duplicates skew everything. Employers often post one role in several cities, so a company with one opening can look like it has eight. A key of company plus normalized title is usually enough to collapse them:

def role_key(job): title = re.sub(r"\W+", " ", job["title"].lower()).strip() return (job["company"].lower(), title) def unique_roles(jobs): seen, out = set(), [] for j in jobs: if role_key(j) not in seen: seen.add(role_key(j)) out.append(j) return out 

Part 3: normalize the pay

Pay ranges come in every shape: structured fields on some ATS platforms, free text like "$70K - $100K" or "$55/hour" in the description on others. For a benchmark you want one shape: a minimum and maximum per year, in one currency, with a note of where the number came from. Hourly rates need annualizing (2,080 hours is the usual convention), and anything you parse from text deserves less trust than a structured field.

Doing the search part without building a crawler

Parts 1 to 3 are a few dozen lines. The part that isn't is the index: a list of tens of thousands of board identifiers, a parser per platform, a daily refresh with retries and rate limits, and a store you can search in seconds. That's a crawler and a database, not a script.

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Job Search API: Career Site Job Listings (steady-scrape/career-jobs-search) is an Apify Actor that searches an index of 1.3 million+ open jobs from 37,000+ company career sites (Greenhouse, Lever, Ashby, Workday, BambooHR, Workable, Recruitee, Personio, Rippling, Teamtailor, Breezy HR, Gem and Oracle Cloud HCM), rebuilt daily from the employers' public job board APIs. It uses the same title rules as the matcher above and removes duplicates. With enrichJobs on, it reads each returned job live from the employer's board and adds an enrichment object: an annualized salary range with its source, a list of skills and tools, seniority, years of experience, work arrangement and a short summary. Recruiter names, emails and phone numbers are stripped.

Here's a small salary and skills benchmark for one title:

import os from collections import Counter from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("steady-scrape/career-jobs-search").call(run_input={ "keywords": ['"analytics engineer"'], "countries": ["US"], "postedWithinDays": 14, "salaryOnly": True, # only postings that publish pay "enrichJobs": True, # salary, skills, seniority "maxResults": 50, }) jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items()) skills = Counter() for j in jobs: e = j.get("enrichment") or {} pay = e.get("salary") or {} skills.update(e.get("skills", [])) print(f'{j["title"]} | {j["company"]} | {j["locationText"]} | ' f'${pay.get("annualMin"):,}-${pay.get("annualMax"):,} ({pay.get("source")})') print(skills.most_common(5)) 

What came back

We ran it on November 9, 2026 with the run's maximum charge capped at $0.02, so it stopped after 3 jobs. Seven US postings with a published pay range matched in the previous 14 days. One had closed by the time the Actor fetched it, so it was skipped and not charged. The first three:

The skills extracted from those three descriptions overlapped a lot: dbt and SQL were in all three, and Snowflake and Looker were in two, alongside BigQuery, Airflow, Mixpanel and Python. Three jobs aren't a benchmark, but the same script with a higher cap gives you one for any title and country. The run took about 10 seconds, and its OUTPUT record reported an index of 1,373,175 jobs from 37,515 career sites, refreshed that morning.

Each job also has the same normalized fields regardless of ATS: title, company, department, locationText, parsed locations with country codes, isRemote, workplaceType, employmentType, salary (when the employer publishes it), postedAt, jobUrl and applyUrl.

A few honest limits:

• Titles, not full text. Keywords match job titles (and optionally departments). Descriptions can be added to results, but they aren't searched.
• Coverage is the supported ATS platforms. Companies with a custom career site or another ATS aren't in the index. If you already have a fixed list of companies, ATS Job Feed reads named boards live instead (that's the tool behind the Slack alert post).
• Pay is what the employer wrote. Check salary.source: a structured ATS field is more reliable than a range parsed from text. Postings without pay are excluded when salaryOnly is on, so the sample leans toward states and companies that publish ranges.
• Some platforms don't publish everything. BambooHR and Rippling don't publish a posting date, and Workday only gives relative ages.

What it costs

On Apify's free plan the price is $0.002 per job returned ($2 per 1,000), plus Apify's $0.00005 start fee per run. On paid Apify plans it's $0.0017 to $0.0013 per job.

Enrichment is an optional add-on: +$0.004 per enriched job on the free plan ($0.0036 to $0.003 on paid plans), charged only when enrichJobs is on and the job comes back with its enrichment object. An enriched job therefore costs $0.006 in total on the free plan.

Examples at free-plan prices: the 50-job benchmark above, uncapped, would cost at most 50 × $0.006 = $0.30. A one-off pull of 10,000 jobs without enrichment is $20. You can cap any run with Max results and the run's maximum charge.

Other uses for the same search

• Sales signals: a company hiring its first "data engineer" or a "Salesforce administrator" is a likely buyer for related services.
• Niche job boards and newsletters: fill a board with fresh roles that link straight to the employer's apply page.
• Labor-market research: count unique roles by title, country, remote share or ATS.

If you try a title the matching rules don't handle well, tell us in the comments.