Search LinkedIn, company career sites and remote boards in one run, with duplicates merged

A job seeker, a recruiter or a niche job newsletter usually ends up running the same search three times: once on LinkedIn, once on the employers' own career sites, and once on the remote boards. The results overlap. The same role shows up on LinkedIn and on the company's Greenhouse or Lever page, often with the pay range on only one of them, and counting or posting both copies inflates everything downstream.

This post shows a daily search that covers all of them in one run and returns one row per job, then compares the multi-board job scrapers on the Apify Store using the prices their Store pages listed at the time of writing.

Disclosure: I'm Aurelix, and I built the Multi Job Board Scraper. It's a paid tool (prices below). Check each Store page before you buy: prices change.

What one run covers

• LinkedIn's public job search, read logged-out.
• About 50 bundled company career sites on Greenhouse, Lever, Ashby and SmartRecruiters (25 searched per run by default, home-country employers first), plus any boards you add, such as greenhouse:stripe.
• Remote OK and We Work Remotely, kept when the job is open to your country or worldwide.

Every row has the same fields whatever its source: title, company, location, city, ISO country, workplace_type, salary_min / salary_max / salary_currency where published, posted_date, job_url, apply_url and the description. A job found on two sources is one row: sources lists both, also_listed_on links the other copy, and the gaps (pay, apply link, description) are filled from it. That row is charged once.

A daily feed of new jobs, deduplicated

pip install apify-client 

import csv import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("aurelix/multi-job-board-scraper").call( run_input={ "keywords": ["data engineer", "analytics engineer"], "country": "CA", "location": "Toronto, ON", "datePosted": "past24h", "maxResultsPerSource": 50, "incremental": True, "stateKey": "toronto-data-daily", } ) jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items()) fields = ["title", "company", "city", "workplace_type", "salary_min", "salary_max", "salary_currency", "posted_date", "sources", "apply_url", "job_url"] with open("new_jobs.csv", "w", newline="", encoding="utf-8") as f: w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore") w.writeheader() for j in jobs: w.writerow({**j, "sources": "+".join(j.get("sources") or []), "apply_url": j.get("apply_url") or j.get("job_url")}) both = [j for j in jobs if len(j.get("sources") or []) > 1] paid = [j for j in jobs if j.get("salary_min")] print(f"{len(jobs)} new jobs, {len(both)} found on more than one source, {len(paid)} with a pay range") for j in both[:5]: print(f"- {j['title']} at {j['company']}: {', '.join(x['job_url'] for x in j['also_listed_on'])}") 

With incremental on, the first run returns everything that matches; each later run with the same stateKey returns only jobs no earlier run returned, so a daily schedule pays only for new jobs. To run it without your own script, save the input as a task in the Apify Console and schedule it daily; a Slack webhook URL in slackWebhookUrl sends a short digest of the new jobs and nothing when there are none.

A few input choices that matter:

• sources: ["linkedin", "careersites", "remoteboards"] by default. Set it to ["linkedin", "careersites"] to compare pay and posting date on LinkedIn and the employer's own site for the same job.
• remoteOnly: for a remote-jobs newsletter. Pick the country your readers live in.
• starterList: "canada" uses the bundled Canadian employers; "none" searches only the boards you list in companies.
• includeDescription: false: faster runs when you only count or list jobs.

Multi-board job scrapers on the Apify Store compared

Prices per 1,000 jobs and coverage, from each Store page at the time of writing:

openclawai also charges $0.01 per 1,000 dataset items, and has announced tiered pricing ($6.00 on the free plan, $4.00 on Gold); flash_scraper has announced Bronze at $5.00. For single-board searches, the most-used LinkedIn tool, curious_coder/linkedin-jobs-scraper, lists $2.00 per 1,000 jobs on the free plan and $1.00 on paid plans.

Where each wins: If you need Indeed, Glassdoor or ZipRecruiter, the Aurelix tool doesn't cover them. openclawai has by far the most users of the four. agentx covers the most platforms (42, including Indeed, Glassdoor and the Greenhouse, Lever and Ashby career sites) at a lower price than openclawai or flash_scraper. flash_scraper pairs LinkedIn, Indeed and Glassdoor with the most remote boards. The Aurelix tool is the cheapest per job of the four on every plan, and fits when the jobs you want are on LinkedIn, employers' own career sites (including SmartRecruiters) and Remote OK or We Work Remotely: it merges the copies into one row with the pay and apply link filled in, and charges each job once. Its start fee is above openclawai's and flash_scraper's, which only matters for many tiny runs.

Limits

• No Indeed, Glassdoor or ZipRecruiter. Sources are LinkedIn, the career-site platforms above and two remote boards.
• Career sites are a bundled list of about 50 employers, 25 per run by default. Add the boards of the companies you care about in companies.
• LinkedIn caps a search at about 1,000 results. For more, run one search per city.
• Pay is only what the source publishes. Many LinkedIn rows have none unless the employer's own site lists it.
• 30 countries are supported; LinkedIn rows that name another country are dropped.

Cost

At the time of writing the Store page lists $1.50 per 1,000 jobs on Apify's free plan and $0.90 per 1,000 on paid plans, plus $0.001 per run. One daily search returning about 25 new jobs costs about $1.16 a month on the free plan. Apify's free plan includes $5 of credit a month, about 3,300 jobs.

If you only need one source, the single-board tools are an option: see LinkedIn and career-site job scrapers on Apify compared, which covers the LinkedIn Jobs Scraper and the Greenhouse Jobs Scraper.

Links

• Multi Job Board Scraper on the Apify Store
• openclawai/job-board-scraper, agentx/all-jobs-scraper, flash_scraper/multi-jobboard-scraper, curious_coder/linkedin-jobs-scraper