Technology detection API for lead enrichment
Which tool does each prospect's website run? AI scrapers gave a different answer on every run. This loads the real page, watches what it requests, and returns the vendor with the URL that proves it.
- Problem
- A sales team needed to know which chat vendor each prospect's website uses. AI-based scrapers in Clay were inconsistent and sometimes named vendors that were not on the page.
- What runs
- An HTTP endpoint called once per prospect row from Clay, scaled across two US regions.
- How it is watched
- Every answer carries its evidence. Bot challenges are reported as errors, not as "no chat".
- Stack
- Node.js, Playwright, Express, Google Cloud Run, Cloud Build on push.
- Code
- Private repository. A walkthrough is available on request.
The problem
For a sales team, knowing which tools a prospect already runs decides the pitch. Here the question was the live chat and support widget: Intercom, Zendesk, HubSpot, Gorgias and dozens of others.
Clay offers AI agents that can read a website and answer "which chat tool does this company use". In a reference test on 14 URLs with 4 models, the same URL got the same answer only about 60% of the time, and one model reported a vendor that was not on the page at all. Most chat widgets load asynchronously with JavaScript, so anything reading static HTML misses them.
What it does
- Opens the page in headless Chromium and records every network request it makes.
- Dismisses cookie consent banners, including those inside cross-origin iframes, so the widget gets a chance to load.
- Checks requests, window globals, DOM and shadow DOM against a fingerprint database of more than 80 vendors.
- Looks again for slow sites, then follows Contact, Help and FAQ links if the home page shows nothing.
- Returns the vendor, a confidence level and the evidence: the actual resource URL that identified it.
How it stays under control
- A proxy service queues requests and spreads them across backends in two regions, retrying with backoff, so Clay never sees a 5xx or a 429 during bursts.
- Identical URLs requested in parallel are scraped once; results are cached per domain for five days.
- Each page has a hard deadline, so one slow site cannot hold a worker.
- Cloudflare, AWS WAF and similar bot challenges are detected and returned as an error, so "blocked" is never mistaken for "no chat".
- Each service redeploys through Cloud Build on push, with unit tests for the fingerprints.
What a business could reuse from this
Technology detection for sales and partnerships, compliance checks across many websites, any question about a list of sites where the answer must be repeatable and provable rather than plausible.