Exporting every listing of a marketplace seller
Classic scraping found nothing on seller pages. The fix was to use the same data source the page uses, which also returned more fields.
- Problem
- The full inventory of one seller on OLX, Poland's largest classifieds marketplace, was needed in a spreadsheet. Seller pages contain no listings in their HTML.
- What runs
- One command per seller: decode the profile ID, page through the API 50 listings at a time, write the file.
- How it is watched
- Progress is printed against the total the API reports, and a seller with no listings stops with a clear message instead of an empty file.
- Stack
- Python, aiohttp, pandas and openpyxl for the export.
The problem
OLX search results are rendered on the server, so a normal HTML scraper sees them. Seller profile pages are different: about 3 MB of HTML and not one listing in it, because the list is fetched by JavaScript after the page loads. No amount of selector tweaking fixes that.
What it does
- Takes a profile URL, its short slug or a numeric ID. The slug turned out to be the user ID in base62, so no lookup request is needed.
- Pages through the marketplace's public JSON API, 50 offers at a time with a short pause between requests.
- Exports to CSV, XLSX or JSON with more than the page ever showed: full descriptions, every photo in high resolution, brand, condition, district and listing expiry date.
What it would take to run it every day
As it stands it is a one-off export. Scheduled daily, with each run compared to the previous one, it becomes monitoring: new listings, price changes and items that disappeared, sent as a short report. That step is a few dozen lines and a timer, which is the point of starting from the right data source.
What a business could reuse from this
Tracking competitor prices and stock, watching a supplier's catalogue, collecting second-hand market prices for valuations. Anywhere a website shows data you need in a sheet, there is usually a cleaner source behind it.