GPT Web Scraper: Describe It, Extract It

Traditional web scraping is fragile. CSS selectors break when sites update. XPath expressions need constant maintenance. Every new website means new parsing logic.

GPT-4 Vision changed the game. Show it a webpage, describe what you want, get structured data. No selectors. No parsing. Just language.

How It Works

You provide a URL and a prompt: "Extract all product names and prices from this page."

The scraper: 1. Renders the page with Playwright (handles JavaScript) 2. Captures a screenshot 3. Sends to GPT-4 Vision with your extraction prompt 4. Returns structured JSON

The AI understands layout, context, relationships. It finds the data you described regardless of how the HTML is structured.

Why Vision Matters

Text-based scraping misses context. A price next to a product name makes sense visually — the DOM might have them in completely different branches.

Vision models see the page like humans do. Proximity matters. Visual hierarchy matters. The extraction prompt can reference what things look like, not just what elements contain.

Use Cases

• Competitive intelligence — Monitor competitor pricing
• Research automation — Extract data from unfamiliar sources
• Content aggregation — Pull structured data from unstructured pages
• Rapid prototyping — Skip the selector engineering phase

Purpose: Natural language web data extraction

Stack: Python, GPT-4 Vision, Playwright, BeautifulSoup

Approach: Screenshot + prompt = structured JSON

Link: GitHub