How to Scrape Google Search Results

Scraping Google directly is a constant maintenance headache. Google updates its HTML structure every few weeks, your Playwright selectors break in production, and you end up fighting CAPTCHAs instead of building the actual product you care about. The SERP Search API removes all of that: you send a query, you get structured JSON back. This guide covers every parameter, the full response shape, and common patterns for real use cases.

What you'll need

• A SERP Search account (sign up here, takes 30 seconds)
• An API key from the dashboard
• cURL, Python, or JavaScript (any of the three works)

Your first request

The only required parameter is query. Everything else has sensible defaults.

cURL

curl -G https://api.serpsearch.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  --data-urlencode "query=best project management tools"

Python

import requests
 
response = requests.get(
    "https://api.serpsearch.com/api/v1/search",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    params={"query": "best project management tools"},
)
response.raise_for_status()
 
data = response.json()
for result in data["organic_results"]:
    print(f"{result['position']}. {result['title']}")
    print(f"   {result['url']}")

JavaScript

const res = await fetch(
  "https://api.serpsearch.com/api/v1/search?" +
    new URLSearchParams({ query: "best project management tools" }),
  { headers: { Authorization: "Bearer YOUR_API_KEY" } },
);
const data = await res.json();
data.organic_results.forEach((r) => console.log(r.position, r.title, r.url));

That's the happy path. If you're seeing a 401, your key is wrong or missing. A 429 means you've hit the rate limit. Slow down and retry with backoff.

All request parameters

The web search endpoint (GET /api/v1/search) accepts these parameters:

For most use cases you'll only use query, page, and occasionally location.

Understanding the response

A successful response looks like this:

{
  "search_info": {
    "total_results": "About 21,600,000 results",
    "time_taken": "0.29s"
  },
  "organic_results": [
    {
      "title": "Beautiful Soup: Build a Web Scraper With Python",
      "url": "https://realpython.com/beautiful-soup-web-scraper-python/",
      "website": "Real Python",
      "position": 1,
      "description": "Beautiful Soup is a Python library designed for parsing HTML and XML documents. It creates parse trees that make it straightforward to extract data from HTML...",
      "visible_url": "https://realpython.com › beautiful-soup-web-scraper-py...",
      "rating": 4.8,
      "reviews": "317",
      "duration": "12:41",
      "sitelinks": [
        {
          "title": "Parse HTML",
          "url": "https://realpython.com/beautiful-soup-web-scraper-python/#parse-html",
          "description": "Inspect and extract structured content."
        }
      ]
    }
  ],
  "knowledge_graph": {
    "title": "Beautiful Soup",
    "category": "Python library",
    "description": "A Python package for parsing HTML and XML documents.",
    "source_name": "Wikipedia",
    "source_link": "https://en.wikipedia.org/wiki/Beautiful_Soup_(HTML_parser)"
  },
  "people_also_ask": [{ "question": "Is Python good for web scraping?" }],
  "related_searches": [{ "query": "python web scraping library" }],
  "calculator_result": {
    "displayed_expression": "12 345 * 6789 =",
    "normalized_expression": "12345*6789=",
    "result": "83810205",
    "mode": "Rad"
  }
}

A few things to know before you start parsing:

The response is sparse. A field is omitted when Google did not render that module; the API does not fill the object with null placeholders. knowledge_graph shows up for branded and entity searches, local_pack for local-intent queries, and top_stories for news-heavy topics. Check whether a property exists before reading it.

The organic_results fields: position is the 1-based rank, url is the absolute publisher link, website is Google's site label, and description is only the visible snippet. When Google shows richer metadata, a result can also include answer, rating, reviews, price, duration, answer_count, date, thumbnail, favicon, and sitelinks. Optional fields are omitted when absent.

Knowledge and local data are typed: knowledge_graph can include category, numeric rating, reviews, cleaned facts, and unique image URLs. local_pack entries can include numeric rating, reviews, price_range, type, distance, address, phone, status, description, services, and website.

Rich result modules

The search endpoint now returns dedicated objects for Google's major native widgets instead of flattening them into unrelated result fields:

Read the full AI response from ai_overview.text. Headings, prose, and list content stay together in that string rather than being reclassified into a steps array; citations and products remain separate typed arrays.

A plain currency conversion is returned as currency_converter with from_currency, to_currency, from_amount, and to_amount. It does not fabricate a market_summary; market data remains reserved for actual stocks, indexes, and crypto widgets. The API reference documents every field and nested object.

related_searches is genuinely useful for keyword research. If you're building a content tool, these are the queries Google itself is recommending.

Pagination

The API returns 10 results per page, which matches Google's default. Pages are 1-based: page=1 is positions 1–10, page=2 is 11–20.

Here's a Python snippet that collects the first 30 results across 3 pages:

import requests
import time
 
API_KEY = "YOUR_API_KEY"
all_results = []
 
for page in range(1, 4):
    resp = requests.get(
        "https://api.serpsearch.com/api/v1/search",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params={"query": "python web scraping", "page": page},
    )
    resp.raise_for_status()
    all_results.extend(resp.json()["organic_results"])
    time.sleep(1)  # stay within rate limits on Starter plan
 
print(f"Collected {len(all_results)} results")

The time.sleep(1) matters. Lower plans are rate-limited to 1 req/s, so fetching 10 pages without a delay will get you a 429. Higher plans have higher limits. Check the pricing page for the specifics.

Geo-targeting

There are two ways to pin results to a location.

Option 1: location string. Easiest to use, readable in logs.

curl -G https://api.serpsearch.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  --data-urlencode "query=coffee shops" \
  --data-urlencode "location=London, England, UK"

Option 2: lat / lng coordinates. More precise, good for hyperlocal queries.

curl -G https://api.serpsearch.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  --data-urlencode "query=coffee shops" \
  -d "lat=51.5074&lng=-0.1278"

The main gotcha with location: format sensitivity. "New York, NY, US" works reliably. Vague strings like "United States" or "NYC" might not produce meaningfully localised results. If you're building a feature where users type their own location, validate the string against the autocomplete endpoint first rather than passing raw user input.

International and localized search

Beyond location and lat/lng, the API offers several parameters for fine-grained control over where and how Google returns results.

gl and hl are the most commonly used. gl sets the country context (ISO 3166-1 alpha-2 code) and hl sets the interface language (ISO 639-1 code). Together they control the "Google experience" your query runs against:

curl -G https://api.serpsearch.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  --data-urlencode "query=best restaurants" \
  -d "gl=de" -d "hl=de"

google_domain lets you query a specific regional Google domain directly, e.g. google.de, google.co.uk, or google.co.jp. This is useful when you need results exactly as they appear on that domain:

curl -G https://api.serpsearch.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  --data-urlencode "query=best restaurants" \
  -d "google_domain=google.de"

lr and cr provide stricter filtering. lr restricts results to pages in a specific language (lang_en, lang_de), while cr restricts to pages from a specific country (countryUS, countryDE). These are more aggressive than gl/hl — use them when you need hard filters rather than preference signals.

uule accepts a pre-encoded UULE string for precise geo-targeting at the city or neighborhood level. This is the same parameter Google uses internally. If you already have UULE strings from another tool, pass them directly.

json_restrictor filters the response JSON to only include specific fields. Pass a comma-separated list of keys: json_restrictor=local_pack,organic_results[0] returns only those fields. This is useful when you only need a subset of the response and want to reduce payload size.

Response formats

The default response_type=json is almost always what you want. The other options exist for specific cases:

• html: returns the raw rendered page HTML. Useful when you need elements the parser doesn't capture, or when you want to do your own extraction.
• both: returns the parsed JSON and the raw HTML in the same response.
• correlated: attempts to link parsed JSON fields back to their source HTML elements. Handy for debugging but not needed for typical data extraction.

The js flag only applies when you're requesting HTML. Set js=true to preserve script tags; the default strips them. For nearly every use case, ignore both of these parameters and stick with json.

Handling errors

The API uses standard HTTP status codes:

A minimal retry wrapper in Python:

import requests
import time
 
def search(query, page=1, retries=3):
    for attempt in range(retries):
        resp = requests.get(
            "https://api.serpsearch.com/api/v1/search",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            params={"query": query, "page": page},
        )
        if resp.status_code == 429:
            time.sleep(2 ** attempt)  # 1s, 2s, 4s
            continue
        resp.raise_for_status()
        return resp.json()
    raise RuntimeError(f"Failed after {retries} retries")

What you can build with this

A few real-world use cases that work well with this API:

Rank tracker. Query your target keywords daily, store position for each URL, and chart movement over time. 10 keywords × daily = 300 calls/month, well within the Starter plan.

Competitor monitoring. Watch where your competitors rank for shared keywords. When a competitor jumps to position 1 on a keyword you care about, you'll know immediately.

Content research. Use related_searches and people_also_ask to surface what your audience is actually searching for. These are Google's own suggestions, which makes them reliable signals.

SERP feature tracking. Check whether you're winning an inline answer, AI Overview citation, knowledge panel, or local pack on a given keyword. The organic_results[].answer, ai_overview, knowledge_graph, definition_result, and local_pack fields tell you what's appearing above the fold.

Next steps

• Full endpoint reference (news, images, videos, maps, reviews): API docs
• Language and framework examples: Examples page
• Pricing: the free tier works for testing, Starter covers most ongoing projects