Your URL fetcher is probably an SSRF vulnerability

I spent last week building a site that lets anyone type in a URL and have my server go and look at it. Security header scanner, link expander, that sort of thing.

About an hour in I realised I had built a machine for attacking my own server, and had been quite pleased with myself about it.

Here is the line of code responsible. It appears in a great many codebases:

requests.get(user_supplied_url)

That looks like fetching a URL. What it actually is, is letting a stranger make HTTP requests from inside your network, using your server's credentials and its position behind your firewall.

The address that ruins your day

Try imagining what happens when the user types this:

http://169.254.169.254/latest/meta-data/

On AWS, GCP and Azure, that address is the instance metadata service. It's link-local, so it isn't routable from the internet — only from the machine itself. Which is exactly why your server can reach it and the attacker can't.

Until your code fetches it for them and hands back the response.

On an EC2 instance with IMDSv1 enabled, /latest/meta-data/iam/security-credentials/ returns temporary AWS credentials for whatever role the instance has. Not a hint of them. The actual keys.

In fairness: AWS's IMDSv2 requires a PUT to get a session token first, which defeats a naive GET like this, and it's the default on newer instances. But IMDSv1 is still switched on in an enormous number of accounts, GCP and Azure have their own metadata quirks, and "the cloud provider probably fixed it" is a poor foundation for your security model.

And metadata is only the most dramatic target. The same line reaches:

This is Server-Side Request Forgery, it's in the OWASP Top 10, and it is depressingly easy to introduce by accident. If your app does link previews, webhook testing, avatar imports by URL, RSS fetching, PDF generation from a URL, or image proxying — you have this code somewhere.

The naive fix, and why it fails

Most people reach for something like this:

from urllib.parse import urlparse
import ipaddress, socket

def is_safe(url):
    host = urlparse(url).hostname
    ip = ipaddress.ip_address(socket.gethostbyname(host))
    return not (ip.is_private or ip.is_loopback)

Better than nothing. Still broken, in three separate ways.

1. It only checks the first address

socket.gethostbyname returns one address. A hostname can resolve to several, and an attacker who controls DNS can return one public address and one private one. You validate the public one; the connection picks whichever it likes.

You need getaddrinfo and you need to check every address it returns.

2. It forgets most of the address space

is_private and is_loopback miss link-local — which is where 169.254.169.254 lives. It's the single most valuable SSRF target and the naive check sails straight past it.

The full set worth blocking:

ip.is_private or ip.is_loopback or ip.is_link_local
    or ip.is_multicast or ip.is_reserved or ip.is_unspecified

Python's ipaddress module already classifies all of these. Use it rather than hand-writing CIDR lists that go stale.

3. It only checks the URL the user typed

This is the one that gets people, and it's worth dwelling on.

You validate https://totally-innocent.com. It's public, it passes, you connect. The server replies:

HTTP/1.1 302 Found
Location: http://169.254.169.254/latest/meta-data/

Your HTTP library helpfully follows the redirect. Your check ran once, on a URL that was never the target.

Every hop needs re-validating, not just the first. That means turning off automatic redirects and following them yourself.

What a working version looks like

Three things have to be true: check every resolved address, block the full set of internal ranges, and re-validate on every redirect.

import ipaddress, socket, urllib.request, urllib.error
from urllib.parse import urljoin, urlparse, urlunparse

class SafeFetchError(Exception):
    pass

def is_blocked_ip(ip_str):
    try:
        ip = ipaddress.ip_address(ip_str)
    except ValueError:
        return True                      # unparseable: refuse, don't guess

    # ::ffff:127.0.0.1 must be judged on the IPv4 address it represents
    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped:
        ip = ip.ipv4_mapped

    return (ip.is_private or ip.is_loopback or ip.is_link_local
            or ip.is_multicast or ip.is_reserved or ip.is_unspecified)

def resolve_and_check(hostname):
    try:
        infos = socket.getaddrinfo(hostname, None)
    except socket.gaierror:
        raise SafeFetchError("Could not resolve %r." % hostname)

    for info in infos:                    # EVERY address, not just the first
        addr = info[4][0]
        if is_blocked_ip(addr):
            raise SafeFetchError(
                "%r resolves to %s, which is internal." % (hostname, addr))

Then suppress automatic redirects so you can inspect each one:

class _NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None                       # handle them ourselves

def fetch(url, timeout=8, max_redirects=5, max_body=65536):
    opener = urllib.request.build_opener(_NoRedirect)
    current = normalise_url(url)

    for _ in range(max_redirects + 1):
        parsed = urlparse(current)
        resolve_and_check(parsed.hostname)          # <- runs on every hop

        req = urllib.request.Request(current, method="GET")
        try:
            resp = opener.open(req, timeout=timeout)
            return {"url": current,
                    "status": resp.status,
                    "headers": {k.lower(): v for k, v in resp.headers.items()},
                    "body": resp.read(max_body)}    # cap it
        except urllib.error.HTTPError as e:
            if e.code in (301, 302, 303, 307, 308):
                location = e.headers.get("Location")
                if not location:
                    raise SafeFetchError("Redirect with no destination.")
                current = normalise_url(urljoin(current, location))
                continue
            raise

    raise SafeFetchError("Too many redirects.")

Two more things belong in normalise_url:

if parsed.scheme not in ("http", "https"):
    raise SafeFetchError("Only http and https.")

file://, gopher:// and dict:// are standard SSRF escalation paths. Allow-list the two schemes you want rather than trying to enumerate the bad ones.

And rebuild the authority from host and port only, dropping anything else:

netloc = host if not parsed.port else "%s:%d" % (host, parsed.port)
return urlunparse((parsed.scheme, netloc, parsed.path or "/", "", "", ""))

That quietly strips embedded credentials, so a URL like https://user:pass@example.com/ doesn't forward someone's password to the target.

Test it like an attacker

Assertions beat intentions. Every one of these must be refused:

BLOCKED = [
    "http://127.0.0.1/", "http://localhost/", "http://[::1]/",
    "http://169.254.169.254/latest/meta-data/",   # cloud metadata
    "http://10.0.0.1/", "http://192.168.1.1/", "http://172.16.0.1/",
    "http://0.0.0.0/",
    "file:///etc/passwd", "gopher://x/", "dict://x:11/",
]

for url in BLOCKED:
    try:
        fetch(url)
        raise AssertionError("SHOULD HAVE BEEN BLOCKED: " + url)
    except SafeFetchError:
        pass                              # correct

Then the redirect case, which is the one worth being paranoid about — stand up a server that 302s to 169.254.169.254 and confirm you don't follow it.

What this does not fix

I'd rather say this plainly than let you think you're done.

DNS rebinding. There is a gap between your check and your connection. An attacker running a DNS server with a one-second TTL can answer with a public address for the check and a private one for the connection. Closing it properly means resolving once, connecting to that pinned IP, and passing the hostname separately for TLS SNI and the Host header. That's awkward with urllib, so the code above doesn't. For most apps this check is proportionate; if your input is genuinely hostile, pin the address.

Your own public IP. If your server has a public address and also runs something private on it, none of this helps — the address isn't in a private range. Use a firewall too.

Being a proxy. If you return the fetched body verbatim to the user, you've built an open proxy regardless. Return only what you need.

Rate limiting. This stops you reaching inside. It doesn't stop someone using your server to hammer a third party.

Take it

I pulled this out of my own project into a single dependency-free file, MIT licensed:

github.com/Rehanfaisal/ssrf-safe-fetch

safe_fetch.py is one file with no dependencies — copy it into your project. The test suite covers every blocked range, IPv4-mapped IPv6, scheme rejection and credential stripping.

It came out of IsSiteSafe, where the whole premise is that strangers type in URLs and my server goes and looks at them. Nothing sharpens your interest in SSRF like shipping the exact feature that causes it.

If your app fetches user-supplied URLs, go and try http://169.254.169.254/latest/meta-data/ against your own staging environment right now. Finding out it works is a much better afternoon than finding out from someone else.