URL parts explained: how to parse a URL in Python and JavaScript

TL;DR

• A URL has up to seven parts: scheme, user info, host, port, path, query, fragment. The query runs from the first ? to the first #, and the fragment never reaches the server.
• Python parses with urlparse + parse_qs, JavaScript with new URL + searchParams. They disagree on blank values, repeated keys, missing schemes, relative URLs and default ports (table below).
• Four traps cause most URL bugs: the missing scheme, the encoded slash, data hidden in the fragment, and punycode hostnames. Each one has a check you can paste, and the last section turns them into one validator.

Run parse_qs("color=red&debug=&page=2") in Python and debug is gone. Run the same query through searchParams in JavaScript and debug is still there, as an empty string.

One example URL runs through every snippet below.

The seven parts of a URL

https://shop.example.com:8443/catalog/shoes?color=red&size=41&size=42&page=2#reviews

Two details cause most "weird parameter" tickets. A ? inside a value does not start a new query, so it gets percent-encoded. And a reserved character such as & inside a value is encoded too: q=a%26b is one parameter with the value a&b, not two. Spaces show up as %20 or +, depending on who encoded them.

Parsing a URL in Python: urlparse and parse_qs

urlparse splits the URL, and parse_qs turns the query into a dict of lists. Both live in the standard library's urllib.parse.

from urllib.parse import urlparse, parse_qs, urlencode

url = "https://shop.example.com:8443/catalog/shoes?color=red&size=41&size=42&page=2#reviews"
p = urlparse(url)

p.scheme    # 'https'
p.netloc    # 'shop.example.com:8443' (keeps user:pw@ if present)
p.hostname  # 'shop.example.com' (lowercased, port removed)
p.port      # 8443 (an int, or None)
p.path      # '/catalog/shoes'
p.query     # 'color=red&size=41&size=42&page=2' (still encoded)
p.fragment  # 'reviews' (no '#')

Compare or connect with hostname and port, because netloc can carry user:pw@.

q = parse_qs(p.query)
# {'color': ['red'], 'size': ['41', '42'], 'page': ['2']}

q["size"]         # ['41', '42']
q.get("sort")     # None  (q["sort"] would raise KeyError)

urlencode({"q": "red shoes"})   # 'q=red+shoes' (spaces become +, like most forms send)

Three behaviors bite everyone once:

1. Blank values are dropped. parse_qs("color=red&debug=&page=2") loses debug unless you pass keep_blank_values=True, which brings it back as [''].
2. Every value is a list. Repeated keys fold together, and a key that appears once still gets a one-item list.
3. urlparse does not decode. p.query is still encoded, and parse_qs is the step that decodes (parse_qs("q=a%26b") gives {'q': ['a&b']}). Don't parse an already-decoded string, and don't decode twice.

Parsing a URL in JavaScript: URL and searchParams

The URL class is built into every browser and into Node, and searchParams is the friendlier way to read a query. The details are in the MDN URL docs.

const url = "https://shop.example.com:8443/catalog/shoes?color=red&size=41&size=42&page=2#reviews";
const u = new URL(url);

u.protocol  // 'https:'  (keeps the colon)
u.hostname  // 'shop.example.com'
u.port      // '8443'    ('' when it is the default port)
u.pathname  // '/catalog/shoes'
u.search    // '?color=red&size=41&size=42&page=2'  (keeps the '?')
u.hash      // '#reviews'  (keeps the '#')

u.searchParams.get("color")     // 'red'
u.searchParams.get("size")      // '41'  (first value only)
u.searchParams.getAll("size")   // ['41', '42']
u.searchParams.get("sort")      // null
new URL("https://shop.example.com/?debug=").searchParams.get("debug")   // ''  (kept)

searchParams decodes values as it reads them.

Where Python and JavaScript disagree

Each row was run on Python 3 and Node 22 or newer. Code ported across changes behavior at each one, and most without an error.

Four traps that cause most URL bugs

1. The missing scheme. Browsers hide the scheme, so people paste addresses without one. Strip https:// from the example and Python accepts the string quietly:

p = urlparse("shop.example.com/catalog/shoes?color=red&size=41&size=42&page=2#reviews")
p.netloc   # ''  (empty)
p.path     # 'shop.example.com/catalog/shoes'  (everything lands in path)

JavaScript throws on the same input. Python's version is the harder one to catch, because the bad value travels on until something further down breaks. The check goes where the URL enters your code (a ready-made one is in the last section).

2. The encoded slash. A callback URL can pass in staging and 404 in production with identical code, because some servers decode %2F early and others reject it outright. Both parsers hand back /files/a%2Fb untouched, so the difference sits in whatever server or proxy handles the request first. Some APIs expect double-encoded slashes (%252F), so read the docs of the one you call. Test each environment with the same two requests:

curl -s -o /dev/null -w '%{http_code}\n' 'https://your-host/files/a%2Fb'
curl -s -o /dev/null -w '%{http_code}\n' 'https://your-host/files/a%252Fb'

Different status codes in staging and production mean the stack in front of your app handles slashes differently.

3. Data in the fragment. The browser strips everything after # before the request leaves the machine. Picture a campaign link that carries its tracking token after the #: the analytics dashboard shows zero attributed visits, and the server log has nothing to explain it. curl strips the fragment the same way, so you can reproduce it:

curl -s -o /dev/null 'https://your-host/page?a=1#token=abc'

Grep your access log. You will find /page?a=1 and no #token=abc. Anything the server needs belongs in the path or the query.

4. Punycode hostnames. Internationalized domains travel as punycode, so münchen.example and xn--mnchen-3ya.example are the same host, and both forms show up in logs and redirects. Say your redirect allowlist was copied from a log and holds xn--mnchen-3ya.example. Your JavaScript front end sees that form, but Python reports the Unicode one, so a valid redirect gets rejected:

urlparse("https://münchen.example").hostname    # 'münchen.example'  (Unicode kept)
"münchen.example" == "xn--mnchen-3ya.example"   # False
"münchen.example".encode("idna")                # b'xn--mnchen-3ya.example'

new URL("https://münchen.example").hostname     // 'xn--mnchen-3ya.example'  (converted)

Convert both sides with IDNA before you compare.

Validate once, at the boundary

The ?debug= flag from the top disappears in Python because parse_qs drops blank values by default. Don't patch the four traps one by one where they surface. Put one function where URLs enter your code and parse everything through it:

from urllib.parse import urlparse, parse_qs

def parse_url_strict(raw):
    p = urlparse(raw)
    if p.scheme not in ("http", "https") or not p.hostname:
        raise ValueError("expected an absolute http(s) URL")
    host = p.hostname.encode("idna").decode()
    return p, host, parse_qs(p.query, keep_blank_values=True)

Then use the five inputs from the disagreement table as its test set. ?debug= must come back as {"debug": [""]}, the repeated size must give two values, a URL with no port must give port as None, and both shop.example.com/catalog and /a?b=1 must raise. Add a münchen.example case that returns the punycode host.

Try this: paste a real callback URL from your app into parse_url_strict, then remove the scheme and run it again. The second run should raise.