Skip to content

urllib.parse: port validation at parse time (gh-88037) breaks multi-host URLs; backported to 3.14 and 3.15 #161987

Closed

Description

@thejesh23

Since gh-159122, urlsplit(), urlparse(), urljoin() and urldefrag() raise ValueError for any netloc whose port isn't a valid integer. That breaks URLs that worked before:

>>> from urllib.parse import urlsplit
>>> urlsplit('mongodb://h1:27017,h2:27017/db')
ValueError: Port could not be cast to integer value as '27017,h2:27017'

The same happens with multi-host URLs for PostgreSQL (postgresql://u@h1:5432,h2:5433/db), Kafka and Redis Sentinel, with pip's git+ssh://git@github.com:org/repo.git, and with templates like http://localhost:{port}/. Code that only reads .hostname or .path used to work.

The change was backported to 3.14 (gh-161621) and 3.15 (gh-161619), but isn't released yet (3.14.10 and 3.15.2 don't have it), so it would first ship in 3.14.11 and 3.15.3. Could the backports be reverted, since this is a behaviour change in bugfix releases? For main, maybe a deprecation period first.

I've opened PRs that revert the two backports.

Linked PRs

Activity

  1. thejesh23 commented on Nov 14, 2026

    @thejesh23
    ContributorAuthor

    @orsenthil Can you please take a look at this regression ? introduced in #159122

  2. added
    3.14bugs and security fixes
    3.15bugs and security fixes
    on Nov 14, 2026
  3. orsenthil commented on Nov 14, 2026

    @orsenthil
    Member

    Thanks for raising and sharing the details. Surprised about the multi-host formats. I will take this up.

  4. orsenthil commented on Nov 14, 2026

    @orsenthil
    Member

    Hi Thejesh,
    Thanks for reporting this. I had to dig a bit into the RFC spec to understand what are multi-host formats and how are they expected to be parsed.

    I understand, the previous "lenient" behavior of urllib was removed by the recent changes and it was intentional and keep in reference only with RFC expectations. And yes, many real world behavior differ from RFC and in those cases, I often consult with libcurl or curl behavior as reference for urllib to lean upon. This was been my guidance.

    RFC 3986 3.2:
    
    authority = [ userinfo "@" ] host [ ":" port ]
    host      = IP-literal / IPv4address / reg-name
    reg-name  = *( unreserved / pct-encoded / sub-delims )   ; sub-delims includes ","
    port      = *DIGIT
    
    • mongodb://h1,h2/db (no ports) is valid. The host is the single reg-name "h1,h2", because commas are allowed in a reg-name.
    • mongodb://h1:27017,h2:27017/db is not valid. A reg-name can't contain :, so the host ends at h1 and the port becomes 27017,h2:27017, which isn't *DIGIT.

    Browsers will fail to parse these too.

    Tried this on devconsole in Chrome.

    for (const s of [
      "mongodb://h1:27017,h2:27017/db",
      "postgresql://u@h1:5432,h2:5433/db",
      "mongodb://h1,h2/db",
      "git+ssh://git@github.com:org/repo.git",
      "http://localhost:{port}/",
      "http://h1:8080/",
    ]) {
      try {
        const u = new URL(s);
        console.log("OK  ", s, "-> hostname=%o port=%o", u.hostname, u.port);
      } catch (e) {
        console.log("FAIL", s, "->", `${e.name}: ${e.message}`);
      }
    }
    

    And Node

    ~ ❯ node
    Welcome to Node.js v26.13.0.
    Type ".help" for more information.
    > ...
    FAIL mongodb://h1:27017,h2:27017/db -> TypeError: Invalid URL
    FAIL postgresql://u@h1:5432,h2:5433/db -> TypeError: Invalid URL
    OK   mongodb://h1,h2/db -> hostname=%o port=%o h1,h2 
    FAIL git+ssh://git@github.com:org/repo.git -> TypeError: Invalid URL
    FAIL http://localhost:{port}/ -> TypeError: Invalid URL
    OK   http://h1:8080/ -> hostname=%o port=%o h1 8080
    undefined
    

    I also summoned on claude for how those parse these "in-application" format URLs.

    • PyMongo (pymongo/uri_parser_shared.py) splits the string by hand and doesn't use urlsplit(). The only thing it imports from urllib.parse is unquote_plus.
    • Node driver (mongodb-connection-string-url/src/index.ts) works around the WHATWG parser. pulls out the hosts group with a regex.
    • How PostgreSQL (libpq) parses them - src/interfaces/libpq/fe-connect.c, function conninfo_uri_parse_options() (around line 6856 on master). It's a hand-written character scanner.

    So, these special parsers do not expect urllib to handle these formats for them, and rejection moved it closer to the what node/ browsers do.

    That said, while investigating I did find a use case that is commonly used which is not muti-host, but used by pip to install from github git+ssh://git@github.com:org/repo.git - I will see how pip handles this and see if this is a regression.

  5. orsenthil commented on Nov 14, 2026

    @orsenthil
    Member

    That said, while investigating I did find a use case that is commonly used which is not muti-host, but used by pip to install from github git+ssh://git@github.com:org/repo.git - I will see how pip handles this and see if this is a regression.

    ~/pip-repro ❯ which pip
    /home/senthil/pip-repro/venv-head/bin/pip
    
    ~/pip-repro ❯ pip --version
    pip 27.0.dev0 from /home/senthil/pip-repro/pip-src/src/pip (python 3.16)
    
    # Commonly expected URL format - works.
    
    ~/pip-repro ❯  pip install "git+ssh://git@github.com/pypa/sampleproject.git"
    Collecting git+ssh://****@github.com/pypa/sampleproject.git
    ...
    Installing collected packages: peppercorn, sampleproject
    Successfully installed peppercorn-0.6 sampleproject-4.0.0
    
    # Change since https://github.com/python/cpython/pull/159122
    # On an _invalid_ URL
    
    
    ~/pip-repro ❯  pip install "git+ssh://git@github.com:pypa/sampleproject.git"
    ERROR: Exception:
    Traceback (most recent call last):
    
      File "/home/senthil/git/cpython/Lib/urllib/parse.py", line 166, in _validate_port
        raise ValueError(f"Port could not be cast to integer value as {port!r}")
    ValueError: Port could not be cast to integer value as 'pypa'
    

    The stable version of pip will fail with a different error and not ValueError

    ~ ✗ pip --version
    pip 26.3 from /home/senthil/.local/share/mise/installs/python/3.15.2/lib/python3.15/site-packages/pip (python 3.15)
    
    ~ ❯ pip install "git+ssh://git@github.com:pypa/sampleproject.git"
    Collecting git+ssh://****@github.com:pypa/sampleproject.git
      Cloning ssh://****@github.com:pypa/sampleproject.git to /tmp/pip-req-build-re2_lzbd
      Running command git clone --filter=blob:none --quiet 'ssh://****@github.com:pypa/sampleproject.git' /tmp/pip-req-build-re2_lzbd
      ssh: Could not resolve hostname github.com:pypa: Name or service not known
      fatal: Could not read from remote repository.
    
      Please make sure you have the correct access rights
      and the repository exists.
      error: subprocess-exited-with-error
      
      × git clone --filter=blob:none --quiet 'ssh://****@github.com:pypa/sampleproject.git' /tmp/pip-req-build-re2_lzbd did not run successfully.
      │ exit code: 128
      ╰─> No available output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed to build 'git+ssh://****@github.com:pypa/sampleproject.git' when git clone --filter=blob:none --quiet 'ssh://****@github.com:pypa/sampleproject.git' /tmp/pip-req-build-re2_lzbd
    
    ~ ✗ 
    

    The real issue is,

    The internal parsing of pip is now broken, but external interface is not.

    ~/pip-repro ❯ python
    Python 3.16.0a2+dev (heads/gh-158907-proxy-auth-direct-407:f8671ac0fda, Nov 12 2026, 15:51:42) [GCC 16.2.1 20260810] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from pip._internal.vcs.git import Git
    >>> print(Git.get_url_rev_and_auth('git+git@github.com:pypa/sampleproject.git'))
    Traceback (most recent call last):
      File "<python-input-2>", line 1, in <module>
        print(Git.get_url_rev_and_auth('git+git@github.com:pypa/sampleproject.git'))
              ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/senthil/pip-repro/pip-src/src/pip/_internal/vcs/git.py", line 522, in get_url_rev_and_auth
        url, rev, user_pass = super().get_url_rev_and_auth(url)
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
      File "/home/senthil/pip-repro/pip-src/src/pip/_internal/vcs/versioncontrol.py", line 381, in get_url_rev_and_auth
        scheme, netloc, path, query, frag = urllib.parse.urlsplit(url)
                                            ~~~~~~~~~~~~~~~~~~~~~^^^^^
      File "/home/senthil/git/cpython/Lib/urllib/parse.py", line 604, in urlsplit
        scheme, netloc, url, query, fragment = _urlsplit(url, scheme, allow_fragments)
                                               ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/senthil/git/cpython/Lib/urllib/parse.py", line 646, in _urlsplit
        _checknetloc(netloc)
        ~~~~~~~~~~~~^^^^^^^^
      File "/home/senthil/git/cpython/Lib/urllib/parse.py", line 547, in _checknetloc
        _validate_port(port)
        ~~~~~~~~~~~~~~^^^^^^
      File "/home/senthil/git/cpython/Lib/urllib/parse.py", line 166, in _validate_port
        raise ValueError(f"Port could not be cast to integer value as {port!r}")
    ValueError: Port could not be cast to integer value as 'pypa'
    >>> 
    

    Against a stable version

    ~/pip-repro ❯ python
    Python 3.15.2 (main, Oct 26 2026, 14:06:56) [Clang 23.1.3 ] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from pip._internal.vcs.git import Git
    >>> print(Git.get_url_rev_and_auth('git+git@github.com:pypa/sampleproject.git'))
    ('git@github.com:pypa/sampleproject.git', None, (None, None))
    >>> 
    

    I will revert the 3.14 and 3.15 changes, and then, raise this against the pip tracker for the 3.16 since this is behavior change of urlsplit that can catch users in an un-expected way. I will also enhance the whatsnew section for Python3.16

  6. orsenthil commented on Nov 14, 2026

    @orsenthil
    Member

    Raised this against pip pypa/pip#14438

  7. added 2 commits that reference this issue on Nov 14, 2026
  8. orsenthil commented on Nov 14, 2026

    @orsenthil
    Member

    Updated WhatsNew Section - #162011

    I think all these changes are sufficient. I am reluctant to change in main as the behavior is in conformance with what generic url parsers are expected to do.

  9. thejesh23 commented on Nov 15, 2026

    @thejesh23
    ContributorAuthor

    @orsenthil thanks for detailed analysis and fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.14bugs and security fixes3.15bugs and security fixes

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions