Pydantic v2 #19071

V02460 · 2025-10-17T15:15:31Z

Update Pydantic to v2.

Steps I took to create this change:

Remove import indirection
Run the Pydantic migration tool
Fix remaining issues
Enable strict mode and remove check_pydantic_models.py

Notable changes are

differing return type for threepid requests with an invalid email
changed logic for _metadata_url and _introspection_endpoint
changed definition of AnyEventId
Disabled strict mode for tuple and set types

Closes #15858
Closes #15979

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

anoadragon453

Thank you very much for filing this - it saves the dev team a ton of effort ❤️

The sytest failures appear to be due to Sytest not following the spec - hooray for validation!

I've started on fixing them - by converting the tests to Complement and debugging them there, starting with: matrix-org/complement#811

Some minor comments below.

synapse/types/handlers/sliding_sync.py

synapse/config/mas.py

tests/rest/client/test_account.py

scripts-dev/lint.sh

pyproject.toml

anoadragon453 · 2025-10-21T09:12:22Z

pyproject.toml

 # We need packaging.verison.Version(...).major added in 20.0.
 packaging = ">=20.0"
-pydantic = "~=2.1"
+pydantic = "~=2.12"


Do we require 2.12?

Several Linux distributions do not carry that version, so will struggle to package Synapse: https://repology.org/project/python%3Apydantic/versions

In general we should aim to specify the lowest possible version in pyproject.toml that's compatible with the code. Then in poetry.lock we specify highest possible (so those distributing Synapse using pyenv's can get the latest fixes and improvements).

It’s dependent on the Python version. Pydantic 2.7.4 was the first version to give correct results. For Python 3.13 and Python 3.14 later versions are required, with Pydantic 2.12 being the first version to support Python 3.14.

We could discriminate based on Python version, something like pydantic>=2.7.4;python_version<'3.13', etc.

Pydantic 2.8 added support for Python 3.13. Because most of the distros in the table packaged at least Pydantic 2.8, I’d suggest the following:

pydantic = [ { version = "~= 2.8", python = "< 3.14" }, { version = "~= 2.12", python = ">= 3.14" }, ]

I see, I didn't realise >=2.7.4 was necessary. Indeed, I don't see any distros in that list with 2.7.*, so it would make sense to go to the next minor version up, 2.8*.

Your suggestion looks sound!

Oh man, trial-olddeps is complaining. But I don’t get what’s going on just yet.

Edit: My first guess is a logic error in check_dependencies.py.

You're correct. check_dependencies.py doesn't take the python attribute into account when determining which dependencies it needs to install. It then fails when attempting to install 2.12.x even though it's on Python 3.9.

I did some preliminary work on updating it in a separate branch. Let me bring that up into a PR.

synapse/util/pydantic_models.py

Co-authored-by: Andrew Morgan <[email protected]>

synapse/util/events.py

Committing this to this branch to check whether it fixes `tests-olddeps`. If so, will extract this to a separate PR for review from the team.

This appears to be tripping old-deps. First, see if removing fixes it.

anoadragon453 · 2025-10-28T23:35:31Z

@V02460 I believe I've bypassed the previous old-deps errors and now we've moved on to something else (likely just changing a set to a list in the defined type?).

Heading to bed now, and can pick it up in the morning.

V02460 · 2025-10-29T08:14:28Z

synapse/rest/synapse/mas/devices.py

    class PostBody(RequestBodyModel):
        localpart: StrictStr
-        devices: set[StrictStr]
+        devices: set[str]


I believe I've bypassed the previous old-deps errors and now we've moved on to something else (likely just changing a set to a list in the defined type?).

Nice!

I found v2.9.0 to be the first version that no longer has this bug. Raising the lower version bound of Pydantic would fix the concrete error. Would that be ok?

Another approach would be to fix all the call sites to actually pass a set.

Looks like there was only one call site to fix? After 13fe3fa the tests now pass!

This PR is now blocked on #19110 merging, which contains the fixes to check_dependencies.py.

I'll revert the commits in this branch for now.

JSON has no way to translate to `set`, and the type is converted to a `set` below anyways.

A regression test for the issue we were seeing in #19071 (comment)

anoadragon453 · 2025-10-29T14:38:14Z

pyproject.toml

+pydantic = [
+    { version = "~=2.8", python = "<3.14" },
+    { version = "~=2.12", python = ">=3.14" },
+]


@V02460 could you also update (probably remove) the comment above pydantic under the [tool.poetry.group.dev.dependencies] header?

anoadragon453 · 2025-10-29T14:53:35Z

synapse/rest/synapse/mas/devices.py

    class PostBody(RequestBodyModel):
        localpart: StrictStr
-        devices: set[StrictStr]
+        devices: set[str]


This PR is now blocked on #19110 merging, which contains the fixes to check_dependencies.py.

I'll revert the commits in this branch for now.

This reverts commit 5a4eb77.

V02460 added 5 commits October 15, 2025 15:37

Use pydantic imports directly

7fe7d4b

Apply conflict-free pydantic migrations

6d4fb7e

Add debugging infos to asserts

4022d21

Migrate to pydantic v2

123fe26

Enable strict mode

32ea42a

V02460 requested a review from a team as a code owner October 17, 2025 15:15

V02460 force-pushed the pydantic-v2 branch from 5acf7a2 to f6d5031 Compare October 17, 2025 15:25

V02460 added 3 commits October 17, 2025 18:16

Update pyproject.toml

a7a5549

Add newsfragment

8879e00

Remove lint-pydantic CI job

1bb5a63

V02460 force-pushed the pydantic-v2 branch from 8edcc92 to 1bb5a63 Compare October 17, 2025 16:17

Bump pydantic to support Python 3.14

6183c4f

anoadragon453 mentioned this pull request Oct 20, 2025

Convert Sytest Name/topic keys are correct to Complement matrix-org/complement#811

Open

anoadragon453 requested changes Oct 21, 2025

View reviewed changes

V02460 and others added 4 commits October 21, 2025 15:16

Apply suggestions from code review

4ac5722

Co-authored-by: Andrew Morgan <[email protected]>

Remove upper version bound for typing-extensions

ab66dc1

Return more generic HTTP error for email requests

865ecf2

Use older Pydantic version for Python < 3.14

ee08ddc

anoadragon453 mentioned this pull request Oct 22, 2025

Convert sytest Can search public room list to Complement matrix-org/complement#812

Merged

anoadragon453 reviewed Oct 22, 2025

View reviewed changes

synapse/util/events.py Show resolved Hide resolved

V02460 and others added 8 commits October 22, 2025 21:01

Use model_validate instead of deprecated parse_obj

f04e690

Update parse_obj_as to TypeAdapter.validate_python

454b3a7

Disable strict mode for room topic content

e6eed4b

Allow tuples for m.text

bae991f

Merge branch 'develop' into pydantic-v2

80e3e8f

REVERT: Add support for markers to check_dependencies.py

5a4eb77

Committing this to this branch to check whether it fixes `tests-olddeps`. If so, will extract this to a separate PR for review from the team.

Merge branch 'develop' of github.com:element-hq/synapse into pydantic-v2

e02beb1

Try removing strict Field annotation to fix olddeps

2719f33

This appears to be tripping old-deps. First, see if removing fixes it.

V02460 commented Oct 29, 2025

View reviewed changes

Use a list rather than set in Pydantic model

13fe3fa

JSON has no way to translate to `set`, and the type is converted to a `set` below anyways.

anoadragon453 added a commit that referenced this pull request Oct 29, 2025

Add a unit test for checking dep versions based on python version

23e0571

A regression test for the issue we were seeing in #19071 (comment)

anoadragon453 mentioned this pull request Oct 29, 2025

Update check_dependencies to support markers #19110

Open

3 tasks

anoadragon453 reviewed Oct 29, 2025

View reviewed changes

Revert "REVERT: Add support for markers to check_dependencies.py"

59d6adf

This reverts commit 5a4eb77.

Uh oh!

Pydantic v2 #19071

Are you sure you want to change the base?

Pydantic v2 #19071

Uh oh!

Conversation

V02460 commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

V02460 Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anoadragon453 commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

V02460 commented Oct 17, 2025 •

edited

Loading

V02460 Oct 21, 2025 •

edited

Loading