This documentation site is for the versions of Synapse maintained by the Matrix.org Foundation (github.com/matrix-org/synapse), available under the Apache 2.0 licence.
Synapse uses a variety of third-party Python packages to function as a homeserver.
Some of these are direct dependencies, listed in pyproject.toml under the
[tool.poetry.dependencies] section. The rest are transitive dependencies (the
things that our direct dependencies themselves depend on, and so on recursively.)
We maintain a locked list of all our dependencies (transitive included) so that
we can track exactly which version of each dependency appears in a given release.
See here
for discussion of why we wanted this for Synapse. We chose to use
poetry to manage this locked list; see
this comment
for the reasoning.
The locked dependencies get included in our "self-contained" releases: namely,
our docker images and our debian packages. We also use the locked dependencies
in development and our continuous integration.
Separately, our "broad" dependencies—the version ranges specified in
pyproject.toml—are included as metadata in our "sdists" and "wheels" uploaded
to PyPI. Installing from PyPI or from
the Synapse source tree directly will not use the locked dependencies; instead,
they'll pull in the latest version of each package available at install time.
[[package]]
name = "phonenumbers"
version = "8.12.44"
description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
category = "main"
optional = false
python-versions = "*"
The lockfile also includes a
cryptographic checksum
of the sdists and wheels provided for this version of phonenumbers.
direnv is a tool for activating environments in your
shell inside a given directory. Its support for poetry is unofficial (a
community wiki recipe only), but works solidly in our experience. We thoroughly
recommend it for daily use. To use it:
Install direnv - it's likely
packaged for your system already.
Teach direnv about poetry. The shell config here
needs to be added to ~/.config/direnv/direnvrc (or more generally $XDG_CONFIG_HOME/direnv/direnvrc).
Mark the synapse checkout as a poetry project: echo layout poetry > .envrc.
Convince yourself that you trust this .envrc configuration and project.
Then formally confirm this to direnv by running direnv allow.
Then whenever you navigate to the synapse checkout, you should be able to run
e.g. mypy instead of poetry run mypy; python instead of
poetry run python; and your shell commands will automatically run in the
context of poetry's venv, without having to run poetry shell beforehand.
# Stop the current virtualenv if active
$ deactivate
# Remove all of the files from the current environment.
# Don't worry, even though it says "all", this will only
# remove the Poetry virtualenvs for the current project.
$ poetry env remove --all
# Reactivate Poetry shell to create the virtualenv again
$ poetry shell
# Install everything again
$ poetry install --extras all
Use poetry run cmd args when you need the python virtualenv context.
To avoid typing poetry run all the time, you can run poetry shell
to start a new shell in the poetry virtualenv context. Within poetry shell,
python, pip, mypy, trial, etc. are all run inside the project virtualenv
and isolated from the rest o the system.
Roughly speaking, the translation from a traditional virtualenv is:
env/bin/activate -> poetry shell, and
deactivate -> close the terminal (Ctrl-D, exit, etc.)
See also the direnv recommendation above, which makes poetry run and
poetry shell unnecessary.
# Current env only
poetry env info
# All envs: this allows you to have e.g. a poetry managed venv for Python 3.7,
# and another for Python 3.10.
poetry env list --full-path
poetry run pip list
Note that poetry show describes the abstract lock file rather than your
on-disk environment. With that said, poetry show --tree can sometimes be
useful.
ought to do the trick. Alternatively, manually update pyproject.toml and
poetry lock --no-update. Include the updated pyproject.toml and poetry.lock
files in your commit.
to use the latest version of packagename in the locked environment, without
affecting the broad dependencies listed in the wheel.
There doesn't seem to be a way to do this whilst locking a specific version of
packagename. We can workaround this (crudely) as follows:
poetry add packagename==1.2.3
# This should update pyproject.lock.
# Now undo the changes to pyproject.toml. For example
# git restore pyproject.toml
# Get poetry to recompute the content-hash of pyproject.toml without changing
# the locked package versions.
poetry lock --no-update
Either way, include the updated poetry.lock file in your commit.
Synapse uses Dependabot to keep the poetry.lock and Cargo.lock file
up-to-date with the latest releases of our dependencies. The changelog check is
omitted for Dependabot PRs; the release script will include them in the
changelog.
When reviewing a dependabot PR, ensure that:
the lockfile changes look reasonable;
the upstream changelog file (linked in the description) doesn't include any
breaking changes;
continuous integration passes.
In particular, any updates to the type hints (usually packages which start with types-)
should be safe to merge if linting passes.
The minimum version of poetry supported by Synapse is 1.3.2.
It can also be useful to check the version of poetry-core in use. If you've
installed poetry with pipx, try pipx runpip poetry list | grep poetry-core.
Poetry caches a bunch of information about packages that isn't readily available
from PyPI. (This is what makes poetry seem slow when doing the first
poetry install.) Try poetry cache list and poetry cache clear --all <name of cache> to see if that fixes things.