Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Standardise the module interface #10062

Merged
merged 38 commits into from
Jun 18, 2021
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e388397
First cut at a standardised module interface
babolivier May 25, 2021
11f525c
Don't use a centralised handler and let modules register what they ne…
babolivier May 26, 2021
ceb9904
Specify where the new methods need to be called from
babolivier May 26, 2021
c4d09a8
Implement new module interface for the spam checker
babolivier May 26, 2021
f5098c9
Don't centralise registration of hooks and web resources
babolivier May 26, 2021
7da2fd3
Don't use a class if a simple function works just as well
babolivier May 26, 2021
f1c0889
Fix CI
babolivier May 26, 2021
817fc75
Lint
babolivier May 26, 2021
a988b8c
Incorporate comments
babolivier May 27, 2021
ba4e678
Lint
babolivier May 27, 2021
a06649c
Don't inhibit rejection reason from spamchecker
babolivier May 28, 2021
d55b17b
Make mypy happy
babolivier May 28, 2021
2c8d6d5
Fix tests
babolivier May 28, 2021
10153fc
Lint
babolivier May 28, 2021
eda9658
Merge branch 'develop' into babolivier/modules
babolivier May 28, 2021
1c9e3d4
Document the new module interface
babolivier Jun 4, 2021
870647d
Merge branch 'develop' into babolivier/modules
babolivier Jun 4, 2021
b92965c
Add new doc to the summary, and add a deprecation notice to the spam …
babolivier Jun 4, 2021
d440297
Fix a typo in registration docs
babolivier Jun 4, 2021
ce4347b
Point to the new docs in the sample configuration
babolivier Jun 4, 2021
79ee967
Improve example
babolivier Jun 4, 2021
7bf8fdb
Apply suggestions from code review
babolivier Jun 16, 2021
a63a060
Merge branch 'develop' into babolivier/modules
babolivier Jun 16, 2021
c6ed049
Incorporate review comments
babolivier Jun 16, 2021
39a02b1
Lint
babolivier Jun 17, 2021
8e28b3e
Use async callbacks in tests
babolivier Jun 17, 2021
9c5bffd
Correctly wrap check_registration_for_spam
babolivier Jun 17, 2021
468b900
Lint
babolivier Jun 17, 2021
5a9f391
Move support for 3-arg check_registration_for_spam to legacy code
babolivier Jun 18, 2021
6a326f9
Remove unused import
babolivier Jun 18, 2021
575556f
Remove other unused import
babolivier Jun 18, 2021
12774dc
Explicitely type legacy callback as not None
babolivier Jun 18, 2021
b12855c
Don't import cast again
babolivier Jun 18, 2021
cd596f5
Be more vague in upgrade notes and add deprecation notice to changelog
babolivier Jun 18, 2021
3a28f6a
Phrasing
babolivier Jun 18, 2021
9cbe1e6
Merge branch 'develop' into babolivier/modules
babolivier Jun 18, 2021
387d41b
Types don't like commas
babolivier Jun 18, 2021
249c607
Fix tests and phrasing
babolivier Jun 18, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/10062.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Standardised the module interface.
2 changes: 1 addition & 1 deletion docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
- [URL Previews](url_previews.md)
- [User Directory](user_directory.md)
- [Message Retention Policies](message_retention_policies.md)
- [Pluggable Modules]()
- [Pluggable Modules](modules.md)
- [Third Party Rules]()
- [Spam Checker](spam_checker.md)
- [Presence Router](presence_router_module.md)
Expand Down
229 changes: 229 additions & 0 deletions docs/modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
# Modules

Synapse supports extending its functionalities by configuring external modules.
babolivier marked this conversation as resolved.
Show resolved Hide resolved

## Using modules

To use a module on Synapse, add it to the `modules` section of the configuration file:

```yaml
modules:
- module: my_super_module.MySuperClass
config:
do_thing: true
- module: my_other_super_module.SomeClass
config: {}
```

Each module is defined by a path to a Python class as well as a configuration. This
information for a given module should be available in the module's own documentation.

**Note**: When using third-party modules, you effectively allow someone else to run
custom code on your Synapse homeserver. Server admins are encouraged to verify the
provenance of the modules they use on their homeserver and make sure the modules aren't
running malicious code on their instance.

babolivier marked this conversation as resolved.
Show resolved Hide resolved
## Writing a module

A module is a Python class that uses Synapse's module API to interact with the
homeserver. It can register callbacks that Synapse will call on specific operations, as
well as web resources to attach to Synapse's web server.

A module must implement the following static method:
babolivier marked this conversation as resolved.
Show resolved Hide resolved

```python
@staticmethod
def parse_config(config: dict) -> dict
```

This method is given a dictionary resulting from parsing the YAML configuration for the
module. It may modify it (for example by parsing durations expressed as strings (e.g.
"5d") into milliseconds, etc.), and return the modified dictionary. If no change is
necessary, this method should just return `config`.
babolivier marked this conversation as resolved.
Show resolved Hide resolved

When instantiated, a module is given its parsed configuration (i.e. the output of
`parse_config`) as well as an instance of the `synapse.module_api.ModuleApi` class.

See the documentation for the `ModuleApi` class [here](/synapse/module_api/__init__.py).

### Registering a web resource

Modules can register web resources onto Synapse's web server using the following module
API method:

```python
def ModuleApi.register_web_resource(path: str, resource: IResource)
```

The path is the full absolute path to register the resource at. For example, if you
register a resource for the path `/_synapse/client/my_super_module/say_hello`, Synapse
will serve it at `http(s)://[HS_URL]/_synapse/client/my_super_module/say_hello`. Note
that Synapse does not allow registering resources for several sub-paths in the `/_matrix`
namespace (such as anything under `/_matrix/client` for example). It is strongly
recommended that modules register their web resources under the `/_synapse/client`
namespace.

The provided resource is a Python class that implements Twisted's [IResource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html)
interface (such as [Resource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resource.html)).

Only one resource can be registered for a given path. If several modules attempt to
register a resource for the same path, the module that appears first in Synapse's
configuration file takes priority.

Modules **must** register their web resources in their `__init__` method.

### Registering a callback

Modules can use Synapse's module API to register callbacks. Callbacks are functions that
Synapse will call when performing specific actions. Callbacks can be either asynchronous
or synchronous, and are split in categories. A single module may implement callbacks from
multiple categories, and is under no obligation to implement all callbacks from the
category(ies) it registers callbacks for.
babolivier marked this conversation as resolved.
Show resolved Hide resolved

#### Spam checker callbacks

To register one of the callbacks described in this section, a module needs to use the
module API's `register_spam_checker_callbacks` method. The callback functions are passed
to `register_spam_checker_callbacks` as keyword arguments, with the callback name as the
argument name and the function as its value. This is demonstrated in the example below.

The available spam checker callbacks are:

```python
def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool, str]
```

Called when receiving an event from a client or via federation. The module can return
either a `bool` to indicate whether the event must be rejected because of spam, or a `str`
to indicate the event must be rejected because of spam and to give a rejection reason to
forward to clients.

```python
def user_may_invite(inviter: str, invitee: str, room_id: str) -> bool
```

Called when processing an invitation. The module must return a `bool` indicating whether
the inviter can invite the invitee to the given room. Both inviter and invitee are
represented by their Matrix user ID (i.e. `@alice:example.com`).

```python
def user_may_create_room(user: str) -> bool
```

Called when processing a room creation request. The module must return a `bool` indicating
whether the given user (represented by their Matrix user ID) is allowed to create a room.

```python
def user_may_create_room_alias(user: str, room_alias: "synapse.types.RoomAlias") -> bool
```

Called when trying to associate an alias with an existing room. The module must return a
`bool` indicating whether the given user (represented by their Matrix user ID) is allowed
to set the given alias.

```python
def user_may_publish_room(user: str, room_id: str) -> bool
```

Called when trying to publish a room to the homeserver's public rooms directory. The
module must return a `bool` indicating whether the given user (represented by their
Matrix user ID) is allowed to publish the given room.

```python
def check_username_for_spam(user_profile: Dict[str, str]) -> bool
```

Called when computing search results in the user directory. The module must return a
`bool` indicating whether the given user profile can appear in search results. The profile
is represented as a dictionary with the following keys:

* `user_id`: The Matrix ID for this user.
* `display_name`: The user's display name.
* `avatar_url`: The `mxc://` URL to the user's avatar.

The module is given a copy of the original dictionary, so modifying it from within the
module cannot modify a user's profile when included in user directory search results.

```python
def check_registration_for_spam(
email_threepid: Optional[dict],
username: Optional[str],
request_info: Collection[Tuple[str, str]],
auth_provider_id: Optional[str] = None,
) -> "synapse.spam_checker_api.RegistrationBehaviour"
```

Called when registering a new user. The module must return a `RegistrationBehaviour`
indicating whether the registration can go through or must be denied, or whether the user
may be allowed to register but will be shadow banned.

The arguments passed to this callback are:

* `email_threepid`: The email address used for registering, if any.
* `username`: The username the user would like to register. Can be `None`, meaning that
Synapse will generate one later.
* `request_info`: A collection of tuples, which first item is a user agent, and which
second item is an IP address. These user agents and IP addresses are the ones that were
used during the registration process.
* `auth_provider_id`: The identifier of the SSO authentication provider, if any.

```python
def check_media_file_for_spam(
file_wrapper: "synapse.rest.media.v1.media_storage.ReadableFileWrapper",
file_info: "synapse.rest.media.v1._base.FileInfo"
) -> bool
```

Called when storing a local or remote file. The module must return a boolean indicating
whether the given file can be stored in the homeserver's media store.

### Example

The example below is a module that implements the spam checker callback
`user_may_create_room` to deny room creation to user `@evilguy:example.com`, and registers
a web resource to the path `/_synapse/client/demo/hello` that returns a JSON object.

```python
import json

from twisted.web.resource import Resource
from twisted.web.server import Request

from synapse.module_api import ModuleApi


class DemoResource(Resource):
def __init__(self, config):
super(DemoResource, self).__init__()
self.config = config

async def render_GET(self, request: Request):
babolivier marked this conversation as resolved.
Show resolved Hide resolved
name = request.args.get(b"name")[0]
request.setHeader(b"Content-Type", b"application/json")
return json.dumps({"hello": name})


class DemoModule:
def __init__(self, config: dict, api: ModuleApi):
self.config = config
self.api = api

self.api.register_web_resource(
path="/_synapse/client/demo/hello",
resource=DemoResource(self.config),
)

self.api.register_spam_checker_callbacks(
user_may_create_room=self.user_may_create_room,
)

@staticmethod
def parse_config(config):
return config

async def user_may_create_room(self, user: str) -> bool:
if user == "@evilguy:example.com":
return False

return True
```
17 changes: 17 additions & 0 deletions docs/sample_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,23 @@
#
# [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html


## Modules ##

# Server admins can expand Synapse's functionalities by using external modules
# to complement certain operations.
#
# See https://matrix-org.github.io/synapse/develop/modules.html for more
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}


## Server ##

# The public-facing domain of the server
Expand Down
4 changes: 4 additions & 0 deletions docs/spam_checker.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
**Note: this page of the Synapse documentation is now deprecated. For up to date
babolivier marked this conversation as resolved.
Show resolved Hide resolved
documentation on setting up or writing a spam checker module, please see
[this page](https://matrix-org.github.io/synapse/develop/modules.html).**

babolivier marked this conversation as resolved.
Show resolved Hide resolved
# Handling spam in Synapse

Synapse has support to customize spam checking behavior. It can plug into a
Expand Down
9 changes: 9 additions & 0 deletions synapse/app/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
from synapse.app.phone_stats_home import start_phone_stats_home
from synapse.config.homeserver import HomeServerConfig
from synapse.crypto import context_factory
from synapse.events.spamcheck import load_legacy_spam_checkers
from synapse.logging.context import PreserveLoggingContext
from synapse.metrics.background_process_metrics import wrap_as_background_process
from synapse.metrics.jemalloc import setup_jemalloc_stats
Expand Down Expand Up @@ -334,6 +335,14 @@ def run_sighup(*args, **kwargs):
# Start the tracer
synapse.logging.opentracing.init_tracer(hs) # type: ignore[attr-defined] # noqa

# Instantiate the modules so they can register their web resources to the module API
# before we start the listeners.
module_api = hs.get_module_api()
for module, config in hs.config.modules.loaded_modules:
module(config=config, api=module_api)

load_legacy_spam_checkers(hs)

# It is now safe to start your Synapse.
hs.start_listening()
hs.get_datastore().db_pool.start_profiling()
Expand Down
4 changes: 4 additions & 0 deletions synapse/app/homeserver.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,10 @@ def _listener_http(self, config: HomeServerConfig, listener_config: ListenerConf
)
resources[path] = resource

# Attach additional resources registered by modules.
resources.update(self._module_web_resources)
self._module_web_resources_consumed = True

babolivier marked this conversation as resolved.
Show resolved Hide resolved
# try to find something useful to redirect '/' to
if WEB_CLIENT_PREFIX in resources:
root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
Expand Down
2 changes: 2 additions & 0 deletions synapse/config/_base.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ from synapse.config import (
key,
logger,
metrics,
modules,
oidc,
password_auth_providers,
push,
Expand Down Expand Up @@ -85,6 +86,7 @@ class RootConfig:
thirdpartyrules: third_party_event_rules.ThirdPartyRulesConfig
tracer: tracer.TracerConfig
redis: redis.RedisConfig
modules: modules.ModulesConfig

config_classes: List = ...
def __init__(self) -> None: ...
Expand Down
5 changes: 3 additions & 2 deletions synapse/config/homeserver.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Copyright 2014-2016 OpenMarket Ltd
# Copyright 2018 New Vector Ltd
# Copyright 2021 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -30,6 +29,7 @@
from .key import KeyConfig
from .logger import LoggingConfig
from .metrics import MetricsConfig
from .modules import ModulesConfig
from .oidc import OIDCConfig
from .password_auth_providers import PasswordAuthProviderConfig
from .push import PushConfig
Expand All @@ -56,6 +56,7 @@
class HomeServerConfig(RootConfig):

config_classes = [
ModulesConfig,
ServerConfig,
TlsConfig,
FederationConfig,
Expand Down
49 changes: 49 additions & 0 deletions synapse/config/modules.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Copyright 2021 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from synapse.config._base import Config, ConfigError
from synapse.util.module_loader import load_module


class ModulesConfig(Config):
section = "modules"

def read_config(self, config: dict, **kwargs):
self.loaded_modules = []
babolivier marked this conversation as resolved.
Show resolved Hide resolved

configured_modules = config.get("modules") or []
for i, module in enumerate(configured_modules):
config_path = ("modules", "<item %i>" % i)
if not isinstance(module, dict):
raise ConfigError("expected a mapping", config_path)

self.loaded_modules.append(load_module(module, config_path))

def generate_config_section(self, **kwargs):
return """
## Modules ##

# Server admins can expand Synapse's functionalities by using external modules
# to complement certain operations.
babolivier marked this conversation as resolved.
Show resolved Hide resolved
#
# See https://matrix-org.github.io/synapse/develop/modules.html for more
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}
babolivier marked this conversation as resolved.
Show resolved Hide resolved
"""
Loading