Extend UserAPI to handle threepid validation sessions

[PiotrKozimor]

• UserInternalAPI is extended to handle validation sessions.
• Tests are implemented for new methods using testMailer.
• Tests for storage are written using sqlite3 with in-memory mode.
• Sending emails is abstracted with Mailer interface.
• SmtpMailer implementation of Mailer is provided.
• UserAPI config is extended to configure SmtpMailer and email templates path.
• ThreepidSessionType is defined. Three different templates are defined - account_password, account_threepid, register for respective /requestToken endpoints

Uncertainties

• sqlite3 and postgres Database implementations share statements but diverge on schema definition. Is that ok?
• New token is not generated when /requestToken is made for the same client_secret and email pair with sent_attempt bumped (both synapse and dendrite generate new tokens). I did not see any rationale behind generating new one. Proposed approach requires one SQL table instead of two.
• github.com/matryer/is was used as extension to go testing framework. I found it minimal yet powerful to work with. Can we accept it as a test dependency?
• I have tested SmtpMailer manually by calling docker run --rm -p 25:25 -v $(pwd)/emails:/var/lib/mock-smtp -d flaviovs/mock-smtp in userapi/mail and then go test -tags=test_mail and inspecting output manually. Can we automate these tests in Buildkite?

What's next

• We can handle msisdn in simple way - just send sms instead of email in CreateSession
• Implement routing in Client API
• Implement registering and login with email

Pull Request Checklist

• I have added any new tests that need to pass to sytest-whitelist as specified in docs/sytest.md
• Pull request includes a sign off

Signed-off-by: Piotr Kozimor <[email protected]>

[PiotrKozimor]

I made a progress with CI checks. I struggle however to understand why Local device key changes appear in /keys/changes test fails. I see that there are similar errors for latest commit in master: https://buildkite.com/matrix-dot-org/dendrite/builds/5441. @kegsay can you help?

[kegsay]

Local device key changes appear in /keys/changes is a flakey test annoyingly, so feel free to ignore it. It is already ignored https://github.com/matrix-org/dendrite/blob/master/sytest-blacklist#L75 but for some reason sytest doesn't pick this up.

[PiotrKozimor]

Nice, thanks! I think this PR is ready to be reviewed - I want to route extended User API in separate PR. Can you please refer to Uncertainties section? Also, sytest are failing now on Inviting an AS-hosted user asks the AS server, looks like join event for wrong room is received. I am not sure how to fix it. In logs there is a line: WARN: Ignoring incoming AS event of type m.room.member - maybe the right event was ignored?

[PiotrKozimor]

Can login with 3pid and password using m.login.password fails because no email configuration is provided. Solved in matrix-org/sytest#1143

[PiotrKozimor]

Sytest is passing 🎉 I have failure in Upgrades test. I will try to fix it soon.

[neilalexander]

I was trying to merge the latest master commits into this PR to re-run it through CI, but it seems there's a protected branch restriction in place in your fork that prevents me from doing so?

[PiotrKozimor]

I have played with it today, please check now, should be fixed. Sorry for inconvenience.