RethinkDB

[leetal]

So, i have finally added RethinkDB. All tests pass (1. see below) locally and some changes have been made to the base implementation of Hydra (no BC Breaks?). I have for example introduced the new Context interface that makes it easier to extend Hydra in the future by adding more storage contexts. I also added the ability to select which storage backend to use via the new DATABASE environment variable (defaults to postgres). There is so much to write, but i think that a code review would definitely be in place now!

Since dockertest does not support rethinkdb, all rethinkdb tests require the environment variable RETHINKDB_URL to be set prior to testing (and obviously a rethinkdb instance up and running).

There are probably lots of improvements that can be made, but at least this is as far as I have gone.
To mention a few:

• For example, i don't know if travis tests work at all right now since the code isn't merged to ory-am/hydra.
• The Close() function in the Contexts is never called right now. We need to find a way to intercept shutdown and interrupt signals and call Close() from there.

1. Sadly i could not test the postgres version at all due to limitations on my macbook. So all Postgres-tests fails for me, but everything else passes. BUT, I have not touched the postgres parts at all, so they should probably work. So neither vagrant (won't work until after merge) nor postgres versions tested. (But hey as i said, they should work ;) )

Please see #39 for more discussions regarding this PR.

[leetal]

So, of course Ladon has to be merged for the tests to even run... ;)

[aeneasr]

Wow cool stuff, looks like a lot of work! I'll go over everything in detail.

[leetal]

Yeah. There was quite a few storage interfaces to cover ;)

[aeneasr]

Man I hope I won't break too much of this once we go fosite...

[leetal]

I count on it! But the best part with that is that then both of us knows just a little bit more of how the interfaces should work for extensibility :)

[aeneasr]

Ha! Yeah absolutely :)

[aeneasr]

Okay so priorities for me right now are:

1. Add rethink db to dockertest
2. Review and merge ladon PR
3. Review and merge this PR

[leetal]

That should do it. BUT, I think that the tests actually will pass anyways
since I have added rethinkdb to travis. Ladon just passed and I think that
the same will go for hydra as soon as ladon is merged (Ladon´s tests all
pass now, just coverage that has decreased due to some errors being hard to
reproduce like connection errors to rethinkdb) :)

[aeneasr]

Ah I see :) I still prefer dockertest because I don't have to set up rethink db on my local environment to get tests passing :)

[leetal]

Then we are two! :D

[aeneasr]

I'll try to get that done tonight. Workload is kinda insane right now but these two PRs are high priority for me :)

[aeneasr]

ory/dockertest#23

[aeneasr]

dockertest is merged

[leetal]

Awesome! I'll fix the tests and update the PR's.

[leetal]

As soon as ladon is merged and usable, i'll rebase and squash the changes and rewrite the commits to a better format.

[aeneasr]

Hey, thanks for cleaning up! I will need a little time to go through all the changes. Are you planing on using Hydra in a production environment? If so, it would be very good if we two could get together for example on the gitter channel and discuss next steps, caveats and architecture.

There are still some areas and concepts in Hydra which I've grown to dislike and if you want to use Hydra in production we need a clear path so you won't get hit by unexpected changes or issues.

We will use Hydra at ORY in closed beta "production" starting first of march (1.3.2016) and hopefully get some insights into stability, scalability, security and so on.

[aeneasr]

One major issue for me is upgrading / migrating the database - automating this is something where I have very little experience in. But I don't want to break your production system because I decided to rename some struct or add some functionality..

[leetal]

We are a few guys doing a project at home right now that's not connected to work. In this project we plan on using hydra (with fosite) and are also well aware of the current status if the project. Personally I like to have things break because that means that you can improve on those areas by finding out that the base implementation was incorrect. Refactoring is what I do for a living so I'm well aware of that things WILL break in hydra. Especially when fosite comes into picture. Also, any modifications that is made on the interfaces (for example automatic db migrations) will break the base implementation in a irreversible way, which at least I know is subject to change. So please, since the status is very well explained in the Readme, do what you have to. You guys own the rights so you also own the rights to change what needs to be changed. The status of our personal project is like 40% done. We actually have a standalone oauth2 service that we've written ourselves based on osin. Hydra looked so much more promising so that's why I'm eager to help out with the stuff I know I can handle. A few things wanted with hydra: 1. Db migrations (only useful for sql) 2. Cluster functionality/scalability (ability to connect multiple hydra instances together and issue commands between them) 3. Improved documentation ;) Right now I really don't know how to pass arbitrary data in the jwt so other services using the jwt can use that data in a scaleable way. One thing I've wanted to write for a while is a intra-communication system that is plug and play for more or less all go services. And to this intra-communication system write a dashboard where the status of the service is shown, with ability to send commands to each instance (or group of instances). I think a system like that in hydra would make hydra more enterprise. Plus, then almost all services would be guarded and controlled remotely. :) I'll see what I can come up with in the coming month. We really want such a system in our project so we can see the status of all micro-services and issue commands to them, making them perform things like rolling updates and backups and even scaling dynamically.

[aeneasr]

Okay very cool, glad we're on the same page then :) I will try to answer you in detail tomorrow

[aeneasr]

Found a couple of free minutes... :)

I do for a living so I'm well aware of that things WILL break in hydra.

Perfect. :) We will have to find a good way to migrate databases but we could for example use a version table and have an array of incremental upgrade commands (ALTER TABLE, ..).

1. Db migrations (only useful for sql)

See above. We need to find a good way to do this for both rethinkdb and sql

1. Cluster functionality/scalability (ability to connect multiple hydra instances together and issue commands between them)

Not needed! Hydra is written 12-factor so it's completely stateless. Hydra relies on a highly available database cluster (rethink or postgres) - so if you have a good database cluster hydra will be a good cluster as well.

What I want to do is a CLI that uses the hydra RESTful api for managing clients, policies and accounts.

1. Improved documentation ;) Right now I really don't know how to pass
   arbitrary data in the jwt so other services using the jwt can use that data
   in a scaleable way.

Absolutely, I won't be able to go into details now but this is definitely something. If the CLI was available, I however think that reading the code would already help a lot.

Regarding JWT - I will probably remove JWTs as access tokens and introduce OpenID Connect ID tokens instead. This is truly an area which must be covered.

One thing I've wanted to write for a while is a intra-communication system
that is plug and play for more or less all go services. And to this
intra-communication system write a dashboard where the status of the
service is shown, with ability to send commands to each instance (or group
of instances). I think a system like that in hydra would make hydra more
enterprise.

Sound like a cool idea, really glad you consider hydra :)

The following is a list to keep my train of thought. I will explain the things in there another time :)

I think these are the things that need to be done / talked about are:

• implement fosite
• implement OpenID Connect (where should the data come from? what claims/scopes to support?)
• refactor login and consent logic. maybe login/consent page should be responsible for getting user data
• consider to what extend account / profile management should be implemented
• clean up messy tests (replace database integration tests in handlers with mocks)
• implement better granted endpoint

[aeneasr]

I forgot to add something.

Cluster functionality/scalability (ability to connect multiple hydra instances together and issue commands between them)

Not needed! Hydra is written 12-factor so it's completely stateless. Hydra relies on a highly available database cluster (rethink or postgres) - so if you have a good database cluster hydra will be a good cluster as well.

Right now it is not anticipated that hydra will consist of multiple executables. I want Hydra to never get on a level of complexity, where it isn't considered "micro service" any more. This why I don't want Hydra to have any sort of HTML output nor a complex user management and it should not become a service for connecting legacy enterprise standards / software. I think there are better solutions already available for that! Instead, hydra should be a fast, easy, powerful and yet very secure service one can use for a new (web) app or micro service environment which will provide a secure experience to millions of users. This is my vision for hydra! :)

[aeneasr]

couldn't this be done via:

• postgres://user:password@host:port/database - postgres
• rethinkdb://... - rethinkdb
• mysql://...

[leetal]

Yes it could, no standard way of doing it was implemented so i just chose the easier way. I can fix this today. I think that prefixing it would me much more neat.

[aeneasr]

I've took a closer look at about 50% and everything looks really well. Give me a little bit more time to review everything in detail so I don't miss anything :)

[aeneasr]

Done! :) Everything else LGTM

[aeneasr]

Thanks (y)