feat: use etcd to store the JSON schema of APISIX, then the manager API can read it from etcd

[membphis]

here is the discuss mail:

https://lists.apache.org/thread.html/rb1c19fe9b6e51ede3b5aead122e2bcd4de6249f42b971eabfb8a79ac%40%3Cdev.apisix.apache.org%3E

[membphis]

@nic-chen do you want to handle this job?

[nic-chen]

OK, let me try.

[nic-chen]

I think the main logic should be:

When APISIX starts, read the key /apisix/jsonschema of ETCD.

If the key does not exist or is empty, then read the JSON schema from APISIX and save to the ETCD key /apisix/jsonschema.

If the key already exists and is not empty, nothing will be done.

APISIX provides a command line tool that can be used to manually update the JSON schema, like: apisix sync-schema,
users can use this tool to update JSON schema when needed.

What do you think ?

cc @membphis @moonming @liuxiran @dabue @tokers @ShiningRush

[tokers]

I think the main logic should be:

When APISIX starts, read the key /apisix/jsonschema of ETCD.

If the key does not exist or is empty, then read the JSON schema from APISIX and save to the ETCD key /apisix/jsonschema.

If the key already exists and is not empty, nothing will be done.

APISIX provides a command line tool that can be used to manually update the JSON schema, like: apisix sync-schema,
users can use this tool to update JSON schema when needed.

What do you think ?

cc @membphis @moonming @liuxiran @dabue @tokers @ShiningRush

So all schemas are put inside a single JSON string here?

BTW, have you consider the grayscale of APISIX itself? If the new version of APISIX has some incompatible changes for some schemas, once the schema is synced, all instances which shares the same ETCD cluster will be effected. So maybe versioning or other way like naming isolation is required.

[liuxiran]

Is it necessary to automatically trigger schema synchronization when adding a new schema(e.g: add a new plugin) or modifying a schema, so that the consumer of the schema (e.g manager-api) can use the feature in a non-perceptible way?

[nic-chen]

I think the main logic should be:
When APISIX starts, read the key /apisix/jsonschema of ETCD.
If the key does not exist or is empty, then read the JSON schema from APISIX and save to the ETCD key /apisix/jsonschema.
If the key already exists and is not empty, nothing will be done.
APISIX provides a command line tool that can be used to manually update the JSON schema, like: apisix sync-schema,
users can use this tool to update JSON schema when needed.
What do you think ?
cc @membphis @moonming @liuxiran @dabue @tokers @ShiningRush

So all schemas are put inside a single JSON string here?

BTW, have you consider the grayscale of APISIX itself? If the new version of APISIX has some incompatible changes for some schemas, once the schema is synced, all instances which shares the same ETCD cluster will be effected. So maybe versioning or other way like naming isolation is required.

Thanks for suggestion, that's really helpful.

So we could use key like /apisix/jsonschema/$schemaversion for isolation.

And I think we need to manually maintain the schema version in both APISIX and manager api.

what do you think ?

[tokers]

I think the main logic should be:
When APISIX starts, read the key /apisix/jsonschema of ETCD.
If the key does not exist or is empty, then read the JSON schema from APISIX and save to the ETCD key /apisix/jsonschema.
If the key already exists and is not empty, nothing will be done.
APISIX provides a command line tool that can be used to manually update the JSON schema, like: apisix sync-schema,
users can use this tool to update JSON schema when needed.
What do you think ?
cc @membphis @moonming @liuxiran @dabue @tokers @ShiningRush

So all schemas are put inside a single JSON string here?
BTW, have you consider the grayscale of APISIX itself? If the new version of APISIX has some incompatible changes for some schemas, once the schema is synced, all instances which shares the same ETCD cluster will be effected. So maybe versioning or other way like naming isolation is required.

Thanks for suggestion, that's really helpful.

So we could use key like /apisix/jsonschema/$schemaversion for isolation.

And I think we need to manually maintain the schema version in both APISIX and manager api.

what do you think ?

What about using a separate key for each schema? just use /apisix/jsonschema/ as the prefix, like other resources?

/apisix/jsonschema/route
/apisix/jsonschema/upstream
/apisix/jsonschema/plugin/echo

I am not sure whether the versioning is a good way to solve this, because APISIX should remember each schema's version that it needs.

[nic-chen]

Is it necessary to automatically trigger schema synchronization when adding a new schema(e.g: add a new plugin) or modifying a schema, so that the consumer of the schema (e.g manager-api) can use the feature in a non-perceptible way?

Thanks for suggestion. I have considered this before, but this will cause version problems, so I think it needs to be automatically generated if it does not exist, and the update should be triggered manually by the user. After solved the version issue, it could be automatically updated.

[nic-chen]

I think the main logic should be:
When APISIX starts, read the key /apisix/jsonschema of ETCD.
If the key does not exist or is empty, then read the JSON schema from APISIX and save to the ETCD key /apisix/jsonschema.
If the key already exists and is not empty, nothing will be done.
APISIX provides a command line tool that can be used to manually update the JSON schema, like: apisix sync-schema,
users can use this tool to update JSON schema when needed.
What do you think ?
cc @membphis @moonming @liuxiran @dabue @tokers @ShiningRush

So all schemas are put inside a single JSON string here?
BTW, have you consider the grayscale of APISIX itself? If the new version of APISIX has some incompatible changes for some schemas, once the schema is synced, all instances which shares the same ETCD cluster will be effected. So maybe versioning or other way like naming isolation is required.

Thanks for suggestion, that's really helpful.
So we could use key like /apisix/jsonschema/$schemaversion for isolation.
And I think we need to manually maintain the schema version in both APISIX and manager api.
what do you think ?

What about using a separate key for each schema? just use /apisix/jsonschema/ as the prefix, like other resources?

/apisix/jsonschema/route
/apisix/jsonschema/upstream
/apisix/jsonschema/plugin/echo

I am not sure whether the versioning is a good way to solve this, because APISIX should remember each schema's version that it needs.

I think separating them will increase the complexity, especially the version issue.

[membphis]

JSON Schema version is a useful field, we need to store and compare it.

Needs to show a warning message if the JSON schema version is different from APISIX version.

[ShiningRush]

Here is no need create version for schema, because the schema is build from APISIX, so schema version should same as APISIX.
I think we can do like that:

• APISIX will automatically write schema and APISIX-VERSION to /apisix/jsonschema at start
• Dashboard maintain a white/black version list in config and watch the schema changes, if the APISIX version is incompatible we can show message to user such as the current dashboard version is incompatible with APISIX(x.x.x), please update dashboard or APISIX

Automation is very important to a product, we should not let users decide when to synchronize and when not to synchronize schema when it's effect scope is controllable( schema mismatch is will just effect dashboard not data plane), it will burden users.

[tokers]

Here is no need create version for schema, because the schema is build from APISIX, so schema version should same as APISIX.
I think we can do like that:

• APISIX will automatically write schema and APISIX-VERSION to /apisix/jsonschema at start
• Dashboard maintain a white/black version list in config and watch the schema changes, if the APISIX version is incompatible we can show message to user such as the current dashboard version is incompatible with APISIX(x.x.x), please update dashboard or APISIX

Automation is very important to a product, we should not let users decide when to synchronize and when not to synchronize schema when it's effect scope is controllable( schema mismatch is will just effect dashboard not data plane), it will burden users.

The first step can be done in the apisix init step.

[membphis]

The first step can be done in the apisix init step.

that is not a good idea, it is unsafe

[tokers]

If we want to make sure the data plane doesn't modify etcd, another control plane component should be introduced to take over the uploading of jsonschema from APISIX, status collection or even the configuration delivery.

[membphis]

another control plane component should be introduced to take over the uploading of jsonschema from APISIX,

yes, that is the right way. How about this? apisix-admin sync-schema

[tokers]

another control plane component should be introduced to take over the uploading of jsonschema from APISIX,

yes, that is the right way. How about this? apisix-admin sync-schema

That's good. But the jsonschema source still in APIS
IX, we have to solve another problem that how apisix-admin gets the jsonschema, maybe apisix-admin could be written based on ngx_lua and share the code base of APISIX.

[nic-chen]

another control plane component should be introduced to take over the uploading of jsonschema from APISIX,

yes, that is the right way. How about this? apisix-admin sync-schema

That's good. But the jsonschema source still in APIS
IX, we have to solve another problem that how apisix-admin gets the jsonschema, maybe apisix-admin could be written based on ngx_lua and share the code base of APISIX.

then we go back to the old issue again..

[ShiningRush]

that is not a good idea, it is unsafe

@membphis Why it is unsafe, I can think of the worst result that is not compatible with manager-api, but this does not affect the normal works of the data plane, does it?
Because the synchronization schema is only a system bypass

[tokers]

another control plane component should be introduced to take over the uploading of jsonschema from APISIX,

yes, that is the right way. How about this? apisix-admin sync-schema

That's good. But the jsonschema source still in APIS

IX, we have to solve another problem that how apisix-admin gets the jsonschema, maybe apisix-admin could be written based on ngx_lua and share the code base of APISIX.

then we go back to the old issue again..

We have to if we wanna the data plane reads etcd only. 😂

[membphis]

I gona to close this issue, we have discussed at maillist: https://lists.apache.org/thread.html/rb1c19fe9b6e51ede3b5aead122e2bcd4de6249f42b971eabfb8a79ac%40%3Cdev.apisix.apache.org%3E

I'll create a new issue with the latest result.