Add WebserverAPI for obtaing tags as json #4558

idotobi · 2020-04-13T11:05:09Z

This commit creates a new route /recipes/default/tags.json (boldly copying from /recipes/default/tiddlers.json).
It returns both tags for which a tiddler exists (i.e. "tiddlerExists": true) and tags for which no tiddler exists (i.e. "tiddlerExists": false). Furthermore, one can use the queryParameter count_tagged_tiddlers` to get the count of tiddlers having that particular tag. However, the default is not getting the count for performance concerns.

To allow for the internal qury used for /recipes/default/tags.json (i.e. [tags[]!is[system]sort[title]]) a new config tiddler was added.

Example default:

GET http://127.0.0.1:8080/recipes/default/tags.json

[
  {
    "title": "baretag",
    "tiddlerExists": false
  },
  {
    "title": "colortag",
    "color": "#eda9a9",
    "created": "20200413104646112",
    "modified": "20200413104647214",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "tiddlerExists": true
  },
  {
    "title": "icontag",
    "created": "20200413104652331",
    "icon": "$:/core/images/add-comment",
    "modified": "20200413104652353",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "tiddlerExists": true
  },
  {
    "title": "tiddlertag",
    "created": "20200413104659294",
    "modified": "20200413104710342",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "tiddlerExists": true
  }
]

Example queryParameters:

GET http://127.0.0.1:8080/recipes/default/tags.json

Parameters:
filter = "[tags[]!is[system]sort[title]]" # this is the only query allowed by default.
count_tagged_tiddlers = "true"            # count usages of tag
exclude = ","                             # prevent exclusion of textfield

[
  {
    "title": "baretag",
    "tiddlerExists": false,
    "taggedCount": 1
  },
  {
    "title": "colortag",
    "color": "#eda9a9",
    "created": "20200413104646112",
    "modified": "20200413104647214",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "tiddlerExists": true,
    "taggedCount": 1
  },
  {
    "title": "icontag",
    "created": "20200413104652331",
    "icon": "$:/core/images/add-comment",
    "modified": "20200413104652353",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "tiddlerExists": true,
    "taggedCount": 1
  },
  {
    "title": "tiddlertag",
    "created": "20200413104659294",
    "modified": "20200413104710342",
    "type": "text/vnd.tiddlywiki",
    "text": "this is a tag which is described by a tiddler",
    "revision": 0,
    "tiddlerExists": true,
    "taggedCount": 1
  }
]

This commit creates a new route `/recipes/default/tags.json` (boldly copying from `/recipes/default/tiddlers.json`). It returns both tags for which a tiddler exists (i.e. `"tiddlerExists": true`) and tags for which no tiddler exists (i.e. "tiddlerExists": false`). Furthermore, one can use the queryParameter `count_tagged_tiddlers` to get the count of tiddlers having that particular tag. However, the default is not getting the count for performance concerns. To allow for the internal qury used for `/recipes/default/tags.json` (i.e. `[tags[]!is[system]sort[title]]`) a new config tiddler was added. Example default: ``` GET http://127.0.0.1:8080/recipes/default/tags.json ``` ```json [ { "title": "baretag", "tiddlerExists": false }, { "title": "colortag", "color": "#eda9a9", "created": "20200413104646112", "modified": "20200413104647214", "type": "text/vnd.tiddlywiki", "revision": 0, "tiddlerExists": true }, { "title": "icontag", "created": "20200413104652331", "icon": "$:/core/images/add-comment", "modified": "20200413104652353", "type": "text/vnd.tiddlywiki", "revision": 0, "tiddlerExists": true }, { "title": "tiddlertag", "created": "20200413104659294", "modified": "20200413104710342", "type": "text/vnd.tiddlywiki", "revision": 0, "tiddlerExists": true } ] ``` Example queryParameters: ``` GET http://127.0.0.1:8080/recipes/default/tags.json Parameters: filter = "[tags[]!is[system]sort[title]]" # this is the only query allowed by default. count_tagged_tiddlers = "true" # count usages of tag exclude = "," # prevent exclusion of textfield ``` ```json [ { "title": "baretag", "tiddlerExists": false, "taggedCount": 1 }, { "title": "colortag", "color": "#eda9a9", "created": "20200413104646112", "modified": "20200413104647214", "type": "text/vnd.tiddlywiki", "revision": 0, "tiddlerExists": true, "taggedCount": 1 }, { "title": "icontag", "created": "20200413104652331", "icon": "$:/core/images/add-comment", "modified": "20200413104652353", "type": "text/vnd.tiddlywiki", "revision": 0, "tiddlerExists": true, "taggedCount": 1 }, { "title": "tiddlertag", "created": "20200413104659294", "modified": "20200413104710342", "type": "text/vnd.tiddlywiki", "text": "this is a tag which is described by a tiddler", "revision": 0, "tiddlerExists": true, "taggedCount": 1 } ] ```

Jermolene · 2020-04-13T11:13:54Z

Thanks @idotobi, and welcome to the project. One minor thing is to ask if you could please sign the Contributor License Agreement, following the instructions in contributing.md in the root of this repo.

A couple of thoughts on this PR:

I think the same thing could be accomplished using the existing route /recipes/default/tiddlers.json and the filter [tags[]!is[system]sort[title]] specified as a query string?
As a general policy, in the interests of keeping the core slim I'm not keen on adding routes that aren't used by the core itself

idotobi · 2020-04-19T15:19:29Z

I just sent out the CLA signature PR (see #4581).

idotobi · 2020-04-19T15:56:29Z

@Jermolene:
Thanks for the comments!

I think the same thing could be accomplished using the existing route /recipes/default/tiddlers.json and the filter [tags[]!is[system]sort[title]] specified as a query string?

Yes, this is doable and this is actually what I did in the first place. The problem with this solution is that it does not give you tags which are not at the same time tiddlers. The problem here is if(tiddler) in the following part of the route:
https://github.com/Jermolene/TiddlyWiki5/blob/e71a27ac2d71f2e48f9e4e9156b59bb3ecc2a105/core/modules/server/routes/get-tiddlers-json.js#L35-L43

Is there a better way to not only get tiddlers but also 'bare'-tags? If this is possible with existing APIs that would be great.

As a general policy, in the interests of keeping the core slim I'm not keen on adding routes that aren't used by the core itself

I totally understand that you want to keep the API as small as possible. I was tinkering with the idea of easily creating reference tiddlers for the current website (similar to bookmarks) with a firefox extension. For this it would have been handy to be able to get the tags for auto-completion. However, my project does not necessarily depend on this feature, so there is no need to go out of the way to implement it. I just thought it might be a useful route to have 😇 .

Just for my future reference can something like a route be also implemented as a plugin?

Jermolene · 2020-04-20T12:09:47Z

Hi @idotobi that makes perfect sense. Perhaps the way to go might be to add a new query string parameter "include-missing=yes" that would cause missing tiddlers to be also returned? We could return them as a tiddler consisting of just the title?

idotobi · 2020-04-28T06:21:07Z

@Jermolene : That sounds like a good idea. I would propose to call the query string parameter include-non-tiddlers=yes. Than it would be very self explanatory.

On the expected response:
Currently a tag-tiddler returns as

  {
    "title": "tiddlertag",
    "created": "20200413104659294",
    "modified": "20200413104710342",
    "type": "text/vnd.tiddlywiki",
    "text": "this is a tag which is described by a tiddler",
    "revision": 0,
  }

What should be return for non-tiddler-tags? Do you mean a simple json object like the following:

{
      "title": "non-tiddler-tag"
}

This way it would be easy to iterate over it and by testing whether the json-object has a "type": "text/vnd.tiddlywiki" one could distinguish tiddlers from non-tiddlers.

One could also give it an explicit type, but I'm not sure what the correct type for non-tiddlers should be.

{
      "title": "non-tiddler-tag",
      "type": ???
}

pmario · 2020-04-28T08:30:11Z

The only field, that is important for a tiddler is "title". So you are good. If no type is used, it will default to "text/vnd.tiddlywiki"

idotobi · 2020-04-28T15:26:08Z

Ok, so @pmario you would propose to represent non-tiddlers as the following?

{
      "title": "non-tiddler-tag"
}

pmario · 2020-04-28T20:14:32Z

hmmmm, ... yes. BUT You are not allowed to create them as real tiddlers.

In reality they don't exist. You only get them back because you explicitly requested: "include-missing=yes"

idotobi · 2020-05-01T10:31:59Z

@pmario : Correct. I'm just trying to figure out how to properly represent non-tiddlers in the json response.

As I see it, whatever the response would be it should be easy to parse for the downstream application using it. The two important criteria I see for that purpose are:

response should be easy to loop through
- => non-tiddlers and tiddlers need to be represented by similar data structures inside a list (e.g json-objects{})
non-tiddlers and tiddlers should be easy to distinguish
- => This is where the question about if there's a type for non-tiddlers comes into play

pmario · 2020-05-02T08:08:15Z

In the OP there has been a tiddlerExists flag. IMO it may be possible to use this info for tag-tiddlers. Just set it to false. Or you can create a new flag. eg: virtualTag=true or something similar. ... Since the flag isn't visible for the user, it doesn't really matter.

Move the functionality to include non-tiddlers (useful for obtaining all tags) from the previously new created route `/recipes/default/tags.json` to the already existing route `/recipes/default/tiddlers.json`.

idotobi · 2020-05-02T10:20:26Z

I updated the pull request.

Following request:

http://127.0.0.1:8080/recipes/default/tiddlers.json?filter=%5Btags%5B%5D!is%5Bsystem%5Dsort%5Btitle%5D%5D&include_non_tiddlers=true

will yield:

[
  {
    "title": "baretag",
    "non_tiddler": true
  },
  {
    "title": "colortag",
    "color": "#eda9a9",
    "created": "20200413104646112",
    "modified": "20200413104647214",
    "type": "text/vnd.tiddlywiki",
    "revision": 0
  },
  {
    "title": "icontag",
    "created": "20200413104652331",
    "icon": "$:/core/images/add-comment",
    "modified": "20200413104652353",
    "type": "text/vnd.tiddlywiki",
    "revision": 0
  },
  {
    "title": "tiddlertag",
    "created": "20200413104659294",
    "modified": "20200413104710342",
    "type": "text/vnd.tiddlywiki",
    "revision": 0
  }
]

@pmario and @Jermolene : What do you think about that approach?

Jermolene · 2020-05-02T11:44:56Z

Hi @pmario @idotobi

hmmmm, ... yes. BUT You are not allowed to create them as real tiddlers.

It is in fact legal in TW5 to have a tiddler with a title but no other field (try executing $tw.wiki.addTiddler({title:"HelloThere"}) in the browser console of tiddlywiki.com).

will yield:

[
  {
    "title": "baretag",
    "non_tiddler": true
  },
etc

Care would have to be taken to distinguish the case where "non_tiddler" is set to a string, which would mean that it was a real tiddler with a field of that name.

I think the terminology could be improved. Perhaps the query parameter should be include-missing instead of include_non_tiddlers, and the response indicator be "missing": true rather than "non_tiddler": true.

I've some other minor comments on the code, too.

Jermolene

Thanks @idotobi!

Jermolene · 2020-05-02T11:46:07Z

core/modules/server/routes/get-tiddlers-json.js

@@ -13,13 +13,15 @@ GET /recipes/default/tiddlers/tiddlers.json?filter=<filter>
 "use strict";

 var DEFAULT_FILTER = "[all[tiddlers]!is[system]sort[title]]";
+var DEFAULT_INCLUDE_NON_TIDDLERS = false;


Is there any reason for pulling this out as a constant? We tend to do this for readability (e.g. making the default tiddler readable and obvious) but here there's not much benefit.

Especially together with your comment below you're right the apparent flexibility that comes from representing it as a default fallback parameter could cause more harm then good.
My intention was to improve readability and be consistent with the DEFAULT_FILTER, but I see they are actually semantically different from one another.

Jermolene · 2020-05-02T11:47:20Z

core/modules/server/routes/get-tiddlers-json.js


 exports.method = "GET";

 exports.path = /^\/recipes\/default\/tiddlers.json$/;

 exports.handler = function(request,response,state) {
 	var filter = state.queryParameters.filter || DEFAULT_FILTER;
+	var include_non_tiddlers = state.queryParameters.include_non_tiddlers || DEFAULT_INCLUDE_NON_TIDDLERS;


Surely we need to test that state.queryParameters.include_non_tiddlers is set to "yes"?

This is dangerous because if DEFAULT_INCLUDE_NON_TIDDLERS were true and state.queryParameters.include_non_tiddlers were false then the setting would end up being true. But as I say, I don't see the value in having a default fallback.

You're right. If the line were state.queryParameters.include_non_tiddlers || false would this already resolve your concern, or do think it's still better to require yes as the positive value for the query parameter.

We should test explicitly for the positive value "yes".

Jermolene · 2020-05-02T11:48:11Z

core/modules/server/routes/get-tiddlers-json.js

+			response_body.push(tiddlerFields);
+		} else {
+			if(include_non_tiddlers) {
+				let entry = {};


We stick to ECMAScript 2015 to keep compatibility with older JS environments.

Do you mean that let should be var? I just started to learn javascript so I'm not yet solid in my distinction between the different standards.
I also tried to seach for ECMAScript 2015 and it seems to support let (https://www.ecma-international.org/ecma-262/6.0/#sec-let-and-const-declarations).

hmm .. ES2015 is ES6 which would allow letand arrow functions.

Apologies, I meant to say ES5.1

Jermolene · 2020-05-02T11:48:58Z

core/modules/server/routes/get-tiddlers-json.js

-			tiddlers.push(tiddlerFields);
+			response_body.push(tiddlerFields);
+		} else {
+			if(include_non_tiddlers) {


We would generally use "includeNonTiddlers" in preference to underscores. But with the changes I suggested the variable should be "includeMissing".

Would you also prefer the camelCase in the json response?

The only problem I'd see with "includeMissing"is that it sounds a little bit paradox at first sight. How can one include something that is missing? That being said I don't have strong feelings on "includeMissing" vs. "includeNonTiddlers" and since you are definitely the one to judge what's intuitive for the TiddlyWiki community I'll go with whatever you prefer. Just let me know and I'll implement it.

Would you also prefer the camelCase in the json response?

I think the JSON response should also be includeMissing: true

The only problem I'd see with "includeMissing"is that it sounds a little bit paradox at first sight

Yes quite! I think it's consistent, though.

Jermolene · 2020-05-02T11:52:03Z

core/wiki/config/ServerExternalFiltersTags.tid

@@ -0,0 +1,2 @@
+title: $:/config/Server/ExternalFilters/[tags[]!is[system]sort[title]]


I don't think that the core should be ship with too many external filters enabled, especially ones that aren't used by the core.

That makes sense. Would there be a good spot in the documentation were I could include a section on how people can use the "new" extended server API? Otherwise, I'd expect it to stay a hidden feature.

Would there be a good spot in the documentation were I could include a section on how people can use the "new" extended server API?

Yes we should update the existing "WebServer API: Get All Tiddlers" docs tiddler.

idotobi · 2020-12-05T18:49:00Z

@Jermolene : It turned out that I didn't need this feature for my project so I didn't spend much time on this lately. If you still think this is a valuable addition I could finish the PR, otherwise I would just close it. What would you prefer?

Jermolene · 2020-12-07T16:12:49Z

Apologies @idotobi I do think this is a good implementation that satisfies a solid use case, but I have now left it too late for v5.1.23. I've put on the list for v5.1.24 (see #5231).

idotobi · 2020-12-07T21:01:49Z

Nothing to apologize :) I'll rebase the PR then and take in your latest comments to make it ready to merge (probably this weekend).

Jermolene · 2021-10-27T11:26:46Z

Hi @idotobi we're at a good point to work on this PR if you're still available? It looks like there's a merge conflict to be resolved, and I think some or all of the code comments still need resolution.

Move non-tiddler functionality to existing route

e9bae9d

Move the functionality to include non-tiddlers (useful for obtaining all tags) from the previously new created route `/recipes/default/tags.json` to the already existing route `/recipes/default/tiddlers.json`.

Jermolene reviewed May 2, 2020

View reviewed changes

Jermolene mentioned this pull request Dec 7, 2020

Tickets to be considered for post-v5.1.23 #5231

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WebserverAPI for obtaing tags as json #4558

Add WebserverAPI for obtaing tags as json #4558

idotobi commented Apr 13, 2020

Jermolene commented Apr 13, 2020

idotobi commented Apr 19, 2020

idotobi commented Apr 19, 2020

Jermolene commented Apr 20, 2020

idotobi commented Apr 28, 2020

pmario commented Apr 28, 2020 •

edited

Loading

idotobi commented Apr 28, 2020

pmario commented Apr 28, 2020

idotobi commented May 1, 2020

pmario commented May 2, 2020

idotobi commented May 2, 2020

Jermolene commented May 2, 2020

Jermolene left a comment

Jermolene May 2, 2020

idotobi May 3, 2020

Jermolene May 2, 2020

idotobi May 3, 2020

Jermolene Aug 28, 2020

Jermolene May 2, 2020

idotobi May 3, 2020

pmario May 3, 2020

Jermolene Aug 28, 2020

Jermolene May 2, 2020

idotobi May 3, 2020

Jermolene Aug 28, 2020

Jermolene May 2, 2020

idotobi May 3, 2020

Jermolene Aug 28, 2020

idotobi commented Dec 5, 2020

Jermolene commented Dec 7, 2020

idotobi commented Dec 7, 2020

Jermolene commented Oct 27, 2021

		@@ -0,0 +1,2 @@
		title: $:/config/Server/ExternalFilters/[tags[]!is[system]sort[title]]

Add WebserverAPI for obtaing tags as json #4558

Are you sure you want to change the base?

Add WebserverAPI for obtaing tags as json #4558

Conversation

idotobi commented Apr 13, 2020

Jermolene commented Apr 13, 2020

idotobi commented Apr 19, 2020

idotobi commented Apr 19, 2020

Jermolene commented Apr 20, 2020

idotobi commented Apr 28, 2020

pmario commented Apr 28, 2020 • edited Loading

idotobi commented Apr 28, 2020

pmario commented Apr 28, 2020

idotobi commented May 1, 2020

pmario commented May 2, 2020

idotobi commented May 2, 2020

Jermolene commented May 2, 2020

Jermolene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

idotobi commented Dec 5, 2020

Jermolene commented Dec 7, 2020

idotobi commented Dec 7, 2020

Jermolene commented Oct 27, 2021

pmario commented Apr 28, 2020 •

edited

Loading