feature(paging) Add ability to opt-out of force paging

[atruskie]

Currently when the query results of an API endpoint exceed 500 items, the results are automatically paged.

Add the option to opt out of this behaviour.

e.g. add a dont_page QSP for normal routes
and a dont_page attribute to the paging object used in filter objects

[cofiem]

The whole point of having the paging there is to guard against the API accidentally (or on purpose) slowing the website to a crawl. If the paging can be turned off, what's the point of having it?

Some other options:

• can the client cope with paging the results?
• is there some other way to aggregate the results on the server?

[atruskie]

Guarding against accidents is fine. Choosing to do advanced things is something else.

Simply put, I'm currently, in prod, just setting the items per page property to 100,000 - I'm getting around the limit with a hack. I thought it would be better to make the option explicit - because at least the intention is explicit - readable and even deprecated able.

I'm not really sure what advantage paging that much data would have in the client. For the visualise case I need all of it... so I'd just fetch all the pages sequentially.... which is slower for the client and is effectively the same load on the server.

Aggregating results is a possibility - but at this stage it seems quite complicated... at some point I potentially need the information for arbitrary items...

Additionally, rails doing a ton of useless processing (5:30 worth in the current example) is sheerly ridiculous... I'd say the server should be working smarter, not imposing arbitrary limits.

[cofiem]

Good points. I'll have a look at how this could be done in a smarter/less resource-intensive way.

[cofiem]

add qsp disable_paging=true or

"paging": {
    "disable_paging": "true"
}

[atruskie]

👍

[atruskie]

I updated the relevant wiki entries

[cofiem]

Thanks!