Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a retry mechanism for authorization tests #118

Merged
merged 5 commits into from
Mar 20, 2024
Merged

Add a retry mechanism for authorization tests #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8047840
Add retries to authorization tests
acoburn Mar 19, 2024
00efb1e
Add match clause
acoburn Mar 19, 2024
f152a06
Using a util function for retries
jholleran Mar 19, 2024
1e20a30
Use shared function
acoburn Mar 19, 2024
7f147ea
Return unexpected change
acoburn Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions web-access-control/protected-operation/common.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ Scenario:
return agentLowerCase !== 'public' ? clients[agentLowerCase].getAuthHeaders(method, url) : {}
}
"""
* def includesExpectedStatus =
"""
function (actual, expected) {
return expected.includes(actual);
}
"""
* def getRequestData =
"""
function (type) {
Expand Down
10 changes: 5 additions & 5 deletions web-access-control/protected-operation/read-access-agent.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | can | GET | plain | no | R | 200 |
Expand Down Expand Up @@ -57,8 +57,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/turtle'
And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | rdf | no | R | [403] |
Expand Down Expand Up @@ -88,8 +88,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/n3'
And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PATCH | rdf | no | R | 403 |
Expand All @@ -109,8 +109,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/plain'
And request "Bob's text"
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | plain | no | R | [403] |
Expand All @@ -136,8 +136,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | DELETE | plain | no | R | [403] |
Expand Down
10 changes: 5 additions & 5 deletions web-access-control/protected-operation/read-access-bob.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | can | GET | plain | no | R | 200 |
Expand Down Expand Up @@ -57,8 +57,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/turtle'
And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | rdf | no | R | [403] |
Expand Down Expand Up @@ -88,8 +88,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/n3'
And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PATCH | rdf | no | R | 403 |
Expand All @@ -109,8 +109,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/plain'
And request "Bob's text"
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | plain | no | R | [403] |
Expand All @@ -136,8 +136,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | DELETE | plain | no | R | [403] |
Expand Down
10 changes: 5 additions & 5 deletions web-access-control/protected-operation/read-access-public.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | can | GET | plain | no | R | 200 |
Expand Down Expand Up @@ -61,8 +61,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/turtle'
And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | rdf | no | R | [403] |
Expand Down Expand Up @@ -92,8 +92,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/n3'
And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PATCH | rdf | no | R | 403 |
Expand All @@ -113,8 +113,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = 'text/plain'
And request "Bob's text"
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | PUT | plain | no | R | [403] |
Expand All @@ -140,8 +140,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | DELETE | plain | no | R | [403] |
Expand Down
13 changes: 7 additions & 6 deletions web-access-control/protected-operation/write-access-agent.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | GET | plain | no | WAC | 403 |
Expand Down Expand Up @@ -58,15 +58,15 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain

Given headers utils.authHeaders('GET', testResource.url, agent)
And retry until responseStatus == <readStatus>
When method GET
Then status <readStatus>

Examples:
| agent | result | method | type | container | resource | writeStatus | readStatus |
Expand Down Expand Up @@ -94,15 +94,15 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain

Given headers utils.authHeaders('GET', testResource.url, agent)
And retry until responseStatus == <readStatus>
When method GET
Then status <readStatus>

Examples:
| agent | result | method | type | container | resource | writeStatus | readStatus |
Expand All @@ -120,11 +120,12 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
| Public | cannot | PATCH | fictive | WAC | inherited | [401] | 401 |

Scenario Outline: <agent> <result> <method> a <type> resource, when an authenticated agent has <container> access to the container and <resource> access to the resource

* def testResource = utils.createResource(container, resource, type, 'authenticated')
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | DELETE | plain | no | C | [403] |
Expand Down
12 changes: 6 additions & 6 deletions web-access-control/protected-operation/write-access-bob.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Feature: Only Bob can write (and only that) a resource when granted write access
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | GET | plain | no | WAC | 403 |
Expand Down Expand Up @@ -58,15 +58,15 @@ Feature: Only Bob can write (and only that) a resource when granted write access
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain

Given headers utils.authHeaders('GET', testResource.url, agent)
And retry until responseStatus == <readStatus>
When method GET
Then status <readStatus>

Examples:
| agent | result | method | type | container | resource | writeStatus | readStatus |
Expand Down Expand Up @@ -94,15 +94,15 @@ Feature: Only Bob can write (and only that) a resource when granted write access
And headers utils.authHeaders(method, testResource.url, agent)
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain

Given headers utils.authHeaders('GET', testResource.url, agent)
And retry until responseStatus == <readStatus>
When method GET
Then status <readStatus>

Examples:
| agent | result | method | type | container | resource | writeStatus | readStatus |
Expand All @@ -123,8 +123,8 @@ Feature: Only Bob can write (and only that) a resource when granted write access
* def testResource = utils.createResource(container, resource, type, 'agent', webIds.bob)
Given url testResource.url
And headers utils.authHeaders(method, testResource.url, agent)
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Bob | cannot | DELETE | plain | no | C | [403] |
Expand Down
8 changes: 4 additions & 4 deletions web-access-control/protected-operation/write-access-public.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
Scenario Outline: <agent> <result> read a <type> resource (<method>), when a public agent has <container> access to the container and <resource> access to the resource
* def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
Given url testResource.url
And retry until responseStatus == <status>
When method <method>
Then status <status>
Examples:
| agent | result | method | type | container | resource | status |
| Public | cannot | GET | plain | no | WAC | 401 |
Expand All @@ -43,8 +43,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
Given url testResource.url
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain
Expand All @@ -71,8 +71,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
Given url testResource.url
And header Content-Type = requestData.contentType
And request requestData.requestBody
And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
When method <method>
Then match <writeStatus> contains responseStatus
# Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
And string responseString = response
And match responseString !contains requestData.responseShouldNotContain
Expand All @@ -96,8 +96,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
Scenario Outline: <agent> <result> <method> a <type> resource, when a public agent has <container> access to the container and <resource> access to the resource
* def testResource = utils.createResource(container, resource, type, 'public')
Given url testResource.url
And retry until utils.includesExpectedStatus(responseStatus, <status>)
When method <method>
Then match <status> contains responseStatus
Examples:
| agent | result | method | type | container | resource | status |
| Public | cannot | DELETE | plain | no | C | [401] |
Expand Down