Add a retry mechanism for authorization tests (#118)

* Add retries to authorization tests --------- Co-authored-by: Jarlath Holleran <[email protected]>
solid-contrib · Mar 20, 2024 · a430fc9 · a430fc9
1 parent 6fe7818
commit a430fc9
Show file tree

Hide file tree

Showing 7 changed files with 38 additions and 31 deletions.
diff --git a/web-access-control/protected-operation/common.feature b/web-access-control/protected-operation/common.feature
@@ -9,6 +9,12 @@ Scenario:
         return agentLowerCase !== 'public' ? clients[agentLowerCase].getAuthHeaders(method, url) : {}
       }
     """
+  * def includesExpectedStatus =
+    """
+      function (actual, expected) {
+        return expected.includes(actual);
+      }
+    """
   * def getRequestData =
     """
       function (type) {

diff --git a/web-access-control/protected-operation/read-access-agent.feature b/web-access-control/protected-operation/read-access-agent.feature
@@ -18,8 +18,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Bob    | can    | GET     | plain     | no        | R         | 200    |
@@ -57,8 +57,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/turtle'
     And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | PUT    | rdf       | no        | R         | [403]      |
@@ -88,8 +88,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/n3'
     And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method | type      | container | resource  | status |
       | Bob    | cannot | PATCH  | rdf       | no        | R         | 403    |
@@ -109,8 +109,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/plain'
     And request "Bob's text"
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type    | container | resource  | status          |
       | Bob    | cannot | PUT    | plain   | no        | R         | [403]           |
@@ -136,8 +136,8 @@ Feature: Only authenticated agents can read (and only that) a resource when gran
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | DELETE | plain     | no        | R         | [403]      |

diff --git a/web-access-control/protected-operation/read-access-bob.feature b/web-access-control/protected-operation/read-access-bob.feature
@@ -18,8 +18,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Bob    | can    | GET     | plain     | no        | R         | 200    |
@@ -57,8 +57,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/turtle'
     And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | PUT    | rdf       | no        | R         | [403]      |
@@ -88,8 +88,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/n3'
     And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method | type      | container | resource  | status |
       | Bob    | cannot | PATCH  | rdf       | no        | R         | 403    |
@@ -109,8 +109,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/plain'
     And request "Bob's text"
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type    | container | resource  | status          |
       | Bob    | cannot | PUT    | plain   | no        | R         | [403]           |
@@ -136,8 +136,8 @@ Feature: Only Bob can read (and only that) a resource when granted read access
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | DELETE | plain     | no        | R         | [403]      |

diff --git a/web-access-control/protected-operation/read-access-public.feature b/web-access-control/protected-operation/read-access-public.feature
@@ -18,8 +18,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Bob    | can    | GET     | plain     | no        | R         | 200    |
@@ -61,8 +61,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/turtle'
     And request '@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. <> rdfs:comment "Bob added this.".'
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | PUT    | rdf       | no        | R         | [403]      |
@@ -92,8 +92,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/n3'
     And request '@prefix solid: <http://www.w3.org/ns/solid/terms#>. _:insert a solid:InsertDeletePatch; solid:inserts { <> a <http://example.org#Foo> . }.'
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method | type      | container | resource  | status |
       | Bob    | cannot | PATCH  | rdf       | no        | R         | 403    |
@@ -113,8 +113,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = 'text/plain'
     And request "Bob's text"
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type    | container | resource  | status          |
       | Bob    | cannot | PUT    | plain   | no        | R         | [403]           |
@@ -140,8 +140,8 @@ Feature: Public agents can read (and only that) a resource when granted read acc
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status     |
       | Bob    | cannot | DELETE | plain     | no        | R         | [403]      |

diff --git a/web-access-control/protected-operation/write-access-agent.feature b/web-access-control/protected-operation/write-access-agent.feature
@@ -18,8 +18,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Bob    | cannot | GET     | plain     | no        | WAC       | 403    |
@@ -58,15 +58,15 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
 
     Given headers utils.authHeaders('GET', testResource.url, agent)
+    And retry until responseStatus == <readStatus>
     When method GET
-    Then status <readStatus>
 
     Examples:
       | agent  | result | method | type      | container | resource  | writeStatus          | readStatus |
@@ -94,15 +94,15 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
 
     Given headers utils.authHeaders('GET', testResource.url, agent)
+    And retry until responseStatus == <readStatus>
     When method GET
-    Then status <readStatus>
 
     Examples:
       | agent  | result | method | type    | container | resource  | writeStatus          | readStatus |
@@ -120,11 +120,12 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
       | Public | cannot | PATCH  | fictive | WAC       | inherited | [401]                | 401        |
 
   Scenario Outline: <agent> <result> <method> a <type> resource, when an authenticated agent has <container> access to the container and <resource> access to the resource
+
     * def testResource = utils.createResource(container, resource, type, 'authenticated')
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status               |
       | Bob    | cannot | DELETE | plain     | no        | C         | [403]                |

diff --git a/web-access-control/protected-operation/write-access-bob.feature b/web-access-control/protected-operation/write-access-bob.feature
@@ -18,8 +18,8 @@ Feature: Only Bob can write (and only that) a resource when granted write access
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Bob    | cannot | GET     | plain     | no        | WAC       | 403    |
@@ -58,15 +58,15 @@ Feature: Only Bob can write (and only that) a resource when granted write access
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
 
     Given headers utils.authHeaders('GET', testResource.url, agent)
+    And retry until responseStatus == <readStatus>
     When method GET
-    Then status <readStatus>
 
     Examples:
       | agent  | result | method | type      | container | resource  | writeStatus          | readStatus |
@@ -94,15 +94,15 @@ Feature: Only Bob can write (and only that) a resource when granted write access
     And headers utils.authHeaders(method, testResource.url, agent)
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
 
     Given headers utils.authHeaders('GET', testResource.url, agent)
+    And retry until responseStatus == <readStatus>
     When method GET
-    Then status <readStatus>
 
     Examples:
       | agent  | result | method | type    | container | resource  | writeStatus          | readStatus |
@@ -123,8 +123,8 @@ Feature: Only Bob can write (and only that) a resource when granted write access
     * def testResource = utils.createResource(container, resource, type, 'agent', webIds.bob)
     Given url testResource.url
     And headers utils.authHeaders(method, testResource.url, agent)
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status               |
       | Bob    | cannot | DELETE | plain     | no        | C         | [403]                |

diff --git a/web-access-control/protected-operation/write-access-public.feature b/web-access-control/protected-operation/write-access-public.feature
@@ -17,8 +17,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
   Scenario Outline: <agent> <result> read a <type> resource (<method>), when a public agent has <container> access to the container and <resource> access to the resource
     * def testResource = utils.testResources[utils.getResourceKey(container, resource, type)]
     Given url testResource.url
+    And retry until responseStatus == <status>
     When method <method>
-    Then status <status>
     Examples:
       | agent  | result | method  | type      | container | resource  | status |
       | Public | cannot | GET     | plain     | no        | WAC       | 401    |
@@ -43,8 +43,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
     Given url testResource.url
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
@@ -71,8 +71,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
     Given url testResource.url
     And header Content-Type = requestData.contentType
     And request requestData.requestBody
+    And retry until utils.includesExpectedStatus(responseStatus, <writeStatus>)
     When method <method>
-    Then match <writeStatus> contains responseStatus
     # Server may return payload with information about the operation e.g. "Created" so check it hasn't leaked the data which was PUT
     And string responseString = response
     And match responseString !contains requestData.responseShouldNotContain
@@ -96,8 +96,8 @@ Feature: Only authenticated agents can write (and only that) a resource when gra
   Scenario Outline: <agent> <result> <method> a <type> resource, when a public agent has <container> access to the container and <resource> access to the resource
     * def testResource = utils.createResource(container, resource, type, 'public')
     Given url testResource.url
+    And retry until utils.includesExpectedStatus(responseStatus, <status>)
     When method <method>
-    Then match <status> contains responseStatus
     Examples:
       | agent  | result | method | type      | container | resource  | status               |
       | Public | cannot | DELETE | plain     | no        | C         | [401]                |