DependsOn Ids for Multipart/Mixed Batch #1020

biaol-odata · 2017-12-20T18:03:59Z

Reader extracts contentId header value for both changeset and top-level requests;
Uses DependsOnIdsTracker to track request ids in Multipart/Mixed batch;
DependsOnIds validation is performed only for JsonBatch since it is client provided; not for Multipart/Mixed since it is not provided by client.

Issues

This pull request is to introduce DependsOn ids to Multipart/Mixed batch format. This makes DependsOn Ids available for query when service processes the batch request.

Description

Introducing DependsOn IDs to Multipart/Mixed batch so that both formats are symmetrical in semantics. The dependsOn ids for multipart/mixed format is derived from protocol and is not explicitly provided by client.

Checklist (Uncheck if it is not completed)

Test cases added
Build and test with one-click build and test script passed

Additional work necessary

If documentation update is needed, please add "Docs Needed" label to the issue and provide details about the required document change in the issue.

- Reader extracts contentId header value for both changeset and top-level requests; - Uses DependsOnIdsTracker to track request ids in Multipart/Mixed batch; - DependsOnIds validation is performed only for JsonBatch since it is client provided; not for Multipart/Mixed since it is not provided by client.

mikepizzo · 2017-12-21T18:43:12Z

src/Microsoft.OData.Core/ODataBatchWriter.cs

            {
+                // Validate dependsOn ids in Json batch.


Base class should not refer to derived classes, even in comments. We validate dependsOnIds if dependsOnIdsValidationRequired is true. Why it's true is outside of the scope of this class. #Closed

Please remove references to JSON and Multipart from comments in the base class. Comment should read something like "Validate dependsOnIds if specified".

In reply to: 158349752 [](ancestors = 158349752)

done for base classes reader and writer.

In reply to: 160554679 [](ancestors = 160554679,158349752)

mikepizzo · 2017-12-21T18:48:15Z

src/Microsoft.OData.Core/MultipartMixed/ODataMultipartMixedBatchWriter.cs

+                throw new ODataException(
+                    "For Multipart/Mixed batch format, request dependency is implicit by the protocol and should be null.");
+            }
+
            // write pending message data (headers, response line) for a previously unclosed message/request


Do not enforce null. We should validate that any ids specified are in the ids read by the tracker for this scope (changeset or toplevel). #Closed

removed null check / exception thrown

In reply to: 158350729 [](ancestors = 158350729)

mikepizzo · 2017-12-21T18:51:22Z

src/Microsoft.OData.Core/MultipartMixed/ODataMultipartMixedBatchWriter.cs

+                method, uri, contentId,
+                ODataMultipartMixedBatchWriterUtils.GetChangeSetIdFromBoundary(this.changeSetBoundary),
+                this.dependsOnIdsTracker.GetDependsOnIds(),
+                /*dependsOnIdsValidationRequired*/ false);


/dependsOnIdsValidationRequired/ false [](start = 16, length = 40)

Not sure why we wouldn't pass true in order to validate against the ids already read? Is this just for backward compatibility? Seems like the request is destined to fail if it doesn't reference a valid id. #Closed

Updated to use only dependsOnIds which is only null for the implicit dependsOnIds Multipart/Mixed case; and the GetDependsOnIds overrides in derived classes getting the convertedDependsOnIds in the BuildOperationRequestMessage method.

In reply to: 158351447 [](ancestors = 158351447)

mikepizzo · 2018-01-02T22:51:10Z

src/Microsoft.OData.Core/DependsOnIdsTracker.cs

+        /// <summary>
+        /// List of top-level dependsOn ids seen so far.
+        /// </summary>
+        private readonly IList<string> topLevelDepeondsOnIds;


topLevelDepeondsOnIds [](start = 39, length = 21)

Spelling; should be topLevelDependsOnIds #Closed

oops! fixed.

In reply to: 159333403 [](ancestors = 159333403)

mikepizzo · 2018-01-03T19:07:35Z

            if (HttpUtils.IsQueryMethod(method))

This restriction (inability to have query methods in a changeset) is no longer true. Please add "&& this.outputContext.MessageWriterSettings.Version <= ODataVersion.V4" #ByDesign

Refers to: src/Microsoft.OData.Core/ODataBatchWriter.cs:811 in 2ce1695. [](commit_id = 2ce1695, deletion_comment = False)

…h as a flag to uniquely indicate implicit dependsOnIds case

biaol-odata · 2018-01-09T23:24:20Z

            if (HttpUtils.IsQueryMethod(method))

I think this is aligned to the protocol and we keep this restriction at least for now per offline discussion with the group. Right?

In reply to: 355098382 [](ancestors = 355098382)

Refers to: src/Microsoft.OData.Core/ODataBatchWriter.cs:811 in 2ce1695. [](commit_id = 2ce1695, deletion_comment = False)

mikepizzo · 2018-01-09T23:50:39Z

src/Microsoft.OData.Core/DependsOnIdsTracker.cs

+//---------------------------------------------------------------------
+
+namespace Microsoft.OData
+{


If this is only used by multipart/mixed, should it go in the multipart/mixed namespace? #Closed

yes, moved.

In reply to: 160557298 [](ancestors = 160557298)

mikepizzo · 2018-01-10T00:01:25Z

src/Microsoft.OData.Core/ODataBatchWriter.cs

@@ -554,8 +549,7 @@ protected virtual IEnumerable<string> GetDependsOnRequestIds(IEnumerable<string>
                }
            }

-            ODataBatchUtils.ValidateReferenceUri(uri, flattenDependsOnIds,
-                this.outputContext.MessageWriterSettings.BaseUri, batchFormat);
+            ODataBatchUtils.ValidateReferenceUri(uri, convertedDependsOnIds, this.outputContext.MessageWriterSettings.BaseUri);


convertedDependsOnIds [](start = 54, length = 21)

I don't think this is the right logic. I thought that, if the derived class passed in null for dependsOnIds, we were going to use the ids from the payloadUriConverter (which would be be correctly scoped for the odata version). The ids from the DependsOnIdsTracker will be return only the ids in the changeset (if currently in changeset) otherwise top level ids, which is correct for 4.0, but wrong for 4.01. I actually don't think we need the dependsOnIdsTracker at all for the ODataMultipartMixedBatchWriter, only for the ODataMultipartMixedBatchReader. #Closed

There are pros and cons of using payloadUrlConverter. I thought about using it but ended up not doing it because it is undesirable to couple with legacy issue of difference between changeset scope in V4.0 implementation and batch scope in V4.0 protocol. In other words, current payloadUrlConverter is actually implemented incorrectly against the V4.0 protocol.
Furthermore, Json Batch Format 19.1: (for referencing url...)"The id of this request MUST be specified in the dependsOn name/value pair", it is natural extension to validate the referenced Id in url using the dependsOnIds?

I understand doing this will allow, on the writer side, the referencing of preceding top level request, e.g. request_2's url uses "$request_1", which should not be allowed in V4.0 implementation due to scope issue above but should not be an problem in V4.01.

All these said, I just don't want to let the scope issue above limit the validation on the writer side, and the referencing urls in both cases above will still be validated / processed correctly on the reader side.

In reply to: 160558905 [](ancestors = 160558905)

For the case of the non-null dependsOnIds (JSON batch and multipartmixed with explicit dependsOnIds) yes, absolutely we should use the (flattened) list of dependsOnIds explicitly passed by the user. However, for the dependsOnIds==null (multipartmixed implicit) case, we should default to the set of Ids in the payloadUrlConverter.

In reply to: 160567393 [](ancestors = 160567393,160558905)

updated with tests added.

In reply to: 160573714 [](ancestors = 160573714,160567393,160558905)

- Changing to use payloadUrlConverter to validate referencing uri for the case of Multipart batch with implicit dependsOnIds. - Tests added.

mikepizzo · 2018-01-11T01:22:03Z

src/Microsoft.OData.Core/MultipartMixed/ODataMultipartMixedBatchWriter.cs

            ODataBatchOperationRequestMessage operationRequestMessage = BuildOperationRequestMessage(
                this.RawOutputContext.OutputStream,
-                method, uri, contentId, /*groupId*/null, dependsOnIds, ODataFormat.Batch);


/groupId/null [](start = 40, length = 15)

Why did we start passing null, rather than ODataMultipartMixedBatchWriterUtils.GetChangeSetIdFromBoundary(this.changeSetBoundary)? #Closed

groupId is introduced with JSON Batch. For MultipartMixed batch, we have changeset boundary instead and we originally considered groupId value is always null. And then we decided to use the value extracted from changeset boundary as group Id, likely from one of your comments.

In reply to: 160844104 [](ancestors = 160844104)

Sorry; I was reacting to an apparent change to pass null here, but that was an old iteration. Now we pass GetChangeSetIdFromBoundary, which is what I would expect.

In reply to: 160846771 [](ancestors = 160846771,160844104)

mikepizzo · 2018-01-11T01:31:19Z

            if (HttpUtils.IsQueryMethod(method))

Actually, I don't think this restriction is in the json specification. We should probably make it either 4.0 specific (here), move it to multipart/mixed, or remove it all together. In general, if a service can support GET requests in a changeset I'd rather not get in their way, so we should probably just make this a 4.0 restriction.

In reply to: 356447377 [](ancestors = 356447377,355098382)

Refers to: src/Microsoft.OData.Core/ODataBatchWriter.cs:811 in 2ce1695. [](commit_id = 2ce1695, deletion_comment = False)

mikepizzo · 2018-01-11T01:38:16Z

src/Microsoft.OData.Core/ODataBatchReader.cs

@@ -333,9 +327,9 @@ protected virtual string GetCurrentGroupIdImplementation()
            string contentId,
            string groupId,
            IEnumerable<string> dependsOnRequestIds,
-            ODataFormat batchFormat)
+            bool dependsOnIdsValidationRequired)


bool dependsOnIdsValidationRequired [](start = 12, length = 35)

Can we do the same thing that we did in the writer, and pass null versus an empty list of requestIds instead of a boolean flag here, and validate the Url against the payloadUriConverter in the null dependsOnIds case? #Closed

I thought about that but we are out of luck / by design here. The dependsOnRequestIds is always not null, and it is always derived (implicit) for Multipart/Mixed batch, and always explicit for Json batch.

In reply to: 160846029 [](ancestors = 160846029)

What would happen if we removed the dependsOnIdsValidationRequired flag and always validated the dependsOnRequestIds? Shouldn't the implicit dependsOnRequestIds accumulated in the multipart/mixed case always be valid?

In reply to: 160847347 [](ancestors = 160847347,160846029)

In V4, since the payloadUriConverter is reset after changset, the validation (L336) will throw because implicit dependsOnIds could contain Content-IDs from preceding top-level requests --> error.

In reply to: 160848457 [](ancestors = 160848457,160847347,160846029)

Right; in 4.0, requests can only reference other requests within the same changeset, so payloadUriConverter only has the set of ids in the changeset, which is a subset of the requests that this request depends on coming before this request. Makes sense.

In reply to: 160849419 [](ancestors = 160849419,160848457,160847347,160846029)

…g> to IEnumerable<string>, not allowing element updates.

mikepizzo

mikepizzo reviewed Dec 21, 2017

View reviewed changes

biaol-odata mentioned this pull request Dec 21, 2017

Add dependsOn Ids semantics to Multiapart/Mixed batch format. #1012

Closed

mikepizzo reviewed Jan 2, 2018

View reviewed changes

biaol-odata added 2 commits January 5, 2018 16:58

CR comments.

243bc90

Addressing CR comment: using null dependsOnIds of MultipartMixed batc…

69b81cf

…h as a flag to uniquely indicate implicit dependsOnIds case

mikepizzo reviewed Jan 9, 2018

View reviewed changes

mikepizzo reviewed Jan 10, 2018

View reviewed changes

Addressed addtional CR comments, primarily:

a745e95

- Changing to use payloadUrlConverter to validate referencing uri for the case of Multipart batch with implicit dependsOnIds. - Tests added.

biaol-odata force-pushed the master_dependsOnIdsForMultipartBatch branch from fb2998f to a745e95 Compare January 10, 2018 23:15

mikepizzo reviewed Jan 11, 2018

View reviewed changes

Restricting BatchOperationRequestMessage.dependsOnIds from List<strin…

c7f10a4

…g> to IEnumerable<string>, not allowing element updates.

mikepizzo approved these changes Jan 12, 2018

View reviewed changes

biaol-odata merged commit ff6102b into OData:master Jan 12, 2018

biaol-odata deleted the master_dependsOnIdsForMultipartBatch branch January 12, 2018 22:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DependsOn Ids for Multipart/Mixed Batch #1020

DependsOn Ids for Multipart/Mixed Batch #1020

biaol-odata commented Dec 20, 2017

mikepizzo Dec 21, 2017 •

edited

Loading

mikepizzo Jan 9, 2018

biaol-odata Jan 10, 2018

mikepizzo Dec 21, 2017 •

edited

Loading

biaol-odata Jan 9, 2018

mikepizzo Dec 21, 2017 •

edited

Loading

biaol-odata Jan 9, 2018

mikepizzo Jan 2, 2018 •

edited

Loading

biaol-odata Jan 9, 2018

mikepizzo commented Jan 3, 2018 •

edited by biaol-odata

Loading

biaol-odata commented Jan 9, 2018

mikepizzo Jan 9, 2018 •

edited

Loading

biaol-odata Jan 10, 2018

mikepizzo Jan 10, 2018 •

edited

Loading

biaol-odata Jan 10, 2018

mikepizzo Jan 10, 2018

biaol-odata Jan 10, 2018

mikepizzo Jan 11, 2018 •

edited

Loading

biaol-odata Jan 11, 2018

mikepizzo Jan 11, 2018

mikepizzo commented Jan 11, 2018

mikepizzo Jan 11, 2018 •

edited

Loading

biaol-odata Jan 11, 2018

mikepizzo Jan 11, 2018

biaol-odata Jan 11, 2018

mikepizzo Jan 11, 2018

mikepizzo left a comment

DependsOn Ids for Multipart/Mixed Batch #1020

DependsOn Ids for Multipart/Mixed Batch #1020

Conversation

biaol-odata commented Dec 20, 2017

Issues

Description

Checklist (Uncheck if it is not completed)

Additional work necessary

mikepizzo Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo Jan 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo commented Jan 3, 2018 • edited by biaol-odata Loading

biaol-odata commented Jan 9, 2018

mikepizzo Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo Jan 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo commented Jan 11, 2018

mikepizzo Jan 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikepizzo left a comment

Choose a reason for hiding this comment

mikepizzo Dec 21, 2017 •

edited

Loading

mikepizzo Dec 21, 2017 •

edited

Loading

mikepizzo Dec 21, 2017 •

edited

Loading

mikepizzo Jan 2, 2018 •

edited

Loading

mikepizzo commented Jan 3, 2018 •

edited by biaol-odata

Loading

mikepizzo Jan 9, 2018 •

edited

Loading

mikepizzo Jan 10, 2018 •

edited

Loading

mikepizzo Jan 11, 2018 •

edited

Loading

mikepizzo Jan 11, 2018 •

edited

Loading