Container tooling #885

zslayton · 2024-12-19T22:17:53Z

This PR adds lots of new tooling methods and bookkeeping:

Tags each value with the variable that produced it, if any.
Renames UnexpandedField to FieldExpr, which is much clearer.
Adds container iterators that yield both macro invocations and their expansions, allowing tooling to see both and work with whatever they need.
Adds Span accessor for the delimited END byte.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2024-12-19T22:28:14Z

Codecov Report

Attention: Patch coverage is 57.78302% with 179 lines in your changes missing coverage. Please review.

Project coverage is 77.62%. Comparing base (75b0763) to head (4974541).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/lazy/expanded/struct.rs	61.76%	50 Missing and 28 partials ⚠️
src/lazy/expanded/sequence.rs	70.37%	2 Missing and 30 partials ⚠️
src/lazy/expanded/macro_evaluator.rs	61.90%	15 Missing and 1 partial ⚠️
src/lazy/expanded/template.rs	25.00%	15 Missing ⚠️
src/lazy/binary/raw/value.rs	0.00%	13 Missing ⚠️
src/lazy/decoder.rs	35.29%	11 Missing ⚠️
src/lazy/any_encoding.rs	0.00%	7 Missing ⚠️
src/lazy/binary/raw/v1_1/value.rs	0.00%	3 Missing ⚠️
src/lazy/text/value.rs	0.00%	3 Missing ⚠️
src/lazy/expanded/mod.rs	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #885      +/-   ##
==========================================
- Coverage   77.85%   77.62%   -0.24%     
==========================================
  Files         136      136              
  Lines       34703    35070     +367     
  Branches    34703    35070     +367     
==========================================
+ Hits        27017    27222     +205     
- Misses       5683     5791     +108     
- Partials     2003     2057      +54

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zslayton

🗺️ PR Tour 🧭

zslayton · 2024-12-19T22:31:59Z

src/lazy/any_encoding.rs

@@ -1047,6 +1048,15 @@ impl<'top> LazyRawValue<'top, AnyEncoding> for LazyRawAnyValue<'top> {
        }
    }

+    fn is_delimited(&self) -> bool {


🪧 Each encoded value kind can now report whether it is delimited. For text 1.0 and binary 1.0, this method just returns false.

zslayton · 2024-12-19T22:32:41Z

src/lazy/binary/raw/value.rs

+    fn delimited_end_span(&self) -> Span<'top> {
+        let bytes = self.span().bytes();
+        let end = bytes.len();
+        let range = if !self.is_delimited() {
+            end..end
+        } else {
+            debug_assert!(bytes[end - 1] == 0xF0);
+            end - 1..end
+        };
+        let end_bytes = bytes.get(range).unwrap();
+        Span::with_offset(self.range().end - end_bytes.len(), end_bytes)
+    }


🪧 For scalars and length-prefixed containers, this returns an empty slice.

zslayton · 2024-12-19T22:33:56Z

src/lazy/decoder.rs

+            EExp(eexp) => FieldExpr::EExp(eexp.resolve(context)?),
+        };
+        Ok(field)
+    }


🪧 A convenience method pattern used widely throughout the crate; resolve turns a raw value (or in this case, field) into its higher-level, context-ful equivalent.

zslayton · 2024-12-19T22:36:28Z

src/lazy/decoder.rs

-        /// Creates an iterator that converts each raw struct field into an `UnexpandedField`, a
+        /// Creates an iterator that converts each raw struct field into an `FieldExpr`, a
        /// common representation for both raw fields and template fields that is used in the
        /// expansion process.
-        fn unexpanded_fields(
+        fn field_exprs(


🪧 UnexpandedField was a clunky name, hopefully this is better. The new relationship between LazyExpandedField and FieldExpr mirrors the existing relationship between LazyExpandedValue and ValueExpr.

zslayton · 2024-12-19T22:37:09Z

src/lazy/encoder/binary/v1_1/flex_sym.rs

-                if input.len() < flex_int_len {
+                if input.len() <= flex_int_len {


🪧 This bug surfaced while I was working on an example file from @jobarr-amzn. This fixed it.

zslayton · 2024-12-19T22:37:54Z

src/lazy/expanded/macro_evaluator.rs

-    variable: Option<TemplateVariableReference<'top>>,
+    pub(crate) variable: Option<TemplateVariableReference<'top>>,


🪧 Macros now remember the variable (if any) that produced them, allowing them in turn to tag the values they produce.

zslayton · 2024-12-19T22:38:56Z

src/lazy/expanded/sequence.rs

+/// Like an [`ExpandedListIterator`], but will yield macro invocations before also yielding
+/// that macro's expansion.
+#[derive(Debug)]
+pub struct ListValueExprIterator<'top, D: Decoder> {


🪧 This section introduces iterators that return both the expressions and the values they expand to.

zslayton · 2024-12-19T22:40:50Z

src/lazy/expanded/struct.rs

+    impl<'top, D: Decoder> Iterator for FieldExprIterator<'top, D> {
+        type Item = IonResult<FieldExpr<'top, D>>;
+
+        fn next(&mut self) -> Option<Self::Item> {


🪧 This is pretty close to the regular struct iterator, expect that any time we would change states and keep reading, we now return the expression that caused us to change states. The following call to next() will resume reading.

src/lazy/expanded/struct.rs

jobarr-amzn · 2024-12-31T21:19:54Z

src/lazy/expanded/struct.rs

+    /// of the name/values pairs in the expansion: `(bar, 1)`, `(bar, 2)`, and `(bar, 3)`.
+    ///
+    /// In contrast, the `FieldExprIterator` would yield a `FieldExpr` for the name/macro
+    /// field expression (`NameMacro("foo", MacroExpr)`) followed by a `FieldExpr` for each of


Suggested change

/// field expression (`NameMacro("foo", MacroExpr)`) followed by a `FieldExpr` for each of

/// field expression (`NameMacro("bar", MacroExpr)`) followed by a `FieldExpr` for each of

jobarr-amzn · 2024-12-31T21:46:40Z

src/lazy/expanded/struct.rs

+            assert!(matches!(
+                fields.next().unwrap()?,
+                FieldExpr::NameMacro(name, invocation)
+                    if name.read()?.text() == Some("bar") && matches!(invocation.kind(), MacroExprKind::EExp(eexp) if eexp.invoked_macro.name() == Some("three_values"))
+            ));


On the one hand, it's cool that we can do this in Rust. On the other hand... 😨

I wish there was a better syntax for expressing a set of constraints like this.

A lot of the lower-level types are missing convenience methods like expect_next(), expect_name_macro(), etc that would make this prettier. 😞 The unit tests are the only place that would really benefit, but that might be reason enough to implement them.

Co-authored-by: Joshua Barr <[email protected]>

zslayton added 8 commits December 17, 2024 10:55

Records which variable produced each expr

50e0ae7

Rename UnexpandedField to FieldSource

9a775ee

Adds iterator over struct field expressions _and_ expansions

bc7a5e7

Adds iterators over ValueExpr in a list or sexp

64615df

Renames FieldSource to FieldExpr

8bf5fec

Additional tooling methods for binary encoded values

080b815

Feature gate type import

c738812

Removes unused field

535a0c1

zslayton marked this pull request as ready for review December 19, 2024 22:24

zslayton added 2 commits December 19, 2024 17:29

Feature gate fix

4f8386c

doc comment fix

b738f70

zslayton commented Dec 19, 2024

View reviewed changes

zslayton mentioned this pull request Dec 19, 2024

Shows e-exprs inside container literals, delimited END amazon-ion/ion-cli#186

Merged

jobarr-amzn approved these changes Dec 31, 2024

View reviewed changes

zslayton and others added 2 commits January 3, 2025 15:52

Update src/lazy/expanded/struct.rs

2ec8858

Co-authored-by: Joshua Barr <[email protected]>

Merge branch 'main' into container-tooling

4974541

zslayton merged commit 54b16d8 into main Jan 3, 2025
32 of 34 checks passed

zslayton deleted the container-tooling branch January 3, 2025 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Container tooling #885

Container tooling #885

zslayton commented Dec 19, 2024 •

edited

Loading

codecov bot commented Dec 19, 2024 •

edited

Loading

zslayton left a comment

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

zslayton Dec 19, 2024

jobarr-amzn Dec 31, 2024

jobarr-amzn Dec 31, 2024

zslayton Jan 3, 2025

		if input.len() < flex_int_len {
		if input.len() <= flex_int_len {

		variable: Option<TemplateVariableReference<'top>>,
		pub(crate) variable: Option<TemplateVariableReference<'top>>,

	/// field expression (`NameMacro("foo", MacroExpr)`) followed by a `FieldExpr` for each of
	/// field expression (`NameMacro("bar", MacroExpr)`) followed by a `FieldExpr` for each of

Container tooling #885

Container tooling #885

Conversation

zslayton commented Dec 19, 2024 • edited Loading

codecov bot commented Dec 19, 2024 • edited Loading

Codecov Report

zslayton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zslayton commented Dec 19, 2024 •

edited

Loading

codecov bot commented Dec 19, 2024 •

edited

Loading