feat: add contains method and improve index config schema

adobe · Dec 2, 2021 · 97c5134 · 97c5134
1 parent 7269fb3
commit 97c5134
Show file tree

Hide file tree

Showing 58 changed files with 5,625 additions and 2,067 deletions.
diff --git a/docs/API.md b/docs/API.md
@@ -270,6 +270,23 @@ but provides better error messages.</p>
 <dt><a href="#dumpDOM">dumpDOM(actual, expected, level)</a></dt>
 <dd><p>prints dom in order for changes to be more discernible.</p>
 </dd>
+<dt><a href="#match">match(globs, path, defaultValue)</a> ⇒ <code>boolean</code></dt>
+<dd><p>Return a flag indicating whether a particular path is matches all given glob patterns.</p>
+</dd>
+<dt><a href="#contains">contains(cfg, path)</a> ⇒ <code>boolean</code></dt>
+<dd><p>Return a flag indicating whether a particular path is contained
+in the indexing configuration (include or exclude element). This
+is true if a path is included and <em>not</em> excluded.</p>
+</dd>
+<dt><a href="#getDOMValue">getDOMValue(elements, expression, log, vars)</a></dt>
+<dd><p>Return a value in the DOM by evaluating an expression</p>
+</dd>
+<dt><a href="#indexResource">indexResource(path, response, config, log)</a> ⇒ <code>object</code></dt>
+<dd><p>Given a response, extract a value and evaluate an expression
+on it. The index contains the CSS selector that will select the
+value(s) to process. If we get multiple values, we return an
+array.</p>
+</dd>
 <dt><a href="#processQueue">processQueue(queue, fn, [maxConcurrent])</a> ⇒</dt>
 <dd><p>Processes the given queue concurrently. The handler functions can add more items to the queue
 if needed.</p>
@@ -1360,6 +1377,67 @@ prints dom in order for changes to be more discernible.
 | expected | <code>object</code> |  | node from test domain page |
 | level | <code>number</code> | <code>0</code> | current level in recursion tree return dump of dom that is indented at every level by level*2 spaces |
 
+<a name="match"></a>
+
+## match(globs, path, defaultValue) ⇒ <code>boolean</code>
+Return a flag indicating whether a particular path is matches all given glob patterns.
+
+**Kind**: global function  
+**Returns**: <code>boolean</code> - whether path matches the globs  
+
+| Param | Type | Description |
+| --- | --- | --- |
+| globs | <code>Array.&lt;string&gt;</code> | globbing patterns |
+| path | <code>string</code> | path to check |
+| defaultValue | <code>boolean</code> | what to return if `globs` is undefined |
+
+<a name="contains"></a>
+
+## contains(cfg, path) ⇒ <code>boolean</code>
+Return a flag indicating whether a particular path is contained
+in the indexing configuration (include or exclude element). This
+is true if a path is included and *not* excluded.
+
+**Kind**: global function  
+**Returns**: <code>boolean</code> - whether path is included in configuration  
+
+| Param | Type | Description |
+| --- | --- | --- |
+| cfg | <code>Index</code> | indexing configuration's |
+| path | <code>string</code> | path to check |
+
+<a name="getDOMValue"></a>
+
+## getDOMValue(elements, expression, log, vars)
+Return a value in the DOM by evaluating an expression
+
+**Kind**: global function  
+
+| Param | Type |
+| --- | --- |
+| elements | <code>Array.&lt;HTMLElement&gt;</code> | 
+| expression | <code>string</code> | 
+| log | <code>Logger</code> | 
+| vars | <code>object</code> | 
+
+<a name="indexResource"></a>
+
+## indexResource(path, response, config, log) ⇒ <code>object</code>
+Given a response, extract a value and evaluate an expression
+on it. The index contains the CSS selector that will select the
+value(s) to process. If we get multiple values, we return an
+array.
+
+**Kind**: global function  
+**Returns**: <code>object</code> - extracted properties  
+
+| Param | Type | Description |
+| --- | --- | --- |
+| path | <code>string</code> | Path of document retrieved |
+| response | <code>object</code> | response containing body and headers |
+| config | <code>Index</code> | indexing configuration |
+| log | <code>Logger</code> | logger |
+
 <a name="processQueue"></a>
 
 ## processQueue(queue, fn, [maxConcurrent]) ⇒
@@ -1371,8 +1449,8 @@ if needed.
 
 | Param | Type | Default | Description |
 | --- | --- | --- | --- |
-| queue | <code>Array.&lt;\*&gt;</code> |  | A list of tasks |
-| fn | <code>function</code> |  | A handler function `fn(task:any, queue:array, results:array)` |
+| queue | <code>Iterable</code> \| <code>Array</code> |  | A list of tasks |
+| fn | <code>ProcessQueueHandler</code> |  | A handler function `fn(task:any, queue:array, results:array)` |
 | [maxConcurrent] | <code>number</code> | <code>8</code> | Concurrency level |
 
 <a name="pruneEmptyValues"></a>

diff --git a/docs/README.md b/docs/README.md
@@ -18,8 +18,6 @@
 
 *   [Index Configuration](./indexconfig.md) – `https://ns.adobe.com/helix/shared/indexconfig`
 
-*   [Language](./language.md) – `https://ns.adobe.com/helix/shared/language`
-
 *   [Markup Configuration](./markup.md "A container for markup mappings") – `https://ns.adobe.com/helix/shared/markup`
 
 *   [Markup Mapping](./markupmapping.md "A markup mapping defines how Helix should generate markup for certain Markdown or DOM patterns") – `https://ns.adobe.com/helix/shared/markupmapping`
@@ -48,10 +46,12 @@
 
 *   [Runtime Strain](./runtimestrain.md "A runtime strain is a combination of code and content that enables the creation of a digital experience") – `https://ns.adobe.com/helix/shared/runtimestrain`
 
-*   [Sitemap](./sitemap.md) – `https://ns.adobe.com/helix/shared/sitemap`
+*   [Sitemap](./sitemap.md "The sitemap configuration for a Project Helix website") – `https://ns.adobe.com/helix/shared/sitemap`
 
 *   [Sitemap Configuration](./sitemapconfig.md) – `https://ns.adobe.com/helix/shared/sitemapconfig`
 
+*   [Sitemap Language](./sitemap-language.md "The sitemap language configuration for a sitemap") – `https://ns.adobe.com/helix/shared/sitemap-language`
+
 *   [Strains](./strains.md) – `https://ns.adobe.com/helix/shared/strains`
 
 *   [Vanity URL mapping](./vanity.md) – `https://ns.adobe.com/helix/shared/vanity`
@@ -66,42 +66,20 @@
 
 ### Objects
 
-*   [Git URL](./runtimestrain-properties-code-oneof-git-url.md "Representation of the fragments of a Git URL") – `https://ns.adobe.com/helix/shared/giturl#/properties/code/oneOf/1`
-
-*   [Git URL](./runtimestrain-properties-static-oneof-git-url.md "Representation of the fragments of a Git URL") – `https://ns.adobe.com/helix/shared/staticgiturl#/properties/static/oneOf/1`
-
-*   [Languages](./sitemap-properties-languages.md "The languages to add to the sitemap") – `https://ns.adobe.com/helix/shared/sitemap#/properties/languages`
-
-*   [Markup Mapping](./markup-markup-mapping.md) – `https://ns.adobe.com/helix/shared/markupmapping#/additionalProperties`
-
 *   [Multi-Value Property](./property-oneof-multi-value-property.md "The property in an index") – `https://ns.adobe.com/helix/shared/property#/oneOf/1`
 
 *   [Multi-Value Property](./property-oneof-multi-value-property.md "The property in an index") – `https://ns.adobe.com/helix/shared/property#/oneOf/1`
 
-*   [Origin](./proxystrain-properties-origin-anyof-origin.md "Representation of a origin host for a proxy strain") – `https://ns.adobe.com/helix/shared/origin#/properties/origin/anyOf/1`
-
-*   [Performance](./proxystrain-properties-performance.md "Performance testing details") – `https://ns.adobe.com/helix/shared/performance#/properties/perf`
-
 *   [Properties](./index-properties-properties.md "The properties to add to the index") – `https://ns.adobe.com/helix/shared/index#/properties/properties`
 
-*   [Property](./index-properties-properties-property.md) – `https://ns.adobe.com/helix/shared/property#/properties/properties/additionalProperties`
-
-*   [Proxy Strain](./strains-definitions-anystrain-oneof-proxy-strain.md "A strain is a combination of code and content that enables the creation of a digital experience") – `https://ns.adobe.com/helix/shared/proxystrain#/definitions/anystrain/oneOf/0`
-
 *   [Queries](./index-properties-queries.md "Named queries that can be executed against this index") – `https://ns.adobe.com/helix/shared/index#/properties/queries`
 
-*   [Query](./index-properties-queries-query.md "A named query that can be run against an index") – `https://ns.adobe.com/helix/shared/query#/properties/queries/additionalProperties`
-
-*   [Redirect Rule](./proxystrain-properties-redirects-redirect-rule.md "A strain is a combination of code and content that enables the creation of a digital experience") – `https://ns.adobe.com/helix/shared/redirectrule#/properties/redirects/items`
-
-*   [Row](./sheet-properties-data-row.md "A JSON representation of an Excel or Google Sheets data row") – `https://ns.adobe.com/helix/data-embed/row#/properties/data/items`
-
-*   [Runtime Strain](./strains-definitions-anystrain-oneof-runtime-strain.md "A runtime strain is a combination of code and content that enables the creation of a digital experience") – `https://ns.adobe.com/helix/shared/runtimestrain#/definitions/anystrain/oneOf/1`
-
 *   [Single-Value Property](./property-oneof-single-value-property.md "The property in an index") – `https://ns.adobe.com/helix/shared/property#/oneOf/0`
 
 *   [Single-Value Property](./property-oneof-single-value-property.md "The property in an index") – `https://ns.adobe.com/helix/shared/property#/oneOf/0`
 
+*   [Sitemap Languages](./sitemap-properties-sitemap-languages.md "The languages to add to the sitemap") – `https://ns.adobe.com/helix/shared/sitemap#/properties/languages`
+
 *   [Untitled object in Configuration](./config-properties-definitions.md "A container for referencable objects that can be re-used elsewhere in the configuration") – `https://ns.adobe.com/helix/shared/config#/properties/definitions`
 
 *   [Untitled object in Configuration](./markupconfig-properties-definitions.md "A container for referencable objects that can be re-used elsewhere in the configuration") – `https://ns.adobe.com/helix/shared/markupconfig#/properties/definitions`
@@ -122,15 +100,11 @@
 
 *   [Vanity URL extractor](./redirects-properties-vanity-url-extractor.md "Configure one or multiple vanity URL extractors, each extractor is a property of this object") – `https://ns.adobe.com/helix/shared/redirects#/properties/vanity`
 
-*   [Vanity URL mapping](./redirects-properties-vanity-url-extractor-vanity-url-mapping.md) – `https://ns.adobe.com/helix/shared/vanity#/properties/vanity/additionalProperties`
-
-*   [Version Lock](./proxystrain-properties-version-lock.md) – `https://ns.adobe.com/helix/shared/version-lock#/properties/version-lock`
-
-*   [Workbook](./data-embed-response-allof-0-oneof-workbook.md "A JSON representation of an Excel or Google Sheets workbook containing multiple sheets") – `https://ns.adobe.com/helix/data-embed/workbook#/allOf/0/oneOf/1`
+### Arrays
 
-*   [Worksheet](./data-embed-response-allof-0-oneof-worksheet.md "A JSON representation of an Excel or Google Sheets worksheet containing rows and columns") – `https://ns.adobe.com/helix/data-embed/sheet#/allOf/0/oneOf/0`
+*   [Exclude](./index-properties-exclude.md "Glob patterns for paths where this index must not be used") – `https://ns.adobe.com/helix/shared/index#/properties/exclude`
 
-### Arrays
+*   [Include](./index-properties-include.md "Glob patterns for paths where this index is used") – `https://ns.adobe.com/helix/shared/index#/properties/include`
 
 *   [Untitled array in Conditions](./conditions-properties-and.md "All conditions in this list must be met") – `https://ns.adobe.com/helix/shared/conditions#/properties/and`
 

diff --git a/docs/conditions-properties.md b/docs/conditions-properties.md
diff --git a/docs/config-properties.md b/docs/config-properties.md
diff --git a/docs/fstab-properties.md b/docs/fstab-properties.md
diff --git a/docs/giturl-properties.md b/docs/giturl-properties.md
diff --git a/docs/language-properties-path.md → docs/index-properties-exclude-items.md b/docs/language-properties-path.md → docs/index-properties-exclude-items.md
@@ -1,15 +1,15 @@
-# Path Schema
+# Untitled string in Index Schema
 
 ```txt
-https://ns.adobe.com/helix/shared/language#/properties/destination
+https://ns.adobe.com/helix/shared/index#/properties/exclude/items
 ```
 
-The destination contentbus path to store sitemap to.
 
-| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                           |
-| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------- |
-| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [language.schema.json*](language.schema.json "open original schema") |
 
-## destination Type
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                     |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------- |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [index.schema.json*](index.schema.json "open original schema") |
 
-`string` ([Path](language-properties-path.md))
+## items Type
+
+`string`
diff --git a/docs/language-properties-href-language.md → docs/index-properties-exclude.md b/docs/language-properties-href-language.md → docs/index-properties-exclude.md
@@ -1,15 +1,22 @@
-# HREF Language Schema
+# Exclude Schema
 
 ```txt
-https://ns.adobe.com/helix/shared/language#/properties/hreflang
+https://ns.adobe.com/helix/shared/index#/properties/exclude
 ```
 
-The HREF language to use when listing alternates.
+Glob patterns for paths where this index must not be used.
 
-| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                           |
-| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------- |
-| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [language.schema.json*](language.schema.json "open original schema") |
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                     |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------- |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [index.schema.json*](index.schema.json "open original schema") |
 
-## hreflang Type
+## exclude Type
 
-`string` ([HREF Language](language-properties-href-language.md))
+`string[]`
+
+## exclude Examples
+
+```yaml
+/drafts/**
+
+```