XMLStreamReader exceeds Quota maxElement limit of 5000

[paramdas11]

We are using XMLStreamReader to read a large XML rating file imported from Bazaarvoice.
But the Job is failing due to com.demandware.core.quota.QuotaLimitExceededException:
Limit for quota 'api.dw.io.XMLStreamReader.getXMLObject().maxElements' exceeded. Limit is 5000, actual is 5001

Is there any alternative approach to read the file ?

[adam-clason]

It seems you are not using the streaming capabilities of the XMLStreamReader class. I suspect you are reading almost the entire xml file with that getXMLObject() function and stuffing it into a variable. You will need to provide more code for context, but you want to loop over the individual nodes using the next() function, and only use the getXMLObject() function to pull small bits into variables as needed.

[pravngaur]

Hi Param, I believe you would be getting this error as soon as the XMLSTREAMREADER object is instantiated, it would be resulting in error. In that case, you have to cut short the xml file so that it stays within the permissible limit of nodes. Best way to do it is at the etl layer if you have any. Else you would need to see what suits best in ur architecture. But advise is not to do file churning in sfcc.

…

On Wed, Apr 24, 2019, 10:33 PM paramdas11 ***@***.***> wrote: We are using XMLStreamReader to read a large XML rating file imported from Bazaarvoice. But the Job is failing due to com.demandware.core.quota.QuotaLimitExceededException: Limit for quota 'api.dw.io.XMLStreamReader.getXMLObject().maxElements' exceeded. Limit is 5000, actual is 5001 Is there any alternative approach to read the file ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/orgs/SalesforceCommerceCloud/teams/community/discussions/30>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKECNZMQNSXCSRZMLSYYGX3PSCHFVANCNFSM4HIGEB2Q> .

[paramdas11]

Hi Praveen Thanks for you response. We are downloading the XML file from a SFTP location and then process it through a JOB controller and script files. Then we need to read the file to parse few data from XML elements. During data reading through XMLStreamReader we get the exception. The file is directly uploaded by our Review partner in the SFTP location, looks like can't split it. Thanks & Regards, Paramita Das Application Architect-eCommerce Phone: 91-33-400 97942 | Mobile: 91-9831032762 E-mail: [email protected] Chat: [email protected] [email protected] Find me on: "Life is not about waiting for the storms to pass, it's about learning how to dance in the rain" DLF IT Park, New Town Kolkata, WB 700156 India From: Praveen Gaur <[email protected]> To: SalesforceCommerceCloud/community <[email protected]> Cc: paramdas11 <[email protected]>, Author <[email protected]> Date: 04/25/2019 12:19 AM Subject: Re: [SalesforceCommerceCloud/community] XMLStreamReader exceeds Quota maxElement limit of 5000 (#30) Hi Param, I believe you would be getting this error as soon as the XMLSTREAMREADER object is instantiated, it would be resulting in error. In that case, you have to cut short the xml file so that it stays within the permissible limit of nodes. Best way to do it is at the etl layer if you have any. Else you would need to see what suits best in ur architecture. But advise is not to do file churning in sfcc.

On Wed, Apr 24, 2019, 10:33 PM paramdas11 ***@***.***> wrote: We are using XMLStreamReader to read a large XML rating file imported

from

Bazaarvoice. But the Job is failing due to com.demandware.core.quota.QuotaLimitExceededException: Limit for quota 'api.dw.io.XMLStreamReader.getXMLObject().maxElements' exceeded. Limit is 5000, actual is 5001 Is there any alternative approach to read the file ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <

https://github.com/orgs/SalesforceCommerceCloud/teams/community/discussions/30

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AKECNZMQNSXCSRZMLSYYGX3PSCHFVANCNFSM4HIGEB2Q

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[GeneralPatten]

@paramdas11 — based on the quota error message, I suspect that you are likely trying to call XMLStreamReader.getXMLObject() on the top level document element. This would load the whole file into memory (generally not a good practice), so the quota ensures that this does not occur.

Instead, read through the XML stream, with a condition to match on the START_ELEMENT with the local name equal to the individual elements you will be pulling data from. Then call getXMLObject() on that element, and process as needed. In your case, maybe the feed is something like:

<feed>
  <products>
    <product>
    ...
    </product>
  </products>
</feed>

Then your script would be similar to

let fr = new FileReader(file, "UTF-8");
let xsr = new XMLStreamReader(fr);

 while (xsr.hasNext()) {
   if (xsr.next() === XMLStreamConstants.START_ELEMENT) {
     if (xsr.getLocalName() === "product") {
       // read single "product" as XML
       let productXml = xsr.getXMLObject();

       // process productXml
     }
   }
 }
 xsr.close();
 fr.close();

If individual product elements have more than 5000 children, then you would probably have to use local name checks for each individual element and grab the attribute and/or text values.

See here for more info:
https://documentation.b2c.commercecloud.salesforce.com/DOC1/index.jsp?topic=%2Fcom.demandware.dochelp%2FDWAPI%2Fscriptapi%2Fhtml%2Fapi%2Fclass_dw_io_XMLStreamReader.html

[pravngaur]

I am assuming that you have validated and the script error outs at the object instantiation only. Again, in this case, file splitting has to happen anyhow. Should not be done in sfcc. If your partner is not willing to do it then do it thru etl. And if the script error outs at the point where instantiation is already done then have a look at the advised code snippet on this thread where you first check the nodes before accessing it.

…

On Thu, Apr 25, 2019, 1:42 PM paramdas11 ***@***.***> wrote: Hi Praveen Thanks for you response. We are downloading the XML file from a SFTP location and then process it through a JOB controller and script files. Then we need to read the file to parse few data from XML elements. During data reading through XMLStreamReader we get the exception. The file is directly uploaded by our Review partner in the SFTP location, looks like can't split it. Thanks & Regards, Paramita Das Application Architect-eCommerce Phone: 91-33-400 97942 | Mobile: 91-9831032762 E-mail: ***@***.*** Chat: ***@***.*** ***@***.*** Find me on: "Life is not about waiting for the storms to pass, it's about learning how to dance in the rain" DLF IT Park, New Town Kolkata, WB 700156 India From: Praveen Gaur ***@***.***> To: SalesforceCommerceCloud/community ***@***.***> Cc: paramdas11 ***@***.***>, Author ***@***.***> Date: 04/25/2019 12:19 AM Subject: Re: [SalesforceCommerceCloud/community] XMLStreamReader exceeds Quota maxElement limit of 5000 (#30) Hi Param, I believe you would be getting this error as soon as the XMLSTREAMREADER object is instantiated, it would be resulting in error. In that case, you have to cut short the xml file so that it stays within the permissible limit of nodes. Best way to do it is at the etl layer if you have any. Else you would need to see what suits best in ur architecture. But advise is not to do file churning in sfcc. On Wed, Apr 24, 2019, 10:33 PM paramdas11 ***@***.***> wrote: > We are using XMLStreamReader to read a large XML rating file imported from > Bazaarvoice. > But the Job is failing due to > com.demandware.core.quota.QuotaLimitExceededException: > Limit for quota 'api.dw.io.XMLStreamReader.getXMLObject().maxElements' > exceeded. Limit is 5000, actual is 5001 > > Is there any alternative approach to read the file ? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > < https://github.com/orgs/SalesforceCommerceCloud/teams/community/discussions/30 >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AKECNZMQNSXCSRZMLSYYGX3PSCHFVANCNFSM4HIGEB2Q > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/orgs/SalesforceCommerceCloud/teams/community/discussions/30/comments/3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKECNZM7ZAMCWUGUFLY2YQ3PSFRXTANCNFSM4HIGEB2Q> .

[GeneralPatten]

@pravngaur – it does not need to be split. It just needs to be read as an XML stream as opposed to being loaded into memory. Very standard, very efficient, fastest execution for any large files, regardless of platform.

How do you think large catalog XML files get imported into and exported from SFCC? This is exactly what is happening when running import/export on the instances. It’s whats happening when data is deployed to development and production instances.

[paramdas11]

Hi Todd, Many thanks for your response ! However this is the full feed file which we are trying to import and read for first time. There is the existing code already which parses the delta feed file efficiently. So here the main concern is not parsing an XML file rather a giant XML file, which is 648157 KB vs the delta feed file which is around 700 KB and already being parsed. Our Feed file format is as below - <xml> <feed> <Product id=1>.......... <OverallRatingRange>5</OverallRatingRange><RatingsOnlyReviewCount>0</RatingsOnlyReviewCount><TotalReviewCount>0</TotalReviewCount>........</Product> <Product id=2>.......... <OverallRatingRange>5</OverallRatingRange><RatingsOnlyReviewCount>0</RatingsOnlyReviewCount><TotalReviewCount>0</TotalReviewCount>........</Product> ................................... ............................... <Product id=5000>.......... <OverallRatingRange>5</OverallRatingRange><RatingsOnlyReviewCount>0</RatingsOnlyReviewCount><TotalReviewCount>0</TotalReviewCount>........</Product> </feed> We are already parsing the product_id and rating XML elements by below code- while(xmlReader.hasNext()) { if (xmlReader.next() == XMLStreamCons tants.START_ELEMENT && xmlReader.getLocalName() == 'Product') { productXML = xmlReader.readXMLObject(); id = ''; bvAverageRating = ''; bvReviewCount = ''; bvRatingRange = ''; var ns = productXML.namespace(); if(!empty( productXML.ns::ExternalId.toString())) { id = productXML.ns::ExternalId.toString(); if (!empty(productXML.ns::ReviewStatistics.ns::AverageOverallRat ing.toString())) { bvAverageRating = productXML.ns::ReviewStatistics.ns::AverageOverallRating.toString(); } if (!empty(productXML.ns::ReviewStatistics.ns::TotalReviewCount.toString())) { bvReviewCount = productXML.ns::ReviewStatistics.ns::TotalReviewCount.toString(); } if (!empty(productXML.ns::ReviewStatistics.ns::OverallRatingRang e.toString())) { bvRatingRange = productXML.ns::ReviewStatistics.ns::OverallRatingRange.toString(); } var product = ProductMgr.getProduct(BVHelper.decodeId(id)); if (product != null) { product.custom.bvAverageRating = bvAverageRating; product.custom.bvReviewCount = bvReviewCount; product.custom.bvRatingRange = bvRatingRange; productIds.push(product.ID); iProcessed++; } else { iFailed++; } } } } xmlReader.close(); fileReader.close(); Thanks & Regards, Paramita Das Application Architect-eCommerce Phone: 91-33-400 97942 | Mobile: 91-9831032762 E-mail: [email protected] Chat: [email protected] [email protected] Find me on: "Life is not about waiting for the storms to pass, it's about learning how to dance in the rain" DLF IT Park, New Town Kolkata, WB 700156 India From: Todd Patten <[email protected]> To: SalesforceCommerceCloud/community <[email protected]> Cc: paramdas11 <[email protected]>, Mention <[email protected]> Date: 04/25/2019 04:47 PM Subject: Re: [SalesforceCommerceCloud/community] XMLStreamReader exceeds Quota maxElement limit of 5000 (#30) @paramdas11 — based on the quota error message, I suspect that you are likely trying to call XMLStreamReader.getXMLObject() on the top level document element. This would load the whole file into memory (generally not a good practice), so the quota ensures that this does not occur. Instead, read through the XML stream, with a condition to match on the START_ELEMENT with the local name equal to the individual elements you will be pulling data from. Then call getXMLObject() on that element, and process as needed. In your case, maybe the feed is something like: <feed> <products> <product> ... </product> </products> </feed> Then your script would be similar to let fr = new FileReader(file, "UTF-8"); let xsr = new XMLStreamReader(fr); while (xsr.hasNext()) { if (xsr.next() === XMLStreamConstants.START_ELEMENT) { xmlStreamReader.getLocalName(); if (xsr.getLocalName() === "product") { // read single "product" as XML let productXml = xsr.getXMLObject(); // process productXml } } } xsr.close(); fr.close(); If individual product elements have more than 5000 children, then you would probably have to use local name checks for each individual element and grab the attribute and/or text values. See here for more info: https://documentation.b2c.commercecloud.salesforce.com/DOC1/index.jsp?topic=%2Fcom.demandware.dochelp%2FDWAPI%2Fscriptapi%2Fhtml%2Fapi%2Fclass_dw_io_XMLStreamReader.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pravngaur]

@todd dont get me wrong, please read through my response, i am still requesting to validate where exactly the script errors out, if at the time of reader object instantiation then only the split is suggested otherwise i have mentioned to follow the code snippets provided by other community members as for the same reason to check first for the child node type and then only load the chunk. Also, the deployment to prod and development is different that's replication, nothing has to do with xml parsing. @param please check on the todd's response.

…

On Thu, Apr 25, 2019, 6:19 PM Todd Patten ***@***.***> wrote: @pravngaur <https://github.com/pravngaur> – it does not need to be split. It just needs to be read as an XML stream as opposed to being loaded into memory. Very standard, very efficient, fastest execution for any large files, regardless of platform. How do you think large catalog XML files get imported into and exported from SFCC? This is exactly what is happening when running import/export on the instances. It’s whats happening when data is deployed to development and production instances. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/orgs/SalesforceCommerceCloud/teams/community/discussions/30/comments/6>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKECNZM6AOYIUZBMCFF2WCDPSGSD3ANCNFSM4HIGEB2Q> .

[GeneralPatten]

@paramdas11 – What do the ellipsis in the element represent? Is it possible that a single element may have 5000+ descendants within it? I'm very familiar with the bv import code. The only place this exception can be thrown is on the line with productXML = xmlReader.getXMLObject(). Which indicates that it's not a file size issue, but an issue with one or more elements having > 5000 descendants. In this case, you will need to stick with using the XMLStreamReader only to get the data. I would expect that the following psuedo code would work:

var fr = new FileReader(file, "UTF-8");
var xsr = new XMLStreamReader(fr);
var bvProduct;
var localName;

var localNames = [
  "ExternalId",
  "AverageOverallRating",
  "TotalReviewCount",
  "OverallRatingRange"
];

while (xsr.hasNext()) {
  if (xsr.next() === XMLStreamConstants.START_ELEMENT) {
    localName = xsr.getLocalName();
    if (localName === "product") {

      if (!empty(bvProduct)) {
        // Call function to update SFCC Product with bvProduct ExternalId, 
        // AverageOverallRating, TotalReviewCount, OverallRatingRange
      }
      bvProduct = {}; // initialize bvProduct to empty object
    }
    else if (~localNames.indexOf(localName)) {
      // if localName is found in localNames whitelist, store the value
      bvProduct[localName] = xsr.readElementText();
    }
  }
}

// Call function to update SFCC Product using last bvProduct object

xsr.close();
fr.close();