Print long flash string?

[nouser2013]

Hi, I'm using the class and it seems to work very nice. Only one question arises: How do I print/write a very long (8k) flash string? I have:

const char txt[]  PROGMEM = "..........."; // ~8k text, no \0s in it

AsyncClient *c;
AsyncPrinter *p; //Both are populated
// [...]
p->print("Hello World!"); // Works
p->write("Hello World!", strlen("Hello World!")); // Works
p->write(txt, strlen_P(txt)); // Does not compile
p->print(txt); // Compiles, but crashes the ESP

Thanks.

[me-no-dev]

where is this code located? is it in one of the callbacks? like onData maybe?

[nouser2013]

it is in the onData callback, yes.

[me-no-dev]

OK :) I have not yet gotten the time to put up some good explanations, but the Printer and SyncClinet classes are made to operate outside of the callbacks. You wrap the client in one of those and save their reference for access from the loop.
If you want to really do async send of large content, you need to use straight the TCP Client and attach to onAck.
Sending the first chunk onData, then onAck send the next one and so on until data is send.
The whole Async way of thinking needs some explanation I know :) Have put up some stuff for the WebServer already. Hope to get some time to do those as well.

[nouser2013]

hmmm, can I just give the AsyncPrinter *p as anotherPointer to the main loop and then anotherPointer->print(txt); from there?

[me-no-dev]

Something like this (take as pseudo code)

AsyncPrinter *printer = NULL;

void onConnect(AsyncClient *c){
  if(printer != NULL && printer->connected()){
    c->onDisconnect([](AsyncClient *c){ c->free(); delete c; });
    c->close();
  } else {
    if(printer != NULL){
      AsyncPrintr *p = printer;
      printer = NULL;
      delete p;
    }
    printer = new AsyncPrinter(c);
    printer->onData(dataCb);
    .....
  }
}

void loop(){
  if(printer != NULL && printer->connected()){
    printer.print(myLongLongString);
    ......
  } else if(printer != NULL){
    AsyncPrintr *p = printer;
    printer = NULL;
    delete p;
  }
}

[nouser2013]

alright, I tried something similar (as I said, had a different pointer to notify main loop) but it still crashes on anotherPointer->print(txt); // TXT=PGM_P type!, I had to do this:

  if (anotherPointer != NULL) {
    Serial.println(" DBG: Sending index.html");
    char reply[1450+1];
    char len[6];
    uint16_t sent=0;
    strcpy_P(reply, indexheader); // indexheader = PGM_P
    strcat_P(reply, PSTR("Content-Length:"));
    itoa(strlen_P(indexhtml), len, 10); // indexheader = PGM_P
    strcat(reply, len);
    strcat_P(reply, PSTR("\r\n\r\n"));
    anotherPointer ->print(reply);
    while (1) {
      uint16_t tmpsent;
      strncpy_P(reply, indexhtml+sent, sizeof(reply)-1); // indexheader = PGM_P
      reply[sizeof(reply)-1] = 0; //manually terminate
      tmpsent = anotherPointer ->print(reply);
      if (tmpsent==0) { delay(5); }
      sent+=tmpsent;
      Serial.printf(" DBG: s='%d'\n", sent);
      if (sent>=strlen_P(indexhtml)) break;
    }
    anotherPointer ->close();
    // ->free(); missing?
    anotherPointer = NULL;
  }

Also, when I do this for like 10 or 20 times, the ESP would crash. If I send smaller data only (~100b), it's still running after 30 minutes.

[me-no-dev]

can you give me some test code so I can look at it and make it crash?

[nouser2013]

I'll try. Unfortunately, it happens only sporadically, I could not isolate one single line of code responsible.

Also, I noticed that when AsyncPrint::close() is called, there is no cleanup code (such as free()) being called. Printer relies on the underlying AsyncClient to call a disconnect. But if the TCP remote site never closes the connection, something probably goes wrong.

Perhaps it also has to do with the interaction with the garbage collector not freeing memory as intended.

AsyncServer webServer(80);
AsyncPrinter webClient[8];  // 8 clients supported!
AsyncPrinter *mainSendIndexTo;

const char indexheader[] PROGMEM = "...............";
const char indexhtml[] PROGMEM = "...............";

void webServer_processData(void *arg1, AsyncPrinter *c, void *buf, uint16_t len);

void setup() {
  webServer.onClient([](void *obj, AsyncClient* c) {
    for (int i=0;i<8;i++) {
      // Find a free slot...
      if (webClient[i].connected()) {
        continue;
      } else {
        Serial.printf(" DGB: Web->slot %d.\n", i);
        webClient[i] = AsyncPrinter(c);
        webClient[i].onData(webServer_processData, 0);
        break;
      }
      Serial.println(" DGB: No free Webclient Slot!.");
      c->close();
    }
  }, 0); // register onConnect callback
  webServer.begin();

}
void loop() {
  // Handle Webserver
  if (mainSendIndexTo != NULL) {
    Serial.println(" DBG: Sending index.html");
    char reply[1450+1];
    char len[6];
    uint16_t sent=0;
    strcpy_P(reply, indexheader);
    strcat_P(reply, PSTR("Content-Length:"));
    itoa(strlen_P(indexhtml), len, 10);
    strcat(reply, len);
    strcat_P(reply, PSTR("\r\n\r\n"));
    mainSendIndexTo->print(reply);
    while (1) {
      uint16_t tmpsent;
      strncpy_P(reply, indexhtml+sent, sizeof(reply)-1);
      reply[sizeof(reply)-1] = 0;
      tmpsent = mainSendIndexTo->print(reply);
      sent+=tmpsent;
      if (sent>=strlen_P(indexhtml)) break;
    }
    mainSendIndexTo->close();
    mainSendIndexTo = NULL;
  }
}

// only GET supported
void webServer_processData(void *arg1, AsyncPrinter *c, void *buf, uint16_t len) {
  char _METHOD[4];
  char _URL[400];
  char *firstBlank;
  char *secondBlank;
  // Fish Method
  strncpy(_METHOD,(char*) buf, 3);
  _METHOD[3]=0;
  Serial.printf(" DBG: m='%s'\n", _METHOD);
  if (strcmp_P(_METHOD, PSTR("GET"))) {
    c->print("HTTP/1.0 501 Not Implemented\r\n\r\n");
    c->close();
    return;
  }
  // Fish URL
  firstBlank = strchr((char*)buf, ' ');
  secondBlank = strchr(firstBlank+1, ' ');
  strncpy(_URL, firstBlank+1, secondBlank-firstBlank-1);
  _URL[secondBlank-firstBlank-1] = 0;
  Serial.printf(" DBG: u='%s'\n", _URL);

  // Handle URL "/"
  if (!strcmp_P(_URL, PSTR("/")) || !strncmp_P(_URL, PSTR("/index.htm"), 10)) {
    mainSendIndexTo = c; // Notify loop() to send index.html on this AsyncPrinter
    return;
  }

  // "/a?cmd=......"
  if (!strncmp_P(_URL, PSTR("/a?cmd="), 7)) {
    char cmd[6];
    cmd[0] = 'l';
    strncpy(cmd+1, _URL+7, 4);
    cmd[5] = 0;
    cmdProcess(cmd, strlen(cmd));
    c->print("HTTP/1.0 200 OK\r\nServer:" PMS_DEVICE_ID "\r\nContent-Length:3\r\n\r\nOK.");
    c->close();
    return;
  }

  //OK NOT FOUND
  c->print("HTTP/1.0 404 Not Found\r\nServer:" PMS_DEVICE_ID "\r\n\r\n");
  c->close();
}

Is the teardown in loop() done correctly? it seems to work within the callback...

[me-no-dev]

first about closing the client that you will not handle at all, do this:

c->onDisconnect([](AsyncClient *c){ c->free(); delete c; });
c->close(true);

second, there are so many things about HTML and sending data that you will be much better off using the AsyncWebServer than trying to server requests yourself.
Since you can packetize the response you will be really happy with the result ;) And so will be any client that connects to that server.

[me-no-dev]

I'm thinking of a way similar to the Printer class that will let you handle web clients in the loop as well for certain cases. Need to come up with elegant solution though :)

[nouser2013]

first about closing the client that you will not handle at all, do this:

I don't understand. In the AsyncPrinter-Class, there is no onDisconnect(). But the Printer Class registers its client's onDisconnect and does stuff in this anonymous function (see here). I have no idea how the set up and teardown of the combination of AsyncClient and AsyncPrinter is supposed to work, but just calling AsyncPrinter::close() (and not doing anything with its AsyncClient) does not seem sufficient.

Wouldn't we need more cleanup in the AsyncPrinter::close() method, e.g.:

void AsyncPrinter::close(){
  if(_client != NULL) {
    _client->close(true);
    _client->free();
    _client = NULL;
  }
}

second, there are so many things about HTML and sending data that you will be much better off using the AsyncWebServer than trying to server requests yourself.

I'd politely disagree, since I need only basic GET requests and fish the host header, instanciating a whole webserver, either Async or the regular one, is way too much overkill. I realize that I expect the webbrowser to send the complete request (at least method, URL, and Host header field) in one single packet.

When requesting the /a?cmd= from my example above I'm done in 10ms overall time. Doing the same thing with AsyncWebserver is 100ms easy.

I of course tried with your AsyncServer and the regular Webserver. Same behaviour. As soon as I reply more than one packet back to the requesting webbrowser, the ESP would freeze its IP subsystem at some point or go into a WDT reset (and halt) completely.

[me-no-dev]

for the close, _client->close(true); is sufficient as it will call the _on_close method and clean the client
doing _client->close(); will run another milliseconds to close the client cleanly from outside the callback.
And for the other thing, maybe I should look into the server and try to figure out a faster way to parse the response. I can not imagine what am I doing the other 90ms (as most requests are single packets)
The WDT resets you explain I'm not sure I understand :) How is it freezing? What are those packets? If we are able to send megabyte files, there should be a reason behind this.

[nouser2013]

Alright, thanks, but note that there is not close(true), but only close(). Could it be that the poll() never gets called?

The WDT resets you explain I'm not sure I understand :) How is it freezing? What are those packets? If we are able to send megabyte files, there should be a reason behind this.

I'm capturing a complete session with Wireshark now. As soon as I have something, I'll post it.

Then, when enabling wireshark, I seem to have gotten issues with the delayed ACK again:

As you can see, Windows ACK's after 200ms, which is way too long. Is there a way to enable some "Windows" mode which sends two packets instead of one without waiting for an ACK? I cannot hock onAck(), since there is no ACK for the first packet. Windows would respond instantly that way...

It seems to be an issue only if windows expects a large content length. When cramming everything into one packet, all seems fine(see timestamps here):

And yes, I disabled nagle on the Client, however, this does not affect the webbrowser. If he has the Nagle enabled, he will wait for the second packet no matter what.

[me-no-dev]

there was some info on the web what to touch in the registry to disable this nonsense :)
It was one of the first results so give it a go.
There is now close(true) maybe you have not pulled since yesterday? The lib is pretty "dynamic" as I'm trying to get it stable (and comments like the one above helps as I already have ideas how to speed the parsing).
Based on my captures, I see lots of time spent on parsing headers and I can get that down to nothing using more C approach :) we will see if it will work.

[me-no-dev]

and by the way for web clients if you do your job and send proper response, the client will close the connection.

[nouser2013]

there was some info on the web what to touch in the registry to disable this nonsense :) It was one of the first results so give it a go.

This would work for me, but not the tons of ppl I'd ship the thing out to.

and by the way for web clients if you do your job and send proper response, the client will close the connection.

Yes. But should we rely on this final RST-flagged packet to arrive for internal cleanup? I'd argue that this final packet can get lost (although I've not seen this in Wireshark yet) and thus we'd have an undefined state in ESP.

I just had an IP freeze again, where the ESP would simply stop responding to IP requests and only loop() and IO PIN callbacks would work. Serial also works RX and TX. No error output on Serial. I have no idea what is wrong...

[me-no-dev]

there is a 2 second timeout :) do not worry if something goes wrong there are many things implemented to take care of cleanup. I don't have a single sketch with leak. Sure there are some edge cases, but that is why there are the timeouts :)
As for your network... I have never ever seen that happen. If something goes wrong with it, the ESP will go crazy in all cases. Could something else be doing it? Changing network? I'm really clueless...

[nouser2013]

hmmm, I tried again to isolate the issue. Didn't work. One time it ran for 30 minutes and no error, then reboot and it froze after 30 seconds. Reboot, no error for 10 minutes, reboot, no error for 5 minutes, then complete crash. It may be hardware related.

Short of sending you the complete source I'm out of options. Perhaps just compile it and put it on one of your devices and see if you can crash it as well...

[me-no-dev]

sure thing :) I have a couple sitting anyway

[nouser2013]

this may sound like a very stupid question, but how do I send you a message here on github? Wouldn't want to post everything publicly....

[nouser2013]

HEUREKA. I think I've tracked down the issue. I believe the error is in SPIFFS implementation. With heavy FS reading, there seem to be issues with SPI arbitration: on one hand, the MCU tries to get new instructions, on the other hand, my user code tries to read new file data (and I'm talking ~200 bytes every 10ms).

I got this by just uncommenting the reading of SPIFFS files (while still having the lib and init code in there) and bombarding the device with http requests. It's still alive after 30 minutes.

Unfortunately, not even a clue how to fix it. But will issue this to the official github repo...

THANKS for all your help and support.

PS: as soon as I started reading from FS again while bombarding HTTP requests, dead after 1 minute and 32 seconds ^^

[me-no-dev]

that really sucks :( I hope you get this resolved for all of us. I have a more basic SPIFFS implementation somewhere here where you can access it through C and not jump from object to object, but I'm not sure how/if that would help... it's the implementation that Espressif shared some time ago

[diraniyoussef]

The guidelines here can be useful especially the memory.