SD size() returns random value until file has non-zero bytes of content #8625

frankcohen · 2023-09-12T00:43:39Z

Board

ESP32-S3 Dev Module

Device Description

I use my Reflections board, it is an open-source Arduino compatible ESP32-S3 board built from the ESP32-S3 Dev Module. Details at https://github.com/frankcohen/ReflectionsOS

Hardware Configuration

// SPI Bus
#define SPI_MOSI 35
#define SPI_MISO 37
#define SPI_SCK 36

// NAND Storage
#define NAND_SPI_CS 15

Version

latest master (checkout manually)

IDE Name

Arduino IDE 2.1.0 nightly

Operating System

MacOS 13.5.1

Flash frequency

80 Mhz

PSRAM enabled

yes

Upload speed

921600

Description

I'm using an external SD/NAND with an ESP32-S3 to create and save log values to a file. NAND is a surface mount version of an SD card. I connect the NAND over SPI using pin 15 for the CS. This is part of my Reflections open-source project https://github.com/frankcohen/ReflectionsOS.

File mylog = SD.open( "/log1", FILE_WRITE );
if( !mylog ){
  Serial.print( F( "Unable to create log file " ) );
}
Serial.print( "mylog.size() = " );
Serial.println( (long) mylog.size() );

I see random size values in the Serial monitor. For example mylog.size = 1070537564. When I write the the file first, then the size values return correctly.

File mylog = SD.open( "/log1", FILE_WRITE );
if( !mylog ){
  Serial.print( F( "Unable to create log file " ) );
}
int logval2 = mylog.print( "Starting log" );
Serial.print( "mylog.size() = " );
Serial.println( (long) mylog.size() );

size returns 14, which is correct.

-Frank

Sketch

/*
 Reflections, mobile connected entertainment device

 Repository is at https://github.com/frankcohen/ReflectionsOS
 Includes board wiring directions, server side components, examples, support

 Licensed under GPL v3 Open Source Software
 (c) Frank Cohen, All rights reserved. [email protected]
 Read the license in the license.txt file that comes with this code.

 Depends on:
 https://github.com/plageoj/urlencode

Example log entry:
CALLIOPE-7A,Logger started correctly
then the node.js function adds the date
20230808-11:45AM,CALLIOPE-7A,Logger started correctly

Implemented the log receiver service in node.js using:
// GET to make a log entry
var timestamp = require('log-timestamp');
router.get( "/logit", (req, res) => {
  console.log( req.query.message );
  res.send( "<html><body>Logged</body></html>" );
}); 

requires timestamp module, install using:
npm install log-timestamp

Example,
https://cloudcity.starlingwatch.com/api/logit?message=thisismyfirstlogentry5

*/

#include "LOGGER.h"

extern const char* root_ca;   // Defined in secrets.h

LOGGER::LOGGER(){}

void LOGGER::info( String message )
{
  String lm = "Info,";
  lm += message;
  logit( lm );
}

void LOGGER::warning( String message )
{
  String lm = "Warning,";
  lm += message;
  logit( lm );
}

void LOGGER::error( String message )
{
  String lm = "Error,";
  lm += message;
  logit( lm );
}

void LOGGER::critical( String message)
{
  String lm = "Critical,";
  lm += message;
  logit( lm );
}

/*
* Adds host name and sends to active file
*/

void LOGGER::logit( String msg )
{
  if ( mylogopen != 1 )
  {
    Serial.println( "Not logging, mylog is not open" );
    return;    
  }
 
  String logmsg = devname.c_str();
  logmsg += ",";
  logmsg += msg;
  logmsg += "\n";

  long mysize = (long) mylog.size();

  int logval = mylog.print( logmsg );
  mylog.flush();

  Serial.print( "Logged to " );
  Serial.print( mylogname );
  Serial.print( ", written = ");
  Serial.print( logval );
  Serial.print( ", characters " );
  Serial.print( logmsg.length() );
  Serial.print( ", file size before " );
  Serial.print( mysize );
  Serial.print( ", file size after " );
  Serial.print( mylog.size() );
  if ( mylog.size() == mysize + logmsg.length() )
  {
    Serial.println( ", OK" );
  }
  else
  {
    Serial.println( ", FAIL" );
  }
  
  if ( mylog.size() < log_size_upload ) return;

  if ( ! scanLogNumbers() ) return;

  mylog.close();
  mylogopen = 0;

  String elname = logname_start;
  elname += String( highLogNumber + 1 );
  
  lognext = SD.open( elname, FILE_WRITE );
  if( !lognext )
  {
    Serial.print( F( "Unable to create log file " ) );
    Serial.println( elname );
  }    
  
  mylog = lognext;
  mylogname = elname;
  mylogopen = 1;
  
  Serial.print( F( "Opened log file " ) );
  Serial.print( mylogname );
  Serial.print( ", mylog.size() = " );
  Serial.println( (long) mylog.size() );

  // I need to write something to the log file here
  // because the Espressif SD library returns a random
  // value for mylog.size() before something is
  // written to the log file.

  int logval2 = mylog.print( "Starting log " );
  logval2 += mylog.print( mylogname );
  logval2 += mylog.print( "\n" );
  mylog.flush();
}

/*
* Sends message to the server over HTTPS GET protocol
*/

bool LOGGER::sendToServer( String message )
{
  Serial.print( "sendToServer: ");
  Serial.println( message );

  String logurl = cloudCityLogURL;
  logurl += urlEncode( message );

  http.begin( logurl, root_ca );
  http.setReuse(true);

  int httpCode = http.GET();

  if( httpCode != HTTP_CODE_OK )
  {
    Serial.print( F( "Server response code: " ) );
    Serial.println( httpCode );
    return false;
  }

  http.end();

  return true;
}

bool LOGGER::scanLogNumbers()
{
  lowLogNumber = 1000000;
  highLogNumber = 0;

  File root = SD.open( "/" );
  if( !root ){
      Serial.println( F( "scanLogNumbers failed to open directory /" ) );
      return false;
  }

  while ( 1 )
  {
    File file = root.openNextFile();
    if ( file )
    {
      if ( ! file.isDirectory() )
      {
        String myfilename = file.name();

        if ( myfilename.startsWith( "log" ) )
        {
          String mynum = myfilename.substring( 3 );
          int mynumbr = mynum.toInt();

          if ( mynumbr < lowLogNumber )
          {
            lowLogNumber = mynumbr;
          }

          if ( mynumbr > highLogNumber )
          {
            highLogNumber = mynumbr;
          }
        }
      }
    }
    else
    {
      break;
    }
  }

  return true;
}

/*
* Begin picks-up the state from where we left off (from the file system)
* or creates the initial state (and file system logs)
*/

void LOGGER::begin()
{
  Serial.println( "Logger begin");

  root_ca = ssl_cert;   // Defined in secrets.h

  mylogopen = 0;
  myuploadopen = 0;
  uploading = 0;
  uploadcount = 0;

  devname = host_name_me;
  std::string mac = WiFi.macAddress().c_str();
  devname.append( mac.substr( 15, 2 ) );
  devicename = devname.c_str();

  // Open a new log file

  if ( ! scanLogNumbers() ) return;

  mylogname = logname_start;
  mylogname += String( highLogNumber + 1 );
  
  mylog = SD.open( mylogname, FILE_WRITE );
  if( !mylog )
  {
    Serial.print( F( "Unable to create log file " ) );
    Serial.println( mylog );
  }    
  
  mylogopen = 1;
  
  Serial.print( F( "Opened log file " ) );
  Serial.println( mylogname );

  uploadchecktime = millis();

  Serial.println( "Logger begin done");
}

void LOGGER::loop()
{

  if ( ( ( millis() - uploadchecktime ) > 10000 ) )
  {
    uploadchecktime = millis();
    
    Serial.println( "Checking for upload" );

    if ( !uploading )
    {

      if ( scanLogNumbers() )
      {
        uploadfilename = logname_start;
        uploadfilename += String( lowLogNumber );

        if ( ( mylogopen ) && ( uploadfilename.equals( mylogname ) ) )
        {
          Serial.print( F( "Skipping log file because it is open " ) );
          Serial.println( uploadfilename );
        }
        else
        {
          myupload = SD.open( uploadfilename );
          if( !myupload )
          {
            Serial.print( F( "Unable to open upload file " ) );
            Serial.println( uploadfilename );
            myuploadopen = 1;
            uploadcount = 0;
            uploading = 0;
          }
          else
          {
            uploading = 1;
            myuploadopen = 1;
            uploadcount = 0;
            
            Serial.print( F( "Opened upload file " ) );
            Serial.println( uploadfilename );
          }
        }
      }
    }
    else
    {
      Serial.println( F( "We're uploading, so I'm fine." ) );
    }
  }

  if ( uploading )
  {
    long avail = myupload.available();

    if ( avail > 0 )
    {
      char line[250];
      char fchr;

      int i=0;
      while ( i < 249 )
      {
        fchr = myupload.read();
        if ( fchr == '\n' ) break;
        line[i] = fchr;
        i++;
      }
      line[ i ] = 0;

      String mys = line;

      sendToServer( mys );
      uploadcount++;
    }
    else
    {
      Serial.print( "Uploading complete, log sent to service " );
      Serial.println( uploadcount );

      myupload.close();
      myuploadopen = 0;
      SD.remove( uploadfilename );
      uploading = 0;
    }
  }
}

Debug Message

Opened log file, mylog.size()=1828101982

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

I confirm I have checked existing issues, online documentation and Troubleshooting guide.

The text was updated successfully, but these errors were encountered:

me-no-dev · 2023-09-12T08:21:54Z

@P-R-O-C-H-Y PTAL

everslick · 2023-09-13T13:25:31Z

Probably related: #7759

Parsaabasi · 2025-01-16T12:51:11Z

Hello,

Due to the overwhelming volume of issues currently being addressed, we have decided to close the previously received tickets. If you still require assistance or if the issue persists, please don't hesitate to reopen the ticket.

Thanks.

frankcohen added the Status: Awaiting triage Issue is waiting for triage label Sep 12, 2023

me-no-dev assigned P-R-O-C-H-Y Sep 12, 2023

me-no-dev added Area: Libraries Issue is related to Library support. Status: Needs investigation We need to do some research before taking next steps on this issue and removed Status: Awaiting triage Issue is waiting for triage labels Sep 12, 2023

Parsaabasi removed the Status: Needs investigation We need to do some research before taking next steps on this issue label Jan 16, 2025

Parsaabasi closed this as completed Jan 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SD size() returns random value until file has non-zero bytes of content #8625

SD size() returns random value until file has non-zero bytes of content #8625

frankcohen commented Sep 12, 2023 •

edited by me-no-dev

Loading

me-no-dev commented Sep 12, 2023

everslick commented Sep 13, 2023

Parsaabasi commented Jan 16, 2025

SD size() returns random value until file has non-zero bytes of content #8625

SD size() returns random value until file has non-zero bytes of content #8625

Comments

frankcohen commented Sep 12, 2023 • edited by me-no-dev Loading

Board

Device Description

Hardware Configuration

Version

IDE Name

Operating System

Flash frequency

PSRAM enabled

Upload speed

Description

Sketch

Debug Message

Other Steps to Reproduce

I have checked existing issues, online documentation and the Troubleshooting Guide

me-no-dev commented Sep 12, 2023

everslick commented Sep 13, 2023

Parsaabasi commented Jan 16, 2025

frankcohen commented Sep 12, 2023 •

edited by me-no-dev

Loading