LoLin NodeMCU V3 ESP8266 requires header patching

[abishekmuthian]

Greetings developers & contributors.

Header patching was removed(598c556) as it was perceived to be useless for ESP32 boards but ESP8266 boards like LoLin NodeMCU V3 uses 26MHz crystal(Including latest ones) and so without header patching the flashing results in reset loop when using esp-web-tools.

Allowing user to set the right flashing mode is perhaps the correct way to go regarding this, But for that to happen esp-web-flasher needs to accept the parameters and execute the patching as it was in version 4.0.0 and prior. Then esp-web-tools could possibly call esp-web-flasher with the correct flashing mode(esphome/esp-web-tools#186) from user input.

Considering ESP8266 continues to be most accessible and affordable development board and its likely to remain that way for next few years due to semiconductor shortage & shipping issues; I think accepting flashing mode from user would be a valuable addition.

[abishekmuthian]

This is how ESPTool is currently detecting the crystal frequency (https://github.com/espressif/esptool/blob/ded13e025628d4c5ca350378b391e9b0d38bcb57/esptool.py#L1223),

    def get_crystal_freq(self):
        # Figure out the crystal frequency from the UART clock divider
        # Returns a normalized value in integer MHz (40 or 26 are the only supported values)
        #
        # The logic here is:
        # - We know that our baud rate and the ESP UART baud rate are roughly the same, or we couldn't communicate
        # - We can read the UART clock divider register to know how the ESP derives this from the APB bus frequency
        # - Multiplying these two together gives us the bus frequency which is either the crystal frequency (ESP32)
        #   or double the crystal frequency (ESP8266). See the self.XTAL_CLK_DIVIDER parameter for this factor.
        uart_div = self.read_reg(self.UART_CLKDIV_REG) & self.UART_CLKDIV_MASK
        est_xtal = (self._port.baudrate * uart_div) / 1e6 / self.XTAL_CLK_DIVIDER
        norm_xtal = 40 if est_xtal > 33 else 26
        if abs(norm_xtal - est_xtal) > 1:
            print("WARNING: Detected crystal freq %.2fMHz is quite different to normalized freq %dMHz. Unsupported crystal in use?" % (est_xtal, norm_xtal))
        return norm_xtal

[balloob]

I don't have enough knowledge to understand how crystal speed comes into play with flashing.

[abishekmuthian]

Okay I will try to study that and clarify what's really happening with the crystal frequency and the flashing modes.

But regardless of the deeper relationship, isn't getting the flashing mode as a parameter from esp-web-tools and then passing to the function which were existing earlier a straightforward fix?

[balloob]

We don't want to ask the user about the crystal speed of their device, it should be automatically detected.

ESP Web Tools and ESP Web Flasher shouldn't do any header patching anymore. The developer putting up the website with ESP Web Tools should create the correct firmware versions. ESP Web Tools does the selecting based on data provided by ESP Web Flasher.

It could probably become a new property for the builds specified in manifest.json such that ESP Web Tools can detect crystal speed and select a different build if a 26Mhz build is available.

[abishekmuthian]

I respectfully disagree, I think the 'Flashing mode' should be handled by the 'flashing tool' and not the 'firmware compilation' tool and hence both ESPTool & Adafruit's ESP tool does the patching.

The developer putting up the website with ESP Web Tools should create the correct firmware versions.

I don't think that's even possible, If so esp-sdks which compiles the firmware would have done that and wouldn't be handled during flashing like I mentioned above.

I think this was not brought before in earlier discussions on this subject(#83, esphome/esp-web-tools#151) because the users were using ESP32 and the current default flashing mode works for them i.e. No one is supplying different firmware versions based on the crystal now.

[balloob]

Maybe I misunderstand, but this is not about 'flashing mode' but about flashing the correct bootloader?

ESP Web Flasher has no knowledge if something is a bootloader. It just flashes what it is told to flash. It's up to the tool using ESP Web Flasher to provide the right files (in your case ESP Web Tools).

Now ESP Web Tools doesn't generate/patch any files either but instead relies on the manifest file to point at the right data for each ESP variation.

[abishekmuthian]

The misunderstanding is because unlike ESP32, the bootloader is not flashed separately(it's not a multi-stage) for ESP8266(It's a single firmware.bin as in the manifest for ESP Web Tools) and header patching 4th byte to 0x20 was done to set the 'Flashing Mode' to DIO and 'Flashing Speed' to 40 Mhz(40m).

More information about the flashing modes can be found here and here.

During the earlier discussion(#83) @TD-er raised concern that removing header patching could affect ESP82666 boards but wrongly concluded that all ESP boards after 2017 doesn't use this flashing mode as they don't contain 26 MHz crystal; That might be the case with ESP32 but not with ESP8266.

My board uses the latest revision of ESP8266 i.e. ESP8266EX so it's likely that removal of header patching prevents all recent ESP8266 boards from being flashed by esp-web-flasher.

[TD-er]

I have not seen ESP8266 boards using a crystal other than 26 MHz (as far as I can remember), so building for 26 MHz crystal should work on (almost?) all boards out there.
This can be set in the IDE or PlatformIO when selecting the board. (or define your own config where you explicitly set the xtal freq. to 26 MHz)

I am not entirely sure whether the web flasher should patch it, but on the other hand esptool.py and the Espressif download tool patch it. (Espressif download tool has a flag to no change the bin to disable this)
Do you have an example of an ESP8266 board that does have a crystal other than 26 MHz?

[abishekmuthian]

This can be set in the IDE or PlatformIO when selecting the board. (or define your own config where you explicitly set the xtal freq. to 26 MHz)

Is this true for ESP8266 as well? I'm using micropython and only ESP32 seems to accept XTAL as a config option (micropython/micropython@f431795),

I am not entirely sure whether the web flasher should patch it, but on the other hand esptool.py and the Espressif download tool patch it. (Espressif download tool has a flag to no change the bin to disable this)

My reasoning is that a developer doesn't have access to all available boards for a particular microcontroller and so the flashing tool takes care of the messy bits. Espressif knows if there's a new board with X frequency and so they just update their tool.

Now esp-web-flasher being a flashing tool as well, In theory one might expect it behave similar to Espressif's tools.

Do you have an example of an ESP8266 board that does have a crystal other than 26 MHz?

No my boards are 26MHz and hence I ran into the reset loop as there's no header patching in latest version and had to use v4.0.0.

[TD-er]

Now esp-web-flasher being a flashing tool as well, In theory one might expect it behave similar to Espressif's tools.

Sure, but my point is that if there is only 1 crystal frequency for all ESP8266 boards, then the bin files should be made for 26 MHz and thus no patching is needed.

If there are multiple crystal frequencies, then the flash tool should at least be able to detect the frequency and either pick the right one, or patch it, or fail to flash with a proper error message.

[abishekmuthian]

That's a reasonable assessment from your position.

but my point is that if there is only 1 crystal frequency for all ESP8266 boards

But are we 100% certain that this is the case?
Espressif's documents, specs, tools suggests that there can be ESP82666 boards with cyrstal frequencies other than 26MHz. Shouldn't we adhere to the specs, rather personal experience?

Besides projects like micropython depend upon the flashing tool for patching and doesn't expect the average developer to build the firmware(let alone setting the correct crystal frequency, which I haven't found yet). So web flashing using esp-web-flasher is out of the scope for such developers and projects now.

I have found at least one project using esp-web-tools(6.1.1) in the wild to flash official micropython build to ESP8266, Now they have to resort to building custom micropython firmware with hard coded crystal frequency if that's possible.

Removing header patching has made the barrier for entry much higher for using esp-web-tools.

[balloob]

Your conclusion is incorrect. Header patching was removed because it was breaking installs and causing a high barrier. Instead of developers relying on magic in ESP Web Flasher, we prefer developers create the right images and we can help with the selection process.

[abishekmuthian]

I didn't mean the issue which caused ESP32 devices from being flashed correctly shouldn't be fixed, I meant the solution for fixing it has increased barrier for entry i.e. developers should know how to build custom firmware to use esp-web-tools.

[balloob]

ESP Web Tools is meant for firmware creators to help users install and update devices. Creating custom builds is what firmware creators do? I guess MicroPython is the only exception here as it's a generic platform.

[abishekmuthian]

Disclaimer: I myself use esp-web-tools to flash my custom micropython firmware to my customers device.

That said, I see scenarios where people would like to flash a generic firmware to their customer devices to offer other services like education(IDE), configuration(IoT) etc.

I guess MicroPython is the only exception here as it's a generic platform.

There are at least about dozen such generic firmwares.

ESP Web Tools is meant for firmware creators to help users install and update devices.

No contention there, It's your choice as a project maintainer.

[Jason2866]

I do not understand what the problem is. Since the header patching was removed we had not a single user of Tasmota complaining that the flashed firmware is NOT working.
We us the webflasher for flashing all Tasmota variants (esp8266, esp32, esp32-s2 and esp32-c3). @balloob any news for supporting the ESP32-S3? Official Arduino support will be the next days ;-)

[abishekmuthian]

We us the webflasher for flashing all Tasmota variants (esp8266, esp32, esp32-s2 and esp32-c3)

@Jason2866 Can you check the crystal frequency of the "esp8266 Tasmota variant" you are using with the latest version of this tool?

[Jason2866]

Which tool?

[abishekmuthian]

Sorry, I'll rephrase; What's the crystal frequency of the esp8266 board you're able to flash successfully after the header patching was removed in esp-web-flasher.

[Jason2866]

26MHz

Just flash Tasmota and it will work https://arendst.github.io/Tasmota-firmware/

[Jason2866]

If it does not work for you have either a very! rare board with 40Mhz and builded the firmware without setting the crystal speed to 40Mhz. It is no problem to flash this firmware with webflasher too. Just do a extra manifest and provide the according firmware.

[abishekmuthian]

Just flash Tasmota and it will work https://arendst.github.io/Tasmota-firmware/

Flashing Tasmota firmware through that website on the board in question(LoLin NodeMCU Lua V3) works fine, The flashing mode (DIO) seems to have been set in the firmware itself during compilation(likely via PlatformIO) as @TD-er mentioned earlier.

Unfortunately PlatformIO doesn't support Micropython(I'm using esp-open-sdk to build mine) and I haven't figured out a way for setting the flashing mode during firmware compilation if that's possible. So for those in my position, Using esp-web-flasher v4.0.0 is the temporary solution.

[Jason2866]

Thats the point. Your platform generating the firmware has to generate a valid firmware for flashing. Do not expect other tools fixes issues caused by a tool generating a faulty firmware.
So in your build process you have to write a tool fixing your wrong flash settings in the firmware.

[abishekmuthian]

Do not expect other tools fixes issue caused by a faulty tool generating a firmware.

That's harsh and inaccurate.

I am not entirely sure whether the web flasher should patch it, but on the other hand esptool.py and the Espressif download tool patch it. (Espressif download tool has a flag to no change the bin to disable this).

[Jason2866]

At least your title of this opened issue is not correct. Patching is NOT required.

[balloob]

Going to close this as it's not a ESP Web Flasher issue and should be fixed when firmware created.