Unable to compile master branch, scons ModuleNotFoundError

[id3nom]

Godot version

4.0

System information

Debian 11

Issue description

scons report the following error:

scons: Reading SConscript files ...
ModuleNotFoundError: No module named 'editor.template_builders'; 'editor' is not a package:

Steps to reproduce

clone the master repository and try to compile with:
/usr/bin/env python3 /usr/local/bin/scons --jobs=8 platform=linuxbsd target=release_debug

Minimal reproduction project

No response

[Calinou]

I can't reproduce this on commit b6b81e8 (Fedora 34).

4.0

4.0 is a moving target. Please specify the exact Git commit hash you're using locally when reporting issues 🙂

Does this occur if you remove system-wide SCons and install SCons from pip using pip3 install --user --upgrade scons?

[soloparatolos]

I've been trying to compile master for a week but I'm getting the same error (arch/gnome). I'll try Calinou's proposed workaround and report back.

[id3nom]

I have tried the SCons from pip with the commit 7faf023 but I'm having the same error.

Here is the scons stack trace:

scons: Reading SConscript files ...
ModuleNotFoundError: No module named 'editor.template_builders'; 'editor' is not a package:
  File ".../godot/SConstruct", line 738:
    SConscript("modules/SCsub")
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File ".../godot/modules/SCsub", line 46:
    SConscript(name + "/SCsub")  # Built-in.
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File ".../godot/modules/gdscript/SCsub", line 25:
    SConscript("editor_templates/SCsub")
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.9/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File ".../godot/modules/gdscript/editor_templates/SCsub", line 5:
    import editor.template_builders as build_template_gd

[soloparatolos]

I tried the SCons from pip too; same error. Commit 4948296.

ModuleNotFoundError: No module named 'editor.template_builders'; 'editor' is not a package:
  File "/home/.../Documentos/git/godot/SConstruct", line 738:
    SConscript("modules/SCsub")
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/.../Documentos/git/godot/modules/SCsub", line 46:
    SConscript(name + "/SCsub")  # Built-in.
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/.../Documentos/git/godot/modules/gdscript/SCsub", line 25:
    SConscript("editor_templates/SCsub")
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 660:
    return method(*args, **kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/.../.local/lib/python3.10/site-packages/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/.../Documentos/git/godot/modules/gdscript/editor_templates/SCsub", line 5:
    import editor.template_builders as build_template_gd

[Calinou]

@soloparatolos Could you try to bisect the regression to determine when this started to occur? Thanks in advance 🙂

[akien-mga]

There's no need to bisect, the error is clear, as this code was added in #53957.

[jmb462]

Cannot reproduce on Ubuntu 21.10 (Commit 8227282)

[akien-mga]

I suspect that affected users might have a Python module named editor.py installed somewhere, e.g. https://pypi.org/project/python-editor/
That would conflict with the syntax used to import editor/templates_builders.py.

Edit: Yep, confirmed.

$ pip install python-editor
Defaulting to user installation because normal site-packages is not writeable
Collecting python-editor
  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Installing collected packages: python-editor
Successfully installed python-editor-1.0.4
$ gobuild_linux 
/usr/bin/scons LINKFLAGS=-fuse-ld=gold -j7 p=linuxbsd verbose=yes warnings=extra werror=yes tests=yes 2>&1 | tee -a build.log
scons: Reading SConscript files ...
Note: Building a debug binary (which will run slowly). Use `target=release_debug` to build an optimized release binary.
ModuleNotFoundError: No module named 'editor.template_builders'; 'editor' is not a package:
  File "/home/akien/Projects/godot/godot.git/SConstruct", line 738:
    SConscript("modules/SCsub"

[akien-mga]

So the part that needs to be changed in:

godot/modules/gdscript/editor_templates/SCsub

Line 5 in 8227282

import editor.template_builders as build_template_gd

(Same in Mono module)

Not sure what's the best way to do this, I remember it was a hassle when we had to workaround similar issues with conflicts on methods.py or detect.py. Not the greatest bit of design from Python IMO :)

Any suggestion @touilleMan? I guess we can do some sys.path wrangling, or move the file somewhere else, or duplicate it in GDScript and Mono modules as it was done initially. Or find the magic keyword that lets Python interpret a project-absolute path as project-absolute and not poking at pip packages.

[fabriceci]

That's the part of the PR where I bruteforced the thing, it would be a shame to have to duplicate the code and Moving the file, if I understand correctly, will produce the same issue if a module has the good idea to be called (core, misc, main, etc.). @jmb462 tries with spec_from_file_location()

[jmb462]

#import editor.template_builders as build_template_gd

import importlib.util
spec = importlib.util.spec_from_file_location("editor.template_builders", "../../../editor/template_builders.py")
build_template_gd = importlib.util.module_from_spec(spec)
spec.loader.exec_module(build_template_gd)

Replacing the import line with these 4 lines does the trick.
It compiles fine and generate template file even with the conflicting module installed.

I propose that I open a PR and maybe a Python guru could confirm that it's ok.

[touilleMan]

In a traditional Python project we should have all our code in a single folder (a "package" in Python vocabulary). This folder (and all it subfolders) would contain a __init__.py to indicate to Python they are packages.

This way we would be able to use relative import (e.g. from .. import editor.template_builders) to import the correct item.

However we don't have a single package here, but instead have .py files scattered around the codebase (no judgement, this is normal given Python is only used a convenient tool here ^^).

The bad thing is it's not possible to differentiate between "this editor.py has been manually added and shouldn't have been taken into account for the build" and "this platform_methods.py is totally part of the build !"

I see two solutions to solve this:

1. be explicit when importing helper modules
2. be explicit when declaring helper modules

@jmb462 PR #56690 does 1), however I'm not convinced about this approach given it requires to ban regular import and replace them with rather complex custom import code (and this code could not be factorized given we cannot import before having it defined 😄 )

Approach 2) on the other hand consist of ensuring that our helper modules are first in line when importing their name.
This can be done:

• Putting the helper modules in a site_scons folder (Scons look for this folder at startup and do the prepend to sys.path automatically)
• explicitly load the helper modules at the very start of SCsub

I think the second solution is better for our need (given we want to keep the helper code in their current folder for readability)

I've created PR #56698 which implement this solution ;-)