Python3 import issues

[cromefire]

What language does this apply to?

python (3)

Describe the problem you are trying to solve.

See #1491

Structure

proto/ <-- git module
    test1.proto
    test2.proto
src/ <-- in PYTHONPATH
    proto_test/
        proto/
            test1_pb2.py
            test2_pb2.py
test_it.py

Files

test1.proto:

syntax = "proto3";

test2.proto:

syntax = "proto3";
import public "test1.proto"

test_it.py:

import "proto_test.proto.test1_pb2"

Run

CMD:
protoc "--proto_path=$(pwd)/proto" "--python_out=$(pwd)/src/proto_test/proto"

Run the python file:

export PYTHONPATH="$(pwd)/src"
python3 test_it.py

Result

ModuleNotFoundError: No module named 'student_pb2'

Describe the solution you'd like

There are 4 proposed solutions that do not prevent parallelism:

1. Just output them relatively:

   import public "test1.proto";
   

   gets:

   from .test1_pb2 import *

   import public "../test1.proto";
   

   gets:

   from ..test1_pb2 import *

   There may be some other cases that need more lookups on how these imports work
2. Use import by file path:
   https://stackoverflow.com/a/67692

   import public "test1.proto";
   

   gets:

   import importlib.util
   from os.path import join, dirname
   test1__pb2_spec = importlib.util.spec_from_file_location("test1_pb2", join(dirname(__file__), "test1_pb2.py"))
   test1__pb2 = importlib.util.module_from_spec(test1__pb2_spec)
   test1__pb2_spec.loader.exec_module(test1__pb2)

   (could live in a helper function)
3. Implement python_package similar to java_package
4. Do python_package as a CLI flag

Additionally I think for method 1 and 2 it should be optionally triggered by a commandline flag, like --python3_relative

Describe alternatives you've considered

You could do:

proto/ <-- git module
    proto_test/
        proto/
            test1.proto
            test2.proto
src/ <-- in PYTHONPATH
    proto_test/
        proto/
            test1_pb2.py
            test2_pb2.py
test_it.py

Which would break other things, if you build for multiple languages (which is a core principle of protobuf)

Additional context

I can create a github repo with an example on request.

[anandolee]

We are doing experiments on relative imports.
The results show relative imports breaks some of our internal projects and we are still working on it.

[cromefire]

I guess the most promising methods are 2, 3 and 4 in a helper method, because "../../some.proto" does not seem work with relative imports.

[cromefire]

That should work:

https://gist.github.com/cromefire/1677b4944458c83cbe566653232eaac9

[cromefire]

Method 4 (and 3) would work out of the box with no additional work required

[erwindassen]

Any update on this?

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.