Cannot run pip during pre-build command

[dimbleby]

Bug Report

• At what date and time did you most recently experience the problem?
  May 1st 00:00 UTC

• Where did you experience the problem? E.g. Azure Web Apps, Azure Functions, Azure Container Registry, or offline use.
  Azure Web Apps

• If your repo is publicly available please share its URL:
  https://github.com/dimbleby/e-treasure-hunt

• What happened?
  prebuild script fails to run a pip command, claiming /opt/python/3.9.7/bin/python3.9: bad interpreter: No such file or directory

• What did you expect or want to happen?
  https://docs.microsoft.com/en-us/azure/app-service/configure-language-python#customize-build-automation says that:

The script can itself run other Python and Node.js scripts, pip and npm commands, ...

So I expect that running a pip command should work

• How can we reproduce it?

PRE_BUILD_COMMAND=prebuild.sh
prebuild.sh contains

#!/bin/bash

pip install poetry

• Do you have log output? Please include between the backticks:

$ git push azure main:master --force
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1012 bytes | 1012.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Deploy Async
remote: Updating branch 'master'.
remote: Updating submodules.
remote: Preparing deployment for commit id '700b994657'.
remote: Repository path is /home/site/repository
remote: Running oryx build...
remote: Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
remote: You can report issues at https://github.com/Microsoft/Oryx/issues
remote:
remote: Oryx Version: 0.2.20211207.1, Commit: 46633df49cc8fbe9718772a3c894df221273b2af, ReleaseTagName: 20211207.1
remote:
remote: Build Operation ID: |UpymsBalEyg=.7e8df737_
remote: Repository Commit : 700b99465783fc2b6481c08dcec082a880d44597
remote:
remote: Detecting platforms...
remote: Detected following platforms:
remote:   python: 3.9.7
remote:
remote: Using intermediate directory '/tmp/8da2b061bda3711'.
remote:
remote: Copying files to the intermediate directory...
remote: Done in 0 sec(s).
remote:
remote: Source directory     : /tmp/8da2b061bda3711
remote: Destination directory: /home/site/wwwroot
remote:
remote: Executing pre-build command...
remote: /home/site/repository/prebuild.sh: /tmp/oryx/platforms/python/3.9.7/bin/pip: /opt/python/3.9.7/bin/python3.9: bad interpreter: No such file or directory
remote: Deployment Failed.
remote: Deployment Logs : 'https://e-treasure-hunt.scm.azurewebsites.net/jsonviewer?view_url=/api/deployments/700b99465783fc2b6481c08dcec082a880d44597/log'
To https://e-treasure-hunt.scm.azurewebsites.net/e-treasure-hunt.git
 + 05488a3...700b994 main -> master (forced update)

[csuriano23]

Same here

[cormacpayne]

@dimbleby @csuriano23 Hey folks, sorry about this issue -- taking a look at what's going on here, it seems that the pip command that's used during pip install poetry that you're providing as a part of the PRE_BUILD_COMMAND is looking for a python3.9 command in a folder that doesn't exist (more specifically, it's looking for the command in the folder where the Python 3.9.7 SDK would be if it were pre-baked into the Oryx build image that's being used by App Service).

We're looking at ways to migrate the symlinks that are set up in the build script generated by Oryx to point to the Python 3.9.7 SDK that the pip command is being called from (the /tmp/oryx/platforms/python path it's found in is reserved for SDKs pulled down dynamically, not pre-baked into the image). This change may take some time to be (1) identified and made on the Oryx side, and (2) adopted by App Service in their pipelines, so we're taking a look at a set of commands you can include in your PRE_BUILD_COMMAND to work around this issue temporarily.

For your repository @dimbleby , I see that you're using a pyproject.toml file, which is something that Oryx actually explicitly checks for in our build script, and runs pip install poetry if it finds:

elif [ -e "pyproject.toml" ]
    then
        set +e
        echo "Running pip install poetry..."
        InstallPipCommand="pip install poetry"
        printf %s " , $InstallPipCommand" >> "$COMMAND_MANIFEST_FILE"
        pip install poetry
        echo "Running poetry install..."
        ....

Out of curiosity, is there a need for you to provide the pip install poetry command as a part of PRE_BUILD_COMMAND, or would you be able to scrap this call and defer to the same command call that we would make above in the build script that's generated? As a note: this call is made within a Python virtual environment that we create, so that may or may not change if you can defer to the call here. Just let us know 😃

[dimbleby]

If Oryx supports poetry directly then that's good for me - is that documented?!

Currently I am installing poetry to export a requirements.txt, I'd be happy to avoid this indirection.

Edit to answer my own question: support for poetry is not documented. Per the link I gave when raising the issue, the description of install demands requirements.txt explicitly, accidentally hints that setup.py would also work, and makes no mention of pyproject.toml.

2. Run pip install -r requirements.txt. The requirements.txt file must be present in the project's root folder. Otherwise, the build process reports the error: "Could not find setup.py or requirements.txt; Not running pip install."

[dimbleby]

By the way, I see that your script goes plain poetry install.

Probably poetry install --no-dev would be preferable, I don't need to install flake8 and black and mypy and so on into my deployment environment.

[real-person]

@cormacpayne Is it possible to use the PRE_BUILD_COMMAND (or another method) to upgrade pip before Oryx installs dependencies from requirements.txt?

[cormacpayne]

@real-person Sorry for the delayed response -- for a sample Django Python application, I was able to provide the following:

docker run -it --rm -e PRE_BUILD_COMMAND='pip install --upgrade pip' -v ${PWD}:/app mcr.microsoft.com/oryx/build:azfunc-jamstack oryx build /app

Which executed the following early on in the build:

Executing pre-build command...
Collecting pip
  Downloading pip-22.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-22.1
Finished executing pre-build command.

It was able to update pip from 20.2.3 to 22.1, which I believe would solve your ask above. Please let me know otherwise 🙂

[dimbleby]

If Oryx supports poetry directly then that's good for me

turns out: not good enough. My application is a django application, and without a requirements.txt I don't hit this

Oryx/src/BuildScriptGenerator/Python/PythonBashBuildSnippet.sh.tpl

Line 233 in d1b2060

if grep -iq "Django" "$SOURCE_DIR/$REQUIREMENTS_TXT_FILE"

(Also, if I am reading the code right, you install poetry in the same virtual environment that the application is then installed in. That's probably a mistake - what if they have conflicting requirements?)

[xK3v]

@cormacpayne has there been any progress/solution/workaround on this? How do i get a pip install working in the PRE_BUILD_COMMAND ? I am trying to install Cython before I get to my requirements.txt, since a package i'm installing needs Cython already there.

I need this, because i am trying to install PyCocoTools straight from github in my requirements.txt using git+https://github.com/gautamchitnis/cocoapi.git@cocodataset-master#egg=pycocotools&subdirectory=PythonAPI , but that fails if i install Cython in the same step saying "No module named Cython". My solution is to install Cython before I get to the requirements.txt, so I am trying to run pip install Cython==0.29.30 in my PRE_BUILD_COMMAND in a Python 3.9 Azure Function.

I am getting the same error
/tmp/oryx/platforms/python/3.9.7/bin/pip: /opt/python/3.9.7/bin/python3.9: bad interpreter: No such file or directory

Thank you!

[riptusk331]

@cormacpayne just bumping here to see if there's been any progress on this. thanks

[EPanni]

The same here! In my case I need to upgrade pip and wheel before Oryx proceeds and install the dependencies from requirements.txt, otherwise I always get " ERROR: Failed building wheel for ...".

[cormacpayne]

@xK3v @riptusk331 @EPanni -- Hey folks, apologies that this issue is still affecting you; I attempted to reproduce this bad interpreter error with a few different images and command configurations, but have yet to run into it locally, so I was hoping that one of you would be able to provide the following if you have the time to gather it and are still hitting this issue:

• A link to your application source code (if publicly available, else if you could let me know what type of app you're running [Django, Flask, etc.] that would be great)
• What type of app deployment you're attempting to do (Azure Web App zip deploy, Azure Functions, Static Web Apps, etc.)
• The full logs of the unsuccessful build (if publicly available)

[erkib]

Here's one example:
2023-04-14T08:32:02.1164439Z ==============================================================================
2023-04-14T08:32:02.5597146Z Got service connection details for Azure App Service:'xxx'
2023-04-14T08:32:05.2299747Z Package deployment using ZIP Deploy initiated.
2023-04-14T08:32:37.9926898Z Updating submodules.
2023-04-14T08:32:37.9929071Z Preparing deployment for commit id '5355d5dd-b'.
2023-04-14T08:32:37.9929446Z PreDeployment: context.CleanOutputPath False
2023-04-14T08:32:37.9930120Z PreDeployment: context.OutputPath /home/site/wwwroot
2023-04-14T08:32:37.9930350Z Repository path is /tmp/zipdeploy/extracted
2023-04-14T08:32:37.9930674Z Running oryx build...
2023-04-14T08:32:38.5664178Z Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.10 -p virtualenv_name=antenv --log-file /tmp/build-debug.log -i /tmp/8db3cc2c04d96b8 --compress-destination-dir | tee /tmp/oryx-build.log
2023-04-14T08:32:38.5664656Z Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
2023-04-14T08:32:38.5664938Z You can report issues at https://github.com/Microsoft/Oryx/issues
2023-04-14T08:32:38.5665064Z
2023-04-14T08:32:38.5665273Z Oryx Version: 0.2.20220825.1, Commit: 2403244, ReleaseTagName: 20220825.1
2023-04-14T08:32:38.5665430Z
2023-04-14T08:32:38.5665577Z Build Operation ID: |7To9rV0cCGc=.92eea6cc_
2023-04-14T08:32:38.5665895Z Repository Commit : 5355d5dd-bc31-4245-bf8e-ccaae8aef62b
2023-04-14T08:32:38.5668847Z
2023-04-14T08:32:38.5672043Z Detecting platforms...
2023-04-14T08:32:38.5675142Z Detected following platforms:
2023-04-14T08:32:38.5678035Z python: 3.10.8
2023-04-14T08:32:38.5681409Z Version '3.10.8' of platform 'python' is not installed. Generating script to install it...
2023-04-14T08:32:38.5684276Z
2023-04-14T08:32:38.5687233Z Using intermediate directory '/tmp/8db3cc2c04d96b8'.
2023-04-14T08:32:38.5690056Z
2023-04-14T08:32:38.5692903Z Copying files to the intermediate directory...
2023-04-14T08:32:38.5695792Z Done in 1 sec(s).
2023-04-14T08:32:38.5698540Z
2023-04-14T08:32:38.5701394Z Source directory : /tmp/8db3cc2c04d96b8
2023-04-14T08:32:38.5704300Z Destination directory: /home/site/wwwroot
2023-04-14T08:32:38.5707075Z
2023-04-14T08:32:38.5709779Z
2023-04-14T08:32:38.5712819Z Downloading and extracting 'python' version '3.10.8' to '/tmp/oryx/platforms/python/3.10.8'...
2023-04-14T08:32:38.5715788Z Detected image debian flavor: bullseye.
2023-04-14T08:32:38.5719182Z Downloaded in 3 sec(s).
2023-04-14T08:32:38.5722777Z Verifying checksum...
2023-04-14T08:32:38.5726070Z Extracting contents...
2023-04-14T08:32:38.5729344Z performing sha512 checksum for: python...
2023-04-14T08:32:38.5732593Z Done in 9 sec(s).
2023-04-14T08:32:38.5735740Z
2023-04-14T08:32:38.5738966Z image detector file exists, platform is python..
2023-04-14T08:32:38.5742409Z OS detector file exists, OS is bullseye..
2023-04-14T08:32:38.5745335Z Executing pre-build command...
2023-04-14T08:32:38.5748310Z /tmp/BuildScriptGenerator/2a92626ed0d2470b8d54daaed373308c/build.sh: /tmp/oryx/platforms/python/3.10.8/bin/pip: /opt/python/3.10.8/bin/python3.10: bad interpreter: No such file or directory
2023-04-14T08:32:38.5751957Z /tmp/BuildScriptGenerator/2a92626ed0d2470b8d54daaed373308c/build.sh: /tmp/oryx/platforms/python/3.10.8/bin/pip: /opt/python/3.10.8/bin/python3.10: bad interpreter: No such file or directory\n/bin/bash -c "oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.10 -p virtualenv_name=antenv --log-file /tmp/build-debug.log -i /tmp/8db3cc2c04d96b8 --compress-destination-dir | tee /tmp/oryx-build.log ; exit $PIPESTATUS "
2023-04-14T08:32:38.5755242Z
2023-04-14T08:32:38.5757925Z Generating summary of Oryx build
2023-04-14T08:32:38.5760591Z Parsing the build logs
2023-04-14T08:32:38.5763165Z Found 0 issue(s)
2023-04-14T08:32:38.5790856Z
2023-04-14T08:32:38.5791181Z Build Summary :
2023-04-14T08:32:38.5791325Z ===============
2023-04-14T08:32:38.5791440Z Errors (0)
2023-04-14T08:32:38.5791564Z Warnings (0)
2023-04-14T08:32:38.5791857Z
2023-04-14T08:32:38.5792051Z Deployment Failed. deployer = VSTS_ZIP_DEPLOY deploymentPath = ZipDeploy. Extract zip. Remote build.
2023-04-14T08:32:38.5825372Z ##[error]Failed to deploy web package to App Service.
2023-04-14T08:32:38.5833155Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

Deployment log:
[
{
"log_time": "2023-04-14T08:32:39.6755718Z",
"id": "b491ab7b-12e0-45b2-a4f3-da4c5e1fb98f",
"message": "Deployment failed.",
"type": 2,
"details_url": null
}
]

I assume this /tmp/build-debug.log is not retained or it should have ended in above...

[pauld-msft]

Hello all, sorry for the delay in getting back to you.

I believe that this issue is caused because the language-agnostic prebuild command is executed before we activate the python virtual environment. We have created a ticket on our side to introduce a new environment variable that would execute a command within the virtual environment.

Please let me know if that solution does not sound like it would resolve the issue for anyone.

For those who would like to execute a command outside of the virtual environment, you should be able to run pip as follows to circumvent the bad interpreter issue (providing the version of python that is used to build your project):
export PRE_BUILD_COMMAND="source /opt/oryx/benv python=3.8.16 dynamic_install_root_dir="/tmp/oryx/platforms" && python3 -m pip install Cython==0.29.30"

[simonjj]

Internal issue is User Story 1767404. Issue has not been addressed beyond the workaround above.

[avineshwar]

Is there any update here? Can I upgrade pip (or change pip version) before Oryx uses it? Latest pip is not used by Oryx.

If that is not possible, what combination of .deployment and deploy.sh can help me in deploying my Python code (Linux, App Service, Local Git)?

I need to do the following:

• Install a particular version of pip (latest in my case)
• Install the requirements

Something as simple as this is not working in my case since I need to use a particular version of pip.

I cannot find any documentation as to how can I do this for my configuration:

• Local Git
• Oryx (or Oryx)
• Python 3.x
• App Service (code project)