python_runtime error on python3, python_version == '3.5'

[mechanicalgirl]

I'm trying to upgrade to Airflow 1.10.1 (with Python3) on CentOS 7 and getting an error related to the python_runtime version. I'm using the version numbers from your default and it's still failing.

I realize that this might not be addressed right away, if at all - it looks like this is because of a dependency in poise-python (which was archived without support for Python 3.5 or 3.6 for RHEL6).

# my Python config
default['airflow']['python_runtime'] = '3'
default['airflow']['python_version'] = '3.5'

error stack from chef-client:

  * python_runtime[3] action install
    * poise_languages_system[python] action install
      
      ================================================================================
      Error executing action `install` on resource 'poise_languages_system[python]'
      ================================================================================
      
      PoiseLanguages::Error
      ---------------------
      Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
      
      Cookbook Trace:
      ---------------
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
      /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
      
      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb
      
       32:         poise_languages_system system_package_name do
       33:           # Otherwise use the default install action.
       34:           action(:upgrade) if options['package_upgrade']
       35:           parent new_resource
       36:           # Don't pass true because we want the default computed behavior for that.
       37:           dev_package options['dev_package'] unless options['dev_package'] == true
       38:           dev_package_overrides dev_package_overrides
       39:           package_version options['package_version'] if options['package_version']
       40:           version options['version']
       41:           instance_exec(&block) if block
       42:         end
       43:       end
      
      Compiled Resource:
      ------------------
      # Declared in /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb:32:in `install_system_packages'
      
      poise_languages_system("python") do
        action [:install]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        declared_type :poise_languages_system
        cookbook_name "vox-airflow"
        parent # Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
      
      python_runtime("3") do
        provider PoisePython::PythonProviders::System
        action [:install]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
        declared_type :python_runtime
        cookbook_name "vox-airflow"
        recipe_name "packages"
        version "3.5"
        pip_version "10.0.1"
        get_pip_url "https://bootstrap.pypa.io/get-pip.py"
        setuptools_version true
        virtualenv_version true
        wheel_version true
      end
      
        version "3.5"
        package_name "python"
        dev_package "python-devel"
      end
      
      System Info:
      ------------
      chef_version=12.22.3
      platform=centos
      platform_version=7.4.1708
      ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
      program_name=chef-client worker: ppid=30126;start=23:31:37;
      executable=/opt/chef/bin/chef-client
      
    
    ================================================================================
    Error executing action `install` on resource 'python_runtime[3]'
    ================================================================================
    
    PoiseLanguages::Error
    ---------------------
    poise_languages_system[python] (/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb line 32) had an error: PoiseLanguages::Error: Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
    
    Cookbook Trace:
    ---------------
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
    
    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb
    
      1: python_runtime node['airflow']['python_runtime'] do
      2:   version node['airflow']['python_version']
      3:   provider :system
      4:   pip_version node['airflow']['pip_version']
      5: end
      6: 
    
    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
    
    python_runtime("3") do
      provider PoisePython::PythonProviders::System
      action [:install]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
      declared_type :python_runtime
      cookbook_name "vox-airflow"
      recipe_name "packages"
      version "3.5"
      pip_version "10.0.1"
      get_pip_url "https://bootstrap.pypa.io/get-pip.py"
      setuptools_version true
      virtualenv_version true
      wheel_version true
    end
    
    System Info:
    ------------
    chef_version=12.22.3
    platform=centos
    platform_version=7.4.1708
    ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
    program_name=chef-client worker: ppid=30126;start=23:31:37;
    executable=/opt/chef/bin/chef-client

[cloneluke]

hmm, I know poise is being deprecated but we definitely use airflow and this recipe with

default['airflow']['python_runtime'] = '3'
default['airflow']['python_version'] = '3.5'

please upgrade pip and try again:

# cant upgrade to pip 19 until this is fixed: https://github.com/poise/poise-python/pull/142
# https://github.com/poise/poise-python/issues/140
default['airflow']['pip_version'] = '18.0'
# can't use https://bootstrap.pypa.io/3.3/get-pip.py because it doesnt support 3.3
# added this revision url because of this issue: https://github.com/pypa/get-pip/issues/37
# added get_pip_url to airflow-cookbook: https://github.com/pypa/get-pip/issues/37
# default['airflow']['get_pip_url'] = 'https://raw.githubusercontent.com/pypa/get-pip/f88ab195ecdf2f0001ed21443e247fb32265cabb/get-pip.py'
default['airflow']['get_pip_url'] = 'https://bootstrap.pypa.io/3.3/get-pip.py'

[cloneluke]

also full disclosure, I have only tested with ubuntu, not centos