Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use gemspec executable (the only one or the first specified executable) as default gem entry point for tebako package #233

Open
maxirmx opened this issue Jan 12, 2025 · 18 comments
Labels
enhancement New feature or request

Comments

@maxirmx
Copy link
Member

maxirmx commented Jan 12, 2025

  • gem: check gemspec for executables (but we can only have one..., so the user still needs to choose)

Originally posted by @ronaldtse in #215 (comment)

@maxirmx maxirmx added the enhancement New feature or request label Jan 12, 2025
@bradgessler
Copy link
Contributor

bradgessler commented Jan 14, 2025

I keep looking at https://github.com/tamatebako/tebako?tab=readme-ov-file#packaging-scenarios-with-ruby and find it mostly confusing, mainly because I don't really know what combination is being used from that table, which leads to surprises.

I think a better approach might be to instead use flags in the press method or have different press commands. For example, I'd like to give tebako a Gemfile and reference a binary in there like this:

ruby "3.3.6"

source "https://rubygems.org"

gem "terminalwire", "~> 0.3.0.alpha"

Then I'd want to run a command like this to generate my executable:

$ tebako press -b Gemfile -e terminalwire-exec

The -o would match the -e by default to keep the command less verbose.

Similarly, perhaps I could pass a gem or gem path into press:

$ tebako press -g terminalwire -v "0.3.0.alpha1" -e terminalwire-exec

Files might work like this:

$ tebako press -b Gemfile -f my-one-off-script.rb -o my-app

This approach would take a lot of the guess work and wondering out of how the binary is being built and probably yield better error messages for misconfigurations.

@maxirmx
Copy link
Member Author

maxirmx commented Jan 15, 2025

  1. Tebako packages directory trees
  2. The directory tree has a root
  3. In the root of the directory tree there may be:
    Gemfile - none or one
    *.gemspec - none, one or many ("gemspecs")
    *.gem - none, one or many ("gems")
  4. Scenarios:
    Error: If the there are many gemspecs it is an error
    Bundled gem: If there is one Gemfile and one gemspec tebako runs
    bundle install
    gem build 
    gem install
    
    Gem: If there is no Gemfile and one gemspec tebako runs
    gem build 
    gem install
    
    Bundle: If there is one Gemfile and no gemspec tebako runs
    bundle install
    
    Packaged gems: If there is no Gemfile and no gemspec but one or many gems tebako runs for each gem
    gem install
    
    Script: If there is no Gemfile, no gemspec and gems tebako just copies files

IMHO there is nothing that is confusing in these rules. One does not need to be Ruby programmer or software engineer to understand them. I do not think that passing \home\maxirmx\project\Gemfile and not \home\maxirmx\project\ as a parameter will change anything.

@maxirmx
Copy link
Member Author

maxirmx commented Jan 15, 2025

This approach would take a lot of the guess work and wondering out of how the binary is being built and probably yield better error messages for misconfigurations.

Frankly, I do not understand the message above

Here follows a piece of real tebako log. Lines starting with @ are real commands that tebako executes to deploy the solution it packages.

-- Running init script
   ... creating packaging environment at /Users/runner/.tebako/o/s
-- Running deploy script
   ... installing tebako-runtime gem
   ... @ /Users/runner/.tebako/o/s/bin/gem install tebako-runtime --no-document --install-dir /Users/runner/.tebako/o/s/lib/ruby/gems/3.2.0
   ... collecting gem from gemspec /Users/runner/work/tebako-samples/tebako-samples/tutorial/2_packaging_scenarios/gemspec_and_gemfile.sample/tebako-table.gemspec and Gemfile
   ... @ /Users/runner/.tebako/o/s/bin/bundle config set --local build.ffi --disable-system-libffi
   ... @ /Users/runner/.tebako/o/s/bin/bundle config set --local build.nokogiri --no-use-system-libraries
   ... @ /Users/runner/.tebako/o/s/bin/bundle config set --local force_ruby_platform false
   *** It may take a long time for a big project. It takes REALLY long time on Windows ***
   ... @ /Users/runner/.tebako/o/s/bin/bundle install --jobs=3
   ... @ /Users/runner/.tebako/o/s/bin/bundle exec /Users/runner/.tebako/o/s/bin/gem build /Users/runner/work/tebako-samples/tebako-samples/tutorial/2_packaging_scenarios/gemspec_and_gemfile.sample/tebako-table.gemspec
   ... installing /Users/runner/.tebako/o/r/tebako-test-0.0.2.gem gem
   ... @ /Users/runner/.tebako/o/s/bin/gem install /Users/runner/.tebako/o/r/tebako-test-0.0.2.gem --no-document --install-dir /Users/runner/.tebako/o/s/lib/ruby/gems/3.2.0
   ... target entry point will be at /__tebako_memfs__/bin/tebako-table-cli
   ... stripping the output
-- Running mkdwarfs script
   ... @ /Users/runner/.tebako/deps/bin/mkdwarfs -o /Users/runner/.tebako/o/p/fs.bin -i /Users/runner/.tebako/o/s --no-progress

@bradgessler
Copy link
Contributor

Thanks for listing that out! I think I can put together a tebako package Gemfile.exe -e terminalwire-exec command together that:

  1. Reads the Ruby Version from the Gemfile.
  2. Grab all the gems (and dependencies), and work with path:, github:, etc. and puts them in a folder, probably a bunch of *.gem files.
  3. Generates a Tabfile with all the info needed to tebako press with the expected configuration.

This can then be passed into tebako press command to create a package.

IMHO there is nothing that is confusing in these rules. One does not need to be Ruby programmer or software engineer to understand them. I do not think that passing \home\maxirmx\project\Gemfile and not \home\maxirmx\project\ as a parameter will change anything.

It gets confusing when what should be the same commands fails differently on different platforms. I currently have a tebako press command that works on amd64 macOS:

# This works from macOS
$ tebako press -r gem/terminalwire -e terminalwire-exec -R 3.3.6 -o build/macos/amd64/bin/terminalwire-exec
Tebako executable packager version 0.12.1
No prefix specified, using ~/.tebako
Running tebako press at /Users/bradgessler/.tebako
   Mode:                      'bundle'
   Ruby version:              '3.3.6'
   Project root:              '/Users/bradgessler/Projects/terminalwire/ruby/gem/terminalwire'
   Application entry point:   'terminalwire-exec'
   Package file name:         '/Users/bradgessler/Projects/terminalwire/ruby/build/macos/amd64/bin/terminalwire-exec'
   Loging level:              'error'
   Package working directory: '<Host current directory>'
# Lots of log messages ...
[ 94%] Building CXX object CMakeFiles/tebako-fs.dir/src/tebako-main.cpp.o
[ 94%] Building CXX object CMakeFiles/tebako-fs.dir/Users/bradgessler/.tebako/deps/src/tebako/tebako-fs.cpp.o
[100%] Linking CXX static library libtebako-fs.a
[100%] Built target tebako-fs
-- Running finalize script
   ... building tebako package
   ... @ make ruby -j12
   ... @ make -j12
Created tebako package at "/Users/bradgessler/Projects/terminalwire/ruby/build/macos/amd64/bin/terminalwire-exec"
[100%] Built target tebako

But a similar command fails inside an Ubuntu Tebako container:

# This fails from Ubuntu
$ docker run -it -v $PWD:/host tebako-ubuntu-base bash
$(docker-host) tebako press -r /host/gem/terminalwire -e terminalwire-exec -R 3.3.6 -o terminalwire-exec-ubuntu
Tebako executable packager version 0.12.1
Using TEBAKO_PREFIX environment variable as prefix
Running tebako press at /root/.tebako
   Mode:                      'bundle'
   Ruby version:              '3.3.6'
   Project root:              '/host/gem/terminalwire/'
   Application entry point:   'terminalwire-exec'
   Package file name:         '/terminalwire-exec-ubuntu'
   Loging level:              'error'
   Package working directory: '<Host current directory>'
-- Generating files
   ... tebako-version.h
   ... tebako-fs.cpp
   ... deploy.rb
-- Running tebako press script
Running tebako press script
-- OSTYPE: 'linux-gnu'
-- NCORES: 4
Configuration summary:
-- ruby: v3.3.6 at /root/.tebako/deps/src/_ruby_3.3.6
-- dwarfs with tebako wrapper: @v0.9.2 at /root/.tebako/deps/src/_dwarfs_wr
-- DATA_SRC_DIR: /root/.tebako/o/s
-- DATA_PRE_DIR: /root/.tebako/o/r
-- DATA_BIN_DIR: /root/.tebako/o/p
-- DATA_BIN_FILE: /root/.tebako/o/p/fs.bin
-- Target binary directory: /root/.tebako/o/s/bin
-- Target library directory: /root/.tebako/o/s/lib
-- Target local directory: /root/.tebako/o/s/local
-- Target Gem directory:: /root/.tebako/o/s/lib/ruby/gems/3.3.0
-- FS_MOUNT_POINT: /__tebako_memfs__
-- Building for Win32 Ruby (RB_W32): OFF
-- Removing GLIBC_PRIVATE reference: OFF
-- Not building Ruby extensions: dbm,win32,win32ole,-test-/*
-- Ruby build cflags='-fPIC -I/root/.tebako/deps/include -I/usr/local/lib/ruby/gems/3.1.0/gems/tebako-0.12.1/include -fdeclspec'
-- Ruby build LDFLAGS='-L/root/.tebako/deps/lib -L/root/.tebako/o'
-- openssl Ruby option=''
-- libyaml Ruby option=''
-- Configuring done
-- Generating done
-- Build files have been written to: /root/.tebako/o
[ 60%] Built target _patchelf
[ 60%] Built target _dwarfs_wr
[ 89%] Built target _ruby_3.3.6
Tebako setup has been verified
[ 89%] Built target setup
-- Running init script
   ... creating packaging environment at /root/.tebako/o/s
-- Running deploy script
   ... installing tebako-runtime gem
   ... @ /root/.tebako/o/s/bin/gem install tebako-runtime --no-document --install-dir /root/.tebako/o/s/lib/ruby/gems/3.3.0 --bindir /root/.tebako/o/s/bin
   ... collecting gem from gemspec /host/gem/terminalwire/terminalwire.gemspec
   ... @ /root/.tebako/o/s/bin/gem build /host/gem/terminalwire/terminalwire.gemspec
   ... installing /root/.tebako/o/r/terminalwire-0.3.0.alpha1.gem gem
   ... @ /root/.tebako/o/s/bin/gem install /root/.tebako/o/r/terminalwire-0.3.0.alpha1.gem --no-document --install-dir /root/.tebako/o/s/lib/ruby/gems/3.3.0 --bindir /root/.tebako/o/s/bin
   ... target entry point will be at /__tebako_memfs__/bin/terminalwire-exec
tebako-packager failed: Entry point bin/terminalwire-exec does not exist or is not accessible [106]
make[3]: *** [CMakeFiles/packaged_filesystem.dir/build.make:70: CMakeFiles/packaged_filesystem] Error 106
make[2]: *** [CMakeFiles/Makefile2:198: CMakeFiles/packaged_filesystem.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:258: CMakeFiles/tebako.dir/rule] Error 2
make: *** [Makefile:202: tebako] Error 2
Tebako script failed: 'tebako press' build step failed [104]

The log messages make sense, but the error messages don't because I don't understand how tebako arrives at different conclusions about the presence of `bin/terminalwire.

I'm sure I'm doing something stupid or not interpreting the log files correctly, but that's true for most things in software development.

@bradgessler
Copy link
Contributor

bradgessler commented Jan 15, 2025

I'm sure I'm doing something stupid or not interpreting the log files correctly, but that's true for most things in software development.

I'm usually right when I say this 😂

I've worked out that when tebako builds this gem inside of a container, its running into this, which is the default spec.files provided by gems created with bundler gem:

Gem::Specification.new do |spec|
  # code ...
  spec.files = IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL) do |ls|
    ls.readlines("\x0", chomp: true).reject do |f|
      (f == gemspec) ||
        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile])
    end
  end
  # code 
end

When I run this from irb inside the container, I get nothing back:

root@0ac7b8e44b97:/host/gem/terminalwire# irb
irb(main):001:0> IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL).read
=> ""

When I run from the host machine, I get something back:

irb(main):023> IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL).read
=> ".env\u0000.github/workflows/Dockerfile\u0000.github/workflows/main.yml\u0000.gitignore\u0000.rspec\u0000Brewfile\u0000CHANGELOG.md\u0000CODE_OF_CONDUCT.md\u0000Gemfile\u0000Gemfile.lock\u0000LICENSE.txt\u0000README.md\u0000Rakefile\u0000bin/console\u0000bin/setup\u0000builder/build.sh\u0000builder/template/app/.gitkeep\u0000builder/template/bin/.gitkeep\u0000builder/template/lib/.gitkeep\u0000builder/template/lib/boot.rb\u0000builder/template/lib/boot.sh\u0000builder/template/vendor/.gitkeep\u0000examples/exec/localrails\u0000gem/terminalwire-client/lib/terminalwire-client.rb\u0000gem/terminalwire-client/lib/terminalwire/client.rb\u0000gem/terminalwire-client/lib/terminalwire/client/entitlement.rb\u0000gem/terminalwire-client/lib/terminalwire/client/entitlement/environment_variables.rb\u0000gem/terminalwire-client/lib/terminalwire/client/entitlement/paths.rb\u0000gem/terminalwire-client/lib/terminalwire/client/entitlement/policy.rb\u0000gem/terminalwire-client/lib/terminalwire/client/entitlement/schemes.rb\u0000gem/terminalwire-client/lib/terminalwire/client/exec.rb\u0000gem/terminalwire-client/lib/terminalwire/client/handler.rb\u0000gem/terminalwire-client/lib/terminalwire/client/resource.rb\u0000gem/terminalwire-client/lib/terminalwire/client/server_license_verification.rb\u0000gem/terminalwire-client/spec/entitlement/environment_variables_spec.rb\u0000gem/terminalwire-client/spec/entitlement/paths_spec.rb\u0000gem/terminalwire-client/spec/entitlement/policy_spec.rb\u0000gem/terminalwire-client/spec/entitlement/schemes_spec.rb\u0000gem/terminalwire-client/spec/entitlement_spec.rb\u0000gem/terminalwire-client/spec/resource_spec.rb\u0000gem/terminalwire-client/spec/spec_helper.rb\u0000gem/terminalwire-client/terminalwire-client.gemspec\u0000gem/terminalwire-core/lib/terminalwire-core.rb\u0000gem/terminalwire-core/lib/terminalwire.rb\u0000gem/terminalwire-core/lib/terminalwire/adapter.rb\u0000gem/terminalwire-core/lib/terminalwire/cache.rb\u0000gem/terminalwire-core/lib/terminalwire/logging.rb\u0000gem/terminalwire-core/lib/terminalwire/transport.rb\u0000gem/terminalwire-core/lib/terminalwire/version.rb\u0000gem/terminalwire-core/spec/cache_spec.rb\u0000gem/terminalwire-core/spec/spec_helper.rb\u0000gem/terminalwire-core/spec/terminalwire_spec.rb\u0000gem/terminalwire-core/terminalwire-core.gemspec\u0000gem/terminalwire-rails/lib/generators/terminalwire/install/USAGE\u0000gem/terminalwire-rails/lib/generators/terminalwire/install/install_generator.rb\u0000gem/terminalwire-rails/lib/generators/terminalwire/install/templates/application_terminal.rb.tt\u0000gem/terminalwire-rails/lib/generators/terminalwire/install/templates/bin/terminalwire\u0000gem/terminalwire-rails/lib/generators/terminalwire/install/templates/main_terminal.rb\u0000gem/terminalwire-rails/lib/terminalwire-rails.rb\u0000gem/terminalwire-rails/lib/terminalwire/rails.rb\u0000gem/terminalwire-rails/terminalwire-rails.gemspec\u0000gem/terminalwire-server/lib/terminalwire-server.rb\u0000gem/terminalwire-server/lib/terminalwire/server.rb\u0000gem/terminalwire-server/lib/terminalwire/server/context.rb\u0000gem/terminalwire-server/lib/terminairb(main):024>

I'm pretty sure bundler gem does this to avoid people accidentally releasing gems with files that are not checked into their repos.

@bradgessler
Copy link
Contributor

Running this from the container:

$ git config --global --add safe.directory '*'

And mounting the project root as a volume, with access to the git repo, results in a successful build.

This is a bit brittle though because if the person running press doesn't have their git repo mounted, nothing will build.

@maxirmx
Copy link
Member Author

maxirmx commented Jan 15, 2025

@bradgessler thank you for reporting this issue

I suggest the following approach

  1. Enter container
  2. Run tebako scenario commands
    In the case under discussion it will be
gem build ...
gem install ...
  1. Run gem executable
  2. Run
tebako press ...
  1. Run tebako package

If step 2 works and step 3 works but step 4 fails or step 5 fails it is tebako container or tebako issue.

@bradgessler
Copy link
Contributor

I'm confused about this step:

  1. Enter container

I searched through the list of tebako commands on the README.md and I'm not sure what it means to "enter container". My mental model for this is based on how Docker works, which means I'd run some sort of command like docker run -it $CONTAINER bash. Obviously Tebako doesn't work that way, but I'm not sure how to build a mental model around this.

Is there a command I can run to "enter [a Tebako] container"? Or do I extract the contents of a tebako package somewhere, do a bunch of things to it, and then somehow package it back up?

Thanks for any pointers on how I'd approach this 🙏

@maxirmx
Copy link
Member Author

maxirmx commented Jan 23, 2025

This response was generated by ChatGpt

What Does It Mean to "Enter a Docker Container" and How to Do It?

Entering a Docker container means accessing the environment inside the running container. This allows you to execute commands, inspect files, and debug processes directly within the container.


Steps to Enter a Docker Container

1. List Running Containers

To identify the container you want to enter, list all currently running containers:

docker ps

This command will display a table of running containers with details such as CONTAINER ID, NAME, and status.

2. Use docker exec to Enter the Container

Once you know the CONTAINER ID or NAME, use the docker exec command to start an interactive shell session inside the container:

docker exec -it <container_id_or_name> /bin/bash
-i: Keeps the input stream open.
-t: Allocates a pseudo-terminal for interaction.

Replace /bin/bash with /bin/sh if the container uses a minimal shell (common in lightweight images like Alpine).
Example:
If the container's name is my_container, use:

docker exec -it my_container /bin/bash

3. Use docker attach (Alternative)

You can also attach directly to the main process of the container:

docker attach <container_id_or_name>

However, this method is less flexible and may inadvertently stop the container if you type exit. It's better suited for monitoring container output.

4. Access a Stopped Container

If the container is not running, start it first:

docker start <container_id_or_name>
docker exec -it <container_id_or_name> /bin/bash

5.Exiting the Container

For docker exec: Type exit to leave the container without stopping it.
For docker attach:
Press Ctrl+P followed by Ctrl+Q to detach without stopping the container.
Typing exit will stop the container.

@bradgessler
Copy link
Contributor

I know how to enter a Docker container, I'm asking "how do I enter a Tebako container", as you stated in this comment:

Image

Did you mean "docker container" and not "tebako container"?

@ronaldtse
Copy link
Contributor

@bradgessler I think @maxirmx was referring to the Tebako CI Containers which can be used to build Tebako packages.

The relevant documentation is here:

@ronaldtse
Copy link
Contributor

  • gem: check gemspec for executables (but we can only have one..., so the user still needs to choose)

I think it's better if we force the user to define their executable because "choosing the first" is not a good solution. If there is one we can use it. If there are multiple then the user must choose?

@ByronHRio
Copy link

@bradgessler thank you for reporting this issue

I suggest the following approach

  1. Enter container
  2. Run tebako scenario commands
    In the case under discussion it will be
gem build ...
gem install ...
  1. Run gem executable
  2. Run
tebako press ...
  1. Run tebako package

If step 2 works and step 3 works but step 4 fails or step 5 fails it is tebako container or tebako issue.

I seem to be having this problem as well. I found your debugging instructions (step 3) helpful as it highlighted a problem that I had to fix, but have since done. I'm still experiencing the problem however.

One thing I notice is that the gem build within the tebako container does seem to work, but produces a very different (and wrong) gem file to that when gem build is run outside the container, on the host. The gem file built inside the container has an empty data.tar.gz.

As a side note, my gem is not hosted on rubygems.org but on a private geminabox server. To get step 3 working I have to add my private gem server to gem sources. Then I can gem exec -g <gem> script.rb. Not sure if this is useful or not.

But as you can imagine, with an empty gem file, I can't find the script to execute

@maxirmx
Copy link
Member Author

maxirmx commented Mar 5, 2025

Hi, @ByronHRio
Thank you for using tebako

Could you please look at https://github.com/tamatebako/tebako-ci-containers?tab=readme-ov-file#using-the-tebako-containers and package your gem using Outside the Conatainer method ?

If it does not work please provide the output of tebako press command

@ronaldtse
Copy link
Contributor

One thing I notice is that the gem build within the tebako container does seem to work, but produces a very different (and wrong) gem file to that when gem build is run outside the container, on the host. The gem file built inside the container has an empty data.tar.gz.

I wonder if this is caused by having a different environment given the Docker bind mounted volume?

As a side note, my gem is not hosted on rubygems.org but on a private geminabox server.

This is indeed a case we are not best at handling -- basically gems that are locally packaged (as *.gem files), gems that are not packaged (e.g. still under testing). We will have to find out a way ahead in a separate ticket.

@bradgessler
Copy link
Contributor

bradgessler commented Mar 5, 2025

@ByronHRio check your gemspec—the default from bundler gem ships with:

  spec.files = Dir.chdir(__dir__) do
    `git ls-files -z`.split("\x0").reject do |f|
      (File.expand_path(f) == __FILE__) || f.start_with?(*%w[bin/ test/ spec/ features/ .git .circleci appveyor])
    end
  end

which doesn't work inside of containers.

I think it's a bad default from bundler TBH, but there's so many gemspecs like it that Tebako might have to reckon with it for container builds.

@maxirmx
Copy link
Member Author

maxirmx commented Mar 5, 2025

Everyone, please read https://github.com/tamatebako/tebako-ci-containers?tab=readme-ov-file#using-the-tebako-containers

There are two distinct modes. Both work with default bundler setup.

@ByronHRio
Copy link

@bradgessler I believe the that was the issue yes. When tebako built the gem using both container methods (inside or outside) it would create an empty gem. I was able to solve this by installing tebako gem and build dependencies on the host, although I would've preferred not to do this and use container method. Thanks for pointing that out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants