Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dnm] roachprod: start crdb in a container #61901

Closed
wants to merge 1 commit into from

Conversation

tbg
Copy link
Member

@tbg tbg commented Mar 12, 2021

This isn't something I am planning to finish, but I wanted to see if I
would immediately run into any snags when trying to get roachprod start to spin up a Docker container instead of running bare-metal.

The motivation is twofold:

  • I'm generally interested in how we involve roachprod over time.
    Specifically, the question is whether in the long run systems testing
    should be done in k8s and which, if any, stepping stones there are to
    evolve what we currently do into that direction.
  • The current bare-metal setup goes unresponsive when CRDB goes into
    overdrive, and tests fail in the most opaque ways. We then incur a
    large tax for debugging these situations since we can't access the
    cluster in that state.

Work that would need to be done to really finish this:

  • maintain ubuntu images that come with docker installed (right now
    needs to be manually set up) and the ubuntu user set up and the images
    we need cached
  • we hit the old problem of having to pass the uid/gid to the container
    to avoid creating files as root, I hacked around it by hard-coding
    them
  • Setting the cluster settings, etc, is all done via the ./cockroach
    binary but that is no longer a thing. Ideally that should use SQL
  • The whole premise of uploading binaries is out of the window, we
    need to deal in CRDB containers exclusively, though we could
    conceivably use a wrapper container running the uploaded binary
    if we wanted to retain how roachprod/test work.

cc @irfansharif

Release note: None

This isn't something I am planning to finish, but I wanted to see if I
would immediately run into any snags when trying to get `roachprod
start` to spin up a Docker container instead of running bare-metal.

The motivation is twofold:

- I'm generally interested in how we involve roachprod over time.
  Specifically, the question is whether in the long run systems testing
  should be done in k8s and which, if any, stepping stones there are to
  evolve what we currently do into that direction.
- The current bare-metal setup goes [unresponsive] when CRDB goes into
  overdrive, and tests fail in the most opaque ways. We then incur a
  large tax for debugging these situations since we can't access the
  cluster in that state.

Work that would need to be done to really finish this:

- maintain ubuntu images that come with docker installed (right now
  needs to be manually set up) and the `ubuntu` user set up and the images
  we need cached
- we hit the old problem of having to pass the uid/gid to the container
  to avoid creating files as root, I hacked around it by hard-coding
  them
- Setting the cluster settings, etc, is all done via the `./cockroach`
  binary but that is no longer a thing. Ideally that should use SQL
- The whole premise of uploading binaries is out of the window, we
  need to deal in CRDB containers exclusively, though we could
  conceivably use a wrapper container running the uploaded binary
  if we wanted to retain how roachprod/test work.

[unresponsive]: cockroachdb#59424 (comment)

Release note: None
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants