Restructure codebase to be more monorepo-friendly

[dhruvkb]

This PR prepares the WordPress/openverse-catalog repository to be merged into the WordPress/openverse repository.

Making it a PR helps verify that everything continues to work (like tests and CI) and also make the changes easier to review.

[AetherUnbound]

@dhruvkb I believe I've addressed the issues with the tests! We should now see the --extended flag respected on CI as well, which we can check by making sure not tests are skipped. I want to point out, specifically, the change I made in c833356 - after lots of debugging, I noticed that the quotes around bash -c 'pytest {{ args {{' were actually disappearing when the docker-compose run command was finally called. Here's an example before the change:

j catalog/test --extended
just _mount-test "bash -c \'pytest --extended\'"
env COMPOSE_PROFILES="catalog_dependencies" just ../up 
env COMPOSE_PROFILES="catalog_dependencies" docker-compose -f docker-compose.yml up -d
Recreating openverse-catalog_s3_1 ... 
Recreating openverse-catalog_s3_1 ... done
Recreating openverse-catalog_load_to_s3_1 ... done
just ../run --rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c \'pytest --extended\'
just dc run -u airflow  --rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c 'pytest --extended'
env COMPOSE_PROFILES="api,ingestion_server,frontend,catalog" docker-compose -f docker-compose.yml run -u airflow --rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c pytest --extended

Note that the last line shows as bash -c pytest --extended, meaning --extended (and any other args) get handed to bash rather than the pytest command. Here's what that same command looks like after my change:

j catalog/test --extended
just _mount-test "bash -c \'pytest --extended\'"
env COMPOSE_PROFILES="catalog_dependencies" just ../up 
env COMPOSE_PROFILES="catalog_dependencies" docker-compose -f docker-compose.yml up -d
openverse-catalog_postgres_1 is up-to-date
openverse-catalog_s3_1 is up-to-date
Starting openverse-catalog_load_to_s3_1 ... done
just ../run --rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c \'pytest --extended\'
just dc run -u airflow  "--rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c 'pytest --extended'"
env COMPOSE_PROFILES="api,ingestion_server,frontend,catalog" docker-compose -f docker-compose.yml run -u airflow --rm -e AIRFLOW_VAR_INGESTION_LIMIT=1000000 -w /opt/airflow/catalog --volume /home/aether/git-a8c/openverse-catalog/catalog/../docker:/opt/airflow/docker/ scheduler bash -c 'pytest --extended'

The quotes around 'pytest --extended' now ensure that all extra CLI args handed at the very top make their way down to pytest. Very odd behavior! I want to take a look at the cache warnings too, so I might push one more commit.

[AetherUnbound]

Okay I think I've got it rectified 😄

[dhruvkb]

Thank you @AetherUnbound for fixing nearly every shortcoming left. Also both you and @stacimc for the super-fast turnaround on the reviews.

Should we merge this into main or only consider this branch as the base for the monorepo merge and leave main unaffected? I'm inclined towards the latter so that our prod and infra can work unchanged while we move the code around.

[AetherUnbound]

I'm just realizing that we have a few community PRs (#999, #1055, #1059) that we should probably merge before we merge this one 😅 I'd be okay merging this into main after we merge those and update this PR! What we could do is disable the dag sync cron job on production, at least until we update the infrastructure repo and deploy a new version of the catalog. We're also waiting on the image data refresh right now which currently has a 3 week timeout, so there's not really a rush to deploy a new version - thus it might make more sense to be working off of main while we're at it. What do you think?

[dhruvkb]

I agree with all your points.

• We should merge the community PRs before this, then update this PR with those changes.
• If merging to main doesn't cause any breakages, I'm all for it. It'll probably take one week to fix all the usages with references to the monorepo so if there is no need to deploy in that time, we can merge.

[AetherUnbound]

Excellent! I will shut the dag-sync cron off once we've merged all the community PRs 🙂

[AetherUnbound]

Approving so this is not blocked when time comes to merge, beyond that it's ready to go!

[stacimc]

All of my local testing has gone great! Excited for this 🚀

[AetherUnbound]

I tested all of the commands and a couple of DAGs after the rebase - everything looks good! I only have one comment but after that this can be merged 🥳