Code

750 configure workers#777

Code

Merged

mauch merged 29 commits intodevaskap-vast/vast-pipeline:devfrom 750_configure_workersaskap-vast/vast-pipeline:750_configure_workersCopy head branch name to clipboardNov 5, 2024

+934-715

Commits

Commits on Oct 22, 2024

• Use itertuples over iterrows since iterrows is an enormous memory hog.

  mauch

  committed
  27de763

  Copy full SHA for 27de763

• Drop sources_df columns before renaming id column to avoid a copy of the while dataframe in memory.

  mauch

  committed
  b4c6088

  Copy full SHA for b4c6088

• Decrease default partition size to 15MB

  mauch

  committed
  e180a4e

  Copy full SHA for e180a4e

• Dont split (large-in-memory) list of DataFrames into dask bags (No performance hit).

  mauch

  committed
  9e9668d

  Copy full SHA for 9e9668d

• Don't write forced parquets in parallel (No perfomance hit for this).

  mauch

  committed
  d7e7b21

  Copy full SHA for d7e7b21

• Initial configuration updates for processing options.

  mauch

  committed
  2ee8052

  Copy full SHA for 2ee8052

Commits on Oct 23, 2024

• Dont overwrite input DataFrame when writing parquets.

  mauch

  committed
  0a08b7b

  Copy full SHA for 0a08b7b

• Update CHANGELOG.md

  mauch

  authored
  ff6269c

  Copy full SHA for ff6269c

Commits on Oct 29, 2024

• Merge branch 'dev' into optimise_mem

  mauch

  committed
  2d82245

  Copy full SHA for 2d82245

• Address review comments.

  mauch

  committed
  67ea721

  Copy full SHA for 67ea721

• Copy YAML objects before revalidation so the can be garbage collected.

  mauch

  committed
  c98720b

  Copy full SHA for c98720b

• Appease flake8

  mauch

  committed
  5c24a7d

  Copy full SHA for 5c24a7d

• Add processing options as optional with defaults.

  mauch

  committed
  cf8256e

  Copy full SHA for cf8256e

• filter processing config to parallel association.

  mauch

  committed
  4ca1c8b

  Copy full SHA for 4ca1c8b

Commits on Oct 31, 2024

• Add a funtion to determine the number of workers and partitions for Dask.

  mauch

  committed
  8e710f9

  Copy full SHA for 8e710f9

• Merge branch 'optimise_mem' into 750_configure_workers

  mauch

  committed
  772ec63

  Copy full SHA for 772ec63

Commits on Nov 1, 2024

• Use config values for num_workers and max_partition_size throughout pipeline.

  mauch

  committed
  8cd8eee

  Copy full SHA for 8cd8eee

• Correct working in config template.

  mauch

  committed
  3fed937

  Copy full SHA for 3fed937

Commits on Nov 4, 2024

• Update CHANGELOG.md

  mauch

  authored
  a9e2b5a

  Copy full SHA for a9e2b5a

• Remove unused imports.

  mauch

  committed
  0f596d9

  Copy full SHA for 0f596d9

• Merge branch 'dev' into 750_configure_workers

  mauch

  authored
  15783ef

  Copy full SHA for 15783ef

• Bump strictyaml to 1.6.2

  mauch

  committed
  0231f2c

  Copy full SHA for 0231f2c

• Use YAML 'null' to create Python None for all cores option.

  mauch

  committed
  c23753a

  Copy full SHA for c23753a

• Make None the default in calculate_workers_and_partitions instead of 0

  mauch

  committed
  5f9cd36

  Copy full SHA for 5f9cd36

• Merge branch '750_configure_workers' of https://github.com/askap-vast/vast-pipeline into 750_configure_workers

  mauch

  committed
  c0e6bd1

  Copy full SHA for c0e6bd1

• Updated run config docs

  ddobie

  committed
  137e425

  Copy full SHA for 137e425

• Allow null for num_workers_io and improve validation of processing parameters.

  mauch

  committed
  f27869a

  Copy full SHA for f27869a

Commits on Nov 5, 2024

• Merge branch '750_configure_workers' of https://github.com/askap-vast/vast-pipeline into 750_configure_workers

  mauch

  committed
  5a8f826

  Copy full SHA for 5a8f826

• Update num_workers_io default in docs.

  mauch

  committed
  2789445

  Copy full SHA for 2789445