Skip to content

Commit

Permalink
schema: Memory thresholds should never be exactly 0.0. (#7458)
Browse files Browse the repository at this point in the history
In the config, memory thresholds such as `distributed.worker.memory.terminate` should never be exactly `0.0`.
Instead, config should use `false` to disable memory management.

This one bit me recently.  My older dask config files used `0.0` to disable the memory management features.  That worked because older versions of `distributed` interpreted the value `0.0` to be the equivalent to `false` for these fields.  But in newer versions, only `false` works.  (I suspect the change occurred in #5904.)

Nowadays, if the config says `0.0`, then `distributed` interprets that literally -- and no memory can be used at all without incurring the wrath of the memory manager!

An easy "fix" is to disallow `0.0` in the user's config.  In json schema, `exclusiveMinimum: 0` ensures that the value `0.0` itself is not permitted by the schema.
  • Loading branch information
stuarteberg authored Jan 6, 2023
1 parent 24ba207 commit 3e793f7
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions distributed/distributed-schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -541,15 +541,15 @@ properties:
transfer:
oneOf:
- {type: number, minimum: 0, maximum: 1}
- {type: number, exclusiveMinimum: 0, maximum: 1}
- {enum: [false]}
description: >-
When the total size of incoming data transfers gets above this amount,
we start throttling incoming data transfers
target:
oneOf:
- {type: number, minimum: 0, maximum: 1}
- {type: number, exclusiveMinimum: 0, maximum: 1}
- {enum: [false]}
description: >-
When the process memory (as observed by the operating system) gets
Expand All @@ -558,7 +558,7 @@ properties:
spill:
oneOf:
- {type: number, minimum: 0, maximum: 1}
- {type: number, exclusiveMinimum: 0, maximum: 1}
- {enum: [false]}
description: >-
When the process memory (as observed by the operating system) gets
Expand All @@ -568,7 +568,7 @@ properties:
pause:
oneOf:
- {type: number, minimum: 0, maximum: 1}
- {type: number, exclusiveMinimum: 0, maximum: 1}
- {enum: [false]}
description: >-
When the process memory (as observed by the operating system) gets
Expand All @@ -577,7 +577,7 @@ properties:
terminate:
oneOf:
- {type: number, minimum: 0, maximum: 1}
- {type: number, exclusiveMinimum: 0, maximum: 1}
- {enum: [false]}
description: >-
When the process memory reaches this level the nanny process will kill
Expand Down

0 comments on commit 3e793f7

Please sign in to comment.