Batching and ubiquitous dtypes #144
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Value matching looks for a list-like object instead of directly looking at the value itself.
|
|
Pull Request Test Coverage Report for Build 3622372693Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
Also for a couple references to github.com/pyiron/pyiron/tree/master, which worked because github kindly redirects us, but should still be main now.
A bespoke node causes all sorts of headaches let's switch over to making ports batchable and having a different update method under the hood
|
Actually wrapping nodes in a super-node was hell. The attack I'm trying now is to make individual ports "batchable", so that the same node UI can be used, and we just have a little toggle to say whether it's expecting batched or straight input. This basically delegates the complexity to the back-end, where the node needs to be able to handle updates with (new) and without (standard) any batched input. Still working on this part... |
This is just a proof of concept, needs to be refactored and pulled up to the parent class, with contingencies in place in case the node doesn't support batched running (which should also be reflected in the UI, disallowing the user from marking input as batched).
In anticipation of batching nodes that run jobs
Ironflow IO ports always have dtype, but sometimes it's None, so we can make this check a bit more specific
chr is a method not a type anyhow
Doesn't work for un-typed (output) ports yet! Probably need a new `Untyped` dtype for that, which would also allow me to get rid of a bunch of `if dtype is not None` checks
|
The parent "batchable" class now exists for pure-data nodes (i.e. those that don't run a pyiron job under the hood), and have been applied to the linspace and structure nodes so far.
Then the remaining big step is to apply the batchable parent to the |
Choice behaves differently by spec now
It was only ever set to the default value anyhow
So it will play well with manipulated arrays
|
Error is just the |
|
Coverage is just complaining because of access rights ( I'm going to just merge and see if it's a branch-vs-main access issue somehow. Otherwise the conventional wisdom over on the coveralls github issue is to try deleting the repo on the coveralls side and re-add it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A wrapper node that allows pure data nodes to internally iterate over batches of input data, producing batches of output data.For nodes that have an exec port, this will be more complicated. In general the interface of pyiron (which is heavily state-based) and ironflow (which strives to be as functional as possible) gets really tough when we start having nodes that run/read/parse pyiron jobs. So here the goal is just to get it working for pure the purely functional data nodes, and then probably do some re-organization before trying to hit the job-running nodes.This wound up being a complete rework of the DType system, including adding types to all ports by default, and introducing new constraints one which ports can connect to which. Inside this new framework, the ports can be toggled to represent 1D "batched" data of their original type. Toggling is done in the controller panel, and batch status is easily seen in the graph by ALL CAPS labels.
Glaringly, there is no
MatrixDType, only 1DListtypes. This is the biggest weakness, but the solution in broad strokes is pretty clear: introduce a new dtype for matrices with adimensionsfield, and then have the currentListbe a specialdimension=1case for this.I also had to really relax in the input types for plotting. Even though batching handles a lot of the nitty gritty, we still wind up doing some vector operations like transposition, slicing, and indexing. The right solution is to have these sorts of nodes pass type (well,
valid_class) information through themselves, but that's not implemented yet.How does this compare to other tools? The most mainstream graphical scripting language I can think of is the Unreal engine's "blueprints". Looking at their control structures docs, they don't offer anything like what we have here. IMO that is both a yellow flag that what we are doing is a bad idea, and a nice asset that we can offer this sort of streamlining.
Of course, an explicit for-loop node is still necessary for nested looping, or looping over dimensions with different length, but I do feel that batching covers the most common scenarios with quite some elegance.