Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) #1046

Merged
merged 6 commits into from
Mar 12, 2024
Merged

[EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) #1046

merged 6 commits into from
Mar 12, 2024

Conversation

laemtl
Copy link
Contributor

@laemtl laemtl commented Mar 8, 2024 •
edited
Loading

Support ingestion of datasets already in assembly_bids.
Also support partial dataset ingestion. We can now can run bids_import multiple times to ingest the datasets iteratively.

To import as partial dataset:

python bids_import.py -p database_config.py -d  /data/loris/data/assembly_bids/sub-xxx/ses-V03/eeg --nobidsvalidation --type raw

  • bids_import new option: --nobidsvalidation: Disable validation for datasets without a dataset_desscription.json (partial import). Required for partial ingestion.

  • bids_import new option: --type: Specify the dataset type for datasets without a dataset_desscription.json (partial import)

  • parse the bids_bir to detect if the path is in assembly_bids. If so, disable the dataset copy.

  • Add a new script: ingest_eeg_bids_datasets.py that ingest all the eeg uploads with status: 'extracted'.

  • Update the python dependencies to fix an install error

  • flake8 fixes

  • permissions fixes

Suggestions for improvements

  • rewrite bis_import as a class to limit the number of parameters to be passed between functions
  • centralize the logic for copying files in one single method so that it can be disabled more easily

@laemtl laemtl changed the base branch from main to 24.1-release March 8, 2024 17:11
@cmadjar cmadjar merged commit c4e8e22 into aces:24.1-release Mar 12, 2024
1 check passed
cmadjar added a commit that referenced this pull request Mar 15, 2024
@cmadjar @umn-kbreen @laemtl
Merge tag v24.1.15 into 25.0 release (#1047)
18f77c1 
* Update VERSION to 24.1.13

* Fixes for parameter file value lists (#1025)

* rm conversion to str on value list

* add check to ensure list is not empty

* Add method to check and skip duplicate content uploads to S3 - for 24.1-release (#1037)

* get Kim's code on 24.1-release branch

* fix flake8

* fix flake8

* remove pyblake2 from pip requirements.txt file as LORIS-MRI uses hashlib now (#1038)

* update version to 24.1.14 for bug fix release (#1039)

* Add a script to correct incorrectly saved list in parameter file (#1034)

* query parameter_file for the lists incorrectly saved

* finalize the update query

* finalize script help section

* [EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) (#1046)

* [EEG BIDS] Support assembly_bids source for datasets

* flake8

* Add executable permission

* QA comment

* Add support for config prePackagedDownloadPath and EEGChunksPath

* Update the bids_import script logic to explicitely add the --no-copy option

* fix incorrect conflict resolution

* fix incorrect conflict resolution

* bids_import.py - Missing short option fix

---------

Co-authored-by: breen0074 <[email protected]>
Co-authored-by: Laetitia Fesselier <[email protected]>
cmadjar added a commit that referenced this pull request Mar 15, 2024
@cmadjar @umn-kbreen @laemtl
Merge v24.1.15 into main (#1048)
9fb842c 
* Update VERSION to 24.1.13

* Fixes for parameter file value lists (#1025)

* rm conversion to str on value list

* add check to ensure list is not empty

* Add method to check and skip duplicate content uploads to S3 - for 24.1-release (#1037)

* get Kim's code on 24.1-release branch

* fix flake8

* fix flake8

* remove pyblake2 from pip requirements.txt file as LORIS-MRI uses hashlib now (#1038)

* update version to 24.1.14 for bug fix release (#1039)

* Add a script to correct incorrectly saved list in parameter file (#1034)

* query parameter_file for the lists incorrectly saved

* finalize the update query

* finalize script help section

* [EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) (#1046)

* [EEG BIDS] Support assembly_bids source for datasets

* flake8

* Add executable permission

* QA comment

* Add support for config prePackagedDownloadPath and EEGChunksPath

* Update the bids_import script logic to explicitely add the --no-copy option

* fix flake8

* bids_import.py - Missing short option fix

---------

Co-authored-by: breen0074 <[email protected]>
Co-authored-by: Laetitia Fesselier <[email protected]>
cmadjar added a commit that referenced this pull request Mar 18, 2024
@cmadjar @umn-kbreen @laemtl
Merge v24.1.16 into 25 (#1052)
c63f0e9 
* Update VERSION to 24.1.13

* Fixes for parameter file value lists (#1025)

* rm conversion to str on value list

* add check to ensure list is not empty

* Add method to check and skip duplicate content uploads to S3 - for 24.1-release (#1037)

* get Kim's code on 24.1-release branch

* fix flake8

* fix flake8

* remove pyblake2 from pip requirements.txt file as LORIS-MRI uses hashlib now (#1038)

* update version to 24.1.14 for bug fix release (#1039)

* Add a script to correct incorrectly saved list in parameter file (#1034)

* query parameter_file for the lists incorrectly saved

* finalize the update query

* finalize script help section

* [EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) (#1046)

* [EEG BIDS] Support assembly_bids source for datasets

* flake8

* Add executable permission

* QA comment

* Add support for config prePackagedDownloadPath and EEGChunksPath

* Update the bids_import script logic to explicitely add the --no-copy option

* Update set file path override and chunks logic (#1050)

* Update set file path override and chunks logic

* flake8

* Update set file path override and chunks logic - bugfix (#1051)

---------

Co-authored-by: breen0074 <[email protected]>
Co-authored-by: Laetitia Fesselier <[email protected]>
cmadjar added a commit that referenced this pull request Mar 18, 2024
@cmadjar @umn-kbreen @laemtl
Merge v24.1.16 into main (#1053)
a021200 
* Update VERSION to 24.1.13

* Fixes for parameter file value lists (#1025)

* rm conversion to str on value list

* add check to ensure list is not empty

* Add method to check and skip duplicate content uploads to S3 - for 24.1-release (#1037)

* get Kim's code on 24.1-release branch

* fix flake8

* fix flake8

* remove pyblake2 from pip requirements.txt file as LORIS-MRI uses hashlib now (#1038)

* update version to 24.1.14 for bug fix release (#1039)

* Add a script to correct incorrectly saved list in parameter file (#1034)

* query parameter_file for the lists incorrectly saved

* finalize the update query

* finalize script help section

* [EEG BIDS] Support assembly_bids source for datasets (LORIS 24.1) (#1046)

* [EEG BIDS] Support assembly_bids source for datasets

* flake8

* Add executable permission

* QA comment

* Add support for config prePackagedDownloadPath and EEGChunksPath

* Update the bids_import script logic to explicitely add the --no-copy option

* Update set file path override and chunks logic (#1050)

* Update set file path override and chunks logic

* flake8

* Update set file path override and chunks logic - bugfix (#1051)

* fix bad resolution

* fix bad resolution

---------

Co-authored-by: breen0074 <[email protected]>
Co-authored-by: Laetitia Fesselier <[email protected]>
@cmadjar cmadjar added this to the 24.1.15 milestone Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
24.1.15
Development

Successfully merging this pull request may close these issues.
2 participants
@laemtl @cmadjar