Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Staging for version 2.0 #206

Open
wants to merge 184 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
184 commits
Select commit Hold shift + click to select a range
e887d2a
add rename and stats components
camillescott Nov 7, 2019
0b4ce27
add remap hmmer component
camillescott Nov 7, 2019
daf61bf
remove converted tasks
camillescott Nov 7, 2019
f28101e
move busco parsers to fileio submodule
camillescott Nov 7, 2019
4e0207a
convert and filter components
camillescott Nov 7, 2019
fcf9d04
Add conversion components
camillescott Nov 7, 2019
a37d5dd
Get gff3 merge command working
camillescott Nov 7, 2019
6293805
add busco, transdecoder wrappers
bluegenes Nov 7, 2019
b562c25
ADd annotate component
camillescott Nov 7, 2019
06fd042
Merge branch 'decompose' of github.com:camillescott/dammit into decom…
camillescott Nov 7, 2019
bb4b203
add rename-transcriptome wrapper
camillescott Nov 7, 2019
56ce4b7
add transdecoder test data, rules
bluegenes Nov 8, 2019
8a595bb
add busco rule, busco testdata
bluegenes Nov 8, 2019
6165178
add last
bluegenes Nov 8, 2019
d4f7bf1
add infernal wrapper and rule
bluegenes Nov 8, 2019
cdaf6ec
First round of workflow structuring: database download
camillescott Nov 13, 2019
accacaf
merge download rules
camillescott Nov 13, 2019
9f719eb
add transdecoder, last, infernal, hmmer wrappers
bluegenes Nov 15, 2019
388ce3e
init hmmer, transdecoder workflows
bluegenes Nov 16, 2019
86c25c3
code removal
camillescott Dec 2, 2019
0edabeb
Merge branch 'decompose' of github.com:camillescott/dammit into decom…
camillescott Dec 2, 2019
fd22338
add yaml; init annot snakes
bluegenes Dec 10, 2019
06b5ab6
init config changes; init main snake
bluegenes Dec 10, 2019
aa11da7
yml db file
bluegenes Dec 10, 2019
2778bea
Merge branch 'decompose' of github.com:camillescott/dammit into decom…
camillescott Dec 12, 2019
e111db1
working snakemake calls
bluegenes Dec 15, 2019
2429aaf
fix md5 downloads
bluegenes Dec 15, 2019
654d4a8
ignore metalink for now
bluegenes Dec 15, 2019
faa5671
fix lastdb wrapper
bluegenes Dec 15, 2019
a5564b8
working transdecoder
bluegenes Dec 16, 2019
c4c48a6
getting... closer...
bluegenes Dec 16, 2019
264d037
init env
bluegenes Dec 16, 2019
41b304f
Merge branch 'decompose' of github.com:camillescott/dammit into decom…
camillescott Feb 27, 2020
2c59f70
upd busco env and wrapper
bluegenes Feb 27, 2020
0726f2b
Remove cruft; use paths relative to installation dir for config and w…
camillescott Feb 28, 2020
8c36447
Merge branch 'decompose' of github.com:camillescott/dammit into decom…
camillescott Feb 28, 2020
656660c
Get annotate working with install-relative paths and fix some formatt…
camillescott Feb 28, 2020
579253b
Update requirements
camillescott Apr 17, 2020
fb43ee3
init setup snakefile
bluegenes Apr 17, 2020
13b2b98
Remove sphinx
camillescott Apr 17, 2020
0db1612
Remove _build folder
camillescott Apr 17, 2020
0acc86f
add mkdocs to dev reqs
camillescott Apr 17, 2020
33fbbfe
Merge pull request #171 from dib-lab/update_reqs
camillescott Apr 17, 2020
020c1d7
init file conversion rules
bluegenes Apr 18, 2020
6be0cde
add gff3 file targets; fix rename txome wrapper
bluegenes Apr 18, 2020
e3cf8c0
Merge pull request #175 from dib-lab/add_conversion_rules
camillescott Apr 18, 2020
df7457a
init shmlast wrapper, init shmlast and busco-v4 rules
bluegenes Apr 23, 2020
5a82181
Refactor config and add config components
camillescott Apr 24, 2020
4198dac
First pass unifying argparse and click interfaces
camillescott Apr 24, 2020
7ec2089
Add database dir as envvar
camillescott Apr 24, 2020
a4045e7
add .snakemake to gitignore
camillescott Apr 24, 2020
fe8105b
vendor cloup
camillescott Apr 24, 2020
abb3919
Reorganize components into cloup groups
camillescott Apr 24, 2020
7c74df2
Remove busco tasks
camillescott Apr 24, 2020
0c3f2d8
remove tasks suybmodules
camillescott Apr 24, 2020
7fb46ba
Fix regression in remap hmmer
camillescott Apr 24, 2020
de2bd9e
add user db targets; init write config
bluegenes Apr 24, 2020
c7792bb
Nuke the rest of dammit.tasks
camillescott Apr 24, 2020
dda80e9
Strip out more doit utils
camillescott Apr 24, 2020
7c3b10f
add .vscode to gitignore
camillescott Apr 24, 2020
8ac85f5
Add --version switch
camillescott Apr 24, 2020
97df5e3
Remove help from remap hmmer
camillescott Apr 24, 2020
6936544
enable print config
bluegenes Apr 24, 2020
86e545c
fix configfile printing
bluegenes Apr 25, 2020
480e8f6
get dammit databases subcommand working again
camillescott Apr 25, 2020
13bb949
change cmd order
bluegenes Apr 25, 2020
70cf00c
fix configfiles
bluegenes Apr 25, 2020
caecc4a
add shmlast to pipeline yml; fix targeting
bluegenes Apr 26, 2020
1565730
enable busco
bluegenes Apr 27, 2020
5e69a02
upd snkmk
bluegenes Apr 27, 2020
4f0549b
refactor databases target gen
camillescott May 8, 2020
18af046
Bring in busco support
camillescott May 8, 2020
0d5bc54
change busco wrapper to enable config
bluegenes May 8, 2020
c0f34a9
comment out in progress rule
bluegenes May 9, 2020
befe039
Fix generate_annotation_targets, convert annotate to click parsing
camillescott May 9, 2020
cded805
merge in remove_argparse branch
bluegenes May 9, 2020
f1a2fe6
upd busco databases, fix busco targeting in run
bluegenes May 9, 2020
7af8112
fix busco db in run
bluegenes May 9, 2020
ad03bf8
working databases download
bluegenes May 9, 2020
bdd638d
working busco run!
bluegenes May 9, 2020
f0a5fc9
move addl setup rule to new branch
bluegenes May 9, 2020
0e31171
Merge pull request #179 from dib-lab/add_busco_config
camillescott May 15, 2020
ed37c53
Merge pull request #177 from dib-lab/remove_argparse
camillescott May 15, 2020
00bf252
Add in best-hits step and fix database attribute in gff3 outputs
camillescott May 15, 2020
2bf0ea6
seems to be working
bluegenes May 16, 2020
84198d4
Fix file output naming conventions, start migrating config locations
camillescott May 29, 2020
81d7702
fix paths in ini config
bluegenes May 30, 2020
afb883c
Rewire temp folders and add config output subcommands
camillescott May 30, 2020
eac1321
Merge in BUSCO config fixes
camillescott May 30, 2020
17676d0
Add temp directory cleaning command
camillescott May 30, 2020
50d1bd4
Merge pull request #185 from dib-lab/extricate_fileio
camillescott May 30, 2020
9c12e37
add busco plotting
bluegenes May 30, 2020
2d324ac
Merge branch 'decompose' into plot_busco
bluegenes May 30, 2020
0d091bf
message formatting
camillescott Jun 26, 2020
8de727f
Merge pull request #186 from dib-lab/plot_busco
camillescott Jun 26, 2020
c632c2e
Add tests for wrappers and components (#188)
camillescott Aug 7, 2020
4ba07a9
use mamba for conda installations (#193)
bluegenes Aug 7, 2020
d1a450f
Nuke it from orbit: general clean-up and bugfixing (#196)
camillescott Oct 26, 2020
e07d01f
Refactor GFF3 conversion code to fix ID non-determinism with threadin…
camillescott Oct 30, 2020
4ac8af9
Snakefile formatting and file hierarchy cleanup (#201)
camillescott Jan 13, 2021
7031204
Remove busci.ini download as part of databases pipeline.
camillescott Jan 14, 2021
00bf7da
--n-threads can be set by environment variable
camillescott Jan 14, 2021
330626a
Add nice output to run commands
camillescott Jan 15, 2021
7bc1c38
Remove extraneous print
camillescott Jan 22, 2021
40ef4dc
Add an environment variable DAMMIT_TESTING_TEMP_BASE_DIR to set a sta…
camillescott Jan 22, 2021
99a1fbd
Merge pull request #204 from dib-lab/ui_improvements
camillescott Jan 22, 2021
91202b3
Merge pull request #205 from dib-lab/conda_testing_configuration
camillescott Jan 22, 2021
ee58118
Add dammit-transcriptome-stats to annotation workflow
camillescott Jan 23, 2021
38dd482
Add transcriptome_stats task to all pipelines
camillescott Jan 25, 2021
ba4e3e2
add rich to environment requirements
camillescott Jan 25, 2021
f33cd31
Fix typo in pipelines definitions
camillescott Jan 25, 2021
5032d72
Update shmlast version in requirements.txt; add environment name to e…
camillescott Jan 25, 2021
7b23224
add hmmscan and Pfam-A to quick pipeline (otherwise the gff3 conversi…
camillescott Jan 25, 2021
169dac7
environment suggestions
luizirber Jan 25, 2021
b37b77e
add missing dep on curl
luizirber Jan 25, 2021
7ee1cc2
Merge pull request #207 from luizirber/environment_suggestions
camillescott Jan 25, 2021
6f9bccf
First pass: github actions for conda and pytest
camillescott Jan 28, 2021
58f5995
Remove .travis.yml
camillescott Jan 28, 2021
53743b4
add master and staging_v2 to triggers
camillescott Jan 28, 2021
f10e247
Remove python 3.9 from test matrix (for now)
camillescott Jan 28, 2021
c95d125
Don't set DAMMIT_DB_DIR
camillescott Jan 28, 2021
b46a52a
Remove macos from matrix (for now)
camillescott Jan 28, 2021
b98c49d
Add DAMMIT_DB_DIR back as global environment var
camillescott Jan 28, 2021
49741be
remove tilde from dammit_db_dir
camillescott Jan 28, 2021
a12bee3
Add lengths file output to test_components::TestTranscriptomeStats
camillescott Jan 28, 2021
21e822e
Update annotate_user_database docstring with --n-threads
camillescott Jan 28, 2021
d95b6c3
Update udb data for fixed quick pipeline definition
camillescott Jan 28, 2021
ef1f4a6
Merge branch 'actions_migration' into fix_quick_test_data
camillescott Jan 28, 2021
35fa697
Merge pull request #210 from dib-lab/fix_quick_test_data
camillescott Jan 28, 2021
bb470a8
Add snakemake environment caching step
camillescott Jan 28, 2021
92c898d
Implement database cache
camillescott Jan 28, 2021
60546e4
Export DAMMIT_TESTING_TEMP_BASE_DIR
camillescott Jan 28, 2021
a84d85a
Remove unused annotate.py
camillescott Jan 28, 2021
d3055f3
Extricate original logging
camillescott Jan 28, 2021
5733654
Merge pull request #211 from dib-lab/snakemake_env_cache
camillescott Jan 28, 2021
939f998
Merge branch 'v2_staging' into cleanup
camillescott Jan 28, 2021
94f28c9
Remove import of deprecated logging
camillescott Jan 28, 2021
1dbca81
Merge pull request #212 from dib-lab/cleanup
camillescott Jan 28, 2021
14959eb
Sketch out BUSCO to GFF3 conversion
camillescott Jan 28, 2021
6e62ba2
Add attributes column to BUSCO_to_GFF3 converter and fix starting coo…
camillescott Jan 28, 2021
35708c6
Add BUSCO conversion to Snakemake workflow
camillescott Jan 29, 2021
a346d56
Update test data with BUSCO hits
camillescott Jan 29, 2021
d96775f
Remove unused pom.50.fa test data
camillescott Jan 29, 2021
27c61fe
Add larger pom subset as test data:
camillescott Jan 29, 2021
7399137
Handle case where there are no BUSCO hits
camillescott Jan 29, 2021
a4e4215
Test multiple busco groups
camillescott Jan 29, 2021
9ada547
Merge pull request #213 from dib-lab/busco_gff_output
camillescott Jan 29, 2021
4b22e76
Attempt making paths portable
camillescott Jan 29, 2021
2ff908e
Use env context
camillescott Jan 29, 2021
9fe86f7
Try using runner.temp for dammit data
camillescott Jan 29, 2021
10126ed
dammit how does this work
camillescott Jan 29, 2021
833bd0c
Add support for Snakemake extra args to annotate and databases
camillescott Jan 29, 2021
8514bd3
Add test for extra snakemake args under annotate subcommand
camillescott Jan 30, 2021
d4f07ec
Use quick pipeline for snakemake args test
camillescott Feb 2, 2021
22bfbb4
Merge pull request #219 from dib-lab/extra_snakemake_args
camillescott Feb 2, 2021
41b95f3
Add --name-map option to dammit annotate-fasta to remap names back to…
camillescott Feb 3, 2021
7766172
Add test data for dammit annotate-fasta --name-map
camillescott Feb 3, 2021
2073274
Use iloc instead of bracket indexing, as bracket indexing doesn't act…
camillescott Feb 3, 2021
06ff3bd
Expose --rename/--no-rename in annotate CLI to enable mapping of rena…
camillescott Feb 3, 2021
2fc126d
Expose --split-regex in the dammit run annotate CLI with --regex-rename
camillescott Feb 3, 2021
5f4a869
Fix Infernal typo
camillescott Feb 3, 2021
d35515a
Add a verbosity flag
camillescott Feb 4, 2021
56f7076
BUSCO v5: add a BUSCOTableParser that properly splits out the coordin…
camillescott Feb 4, 2021
9bdd7b9
Use set() to fix strange bug where multiple HMMER targets are generat…
camillescott Feb 4, 2021
221a1df
Pass --global-evalue to BUSCO
camillescott Feb 4, 2021
f0ea818
Handle no BUSCO hits in table parser
camillescott Feb 4, 2021
f907867
Update test data and tests with data that works for BUSCO v5
camillescott Feb 9, 2021
2bac59a
Update busco wrapper test for smaller test data
camillescott Feb 9, 2021
cd8df6d
py.test captures STDOUT and STDERR
camillescott Feb 10, 2021
77b5eee
Add proper --busco-group flag to test_annotate_dbdir
camillescott Feb 10, 2021
7ce1650
Merge branch 'v2_staging' into test_busco_v5
camillescott Feb 10, 2021
27706a9
Merge pull request #223 from dib-lab/test_busco_v5
camillescott Feb 10, 2021
faa4ad5
Merge pull request #222 from dib-lab/verbosity
camillescott Feb 10, 2021
b038df3
Merge branch 'v2_staging' into expose_rename_regex
camillescott Feb 12, 2021
bc82940
Update generate-test-data.sh with --regex-rename
camillescott Feb 12, 2021
627db02
Add --regex-rename test
camillescott Feb 12, 2021
470ca6d
Actually add regex-rename test-data
camillescott Feb 12, 2021
13e9d95
Merge pull request #221 from dib-lab/expose_rename_regex
camillescott Feb 12, 2021
fa3410c
verbose pandas output
camillescott Feb 12, 2021
d4c4636
Sort columns in compare_gff
camillescott Feb 12, 2021
e2159aa
Add a --fail-on-repeats flag to and set it true when user runs annot…
camillescott Mar 15, 2021
4124c0d
Merge pull request #215 from dib-lab/actions_paths
camillescott Mar 18, 2021
d605b70
Update no-rename test-data to use --quick pipeline
camillescott Mar 18, 2021
e51584d
Merge pull request #226 from dib-lab/norename_tests
camillescott May 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Tests

on:
push:
branches: [latest, master, v2_staging]
pull_request:
branches: [latest, master, v2_staging]
schedule:
- cron: "0 0 * * 0" # weekly

jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-18.04]
py: [3.7, 3.8]
defaults:
run:
shell: bash -l {0}
env:
DAMMIT_DB_DIR: ${{ github.workspace }}/data/databases
DAMMIT_CONDA_DIR: ${{ github.workspace }}/data/snakemake-envs-${{ matrix.os }}
DAMMIT_TESTING_TEMP_BASE_DIR: ${{ github.workspace }}/data/snakemake-envs-${{ matrix.os }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- name: cache conda
uses: actions/cache@v1
env:
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key:
${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
hashFiles('environment.yml') }}

- name: setup conda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: ${{ matrix.py }}
mamba-version: "*"
channels: conda-forge,bioconda,defaults
activate-environment: dammit-env
environment-file: environment.yml
auto-activate-base: false
use-only-tar-bz2: true

- name: cache databases
uses: actions/cache@v2
with:
path: |
${{ env.DAMMIT_DB_DIR }}/Pfam-A.hmm
${{ env.DAMMIT_DB_DIR }}/Pfam-A.hmm.h3f
${{ env.DAMMIT_DB_DIR }}/Pfam-A.hmm.h3i
${{ env.DAMMIT_DB_DIR }}/Pfam-A.hmm.h3m
${{ env.DAMMIT_DB_DIR }}/Pfam-A.hmm.h3p
key: databases-${{ hashFiles('dammit/databases.yml') }}

- name: setup databases
run: |
dammit run --pipeline quick databases --install

- name: cache snakemake-conda
uses: actions/cache@v2
with:
path: |
${{ env.DAMMIT_CONDA_DIR }}
key: ${{ matrix.os }}-snakemake-envs

- name: run pytest
run: |
py.test -m "not long and not huge and not requires_databases"
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,7 @@ dammit.egg-info/

.ipynb_checkpoints/
.pytest_cache/

*.snakemake
.vscode/
envs/
29 changes: 0 additions & 29 deletions .travis.yml

This file was deleted.

96 changes: 0 additions & 96 deletions ChangeLog

This file was deleted.

Loading