Tidy up files #1233

gcapes · 2021-09-07T09:16:08Z

An initial stab at #1225.

gcapes · 2021-09-07T09:35:12Z

This will want checking for errors, but also I'd like to discuss the best solution to some of the extra complexity the data and scripts directory has introduced.

Episode 5 has only a small modification to access the goostats.sh script, which is now ../scripts/goostats.sh.
Episode 6 starts to get complicated, which is probably why there hasn't been a separation of data and scripts yet.

If we were to be doing this IRL, we'd add the scripts to PATH so they could be easily accessed, given the script we write calls the other we already have (goostat.sh). PATH isn't covered so we can't do that. The problem then is that one of the scripts makes a copy of the original data files stats-$datafile which means it has to be run from the data directory to avoid polluting the scripts directory with back ups of data files. There is an error I just spotted where the do-stats.sh script calls bash goostats.sh but it should be bash ../scripts/goostats.sh. But anyway, this hard-coding of the script file path is jarring and undermines the flexibility that is taught by using $@ to run on any user input.

So, put data and scripts back into the same file? Yes it's an anti-pattern, but without covering PATH I think it might be the best solution. Having the scripts just one level up from the data directory doesn't solve the relative path issue. It's going to have to be a compromise somewhere, the question is where?

deppen8 · 2021-09-08T00:36:45Z

@gcapes, I think I was the one who raised the "scripts next to data" as a bit "smelly", but I want to stress that I don't feel too strongly about that. If scripts next to data makes the whole thing run with minimal disruption, we should do that.

deppen8 · 2021-09-22T01:53:23Z

@gcapes , is this ready for detailed review yet?

gcapes · 2021-09-22T08:57:18Z

Yep, please go ahead!

deppen8

Thanks for all the hard work you have put in here, @gcapes. To review, I did a fresh make serve of the lesson from the PR then went through the whole thing as an instructor/learner would: download the data, start from episode 1, and carry on through episode 7. There are a few rough edges to smooth out, but hopefully those are not too onerous.

I made individual comments/suggestions where I could on the diffs. There were some places where I couldn't put comments directly in the diffs because they were on parts that you didn't change (yet). Those are:

The path that leads off episode 7 needs to be Desktop/shell-lesson-data/exercise-data/writing
The path in the Little Women exercise at the end of episode 7 also needs to be updated: shell-lesson-data/writing/exercise-data/writing/LittleWomen.txt

deppen8 · 2021-09-24T01:10:30Z

_episodes/02-filedir.md

@@ -433,8 +433,7 @@ $ ls -F
 {: .language-bash}

 ~~~
-amino-acids.txt   elements/     pdb/	        salmon.txt
-animals.txt       morse.txt     planets.txt     sunspot.txt
+animal-counts/  backup/  creatures/  numbers.txt  project/  proteins/  writing/


I think only these directories are available at this stage in the lesson.

Suggested change

animal-counts/ backup/ creatures/ numbers.txt project/ proteins/ writing/

animal-counts/ creatures/ numbers.txt proteins/ writing/

deppen8 · 2021-09-24T01:18:07Z

_episodes/02-filedir.md

+$ ls -s Desktop/shell-lesson-data/exercise-data/
+total 28
+ 4 animal-counts   4 creatures  12 numbers.txt   4 proteins   4 writing


I am seeing:

Suggested change

$ ls -s Desktop/shell-lesson-data/exercise-data/

total 28

4 animal-counts 4 creatures 12 numbers.txt 4 proteins 4 writing

$ ls -s Desktop/shell-lesson-data/exercise-data/

total 8

0 animal-counts 0 creatures 8 numbers.txt 0 proteins 0 writing

This must be an OS thing - I've just double checked the values. Which OS do you use?

deppen8 · 2021-09-24T01:22:07Z

_episodes/02-filedir.md

 $ ls -S Desktop/shell-lesson-data/data
-sunspot.txt  animal-counts  pdb        amino-acids.txt  salmon.txt
-planets.txt  elements       morse.txt  animals.txt
+animal-counts  creatures  proteins  writing  numbers.txt


My sorting looks like this:

Suggested change

animal-counts creatures proteins writing numbers.txt

proteins creatures writing animal-counts numbers.txt

I wonder what is going on here. This is what my bash is reporting...

$ du . | sort -n 4 ./writing/thesis 16 ./animal-counts 40 ./creatures 76 ./proteins 1060 ./writing 1208 .

deppen8 · 2021-09-24T01:25:34Z

_episodes/02-filedir.md

-> November ('11') would come before July ('7'). Similarly, putting the year first
-> means that June 2012 will come before June 2013.
-{: .callout}
+she creates a directory called `data`,


This data/ dir no longer exists. Everything is at the top level of north-pacific-gyre

$ ls north-pacific-gyre/ NENE01729A.txt NENE01751B.txt NENE01971Z.txt NENE02040A.txt NENE02043B.txt NENE01729B.txt NENE01812A.txt NENE01978A.txt NENE02040B.txt goodiff.sh NENE01736A.txt NENE01843A.txt NENE01978B.txt NENE02040Z.txt goostats.sh NENE01751A.txt NENE01843B.txt NENE02018B.txt NENE02043A.txt

deppen8 · 2021-09-24T01:26:24Z

_episodes/02-filedir.md

+She also creates a `scripts` directory to contain the processing script `goostats.sh`
+written by her supervisor, and any new scripts she will write for this project.
+


scripts is also obsolete now.

deppen8 · 2021-09-24T02:48:56Z

_episodes/04-pipefilter.md

+> 2.  `sort -t, -k2,2 animals.csv | uniq -c`
+> 3.  `cut -d, -f 2 animals.csv | uniq -c`
+> 4.  `cut -d, -f 2 animals.csv | sort | uniq -c`
+> 5.  `cut -d, -f 2 animals.csv | sort | uniq -c | wc -l`
 >
 > > ## Solution
 > > Option 4. is the correct answer.


Should refer to shell-lesson-data/exercise-data not shell-lesson-data/data

deppen8 · 2021-09-24T02:50:25Z

_episodes/04-pipefilter.md

@@ -603,11 +605,11 @@ so that you and other people can put those programs into pipes to multiply their
 ## Nelle's Pipeline: Checking Files

 Nelle has run her samples through the assay machines
-and created 17 files in the `north-pacific-gyre/2012-07-03` directory described earlier.
+and created 17 files in the `north-pacific-gyre` directory described earlier.
 As a quick check, starting from her home directory, Nelle types:


Suggested change

As a quick check, starting from her home directory, Nelle types:

As a quick check, starting from the `shell-lesson-data` directory, Nelle types:

deppen8 · 2021-09-24T02:53:33Z

_episodes/04-pipefilter.md

+> The file
+> [`shell-lesson-data/exercise-data/numbers.txt`](../shell-lesson-data/exericse-data/numbers.txt)


Don't think this needs to be a link.

deppen8 · 2021-09-24T02:59:40Z

_episodes/05-loop.md

 > `ls` gives the following output:
 >
 > ~~~


I also have lengths.txt and sorted-lengths.txt in this dir, FWIW.

deppen8 · 2021-09-24T03:28:12Z

_episodes/06-script.md

@@ -550,7 +558,7 @@ Of course, this introduces another tradeoff between flexibility and complexity.

 > ## Script Reading Comprehension
 >
-> For this question, consider the `shell-lesson-data/molecules` directory once again.
+> For this question, consider the `shell-lesson-data/exercise-data/proteins` directory once again.


This one is OK, but there are still a number of references to molecules in this lesson. Probably just need to find/replace those.

gcapes · 2021-09-28T10:57:06Z

Thanks for the review! It's a lot to work through. I think I've addressed all your comments, but there's also some stuff in episode 7 I've overlooked in that the filesystem diagram and subsequent output is now incorrect. I'll add a couple of to-do items below so I don't forget.

Update diagram to match current file structure
Update subsequent output

gcapes · 2021-10-27T15:19:38Z

I taught this today and it went well. I've pushed a fix to an example that gave the wrong answer, but otherwise nothing broke.

gcapes · 2022-01-06T13:46:49Z

The only thing which is (possibly) outstanding is an updated file tree diagram, but I think it would be too large given the new directory layout.
Is this ok to merge now if I fix the conflicts? @swcarpentry/shell-novice-maintainers

deppen8 · 2022-01-06T16:14:16Z

I say merge. The only way we can know the rough edges is if more people start using it.

Remove unused files, and simplify directory structure.

Fix swcarpentry#1225.

The previous config meant that new files or changes were added into the zip file, but deletions weren't removed.

The resulting tree diagram would be quite large, and given it's create manually I've decided to drop it for now.

gcapes · 2022-02-22T14:55:39Z

I've just replaced the ls output with the visually easier output from tree. I don't think it matters that we've not covered tree, and I've not shown the command - just the output. If no-one objects (?) I'll go ahead and merge this, and people can start teaching it.

deppen8 · 2022-03-16T03:22:44Z

Just want to add a comment to commemorate this PR 😄 getting merged. It was a huge lift by @gcapes and I am really excited to see how it improves the lesson. 🥳

Tidy up files

gcapes force-pushed the tidy-up-files branch 3 times, most recently from 7309dad to b287be3 Compare September 9, 2021 11:06

gcapes mentioned this pull request Sep 10, 2021

Lesson Contribution - Pipes and Filters #1215

Closed

gcapes force-pushed the tidy-up-files branch 2 times, most recently from 0e8cbb7 to 063dfc0 Compare September 16, 2021 11:02

deppen8 added the status:in review Being review by the community label Sep 24, 2021

deppen8 requested changes Sep 24, 2021

View reviewed changes

gcapes force-pushed the tidy-up-files branch from a818fc2 to 848f522 Compare October 26, 2021 13:55

gcapes requested a review from gdevenyi November 6, 2021 18:03

deppen8 mentioned this pull request Nov 10, 2021

Ep 3: Add data files for lessons with Sam and Jamie #1261

Open

gcapes added 11 commits January 7, 2022 11:10

Restructure example data files

630ebcc

Remove unused files, and simplify directory structure.

Improve Nelle's project folder

7b2d9cf

Remove hidden bash_profile file

60d2fa3

Update material to reflect new directory structure

f404284

Fix swcarpentry#1225.

Fix Makefile so zip matches source directory

921fb1d

The previous config meant that new files or changes were added into the zip file, but deletions weren't removed.

Update zip file for learners to download

cb8a830

Remove pointless gitkeep

f1b1c2d

Fix long line errors

00a9890

Fix path to Little Women

4854655

Fix output to show correct directories

534d009

Update path for exercise data

9777d09

gcapes added 10 commits January 7, 2022 11:33

Remove reference to scripts and data directories

b41b484

Fix tab completion section with new file structure

fa06f32

Fix backticks typo

af2f852

Fix directory errors

4f78486

Remove unneeded link to file

1966e61

Remove empty thesis directory from zip file

f38d75d

Show correct format of animal counts file

223f917

Screen intermediate files out of listing

8707dbe

Use current file tree for find episode

63056d4

The resulting tree diagram would be quite large, and given it's create manually I've decided to drop it for now.

Correct answer to exercise

8dd3b29

gcapes force-pushed the tidy-up-files branch from 126beb7 to 8dd3b29 Compare January 7, 2022 11:37

Use output from tree to show directory structure

4414bc4

Add code block type

8072b4b

gcapes merged commit 636f637 into swcarpentry:gh-pages Mar 8, 2022

This was referenced May 6, 2022

Suggestion: less changing of data files? #802

Open

Episode 4: Removing Unneeded Files exercise needs rewrite #848

Open

zkamvar pushed a commit that referenced this pull request May 2, 2023

Merge pull request #1233 from gcapes/tidy-up-files

dded18b

Tidy up files

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tidy up files #1233

Tidy up files #1233

gcapes commented Sep 7, 2021

gcapes commented Sep 7, 2021

deppen8 commented Sep 8, 2021

deppen8 commented Sep 22, 2021

gcapes commented Sep 22, 2021

deppen8 left a comment

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

gcapes Sep 28, 2021

deppen8 Sep 24, 2021

gcapes Sep 28, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

deppen8 Sep 24, 2021

gcapes commented Sep 28, 2021 •

edited

Loading

gcapes commented Oct 27, 2021

gcapes commented Jan 6, 2022

deppen8 commented Jan 6, 2022

gcapes commented Feb 22, 2022

deppen8 commented Mar 16, 2022

	animal-counts/ backup/ creatures/ numbers.txt project/ proteins/ writing/
	animal-counts/ creatures/ numbers.txt proteins/ writing/

	animal-counts creatures proteins writing numbers.txt
	proteins creatures writing animal-counts numbers.txt

		She also creates a `scripts` directory to contain the processing script `goostats.sh`
		written by her supervisor, and any new scripts she will write for this project.

	As a quick check, starting from her home directory, Nelle types:
	As a quick check, starting from the `shell-lesson-data` directory, Nelle types:

		> The file
		> [`shell-lesson-data/exercise-data/numbers.txt`](../shell-lesson-data/exericse-data/numbers.txt)

Tidy up files #1233

Tidy up files #1233

Conversation

gcapes commented Sep 7, 2021

gcapes commented Sep 7, 2021

deppen8 commented Sep 8, 2021

deppen8 commented Sep 22, 2021

gcapes commented Sep 22, 2021

deppen8 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcapes commented Sep 28, 2021 • edited Loading

gcapes commented Oct 27, 2021

gcapes commented Jan 6, 2022

deppen8 commented Jan 6, 2022

gcapes commented Feb 22, 2022

deppen8 commented Mar 16, 2022

gcapes commented Sep 28, 2021 •

edited

Loading