Poor example of file names and folder organization

[kekoziar]

I've always read the different folders north-pacific-gyre, creatures, molecules as different projects, and so they would have their own data folders. But, then there's the generically named data folder. A similar example is the creation of the thesis but not using the writing/thesis folder to move files into.

Can either the data files be originally a good model, or the lesson adapted to fix the file names/structure to model good file naming and organization practices?

[gcapes]

I agree - this bugs me too.
Perhaps some of the currently unused files were used at some point in the history of this lesson. I would propose that we cull anything which isn't currently used, and have the shell-lesson-data better organised to start with.
Someone has already put forward a suggestion for better organisation

[cilwerner]

I like that new organisation idea, and also to fix the naming practices. Another exercise, which has been eluded to could be the solution, but at the same time needing to keep it fairly short, MCQ style as an add on to the "Moving files and directories section" as "Nelle's pipeline: Organising Files Part 2", keeping in line with the story of the episode. Could it replace the "Organising Files and Directories" exercise starting in 03-create.md L815

Maybe something like this (suggest edits!);

---

Reorganising a folder structure

At present, Nelle's folder structure looks like this

data-shell/
├── creatures
├── data
             ├ elements
             ├ animal-counts
            └ pdb
├── molecules
├── north-pacific-gyre
            └ 2012-07-03
├── writing
             ├ data
             ├ thesis
            └ tools
                  └ old

Although there is no official right or wrong way to organise files, some are better than others, and Nelle is considering to reorganise her folder structure. Which of the following would be considered a good reorganisation of her file structure

1. Move the contents of data-shell/data into data-shell/writing/data (ANS: No, because you have data corresponding to writing and your other subprojects and it can be very confusing what data belongs to which. Remember that you can have the same folder names in different parent folders)

data-shell/
├── creatures
├── molecules
├── north-pacific-gyre
            └ 2012-07-03
├── writing
             ├ data
                  ├ elements
                  ├ animal-counts
                 └ pdb
             ├ thesis
            └ tools
                  └ old

2. Have a relevant data directory in most folders excluding writing (ANS: Yes, it is much clearer what data corresponds to which directory)

data-shell/
├── creatures
            └ data
                   └ animal-counts
├── molecules
            └ data
                    ├ elements
                   └ pdb
├── north-pacific-gyre
            └ 2012-07-03
├── writing
             ├ data
             ├ thesis
            └ tools
                  └ old

3. (Structure mentioned here (ANS: Yes - explanations therein)

4. Keep the file structure as is. There is no need to reorganise as it is perfectly clear where everything is (ANS: No, although it is a file structure that can work, it can be difficult to pinpoint what the dat corresponds to)

---

[gcapes]

I think having the suggested exercise without actually moving the files is a bit unsatisfactory, but tidying up unorganised files is a huge potential stumbling block for learners if later material refers to a particular directory structure.
So I think it's best to just have a good structure to start with.

I'm going to start work on tidying the files up, and have a bit more of a re-write of the lesson to include #1112.

[gcapes]

Before I sink too much time into this, I've come up with this file structure. It's a compromise between good practice and making the exercises needlessly difficult to follow (e.g. putting shell scripts in a different directory to Nelle's data files).

.
├── exercise-data
│   ├── animal-counts
│   │   └── animals.csv
│   ├── creatures
│   │   ├── basilisk.dat
│   │   ├── minotaur.dat
│   │   └── unicorn.dat
│   ├── numbers.txt
│   ├── proteins
│   │   ├── cubane.pdb
│   │   ├── ethane.pdb
│   │   ├── methane.pdb
│   │   ├── octane.pdb
│   │   ├── pentane.pdb
│   │   └── propane.pdb
│   └── writing
│       ├── haiku.txt
│       └── LittleWomen.txt
└── north-pacific-gyre
    └── 2012-07-03
        ├── goodiff.sh
        ├── goostats.sh
        ├── NENE01729A.txt
        ├── NENE01729B.txt
        ├── NENE01736A.txt
        ├── NENE01751A.txt
        ├── NENE01751B.txt
        ├── NENE01812A.txt
        ├── NENE01843A.txt
        ├── NENE01843B.txt
        ├── NENE01971Z.txt
        ├── NENE01978A.txt
        ├── NENE01978B.txt
        ├── NENE02018B.txt
        ├── NENE02040A.txt
        ├── NENE02040B.txt
        ├── NENE02040Z.txt
        ├── NENE02043A.txt
        └── NENE02043B.txt

Any glaring problems?

[deppen8]

I like the proposed structure a lot. The scripts inside the dated folder are a bit of an anti-pattern, but I agree that putting them in a scripts/ directory or something would be overkill. What about just having them up one level: north-pacific-gyre/goodif.sh, etc?

Have you tried playing through the lesson with it yet? I assume quite a bit needs to be edited.

[gcapes]

Great! Thanks for the feedback. I've not worked through the lesson yet with this new structure - it's going to be a lot of work so I wanted to get feedback on a new structure before I start :)

The problem I envisage with moving the scripts into a different directory might be something of a nothing, but it would certainly be more typing in the script created later in the lesson, and hence another place for learners to fall behind.

Ok, after typing and deleting a long stream-of-consciousness reply, I have come up with the following, which should encourage better directory structure, reduce typing of complicated names, and avoid the anti-pattern:

north-pacific-gyre/
├── data
│   ├── NENE01729A.txt
│   ├── NENE01729B.txt
│   ├── NENE01736A.txt
│   ├── NENE01751A.txt
│   ├── NENE01751B.txt
│   ├── NENE01812A.txt
│   ├── NENE01843A.txt
│   ├── NENE01843B.txt
│   ├── NENE01971Z.txt
│   ├── NENE01978A.txt
│   ├── NENE01978B.txt
│   ├── NENE02018B.txt
│   ├── NENE02040A.txt
│   ├── NENE02040B.txt
│   ├── NENE02040Z.txt
│   ├── NENE02043A.txt
│   └── NENE02043B.txt
└── scripts
    ├── goodiff.sh
    └── goostats.sh

[deppen8]

That's probably what I would do if it were my project, so that's alright with me.