Seeding intermediate material

[gvwilson]

This PR brings in a rough start to intermediate material on:

• Make
• Bash
• sets & dictionaries in Python

None of it is ready for use; we need a volunteer to take over each topic and flesh it out.

[wking]

On Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson wrote:

• Make
• Bash

I'm happy to pick up on Make or Bash unless someone else wants them.
However, I'm tempted to rename 'bash' to 'shell', since this is mostly
about utilities with a few references to the basic POSIX shell, and
hardly at all about Bash itself (as distinct from POSIX shells more
generally).

A hash/intermediate/01-sets.md (341)

Hash? Shouldn't this live under Python? I still prefer my
{subject}/{tool} naming scheme (or {subject}/{tool}/{level}?), but
mixing {tool}/{level} stuff like python/novice with
{subject}/{level} stuff like hash/intermediate is just going to
lead to confusion. What happens when the R folks want to talk about
hashes?

[rgaiacs]

The user should append the public key using cat and not replace the "old" authorized_keys. From ssh man page:

The user should then copy the public key to ~/.ssh/autho‐rized_keys in his/her home directory on the remote machine. The authorized_keys file corresponds to the conventional ~/.rhosts file, and has one key per line, though the lines can be very long. After this, the user can log in without giving the password.

[gvwilson]

On 2014-01-08 12:21 PM, W. Trevor King wrote:

On Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson wrote:

• Make
• Bash

I'm happy to pick up on Make or Bash unless someone else wants them.
However, I'm tempted to rename 'bash' to 'shell', since this is mostly
about utilities with a few references to the basic POSIX shell, and
hardly at all about Bash itself (as distinct from POSIX shells more
generally).
+1 to renaming 'shell' unless someone else objects?
And very happy to have you fix/extend that material - please check with
@ethanwhite about what else should be added (since he's the release
manager for intermediate stuff).

A hash/intermediate/01-sets.md (341)

Hash? Shouldn't this live under Python? I still prefer my
{subject}/{tool} naming scheme (or {subject}/{tool}/{level}?), but
mixing {tool}/{level} stuff like python/novice with
{subject}/{level} stuff like hash/intermediate is just going to
lead to confusion. What happens when the R folks want to talk about
hashes?
Any attempt to turn a table into a tree is going to feel wrong some of
the time, and the set/dict stuff feels too large to go into
'python/intermediate'. That said, I'm happy to put it to a vote.

• G

[wking]

On Wed, Jan 08, 2014 at 12:45:15PM -0800, Greg Wilson wrote:

+1 to renaming 'shell' unless someone else objects?

I'll let this cook, and submit a PR over the weekend if nobody
objects.

And very happy to have you fix/extend that material - please check
with @ethanwhite about what else should be added (since he's the
release manager for intermediate stuff).

If you or @ethanwhite already know stuff that needs to be done, it's
probably best to just create new issues for each fix/addition. I'll
file issues (or comment on existing PRs) if I notice anything.

2014-01-08 12:21 PM, W. Trevor King:

Hash? Shouldn't this live under Python? I still prefer my
{subject}/{tool} naming scheme (or {subject}/{tool}/{level}?),
but mixing {tool}/{level} stuff like python/novice with
{subject}/{level} stuff like hash/intermediate is just going
to lead to confusion. What happens when the R folks want to talk
about hashes?

Any attempt to turn a table into a tree is going to feel wrong some
of the time, and the set/dict stuff feels too large to go into
'python/intermediate'. That said, I'm happy to put it to a vote.

I don't see any need to vote. You and @ethanwhite are guiding the
novice/intermediate content. If I'm not convincing, then I'm not
convincing ;).

[rgaiacs]

Make is TAB dependent. What about replace the spaces with tab so that students can copy and paste?

[ethanwhite]

On Wed, Jan 8, 2014 at 12:21 PM, W. Trevor King
[email protected] wrote:

On Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson wrote:

• Make
• Bash

I'm happy to pick up on Make or Bash unless someone else wants them.
However, I'm tempted to rename 'bash' to 'shell', since this is mostly
about utilities with a few references to the basic POSIX shell, and
hardly at all about Bash itself (as distinct from POSIX shells more
generally).

That would be awesome. +1 to 'shell'.

On Wed, Jan 8, 2014 at 4:01 PM, W. Trevor King [email protected] wrote:

On Wed, Jan 08, 2014 at 12:45:15PM -0800, Greg Wilson wrote:

And very happy to have you fix/extend that material - please check
with @ethanwhite about what else should be added (since he's the
release manager for intermediate stuff).

If you or @ethanwhite already know stuff that needs to be done, it's
probably best to just create new issues for each fix/addition. I'll
file issues (or comment on existing PRs) if I notice anything.

I think we've already discussed everything I've thought about in [1].

[1] #126

2014-01-08 12:21 PM, W. Trevor King:

Hash? Shouldn't this live under Python? I still prefer my
{subject}/{tool} naming scheme (or {subject}/{tool}/{level}?),
but mixing {tool}/{level} stuff like python/novice with
{subject}/{level} stuff like hash/intermediate is just going
to lead to confusion. What happens when the R folks want to talk
about hashes?

Any attempt to turn a table into a tree is going to feel wrong some
of the time, and the set/dict stuff feels too large to go into
'python/intermediate'. That said, I'm happy to put it to a vote.

I don't see any need to vote. You and @ethanwhite are guiding the
novice/intermediate content. If I'm not convincing, then I'm not
convincing ;).

I agree that organizationally this probably belongs in Python, but I
think we do need to find a way to indicate that it's really a separate
chunk of material. What do folks thing about
/python/intermediate/hash. In other words, it's a distinct chunk of
material that is logically part of /python/intermediate.

[wking]

On Thu, Jan 09, 2014 at 05:16:11AM -0800, Ethan White wrote:

Wed, Jan 8, 2014 at 4:01 PM, W. Trevor King:

Wed, Jan 08, 2014 at 12:45:15PM -0800, Greg Wilson:

2014-01-08 12:21 PM, W. Trevor King:

Hash? Shouldn't this live under Python? I still prefer my
{subject}/{tool} naming scheme (or {subject}/{tool}/{level}?),
but mixing {tool}/{level} stuff like python/novice with
{subject}/{level} stuff like hash/intermediate is just going
to lead to confusion. What happens when the R folks want to talk
about hashes?

Any attempt to turn a table into a tree is going to feel wrong some
of the time, and the set/dict stuff feels too large to go into
'python/intermediate'. That said, I'm happy to put it to a vote.

I don't see any need to vote. You and @ethanwhite are guiding the
novice/intermediate content. If I'm not convincing, then I'm not
convincing ;).

I agree that organizationally this probably belongs in Python, but I
think we do need to find a way to indicate that it's really a separate
chunk of material. What do folks thing about
/python/intermediate/hash. In other words, it's a distinct chunk of
material that is logically part of /python/intermediate.

I still prefer {subject}/{tool}/{level} because I think that puts
questions answered (Why? How? Who?) are in order of decreasing
importance. That said, {tool}/{level}/{subject} still has all the
information, and I like it better than {tool|subject}/{level}.

[gvwilson]

On 2014-01-08 4:01 PM, W. Trevor King wrote:

On Wed, Jan 08, 2014 at 12:45:15PM -0800, Greg Wilson wrote:

+1 to renaming 'shell' unless someone else objects?

I'll let this cook, and submit a PR over the weekend if nobody
objects.
Please go ahead and rename 'bash' to 'shell' - you're right, it's what
we call it when we're speaking to each other, so that's what we should
call the directory.

[wking]

On Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson wrote:

• Make

We have a checkered history with Make. Sometimes we don't want it
(swcarpentry/DEPRECATED-boot-camps#33), sometimes we do
(swcarpentry/DEPRECATED-boot-camps#99, but just for 2013-05-krakow). Have we
reached a consensus about wanting it often enough to add it to master?

[gvwilson]

On 2014-01-13 4:20 PM, W. Trevor King wrote:

On Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson wrote:

• Make

We have a checkered history with Make. Sometimes we don't want it
(swcarpentry/DEPRECATED-boot-camps#33), sometimes we do
(swcarpentry/DEPRECATED-boot-camps#99, but just for 2013-05-krakow). Have we
reached a consensus about wanting it often enough to add it to master?

—
Reply to this email directly or view it on GitHub
#221 (comment).

I think Make is worth an hour in a two-day bootcamp for intermediates:
it's about as much time as we give regular expressions.

Greg Wilson
Software Carpentry http://www.software-carpentry.org/
Mozilla Science Lab http://mozillascience.org/
@swcarpentry | @swcannounces | @mozillascience

[wking]

On Tue, Jan 14, 2014 at 05:08:02PM -0800, Greg Wilson wrote:

I think Make is worth an hour in a two-day bootcamp for
intermediates: it's about as much time as we give regular
expressions.

Ok.

Can we split this into separate Make, hash, and Bash PRs? 4010 lines
of insertions are going to be a lot to review in a single, unthreaded
channel ;). I'll go through the Make stuff now; just let me know
where to push the comments.

[wking]

On Sun, Jan 19, 2014 at 09:13:10PM -0800, W. Trevor King wrote:

Can we split this into separate Make, hash, and Bash PRs? 4010
lines of insertions are going to be a lot to review in a single,
unthreaded channel ;). I'll go through the Make stuff now; just let
me know where to push the comments.

I've finished my first pass, and haven't heard back on a Make-specific
issue, so I'll just post my feedback here. I think the overall line
of development here is good, and works through the usual syntax in
nice, tidy steps. Here's a list of my comments by file, as I read
through the Markdown docs:

Wed, Jan 08, 2014 at 07:01:08AM -0800, Greg Wilson:

A make/intermediate/00-intro.md (65)

I think the robot story could use an intro line, so others don't start
off thinking “we have a robot?” ;).

I'd drop “back home” from “colleague back home”. We may spend a lot
of time with our lab partners, but usually don't live with them ;).

I'm amused by “which is usually just a plain text file in some
specialized format”. Maybe “which is a plain text file in a
specialized format”?

There's some fear-mongering in here (cross-platform portability issues
are rarely Make's fault ;), and in later files. I think some amount
of “don't feel bad if this doesn't make sense immediately, it's
complicated stuff” is good, but “these tools are ugly, but they're the
best we've got” and “you won't understand this cryptic syntax” do not
strike me as good marketing ;). If we don't think Make is a good
sell, we shouldn't be selling it at all.

A make/intermediate/01-basics.md (171)

Graphs of these dependency trees would be good. I don't really care
about the tooling. Dia is nice 1. So is TikZ/PFG 2 ;).
Someone's probably written a Makefile → graphviz script too.

Can we use data-1-*.dat instead of data-1-something.dat? Nobody
should be in the Make lesson without hitting the shell lesson first.
I missed the implied ${something} the first time through (and then
felt pretty stupid about it :p).

summary-1.dat only depends on stats.py indirectly via data-1-*.dat.
I'd express that in the Makefile instead of claiming the direct
dependency you do in the text here (although you address this
02-patterns.md and later).

You've got example Makefiles, but don't have the summary-1.dat
source for testing this out in stages. I think that would be nice,
but you'd need to pick a less hypothical example ;). As it stands,
there are few places for students to actively follow along.

Instead of saying “our comments should be more meaningful”, why don't
we just add a meaningful comment? Comments with the filename just
duplicate information already stored on the filesystem itself, and
folks may move/copy files without remembering to change the comments.
I suggest:

# rebuild figure-1.svg using sgr if summary-1.dat changes
figure-1.svg : summary-1.dat
        sgr -N -r summary-1.dat > figure-1.svg

Instead of “action”, the standard name for the body of the rule is the
“recipe” 3. For things like this (and the default rule, etc.), I'd
rather link into the GNU Make docs than to our own glossary. Less
work for us, and more detail and accuracy for our students. Where the
GNU Make docs are unclear, I'm sure they'd be interested in patches to
improve them ;).

You say Make will “will print an error message and stop” for circular
dependencies, but GNU Make actually drops circular dependencies with a
warning and tries to soldier on 4. For example:

$ cat Makefile
a: b
cat "$<" > "$@"

b: c
cat "$<" > "$@"

c: a
cat "$<" > "$@"
$ make
make: Circular c <- a dependency dropped.
cat "" > "c"
cat: : No such file or directory
make: *** [c] Error 1

Now that the first run created c (while dying), the second run
succeeds:

$ make
make: Circular c <- a dependency dropped.
cat "c" > "b"
cat "b" > "a"

A make/intermediate/02-patterns.md (180)

I don't use sticky notes to remember the automatic variables, I:

$ info make

And type i, followed by $@, followed by enter.

You can get the same effect via your web browser:

1. Search for “GNU make docs”, using my favorite search engine.
2. Click on the first link (to
   http://www.gnu.org/software/make/manual/make.html).
3. Click on “Name Index” (http://www.gnu.org/software/make/manual/make.html#Name-Index)
4. Click on “$@” (or whatever)

The benefit of the web approach is that you end up with handy URLs to
paste into your commit message where you explain anything that might
confuse future devs ;).

A make/intermediate/03-rules.md (135)

In the case where the figure-_.svg files are missing, you claim the
figure-_.svgprerequisite will expand to nothing, but it expands to
figure-*.svg and you'll get a:

make: *** No rule to make target `figure-*.sh'…

error.

make -r -d 1 … is a good way to get inside Make's head, if you're
having trouble understanding why it's not acting the way you think it
should. Repeatedly telling folks that Make is hard to debug doesn't
seem productive.

Instead of using false dependencies in false-dependencies.mk, I'd
prefer:

FIGURES = figure-1.svg figure-2.svg
FIGURE_DATA = $(patsubst figure-%.svg, summary-%.dat, $(FIGURES))

paper.pdf : paper.wdp $(FIGURES)
wdp2pdf $<

figure-%.svg : summary-%.dat
sgr -N -r $@ $^

summary-%.dat : data-%-*.dat
stats.py $@ $^

$(FIGURE_DATA) : stats.py

but you'd have to bump that into 04-macros.

A make/intermediate/04-macros.md (187)

Why do you prefer “macro” to “variable”? The GNU Make docs seem to
prefer “variable”, although they say “In some other versions of make,
variables are called macros” 5. Since we're focusing on GNU Make,
I'd rather stick to their conventions.

No space around the equals sign in your STYLE_DIR=c:/papers/ (and
later variable definitions)?

The GNU Make docs prefer the $() syntax to ${…} in
their examples, so I think we should follow their lead 6.

Also, for STYLE_DIR, I'd probably just override it on the command line
7 instead of writing a separate config.mk:

$ make -f with-lots-of-macros.mk STYLE_DIR=c:/papers/

If this was going to happen a lot, I'd write a wrapper script around
make that set my overrides. If other people need to reproduce my
paper, and have their style files outside the standard
/usr/lib/styles, they could read my docs pointing out the easily
personalized STYLE_DIR variable, and override it from the command line
themselves.

This sort of platform adaptation is really a problem that needs to be
resolved at the packaging/linking level (e.g. with a system-wide
wdp2pdf lookup path into which users installed their euphoric styles),
and most systems (C, Python, LaTeX, …) already handle that sort of
thing well out of the box. Jumping through hoops to support users
that refuse to follow their tool's installation conventions doesn't
seem to be a particularly productive line of development. I'd much
rather use that time educating people about how they could be making
both of our lives easier by sticking to those conventions.

[ethanwhite]

On Mon, Jan 20, 2014 at 12:13 AM, W. Trevor King
[email protected] wrote:

Can we split this into separate Make, hash, and Bash PRs? 4010 lines
of insertions are going to be a lot to review in a single, unthreaded
channel ;). I'll go through the Make stuff now; just let me know
where to push the comments.

The hash stuff hash already been moved out into IPython notebooks and
is on my plate to integrate in Intermediate Python #3, so that can
definitely be removed.

[gvwilson]

Replaced by #263 and #264.