#6 - Adds blog post content for Elixir Parallel Letter Frequency walkthrough #14
Conversation
|
This is super-awesome :) One general comment: How about removing the "final code" bit at the end and linking to your solution on Exercism instead? I'll try and find some time to read through this more in detail this week! |
|
@iHiD you're absolutely right, I completely forgot about linking to my solution on Exercism! I think it might be better to completely remove that section from the end and add a link to my solution on Exercism right at the start of the post. Thanks for the feedback, I'll update the PR. |
|
Great. @kytrinyx is going to give some proper feedback to this over the coming days :) |
|
Hi all, I have an unlisted video on YouTube for this tutorial as well: https://youtu.be/inhcxfzVWWM Happy to make any changes you all think could improve the video. Thanks! |
There was a problem hiding this comment.
Hi @percygrunwald,
Thank you so much for diving into this, and I apologize for the delay in digging into it.
I've read through this and discussed a bit with @iHiD to try to wrap my head around how I think we best can tackle this one.
The first thing that strikes me is that I love what you're doing with the videos. I think the video format is kind of perfect for code walkthroughs, as they provide a lot of context—the screen, the chatter and commentary along the way. It's an immersive experience in a way that a blog post cannot be.
For the blog post, my primary thought is that it shouldn't try to do what the video is already doing so well. I wonder if we could take one interesting aspect and do a deep dive or deeper exploration, or discuss the implications of it, rather than doing a full walkthrough.
That might be a great teaser to link to (or embed, if we figure out how to pull that off) your video for your walkthrough of the same exercise.
I would leave this up to you to suggest where you'd prefer to take it, but I find this bit really tantalizing:
The concurrent version of the code ran around 80% faster on my system for certain workloads, but in some cases was over 3x slower than the original code. As the benchmarks later in the post demonstrate, depending on the situation, adding concurrency can actually reduce performance while also increasing the complexity of your code.
This trade-off is something worth agonizing over just a bit :) It has all sorts of potential drama, and there's no simple answer.
(For the record, I think that your video should have all the usual branding that it always does, even if we embed it... it's your video).
| @@ -0,0 +1,605 @@ | |||
| In this post I'll walk you through how I solved the [Parallel Letter Frequency problem](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). This is a medium difficulty problem that I found really interesting to solve, and it's a great opportunity to show how you can add concurrency to your [Elixir](https://elixir-lang.org/) code with the [`Task` module](https://hexdocs.pm/elixir/Task.html). | |||
|
|
|||
|  | |||
There was a problem hiding this comment.
@iHiD Do we want to host images ourselves? If so, how and where?
No problem at all! I'm really impressed that you all are taking a lot of time out of your day to run and promote Exercism.
I was a bit concerned about this aspect as well, so I tried to add some intrigue with the "does concurrency actually make your code faster" aspect. One thing I could do is boil down the article to have just the section exploring how to benchmark Elixir code and the results of the benchmarking, along with my advice about how to think about the clean code/performance tradeoff. I could edit the video to contain only the walkthrough, but not the benchmarking and say "if you want to see how I benchmarked the code and the results, please check out the article on the Exercism blog". Not sure how many people this would drive to the blog post (your audience is much bigger than mine), but it's something that might add some intrigue. My only concern about this approach is that the blog post then becomes something you'd likely find on a general Elixir blog, since the article content excluding the video would be less about Exercism. The other thing I could do is just take the key hints that might prevent someone from solving this problem and combine it with the benchmarking procedure + results, namely:
This way people could get the full walkthrough by watching the video, there's some Exercism-specific hints in the text content as well as some intriguing benchmark results + advice in the article. I'll start playing around with this idea and update the PR with a more chopped down version of the article. Let me know what you think about the suggestions. |
…ncy walkthrough
…ed solution on Exercism at the start of the post
percygrunwald
force-pushed
the
6-elixir-parallel-letter-frequency-walkthrough
branch
from
756e5b3 to
12eace3
Compare
|
@kytrinyx I made the changes I suggested in my comment above. The article is shorter and more engaging in my opinion, and there is less overlap with what the video is doing. Let me know if you have any feedback whenever you have time to review the latest version. Thanks! |
|
@percygrunwald I am so sorry I dropped the ball. I will review this again in the next day and get back to you! |
|
@percygrunwald This reads so much better! My gut reaction as I was reading through it was that we actually have three articles in one, and I'm wondering if we might post it as three separate ones, all of which cross-link to each other, and all of which link to your walk-through. The three articles would be:
@iHiD I'd love your thoughts on this before I send Percy on a wild rabbit chase. |
|
So a three-part series about this task that covers those three topics. That makes sense to me. I don't think it'd involve much extra work - just splitting the post and top and tailing a "Next time we'll talk about X" and "Last time we discussed Y". We could then publish them either as a set of three, or one-per-week. I like the idea. What do you think @percygrunwald? |
|
Thanks @percygrunwald ! |
|
@percygrunwald Fantastic! I'll take a look at these in the next few days. |
|
I like this direction! Ok, here's what I'd like to suggest. I'd like each blog post to stand alone, with a proper intro/conclusion, and that we'd post them one per week. I'd like to start off the first one without announcing that it's a series (so no "part 1" in the title, and no "coming up next" in the conclusion). Once we post the second one, we can cross-link the first to the second, and vice versa, and the same when we post the third. For titles, I'm thinking:
I think we could have the same introduction section for all three exercises. Here's something that I adapted from what you already have here:
Then each post introduces the lesson that that post will talk about, without talking about the other lessons. For the conclusion, I think it would make sense to echo the idea that "synthetic problems can teach useful things about real-world code", and we could grab this bit that you said in one of the posts:
And then say something that lets us link to your video walkthrough. What do you think, @percygrunwald? |
|
@kytrinyx, I'm fine with those suggestions, I'll get to work implementing them and make a new commit for your review. |
|
@kytrinyx I've made changes based on your feedback. I kept the file names the same so it's easier to see what changes I made. Let me know what you think. |
There was a problem hiding this comment.
I think this is the right split. Now there are two main things that I think we need to get right before this is ready for the final edit/polish for grammar etc:
The first thing is that we need each blog post to read well even if you don't ever read the two other blog posts. I think the main thing here is to consider the introduction, transition from introduction to the main content that you've written, and the conclusion. For each post, consider who is the main target for the reader, and what they know/don't know (in particular: what questions would they have going into this, what questions would come up along the way, and what real-world problems will be easier to solve with this in their toolbox?)
The second thing is that I'd like the post to be mostly about the reader—while the source of the post is what you learned, the excitement for the reader is about their struggles and their learning. I think if we can phrase this to be more consistently about you (the reader) rather than me (the author), this will be more engaging. You already do this in a lot of places.
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty problem on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that I found really interesting to solve. You need to write a function that calculates the frequency of letters in a list of strings containing _non-English characters_ with a user-controlled number of workers that do the calculation _concurrently_: | ||
| Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these synthetic problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real world problems more efficiently or in a more expressive way. | ||
|
|
||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). It asks you to write a function that calculates the frequency of letters in a list of strings, and to do so concurrently. This unpacks a surprising number of interesting lessons. |
There was a problem hiding this comment.
We need a transition here to introduce one of those lessons as Unicode, and then a transition from there to explain why the reader is looking at a code example :)
| @@ -1,6 +1,8 @@ | |||
| # "Solving Parallel Letter Frequency in Elixir - Part I: Unicode Matching in Elixir" | |||
| # 1. What Parallel Letter Frequency can teach you about Unicode matching in Elixir | |||
There was a problem hiding this comment.
Let's get rid of the number here, as it won't make sense outside of the context of a series.
| @@ -1,6 +1,8 @@ | |||
| # "Solving Parallel Letter Frequency in Elixir - Part II: Adding Concurrency with `Task`" | |||
| # 2. What Parallel Letter Frequency can teach you about concurrency with Elixir’s `Task` module | |||
| In `Part I` I showed how I applied Unicode matching in Elixir to help me solve the first part of [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency), a medium difficulty problem on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). | ||
| Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these synthetic problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real world problems more efficiently or in a more expressive way. | ||
|
|
||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). It asks you to write a function that calculates the frequency of letters in a list of strings, and to do so concurrently. This unpacks a surprising number of interesting lessons. |
There was a problem hiding this comment.
Here too we need a transition from the intro, in order to introduce the problem that this blog post is talking about.
| draft: true | ||
| --- | ||
| In `Part I` I showed how I applied Unicode matching in Elixir to help me solve the first part of [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency), a medium difficulty problem on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). In `Part II` I tackled the next part of the problem -- adding concurrency -- and explored how you can use Elixir's `Task` module to do this. | ||
| # 3. What Parallel Letter Frequency can teach you about benchmarking in Elixir |
|
|
||
| The concurrent code in my solution ran around **80% faster** on my system for certain workloads, but in some cases was over **3x slower than the original code**. As the benchmarks later in the post demonstrate, depending on the situation, adding concurrency can actually **reduce performance while also increasing the complexity of your code**. | ||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir). It asks you to write a function that calculates the frequency of letters in a list of strings, and to do so concurrently. This unpacks a surprising number of interesting lessons. |
There was a problem hiding this comment.
Here, too, we need a transition from the intro that introduces what this particular blog post is about.
|
Thanks for the feedback @kytrinyx, I'll make some edits today. |
…changes framing to be less "me" and more "you", improves conclusions with examples of practical applications of the lessons from the exercise
|
@kytrinyx, added more improvements to the articles:
I've also made some general improvements to the article content such as making the code examples more standalone by including more code in each post. |
There was a problem hiding this comment.
@percygrunwald I reviewed the unicode article (will review the others after work).
I really love this article now! I have a couple of suggestions, but I think that the basic shape and story are exactly right.
| @@ -0,0 +1,129 @@ | |||
| Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these synthetic problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real world problems more efficiently or in a more expressive way. | |||
|
|
|||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that unpacks a surprising number of interesting lessons. One of those lessons is how to determine whether a character is a letter in non-English language, which has clear benefits for anyone writing a multilingual application or an application in a language other than English. | |||
There was a problem hiding this comment.
This is good!
determine whether a character is a letter in non-English language
I think we need to expand on this. It will check whether a character is a letter in English, too.
Since you talk about unicode and what it covers below, maybe we can work around this with something like:
Parallel Letter Frequency is a medium difficulty exercise on Exercism's Elixir Track that unpacks a surprising number of interesting lessons. The fundamental challenge that the exercise presents is to break a sentence up into individual letters, and count how often each letter occurs. It complicates matters ever so slightly for native English speakers who often get away with the bare minimum when it comes to things like alphabets, as some of the test cases include sentences in languages other than English.
Or something like that, rewritten to feel like it would be natural for you to say it :)
There was a problem hiding this comment.
Made changes to this section based on your suggestions.
|
|
||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that unpacks a surprising number of interesting lessons. One of those lessons is how to determine whether a character is a letter in non-English language, which has clear benefits for anyone writing a multilingual application or an application in a language other than English. | ||
|
|
||
| This exercise requires you to implement a `Frequency.frequency/2` function that determines letter frequency in a list of strings, with the added challenge of counting letters in any language: |
There was a problem hiding this comment.
If the earlier paragraph is tweaked to introduce the added challenge, then we can remove that part here.
There was a problem hiding this comment.
I kept a small preamble to introduce the the code snippet reminding/showing the reader what needs to be implemented, but removed the "added challenge" part.
|
|
||
| It turns out that the `u` modifier -- and [specifically the `\p` pattern](https://www.regular-expressions.info/unicode.html) -- is a really elegant solution. The `\p` pattern lets you match a grapheme (another name for a single Unicode character) in any of the [Unicode character categories](https://en.wikipedia.org/wiki/Unicode_character_property#General_Category). This not only includes specific categories like `Ll` ([Letter, lowercase](https://www.compart.com/en/unicode/category/Ll)) and `Sc` ([Symbol, currency](https://www.compart.com/en/unicode/category/Sc)), but also the parent categories like `L` (Letter) and `S` (Symbol). | ||
|
|
||
| You can match _any_ letter of _any_ case in _any_ [human language covered by Unicode](https://www.unicode.org/faq/basic_q.html) with the pattern `\p{L}` and easily [use it in Elixir regular expressions](https://www.toptechskills.com/elixir-phoenix-tutorials-courses/how-to-match-any-unicode-letter-with-regex-elixir/) to solve the problem: |
There was a problem hiding this comment.
While I suspected (but had to look up) whether Korean and Thai were included, I was surprised and delighted to discover that this includes Chinese characters, too. I didn't realize that. It could be worth making a bigger deal out of how extensive unicode is.
There was a problem hiding this comment.
Added some more examples demonstrating what exactly you can match, and a cheeky gif that shows how I felt when I learned about it.
|
|
||
| It turns out that matching non-English letters becomes pretty simple when you know about Unicode matching, and luckily for us it's a core feature in Elixir's `Regex` module. Prior to solving this Exercism problem I barely knew about this feature, but I would now consider it an indispensable part of my Elixir toolbox. | ||
|
|
||
| You could use this new tool in a number of ways, and a few that spring to my mind are more robust validation of passwords and usernames, or even for determining whether an input string is a valid currency string without needing to manually list [all possible currency symbols](https://www.compart.com/en/unicode/category/Sc): |
There was a problem hiding this comment.
😍
This leads me to consider whether or not I should start devising pass phrases in 한글... though good luck switching keyboards when not using your own computer, I suppose :-)
…cle. Fixes images.
|
@kytrinyx, great suggestions, thank you! I've pushed some updates. |
|
The unicode article is great! I'd like to reconsider the gif. I like the sentiment, but I find animated gifs to be an accessibility issue—I always have to either cover the moving part in order to read the article, or open it with a "readability" plugin. Would you put the unicode article in a PR by itself? That way we can do a final edit for polish, as well as work out the metadata and "marketing" blurb for it while I review the other two articles. |
|
No problem, I'll remove the gif and make a new PR. |
There was a problem hiding this comment.
@percygrunwald I've read through the concurrency article and left a couple of suggestions. I think that it's looking really good, and mostly needs a bit more context in a couple places, and maybe a transition or two.
| } | ||
| ``` | ||
|
|
||
| Let's explore how we can add concurrency to our solution using Elixir. |
There was a problem hiding this comment.
Here you say concurrency and below you say parallelise. These are somewhat related terms but don't mean the same thing, if I understand correctly.
Should we add some clarification about the difference?
There was a problem hiding this comment.
My intention was to use them synonymously, I'll read through the article again and see if I can change the wording to make it more clear.
There was a problem hiding this comment.
Thanks. Since they aren't synonyms I think it's important to be certain which this actually addresses.
My favorite talk on the subject is Rob Pike's "Concurrency is not parallelism" https://blog.golang.org/concurrency-is-not-parallelism
|
|
||
| Here's a diagram of how you could apply `Task.async_stream/5` to parallelize the letter frequency calculation: | ||
|
|
||
|  |
There was a problem hiding this comment.
I can't see this image (getting a 404 on the toptechskills.com site)
There was a problem hiding this comment.
Ah yes, my bad, sorry! I'll update the URL.
|
|
||
| The first option is simple and easy to understand, and lets you ensure the correct number of workers is used even for short strings. | ||
|
|
||
| ## Make the letter frequency calculation concurrent |
There was a problem hiding this comment.
In this section, I wonder if it would help to add a little bit of intro about what is going to happen.
We'll take an existing sequential implementation, and parallelize it by doing [x] (which you've described in the "in words" bullet list). If we do this I think it would make sense to put the diagram after the bullet list, rather than before it.
There was a problem hiding this comment.
I'll go over this one again and see if I can make it clearer.
|
|
||
| [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that unpacks a surprising number of interesting lessons. One of those lessons is how easy it is to add concurrency to Elixir code with core features alone, which has clear benefits for anyone writing applications whose performance could be improved by parallelizing tasks. | ||
|
|
||
| This exercise requires you to implement a `Frequency.frequency/2` function that determines letter frequency in a list of strings, with the added challenge of distributing the calculation to a number of worker processes, set with the `workers` argument: |
There was a problem hiding this comment.
I think we need something more in the intro about why concurrency and parallelism can be hard, what the pain points of it often are, or why people dread it. This will set the reader up to appreciate the rest of the article more, especially the bit at the end where you point out that the parallelized solution is as easy to read as the sequential one.
There was a problem hiding this comment.
Good call, I'll try to add something about this.
…ency vs parallelism plus some other improvements
|
@kytrinyx, I made some pretty major changes to the article, including a full section on parallelism vs concurrency, plus how Elixir makes achieving both simpler than some other popular languages/runtimes (JS/Node, Python). The length of the article has increased a lot, but I think it's probably valuable for readers. Let me know what you think, thanks! |
|
@kytrinyx, made some edits to the benchmarking article as well to make the concurrency/parallelism distinction more clear and uniform. |
|
@percygrunwald The updates to the concurrency article are fantastic. This really feels like a full-fledged, informative, topical article that will be relevant to lots of folks! I still need to read through the changes to the benchmarks article. |
|
Ok, the benchmark article is great now, too. It feels like it gets just the right amount of depth and detail, while still doing a great job of telling people why they should care. Let's get the first article shipped, then extract the second into a separate PR and polish it up (I'll take a quick run through for grammar/typos, that sort of thing), and then do the same with the third article once the second has shipped. |
Hi @iHiD @nicolechalmers, I've added my first iteration of the blog post in this commit.
I'd love your feedback on the post and I can make any improvements you'd like. The post is quite long, but I tried to add an extra level of interest by adding my benchmarks of the concurrent and non-concurrent code.
I didn't add anything to
blog.jsonsince I presume that you all will do that when you decide to release the blog post.I'm currently putting the final touches on the walkthrough video and I can share the unlisted video with you tomorrow for feedback.
Thanks!