Cilkscale documentation

[ailiop]

This is an initial (and currently partial) draft of a User's guide page for Cilkscale.

[netlify]

✅ Deploy Preview for sage-licorice-6da44d ready!

Name	Link
🔨 Latest commit	4c220c7
🔍 Latest deploy log	https://app.netlify.com/sites/sage-licorice-6da44d/deploys/6329fa580f12fb00098af0c4
😎 Deploy Preview	https://deploy-preview-148--sage-licorice-6da44d.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[ailiop]

I believe this is at a point that it could really benefit from an extra set of eyes.

Anyone willing to review? @behoppe @timkaler @cleiserson @neboat

[behoppe]

@ailiop I think this an outstanding starting point. For now I will stick to very high-level feedback. I recommend reconsidering the arrangement of content with an eye to several different goals that seem to be in play:

• How to use Cilkscale.
• Understanding work, span, and parallelism.
• Understanding sample_qsort.

IMO "how to use Cilkscale" is the top goal, and so we want to prioritize every arrangement of content in service to this goal. I think this means pushing sample_qsort down, so that people read about that only after they've made progress understanding how to use Cilkscale.

Understanding work, span, and parallelism is intertwined with how to use Cilkscale. No way around that. But still, I recommend we establish other places that we can leverage for this understanding. For example, "What the #$!@# is parallelism" will be a useful blog post (with a new title). Maybe also a tutorial on work-span analysis? And the glossary of course. This user's guide article needs a clear sense of partnership with those (not-yet-ready) parts of the website, so that here we focus on how to use Cilkscale.

On a more detailed stylistic note, I like how the "fine grained analysis" section breaks code into little pieces that are interspersed with commentary. I recommend more of that. For example, instead of one big block of bash that forces me to parse and scroll up and down etc...

$ /opt/opencilk/bin/clang++ qsort.cpp -fopencilk -fcilktool=cilkscale -O3 -o qsort_cilkscale
$ ./qsort_cilkscale 100000000
Sorting 100000000 random integers
Sort succeeded
Time(sample_qsort) = 2.0279 sec
tag,work (seconds),span (seconds),parallelism,burdened_span (seconds),burdened_parallelism
,24.2875,2.96416,8.19373,2.96449,8.19283

We might break it up and exclude things that need no further commentary.

$ /opt/opencilk/bin/clang++ qsort.cpp -fopencilk -fcilktool=cilkscale -O3 -o qsort_cilkscale
$ ./qsort_cilkscale 100000000

(Commentary here about two new lines before resuming the block of code...)

tag,work (seconds),span (seconds),parallelism,burdened_span (seconds),burdened_parallelism
,24.2875,2.96416,8.19373,2.96449,8.19283

The goal is to allow the reader to parse the story as he goes, without having to back up and do comparisons between two code blocks that are not even showing simultaneously on the screen.

Finally, I see code indents set at 4. We are wanting a standard for this. I personally advocate for 2, or the smallest possible tab we can all agree on. My perspective is that this code is not for editing. It is for display in locations that are sometimes very space-constrained. Wide tabs are an extravagant use of that precious space IMO. (See the homepage for an example.)

I hope this is helpful. Great job!

[behoppe]

PS: I also think that showing the Cilkscale plots near the top (and/or in a tutorial) would help, saving the part about how to create the plots for the end.

[ailiop]

@behoppe Thank you for your comments! I pushed a set of updates to address (most of) them.

[ailiop]

I have added a new page for Cilkscale in the Reference category. While I have not yet made any updates to the User's guide or Tutorial components of the overall Cilkscale documentation, I think the new Cilkscale reference manual page is at a good place for review.

There are 4 places where I think the Cilkscale reference page needs some changes, either in content or in related website infrastructure. These are marked in "TODO" or "BUG" alert boxes in the page; I also summarize them below:

1. The CSS(?) styling for reference pages seems to be different than for the rest of the website. Specifically, notice the way that Markdown sections (## Section more so than ### Subsection) and bullet points are rendered. More problematically, the rendering of the shell-session code block that shows the cilkscale.py script output towards the end of the page gets mangled (the exact same code block gets rendered fine in, say, a page in the User's guide directory).
2. It'd be great to have some infrastructure for typesetting citations from BibTeX entries.
3. Related to (2), I added a couple of references that discuss parallel performance and scalability analysis concepts and measures. At the moment, I added DOI links to them, but those lead to the paywalled ACM website. It'd be great to have publicly accessible links, instead.
4. Once a tutorial page for Cilkscale is ready, it should be linked by the reference page.
5. It may be better if the Cilkscale visualizations at the end were generated on a compute server rather than on my laptop. I will give that a try later.

[ailiop]

@behoppe The Netlify deployment checks are currently failing, but I think the failures are spurious. Let me explain:

• Before committing the new Cilkscale reference page, I first rebased this PR branch onto the main branch of OpenCilk/www.opencilk.org. (I did that so I could check how certain code snippets looked after the recent change to code highlighting style in main.)
• I see no conflicts with the main branch. Moreover, when I check the failed-check logs, I get that the error actually comes from the Introduction to Cilk programming page: specifically the error message tells me that cilkc is an unrecognized language.
• Based on the above, I suspect that the version of the Netlify deployment engine that is used within the PR is "frozen" at the time that the PR is created. That is, the checks are done not with the current (rebased) state of the website, but with an outdated one.

Is it possible to somehow "refresh" the checks engine? If not — or if it's not easy anyway — I am happy to close this PR and create a new one, which I think will fix the problem.

[behoppe]

@ailiop it's easy to refresh the checks engine, and I just did so, and I got another failure. I think I agree with your assessment. I've seen this problem recently, and it had to do with outdated Node packages (specifically prismjs), which are not normally updated with a site build. So I "cleared cache" and redeployed, and the error I see is something about ".eleventy.js". I will look into this after lunch.

[behoppe]

@ailiop I am investigating per https://docs.netlify.com/configure-builds/troubleshooting-tips/. This PR/branch builds fine in my local system, but my local system is using older packages than Netlify. So I am going to update my local system and check again.

[behoppe]

@ailiop the deploy preview is built now. After I updated Node and npm on my local system, I set environment variables on Netlify to fix the same versions I observed locally:

NODE_VERSION 16.17.0
NPM_VERSION 8.15.0

Then I "cleared cache" and redeployed, and it worked.

[ailiop]

Thank you @behoppe! I can now see the new Cilkscale reference page in the deploy preview. The checks still come up as "failed" from my end on the GitHub PR page but I won't worry about that.

[ailiop]

@behoppe I have finished a pass over the Cilkscale documentation and split it into two parts: a user's guide and a reference manual.

1. Cilkscale user's guide (preview)
2. Cilkscale reference manual (preview)

There is some tutorial-like content at the end of the user's guide (small section on reasoning about scalability). I think this would work better in a separate tutorial page but I don't have time to work on one at the moment. Until I or anyone else gets to that, I think it's still helpful to have something in the User's guide that briefly address the "what next?" question that follows naturally after one gets a bunch of results with Cilkscale.

Anyway, this is ready for review!

[ailiop]

@behoppe There are a couple of styling issues in the Cilkscale reference manual page (preview):

1. In the example code-box in the cilkscale.py section, the script output gets mangled. (Compare it to the code boxes in the relevant section of the User's guide.) This only happens if the ```code-session [...] ``` box is within an Eleventy alert box. The mangling seems to be triggered by the lines that start with >> in the cilkscale.py output.
2. The section headers are not styled consistently with the rest of the website.

Perhaps related to (1), alert boxes seem to have some of their own styling rules. Another notable difference is that there is no extra vertical space between paragraphs in alert boxes.

I am hoping these issues are things you could fix, hopefully with little effort.

[behoppe]

Thanks @ailiop. This looks great. Sorry for my delayed review. I will use multiple comments. After reviewing the user's guide, I tested the instructions, and I could not get qsort.cpp to compile on my Ubuntu 20.04 box when using -fcilktool (both benchmarking and work/span). It did compile without -fcilktool. Error dump is below.

bruce@bruce-ThinkPad-X1-Yoga-1st:~/OpenCilk/OpenCilk-2.0.0-x86_64-Linux-Ubuntu-20.04$ ./bin/clang++ qsort.cpp -fopencilk -fcilktool=cilkscale-benchmark -O3 -o qsort_cs_bench
Sync   sync within %syncreg, label %sync.continuehas unwind
; Function Attrs: mustprogress uwtable
define linkonce_odr dso_local void @_Z12sample_qsortIlEvPT_S1_(i64* noundef %begin, i64* noundef %end) local_unnamed_addr #5 comdat personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
entry:
  %syncreg = tail call token @llvm.syncregion.start()
  %sub.ptr.lhs.cast = ptrtoint i64* %end to i64
  %sub.ptr.rhs.cast = ptrtoint i64* %begin to i64
  %sub.ptr.sub = sub i64 %sub.ptr.lhs.cast, %sub.ptr.rhs.cast
  %cmp = icmp slt i64 %sub.ptr.sub, 256
  br i1 %cmp, label %if.then, label %if.else

if.then:                                          ; preds = %entry
  %cmp.not.i.i = icmp eq i64* %begin, %end
  br i1 %cmp.not.i.i, label %if.end, label %if.then.i.i

if.then.i.i:                                      ; preds = %if.then
  %sub.ptr.div.i.i = ashr exact i64 %sub.ptr.sub, 3
  %0 = tail call i64 @llvm.ctlz.i64(i64 %sub.ptr.div.i.i, i1 true) #16, !range !23
  %sub.i.i.i = shl nuw nsw i64 %0, 1
  %mul.i.i = xor i64 %sub.i.i.i, 126
  tail call void @_ZSt16__introsort_loopIPllN9__gnu_cxx5__ops15_Iter_less_iterEEvT_S4_T0_T1_(i64* noundef %begin, i64* noundef %end, i64 noundef %mul.i.i)
  tail call void @_ZSt22__final_insertion_sortIPlN9__gnu_cxx5__ops15_Iter_less_iterEEvT_S4_T0_(i64* noundef %begin, i64* noundef %end)
  br label %if.end

if.else:                                          ; preds = %entry
  %incdec.ptr = getelementptr inbounds i64, i64* %end, i64 -1
  %1 = load i64, i64* %incdec.ptr, align 8, !tbaa !3
  %cmp3942.i.i = icmp eq i64* %incdec.ptr, %begin
  br i1 %cmp3942.i.i, label %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, label %if.else.lr.ph.i.i

if.else.lr.ph.i.i:                                ; preds = %if.else, %while.end18.i.i
  %__last.addr.044.i.i = phi i64* [ %__last.addr.1.i.i, %while.end18.i.i ], [ %incdec.ptr, %if.else ]
  %__first.addr.043.i.i = phi i64* [ %incdec.ptr19.i.i, %while.end18.i.i ], [ %begin, %if.else ]
  br label %if.else.i.i

if.else.i.i:                                      ; preds = %if.then3.i.i, %if.else.lr.ph.i.i
  %__first.addr.140.i.i = phi i64* [ %__first.addr.043.i.i, %if.else.lr.ph.i.i ], [ %incdec.ptr.i.i, %if.then3.i.i ]
  %2 = load i64, i64* %__first.addr.140.i.i, align 8, !tbaa !3
  %cmp.i.i.i = icmp slt i64 %2, %1
  br i1 %cmp.i.i.i, label %if.then3.i.i, label %while.body8.i.i.preheader

while.body8.i.i.preheader:                        ; preds = %if.else.i.i
  br label %while.body8.i.i

if.then3.i.i:                                     ; preds = %if.else.i.i
  %incdec.ptr.i.i = getelementptr inbounds i64, i64* %__first.addr.140.i.i, i64 1
  %cmp.i.i = icmp eq i64* %incdec.ptr.i.i, %__last.addr.044.i.i
  br i1 %cmp.i.i, label %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, label %if.else.i.i, !llvm.loop !24

while.body8.i.i:                                  ; preds = %while.body8.i.i.preheader, %if.else11.i.i
  %__last.addr.0.pn.i.i = phi i64* [ %__last.addr.1.i.i, %if.else11.i.i ], [ %__last.addr.044.i.i, %while.body8.i.i.preheader ]
  %__last.addr.1.i.i = getelementptr inbounds i64, i64* %__last.addr.0.pn.i.i, i64 -1
  %cmp9.i.i = icmp eq i64* %__first.addr.140.i.i, %__last.addr.1.i.i
  br i1 %cmp9.i.i, label %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, label %if.else11.i.i

if.else11.i.i:                                    ; preds = %while.body8.i.i
  %3 = load i64, i64* %__last.addr.1.i.i, align 8, !tbaa !3
  %cmp.i32.i.i = icmp slt i64 %3, %1
  br i1 %cmp.i32.i.i, label %while.end18.i.i, label %while.body8.i.i, !llvm.loop !25

while.end18.i.i:                                  ; preds = %if.else11.i.i
  store i64 %3, i64* %__first.addr.140.i.i, align 8, !tbaa !3
  store i64 %2, i64* %__last.addr.1.i.i, align 8, !tbaa !3
  %incdec.ptr19.i.i = getelementptr inbounds i64, i64* %__first.addr.140.i.i, i64 1
  %cmp39.i.i = icmp eq i64* %incdec.ptr19.i.i, %__last.addr.1.i.i
  br i1 %cmp39.i.i, label %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, label %if.else.lr.ph.i.i, !llvm.loop !26

_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit: ; preds = %while.end18.i.i, %if.then3.i.i, %while.body8.i.i, %if.else
  %__first.addr.136.i.i = phi i64* [ %begin, %if.else ], [ %__first.addr.140.i.i, %while.body8.i.i ], [ %__last.addr.044.i.i, %if.then3.i.i ], [ %__last.addr.1.i.i, %while.end18.i.i ]
  %4 = load i64, i64* %incdec.ptr, align 8, !tbaa !3
  %5 = load i64, i64* %__first.addr.136.i.i, align 8, !tbaa !3
  store i64 %5, i64* %incdec.ptr, align 8, !tbaa !3
  store i64 %4, i64* %__first.addr.136.i.i, align 8, !tbaa !3
  %6 = tail call token @llvm.tapir.runtime.start()
  detach within %syncreg, label %det.achd, label %det.cont unwind label %lpad2

det.achd:                                         ; preds = %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit
  invoke void @_Z12sample_qsortIlEvPT_S1_(i64* noundef %begin, i64* noundef nonnull %__first.addr.136.i.i)
          to label %invoke.cont unwind label %lpad

invoke.cont:                                      ; preds = %det.achd
  reattach within %syncreg, label %det.cont

det.cont:                                         ; preds = %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, %invoke.cont
  %incdec.ptr12 = getelementptr inbounds i64, i64* %__first.addr.136.i.i, i64 1
  invoke void @_Z12sample_qsortIlEvPT_S1_(i64* noundef nonnull %incdec.ptr12, i64* noundef nonnull %end)
          to label %invoke.cont14 unwind label %lpad9.tfsplit.split-lp.csi-split

invoke.cont14:                                    ; preds = %det.cont
  sync within %syncreg, label %sync.continue

sync.continue:                                    ; preds = %invoke.cont14
  invoke void @llvm.sync.unwind(token %syncreg)
          to label %invoke.cont15 unwind label %lpad9.tfsplit.split-lp.csi-split-lp

invoke.cont15:                                    ; preds = %sync.continue
  tail call void @llvm.tapir.runtime.end(token %6)
  br label %if.end

lpad:                                             ; preds = %det.achd
  %7 = landingpad { i8*, i32 }
          cleanup
  invoke void @llvm.detached.rethrow.sl_p0i8i32s(token %syncreg, { i8*, i32 } %7)
          to label %unreachable unwind label %lpad2

lpad2:                                            ; preds = %_ZSt9partitionIPlZ12sample_qsortIlEvPT_S3_EUllE_ES2_S2_S2_T0_.exit, %lpad
  %8 = landingpad { i8*, i32 }
          cleanup
  br label %lpad9

lpad9.tfsplit.split-lp.csi-split-lp:              ; preds = %sync.continue
  %lpad.csi-split-lp = landingpad { i8*, i32 }
          cleanup
  br label %lpad9.tfsplit.split-lp

lpad9.tfsplit.split-lp.csi-split:                 ; preds = %det.cont
  %lpad.csi-split = landingpad { i8*, i32 }
          cleanup
  br label %lpad9.tfsplit.split-lp

lpad9.tfsplit.split-lp:                           ; preds = %lpad9.tfsplit.split-lp.csi-split, %lpad9.tfsplit.split-lp.csi-split-lp
  %lpad.phi141 = phi { i8*, i32 } [ %lpad.csi-split-lp, %lpad9.tfsplit.split-lp.csi-split-lp ], [ %lpad.csi-split, %lpad9.tfsplit.split-lp.csi-split ]
  br label %lpad9

lpad9:                                            ; preds = %lpad9.tfsplit.split-lp, %lpad2
  %lpad.phi = phi { i8*, i32 } [ %8, %lpad2 ], [ %lpad.phi141, %lpad9.tfsplit.split-lp ]
  tail call void @llvm.tapir.runtime.end(token %6)
  resume { i8*, i32 } %lpad.phi

if.end:                                           ; preds = %if.then.i.i, %if.then, %invoke.cont15
  ret void

unreachable:                                      ; preds = %lpad
  unreachable
}

[ailiop]

@behoppe I believe that this is a spurious printout that does not actually correspond to a compilation error. That is, if you look at the files in your directory after compiling with -fcilktool=cilkscale (or similar), you should see the instrumented binary.

This is a known issue with OpenCilk 2.0 and Cilk/C++ (OpenCilk/opencilk-project#129), and has been resolved with OpenCilk 2.0.1.

I think if you either install OpenCilk 2.0.1 or simply ignore these printouts you should be able to go through the Cilkscale user's guide.

[behoppe]

@ailiop thanks for your formatting comments. The headers are an easy fix, which I included with my latest push 6f92f27 to this branch. I am less sure about fixing the jumbled shell-code that's inside alert. I think there's a problem with how alert renders markdown, and I hope that the solution will be found here. That blog post takes a much more thorough approach to creating shortcodes than I have to date.

On to the more substantive comments on the user's guide. (I have not yet reviewed the reference, except its title, as noted below.) This is really coming along!

• I shortened most of the headers, trying to make them less about Cilkscale and more about naming things that people would recognize as tasks they want to do.
• I put qsort.cpp into _includes/code/qsort.cpp. Please see include program code in markdown #156 for context and let me know if you agree that _includes/code is a good standard location for code files. I have just linked include program code in markdown #156 to this PR, assuming that we'll address include program code in markdown #156 within this PR now.
• I tried to elevate the definitions of "benchmarking" and "work/span" modes. The only downside here is that "benchmarking" seems to assume a different meaning in the last section, which I find confusing. My edit makes that problem more obvious.
• I trimmed some places that explained how Cilkscale worked, to try and focus on the user tasks being solved.
• I changed most (not all) LaTeX to plain text, except for the "P" part. I find our LaTeX font quite jarring, especially if we're just writing numbers like "2" and "10." I sense a style guide topic coming on... or an issue "please change mathjax font".
• I got genuinely confused by the section about benchmarking and visualizing. I have not yet used Cilkscale, and this section did not really help me understand why I would want to use this python script. "Benchmarking" takes a new meaning here (different than "benchmarking mode"), and that's a problem IMO. I will not be convinced that I understand this section until I actually do it.
• "Cilkscale reference manual" sounds too big IMO, and I suggest something like "Cilkscale reference guide" when linking to it from other pages.

I am torn about this line:

The steps for compiling and running the Cilkscale-instrumented binary are the same whether or not your program uses the Cilkscale API.
That is another place I got stuck and confused for some time. The problem is that there is no user task being addressed here. It's just information, sitting there and asking me to leave the task at hand. I honestly started scrolling around trying to interpret what this really meant. All that to say -- I recommend removing this line, and lines like it. I am torn only because after I spent a few minutes being confused, then I figured it out and thought it was a nice comment.

I have not yet made it to the plots and "insights" (very last section).

[ailiop]

@behoppe Thank you for these comments! I will go over them in more detail tomorrow, but let me offer a quick reply to some of them in the meantime:

I am torn about this line: "The steps for compiling and running the Cilkscale-instrumented binary are the same whether or not your program uses the Cilkscale API."

I can see how that might be confusing. I think it'd be clearer if a similar comment was moved towards the end of the Cilkscale API section and in a Note box. I can play around with that tomorrow.

"Cilkscale reference manual" sounds too big IMO, and I suggest something like "Cilkscale reference guide" when linking to it from other pages.

"Cilkscale reference guide" is actually the name I used initially but thought it has too much overlap with "Cilkscale user's guide". Maybe then it's better to just say "Cilkscale reference page"?

I got genuinely confused by the section about benchmarking and visualizing. I have not yet used Cilkscale, and this section did not really help me understand why I would want to use this python script. "Benchmarking" takes a new meaning here (different than "benchmarking mode"), and that's a problem IMO. I will not be convinced that I understand this section until I actually do it.

I'll have to think some more about why the text gave rise to this confusion. The notion of "benchmarking" is generally the same throughout: it refers to timing a (sub-)computation. The cilkscale.py script basically does 3 (related) things:

1. It automates the process of running (i) the work/span-analysis-mode instrumented binary and (ii) the benchmark-mode instrumented binary on different numbers of processors.
2. It combines the Cilkscale reports from each of the above runs into a single CSV table.
3. It visualizes the reported measurements to aid human inspection.

In other words, the script does not produce any new analysis/benchmarking measurements that were not covered before. But it makes it way more convenient to collect and view results.

[ailiop]

@behoppe I made some revisions to the user's guide and a couple of minor tweaks to the reference page. My revisions touch most parts of the user's guide but are more substantial in the section that addresses the cilkscale.py script.

I also noticed that the Cilkscale plots were inconsistent with the tabular reports (I think the plots were actually obtained with a different base-case size) so I regenerated them and edited the tutorial-like epilogue accordingly.

Incidentally, I generally like the section title changes you suggested. In the case of the last section, I felt like "Insights" was vague and attempted an alternative: "Discussion: diagnosing performance limitations". My goal was two-fold: (1) make it clear that this section different from the rest (a discussion as opposed to a how-to), and (2) clarify the specific task that is discussed. I am not sure I would want to stick with the current title, but I feel it mostly accomplishes these goals. Alternative suggestions are welcome!

[behoppe]

@ailiop I reviewed the reference and -- pending my edits fcbcc0d and some thoughts on "processors" -- I think it's ready to publish. Great job.

• I made the measurement definitions more consistent with the new glossary.
• I took the start, end, elapsed variables from the reference example and applied them to the user's guide.
• I resized the images to 1000px wide and used "100%" width in the img shortcode.

Processors
IMO we need to look carefully at the definitions of the CPU_COUNTS argument, the x-axis of the Cilkscale plots, and the top-level definitions of things like work and span. We are going to overload the term "processor" no matter what IMO, but I think we could do a better job of naming what exactly Cilkscale is automating. It is running the program on a varying number of _______. What is that word in the blank? We number it with P and CPU_COUNT, and we refer to it as core and processor -- then it gets called "worker" in the x-axis of the plot.

[ailiop]

@behoppe Thank you. I like your edits and can certainly see your point regarding the terminology for processor/core/worker, especially in the context of the cilkscale.py script's -cpus option.

I made an attempt to clarify what cilkscale.py does when benchmarking on multiple numbers of cores (commit e2227dc). Let me know if you think it is still confusing.

I also noticed that I had messed up the Cilkscale visualization plotter base version when I changed the plot face color and added grid lines. I regenerated the plots with the correct (up-to-date) version and updated the reports and related discussion for consistency.

[behoppe]

Thanks, @ailiop. Looks great. I made very minor adjustments to image size and punctuation. Merging with main branch!