Survey simulations and real data analyses

[pkimes]

As a first step to choosing a set of simulation and real data analyses to use for comparing methods, let's take a look at what was used for evaluating each method by their respective authors.

---

To make it easier, let's split this up so one person takes each of the following:
(add a comment if you can help!)

• add notes on IHW simulations here [@stephaniehicks]
• add notes on ASH simulations here [@pkimes]
• add notes on Boca-Leek simulations here [@areyesq89]
• add notes on IHW real data analyses here [@ayshwaryas]
• add notes on ASH real data analyses here [@tengmx]
• add notes on Boca-Leek real data analyses here [@kdkorthauer]

---

Some thoughts on what to include in the summary:

• settings (for simulations) / data type and source (for real data)
• evaluation metrics
• code availability (can we reproduce/modify/extend?)
• other suggestions?

---

Based on 2017/08/28 meeting (notes taken by @kdkorthauer):

try to include a summary in a couple of sentences: What is the big picture of the simulation scenarios and real data characteristics? Under what global characteristics is each method designed to excel?

[pkimes]

Maybe we can agree on a deadline, e.g. 2017/09/01 (next Friday)?

[stephaniehicks]

I presented IHW in journal club, so I'm happy to take on IHW. I'll write up a summary for both simulation study and real data analysis. If anyone else wanted to work on IHW too, feel free to join me.

BTW @pkimes after I click the checkboxes, do I need to click something else to assign that task to myself? Or do I only check the box after the task is complete?

[pkimes]

Thanks, @stephaniehicks!

[pkimes]

@stephaniehicks hmmm... I don't think GitHub has a way of assigning items in a list. 🤔

I was thinking we could keep track by adding our names to the items and checking the boxes when we've completed the tasks.

[kdkorthauer]

I can take on Boca-Leek since I presented it in Journal Club. Deadline next Friday might be tight for me, but I'll do my best.

[pkimes]

Thanks, @kdkorthauer!
If it's tight, don't feel like you need to commit to both simulations and real data (we have 6 people)!
If you want to just take one, we can find someone else to work on the other!
(and obviously, deadlines are somewhat flexible here 😉)

[kdkorthauer]

Got it. Maybe I'll start with real data, and then tackle simulations if no one else has nabbed it yet. Thanks for getting the ball rolling, @pkimes !

[areyesq89]

I would be happy to help @kdkorthauer with the simulations of the Boca-Leek paper.

[tengmx]

then I will take the ASH real data @pkimes

[pkimes]

Thanks, @tengmx, @areyesq89!

[ayshwaryas]

Great, thanks Patrick! I'll do the IHW real data summary.

[pkimes]

Added notes on ASH simulations - don't really expect anyone to write as much.
Also realized that ASH doesn't have any real data analysis! Lucky @tengmx!

[tengmx]

Haha! I should buy powerball @pkimes

[stephaniehicks]

I added notes on IHW simulations. Also added a few notes ("idea" or summary of method, "good" or what's good about it, and "bad" what's bad about the method) here.

[pkimes]

@stephaniehicks, good idea re:'idea/good/bad'!
don't mind me - just going to add some flair to make it easier to see.

[pkimes]

done!