Update microstates of pKa challenge #20

MehtapIsik · 2017-11-23T03:50:52Z

I have updated the microstate lists of pKa challenge molecules, according to following corrections:

Incorporated missing microstates according to suggestions from participants.
Removed duplicate microstates (due to resonance structures and geometric isomers) by visual inspection.
Removed duplicate microstates using an OpenEye script based on @bannanc's method and Geoff Skillman's suggestion to compare the number of hydrogens bound to heavy atoms to detect resonance structures.

We should release these microstate lists with major updates as Version 1.4 of this repo, as soon as possible.

…metric isomers eliminated.

davidlmobley

Looks good to me in terms of organization/content, but I'm not doing any checking on the states. Should we get @bannanc to review also or go ahead and merge?

Also, what do you think -- is this a substantial enough change that we should send out an e-mail blast with news of it (once merged) too in case anyone is not watching the GitHub repo?

…MPL6 into pKa_microstate_corr

MehtapIsik · 2017-11-24T20:19:14Z

@davidlmobley I removed many replicate microstates (about 25% of initial microstates list ) and added some missing microstates. I think once we merge this PR, it will be appropriate to send out an e-mail blast stating following changes:

Microstate files of pKa challenge (SMXX_microstates.csv files) were updated. Some microstates of the initial version were removed and some missing microstates are added.
Deprecated microstates were replicates of others (resonance structures or stereoisomers). Deprecated microstates and their microstate IDs were recorded in SMXX_microstates_deprecated.csv files with "deprecated" comment in "remarks" column. Their microstate IDs were preserved.
Example submission files were added to illustrate expected format when filling 3 types of submission templates of the pKa challenge.

MehtapIsik · 2017-11-24T20:27:49Z

I am done with my updates on this branch.
@bannanc Could you please have a look at the updated microstates and example submission files when you get a chance before we merge?

davidlmobley · 2017-11-25T01:17:13Z

OK, thanks. Tag me once @bannanc reviews and we can send out an e-mail blast. In the meantime I'll ask for the latest e-mail list (I don't have access).

Thanks.

bannanc · 2017-11-28T00:37:23Z

pKa_challenge_instructions.md

@@ -158,6 +167,9 @@ These and blank lines will be ignored.

 The file must contain the following four components in the following order: your predictions, a name for your computational protocol, a list of the major software packages used, and a long-form methods description. 
 Each of these components must begin with a line containing only the corresponding keyword: `Predictions:`, `Name:`, `Software:`, and `Method:`, as illustrated in the example files. 
+
+Example submission files can be found in `physical_properties/pKa/example_submission_files/` directory to illustrate expected format when filling submission templates of the pKa challenge.


If this directory existed before this PR, could you make this a hyperlink so people can click on the path?

I added a link.

bannanc · 2017-11-28T00:38:23Z

pKa_challenge_instructions.md

@@ -130,6 +130,15 @@ Newly added microstates will also be shared with participants and microstates li
 Please do not create a microstate ID yourself. 
 It is important that challenge organizers assign unique microstate IDs and keep track.  

+### Updates on microstates lists
+
+Microstate SMILES strings and microstate IDs can be found in `physical_properties/pKa/microstates/` directory. 


Whenever possible, I think it is useful to provide links so people don't have to click around to figure out where you're pointing them.

If this directory existed before this PR, could you make this a hyperlink so people can click on the path?
If the directory is new a link will need to be added after this PR.

I agree. I added a link.

@bannanc FTR, one can add links in PRs that add the directory (a link is just formatted text). Relative links (like here) will be active even before merging since they'll just refer elsewhere in the branch; absolute links might not be active until merging.

bannanc · 2017-11-28T00:38:56Z

pKa_challenge_instructions.md

+Due to replicate and missing microstates present in the first release of microstate lists, we have updated `SMXX_microstates.csv` files with necessary corrections in Version 1.4 of this repository. 
+The main correction of this update was the removal of resonance structures that were causing dublicate representation of same microstates. 
+We have also added new microstates suggested by participants.
+Newly added microstates were assigned unique microstate IDs, as recorded in `SMXX_microstates.csv` files.


Same thing I said about the directory paths, it would be convenient for the file names to also link to the relevant file.

These files are all in the directory mentioned at the beginning of paragraph. I added a link to the directory. I won't add link to the files.

I think since it's multiple files you can't have each be a link.

bannanc · 2017-11-28T00:41:20Z

physical_properties/pKa/example_submission_files/typeI-MehtapIsik-1.csv

+#
+# PREDICTION SECTION
+#
+# Fill one typeI_microscopic_pKas_and_microstates.csv template for all molecules predicted with the same method.


I understand what you are trying to say here, but its worded in a way that is a little confusing. Maybe something like:

You may submit predictions from multiple methods, but people make a separate [file name] for each different method.
@davidlmobley help on wording?

"Fill one typeI_microscopic_pKas_and_microstates.csv template for all molecules predicted with the same method."

Agree this is awkward. Maybe, "Each submission (method) must have a completed typeI_microscopic_pKas_and_microstates.csv file for all molecules the prediction encompasses," if that's what's meant?

I liked how you clarified this rule. I added these sentences to pKa_challenge_instructions.md

bannanc · 2017-11-28T00:49:40Z

physical_properties/pKa/example_submission_files/typeI-MehtapIsik-1.csv

+# Please include the values of key parameters, with units, and explain how any statistical uncertainties were estimated.
+# Use as many lines of text as you need. 
+# All text following the "Method:" keyword will be regarded as part of your free text methods description. 
+Method: 


It might be informative to have a real example here, maybe with the reference calculation protocol (or approximately that).

Agree that if you can provide people an example of what these would look like it will probably help them (and help prevent an e-mail deluge when people start trying to submit).

I created example submission files directory with 3 filled templates. Method section has a few sentences as example, not exactly the full description of a method.

So I read your example when I wrote this initial comment. Maybe it is fine as is, in the past we put a method for our reference calculations here as an example of how much detail to include, but I guess its up to them how to describe it anyway.

I agree that it would be nicer to have a full detail description here. I can add that in the future when Bas is done with reference calculations.

bannanc · 2017-11-28T00:50:09Z

physical_properties/pKa/example_submission_files/typeII-MehtapIsik-1.csv

+#
+# PREDICTION SECTION
+#
+# Fill one typeII_microstate_fractional_populations.csv template file for all molecules and microstates you have predictions for.


see my comment for this line in the typeI example.

bannanc · 2017-11-28T00:53:07Z

physical_properties/pKa/example_submission_files/typeII-MehtapIsik-1.csv

+# e.g. In column 3 report ln(fractional microstate population) at pH 2.10 of your predicted microstate.
+# Use scientific notation with 3 decimals of precision (e.g. -2.30e0 or 3.14e-4).
+# For microstates predicted to have 0 populations, instead of ln(0), report as "-infinity".
+# Do not report error estimates in this section. You can report error estimates in Method secton.


If I read this I would provide the same lines as below with uncertainty estimates for each measurement. Is that what you're expecting people to do? If you want people to have the option of reporting uncertainties maybe you should make an uncertainties: section for this file type.

We don't plan to evaluate uncertainties of this submission type.

OK, maybe just say that and not tell people they can report error estimates in the method section?

OK. This is reasonable.

I removed " You can report error estimates in Method section." sentence from the template file.

bannanc · 2017-11-28T00:55:28Z

physical_properties/pKa/example_submission_files/typeIII-MehtapIsik-1.csv

+# For molecules with multiple macroscopic pKas report each macroscopic pKa in a new line.
+# Report pKa values to two decimal places (e.g. 10.71).
+# Reporting the standard error of the mean (SEM) is optional and encouraged. If it is reported, SEM should be reported to two decimal places (e.g. 1.02).
+# For values for which you don't have an estimate, leave that cell of the csv table empty.


Does this apply to both the uncertainty and the pKa?
For example, if I can only calculate macro pkas for 5 of the molecules can I still participate? Do I have to list all of the molecule IDs with empty cells next them?

It applies both to uncertainty and pKa.

We can handle both cases for a molecule without prediction:

Excluding a molecule ID line completely

A line with just molecule ID followed by empty cvs cells

ok cool, I just wanted to make sure you had thought about it in a file parsing perspective.

bannanc · 2017-11-28T00:56:16Z

physical_properties/pKa/example_submission_files/typeII-MehtapIsik-1.csv

@@ -0,0 +1,63 @@
+# Submission Type II


Maybe we should specify at the top of each file that any line that begins with a # is considered a comment and will be ignored when parsing.

This is a good idea. I will add this.

bannanc · 2017-11-28T00:58:24Z

physical_properties/pKa/microstates/SM01_microstates.csv

@@ -1,11 +1,9 @@
 microstate ID,canonical isomeric SMILES
 SM01_micro001,c1cc2c(cc1O)c3c(o2)C(=[OH+])NCCC3
 SM01_micro002,c1cc2c(cc1[O-])c3c(o2)C(=NCCC3)[O-]
-SM01_micro003,c1cc2=[O+]C3=C(NCCCC3=c2cc1O)[O-]


I'm going to be honest, I'm not good enough at reading SMILES strings to check these so I'm going to trust your visualization and the openeye code we worked on together.

OK. I am also relying on participants to tell us, if they catch another mistake.

bannanc · 2017-11-28T01:01:50Z

I had a few minor things that I think should be addressed before this is accepted.

I have a general question about resonance vs. microstates/tautomers. What if my method comes up with the resonance structure that doesn't agree with the microstate ID you have assigned it? Should I use the deprecated ID or is it my responsibility to figure out which ID it was supposed to be?

Related to that, if you can't use a deprecated microstate ID are you going to check IDs when files are submitted?

MehtapIsik · 2017-11-28T17:51:24Z

@bannanc I expect people to find the matching ID in the updated microstate file, i.e. find a resonance structure of the same microstate from the SMILES list. If they can't find it, it is they can email me their requested microstate in which case I will check if it is unique or resonance of others first.

But if they insist on using the deprecated microstate ID in their submissions, that would be fine too. When I analyze the prediction, I will pool together microstate IDs that are resonance structures of one another (sum their predicted populations).

MehtapIsik · 2017-11-28T18:19:51Z

@davidlmobley @bannanc I am done with these edits.

bannanc · 2017-11-28T18:35:43Z

pKa_challenge_instructions.md

@@ -87,7 +87,7 @@ Participants are encouraged to submit their results in all or multiple submissio
 #### Prediction Type I - microscopic pKas and related microstates
 Predicting microscopic pKas and related microstate structures. 
 Different protonation states and tautomer combinations constitute different microstates. 
- Fill one `typeI_microscopic_pKas_and_microstates.csv` template for all molecules.
+- Fill one `typeI_microscopic_pKas_and_microstates.csv` template for all molecules predicted with one method. You may submit predictions from multiple methods, but you should fill a separate template file for each different method. 


This is more clear, thanks!

bannanc

I added a few follow-up questions, but I think in general its fine to push these changes.

… file.

MehtapIsik · 2017-11-28T19:53:27Z

I think we addressed everything so far.

I am anxious to merge this branch because of major changes in microstate lists. The sooner we announce this correction the better.

davidlmobley · 2017-11-28T19:57:35Z

Thanks, @MehtapIsik . I have the e-mail list; would you like to do the honors? LMK on Slack.

davidlmobley · 2017-11-28T19:58:17Z

Also, @MehtapIsik - please go ahead and do a release. Tag me if you need anything.

MehtapIsik · 2017-11-28T20:21:25Z

Release version 1.4 is done.

MehtapIsik added 5 commits November 22, 2017 21:56

Update microstates files to v1.3.2. Resonance structures and most geo…

f999f67

…metric isomers eliminated.

Merge branch 'master' into pKa_microstate_corr

d371ecf

Comment on updated microstates in pKa_challenge_instructions.

7854efd

Update main README for version 1.4 with updated pKa microstates.

a27ac60

Clarification of microstates update in pKa instructions.

fff35f9

davidlmobley approved these changes Nov 23, 2017

View reviewed changes

MehtapIsik added 3 commits November 24, 2017 11:13

README added to microstates directory.

cbe43ff

Add example pKa prediction submission files.

05764ae

Merge branch 'pKa_microstate_corr' of https://github.com/MobleyLab/SA…

0a428ab

…MPL6 into pKa_microstate_corr

Update pKa instructions with availability of example submission files.

fcabda6

bannanc suggested changes Nov 28, 2017

View reviewed changes

MehtapIsik added 13 commits November 28, 2017 12:52

Incorporate Caitlin's suggestions to pKa instructions.

31a38e5

Update typeI-MehtapIsik-1.csv

d992603

Update typeII-MehtapIsik-1.csv

0a95fca

Update typeIII-MehtapIsik-1.csv

808de91

Update typeI_microscopic_pKas_and_microstates.csv

53c3e41

Update typeII_microstate_fractional_populations.csv

325fcb4

Update typeIII_macroscopic_pKas.csv

8479a8c

Add "one file for each method" comment to type I template.

5a0ff9d

Add "one file for each method" comment to type I example.

a50dd66

Add "one file for each method" comment to type II template.

6d45391

Add "one file for each method" comment to type II example.

4549830

Add "one file for each method" comment to type III template.

9d2ee02

Add "one file for each method" comment to type III example.

9c714c5

bannanc reviewed Nov 28, 2017

View reviewed changes

bannanc approved these changes Nov 28, 2017

View reviewed changes

MehtapIsik added 3 commits November 28, 2017 14:40

Remove the error estimate comment from type II template.

eba6e27

Remove the error estimate comment from type II example submission.

84a15aa

Clarify type II error estimate comment of type II in pKa instructions…

0e5a080

… file.

davidlmobley merged commit 9be5023 into master Nov 28, 2017

davidlmobley deleted the pKa_microstate_corr branch November 28, 2017 19:58

Update microstates of pKa challenge #20

Update microstates of pKa challenge #20

Conversation

MehtapIsik commented Nov 23, 2017

davidlmobley left a comment

Choose a reason for hiding this comment

MehtapIsik commented Nov 24, 2017

MehtapIsik commented Nov 24, 2017

davidlmobley commented Nov 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bannanc commented Nov 28, 2017

MehtapIsik commented Nov 28, 2017

MehtapIsik commented Nov 28, 2017

Choose a reason for hiding this comment

bannanc left a comment

Choose a reason for hiding this comment

MehtapIsik commented Nov 28, 2017

davidlmobley commented Nov 28, 2017

davidlmobley commented Nov 28, 2017

MehtapIsik commented Nov 28, 2017