SciCrunch · tgbugs · Oct 16, 2017 · Sep 12, 2017 · Sep 12, 2017 · Sep 12, 2017
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 # NIF-Ontology
-Repository for the development of NIF Standard (NIFSTD) Ontologies.
+Repository for the NIF Standard (NIFSTD) Ontologies.
 
 # Notice
 NIFSTD is undergoing significant changes. Please see the
@@ -8,18 +8,22 @@ for upcoming changes. For a full record of changes please see
 the [log](https://github.com/SciCrunch/NIF-Ontology/commits/master) or
 run `git log` for a quick overview.
 
-There is older xml version of the ontology in the
-[xml-final branch](https://github.com/SciCrunch/NIF-Ontology/tree/xml-final)
-but it is no longer being updated or maintained.
-
 # Documentation
+Please see [docs/](docs/) for documentation about how to develop NIFSTD
+using this repository.
+
 Please see the [GitHub wiki](https://github.com/SciCrunch/NIF-Ontology/wiki)
 for documentation about the structure of the ontology and how to load and edit it.
 
 # Contributing
-Please [fork](https://github.com/SciCrunch/NIF-Ontology/commits/master#fork-destination-box)
-and submit [pull requests](https://github.com/SciCrunch/NIF-Ontology/pull/new/master) on GitHub.
-See the [wiki](https://github.com/SciCrunch/NIF-Ontology/wiki) for full documentation.
+Please [fork](https://github.com/SciCrunch/NIF-Ontology#fork-destination-box) and submit
+[pull requests](https://github.com/SciCrunch/NIF-Ontology/pull/new/master) on GitHub.  
+Please read [the docs for development processes](docs/processes.md).  
+See the [wiki](https://github.com/SciCrunch/NIF-Ontology/wiki) for more documentation.  
 In short, load [ttl/nif.ttl](ttl/nif.ttl) in your favourite ontology editor,
-and before commit run [ttlfmt](https://github.com/tgbugs/pyontutils/blob/master/ttlfmt.py)
+and before commit run [ttlfmt](https://github.com/tgbugs/pyontutils/blob/master/pyontutils/ttlfmt.py)
 on any files that you have modified.
+
+# License
+<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/80x15.png" /></a><br />The contents of this repository are licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
+The text of the license can also be found in the [LICENSE](LICENSE) file.
diff --git a/catalog-extras b/catalog-extras
@@ -0,0 +1,9 @@
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/doid.owl" uri="external/doid.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/go.owl" uri="external/go.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/iao.owl" uri="external/iao.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/ncbitaxon/subsets/taxslim.owl" uri="external/taxslim.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/pato.owl" uri="external/pato.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/ro.owl" uri="external/ro.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/so.owl" uri="external/so.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/uberon.owl" uri="external/uberon.owl"/>
+    <uri id="User Entered Import Resolution" name="http://purl.obolibrary.org/obo/uberon/bridge/uberon-bridge-to-nifstd.owl" uri="external/uberon-bridge-to-nifstd.owl"/>
diff --git a/deprecate.txt b/deprecate.txt
diff --git a/dev-reasoner.patch b/dev-reasoner.patch
@@ -1,93 +1,26 @@
 diff --git a/ttl/NIF-Investigation.ttl b/ttl/NIF-Investigation.ttl
-index 3e3a27d..93cedfc 100644
+index 8b9ced0..1311ef4 100644
 --- a/ttl/NIF-Investigation.ttl
 +++ b/ttl/NIF-Investigation.ttl
-@@ -22,8 +22,8 @@
-
+@@ -23,7 +23,7 @@
+ 
  <http://ontology.neuinfo.org/NIF/ttl/NIF-Investigation.ttl> a owl:Ontology ;
      owl:imports <http://ontology.neuinfo.org/NIF/ttl/BIRNLex-OBI-proxy.ttl>,
 -        <http://ontology.neuinfo.org/NIF/ttl/external/cogat_v0.3.owl>,
--        <http://ontology.neuinfo.org/NIF/ttl/external/CogPO.owl>,
 +        <http://www.cognitiveatlas.org/ontology/cogat.owl>,
-+        <http://www.cogpo.org/ontologies/CogPOver2010.owl>,
+         <http://ontology.neuinfo.org/NIF/ttl/external/CogPO.ttl>,
          <http://ontology.neuinfo.org/NIF/ttl/NIF-Government-Granting-Agency.ttl>,
          <http://ontology.neuinfo.org/NIF/ttl/NIF-Resource.ttl>,
-         <http://ontology.neuinfo.org/NIF/ttl/NIF-Scientific-Discipline.ttl>,
-diff --git a/ttl/bridge/doid-bridge.ttl b/ttl/bridge/doid-bridge.ttl
-index fb925b9..062efbb 100644
---- a/ttl/bridge/doid-bridge.ttl
-+++ b/ttl/bridge/doid-bridge.ttl
-@@ -8,7 +8,7 @@
- @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
-
- <http://ontology.neuinfo.org/NIF/ttl/bridge/doid-bridge.ttl> a owl:Ontology ;
--    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/doid.owl>,
-+    owl:imports <http://purl.obolibrary.org/obo/doid.owl>,
-         <http://ontology.neuinfo.org/NIF/ttl/nif_backend.ttl> .
-
- ### Annotations
-diff --git a/ttl/bridge/ero-bridge.ttl b/ttl/bridge/ero-bridge.ttl
-index 7aed46c..f79b55c 100644
---- a/ttl/bridge/ero-bridge.ttl
-+++ b/ttl/bridge/ero-bridge.ttl
-@@ -6,6 +6,6 @@
- @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
-
- <http://ontology.neuinfo.org/NIF/ttl/bridge/ero-bridge.ttl> a owl:Ontology ;
--    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/ero.owl> .
-+    owl:imports <http://purl.obolibrary.org/obo/ero.owl> .
-
- ### Serialized using the nifstd custom serializer v1.0.7
-diff --git a/ttl/bridge/go-bridge.ttl b/ttl/bridge/go-bridge.ttl
-index 1cb24a9..e98785f 100644
---- a/ttl/bridge/go-bridge.ttl
-+++ b/ttl/bridge/go-bridge.ttl
-@@ -9,7 +9,7 @@
- @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
-
- <http://ontology.neuinfo.org/NIF/ttl/bridge/go-bridge.ttl> a owl:Ontology ;
--    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/go.owl>,
-+    owl:imports <http://purl.obolibrary.org/obo/go.owl>,
-         <http://ontology.neuinfo.org/NIF/ttl/nif_backend.ttl> .
-
- ### Classes
 diff --git a/ttl/bridge/pr-bridge.ttl b/ttl/bridge/pr-bridge.ttl
-index c34e372..55b8ec2 100644
+index 983a3b3..41deaef 100644
 --- a/ttl/bridge/pr-bridge.ttl
 +++ b/ttl/bridge/pr-bridge.ttl
-@@ -6,6 +6,6 @@
+@@ -6,7 +6,7 @@
  @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
-
+ 
  <http://ontology.neuinfo.org/NIF/ttl/bridge/pr-bridge.ttl> a owl:Ontology ;
--    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/pr.owl> .
-+    owl:imports <http://purl.obolibrary.org/obo/pr.owl> .
-
+-    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/pr.owl>,
++    owl:imports <http://purl.obolibrary.org/obo/pr.owl>,
+         <http://ontology.neuinfo.org/NIF/ttl/generated/pr-dead.ttl> .
+
  ### Serialized using the nifstd custom serializer v1.0.7
-diff --git a/ttl/bridge/uberon-bridge.ttl b/ttl/bridge/uberon-bridge.ttl
-index 17c21a9..891cad4 100644
---- a/ttl/bridge/uberon-bridge.ttl
-+++ b/ttl/bridge/uberon-bridge.ttl
-@@ -10,7 +10,7 @@
- @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
-
- <http://ontology.neuinfo.org/NIF/ttl/bridge/uberon-bridge.ttl> a owl:Ontology ;
--    owl:imports <http://ontology.neuinfo.org/NIF/ttl/external/uberon.owl> ;
-+    owl:imports <http://purl.obolibrary.org/obo/uberon.owl> ;
-     rdfs:label "NIFSTD Uberon Bridge" ;
-     rdfs:comment "This is the bridge file that holds local NIFSTD additions to uberon. This is also staging for any changes that we want to push upstream." ;
-     owl:versionInfo "2016-11-03" ;
-diff --git a/ttl/nif_backend.ttl b/ttl/nif_backend.ttl
-index 1e95a9e..c8147b7 100644
---- a/ttl/nif_backend.ttl
-+++ b/ttl/nif_backend.ttl
-@@ -11,8 +11,8 @@
- <http://ontology.neuinfo.org/NIF/ttl/nif_backend.ttl> a owl:Ontology ;
-     owl:imports <http://ontology.neuinfo.org/NIF/ttl/BIRNLex-OBO-UBO.ttl>,
-         <http://ontology.neuinfo.org/NIF/ttl/BIRNLex_annotation_properties.ttl>,
--        <http://ontology.neuinfo.org/NIF/ttl/external/pato.owl>,
--        <http://ontology.neuinfo.org/NIF/ttl/external/ro_bfo1-1_bridge.owl>,
-+        <http://purl.obolibrary.org/obo/pato.owl>,
-+        <http://purl.org/obo/owl/ro_bfo1-1_bridge>,
-         <http://ontology.neuinfo.org/NIF/ttl/quality_bfo_bridge.ttl>,
-         <http://purl.obolibrary.org/obo/iao.owl>,
-         <http://purl.obolibrary.org/obo/ro.owl>,
diff --git a/docs/development setup.md b/docs/development setup.md
@@ -0,0 +1,76 @@
+# Intro
+This file documents
+the steps needed to create
+a working development environment
+for NIFSTD.
+
+NIFSTD development uses this repo, git (GitHub), pyontutils (Python3.6), Protege, ELK, FACT++, and InterLex.
+
+NIFSTD development also uses SciGraph (Java8), and OWLAPI (see pyontutils for more information).
+
+## git
+Set up git. [This guide can be helpful.](https://help.github.com/articles/set-up-git/).
+
+**We only accept pull requests with commits that have an email address.**  
+
+Please be sure to
+[set your commit email](https://help.github.com/articles/setting-your-commit-email-address-in-git/).
+**NOTE:** If you have not participated in open source development before I often
+recommend creating a new email specifically for all development related activities.
+
+If you have never used git before check out the
+[git bootcamp guides](https://help.github.com/categories/bootcamp/)
+and if you have never used GitHub before check out the
+[GitHub guides](https://guides.github.com/).
+
+## repo
+If you do not have push rights to
+[SciCrunch/NIF-Ontology](https://github.com/SciCrunch/NIF-Ontology), please
+[fork](https://github.com/SciCrunch/NIF-Ontology#fork-destination-box)
+the repo so that you can push somewhere that can submit
+[pull requests](https://github.com/SciCrunch/NIF-Ontology/pull/new/master).
+```bash
+cd ~/git/
+git clone https://github.com/${github_username}/NIF-Ontology.git
+# OR if you have push rights to the SciCrunch org repo
+git clone https://github.com/SciCrunch/NIF-Ontology.git
+```
+My default development setup assumes that all git repos live in `~/git/`, so if your setup
+is different you should adjust accordingly. Note that many of the scripts in pyontutils
+avoid this issue by using `/tmp/` as the default location for local git repos and the
+`--git-local` option can be used to adjust accordingly. However, not all scripts have been
+updated with these assumptions, so if you encounter a `FileNotFoundError` that is likely the
+cause.
+
+## pyontutils
+Install [pyontutils](https://github.com/tgbugs/pyontutils) by following the installation instructions
+[here](https://github.com/tgbugs/pyontutils/blob/master/README.md#installation).
+
+## Protege
+Download and install the latest version of Protege [here](http://protege.stanford.edu/products.php#desktop-protege).
+
+**NIFSTD is a very large ontology which requires more memory than the Protege defaults.**
+
+Modify `run.sh` (or equivalent) so that it launches java with `jre/bin/java -Xmx8G -Xms500M \`
+(or similar). If you do not you will see the following line in the protege logs:  
+`An error occurred whilst loading the ontology at Java heap space. Cause: {}`.
+
+Make sure that **Expand trees by default is disabled** in menu `File -> Preferences -> General`
+otherwise load times can stretch on for many minutes.
+
+## Reasoners
+You can obtain copies of ELK and FACT++ by launching protege and going to
+menu bar `File -> Check for plugins` and selecting ELK and FACT++.
+**NOTE:** sometimes the plugin repos don't load, if that happens try again.
+Otherwise you can download the reasoner plugins and put them in `Protege-*/plugins/`.
+### ELK
+Get the latest version of `elk-distribution-*-protege-plugin.zip` from https://github.com/liveontologies/elk-reasoner/releases.
+Or get the latest version of https://mvnrepository.com/artifact/org.semanticweb.elk/elk-owlapi.
+### FACT++
+Get the latest version of `uk.ac.manchester.cs.owl.factplusplus-*.jar` from https://bitbucket.org/dtsarkov/factplusplus/downloads/.
+
+## InterLex
+[InterLex](http://interlex.org/) provides all new identifiers for NIFSTD.
+You will need a [SciCrunch account](https://scicrunch.org/register) in order to
+access the InterLex web api and to use InterLex to create new terms.
+**NOTE:** At the moment only curators can add terms so let us know if you need access.
diff --git a/docs/import chain.md b/docs/import chain.md
@@ -0,0 +1,25 @@
+# Logic of the import chain
+The basic principle that guides the structure of the import chain is that
+ALL references to owl:Classes, owl:ObjectProperties, etc. should not require
+a local redefinition of that class. Said another way, all local additions to
+externally defined classes should appear in the Annotations section of the
+serialization, not in the Classes section. It is not always possible (nor
+sensible) to follow this principle, but most of the time it is.
+
+If there is an external class that we want to reference it should either be
+imported directly or it should live in a slim that is imported as a proxy
+for the external ontology and copies exactly the contents of that ontology.
+Any local modifications should NOT be made to that slim itself but should be
+made in the bridge file that imports it (in accordance with the basic principle).
+
+In the case where we want to make bridging axioms between multiple external
+ontologies, those axioms should live in another bridge file that imports
+the other bridge files. This practice increases the depth of the import chain
+but ultimately simplifies the development workflow by providing explicit
+locations where bridging axioms should live that can be easily checked by
+seeing whether the basic principle holds.
+
+Import of `ttl/generated/*-dead.ttl` files should occur at the first point in
+the import chain that we have control over. See for example [doid-bridge.ttl](../ttl/bridge/doid-bridge.ttl)
+which imports both `symp-dead.ttl` and `trans-dead.ttl`. The original ontologies
+are imported by `doid.owl` so we import them in `doid-bridge.ttl`.
diff --git a/docs/processes.md b/docs/processes.md
@@ -0,0 +1,96 @@
+# Intro
+This file documents
+the development processes
+that have not (yet) been automated
+and
+the practices that developers should follow
+when working on NIFSTD.
+
+All of these processes expect the full NIF-Ontology development
+environment to have been set up according to [development setup.md](development%20setup.md).
+
+All relative filepaths listed in code sections start from the base
+folder of this repository `../` from the location of this file.
+
+# DO NOT PUSH TO MASTER
+The `uri.neuinfo.org` resolver points to the `master` branch on GitHub
+(see [the resolver config](https://github.com/tgbugs/pyontutils/blob/088b2f8f28be5e55278e3cde1e0e8a4f3ccfc94f/resolver/nif-ont-resolver.conf#L20-L24)).
+
+Therefore, please do not push to the master branch.
+
+Please submit a pull request (even if it is from another branch
+in the SciCrunch org repo) so that we can run all the requisite
+checks to make sure that the ontology is consistent.
+
+## Prior to commit
+```bash
+ttlfmt $(git status -s | grep ttl | grep M | cut -d' ' -f3)
+# OR if you are working on files that need to be normalized
+qnamefix $(git status -s | grep ttl | grep M | cut -d' ' -f3)
+```
+
+## Adding a new external import
+All external imports should be imported into NIFSTD via a bridge file.
+This file will hold any local additions that we want to make.
+1. Create a new bridge file by copying an existing bridge file and modifying
+   as needed to import the external file from its canonical iri, and to import
+   `filename-dead.ttl`.
+2. Add an entry in [catalog-extras](../catalog-extras) for the new import.
+3. Create `ttl/generated/filename-dead.ttl` by running `necromancy http://myurl.org/filename.owl`.
+4. Whenever there is a new release repeat step 3.
+
+## Changing an ontology file
+1. Edit your file and save.
+2. If using protege `git diff` to make sure that any new triples
+   have landed in the correct file.
+3. Execute prior to commit processes.
+
+## Adding a new ontology file
+```bash
+touch filename.ttl
+add filename.ttl
+`make_catalog`  # NOTE this is broken at the moment
+```
+1. Add an entry in [catalog-v001.xml](../ttl/catalog-v001.xml) by hand.
+
+Before you run `make_catalog` make sure that the import chain is local (unpatched)
+otherwise it will take an extremely long time to fetch files.
+
+## Load remote imports from local copies
+Sometimes we don't want to have to retrieve remote copies of files every time
+we start protege, or we need to run tests for patches that are applied to external
+imports when loading the ontology into SciGraph.
+```bash
+cd ttl/
+cat <(head -n -1 catalog-v001.xml) ../catalog-extras <(tail -n 1 catalog-v001.xml) > catalog-new
+mv catalog-new catalog-v001.xml
+```
+Local copies can be obtained using the following.
+```bash
+cd ttl/external/
+curl --header 'Accept-Encoding: gzip' --compressed --location --remote-name-all $(cat ../../catalog-extras | cut -d'=' -f3 | cut -d'"' -f2)
+```
+
+## Running the reasoner
+Due to a [bug in protege](https://github.com/protegeproject/protege/issues/709) we need to patch
+our import chain in order to for the reasoner to run correctly.
+```bash
+patch -p1 -i dev-reasoner.patch
+# OR
+cd ttl/
+patch -p2 -i ../dev-reasoner.patch
+# Then
+protege  # can be launched as you see fit
+```
+1. In protege menu bar `File -> Open` and choose `ttl/nif.ttl`
+   (after first load `File -> Open recent` is faster).
+2. Make sure reasoner is set to ELK (menu bar `Reasoner -> ELK 0.4.3`)
+3. Ctrl-R
+
+Protege does not have to be run from `ttl/` in order to find `ttl/catalog-v001.xml`,
+however the file opened must be in `ttl/`.
+
+## Check for incorrect or unimported predicates
+1. Remove `/tmp/ttlcmp.patch` if it exists
+2. `ttlcmp.sh`  (lives in `pyontutils/pyontutils`)
+3. Review `/tmp/ttlcmp.patch` for annotation and object properties.