Global augmentation fixes

Import elements are ordered
Message schema fixes
Structures, proxy, appinfo schema documents now generated, not copied
Version 1.0-alpha.2

README.md

@@ -10,67 +10,41 @@ For more information on NIEMOpen, see the project's website at [www.niemopen.org
 
 General questions about OASIS Open Projects may be directed to OASIS staff at [[email protected]](mailto:[email protected])
 
-## What's new in version 1.0-alpha.1
-
-* Create NIEM 6 source and message document piles from NIEM 5 model
-* Every schema document gets a Namespace object; SchemaDocument removed
-* Fixed many URIs in source, resource, test files
-* DefinitionText changed to DocumentationText in CMF 1.0  
-* Recognize adapter representation term in addition to appinfo:externalAdapterTypeIndicator
-* Many changes to the JUnit tests.
-
-## What's new in version 0.7-alpha.6 (12 December)
-
-* CMF to message schema is working
-* Most of the augmentation examples are working (but not augmentations on every object)
-* Built-in schema documents are from the PS01 release
-
-## What's new in version 0.7-alpha.5 (14 November)
-
-* NIEM 6 XSD generation uses builtin schema documents from NIEM 6 PSD01
-* Entire NIEM 6 PSD 01 model converts to CMF and back to XSD
-* Handle several annoying differences between Windows and Linux JRE 
-* Many other bugs fixed
-
-## What's new in version 0.7-alpha.4 (26 September)
-
-* Many many bugs fixed
-* Entire NIEM 5 model converts to CMF and back to XSD
-* The usual builtin schema documents now obtained from the JAR file
-* NIEM 6 XSD generation still doesn't work
-
-## What's new in version 0.7-alpha.3 (9 August)
-
-* New features in CMF to XSD commands
-  * `cmftool m2xref` renamed to `cmftool m2xsrc`
-  * Option to generate xml-catalog.xsd file
-  * Option to specify name of generated XML Catalog file
-  * Option to specify message schema "root namespace" with all needed import elements
-
-## What's new in version 0.7-alpha.2
-
-* Now generates CMF version 0.8, which is based on NIEM 6
-* There are NIEM 6 builtin schema documents in *src/main/dist/share*.  These are based on the Wildcard Augmentation discussions.
-* **n5to6** command converts a NIEM 5 model (CMF or XSD) to NIEM 6, rewriting namespace URIs
-* CMF to XSD command split into three
-  * **m2xn5** command generates NIEM 5 XSD
-  * **m2xref** command generates NIEM 6 reference XSD
-  * **m2xmsg** command will generate NIEM 6 message XSD (not implemented yet)
-
-## What's new in version 0.7-alpha.1
-
-* Handling code-list-instance (CLI) properties
-* Handling code-list-schema-attribute (CLSA) appinfo
-* Augmentations are now primarily recorded as properties in the Namespace object
-* The CMF submodule is now in the OASIS repo
-
-## What's working now
-
-- NIEM XML schema pile to Common Model Format
-- Common Model Format to NIEM XML schema pile
-- NIEM 5.2 niem-core model converts from XSD to CMF and back to XSD with no significant change
-  (except LocalTerm appinfo is not handled yet)
-- Entire NIEM 5.2 model converts from XSD to CMF, and vice versa.  (Haven't looked for significant changes yet.)
+## CMFTool commands
+
+* `x2m` command generates CMF from a source or message XSD pile
+* `m2m` command generates a canonical version of a CMF file
+* `m2o` command generates OWL from CMF (needs work)
+* `m2xn5` command generates NIEM 5 XSD from CMF
+* `m2xs` command generates NIEM 6 source XSD from CMF
+* `m2xm` command generates NIEM 6 message XSD from CMF
+* `n5to6` command converts a NIEM 5 model (CMF or XSD) to NIEM 6 CMF, rewriting namespace URIs (needs fixing)
+* `xcanon` command generates a canonical version of an XSD document
+* `xcmp` command compares two XML schemas
+
+Every command accepts a `–help` argument, for providing, well… help.
+
+## What's new in version 1.0-alpha.2
+
+* CMF now has `ObjectProperty` and `DataProperty`
+* `cmf:ReferenceCode` and `appinfo:referenceCode` is working
+* Global element augmentation is working
+* Message schema generation is working
+* Builtin schema documents are updated to version PS02
+  * Add `structures:appliesToParent` to `AssociationType` and `ObjectType` in *structures.xsd*
+  * Add proxies for xs:Name, NCName, QName to *niem-xs.xsd*
+
+## Still to be done
+
+* Writing partial models in CMF, composing a model from parts
+* Handle `appinfo:relationshipPropertyIndicator`
+* Implement `appinfo:AttributeAugmentation` 
+* Corner cases for `xs:simpleType`
+* Ordered properties and xs:list
+* Definitions
+  * Repeated (with `xml:lang`), including inside `LocalTerm` elements
+  * On element/attribute references (and HasProperty objects)
+  * On `xs:import elements`
 
 ## Quick start
 
@@ -85,15 +59,15 @@ Run "cmftool help" for a list of commands.
 
 ## Examples
 
-There is an "examples" directory, with... examples.  But they are still NIEM 5 based.
+There is an "examples" directory, with... examples.  But they are still NIEM 5 based. 
 
 ## Testing
 
-The directory "src/test/resources" contains resources for the JUnit tests.
+The directory "src/test/resources" contains resources for the JUnit tests.  Many, many examples there.
 
 ## Building
 
-This project was built with NetBeans 10.0, Gradle 8.3, and Oracle JDK 17.0.9
+This project was built with NetBeans 21.0, Gradle 8.6, and Oracle JDK 21.0.2
 Try "gradle installDist" 
 
 ## Other assets

build.gradle

@@ -21,7 +21,7 @@ java {
     sourceCompatibility = JavaVersion.VERSION_17
 }
 
-project.version = '1.0-alpha.2 SNAPSHOT'
+project.version = '1.0-alpha.2'
 
 dependencies {
     implementation 'xerces:xercesImpl:2.12.2'

src/main/java/org/mitre/niem/cmf/AugmentRecord.java

@@ -23,6 +23,9 @@
  */
 package org.mitre.niem.cmf;
 
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
 /**
  * A class for <cmf:NamespaceAugmentingRecord>
  * Records all the augmentations in a namespace.  If every augmentation was an
@@ -39,14 +42,16 @@
  * <a href="mailto:[email protected]">[email protected]</a>
  */
 public class AugmentRecord extends ObjectType implements Comparable<AugmentRecord> {
+    static final Logger LOG = LogManager.getLogger(AugmentRecord.class);
+
     private ClassType classType = null;         // the augmented Class
     private Property property = null;           // the agumenting Property
     private int indexInType = -1;               // index of Property in augmentation type, or -1 if no such type
     private int minQ = 0;                       // minOccurs
     private int maxQ = 0;                       // maxOccurs
     private boolean maxUnbounded = false;       // unbounded
     private boolean orderedProperty = false;    // is this property ordered?
-    private int globalAug = AUG_NONE;           // global augmentation? Of objects, associations, simple content?
+    private String globalAugComp = null;        // QName of global component augmented (eg. structures:ObjectType)
 
     public void setClassType (ClassType c)      { classType = c; }    
     public void setProperty (Property p)        { property = p; }
@@ -64,33 +69,34 @@ public class AugmentRecord extends ObjectType implements Comparable<AugmentRecor
     public boolean maxUnbounded ()              { return maxUnbounded; }
     public boolean orderedProperties ()         { return orderedProperty; }
 
-    // Global augmentation codes are represented internally as a bitmap
-    // AUG_NONE, AUG_ASSOC, and AUG_OBJECT are mutually exclusive!
+    public String getGlobalAugmented ()         { return globalAugComp; }
+    public void setGlobalAugmented (String s)   { globalAugComp = s; }
+
+    // Global augmentation codes 
     public static final int AUG_NONE   = 0;     // not a global augmentation
-    public static final int AUG_ASSOC  = 1;     // global augmentation for associations
-    public static final int AUG_OBJECT = 2;     // global augmentation for objects
-    public static final int AUG_SIMPLE = 4;     // global augmentation for simple content
-    public static final int AUG_MAX    = 4;
-    public static String[] augCode = { "NONE", "ASSOCIATION", "OBJECT", null, "SIMPLE" };
+    public static final int AUG_ASSOC  = 1;     // a global augmentation for associations
+    public static final int AUG_OBJECT = 2;     // a global augmentation for objects
+    public static final int AUG_SIMPLE = 3;     // a global augmentation for simple content
+    public static final int AUG_NUMCODES = 4;   // this many kinds of augmentations
 
-    public void setGlobalAug (int i)            { globalAug = i; }
-    public boolean hasGlobalAug (int i)         { return 0 != (globalAug & i); }
-    public int getGlobalAug ()                  { return globalAug; }
-    public void addGlobalAug (int i)            { globalAug = globalAug | i; }
-    public void addGlobalAug (String s) {
-        if (null == s) return;
-        for (int i = 1; i <= AUG_MAX; i++)
-            if (null != augCode[i] && augCode[i].equals(s)) addGlobalAug(i);
+    public int getGlobalAug () {
+        if (null == globalAugComp) return AUG_NONE;
+        var cname = globalAugComp.substring(globalAugComp.indexOf(":")+1);
+        if (cname.startsWith("Association")) return AUG_ASSOC;
+        if (cname.startsWith("Object")) return AUG_OBJECT;
+        if (cname.startsWith("Simple")) return AUG_SIMPLE;
+        LOG.error("unknown global augmented component {}", globalAugComp);
+        return AUG_NONE;
     }
-    public String getGlobalAugCode (int i)      { return augCode[i]; }
 
     public AugmentRecord () {}
 
     @Override
     public int compareTo (AugmentRecord o) {
-        int rv = this.classType.compareTo(o.classType);
-//        if (0 == rv) rv = this.indexInType - o.indexInType;
+        int rv = 0;
+        if (null != this.classType && null != o.classType) rv = this.classType.compareTo(o.classType);
         if (0 == rv) rv = this.property.getQName().compareTo(o.property.getQName());
+        if (0 == rv) rv = this.globalAugComp.compareTo(o.globalAugComp);
         return rv;
     }
 }

src/main/java/org/mitre/niem/cmf/ClassType.java

@@ -33,7 +33,7 @@
  * <a href="mailto:[email protected]">[email protected]</a>
  */
 public class ClassType extends Component {
-    private String refCode = "";
+    private String refCode = null;
     private boolean isAugmentable = false;
     private ClassType extensionOfClass  = null;
     private final List<HasProperty> hasPropertyList = new ArrayList<>();

src/main/java/org/mitre/niem/cmf/Model.java

@@ -64,6 +64,16 @@ public class Model extends ObjectType {
     public Model () { super(); }
 
     public NamespaceMap namespaceMap () { return nsmap; }
+
+    // Convenience routine to get URI for this prefix from the namespace map.
+    public String getPrefix (String uri) {
+        return nsmap.getPrefix(uri);
+    }
+
+    // Convenience routine to get prefix for this URI from the namespace map.
+    public String getURI (String prefix) {
+        return nsmap.getURI(prefix);
+    }
 
     // Returns a list of Component objects in the model, ordered by QName.
     // Generates sorted list when necessary, caches for later.
@@ -90,10 +100,16 @@ public Component getComponent (String qn) {
     }
 
     public Component getComponent (String nsuri, String lname) {
-        Namespace n = uri2NS.get(nsuri);
-        if (null == n) return null;
-        String qn = n.getNamespacePrefix() + ":" + lname;
-        return components.get(qn);
+        var qn = getQN(nsuri, lname);
+        if (null == qn) return null;
+        else return components.get(qn);
+    }
+
+    // Convenience function to generate QName from namespace and name
+    public String getQN (String nsuri, String lname) {
+        var ns = uri2NS.get(nsuri);
+        if (null == ns) return null;
+        else return ns.getNamespacePrefix() + ":" + lname;
     }
 
     public ClassType getClassType (String qname) {

src/main/java/org/mitre/niem/cmf/ModelToN6.java

@@ -24,13 +24,10 @@
 package org.mitre.niem.cmf;
 
 import java.util.ArrayList;
-import java.util.HashMap;
 import java.util.List;
-import java.util.Map;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import org.apache.logging.log4j.LogManager;
-import static org.mitre.niem.cmf.NamespaceKind.NIEM_NOTBUILTIN;
 
 /**
  * A class to convert a Model object to the NIEM 6 architecture

src/main/java/org/mitre/niem/cmf/NamespaceKind.java

@@ -392,7 +392,7 @@ public static String namespaceUtil2Builtin (int util) {
 
     // Returns the namespace URI for a utility namespace given the architecture
     // and version.  
-    public static String getBuiltinNS(int util, String arch, String version) {
+    public static String getBuiltinNS (int util, String arch, String version) {
         if (NIEM_XML == util) return XML_NS_URI;
         String ustr = namespaceUtil2Builtin(util);
         String uri = null;
@@ -410,6 +410,14 @@ public static String getBuiltinNS(int util, String arch, String version) {
         return uri;
     }
 
+    // Returns the namespace URI for a utility namespace, based on the
+    // URI of another utility namespace.
+    public static String getBuiltinNS (int util, String builtinURI) {
+        var arch = uri2Architecture(builtinURI);
+        var vers = uri2Version(builtinURI);
+        return getBuiltinNS(util, arch, vers);
+    }
+
     // Default schema document file paths are properly not part of CMF, but it's
     // way too much trouble to put them anywhere else. 
     private final static String[] defBuiltinPath = {

src/main/java/org/mitre/niem/cmf/Property.java

@@ -32,7 +32,7 @@ public class Property extends Component {
     private Property subPropertyOf = null;
     private ClassType classType = null;
     private Datatype datatype = null;
-    private String refCode = null;          // default to ClassType if not set
+    private String refCode = null;
     private boolean isAttribute = false; 
     private boolean isRefAttribute = false;
     private boolean isRelationship = false;
@@ -61,12 +61,12 @@ public Property (Namespace ns, String lname) {
     public boolean isAttribute()                  { return isAttribute; }
     public boolean isRefAttribute()               { return isRefAttribute; }
     public boolean isRelationship()               { return isRelationship; }
-    public String getReferenceCode ()             { return null == refCode ? "" : refCode ; }
+    public String getReferenceCode ()             { return refCode; } 
 
     public String getInheritedReferenceCode() {
         if (null != refCode) return refCode;
         if (null != classType) return classType.getReferenceCode();
-        return "";
+        return null;
     }
 
     @Override

src/main/java/org/mitre/niem/cmf/ReferenceGraph.java

@@ -94,7 +94,7 @@ private void addDatatypeRefs (Datatype d) {
         var duni = d.getUnionOf();
         var dlst = d.getListOf();
         if (null != dres) addRef(dns, dres.getDatatype().getNamespace());
-        if (null != dlst) addRef(dns, dlst.getListOf().getNamespace());
+        if (null != dlst) addRef(dns, d.getListOf().getNamespace());
         if (null != duni) 
             for (var udt : duni.getDatatypeList()) 
                 addRef(dns, udt.getNamespace());

src/main/java/org/mitre/niem/cmftool/CMFTool.java

@@ -47,7 +47,22 @@ private void run (String[] args) {
 
     String xtd = "src/test/resources/xsd6/";
     if (0 == args.length) {
-        args = new String[]{"m2xm", "-o", "tmp/tmp", "tmp/07-AugOTwithE/augOTwE.cmf"};
+//        var td = "src/main/CMF/";
+//        args = new String[]{"m2xm", "-o", td+"tmp", td+"model.cmf"};
+
+//        var td = "src/test/resources/xsd6/";
+//        args = new String[]{"m2xs", "-o", td+"tmp", td+"literal-2.cmf"};
+
+//        args = new String[]{"m2xs", "-o", "tmp/tmp", "tmp/m60.cmf"};
+//        var td = "../../NIEM/ntac-admin/examples/src/Augmentation/01-N6-Subset/";
+//        args = new String[]{"m2xs", "-o", td+"tmp", td+"subset.cmf"};
+
+//        var td = "../../NIEM/ntac-admin/examples/src/Augmentation/08-AugOTwithA/";
+//        args = new String[]{"m2xm", "-o", td+"tmp", td+"augOTwA.cmf"};
+
+//        var td = "../../NIEM/ntac-admin/examples/src/Augmentation/07-AugOTwithE/";
+//        args = new String[]{"m2xs", "-o", td+"tmp", td+"augOTwE.cmf"};
+//        args = new String[]{"x2m", "-o", td+"foo", td+"model.xsd/messageModel.xsd"};
 //        args = new String[]{"m2xm", "-o", xtd + "tmp", xtd + "augment.cmf"};
     }
 

src/main/java/org/mitre/niem/cmftool/CmdCMFtoCMF.java

@@ -26,18 +26,13 @@
 import com.beust.jcommander.JCommander;
 import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
-import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.io.OutputStream;
-import java.io.OutputStreamWriter;
-import java.io.UnsupportedEncodingException;
 import java.util.List;
-import java.util.logging.Level;
-import java.util.logging.Logger;
 import javax.xml.parsers.ParserConfigurationException;
 import javax.xml.transform.TransformerException;
 import org.mitre.niem.cmf.Model;

src/main/java/org/mitre/niem/cmftool/CmdCMFtoXSD.java

@@ -31,14 +31,10 @@
 import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.util.List;
-import java.util.logging.Level;
-import java.util.logging.Logger;
 import javax.xml.parsers.ParserConfigurationException;
-import javax.xml.transform.TransformerException;
 import org.apache.commons.io.FileUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.core.config.Configurator;
-import org.mitre.niem.cmf.CMFException;
 import org.mitre.niem.cmf.Model;
 import org.mitre.niem.xsd.ModelToMsgXSD;
 import org.mitre.niem.xsd.ModelToN5XSD;

src/main/java/org/mitre/niem/cmftool/CmdXSDtoCMF.java

@@ -58,7 +58,6 @@
 import org.xml.sax.SAXException;
 import static org.mitre.niem.cmf.NamespaceKind.NSK_BUILTIN;
 import static org.mitre.niem.cmf.NamespaceKind.cta2Arch;
-import static org.mitre.niem.cmf.NamespaceKind.cta2Version;
 
 /**
  *