Loading files

One of the main features of ml-javaclient-util is its support for loading modules into a MarkLogic modules database.

Loading modules

Prior to version 3.0.0, this was accomplished via a combination of XCC and the REST API:

DatabaseClient client = DatabaseClientFactory.newClient(...); // Use the ML Java Client API
XccAssetLoader assetLoader = new XccAssetLoader(client); // Can use XCC or the REST API to load asset modules
DefaultModulesLoader modulesLoader = new DefaultModulesLoader(assetLoader);
File modulesDir = new File("src/main/ml-modules");
ModulesFinder modulesFinder = new DefaultModulesFinder(); // Allows for adjusting where modules are stored on a filesystem
modulesLoader.loadModules(modulesDir, modulesFinder, client);

Starting in version 3.0.0, all modules are loaded via the REST API:

DatabaseClient client = DatabaseClientFactory.newClient(...); // Use the ML Java Client API
AssetFileLoader assetFileLoader = new AssetFileLoader(client); // Uses the REST API to load asset modules
DefaultModulesLoader modulesLoader = new DefaultModulesLoader(assetLoader);
File modulesDir = new File("src/main/ml-modules");
ModulesFinder modulesFinder = new DefaultModulesFinder(); // Allows for adjusting where modules are stored on a filesystem
modulesLoader.loadModules(modulesDir, modulesFinder, client);

Loading any kind of file

The support for loading asset modules in 3.0.0 is provided via the AssetFileLoader class, as shown above. This class extends GenericFileLoader, which provides a flexible and extensible API for loading files from multiple directories, where permissions and collections can be specified on a per-file basis.

GenericFileLoader provides the following features:

1. Files can be loaded from one or more paths.
2. Files are loaded via a BatchWriter, which means they can be loaded via the REST API, DMSDK, or XCC.
3. Zero or more FileFilter objects can be used to configure which files are loaded.
4. Zero or more DocumentFileProcessor objects can be used to process each file as it's loaded. A processor can decide not to load the file, or it could modify the target URI, or the collections or permissions (see more details below), etc.
5. A default set of permissions and default set of collections can be defined.
6. Tokens can be replaced in the text of a file before it's loaded.
7. GenericFileLoader is aware of a number of extensions that indicate a file should be loaded as a binary, and more extensions can be added.

Specifying collections and permissions

As of 2.13.0, you can now specify collections and permissions for schemas, and this will soon be supported for modules too. This has been added specifically for making it easier to add ML9 redaction rulesets to specific collections. You can do this by defining either of the following files in any directory containing schemas:

• collections.properties
• permissions.properties

These are expected to have key/value pairs of filename=collection1,collection2 and filename=role,capability,role,capability.

For example, for a file named "my.ruleset", you could have the following in collections.properties:

my.ruleset=coll1,coll2

And in permissions.properties:

my.ruleset=rest-reader,read,rest-writer,update

And your directory would look like this:

collections.properties
my.ruleset
permissions.properties

Note that these special properties files will NOT be loaded into MarkLogic - they're just there to provide metadata for files that you do want to load.