Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fcrepo 3919 - Make solr endpoint credentials configurable #200

Merged
merged 11 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 25 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,18 +124,32 @@ This application listens to Fedora's event stream and
indexes objects into an external Solr server.

#### Properties
| Name | Description| Default Value |
| :--- | :---| :---- |
| Name | Description | Default Value |
| :--- |:---| :---- |
| solr.indexing.enabled | Enables/disables the SOLR indexing service. Disabled by default | false |
| solr.fcrepo.checkHasIndexingTransformation | When true, check for an indexing transform in the resource matadata. | true |
| solr.fcrepo.defaultTransform | The solr default ldpath transform when none is provide in resource metadata. | null |
| solr.input.stream | The JMS topic or queue serving as the message source | broker:topic:fedora |
| solr.reindex.stream | The JMS topic or queue serving as the reindex message source | broker:queue:solr.reindex |
| solr.commitWithin | Milliseconds within which commits should occur | 10000 |
| solr.indexing.predicate | When true, check that resource is of type http://fedora.info/definitions/v4/indexing#Indexable; otherwise do not index it. | false |
| solr.ldpath.service.baseUrl | The LDPath service base url | http://localhost:9085/ldpath |
| solr.filter.containers | A comma-separate list of containers that should be ignored by the indexer | http://localhost:8080/fcrepo/rest/audit |

| solr.fcrepo.checkHasIndexingTransformation | When true, check for an indexing transform in the resource metadata with the predicate http://fedora.info/definitions/v4/indexing#hasIndexingTransformation | true |
| solr.fcrepo.defaultTransform | The solr default XSL transform when none is provide in resource metadata. | null |
| solr.input.stream | The JMS topic or queue serving as the message source | broker:topic:fedora |
| solr.reindex.stream | The JMS topic or queue serving as the reindex message source | broker:queue:solr.reindex |
| solr.commitWithin | Milliseconds within which commits should occur | 10000 |
| solr.indexing.predicate | When true, check that resource is of type http://fedora.info/definitions/v4/indexing#Indexable; otherwise do not index it.| false |
| solr.filter.containers | A comma-separate list of containers that should be ignored by the indexer| http://localhost:8080/fcrepo/rest/audit |
| solr.username | Optional username for a Solr server protected with Basic Auth. Must be used in combination with solr.password | |
| solr.password | Optional password for a Solr server protected with Basic Auth. Must be used in combination with solr.username | |


**Note**: You must start with the `file://` protocol when defining the path to a custom XSLT for either the `solr.fcrepo.defaultTransform`
or within the resource using the `http://fedora.info/definitions/v4/indexing#hasIndexingTransformation` predicate.

For example,
```text
solr.fcrepo.defaultTransform=file:///path/to/your/transform.xsl
```
or
```text
@prefix indexing: <http://fedora.info/definitions/v4/indexing#> .
<> indexing:hasIndexingTransformation <file:///path/to/your/transform.xsl> .
```

### Repository Indexer (Triplestore)

Expand All @@ -157,48 +171,6 @@ indexes objects into an external triplestore.
| triplestore.prefer.include | A list of [valid prefer values](https://fedora.info/2021/05/01/spec/#additional-prefer-values) defining predicates to be included | null |
| triplestore.prefer.omit | A list of [valid prefer values](https://fedora.info/2021/05/01/spec/#additional-prefer-values) defining predicates to be omitted. | http://www.w3.org/ns/ldp#PreferContainment |

### LDPath Service

This application implements an LDPath service on repository
resources. This allows users to dereference and follow URI
links to arbitrary lengths. Retrieved triples are cached locally
for a specified period of time.

More information about LDPath can be found at the [Marmotta website](http://marmotta.apache.org/ldpath/language.html).

Note: The LDPath service requires an LDCache backend, such as `fcrepo-service-ldcache-file`.

#### Usage
The LDPath service responds to `GET` and `POST` requests using any accessible resources as a context.

For example, a request to
`http://localhost:9086/ldpath/?context=http://localhost/rest/path/to/fedora/object`
will apply the appropriate ldpath program to the specified resource. Note: it is possible to
identify non-Fedora resources in the context parameter.

A `GET` request can include a `ldpath` parameter, pointing to the URL location of an LDPath program:

`curl http://localhost:9086/ldpath/?context=http://localhost/rest/path/to/fedora/object&ldpath=http://example.org/ldpath`

Otherwise, it will use a simple default ldpath program.

A `POST` request can also be accepted by this endpoint. The body of a `POST` request should contain
the entire `LDPath` program. The `Content-Type` of the request should be either `text/plain` or
`application/ldpath`.

`curl -XPOST -H"Content-Type: application/ldpath" -d @program.txt http://localhost:9086/ldpath/?context=http://localhost/rest/path/to/fedora/object

#### Properties
| Name | Description| Default Value |
| :--- | :---| :---- |
| ldpath.fcrepo.cache.timeout | The timeout in seconds for the ldpath cache | 0 |
| ldpath.rest.prefix | The LDPath rest endpoint prefix | no | /ldpath|
| ldpath.rest.port| The LDPath rest endpoint port | no | 9085 |
| ldpath.rest.host| The LDPath rest endpoint host | no | localhost |
| ldpath.cache.timeout | LDCache timeout in seconds | no | 86400 |
| ldpath.ldcache.directory | LDCache directory | no | ldcache/ |
| ldpath.transform.path | The LDPath transform file path | classpath:org/fcrepo/camel/ldpath/default.ldpath |

### Reindexing Service

This application implements a reindexing service so that
Expand Down
3 changes: 2 additions & 1 deletion docker-compose/camel-toolbox-config/configuration.properties
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
fcrepo.baseUrl=http://fcrepo:8080/fcrepo/rest
fcrepo.authHost=fcrepo
fcrepo.authHost=localhost
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be fcrepo or authentication against Fedora does not work.


jms.brokerUrl=tcp://fcrepo:61616

solr.indexing.enabled=true
solr.baseUrl=http://solr:8983/solr/fcrepo
solr.fcrepo.defaultTransform=org/fcrepo/camel/indexing/solr/default_transform.xsl

triplestore.indexing.enabled=true
triplestore.baseUrl=http://fuseki:3030/fcrepo
Expand Down
2 changes: 1 addition & 1 deletion fcrepo-audit-triplestore/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>org.fcrepo.camel</groupId>
<artifactId>fcrepo-camel-toolbox</artifactId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-audit-triplestore</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion fcrepo-camel-common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
10 changes: 2 additions & 8 deletions fcrepo-camel-toolbox-app/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down Expand Up @@ -48,12 +48,6 @@
<version>${project.parent.version}</version>
</dependency>

<dependency>
<groupId>${project.parent.groupId}</groupId>
<artifactId>fcrepo-ldpath</artifactId>
<version>${project.parent.version}</version>
</dependency>

<dependency>
<groupId>${project.parent.groupId}</groupId>
<artifactId>fcrepo-fixity</artifactId>
Expand Down Expand Up @@ -117,7 +111,7 @@
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>driver</shadedClassifierName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.fcrepo.camel.toolbox.app.Driver</mainClass>
</transformer>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

import java.nio.file.Path;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;

import static org.fcrepo.camel.common.config.BasePropsConfig.FCREPO_CAMEL_CONFIG_FILE_PROPERTY;
import static org.slf4j.LoggerFactory.getLogger;
Expand Down Expand Up @@ -42,12 +43,20 @@ public Integer call() {
System.setProperty(FCREPO_CAMEL_CONFIG_FILE_PROPERTY, configurationFilePath.toFile().getAbsolutePath());
}
final var appContext = new AnnotationConfigApplicationContext("org.fcrepo.camel");
appContext.registerShutdownHook();
final var countdownLatch = new CountDownLatch(1);

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
LOGGER.info("Shutting down fcrepo-camel-toolbox...");
appContext.stop();
countdownLatch.countDown();
}));
appContext.start();
LOGGER.info("fcrepo-camel-toolbox started.");

while (appContext.isRunning()) {
Thread.onSpinWait();
try {
countdownLatch.await();
} catch (final InterruptedException e) {
// Ignore error because we are exiting anyways.
return 1;
}
return 0;
}
Expand Down
2 changes: 1 addition & 1 deletion fcrepo-fixity/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-fixity</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion fcrepo-http-forwarding/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-http-forwarding</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion fcrepo-indexing-solr/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.1.0-SNAPSHOT</version>
<version>6.2.0-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-indexing-solr</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ static class SolrIndexingEnabled extends ConditionOnPropertyTrue {
@Value("${solr.fcrepo.checkHasIndexingTransformation:true}")
private boolean checkHasIndexingTransformation;

@Value("${solr.fcrepo.defaultTransform:}")
@Value("${solr.fcrepo.defaultTransform:org/fcrepo/camel/indexing/solr/default_transform.xsl}")
private String defaultTransform;

@Value("${solr.input.stream:broker:topic:fedora}")
Expand All @@ -52,15 +52,18 @@ static class SolrIndexingEnabled extends ConditionOnPropertyTrue {
@Value("${solr.indexing.predicate:false}")
private boolean indexingPredicate;

@Value("${solr.ldpath.service.baseUrl:http://localhost:9085/ldpath}")
private String ldpathServiceBaseUrl;

@Value("${solr.filter.containers:http://localhost:8080/fcrepo/rest/audit}")
private String filterContainers;

@Value("${solr.baseUrl:http://localhost:8983/solr/collection1}")
private String solrBaseUrl;

@Value("${solr.username:}")
private String solrUsername;

@Value("${solr.password:}")
private String solrPassword;

public boolean isCheckHasIndexingTransformation() {
return checkHasIndexingTransformation;
}
Expand All @@ -85,10 +88,6 @@ public boolean isIndexingPredicate() {
return indexingPredicate;
}

public String getLdpathServiceBaseUrl() {
return ldpathServiceBaseUrl;
}

public String getFilterContainers() {
return filterContainers;
}
Expand All @@ -97,6 +96,13 @@ public String getSolrBaseUrl() {
return solrBaseUrl;
}

public String getSolrUsername() {
return solrUsername;
}

public String getSolrPassword() {
return solrPassword;
}

@Bean(name = "http")
public HttpComponent http() {
Expand Down
Loading
Loading