Skip to content

Commit

Permalink
Fixes #4123: Add anthropic claude support (#4148)
Browse files Browse the repository at this point in the history 
* Fixes #4123: Add anthropic claude support

* added doc examples
  • Loading branch information
@vga91
vga91 authored Jul 30, 2024
1 parent 30fe1c9 commit b80a8bd
Show file tree
Hide file tree
Showing 8 changed files with 539 additions and 25 deletions.
220 changes: 217 additions & 3 deletions docs/asciidoc/modules/ROOT/pages/ml/openai.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,10 @@ All the following procedures can have the following APOC config, i.e. in `apoc.c
.Apoc configuration
|===
|key | description | default
| apoc.ml.openai.type | "AZURE", "HUGGINGFACE", "OPENAI", indicates whether the API is Azure, HuggingFace or another one | "OPENAI"
| apoc.ml.openai.url | the OpenAI endpoint base url | https://api.openai.com/v1
(or empty string if `apoc.ml.openai.type=<AZURE OR HUGGINGFACE>`)
| apoc.ml.openai.type | "AZURE", "HUGGINGFACE", "OPENAI", indicates whether the API is Azure, HuggingFace, Anthropic or another one | "OPENAI"
| apoc.ml.openai.url | the OpenAI endpoint base url | `https://api.openai.com/v1` by default,
`https://api.anthropic.com/v1` if `apoc.ml.openai.type=<ANTHROPIC>`,
or empty string if `apoc.ml.openai.type=<AZURE OR HUGGINGFACE>`
| apoc.ml.azure.api.version | in case of `apoc.ml.openai.type=AZURE`, indicates the `api-version` to be passed after the `?api-version=` url
|===

Expand Down Expand Up @@ -249,3 +250,216 @@ CALL apoc.ml.openai.chat([{"role": "user", "content": "Explain the importance of
'<apiKey>',
{endpoint: 'https://api.groq.com/openai/v1', model: 'mixtral-8x7b-32768'})
----

=== Anthropic API (OpenAI-compatible)

Another alternative is to use the https://docs.anthropic.com/en/api/getting-started[Anthropic API].

We can use the `apoc.ml.openai.chat` procedure to leverage the https://docs.anthropic.com/en/api/messages[Anthropic Messages API].

These are the default key-value parameters that will be included in the body request, if not specified:

.Default Anthropic key-value parameters
[%autowidth, opts=header]
|===
| key | value
| max_tokens | 1000
| model | "claude-3-5-sonnet-20240620"
|===

For example:

[source,cypher]
----
CALL apoc.ml.openai.chat([
{ content: "What planet do humans live on?", role: "user" },
{ content: "Only answer with a single word", role: "assistant" }
],
$anthropicApiKey,
{apiType: 'ANTHROPIC'}
)
----

.Example result
[%autowidth, opts=header]
|===
| value
| {"id": "msg_01NUvsajthuiqRXKJyfs4nBE",
"content": [{"text": " in lowercase: What planet do humans live on?", type: "text"}],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"usage": {"output_tokens": 13, input_tokens: 20},
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message"
}
|===


Moreover, we can define the Anthropic API Version via the `anthropic-version` config parameter, e.g.:

[source,cypher]
----
CALL apoc.ml.openai.chat([
{ content: "What planet do humans live on?", role: "user" }
],
$anthropicApiKey,
{apiType: 'ANTHROPIC', `anthropic-version`: "2023-06-01"}
)
----

with a result similar to above.


Additionally, we can specify a Base64 image to include in the body, e.g.:

[source,cypher]
----
CALL apoc.ml.openai.chat([
{ role: "user", content: [
{type: "image", source: {type: "base64",
media_type: "image/jpeg",
data: "<theBase64ImageOfAPizza>"} }
]
}
],
$anthropicApiKey,
{apiType: 'ANTHROPIC'}
)
----

.Example result
[%autowidth, opts=header]
|===
| value
| {"id": "msg_01NxAth45myf36njuh1qwxfM",
"content": [{
"text": "This image shows a pizza.....",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"usage": {
"output_tokens": 202,
"input_tokens": 192
},
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message"
}
|===

We can also specify other custom body requests, like the `max_tokens` value, to be included in the config parameter:

[source,cypher]
----
CALL apoc.ml.openai.chat([
{ content: "What planet do humans live on?", role: "user" }
],
$anthropicApiKey,
{apiType: 'ANTHROPIC', max_tokens: 2}
)
----

.Example result
[%autowidth, opts=header]
|===
| value
| {
"id": "msg_01HxQbBuPc9xxBDSBc5iWw2P",
"content": [
{
text": "Hearth",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"usage": {
"output_tokens": 10,
"input_tokens": 20
},
"stop_reason": "max_tokens",
"stop_sequence": null,
"type": "message"
}
|===




Also, we can use the `apoc.ml.openai.completion` procedure to leverage the https://docs.anthropic.com/en/api/complete[Anthropic Complete API].

These are the default key-value parameters that will be included in the body request, if not specified:

.Default Anthropic key-value parameters
[%autowidth, opts=header]
|===
| key | value
| max_tokens_to_sample | 1000
| model | "claude-2.1"
|===


For example:

[source,cypher]
----
CALL apoc.ml.openai.completion('\n\nHuman: What color is sky?\n\nAssistant:',
$anthropicApiKey,
{apiType: 'ANTHROPIC'}
)
----

.Example result
[%autowidth, opts=header]
|===
| value
| {
"id": "compl_016JGWzFfBQCVWQ8vkoDsdL3",
"stop": "Human:",
"model": "claude-2.1",
"stop_reason": "stop_sequence",
"type": "completion",
"completion": " The sky appears blue on a clear day. This is due to how air molecules in Earth's atmosphere scatter sunlight. Shorter wavelengths of light like blue and violet are scattered more, making the sky appear blue to our eyes.",
"log_id": "compl_016JGWzFfBQCVWQ8vkoDsdL3"
}
|===

Moreover, we can specify other custom body requests, like the `max_tokens_to_sample` value, to be included in the config parameter:

[source,cypher]
----
CALL apoc.ml.openai.completion('\n\nHuman: What color is sky?\n\nAssistant:',
$anthropicApiKey,
{apiType: 'ANTHROPIC', max_tokens_to_sample: 3}
)
----

.Example result
[%autowidth, opts=header]
|===
| value
| {
"id": "compl_015yzL9jDdMQnLSN3jkQifZt",
"stop": null,
"model": "claude-2.1",
"stop_reason": "max_tokens",
"type": "completion",
"completion": " The sky is",
"log_id": "compl_015yzL9jDdMQnLSN3jkQifZt"
}
|===


And also, we can specify the API version via `anthropic-version` configuration parameter, like the above example with the apoc.ml.openai.chat procedure.


[NOTE]
====
At the moment Anthropic does not support embedding API.
And at the time, payload with https://docs.anthropic.com/en/api/messages-streaming[`stream: true`] is not supported, since the result of apoc.ml.openai must be a JSON.
====

3 changes: 3 additions & 0 deletions extended/src/main/java/apoc/ml/MLUtil.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,7 @@ public class MLUtil {
public static final String API_VERSION_CONF_KEY = "apiVersion";
public static final String REGION_CONF_KEY = "region";
public static final String MODEL_CONF_KEY = "model";
public static final String MAX_TOKENS = "max_tokens";
public static final String MAX_TOKENS_TO_SAMPLE = "max_tokens_to_sample";
public static final String ANTHROPIC_VERSION = "anthropic-version";
}
71 changes: 49 additions & 22 deletions extended/src/main/java/apoc/ml/OpenAI.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,34 +64,22 @@ static Stream<Object> executeRequest(String apiKey, Map<String, Object> configur
);
OpenAIRequestHandler.Type type = OpenAIRequestHandler.Type.valueOf(apiTypeString.toUpperCase(Locale.ENGLISH));

var config = new HashMap<>(configuration);
// we remove these keys from config, since the json payload is calculated starting from the config map
Stream.of(ENDPOINT_CONF_KEY, API_TYPE_CONF_KEY, API_VERSION_CONF_KEY, APIKEY_CONF_KEY).forEach(config::remove);

switch (type) {
case MIXEDBREAD_CUSTOM -> {
// no payload manipulation, taken from the configuration as-is
}
case HUGGINGFACE -> {
config.putIfAbsent("inputs", inputs);
jsonPath = "$[0]";
}
default -> {
config.putIfAbsent(MODEL_CONF_KEY, model);
config.put(key, inputs);
}
}

var configForPayload = new HashMap<>(configuration);
// we remove these keys from configPayload, since the json payload is calculated starting from the configPayload map
Stream.of(ENDPOINT_CONF_KEY, API_TYPE_CONF_KEY, API_VERSION_CONF_KEY, APIKEY_CONF_KEY).forEach(configForPayload::remove);

final Map<String, Object> headers = new HashMap<>();

handleAPIProvider(type, configuration, path, model, key, inputs, configForPayload, headers);

path = (String) configuration.getOrDefault(PATH_CONF_KEY, path);
OpenAIRequestHandler apiType = type.get();

jsonPath = (String) configuration.getOrDefault(JSON_PATH_CONF_KEY, jsonPath);
path = (String) configuration.getOrDefault(PATH_CONF_KEY, path);

final Map<String, Object> headers = new HashMap<>();
headers.put("Content-Type", "application/json");
apiType.addApiKey(headers, apiKey);

String payload = JsonUtil.OBJECT_MAPPER.writeValueAsString(config);
String payload = JsonUtil.OBJECT_MAPPER.writeValueAsString(configForPayload);

// new URL(endpoint), path) can produce a wrong path, since endpoint can have for example embedding,
// eg: https://my-resource.openai.azure.com/openai/deployments/apoc-embeddings-model
Expand All @@ -100,6 +88,45 @@ static Stream<Object> executeRequest(String apiKey, Map<String, Object> configur
return JsonUtil.loadJson(url, headers, payload, jsonPath, true, List.of(), urlAccessChecker);
}

private static void handleAPIProvider(OpenAIRequestHandler.Type type,
Map<String, Object> configuration,
String path,
String model,
String key,
Object inputs,
HashMap<String, Object> configForPayload,
Map<String, Object> headers) {
switch (type) {
case MIXEDBREAD_CUSTOM -> {
// no payload manipulation, taken from the configuration as-is
}
case HUGGINGFACE -> {
configForPayload.putIfAbsent("inputs", inputs);
configuration.putIfAbsent(JSON_PATH_CONF_KEY, "$[0]");
}
case ANTHROPIC -> {
headers.putIfAbsent(ANTHROPIC_VERSION, configuration.getOrDefault(ANTHROPIC_VERSION, "2023-06-01"));

if (path.equals("completions")) {
configuration.putIfAbsent(PATH_CONF_KEY, "complete");
configForPayload.putIfAbsent(MAX_TOKENS_TO_SAMPLE, 1000);
configForPayload.putIfAbsent(MODEL_CONF_KEY, "claude-2.1");
} else {
configuration.putIfAbsent(PATH_CONF_KEY, "messages");
configForPayload.putIfAbsent(MAX_TOKENS, 1000);
configForPayload.putIfAbsent(MODEL_CONF_KEY, "claude-3-5-sonnet-20240620");
}

configForPayload.remove(ANTHROPIC_VERSION);
configForPayload.put(key, inputs);
}
default -> {
configForPayload.putIfAbsent(MODEL_CONF_KEY, model);
configForPayload.put(key, inputs);
}
}
}

@Procedure("apoc.ml.openai.embedding")
@Description("apoc.openai.embedding([texts], api_key, configuration) - returns the embeddings for a given text")
public Stream<EmbeddingResult> getEmbedding(@Name("texts") List<String> texts, @Name("api_key") String apiKey, @Name(value = "configuration", defaultValue = "{}") Map<String, Object> configuration) throws Exception {
Expand Down
13 changes: 13 additions & 0 deletions extended/src/main/java/apoc/ml/OpenAIRequestHandler.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ enum Type {
HUGGINGFACE(new OpenAi(null)),
MIXEDBREAD_EMBEDDING(new OpenAi(MIXEDBREAD_BASE_URL)),
MIXEDBREAD_CUSTOM(new Custom()),
ANTHROPIC(new Anthropic()),
OPENAI(new OpenAi("https://api.openai.com/v1"));

private final OpenAIRequestHandler handler;
Expand Down Expand Up @@ -94,6 +95,18 @@ public void addApiKey(Map<String, Object> headers, String apiKey) {
}
}

static class Anthropic extends OpenAi {

public Anthropic() {
super("https://api.anthropic.com/v1");
}

@Override
public void addApiKey(Map<String, Object> headers, String apiKey) {
headers.put("x-api-key", apiKey);
}
}

static class Custom extends OpenAi {

public Custom() {
Expand Down
Loading

0 comments on commit b80a8bd

Please sign in to comment.