Image Generation with NearAI defaults to OpenAI

[thomasjvu]

Is the near-eliza-starter intended to be integrated with Image Generation?

It seems like you brought up testing image generation in ElizaOS #3644 as well, and I noticed that there's an IMAGE_NEARAI_MODEL variable to work with, so I assume the image generation feature is supposed to work. However, I'm getting an error when trying to generate an image.

Reproduction Steps

1. Clone the near-eliza-starter project or create a new branch.
2. In the src/character.ts file, add this to the imports: import { imageGenerationPlugin } from "@elizaos/plugin-image-generation";
3. For the character's plugins key, add the imageGenerationPlugin to the array and remove any unnecessary ones.
4. Run Eliza with pnpm start and ask the agent to use the GENERATE_IMAGE action like: "can you GENERATE_IMAGE of a spider"
5. You will encounter an error generating the request. See the logs below.

Logs

Click to see the system logs

[2025-02-25 12:34:46] INFO: Eliza(b850bc30-45f8-0041-a00a-83df46d8555d) - Initializing AgentRuntime with options:
    character: "Eliza"
    modelProvider: "nearai"
    characterModelProvider: "nearai"
You: [2025-02-25 12:34:46] INFO: Eliza(b850bc30-45f8-0041-a00a-83df46d8555d) - Setting Model Provider:
    characterModelProvider: "nearai"
    optsModelProvider: "nearai"
    finalSelection: "nearai"
[2025-02-25 12:34:46] INFO: Eliza(b850bc30-45f8-0041-a00a-83df46d8555d) - Selected model provider: nearai
[2025-02-25 12:34:46] INFO: Eliza(b850bc30-45f8-0041-a00a-83df46d8555d) - Selected image model provider: nearai
[2025-02-25 12:34:46] INFO: Eliza(b850bc30-45f8-0041-a00a-83df46d8555d) - Selected image vision model provider: nearai
[2025-02-25 12:34:46] INFO: Initializing LlamaService...

You: can you GENERATE_IMAGE of a spider
[2025-02-25 12:35:03] INFO: Generating text with options:
    modelProvider: "nearai"
    model: "large"
    verifiableInference: false
[2025-02-25 12:35:03] INFO: Selected model: fireworks::accounts/fireworks/models/llama-v3p1-405b-instruct
Received response from OpenAI model.
[2025-02-25 12:35:06] INFO: Executing handler for action: GENERATE_IMAGE
[2025-02-25 12:35:06] INFO: Generating text with options:
    modelProvider: "nearai"
    model: "medium"
    verifiableInference: false
[2025-02-25 12:35:06] INFO: Selected model: fireworks::accounts/fireworks/models/llama-v3p1-70b-instruct
Received response from OpenAI model.
[2025-02-25 12:35:08] INFO: Generating image with options:
    imageModelProvider: "fireworks::accounts/fireworks/models/playground-v2-5-1024px-aesthetic"
BadRequestError: 400 Invalid model fireworks::accounts/fireworks/models/playground-v2-5-1024px-aesthetic. The model argument should be left blank.
    at APIError.generate (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/error.mjs:41:20)
    at OpenAI.makeStatusError (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/core.mjs:286:25)
    at OpenAI.makeRequest (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/core.mjs:330:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async generateImage (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected]_op_tugtwojwdqumroqz55lkhdu3lm/node_modules/@near-agent/elizaos-core/dist/index.js:37578:24)
    at async Object.handler (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected]_op_ccoqo67yiwazoabsyl2zvw5o6q/node_modules/@elizaos/plugin-image-generation/dist/index.js:193:20)
    at async AgentRuntime.processActions (file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected]_op_tugtwojwdqumroqz55lkhdu3lm/node_modules/@near-agent/elizaos-core/dist/index.js:39729:9)
    at async file:///Users/area/Desktop/repos/near-eliza-starter/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]._twexot5b76headzmcwexq6tdoi/node_modules/@elizaos/client-direct/dist/index.js:4655:9 {
  status: 400,
  headers: {
    'access-control-allow-origin': '*',
    'alt-svc': 'h3=":443"; ma=86400',
    'cf-cache-status': 'DYNAMIC',
    'cf-ray': '9177bbcce8538d15-BOS',
    connection: 'keep-alive',
    'content-length': '238',
    'content-type': 'application/json',
    date: 'Tue, 25 Feb 2025 12:35:09 GMT',
    'openai-organization': 'ain-software',
    'openai-processing-ms': '36',
    'openai-version': '2020-10-01',
    server: 'cloudflare',
    'set-cookie': '__cf_bm=w0mIoFJOwGdlBI7Eqdyz2OV66WY3VBO7Z8K4.l4kjVg-1740486909-1.0.1.1-bMDZ0hWofnYmmw4r_erXpyK5uv.AjMCS70z_HGc7VmOB2zUZ6hCG.AoS4WCYUz_pqRBANVliZF2mzYvByfCdmA; path=/; expires=Tue, 25-Feb-25 13:05:09 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None, _cfuvid=VfbrCfWcr_mkQTdxDvo6O6ZRhstXe9Vs8kS.MJNVSxU-1740486909064-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None',
    'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
    'x-content-type-options': 'nosniff',
    'x-request-id': 'req_2edc939475eec9949fdb0d2ff55c9f5f'
  },
  request_id: 'req_2edc939475eec9949fdb0d2ff55c9f5f',
  error: {
    code: null,
    message: 'Invalid model fireworks::accounts/fireworks/models/playground-v2-5-1024px-aesthetic. The model argument should be left blank.',
    param: null,
    type: 'invalid_request_error'
  },
  code: null,
  param: null,
  type: 'invalid_request_error'
}
[2025-02-25 12:35:08] ERROR: Image generation failed or returned no data.

Additional Information

When testing, I tried having both the multichainPlugin and nearPlugin extensions enabled in my character.ts file, as well as having them both disabled but the result was the same.

I did this because other users have reported that the nearPlugin caused issues with imageGeneration as noted in ElizaOS #1183 and ElizaOS #1935.

[think-in-universe]

@thomasjvu thanks for reporting the issue!

I have tested and it's indeed a bug in our implementation in @elizaos/core for image generation.

I'll propose a fix tomorrow and update in this repo.