feat: Add retry with exponential back-off to PromptNode's OpenAI models

[vblagoje]

Related Issues

• fixes PromptNode should handle OpenAIRateLimitError with back-offs and retry inference attempt before giving up #3879

Proposed Changes:

Added a well-known (to us at least) and previously used decorator @retry_with_exponential_backoff(backoff_in_seconds=10, max_retries=5) that handles the OpenAIRateLimitError with retry and exponential backoff

How did you test it?

Manual tests by forcing OpenAIRateLimitError

Notes for the reviewer

We used this decorator in other OpenAI APIs, I checked it manually and didn't add new tests. We'll see how this works in our CI ;-)

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[ZanSara]

I like the concept a lot but I think the backoff parameters should be controllable by the users. Right now 10 and 5 are really hardcoded and quite invisible... They would be hard to change even with reflection! Can we find a way (any, really) to let the user control such values? I'd be happy with a hacky-ish solution too (env vars, module constants, ...)

[vblagoje]

In an ideal world, yes, but IMHO the cost-benefit of providing such a fine detail of control might be an overkill. IIRC, Bogdan and I worked on this feature; users have used it in other components (EmbeddingEncoder) for some time now, and we have yet to receive such a request. Let's hear the opinion of others and proceed forward. cc @bogdankostic @masci

[bogdankostic]

For context: we had a similar discussion in #3398.

[vblagoje]

I remember now, thanks! Maybe env variables can be used? I'll check tomorrow

[vblagoje]

Ok, I stand corrected. We can use env vars with defaults - and we should. See 962281a

[vblagoje]

What would be the best names for these two variables? Let's use them not only for OPENAI remote models but also for all remote model invocations. If we go down naming env vars for backoff and max retries for every model, we'll have many of these env vars.

[vblagoje]

@ZanSara @bogdankostic @sjrl @masci I came up with these names. Please feel free to suggest others. I liked this approach because all env vars are imported in prompt_node.py with from haystack.environment, which is nice. This PR will help the CI as well. Let's introduce the same env vars in other places but in a separate refactor PR.

[ZanSara]

Great! Looks like a good solution to me 😊

Just as a follow up, do we set these env vars to 0 in the CI? I think that might help speed the tests up a bit

[vblagoje]

Cool, I would leave them either as is or lower HAYSTACK_REMOTE_API_BACKOFF_SEC to 1 or 2 seconds and leave HAYSTACK_REMOTE_API_MAX_RETRIES at 5. If HAYSTACK_REMOTE_API_BACKOFF_SEC is 0, we will attempt another remote call almost immediately. Even with exponential back-off, such a small value would cause rapid-fire re-attempts. We can even leave it as is, and then if we notice issues, we can fix them.

[vblagoje]

Yeah, in fact let's set HAYSTACK_REMOTE_API_BACKOFF_SEC to 1 in CI