-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Cohere Provider #34921
Add Cohere Provider #34921
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need the .svg files to be added to this PR?
f21e41f
to
9a7f559
Compare
Yes, we do need to add them, as the provider dependency changes, the breeze output command interface also changes and that needs to be reflected. Also, they are auto-generated and enforced via |
tests/system/providers/cohere/example_cohere_embedding_operator.py
Outdated
Show resolved
Hide resolved
tests/system/providers/cohere/example_cohere_embedding_operator.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Josh Fell <[email protected]>
Co-authored-by: Josh Fell <[email protected]>
Co-authored-by: Josh Fell <[email protected]>
Co-authored-by: Hussein Awala <[email protected]>
Co-authored-by: Hussein Awala <[email protected]>
Co-authored-by: Pankaj Singh <[email protected]>
Co-authored-by: Josh Fell <[email protected]>
Co-authored-by: Josh Fell <[email protected]>
0bb4e54
to
7f1e0ae
Compare
* Add Cohere Provider * Add Cohere Provider * Move link to seealso sphinx directive * Updated check for parameters * Update dependency of the cohere * Move the dag out of rst and into system tests * Add dependency to of cohere python sdk * Add cache_property for the cohere client * Remove unwanted get_conn method * Add correct label to password field * Expose timeout, max_retries and api_url to user * Fix documentation * Update interface of CohereEmbeddingOperator operator * Updated testcases * Updated testcases * Fix static check and docs build * Update CONTRIBUTING.rst Co-authored-by: Josh Fell <[email protected]> * Update docs/apache-airflow-providers-cohere/operators/embedding.rst Co-authored-by: Josh Fell <[email protected]> * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Josh Fell <[email protected]> * Update airflow/providers/cohere/hooks/cohere.py Co-authored-by: Hussein Awala <[email protected]> * Update airflow/providers/cohere/CHANGELOG.rst Co-authored-by: Hussein Awala <[email protected]> * Address the PR comments * Resolve conflicts * Fix breaking tests * Fix static checks * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Pankaj Singh <[email protected]> * Fix docstring * Add note for initial release * Add security.rst file * Update airflow/providers/cohere/hooks/cohere.py * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Josh Fell <[email protected]> * Update docs/apache-airflow-providers-cohere/operators/embedding.rst Co-authored-by: Josh Fell <[email protected]> * Add ref to sequrity.rst * Update docs/apache-airflow-providers-cohere/security.rst * Add /changelog.rst * Resolve conflicts --------- Co-authored-by: Josh Fell <[email protected]> Co-authored-by: Hussein Awala <[email protected]> Co-authored-by: Pankaj Singh <[email protected]> Co-authored-by: Pankaj <[email protected]>
This PR is part of our larger effort to add first-class integrations to support LLMOps that was presented at Airflow Summit. This PR specifically adds the Cohere Provider. Cohere is a renowned platform offering a range of AI Models tailored for various NLP tasks. In this iteration, we are integrating with their Embeddings Model.
The primary objective of this Provider is to present users with an alternative embedding model. This allows them to generate vectors for their proprietary data, a pivotal step towards establishing integrations with LLM models like ChatGPT.
Example DAG:
The
CohereEmbeddingOperator
can accept either a list of strings or a callable returning a list of strings.Email Discussion related to the effort can be found here - https://lists.apache.org/thread/0d669fmy4hn29h5c0wj0ottdskd77ktp