Installation issue.

[sandeep-krutrim]

Getting this error when using install.sh

`Successfully installed absl-py-2.1.0 aiofiles-23.2.1 altair-5.3.0 annotated-types-0.7.0 anyio-4.4.0 astunparse-1.6.3 attrs-23.2.0 click-7.1.2 colorama-0.4.6 contourpy-1.2.1 ctranslate2-3.9.0 cycler-0.12.1 dill-0.3.8 docopt-0.6.2 exceptiongroup-1.2.1 fastapi-0.110.3 ffmpy-0.3.2 flatbuffers-24.3.25 fonttools-4.53.0 future-1.0.0 gast-0.6.0 google-pasta-0.2.0 gradio-4.26.0 gradio-client-0.15.1 grpcio-1.64.1 h11-0.14.0 h5py-3.11.0 httpcore-1.0.5 httpx-0.27.0 huggingface-hub-0.23.4 importlib-resources-6.4.0 joblib-1.4.2 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 keras-3.4.1 kiwisolver-1.4.5 libclang-18.1.1 lxml-5.2.2 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.9.0 mdurl-0.1.2 ml-dtypes-0.3.2 mock-5.1.0 mosestokenizer-1.2.1 namex-0.0.8 nltk-3.8.1 numpy-1.26.4 openfile-0.0.7 opt-einsum-3.3.0 optree-0.11.0 orjson-3.10.6 pillow-10.4.0 portalocker-2.10.0 promise-2.3 protobuf-3.20.3 pydantic-2.8.2 pydantic-core-2.20.1 pydub-0.25.1 pyparsing-3.1.2 python-multipart-0.0.9 pyyaml-6.0.1 referencing-0.35.1 regex-2024.5.15 rich-13.7.1 rpds-py-0.18.1 ruff-0.5.0 sacrebleu-2.3.1 sacremoses-0.1.1 semantic-version-2.10.0 shellingham-1.5.4 sniffio-1.3.1 starlette-0.37.2 tabulate-0.9.0 tensorboard-2.16.2 tensorboard-data-server-0.7.2 tensorflow-2.16.2 tensorflow-addons-0.23.0 tensorflow-datasets-3.2.1 tensorflow-io-gcs-filesystem-0.37.1 tensorflow-metadata-1.15.0 termcolor-2.4.0 tf2crf-0.1.33 tokenizers-0.13.3 tomlkit-0.12.0 toolwrapper-2.1.0 toolz-0.12.1 tqdm-4.66.4 transformers-4.28.1 typeguard-2.13.3 typer-0.10.0 uctools-1.3.0 urduhack-1.1.1 uvicorn-0.30.1 websockets-11.0.3 werkzeug-3.0.3 wrapt-1.16.0
2024-07-04 06:04:54.259356: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-04 06:04:54.261982: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-04 06:04:54.272369: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:479] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-04 06:04:54.293163: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:10575] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-04 06:04:54.293203: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1442] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-04 06:04:54.305759: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-04 06:04:54.937893: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:

TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).

For more information see: tensorflow/addons#2807

warnings.warn(
/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.13.0 and strictly below 2.16.0 (nightly versions are not supported).
The versions of TensorFlow you are currently using is 2.16.2 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
warnings.warn(
Traceback (most recent call last):
File "", line 1, in
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/init.py", line 6, in
from .pipeline import Pipeline
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/init.py", line 4, in
from .core import Pipeline
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/core.py", line 7, in
from .parsers.ner import NerParser
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/parsers/ner.py", line 4, in
from urduhack.models.ner import predict_ner
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/init.py", line 4, in
from .predict import predict_ner
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/predict.py", line 11, in
from urduhack.models.ner.model import _bi_lstm_crf_model
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/model.py", line 5, in
from tf2crf import CRF
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tf2crf/init.py", line 2, in
from .crf import CRF
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tf2crf/crf.py", line 2, in
import tensorflow_addons as tfa
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/init.py", line 23, in
from tensorflow_addons import activations
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/activations/init.py", line 17, in
from tensorflow_addons.activations.gelu import gelu
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/activations/gelu.py", line 19, in
from tensorflow_addons.utils.types import TensorLike
File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/types.py", line 29, in
from keras.src.engine import keras_tensor
ModuleNotFoundError: No module named 'keras.src.engine'
[nltk_data] Downloading package punkt to
[nltk_data] /home/sandeep.pandey/nltk_data...
[nltk_data] Package punkt is already up-to-date!
Collecting sentencepiece
Using cached sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Using cached sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Installing collected packages: sentencepiece
Successfully installed sentencepiece-0.2.0
fatal: destination path 'fairseq' already exists and is not an empty directory.
Processing /home/sandeep.pandey/local_setup/new/IndicEmbeddings/llm2vec_modified/translation/new/IndicTrans2/fairseq
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting cffi (from fairseq==0.12.2)
Using cached cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting cython (from fairseq==0.12.2)
Using cached Cython-3.0.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
Collecting hydra-core<1.1,>=1.0.7 (from fairseq==0.12.2)
Using cached hydra_core-1.0.7-py3-none-any.whl.metadata (3.7 kB)
Collecting omegaconf<2.1 (from fairseq==0.12.2)
Using cached omegaconf-2.0.6-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.6 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/d0/eb/9d63ce09dd8aa85767c65668d5414958ea29648a0eec80a4a7d311ec2684/omegaconf-2.0.6-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.5-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.5 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/e5/f6/043b6d255dd6fbf2025110cea35b87f4c5100a181681d8eab496269f0d5b/omegaconf-2.0.5-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.4-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.4 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/92/b1/4f3023143436f12c98bab53f0b3db617bd18a7d223627d5030e13a7b4fc2/omegaconf-2.0.4-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.3-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.3 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/29/08/a88210c2c1aa0a3f65f05d8a6c98939ccb84b6fb982aa6567dec4e6773f9/omegaconf-2.0.3-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.2-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.2 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/72/fe/f8d162aa059fb4f327fd75144dd69aa7e8acbb6d8d37013e4638c8490e0b/omegaconf-2.0.2-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.1-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.1 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/86/ec/605805e60abdb025b06664d107335031bb8ebdc52e0a90bdbad6a7130279/omegaconf-2.0.1-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.0-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: numpy>=1.21.3 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (1.26.4)
Requirement already satisfied: regex in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2024.5.15)
Requirement already satisfied: sacrebleu>=1.4.12 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2.3.1)
Requirement already satisfied: torch>=1.13 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2.3.1+cu118)
Requirement already satisfied: tqdm in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (4.66.4)
Collecting bitarray (from fairseq==0.12.2)
Using cached bitarray-2.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting torchaudio>=0.8.0 (from fairseq==0.12.2)
Using cached torchaudio-2.3.1-cp39-cp39-manylinux1_x86_64.whl.metadata (6.4 kB)
Collecting scikit-learn (from fairseq==0.12.2)
Using cached scikit_learn-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Requirement already satisfied: packaging in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (24.1)
Collecting omegaconf<2.1 (from fairseq==0.12.2)
Using cached omegaconf-2.0.6-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.6 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/d0/eb/9d63ce09dd8aa85767c65668d5414958ea29648a0eec80a4a7d311ec2684/omegaconf-2.0.6-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.)
~~~~~~^
Please use pip<24.1 if you need to use this version.
Using cached omegaconf-2.0.5-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.5 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/e5/f6/043b6d255dd6fbf2025110cea35b87f4c5100a181681d8eab496269f0d5b/omegaconf-2.0.5-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators
PyYAML (>=5.1.*)
~~~~~~^
Please use pip<24.1 if you need to use this version.
INFO: pip is looking at multiple versions of hydra-core to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install fairseq and fairseq==0.12.2 because these package versions have conflicting dependencies.

The conflict is caused by:
fairseq 0.12.2 depends on omegaconf<2.1
hydra-core 1.0.7 depends on omegaconf<2.1 and >=2.0.5

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Setup completed!`

[cmrajan]

Pin the pip version to 24.0 and try. You can change below line to python3 -m pip install --upgrade pip==24.0

IndicTrans2/install.sh

Line 15 in f897168

python3 -m pip install --upgrade pip

[Kishlay-notabot]

Hi! I have been facing the same issue, and it is related to fairseq not installing correctly. The omegaconf related dependency issue still persists. Please specify the python version and the pip version you guys have used so that I can replicate it without facing errors

[Kishlay-notabot]

I can't run the custom pipeline, but I am able to run the Huggingface transformer Google Colab code because of the fairseq issue.