Added ctranslate2 translation example script #83

uahmed93 · 2024-09-11T16:40:12Z

Added a ctransalte2 example which works on string tokens instead of integer tokens.

To run the example :

python3 example/custom_ct2_model.py --ct2-model-dir <your-modelp-dir> inp.parquet out.parquet

copy-pr-bot · 2024-09-11T16:40:15Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

uahmed93 · 2024-09-11T16:44:02Z

This script is breaking because it is unable to serialize ct2 model while loading on worker. I have run this after initializing NDC gpu dask cluster on slurm but this is not working here.

VibhuJawa · 2024-09-11T17:10:33Z

@uahmed93 , Can you post the error you saw here please.

uahmed93 · 2024-09-11T17:59:06Z

Here:

Deployed LocalCUDACluster(22f3d4fd, 'tcp://127.0.0.1:37693', workers=1, threads=1, memory=1.79 TiB)...
2024-09-11 10:56:33,005 - distributed.protocol.pickle - ERROR - Failed to serialize <ToPickle: HighLevelGraph with 4 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x15522cf81900>
 0. read-parquet-2f062a30cf4676cd0b9a4ab5cf06fe85
 1. repartition-2-aa356a5cb4dedecfbdd625f941566718
 2. to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1
 3. store-to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1
>.
Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
TypeError: cannot pickle 'ctranslate2._ext.Translator' object
Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 353, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 76, in pickle_dumps
    frames[0] = pickle.dumps(
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
TypeError: cannot pickle 'ctranslate2._ext.Translator' object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/ctransl/ctransl_cf.py", line 145, in <module>
    main()
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/ctransl/ctransl_cf.py", line 141, in main
    outputs.to_parquet(args.output_parquet_path)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/nvtx/nvtx.py", line 116, in inner
    result = func(*args, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask_cudf/core.py", line 264, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/dataframe/io/parquet/core.py", line 1047, in to_parquet
    out = out.compute(**compute_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/base.py", line 379, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/base.py", line 665, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 379, in serialize
    raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 4 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x15522cf81900>\n 0. read-parquet-2f062a30cf4676cd0b9a4ab5cf06fe85\n 1. repartition-2-aa356a5cb4dedecfbdd625f941566718\n 2. to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1\n 3. store-to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1\n>')

VibhuJawa

Thanks, @uahmed93, for implementing this feature. The overall approach looks solid, but I've left some suggestions to enhance the code's readability and design.

The foundation is strong, but with a few refinements, it can be polished for better performance and maintainability.

examples/custom_ct2_model.py

crossfit/backend/torch/op/base.py

VibhuJawa · 2024-09-17T06:20:44Z

Also , CC: @sarahyurick for suggestions .

crossfit/backend/torch/op/base.py

examples/custom_ct2_model.py

crossfit/backend/torch/op/base.py

examples/custom_ct2_model.py

sarahyurick

Thanks @uahmed93 for the updates! I ran black and there are several lines which need to be reformatted, so I added them here.

crossfit/backend/torch/model.py

crossfit/backend/torch/op/base.py

examples/custom_ct2_model.py

sarahyurick · 2024-09-18T23:18:35Z

Also @uahmed93, I was wondering if I could test this PR myself by downloading the model files at https://huggingface.co/ai4bharat/indictrans2-en-indic-1B? Or should I be using something else?

uahmed93 · 2024-09-19T17:13:27Z

Thanks @VibhuJawa and @sarahyurick for detailed analysis. I have now added get_model_output function in Model class to handle postprocessing of outputs as suggested by @VibhuJawa .
I have run black for formatting now.

uahmed93 · 2024-09-19T17:17:01Z

@sarahyurick that is hf model. CT2 model can be downloaded from here Base En-indic model.
After downloading you need to unzip and there will be ct2_fp16 model. That model path need to be provided.

sarahyurick

Tested and it worked for me. Thanks @uahmed93 !

VibhuJawa

Please add pytests, mostly other things look good to me.

crossfit/backend/torch/model.py

examples/custom_ct2_model.py

uahmed93 · 2024-09-23T17:28:11Z

Hi, @VibhuJawa @sarahyurick
Sorry was doing some other priority task due to which I could not handle the reviews immediately.
I have added test case here and corrected some suggestions. can please review them?.

examples/custom_ct2_model.py

VibhuJawa

Last final bits of review which i missed in the initial review, we should be good to merge after these are fixed.

examples/custom_ct2_model.py

tests/op/test_model_function.py

Signed-off-by: Ahmed Umair <[email protected]>

to allow string output types. Signed-off-by: Ahmed Umair <[email protected]>

handle postprocessing of outputs there only. Signed-off-by: Ahmed Umair <[email protected]>

Handle some review comments. Signed-off-by: Ahmed Umair <[email protected]>

changed to type checking instead of string matching in Model. Signed-off-by: Ahmed Umair <[email protected]>

…t function. Signed-off-by: Ahmed Umair <[email protected]>

Signed-off-by: Ahmed Umair <[email protected]>

uahmed93 · 2024-09-25T18:46:35Z

Here:

Deployed LocalCUDACluster(22f3d4fd, 'tcp://127.0.0.1:37693', workers=1, threads=1, memory=1.79 TiB)...
2024-09-11 10:56:33,005 - distributed.protocol.pickle - ERROR - Failed to serialize <ToPickle: HighLevelGraph with 4 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x15522cf81900>
 0. read-parquet-2f062a30cf4676cd0b9a4ab5cf06fe85
 1. repartition-2-aa356a5cb4dedecfbdd625f941566718
 2. to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1
 3. store-to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1
>.
Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
TypeError: cannot pickle 'ctranslate2._ext.Translator' object
Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
AttributeError: Can't pickle local object 'to_parquet.<locals>.<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 353, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 76, in pickle_dumps
    frames[0] = pickle.dumps(
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
TypeError: cannot pickle 'ctranslate2._ext.Translator' object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/ctransl/ctransl_cf.py", line 145, in <module>
    main()
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/ctransl/ctransl_cf.py", line 141, in main
    outputs.to_parquet(args.output_parquet_path)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/nvtx/nvtx.py", line 116, in inner
    result = func(*args, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask_cudf/core.py", line 264, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/dataframe/io/parquet/core.py", line 1047, in to_parquet
    out = out.compute(**compute_kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/base.py", line 379, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/dask/base.py", line 665, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/lustre/fsw/portfolios/llmservice/users/uahmed/test_env/lib/python3.10/site-packages/distributed/protocol/serialize.py", line 379, in serialize
    raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 4 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x15522cf81900>\n 0. read-parquet-2f062a30cf4676cd0b9a4ab5cf06fe85\n 1. repartition-2-aa356a5cb4dedecfbdd625f941566718\n 2. to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1\n 3. store-to-parquet-03d4d012cafd9ed3bdb4ee7374761cf1\n>')

Resolved it.

VibhuJawa

Last nits around testing

crossfit/backend/torch/model.py

tests/op/test_model_function.py

Signed-off-by: Ahmed Umair <[email protected]>

VibhuJawa · 2024-09-27T01:25:04Z

/okay to test

VibhuJawa · 2024-09-27T01:44:36Z

@uahmed93 , Please fix linting issue.

Signed-off-by: Ahmed Umair <[email protected]>

VibhuJawa · 2024-09-27T17:01:14Z

/okay to test

uahmed93 mentioned this pull request Sep 16, 2024

Added a translation pipeline for ctranslate2 inference NVIDIA/NeMo-Curator#245

Draft

3 tasks

VibhuJawa requested changes Sep 17, 2024

View reviewed changes

examples/custom_ct2_model.py Outdated Show resolved Hide resolved

crossfit/backend/torch/op/base.py Outdated Show resolved Hide resolved

VibhuJawa requested a review from sarahyurick September 17, 2024 06:20

sarahyurick suggested changes Sep 17, 2024

View reviewed changes

sarahyurick mentioned this pull request Sep 18, 2024

[FEA] Support for ctranslate2 model translation. #82

Closed

sarahyurick suggested changes Sep 18, 2024

View reviewed changes

sarahyurick approved these changes Sep 19, 2024

View reviewed changes

VibhuJawa requested changes Sep 20, 2024

View reviewed changes

crossfit/backend/torch/model.py Outdated Show resolved Hide resolved

crossfit/backend/torch/model.py Outdated Show resolved Hide resolved

VibhuJawa reviewed Sep 20, 2024

View reviewed changes

crossfit/backend/torch/model.py Outdated Show resolved Hide resolved

VibhuJawa reviewed Sep 20, 2024

View reviewed changes

examples/custom_ct2_model.py Show resolved Hide resolved

VibhuJawa reviewed Sep 20, 2024

View reviewed changes

examples/custom_ct2_model.py Outdated Show resolved Hide resolved

VibhuJawa reviewed Sep 23, 2024

View reviewed changes

examples/custom_ct2_model.py Outdated Show resolved Hide resolved

VibhuJawa reviewed Sep 23, 2024

View reviewed changes

examples/custom_ct2_model.py Outdated Show resolved Hide resolved

VibhuJawa requested changes Sep 24, 2024

View reviewed changes

examples/custom_ct2_model.py Outdated Show resolved Hide resolved

tests/op/test_model_function.py Outdated Show resolved Hide resolved

tests/op/test_model_function.py Show resolved Hide resolved

tests/op/test_model_function.py Show resolved Hide resolved

Umair Ahmed added 6 commits September 25, 2024 23:21

Added ctranslate2 translation example script

bb69676

Signed-off-by: Ahmed Umair <[email protected]>

Added model_output_type attribute in base Model class

b023c34

to allow string output types. Signed-off-by: Ahmed Umair <[email protected]>

Added get_model_output function in base Model class to

4550597

handle postprocessing of outputs there only. Signed-off-by: Ahmed Umair <[email protected]>

Added test_tokenize.py to test get_model_output from Model class.

a27faeb

Handle some review comments. Signed-off-by: Ahmed Umair <[email protected]>

calling load_cfg() when config is required.

5c9248c

changed to type checking instead of string matching in Model. Signed-off-by: Ahmed Umair <[email protected]>

Due to cyclic import, changed import statement inside get_model_outpu…

476b335

…t function. Signed-off-by: Ahmed Umair <[email protected]>

uahmed93 force-pushed the uahmed/ct2_example branch from 9929be2 to 476b335 Compare September 25, 2024 17:51

Added review comment in test functions and in custom_ct2_model.py

22844c7

Signed-off-by: Ahmed Umair <[email protected]>

uahmed93 mentioned this pull request Sep 25, 2024

Added example notebook for translation with ct2 model. NVIDIA/NeMo-Curator#262

Draft

3 tasks

VibhuJawa marked this pull request as ready for review September 25, 2024 23:05

VibhuJawa requested changes Sep 25, 2024

View reviewed changes

crossfit/backend/torch/model.py Show resolved Hide resolved

tests/op/test_model_function.py Outdated Show resolved Hide resolved

tests/op/test_model_function.py Outdated Show resolved Hide resolved

Umair Ahmed and others added 2 commits September 26, 2024 20:57

Changed input_ids from test function.

febc867

Signed-off-by: Ahmed Umair <[email protected]>

Merge branch 'main' into uahmed/ct2_example

4f82e86

VibhuJawa added the enhancement New feature or request label Sep 27, 2024

VibhuJawa linked an issue Sep 27, 2024 that may be closed by this pull request

[FEA] Support for ctranslate2 model translation. #82

Closed

Resolved linting issue.

a170248

Signed-off-by: Ahmed Umair <[email protected]>

VibhuJawa merged commit 23af498 into rapidsai:main Sep 27, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added ctranslate2 translation example script #83

Added ctranslate2 translation example script #83

uahmed93 commented Sep 11, 2024

copy-pr-bot bot commented Sep 11, 2024

uahmed93 commented Sep 11, 2024

VibhuJawa commented Sep 11, 2024

uahmed93 commented Sep 11, 2024

VibhuJawa left a comment

VibhuJawa commented Sep 17, 2024

sarahyurick left a comment

sarahyurick commented Sep 18, 2024

uahmed93 commented Sep 19, 2024

uahmed93 commented Sep 19, 2024

sarahyurick left a comment

VibhuJawa left a comment

uahmed93 commented Sep 23, 2024

VibhuJawa left a comment

uahmed93 commented Sep 25, 2024

VibhuJawa left a comment

VibhuJawa commented Sep 27, 2024

VibhuJawa commented Sep 27, 2024

VibhuJawa commented Sep 27, 2024

Added ctranslate2 translation example script #83

Added ctranslate2 translation example script #83

Conversation

uahmed93 commented Sep 11, 2024

copy-pr-bot bot commented Sep 11, 2024

uahmed93 commented Sep 11, 2024

VibhuJawa commented Sep 11, 2024

uahmed93 commented Sep 11, 2024

VibhuJawa left a comment

Choose a reason for hiding this comment

VibhuJawa commented Sep 17, 2024

sarahyurick left a comment

Choose a reason for hiding this comment

sarahyurick commented Sep 18, 2024

uahmed93 commented Sep 19, 2024

uahmed93 commented Sep 19, 2024

sarahyurick left a comment

Choose a reason for hiding this comment

VibhuJawa left a comment

Choose a reason for hiding this comment

uahmed93 commented Sep 23, 2024

VibhuJawa left a comment

Choose a reason for hiding this comment

uahmed93 commented Sep 25, 2024

VibhuJawa left a comment

Choose a reason for hiding this comment

VibhuJawa commented Sep 27, 2024

VibhuJawa commented Sep 27, 2024

VibhuJawa commented Sep 27, 2024