Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc Quality Transform: update readme and add sample notebook #790

Merged
merged 9 commits into from
Nov 27, 2024
Merged

Doc Quality Transform: update readme and add sample notebook #790

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1a70530
update readme following template https://github.com/IBM/data-prep-kit…
dtsuzuku-ibm Nov 11, 2024
ecb87b0
fix typo and update description
dtsuzuku-ibm Nov 13, 2024
e3fae5d
add name/email of contributor
dtsuzuku-ibm Nov 13, 2024
170af4b
first version of a notebook
shahrokhDaijavad Nov 21, 2024
10851f6
fixed code_location
shahrokhDaijavad Nov 21, 2024
7545872
add link to jupyter notebook
dtsuzuku-ibm Nov 22, 2024
9ee506e
update notebook
dtsuzuku-ibm Nov 22, 2024
cf13388
stop installing data-prep-connector
dtsuzuku-ibm Nov 25, 2024
edb605b
use data-prep-toolkit-transforms==0.2.2.dev3
dtsuzuku-ibm Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 47 additions & 10 deletions transforms/language/doc_quality/python/README.md
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shahrokhDaijavad @dtsuzuku-ibm This shows how to run the example script once we have cloned the repo. I wonder if we should also add a section to it that would explain how to use it in a notebook or a python script without cloning the repo but only with pip install . i.e.:

!pip install data-prep-toolkit
!pip install data-prep-toolkit-transform[doc_quality]

from doc_quality_transform python import ...

params = {
...
}
...

laucher.launch()

Copy link
Collaborator Author

@dtsuzuku-ibm dtsuzuku-ibm Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@touma-I

Please add example. with a few lines of code on how one can do pip install, setup the parameter block for input/output folder and then invoke the runtime launcher using the default parameters . Thanks

We can do all of these in jupyter/collab notebook which will be added in the future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dtsuzuku-ibm . that is great. Once you have the notebook, you can reference it. For now, I don't think this is complete until we have one or the other or both.

Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
# Document Quality Transform

Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This transform will calculate and annotate several metrics related to document, which are usuful to see the quality of document.
## Description
This transform will calculate and annotate several metrics which are useful to assess the quality of the document.
The document quality transform operates on text documents only

### Input

In this transform, following metrics will be included:
| input column name | data type | description |
|-|-|-|
| the one specified in _doc_content_column_ configuration | string | text whose quality will be calculated by this transform |

### Output columns annotated by this transform

| output column name | data type | description | supported language |
|-|-|-|-|
Expand All @@ -27,7 +35,7 @@ In this transform, following metrics will be included:

You can see more detailed backgrounds of some columns in [Deepmind's Gopher paper](https://arxiv.org/pdf/2112.11446.pdf)

## Configuration and command line Options
## Configuration

The set of dictionary keys holding [DocQualityTransform](src/doc_quality_transform.py)
configuration for values are as follows:
Expand All @@ -36,13 +44,19 @@ configuration for values are as follows:
* _doc_content_column_ - specifies column name that contains document text. By default, "contents" is used.
* _bad_word_filepath_ - specifies a path to bad word file: local folder (file or directory) that points to bad word file. You don't have to set this parameter if you don't need to set bad words.

## Running
Example
```
{
text_lang_key: "en",
doc_content_column_key: "contents",
bad_word_filepath_key: os.path.join(basedir, "ldnoobw", "en"),
}
```

## Usage

### Launched Command Line Options
When running the transform with the Ray launcher (i.e. TransformLauncher),
the following command line arguments are available in addition to
the options provided by
the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md).
The following command line arguments are available
```
--docq_text_lang DOCQ_TEXT_LANG language used in the text content. By default, "en" is used.
--docq_doc_content_column DOCQ_DOC_CONTENT_COLUMN column name that contain document text. By default, "contents" is used.
Expand Down Expand Up @@ -70,14 +84,37 @@ ls output
```
To see results of the transform.

### Code example

TBD (link to the notebook will be provided)

### Transforming data using the transform image

To use the transform image to transform your data, please refer to the
[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
substituting the name of this transform image and runtime as appropriate.

## Testing

Following [the testing strategy of data-processing-lib](../../../../data-processing-lib/doc/transform-testing.md)

Currently we have:
- [Unit test](test/test_doc_quality_python.py)
- [Integration test](test/test_doc_quality.py)


## Further Resource

- For those who want to learn C4 heuristic rules
- https://arxiv.org/pdf/1910.10683.pdf
- For those who want to learn Gopher statistics
- https://arxiv.org/pdf/2112.11446.pdf
- For those who want to see the source of badwords used by default
- https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words


## Consideration

## Troubleshooting guide
### Troubleshooting guide

For M1 Mac user, if you see following error during make command, `error: command '/usr/bin/clang' failed with exit code 1`, you may better follow [this step](https://freeman.vc/notes/installing-fasttext-on-an-m1-mac)