[Improve][Doc] Add file_filter_pattern example to doc
#7922
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1、When the ClickHouse connector is set to multi parallelism, the task extraction is completed but cannot be stopped normally [(#7897)](#7897) 2、Added E2E test cases for this issue [(#7897)](#7897) 3、Local developers want to observe **Job Progress Information** in a timely manner, Need to modify the following configuration.The configuration in config is invalid ``` seatunnel engine/seatunnel-engineer-common/src/main/resources/seatunnely.yaml ```
1、When the ClickHouse connector is set to multi parallelism, the task extraction is completed but cannot be stopped normally [(#7897)](#7897) 2、Added E2E test cases for this issue [(#7897)](#7897) 3、Local developers want to observe **Job Progress Information** in a timely manner, Need to modify the following configuration.The configuration in config is invalid ``` seatunnel engine/seatunnel-engineer-common/src/main/resources/seatunnely.yaml ```
1、When the ClickHouse connector is set to multi parallelism, the task extraction is completed but cannot be stopped normally [(#7897)](#7897) 2、Added E2E test cases for this issue [(#7897)](#7897) 3、Local developers want to observe **Job Progress Information** in a timely manner, Need to modify the following configuration.The configuration in config is invalid ``` seatunnel engine/seatunnel-engineer-common/src/main/resources/seatunnely.yaml ```
This reverts commit 2e9162d.
This reverts commit e64b8a6.
This reverts commit 1b80667.
This reverts commit 4276681.
Hisoka-X
reviewed
Oct 28, 2024
| @@ -254,6 +254,72 @@ Specifies Whether to process data using the tag attribute format. | |||
|
|
|||
| Filter pattern, which used for filtering files. | |||
|
|
|||
| The filtering format is similar to wildcard matching file names in Linux. | |||
There was a problem hiding this comment.
We cannot tell users about the ambiguous conclusion. Please tell users directly that we use Java regular expressions.
Comment on lines
259
to
270
| | Wildcard | Meaning | Example | | ||
| |--------------|--------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | * | Match 0 or more characters | f*     Any file starting with f<br/>b*.txt   Any file starting with b, any character in the middle, and ending with. txt | | ||
| | [] | Match a single character in parentheses | [abc]*   A file that starts with any one of the characters a, b, or c | | ||
| | ? | Match any single character | f?.txt   Any file starting with 'f' followed by a character and ending with '. txt' | | ||
| | [!] | Match any single character not in parentheses | [!abc]*   Any file that does not start with abc | | ||
| | [a-z] | Match any single character from a to z | [a-z]*   Any file starting with a to z | | ||
| | {a,b,c}/a..z | When separated by commas, it represents individual characters<br/>When separated by two dots, represents continuous characters | {a,b,c}*   Files starting with any character from abc<br/>{a..Z}*    Files starting with any character from a to z | | ||
|
|
||
| However, it should be noted that unlike Linux wildcard characters, when encountering file suffixes, the middle dot cannot be omitted. | ||
|
|
||
| For example, `abc20241022.csv`, the normal Linux wildcard `abc*` is sufficient, but here we need to use `abc*.*` , Pay attention to a point in the middle. |
There was a problem hiding this comment.
Please replace this part to link to https://en.wikipedia.org/wiki/Regular_expression. Let user to learn regular itself.
Comment on lines
274
to
287
| report.txt | ||
| notes.txt | ||
| input.csv | ||
| abch20241022.csv | ||
| abcw20241022.csv | ||
| abcx20241022.csv | ||
| abcq20241022.csv | ||
| abcg20241022.csv | ||
| abcv20241022.csv | ||
| abcb20241022.csv | ||
| old_data.csv | ||
| logo.png | ||
| script.sh | ||
| helpers.sh |
There was a problem hiding this comment.
Please add some file path, not only match file name.
Optimize the describe about Regex
Hisoka-X
reviewed
Oct 28, 2024
| @@ -254,6 +254,54 @@ Specifies Whether to process data using the tag attribute format. | |||
|
|
|||
| Filter pattern, which used for filtering files. | |||
|
|
|||
| The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. learn it | |||
There was a problem hiding this comment.
Suggested change
| The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. learn it | |
| The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. |
| @@ -254,6 +254,54 @@ Specifies Whether to process data using the tag attribute format. | |||
|
|
|||
| Filter pattern, which used for filtering files. | |||
|
|
|||
| The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. learn it | |||
|
|
|||
| File Structure Example: | |||
There was a problem hiding this comment.
Suggested change
| File Structure Example: | |
| There are some examples. | |
| File Structure Example: |
Optimize document structure
Optimize document structure
|
@Hisoka-X Please review this |
Please provide a description of all connectors that support the file_filter_pattern parameter
Added the following file connector description about file_filter_pattern： CosFile(en)、OssFile(en)、OssJindoFile(en)、HdfsFile(en)
Added the following file connector description about file_filter_pattern： FtpFile(en)、SftpFile(en)、S3File(en)、HdfsFile(zh)
Hisoka-X
changed the title
file_filter_pattern example to doc
Hisoka-X
approved these changes
Oct 29, 2024
|
@zhilinli123 please review |
zhilinli123
approved these changes
Oct 29, 2024
hailin0
approved these changes
Oct 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added file filtering instructions to the localfile connector documentation