[Document] Remove a `fs.oss.credentials.provider` option #7507

loustler · 2024-08-27T08:00:32Z

Purpose of this pull request

Remove a fs.oss.credentials.provider configuration from checkpoint with OSS storage. Because if it provided into a hadoop configuration, it trying to find a constructor has of URL and Configuration, and it throws a exception. Because a AliyunCredentialsProvider class only have a single constructor which requires a Configuration class.
It leads users like me to be confused/misunderstood, so we need to fix documents about it for users.

These are all release hadoop-aliyun source codes about load a credential provider
Hadoop 3.1.4

https://github.com/apache/hadoop/blob/1e877761e8dadd71effef30e592368f7fe66a61b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSUtils.java#L105-L141

https://github.com/apache/hadoop/blob/1e877761e8dadd71effef30e592368f7fe66a61b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunCredentialsProvider.java#L35-L67

Hadoop 3.3.6

https://github.com/apache/hadoop/blob/1be78238728da9266a4f88195058f08fd012bf9c/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSUtils.java#L106-L143

https://github.com/apache/hadoop/blob/1be78238728da9266a4f88195058f08fd012bf9c/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunCredentialsProvider.java#L35-L67

Error stacktrace

2024-08-27 16:43:55,635 WARN  [Log4j2HttpPostCommandProcessor] [hz.main.cached.thread-3] - [192.168.106.239]:5801 [my-seatunnel] [5.1] An error occurred while handling request HttpCommand [HTTP_POST]{uri='/hazelcast/rest/maps/submit-job?jobName=my-job'}AbstractTextCommand[HTTP_POST]{requestId=0}
java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.engine.checkpoint.storage.exception.CheckpointStorageException: Failed to get file system
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:70)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.<init>(HdfsStorage.java:57)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:105)
	at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:288)
	at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:271)
	at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:499)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider constructor exception.  A class specified in fs.oss.credentials.provider must provide an accessible constructor accepting URI and Configuration, or an accessible default constructor.
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:132)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:155)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:349)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:68)
	... 12 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider.<init>()
	at java.lang.Class.getConstructor0(Class.java:3082)
	at java.lang.Class.getDeclaredConstructor(Class.java:2178)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:126)
	... 18 more

	at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375) ~[?:1.8.0_392]
	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947) ~[?:1.8.0_392]
	at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.submitJob(RestHttpPostCommandProcessor.java:239) ~[seatunnel-starter.jar:2.3.7]
	at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handleSubmitJob(RestHttpPostCommandProcessor.java:146) ~[seatunnel-starter.jar:2.3.7]
	at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:82) ~[seatunnel-starter.jar:2.3.7]
	at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:60) ~[seatunnel-starter.jar:2.3.7]
	at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:402) ~[seatunnel-starter.jar:2.3.7]
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[seatunnel-starter.jar:2.3.7]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[seatunnel-starter.jar:2.3.7]
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.7]
Caused by: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.engine.checkpoint.storage.exception.CheckpointStorageException: Failed to get file system
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:70)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.<init>(HdfsStorage.java:57)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:105)
	at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:288)
	at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:271)
	at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:499)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider constructor exception.  A class specified in fs.oss.credentials.provider must provide an accessible constructor accepting URI and Configuration, or an accessible default constructor.
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:132)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:155)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:349)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:68)
	... 12 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider.<init>()
	at java.lang.Class.getConstructor0(Class.java:3082)
	at java.lang.Class.getDeclaredConstructor(Class.java:2178)
	at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:126)
	... 18 more

	at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:506) ~[seatunnel-starter.jar:2.3.7]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_392]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_392]

Does this PR introduce any user-facing change?

YES

seatunnel:
  engine:
    checkpoint:
      interval: 6000
      timeout: 7000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          storage.type: oss
          oss.bucket: your-bucket
          fs.oss.accessKeyId: your-access-key
          fs.oss.accessKeySecret: your-secret-key
          fs.oss.endpoint: endpoint address
-         fs.oss.credentials.provider: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider

How was this patch tested?

It tested with release 2.3.7 with OSS

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

…dException

Hisoka-X

Yes. We did clean in #7332 . But seem like not cleaned up. Thanks @loustler ! cc @Carl-Zhou-CN

Carl-Zhou-CN · 2024-08-27T09:07:57Z

Yes. We did clean in #7332 . But seem like not cleaned up. Thanks @loustler ! cc @Carl-Zhou-CN

@loustler Could you please help me globally search for 'org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider' and completely remove this configuration? Thank you very much for your help.

loustler · 2024-08-27T09:24:09Z

@Carl-Zhou-CN
A fs.oss.credentials.provider found in some tests

A org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider found in some documents and tests

I found those by IntelliJ.

Should I change this?

seatunnel/docs/en/seatunnel-engine/checkpoint-storage.md

Line 51 in c6f627f

    
           The client supports multiple authentication mechanisms and can be configured as to which mechanisms to use, and their order of use. Custom implementations of org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider may also be used.

seatunnel/docs/zh/seatunnel-engine/checkpoint-storage.md

Line 49 in c6f627f

    
           客户端支持多种身份验证机制，并且可以配置使用哪种机制及其使用顺序。也可以使用of org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider的自定义实现。

Carl-Zhou-CN · 2024-08-27T09:37:53Z

I think it can be removed from the test case, @Hisoka-X what do you think?

Hisoka-X · 2024-08-27T09:43:23Z

I think it can be removed from the test case, @Hisoka-X what do you think?

yes.

loustler · 2024-08-27T23:38:35Z

@Carl-Zhou-CN @Hisoka-X
Removed it from test codes

wuchunfu

LGTM

* Remove a `fs.oss.credentials.provider` option caused by a NoSuchMethodException * Add RELEASE NOTE * Remove a `fs.oss.credentials.provider` option from test codes * fix --------- Co-authored-by: Dongyeon <[email protected]>

Remove a fs.oss.credentials.provider option caused by a NoSuchMetho…

6f7fe35

…dException

github-actions bot added document Zeta labels Aug 27, 2024

Add RELEASE NOTE

a52f807

loustler changed the title ~~[Document] Remove a fs.oss.credentials.provider option caused by a NoSuchMethodException~~ [Document] Remove a fs.oss.credentials.provider option Aug 27, 2024

Hisoka-X previously approved these changes Aug 27, 2024

View reviewed changes

github-actions bot added approved reviewed labels Aug 27, 2024

Remove a fs.oss.credentials.provider option from test codes

e0dc71c

loustler dismissed Hisoka-X’s stale review via e0dc71c August 27, 2024 23:38

github-actions bot added e2e and removed approved labels Aug 27, 2024

github-actions bot removed the reviewed label Aug 27, 2024

fix

2acac07

Hisoka-X approved these changes Aug 28, 2024

View reviewed changes

github-actions bot added approved reviewed labels Aug 28, 2024

wuchunfu approved these changes Aug 28, 2024

View reviewed changes

wuchunfu merged commit 657def2 into apache:dev Aug 28, 2024
7 checks passed

loustler deleted the chore/oss-checkpoint-doc branch August 28, 2024 01:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Document] Remove a `fs.oss.credentials.provider` option #7507

[Document] Remove a `fs.oss.credentials.provider` option #7507

loustler commented Aug 27, 2024 •

edited

Loading

Hisoka-X left a comment

Carl-Zhou-CN commented Aug 27, 2024

loustler commented Aug 27, 2024

Carl-Zhou-CN commented Aug 27, 2024

Hisoka-X commented Aug 27, 2024

loustler commented Aug 27, 2024

wuchunfu left a comment

[Document] Remove a fs.oss.credentials.provider option #7507

[Document] Remove a fs.oss.credentials.provider option #7507

Conversation

loustler commented Aug 27, 2024 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X left a comment

Choose a reason for hiding this comment

Carl-Zhou-CN commented Aug 27, 2024

loustler commented Aug 27, 2024

Carl-Zhou-CN commented Aug 27, 2024

Hisoka-X commented Aug 27, 2024

loustler commented Aug 27, 2024

wuchunfu left a comment

Choose a reason for hiding this comment

[Document] Remove a `fs.oss.credentials.provider` option #7507

[Document] Remove a `fs.oss.credentials.provider` option #7507

loustler commented Aug 27, 2024 •

edited

Loading