Adding pinot file system command

[xiangfu0]

Adding Pinot FS command into toolings for quickly testing if the filesystem configs and settings credentials are correct. Also, users can perform some simple actions if needed.

Usage:

pinot-admins.sh FileSystem <sub-command> <sub-command parameters>

There is also a separate script for it:

pinot-fs-util.sh <sub-command> <sub-command parameters>

Currently, supported sub-command operations are coming from PinotFS interfaces:

ls
cp
mv
rm

Default loaded PinotFS are org.apache.pinot.spi.filesystem.LocalPinotFS and org.apache.pinot.plugin.filesystem.S3PinotFS if scheme is not registered by the config file.

Example:

List files recursively on s3 when AWS is configured:

pinot-admin.sh FileSystem ls -r s3://<my-bucket>/

List files recursively on s3 with the pinot fs config:

pinot-admin.sh  FileSystem -config /tmp/fs.conf  -command ls -r s3://<my-bucket>/

The content of fs.conf will be deserailize to org.apache.pinot.spi.ingestion.batch.spec.PinotFSSpec

class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
s3.region=us-west-2
s3.accessKey=<my-access-key>
s3.secretKey=<my-secret-key>

Examples of no access for s3:

bin/pinot-admin.sh  FileSystem ls  s3://<my-bucket>

Unable to list files under: s3://<my-bucket>, exception: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 33KA59XA0ZPBDPN2, Extended Request ID: TILm5sJyqoQttVTNgSS5Hs1N89cQa9MtxRAh5oQSH8Jm9DpOG/yI/raYIw/7FU3fYzARba6qQBA=)

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.37%. Comparing base (c0056dc) to head (35187ae).
Report is 3767 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #8659      +/-   ##
============================================
+ Coverage     70.32%   70.37%   +0.05%     
+ Complexity     4418     4347      -71     
============================================
  Files          1693     1693              
  Lines         88309    88309              
  Branches      13212    13212              
============================================
+ Hits          62104    62149      +45     
+ Misses        21921    21876      -45     
  Partials       4284     4284              

Flag	Coverage Δ
integration1	27.47% <ø> (+0.07%)	⬆️
integration2	25.59% <ø> (+0.05%)	⬆️
unittests1	66.21% <ø> (+0.04%)	⬆️
unittests2	14.34% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.