Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using read(::IO, nb) returns no data on first read with S3Path #197

Closed
omus opened this issue Aug 16, 2021 · 3 comments
Closed

Using read(::IO, nb) returns no data on first read with S3Path #197

omus opened this issue Aug 16, 2021 · 3 comments

Comments

@omus
Copy link
Member

omus commented Aug 16, 2021

julia> using AWSS3

julia> ENV["AWS_DEFAULT_REGION"] = "us-east-1"
"us-east-1"

julia> p = S3Path("s3://fcp-indi/data/test_resources/mpi_smp.conf")  # Random public dataset file
p"s3://fcp-indi/data/test_resources/mpi_smp.conf"

julia> io = open(p)
FilePathsBase.FileBuffer(p"s3://fcp-indi/data/test_resources/mpi_smp.conf", IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), true, false, false)

julia> read(io, 8)
UInt8[]

julia> read(io, UInt8)
0x70

julia> read(io, 8)
8-element Vector{UInt8}:
 0x65
 0x5f
 0x6e
 0x61
 0x6d
 0x65
 0x20
 0x20
@omus omus changed the title Using read(::IO, nb) fails initially with S3Path Using read(::IO, nb) returns no data on first read with S3Path Aug 16, 2021
@omus
Copy link
Member Author

omus commented Aug 16, 2021

Underlying issue seems to be: rofinn/FilePathsBase.jl#126

@nickrobinson251
Copy link
Contributor

Confirmed that with rofinn/FilePathsBase.jl#140 we now get

julia> using AWSS3

julia> ENV["AWS_DEFAULT_REGION"] = "us-east-1"
"us-east-1"

julia> p = S3Path("s3://fcp-indi/data/test_resources/mpi_smp.conf")
p"s3://fcp-indi/data/test_resources/mpi_smp.conf"

julia> io = open(p)
FilePathsBase.FileBuffer(p"s3://fcp-indi/data/test_resources/mpi_smp.conf", IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), true, false, false)

julia> read(io, 8)
8-element Vector{UInt8}:
 0x70
 0x65
 0x5f
 0x6e
 0x61
 0x6d
 0x65
 0x20

So this can be closed once there's a new patch release of FilePathsBase with that fix

@omus
Copy link
Member Author

omus commented Nov 1, 2021

Fixed as of FilePathsBase 0.9.12

@omus omus closed this as completed Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants