Read IOStream for use with S3

[rob-luke]

Is it possible to pass an IOBuffer to MAT.jl?

This would be useful when storing data on services such as Amazon S3 with AWS.jl.
I tried the following code, but it did not work.

using AWS
using AWS.S3
env = AWSEnv(); bkt = "???"

file = S3.get_object(env, bkt, "testfile.mat").obj
#IOBuffer(data=Uint8[...], readable=true, writable=true, seekable=true, append=false, size=1040, maxsize=Inf, ptr=1041, mark=-1)

vars = matread(file)
# ERROR: `matread` has no method matching matread(::IOBuffer)

[dahong67]

Would be great for this to get a more "proper" fix, but I think a simple (inefficient?) work-around for now could be something along the lines of:

function matread(io::IO)
    # Write to a temporary file
    tempfile = tempname()
    write(tempfile, io)

    # Read data from temporary file
    data = matread(tempfile)

    # Clean up temporary file and return data
    rm(tempfile)
    return data
end

Would need a bit of work to make sure errors, etc. are handled correctly (e.g., what should be done if io is empty?).

@JeffFessler, is this something you're still looking at?

[JeffFessler]

I found a workaround of the form data = matread(Downloads.download(dataurl)) for my own use, e.g.,
https://juliaimagerecon.github.io/Examples/generated/mri/5-l-plus-s/#Read-data
(You might be amused to see whose work is cited there.)

That does not address the OP question though, and it still (inefficiently?) uses an intermediate file.

[dahong67]

You might be amused to see whose work is cited there

Fun! Just showed her. :)

[matthijscox]

This should be doable for the old file formats (before v7.3), which are just binary files.

The new format (v7.3) is HDF5. I dont think HDF5.jl supports IOBuffer input. Reading the HDF5.API source code, you can see it calls into the libhdf5 C library. We currently use h5f_open inside MAT.jl, which uses a filepath. I'm not seeing a streaming alternative there. I also can't find information on a libhdf5 streaming options on the internet anywhere yet.

in-memory file support does seem possible: HDF5.jl In-memory-HDF5-files, so maybe we can use that? Do you really need streaming? Or can we just read the full IOBuffer into a byte vector, then turn it into an in-memory HDF5 object, and then read that with MAT.jl? Note this would also solve issue #96