access response content as Tempfile IO object

[khoan]

The current engine returns response content in the body. When content size is large (hundreds of MB), this content will be held in memory.

When there are multiple threads fetching large dataset, it can crash boxes with low memory.
Can we have an engine that returns IO object to response content?

# musing with possible API
RC::Universal.new(
  'response_body' => 'io', # defaults to response content when unspecified, which is the current behavior
  'site' => 'http://large.size.api/',
).get('/')
``

[godfat]

The question for doing this is then how those middleware which handles response should do?
If we just don't care and they shouldn't really do anything, then we could use HIJACK for now:
https://github.com/godfat/rest-core#streaming-responses

sock = client.get('path', {}, RC::HIJACK => true)
p sock.read(10)
p sock.read(10)
p sock.read(10)

This would basically return '' for the original response, so middleware (e.g. JsonResponse) which handles response would probably ignore it. ('' might be parsed to nil depending on your json parser)

You can override request_full or request in the client so that RC::HIJACK is attached to each request by default.

Does this make sense? Is there any better way to do this?

p.s. This was used in building the SSE (Server-Sent Event) client, which was used in https://github.com/CodementorIO/rest-firebase

[speedmax]

Hi Godfat,

@khoan and I found out about the socket hijack this morning.

temp =  TempFile.new('large-response')
temp.binmode

client.get('path', {}, RC::HIJACK => true) do |sock|
   while chunk = sock.read(4096)
       temp << chunk
   end
end

We have built a data processing framework on top of rest-core/more, we are using it to handle some pretty large product xml feed or large JSON responses, so we are constantly looking for a way to keep memory footprint low.

1. Tempfile approach: I think open-uri.rb's buffer is a neat idea, that it creates a StringIO for response smaller than 10KB, Tempfile for larger responses.
2. Stream response from Socket IO to (JSON, Nokogiri) parsers: I am not sure how this would work. You might have done something similar parsing a large data feed.

Example from Oga's README

enum = Enumerator.new do |yielder|
  HTTPClient.get('http://some-website.com/some-big-file.xml') do |chunk|
    yielder << chunk
  end
end

document = Oga.parse_xml(enum)

I am not sure if the temp file idea is something that can go into the library, or some example that we can make rest-core to work inside a Enumerator

[godfat]

Hi,

If you want to make a tempfile, then you could also do:

temp =  TempFile.new('large-response')
temp.binmode
sock = client.get('path', {}, RC::HIJACK => true)
IO.copy_stream(sock, temp)

This way you don't have to read and write by yourself.

As for buffering the response, I am not very sure how we could do this in rest-core,
as some middleware might not be aware of this buffer, and at some point, some
middleware just need the response in order to do stuffs. Simply do a read there
would defeat the purpose.

So maybe it would be easier to introduce another interface for this kind of stuffs.
My original intention was only for handling small API responses. For downloading,
I used to think I could write another library for that, but then I no longer really need
that feature...

That says, if we could make rest-core handle large responses, I am all for it.

Parsing JSON with stream, as far as I know, yajl-ruby could do that.
We could probably make a yajl-ruby only JsonResponse that could take
the advantage of this. I'll try if I could make this integrated seamlessly.

As for Oga, I never heard of this before. But this should work:

enum = Enumerator.new do |yielder|
  client.get('path', {}, RC::HIJACK => true) do |sock|
    while chunk = sock.read(4096)
      yielder << chunk
    end
  end
end

document = Oga.parse_xml(enum)

Of course, this could be done in a middleware, too.
We don't have an XmlResponse middleware right now though.
I haven't added one because XML can't be directly converted to
ruby hash, and we probably always want the document object,
which differs from Oga and Nokogiri.

[godfat]

This could make this parse with stream:

parsed_ruby_object = client.get('path', {}, RC::HIJACK => true)

But frankly I am not sure if this is a good idea, as it would change the socket response to a parsed result. This could be unexpected when using hijacking.

Patch attached:

diff --git a/lib/rest-core/middleware/json_response.rb b/lib/rest-core/middleware/json_response.rb
index 84ec0a8..510fa30 100644
--- a/lib/rest-core/middleware/json_response.rb
+++ b/lib/rest-core/middleware/json_response.rb
@@ -28,11 +28,16 @@ class RestCore::JsonResponse
   end

   def process response
-    # StackExchange returns the problematic BOM! in UTF-8, so we need to
-    # strip it or it would break JSON parsers (i.e. yajl-ruby and json)
     body = response[RESPONSE_BODY].to_s.sub(/\A\xEF\xBB\xBF/, '')
-    response.merge(RESPONSE_BODY => Json.decode("[#{body}]").first)
-    # [this].first is not needed for yajl-ruby
+    if response[HIJACK] && Json.respond_to?(:decode_from_io)
+      response.merge(
+        RESPONSE_SOCKET => Json.decode_from_io(response[RESPONSE_SOCKET]))
+    else
+      # StackExchange returns the problematic BOM! in UTF-8, so we need to
+      # strip it or it would break JSON parsers (i.e. yajl-ruby and json)
+      response.merge(RESPONSE_BODY => Json.decode("[#{body}]").first)
+      # [this].first is not needed for yajl-ruby
+    end
   rescue Json.const_get(:ParseError) => error
     fail(response, ParseError.new(error, body))
   end
diff --git a/lib/rest-core/util/json.rb b/lib/rest-core/util/json.rb
index 2d93285..e781eb3 100644
--- a/lib/rest-core/util/json.rb
+++ b/lib/rest-core/util/json.rb
@@ -23,6 +23,7 @@ module RestCore::Json
     def decode json
       Yajl::Parser.parse(json)
     end
+    alias_method :decode_from_io, :decode
   end

   module Json

[speedmax]

You are right, it's the wrong level of abstraction to stream content like XML, there is another label of domain logic (Ex. RSS, Atom, Google Shopping feed).

We are currently decoding large JSON response via Tempfile first, so it doesn't take up hundreds of mb of memory. Thanks for the tip on IO.copy_stream, I think it might be possible to support TCPSocket IO objects.

Holy grail is to have a set of data processing layer that receive data stream (IO like object) and feed it into streaming JSON (yajl-ruby), XML (Sax XML parsers like sax-machine).

[godfat]

Yes, IO.copy_stream should work on any IO like objects.
If Enumerator could be IO like, we could do this for Oga.
I think Oga should really take IO like objects instead of Enumerator...
Or we could probably write a ChunkIO so that we could do something like:

enum = ChunkIO.new
IO.copy_stream(sock, enum)
Oga.parse_xml(enum)

Might be a good add to Ruby.