Add QuestDB output component#115
Conversation
gregfurman
left a comment
gregfurman
left a comment
There was a problem hiding this comment.
Thanks for submitting this! All is looking good just some minor clarifications and suggestions.
Also, have you used QuestDB against our existing SQL components? Would be curious if it works out-the-box it and would be happy to extend them if need to do accomodate QuestDB!
Otherwise, your tests are passing on my local but I haven't (yet) tested this against a running QuestDB instance. Will do so soon 🥇
| // todo: is this the correct interpretation of max-in-flight? | ||
| qdb.WithMaxSenders(mif)(w.pool) |
There was a problem hiding this comment.
The max_in_flight field is specifically for internal Bento messages. This WithMaxSenders looks like it's referring to the number of QuestDB Senders which I believe are different (from QuestDB dotnet docs):
Sender is single-threaded, and uses a single connection to the database.
If you want to send in parallel, you can use multiple senders and standard async tasking.
cc @jem-davies Thoughts on this?
Reminds me that we could probably add a StreamTestOptMaxInFlight test to the integration tests. Will comment.
There was a problem hiding this comment.
Quick update here: not sure about the above. Have asked Jem to take a look.
What other value would make sense for the sender count?
There was a problem hiding this comment.
The pool defaults to a maximum of 64 senders. Ideally, we would use a sender per bento thread/goroutine, but it's not clear to me how many concurrent executions of WriteBatch can happen at a single time. I'm guessing max_in_flight is a maximum message count then?
There was a problem hiding this comment.
I can take a look tomorrow - just busy at the moment
There was a problem hiding this comment.
Yeah it's a maximum message count. Happy with approving and seeing how performance looks, and iterating on this at another point
Co-authored-by: Greg Furman <31275503+gregfurman@users.noreply.github.com>
Co-authored-by: Greg Furman <31275503+gregfurman@users.noreply.github.com>
I have not. We implement pgwire and should be compatible with standard postgres clients. I'll give that a shot this week, although we really only recommend using SQL for data extraction and querying, not ingestion. |
|
Lastly, can you add the docs for the component? Running Once that's added I'll approve and we can ship this bad-boy |
Done! Thank you :-) |
gregfurman
left a comment
gregfurman
left a comment
There was a problem hiding this comment.
LGTM! Just fixed the formatting of that description string.
3 participants
Adds an output component for writing to QuestDB tables using ILP (Influx Line Protocol) as implemented by the official QuestDB Go Client.