Improve error handling

[egorps]

https://github.com/spilgames/pyvertica/blob/master/pyvertica/batch.py#L92-L93

I'd like to add a parameter to the vertica batch constructor that provide a rescue action for everything remaining in the fifo - if things are just sucked out and dropped, important data could be lost. I'm thinking the default could just be "pass", but for my use, it would be nice to dump everything in the fifo to a file on disk so it could be restored later.

I'll work on this feature.

[lomignet]

That would be a good one. One thing to keep in mind is that as we are talking about big data, filling up a partition is quite easy (and already happen to us with the first uses of the REJECTED DATA option of copy).

[egorps]

Hmmm good point. I think my use case will be like 100 entries per batch. Are you really loading that much data in one batch? How long does it take and how much data are you talking about?

I was thinking about passing in an optional param to the constructor of VerticaBatch, the default value could be "pass"

[lomignet]

We had a project where we were loading daily hundreds of gigs from a binary format, and we really felt the pain of big data (but then it was really fun to work on it). Sadly this data source has been scrapped. Currently we load maybe 1GB/hour, and we are taking about not even 5 minutes of load I believe divided in 16 batches, hence including all startup, checks and so on.

My guess is that a default of 'pass' is safe, and then if data needs to be written to disk, making sure to catch the diskFullException (not sure on top of my head what the real name is) and delete the written file in that case would be enough.