Batch addition of chunks

[frederickmannings]

Is your feature request related to a problem? Please describe.
Adding text chunks can only be done one at a time with the current implementation. Really, we need the ability for many many chunks to be added in one go.

And in DynaRAG style, this needs to be fast.

Describe the solution you'd like
A batch chunk addition function. As we are moving away from this module as an API, all we need is a function that performs this functionality.

The function should accept a slice of the Chunk struct:

type ChunkRequestBody struct {
	Chunk    string                 `json:"chunk"`
	FilePath string                 `json:"filepath"`
	MetaData map[string]interface{} `json:"metadata,omitempty"`
}

The major latency in this operation will be in the embedding process. Even though this is not an IO bound task, Go's concurrency model will help us, as the go scheduler will dynamically allocate goroutine tasks accross available cpu cores (assuming GOMAXPROCS == available cpu cores).

So the way to do this (in my opinion), is to farm off the embedding processing via go routines. Go will manage the spread across cpu cores. We need to manage CPU resource usage via the use of a worker pool. Then, as the embeddings come back, we can farm off the writes to the DB as go routines also.

I also think we should have an "early exit" ability to this function that returns a reference to struct that stores the condition of the embedding process. This allows the use to manage the processing time within their function.

For example, a struct that looks like the following would be sufficient (note the use of a mutex lock and context - these will need to be factored in to the implementation):

type BatchProgress struct {
    Total int
    Completed int
    Failed []string
    mu sync.RWMutex
    // Maybe a context for cancellation
    ctx context.Context
    cancel context.CancelFunc
}

Describe alternatives you've considered
We could write each embedding one by one, but this will be slow, and will force us to exit early before the operation has finished in order to maintain an illusion of performance.