Vectorize Docs for RAG Chatbot

[patriciaflutterflow]

Link to RAG Chatbot PRD

There are 2 tables in BigQuery associated with documentation for the RAG Chatbot:

1. doc_text (includes doc id + doc text)
2. doc_text_vector (includes doc id + chunk id + vectorized chunk + chunk text)
   The code here ultimately processes each documentation page to write to the tables in BigQuery.

What happens:

• Github action run everytime there is a change in a markdown file in docs (vectorize_docs.yaml), function is process_single_file in vectorize.py
• the markdown file is chunked and each chunk is vectorized
• these changes are added to bigquery
• there is also a function that does a backfill for all existing docs

ToDo: right now, I am getting a 429 error quota exceeded after vectorizing sometimes. Check to see what quota limits are and how to get around it.

[bolt-new-by-stackblitz]

Run & review this pull request in StackBlitz Codeflow.

[PoojaB26]

Is there a ticket to this or maybe an explanation of what it does? @patriciaflutterflow

[patriciaflutterflow]

Is there a ticket to this or maybe an explanation of what it does? @patriciaflutterflow

Hello @PoojaB26 ! Yes added a brief overview at the top

[PoojaB26]

Can we close this ticket? @patriciaflutterflow

[patriciaflutterflow]

@PoojaB26 if it's alright with you can I leave it open for now? I may create a new branch for this but using a lot of the existing code here. Basically, this is to update the database for the chatbot automatically every time new documentation is added or when existing documentation is changed

[PoojaB26]

@patriciaflutterflow I mean the branch can be open, but will this PR be merged anytime soon? We can just close the PR and keep the branch for you to compare code?

[patriciaflutterflow]

Ohhhh I gotchu sounds good I'll close it :)