I am developing a Q&A application , that needs to answer the prompts based on the following types of data
- Static documents ( weblinks, pdf, video’s )
- we can use the fine-tuning or RAG for this type of data.
- Dynamically updating documents - New data will be appended to the existing documents at regular intervals
- We can use RAG to include this type of data. Any other options can we use for efficiency ?
- Dynamically generated text and numeric data –
- Is there any best way to generate the embeddings for numeric data ? (categorization of the numeric data is one option, but it is not suitable for all types of data ) ?
- Is there any agents available to fetch the data from data sources like Elasticdb and MangoDB
- How efficient are the LLM’s for forecasting and outlier detection, based on numeric embeddings generated through different methods?
Are there any better approaches to implement the Q&A using the above type of data, otherthan using RAG And Agents ? If, so , can you please mention about the same and at which stage ?