We recently started experimenting with the File-Search-Store, to use it as RAG System for user provided pdf documents in our app. Last week we started to roll it out to a share of our users and encountered the issue that documents are sometimes stuck in a pending state for a very long time or even forever.
Most documents were processed within ~10 seconds, some only finished after minutes or hours and some are still pending today. Not a single document is in the failed state. Roughly 25% of the file uploads are affected by that issue.
I also looked into the document sizes, it happened more often for larger ones, but not only. I can also confirm that the files are not corrupt, as we open them in our backend before uploading them. Also while it happened more often during our peak usage time, it is still happening mostly random as it seems.
We had to roll back the experiment, as the average latency is way too high for our feature. We are currently looking into building a mitigation mechanism, to just start reuploads after some seconds, but this is not a proper solution.
To me it looks like there are some general issues with the processing pipeline maybe?
There is nothing mentioned on the status page, also the API does not allow any more insights into why it sometimes is so slow.
In the current state, we can not really use the feature in a production setup…
Update
After digging a bit more into the files, I noticed that a large part (> ~90%) of the failed file uploads were pdfs without a text layer, while the successful ones all have it. As Gemini supports visual understanding of pdfs, I assumed it would also work for the File-Search-Store. But the documentation does not say anything about it.
So my current assumption is that this is the root cause. Still the documents should fail properly then and not just stay in pending.
Also it’s a bit disappointing that it does not have visual understanding of pdfs.