How to upload files in parallel, asynchronous with genai

Gemini API
,
1

Hello!
When i write sequential code to upload videos to genai, like this (omit what is here async):

for video in videos:
     uploaded_videos.append(self._upload_video(video))

_upload_video:

   video_file = None
        while True:
            try:
                logger.info(f"Uploading video {video.name}")
                video_file = await self.client.aio.files.upload(file=Path(video.path))
            except httpx.TimeoutException:
                logger.warning(f"Timeout while uploading video {video.name}. Retrying...")
                await asyncio.sleep(10)
                continue
            break

Everything works great - but it is very slow.
I want to upload video files (I split 30 min video into 2min chunks) asynchronous, in parallel.
When I do this:

        tasks = [self._upload_video(video) for video in videos]
        return await asyncio.gather(*tasks)

I got all the time DEBUG:httpcore.http11:send_request_body.failed exception=WriteTimeout(TimeoutError()). How to deal with it?
Is it possible to upload files in parallel, how can I fix this problem with WriteTimeout?

I tried to submit the images in groups of three and managed to reduce the loading time twice.

Are there any other options? (I also used semaphores, but did not see any speedup.)

3

The WriteTimeout errors you’re encountering often stem from the server being overwhelmed by simultaneous requests. While the Gemini API supports asynchronous operations, it’s essential to manage concurrency to avoid such issues.

Here’s an approach to handle parallel uploads more effectively:

  1. Limit Concurrent Uploads: Use a semaphore to restrict the number of concurrent uploads. This helps prevent overwhelming the server and reduces the likelihood of timeouts.
  2. Batch Uploads: Instead of uploading all videos simultaneously, process them in smaller batches. This can help manage load and improve stability.
  3. Implement Retry Logic: For transient errors like timeouts, implement a retry mechanism with exponential backoff. This increases the chances of successful uploads without overwhelming the server.

Let me know if you need code snippets to implement these strategies.