Hi all, I’m trying to implement screensharing using the live api in an electron app i’m building. I’m using python as a backend and javascript in the front end. I’m essentially following the example code on the google ai studio website, in the streaming section. https://aistudio.google.com/live
This is the specific bit:
def _get_screen(self):
sct = mss.mss()
monitor = sct.monitors[0]
i = sct.grab(monitor)
mime_type = "image/jpeg"
image_bytes = mss.tools.to_png(i.rgb, i.size)
img = PIL.Image.open(io.BytesIO(image_bytes))
img.thumbnail([1024, 1024])
image_io = io.BytesIO()
img.save(image_io, format="jpeg")
image_io.seek(0)
image_bytes = image_io.read()
return {"mime_type": mime_type, "data": base64.b64encode(image_bytes).decode()}
async def get_screen(self):
while True:
frame = await asyncio.to_thread(self._get_screen)
if frame is None:
break
await asyncio.sleep(1.0)
await self.out_queue.put(frame)
I’m getting the following error.
±±--------------- 1 ----------------
| Traceback (most recent call last):
| File “/Users/…/Documents/app/widget/backend/multimedia_service.py”, line 241, in receive_audio
| async for response in turn:
| File “/opt/anaconda3/envs/widget-python/lib/python3.11/site-packages/google/genai/live.py”, line 416, in receive
| while result := await self._receive():
| ^^^^^^^^^^^^^^^^^^^^^
| File “/opt/anaconda3/envs/widget-python/lib/python3.11/site-packages/google/genai/live.py”, line 497, in _receive
| raw_response = await self._ws.recv(decode=False)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File “/opt/anaconda3/envs/widget-python/lib/python3.11/site-packages/websockets/asyncio/connection.py”, line 322, in recv
| raise self.protocol.close_exc from self.recv_exc
| websockets.exceptions.ConnectionClosedError: received 1007 (invalid frame payload data) Request contains an invalid argument.; then sent 1007 (invalid frame payload data) Request contains an invalid argument.
I have now narrowed the error to:
ValueError('Unsupported input type “<class 'google.genai.types.Content'>” or input content "parts=[Part(video_metadata=None, thought=None, inline_data=Blob(display_name=None, data=b'/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAY…
It seems that gemini does not recognize how the images are encoded and is closing the websocket. What could be causing this? What am I missing in regards to how the images should be encoded?