Tokenization of the text

Google AI Studio
,
1

Is there a way to provide a Tokenize/Detokenize api endpoints?

I am upgrading the gemini binding in lollms and one functionality it can not provide yet is the tokenization. It is useful for lollms’s playground page where users can experiment with AI.

Thanks.

3

Hi @saifeddine_ALOUI ,

Currently, dedicated Tokenize/Detokenize API endpoints are not available for Gemini. However, this section of the Gemini Documentation may still be helpful for understanding how tokens work with the API."