I set Top-K to 1 for Gemini 2.5 Pro, and then set Temperature and Top-P to their maximum values. I observed that the model’s output was not consistently fixed/deterministic.
Does this imply that the Top-K parameter has no effect?
Hi @Duncan_Lean You can achieve consistently fixed and deterministic output from an LLM, you should set the temperature to the lowest possible value, which is typically 0 .
Temperature usually overrides top k.
Thank you
@Pannaga_J Thank you!
I followed your advice and set both temperature and TopP to 0. Then I asked the model to output a random integer array, but the results returned by the model were still not deterministic.
And I also noticed that the details of its thought process differ.
1 Like
The Top-K parameter does have an effect on the model’s output, but it’s essential to understand how it interacts with temperature and Top-P (nucleus sampling).
1.Top-K Sampling:
When you set Top-K to 1,
the model restricts its next word selection to the most probable word in the distribution. Essentially, this means that only the highest-probability token can be chosen as the next token.
2.Temperature:
The temperature parameter controls the randomness of the model’s output. A higher temperature increases randomness, making the distribution more uniform, while a lower temperature makes the model’s selection more deterministic, focusing on the top probabilities.
3.Top-P Sampling:
This parameter allows you to select from the smallest set of tokens whose cumulative probability exceeds the threshold P.
Higher values of Top-P increase randomness and diversity in the output.
When you set Top-K to 1,
it seems at first that the model should always generate the same output, as it can only pick the highest probability token. However, when combined with a high temperature and Top-P, the sampling process allows for variability to some degree (although the variance will be limited to the probabilities of the tokens selected).
In essence, although Top-K limits the choice to the highest-ranked token, the random fluctuation imposed by the temperature can still create different outputs if the model’s internal probabilities for the highest token can vary slightly.
Thus, while Top-K does impact the output by limiting the choice, when combined with high settings of temperature, it can lead to non-deterministic outputs. If you observe inconsistency despite the Top-K setting, it could be due to how the model handles probabilities or internal factors that still allow for some variation. If you want more deterministic results, you may want to lower the temperature or adjust the Top-P setting.
Joseph lual
Hi @Duncan_Lean ,
Setting the Temperature to 0 increases the chance of getting a similar (or nearly identical) response, but it does not guarantee an identical response due to underlying computational process. Thank You!!