Add logprobs support by jschweiz · Pull Request #127 · oracle/wayflow

jschweiz · 2026-03-27T15:58:30Z

Adding support for LogProbs in Wayflow: #126

dhilloulinoracle · 2026-03-27T16:21:05Z

Internal regression failed: Build ID #763

dhilloulinoracle · 2026-03-27T17:28:41Z

Internal regression succeeded 🍏: Build ID #764

dhilloulinoracle · 2026-04-16T08:48:55Z

+    if llm_config == GROK_OCI_API_KEY_CONFIG:
+        pytest.skip("OCI grok returns empty logprobs")
+    if llm_config == COHERE_OCI_API_KEY_CONFIG:
+        pytest.skip("Gtp-OSS does not support returning logprobs")


the string seems wrong

dhilloulinoracle · 2026-04-16T08:49:19Z

+        pytest.skip("OCI grok returns empty logprobs")
+    if llm_config == COHERE_OCI_API_KEY_CONFIG:
+        pytest.skip("Gtp-OSS does not support returning logprobs")
+    if llm_config == COHERE_OCI_API_KEY_CONFIG:


the if seems the same as previous one?

dhilloulinoracle · 2026-04-16T08:49:58Z

+def test_hosted_llm_can_return_logprobs_if_supported(llm_config):
+
+    if llm_config == OLLAMA_MODEL_CONFIG:
+        pytest.skip("Ollama hosted models sometimes does not return logprobs")


instead of skipping, we might want to check that we get an expected exception or expected default output

dhilloulinoracle · 2026-04-16T08:51:34Z

+    token: str
+    """The literal text of the generated token."""
+    logprob: float
+    """The log probability assigned to the generated token."""


what's the range of values? should be documented and normalized across providers

dhilloulinoracle · 2026-04-16T08:52:43Z

        generation_config:
            Optional generation arguments for the LLM generation in this step. See ``LlmGenerationConfig`` for available parameters.
+        top_logprobs:
+            If not None, the step will return the top logprobs for each token.


for each of the top top_logprobs token?

sonleoracle · 2026-04-16T13:31:13Z

        Python script/notebook for this guide.

-Generation parameters, such as temperature, top-p, and the maximum number of output tokens, are important for achieving the desired performance with Large Language Models (LLMs).
+Generation parameters, such as temperature, top-p, the maximum number of output tokens, and token log probabilities, are important for achieving the desired performance with Large Language Models (LLMs).


Suggested change

Generation parameters, such as temperature, top-p, the maximum number of output tokens, and token log probabilities, are important for achieving the desired performance with Large Language Models (LLMs).

Generation parameters, such as temperature, top-p, the maximum number of output tokens, and per-token log-probabilities, are important for achieving the desired performance with Large Language Models (LLMs).

sonleoracle · 2026-04-16T13:47:54Z


 The :ref:`LLM generation config <llmgenerationconfig>` is the set of parameters that control the output of a :ref:`Large Language Model (LLM) <llmmodel>` in WayFlow.
-These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, and the probability threshold for nucleus sampling (``top_p``).
+These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, the probability threshold for nucleus sampling (``top_p``), and optional token log probabilities (``top_logprobs``).


Suggested change

These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, the probability threshold for nucleus sampling (``top_p``), and optional token log probabilities (``top_logprobs``).

These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, the probability threshold for nucleus sampling (``top_p``), and optional per-token log-probabilities (``top_logprobs``).

sonleoracle · 2026-04-16T15:59:16Z

+                    continue
+
+                new_logprob = TextTokenLogProb(
+                    token=max_log_prob_token, logprob=max_log_prob, top_logprobs=top_log_probs


TextTokenLogProb is supposed to represent the actual emitted token and its logprob. Here we’re deriving it from top_logprobs by picking the max-probability candidate, which is only the argmax token, not necessarily the returned token. Under sampling, those can differ. OCI already gives us the canonical emitted token/logprob in choice_dict.logprobs.tokens and choice_dict.logprobs.token_logprobs; top_logprobs should only populate the alternate candidates.

sonleoracle · 2026-04-17T08:29:53Z

                if "summary" not in generation_config.extra_args["reasoning"]:
                    generation_config.extra_args["reasoning"]["summary"] = "auto"

            kwargs.update(generation_config.extra_args)


if generation_config.extra_args contains "include", this would overwrite the kwargs["include"].append("message.output_text.logprobs") you set above. Please check if the same problem with overriding exists for the chat completions processor.

sonleoracle · 2026-04-17T08:47:21Z

            if "usage" in json_object and json_object["usage"] is not None:
                raw_usage = json_object["usage"]
                token_usage = self._extract_usage(raw_usage)
            yield StreamChunkType.TEXT_CHUNK, Message(


Should we not attach logprobs here too? Same for the responses processor. Like delta["logprobs"]?

sonleoracle · 2026-04-17T08:53:30Z

+                    )
+                    logger.warning(warning_message)
+                    raise ValueError(warning_message)
+                outputs[self.LOGPROBS] = text_chunk.logprobs


Would this overwrite a usesr-defined output property also called "logprobs"?

paul-cayet · 2026-04-17T09:26:38Z

+
+  Add token logprobs support with the `top_logprobs` generation config parameter and support returning
+  logprobs in the `PromptExecutionStep`.
+


Please add a

for more information please read the guide on :ref:`how to ... <ref_to_section on request token log probabilities>`

sonleoracle

Please rebase, and please ensure this works for GeminiModel or raise

jschweiz added 5 commits March 27, 2026 11:35

[feat]: logprobs support in llm models and promptexecutionstep

6ac4129

[feat]: add more tests

b7f4964

[fix]: fix serialization tests

40bbd14

[fix]: fix documentation error

5fe3371

[fix]: add doc

ead58b0

jschweiz requested a review from a team March 27, 2026 15:58

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Mar 27, 2026

jschweiz requested a review from sonleoracle March 27, 2026 15:58

jschweiz assigned paul-cayet and dhilloulinoracle Mar 27, 2026

[fix]: fix readme format

bcf6640

jschweiz changed the title ~~Add logprobs support 2~~ Add logprobs support Mar 27, 2026

jschweiz assigned jschweiz, paul-cayet and dhilloulinoracle and unassigned paul-cayet and dhilloulinoracle Mar 27, 2026

jschweiz requested review from dhilloulinoracle and paul-cayet April 16, 2026 08:45

jschweiz unassigned paul-cayet and dhilloulinoracle Apr 16, 2026

dhilloulinoracle reviewed Apr 16, 2026

View reviewed changes

sonleoracle reviewed Apr 17, 2026

View reviewed changes

paul-cayet reviewed Apr 17, 2026

View reviewed changes

paul-cayet approved these changes Apr 17, 2026

View reviewed changes

sonleoracle approved these changes Apr 17, 2026

View reviewed changes

	Generation parameters, such as temperature, top-p, the maximum number of output tokens, and token log probabilities, are important for achieving the desired performance with Large Language Models (LLMs).
	Generation parameters, such as temperature, top-p, the maximum number of output tokens, and per-token log-probabilities, are important for achieving the desired performance with Large Language Models (LLMs).

	These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, the probability threshold for nucleus sampling (``top_p``), and optional token log probabilities (``top_logprobs``).
	These parameters include the maximum number of tokens to generate (``max_tokens``), the sampling ``temperature``, the probability threshold for nucleus sampling (``top_p``), and optional per-token log-probabilities (``top_logprobs``).


		Add token logprobs support with the `top_logprobs` generation config parameter and support returning
		logprobs in the `PromptExecutionStep`.

Conversation

jschweiz commented Mar 27, 2026

Uh oh!

dhilloulinoracle commented Mar 27, 2026

Uh oh!

dhilloulinoracle commented Mar 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonleoracle left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants