FlyteLlm

Package: flyteplugins.agents.google

A BaseLlm that records each model turn via durable_step for replay.

Wraps an inner BaseLlm (resolved from the agent’s model); model is set to the inner model name so ADK behaves identically. Construct via flyteplugins.agents.google.durable_model.

Parameters

class FlyteLlm(
    model: str,
    inner: typing.Any = None,
)

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Methods

generate_content_async()

def generate_content_async(
    llm_request: typing.Any,
    stream: bool = False,
) -> typing.AsyncGenerator[typing.Any, None]

Generates content for a single model turn.

This method handles Server-Sent Events (SSE) streaming for unidirectional content generation. For bidirectional streaming (e.g., Gemini Live API), use the connect() method instead.

Yields:

LlmResponse objects representing the model’s response for one turn.

Non-streaming mode (stream=False):

Yields exactly one LlmResponse containing the complete model output (text, function calls, bytes, etc.). This response has partial=False.

Streaming mode (stream=True):

Yields multiple LlmResponse objects as chunks arrive:

• Intermediate chunks: partial=True (progressive updates)
• Final chunk: partial=False (aggregated content from entire turn, identical to stream=False output)
• Text consolidation: Consecutive text parts of the same type (thought/non-thought) SHOULD merge without separator, but client code must not rely on this - unconsolidated parts are unusual but also valid

Common content in partial chunks:

All intermediate chunks have partial=True regardless of content type. Common examples include:

• Text: Streams incrementally as tokens arrive
• Function calls: May arrive in separate chunks
• Bytes (e.g., images): Typically arrive as single chunk, interleaved with text
• Thoughts: Stream incrementally when thinking_config is enabled

Examples:

1. Simple text streaming:

   LlmResponse(partial=True,  parts=["The weather"])
   LlmResponse(partial=True,  parts=[" in Tokyo is"])
   LlmResponse(partial=True,  parts=[" sunny."])
   LlmResponse(partial=False, parts=["The weather in Tokyo is sunny."])

2. Text + function call:

   LlmResponse(partial=True,  parts=[Text("Let me check...")])
   LlmResponse(partial=True,  parts=[FunctionCall("get_weather", ...)])
   LlmResponse(partial=False, parts=[Text("Let me check..."),
                                     FunctionCall("get_weather", ...)])

3. Parallel function calls across chunks:

   LlmResponse(partial=True,  parts=[Text("Checking both cities...")])
   LlmResponse(partial=True,  parts=[FunctionCall("get_weather", Tokyo)])
   LlmResponse(partial=True,  parts=[FunctionCall("get_weather", NYC)])
   LlmResponse(partial=False, parts=[Text("Checking both cities..."),
                                     FunctionCall("get_weather", Tokyo),
                                     FunctionCall("get_weather", NYC)])

4. Text + bytes (image generation with gemini-2.5-flash-image):

   LlmResponse(partial=True,  parts=[Text("Here's an image of a dog.")])
   LlmResponse(partial=True,  parts=[Text("")])
   LlmResponse(partial=True,  parts=[Blob(image/png, 1.6MB)])
   LlmResponse(partial=True,  parts=[Text("It carries a bone")])
   LlmResponse(partial=True,  parts=[Text(" and running around.")])
   LlmResponse(partial=False, parts=[Text("Here's an image of a dog."),
                                     Blob(image/png, 1.6MB),
                                     Text("It carries a bone and running around.")])

   Note: Consecutive text parts before and after blob merge separately.

5. Text with thinking (gemini-2.5-flash with thinking_config):

   LlmResponse(partial=True,  parts=[Thought("Let me analyze...")])
   LlmResponse(partial=True,  parts=[Thought("The user wants...")])
   LlmResponse(partial=True,  parts=[Text("Based on my analysis,")])
   LlmResponse(partial=True,  parts=[Text(" the answer is 42.")])
   LlmResponse(partial=False, parts=[Thought("Let me analyze...The user wants..."),
                                     Text("Based on my analysis, the answer is 42.")])

   Note: Consecutive parts of same type merge (thoughts→thought, text→text).

Important: All yielded responses represent one logical model turn. The final response with partial=False should be identical to the response that would be received with stream=False.