Skip to content

Chat Interactions

QueryMT provides comprehensive support for chat-based interactions with Large Language Models, enabling you to build conversational AI applications.

Key Components

  • querymt::chat::ChatMessage: Represents a single message in a conversation. Key attributes include:

    • role: Indicates who sent the message (querymt::chat::ChatRole::User or querymt::chat::ChatRole::Assistant).
    • message_type: Specifies the nature of the content (e.g., querymt::chat::MessageType::Text, Image, ToolUse, ToolResult).
    • content: The primary text content of the message.
    • Source: crates/querymt/src/chat/mod.rs

  • querymt::chat::ChatResponse: A trait representing the LLM's response to a chat request. It provides methods to access:

    • text(): The textual content of the LLM's reply.
    • tool_calls(): A list of querymt::ToolCall objects if the LLM decided to use one or more tools.
    • thinking(): Optional "thoughts" or reasoning steps from the model, if supported and enabled.
    • usage(): Optional token usage information (querymt::Usage).
    • Source: crates/querymt/src/chat/mod.rs

  • querymt::chat::BasicChatProvider: A trait that LLM providers implement to support fundamental chat functionality. It has a single method:

    • chat(&self, messages: &[ChatMessage]): Sends a list of messages to the LLM and returns a ChatResponse.
    • Source: crates/querymt/src/chat/mod.rs

  • querymt::chat::ToolChatProvider: Extends BasicChatProvider to include support for tools (function calling). It has one primary method:

    • chat_with_tools(&self, messages: &[ChatMessage], tools: Option<&[Tool]>): Sends messages along with a list of available tools the LLM can use.
    • Source: crates/querymt/src/chat/mod.rs

How It Works

  1. Construct Messages: Your application assembles a sequence of ChatMessage objects representing the conversation history. Use ChatMessage::user() for user messages and ChatMessage::assistant() for assistant messages. This typically starts with alternating User and Assistant messages.
  2. Initiate Chat: You call the chat or chat_with_tools method on an LLMProvider instance, passing the message history and optionally, a list of available tools.
  3. Provider Interaction: The LLMProvider (or its underlying implementation like HTTPLLMProvider) formats the request according to the specific LLM's API, sends it, and receives the raw response.
  4. Parse Response: The provider parses the raw response into an object implementing ChatResponse.
  5. Handle Response: Your application processes the ChatResponse:
    • If text() is present, it's the LLM's textual reply.
    • If tool_calls() is present, the LLM wants to execute one or more functions. Your application needs to:
      • Execute these functions.
      • Send the results back to the LLM as new ChatMessages (typically with MessageType::ToolResult).
      • Continue the chat loop.

Example Flow (Conceptual)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
use querymt::chat::{ChatMessage, ChatMessageBuilder, ChatResponse, MessageType, Tool, ToolCall};
use querymt::LLMProvider;
use serde_json::Value;

// Assuming 'llm_provider' is an instance of Box<dyn LLMProvider> that has tools registered
// and 'my_tools' is a Vec<Tool> describing them.

async fn handle_tool_calling_loop(
    llm_provider: &Box<dyn LLMProvider>,
    initial_messages: Vec<ChatMessage>,
    my_tools: &[Tool],
) -> anyhow::Result<String> {
    let mut messages = initial_messages;

    loop {
        let response = llm_provider.chat_with_tools(&messages, Some(my_tools)).await?;

            // Add the assistant's response to the history. It might contain text and/or tool calls.
            let mut msg_builder = ChatMessage::assistant().text(response.text().unwrap_or_default());
            if let Some(tool_calls) = response.tool_calls() {
                for tc in tool_calls {
                    let args: serde_json::Value = serde_json::from_str(&tc.function.arguments).unwrap_or_default();
                    msg_builder = msg_builder.tool_use(tc.id.clone(), tc.function.name.clone(), args);
                }
            }
            if let Some(thinking) = response.thinking() {
                msg_builder = msg_builder.thinking(thinking);
            }
            messages.push(msg_builder.build());

        if let Some(tool_calls) = response.tool_calls() {
            if tool_calls.is_empty() {
                // No tool calls, so the text response is the final answer.
                return Ok(response.text().unwrap_or_default());
            }

            // The model wants to call one or more tools.
            let mut tool_results = Vec::new();
            for call in tool_calls {
                println!("LLM wants to call tool: {} with args: {}", call.function.name, call.function.arguments);

                // In a real app, you would dispatch to your tool execution logic here.
                // The `LLMProvider` trait has a `call_tool` helper for this.
                let args: Value = serde_json::from_str(&call.function.arguments)?;
                let result_str = llm_provider.call_tool(&call.function.name, args).await?;

                // Create a ToolCall struct containing the result.
                // Note: The result is placed in the `arguments` field for transport.
                let result_call = ToolCall {
                    id: call.id,
                    call_type: "function".to_string(),
                    function: querymt::FunctionCall {
                        name: call.function.name,
                        arguments: result_str,
                    },
                };
                tool_results.push(result_call);
            }

            // Add the tool results back to the conversation history.
            messages.push(ChatMessage::user().tool_result(tool_results).build());
            // Loop again to let the model process the tool results.
        } else {
            // No tool calls in the response, so the text is the final answer.
            return Ok(response.text().unwrap_or_default());
        }
    }
}

QueryMT's chat system is designed to be flexible, supporting simple Q&A, complex multi-turn dialogues, and sophisticated interactions involving external tools. The Tool and ToolChoice mechanisms provide fine-grained control over how LLMs can utilize functions.