LLM Integration

Integrate LLM-powered adaptive intelligence into your behavior trees with the BYOM (Bring Your Own Model) architecture.

BYOM Philosophy

Igris BTree is LLM-agnostic. You choose your model, inference backend, and costs. The system provides the integration points - you bring the intelligence.

Why BYOM?

No vendor lock-in: Use any LLM provider
Full control: Choose model, costs, privacy policy
Flexibility: Local models, cloud APIs, or both
Testability: Mock providers for unit tests
Future-proof: Swap models without code changes

LlmProvider Trait

The LlmProvider trait is the core abstraction for LLM integration.

Trait Definition

#[async_trait]
pub trait LlmProvider: Send + Sync {
    /// Generate text from prompt
    async fn generate(&self, prompt: &str) -> anyhow::Result<String>;

    /// Provider name for logging
    fn name(&self) -> &str;

    /// Optional: streaming generation
    async fn generate_stream(
        &self,
        prompt: &str,
    ) -> anyhow::Result<tokio::sync::mpsc::Receiver<String>> {
        anyhow::bail!("Streaming not supported by this provider")
    }
}

Required Methods

generate()

Synchronous (non-streaming) text generation:

async fn generate(&self, prompt: &str) -> anyhow::Result<String>

Input: Text prompt
Output: Generated text
Errors: Network errors, API errors, timeout, etc.

name()

Provider identifier for logging:

fn name(&self) -> &str

Returns a human-readable name (e.g., "openai", "ollama", "mock").

Optional Methods

generate_stream()

Streaming generation (token-by-token):

async fn generate_stream(
    &self,
    prompt: &str,
) -> anyhow::Result<tokio::sync::mpsc::Receiver<String>>

Default implementation returns an error. Override to support streaming.

MockLlmProvider

Built-in mock provider for testing and development.

Basic Usage

use igris_btree::MockLlmProvider;
use std::sync::Arc;

// Custom responses
let provider = Arc::new(MockLlmProvider::new(vec![
    r#"{"type": "Sequence", "name": "plan1", "children": []}"#.to_string(),
    r#"{"type": "Sequence", "name": "plan2", "children": []}"#.to_string(),
]));

// Use in context
let context = BTreeContext::new().with_llm(provider);

Built-in Scenarios

Navigation Plan

let provider = Arc::new(MockLlmProvider::with_navigation_plan());

Returns a pre-built navigation sequence:

{
  "type": "Sequence",
  "name": "Navigate to warehouse",
  "children": [
    {
      "type": "Action",
      "name": "Move forward",
      "tool": "navigate",
      "args": {"direction": "forward", "distance": 10}
    },
    {
      "type": "Action",
      "name": "Turn right",
      "tool": "navigate",
      "args": {"direction": "right", "angle": 90}
    }
  ]
}

Recovery Plan

let provider = Arc::new(MockLlmProvider::with_recovery_plan());

Returns two plans (for testing ReplanOnFailure):

Primary plan (simulates failure)
Recovery plan (fallback)

Custom Mock

use igris_btree::MockLlmProvider;

let provider = MockLlmProvider::new(vec![
    // First call returns this
    r#"{"type": "Action", "name": "step1", "tool": "tool1", "args": {}}"#.to_string(),

    // Second call returns this
    r#"{"type": "Action", "name": "step2", "tool": "tool2", "args": {}}"#.to_string(),
]);

// Cycles through responses

Implementing Custom Providers

Example: Ollama Provider

use igris_btree::LlmProvider;
use async_trait::async_trait;
use reqwest::Client;
use serde_json::json;

pub struct OllamaProvider {
    client: Client,
    base_url: String,
    model: String,
}

impl OllamaProvider {
    pub fn new(base_url: impl Into<String>, model: impl Into<String>) -> Self {
        Self {
            client: Client::new(),
            base_url: base_url.into(),
            model: model.into(),
        }
    }
}

#[async_trait]
impl LlmProvider for OllamaProvider {
    async fn generate(&self, prompt: &str) -> anyhow::Result<String> {
        let response = self.client
            .post(format!("{}/api/generate", self.base_url))
            .json(&json!({
                "model": self.model,
                "prompt": prompt,
                "stream": false
            }))
            .send()
            .await?;

        let data: serde_json::Value = response.json().await?;
        let text = data["response"]
            .as_str()
            .ok_or_else(|| anyhow::anyhow!("No response field"))?;

        Ok(text.to_string())
    }

    fn name(&self) -> &str {
        "ollama"
    }
}

// Usage
let provider = Arc::new(OllamaProvider::new("http://localhost:11434", "phi3"));
let context = BTreeContext::new().with_llm(provider);

Example: OpenAI Provider

use igris_btree::LlmProvider;
use async_trait::async_trait;
use reqwest::Client;
use serde_json::json;

pub struct OpenAIProvider {
    client: Client,
    api_key: String,
    model: String,
}

impl OpenAIProvider {
    pub fn new(api_key: impl Into<String>, model: impl Into<String>) -> Self {
        Self {
            client: Client::new(),
            api_key: api_key.into(),
            model: model.into(),
        }
    }
}

#[async_trait]
impl LlmProvider for OpenAIProvider {
    async fn generate(&self, prompt: &str) -> anyhow::Result<String> {
        let response = self.client
            .post("https://api.openai.com/v1/chat/completions")
            .header("Authorization", format!("Bearer {}", self.api_key))
            .json(&json!({
                "model": self.model,
                "messages": [{
                    "role": "user",
                    "content": prompt
                }],
                "temperature": 0.7
            }))
            .send()
            .await?;

        let data: serde_json::Value = response.json().await?;
        let text = data["choices"][0]["message"]["content"]
            .as_str()
            .ok_or_else(|| anyhow::anyhow!("No content in response"))?;

        Ok(text.to_string())
    }

    fn name(&self) -> &str {
        "openai"
    }
}

// Usage
let provider = Arc::new(OpenAIProvider::new(
    std::env::var("OPENAI_API_KEY")?,
    "gpt-4"
));
let context = BTreeContext::new().with_llm(provider);

Provider Best Practices

Error Handling: Return descriptive errors
Timeouts: Set reasonable timeouts (igris-rt provides 5s default)
Retries: Implement retry logic for transient failures
Logging: Log requests and responses for debugging
Costs: Track token usage and costs
Caching: Cache responses when appropriate
Rate Limiting: Respect API rate limits

LLMPlannerNode

Generates behavior tree plans using LLM.

How It Works

1. Read task description from blackboard
2. Construct prompt with task and available tools
3. Call LLM provider
4. Parse response as JSON
5. Validate tree structure
6. Store plan in blackboard

Prompt Template

The LLMPlannerNode constructs prompts like this:

Generate a behavior tree plan for the following task:

Task: {task_description}

Available tools:
- navigate(destination, speed): Navigate to location
- pick(object_id): Pick up object
- place(location): Place held object

Generate a JSON behavior tree with nodes:
- Sequence: Execute children in order
- Selector: Try children until one succeeds
- Action: Execute a tool with parameters

Example format:
{
  "type": "Sequence",
  "name": "Complete task",
  "children": [
    {
      "type": "Action",
      "name": "Navigate to target",
      "tool": "navigate",
      "args": {"destination": "warehouse", "speed": 1.0}
    }
  ]
}

Return ONLY the JSON, no explanation.

Usage Example

use igris_btree::prelude::*;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Setup LLM provider
    let llm = Arc::new(MockLlmProvider::with_navigation_plan());
    let mut context = BTreeContext::new().with_llm(llm);

    // Set task in blackboard
    context.blackboard.set(
        "mission_task",
        "Navigate to warehouse and pick up box #42"
    ).await;

    // Create planner node
    let mut planner = LLMPlannerNode::new(
        "mission_planner",
        "mission_task",  // Read task from here
        "execution_plan" // Write plan here
    );

    // Generate plan
    let status = planner.tick(&mut context).await?;

    if status == NodeStatus::Success {
        let plan: String = context.blackboard.get("execution_plan").await?;
        println!("Generated plan: {}", plan);
    }

    Ok(())
}

Configuration

// Planner uses these from context:
// - llm_provider: LLM for generation
// - tool_registry: Available tools for prompt
// - rt_executor: Bounds LLM call to 5s (optional)

Error Handling

The planner handles errors gracefully:

No LLM provider: Returns Failure with error log
LLM timeout: Retries up to 3 times
Invalid JSON: Attempts to extract JSON from markdown
Parse errors: Returns Failure with error details

SubtreeLoader

Loads and executes dynamically generated subtrees.

How It Works

1. Read JSON plan from blackboard
2. Parse JSON into BTreeNode instances
3. Tick loaded subtree
4. Return subtree status

Supported Node Types

Currently parses:

Sequence: Sequential execution
Selector: Fallback logic
Action: ToolAction nodes
Condition: CheckBlackboard nodes

Future support planned for decorators and parallel nodes.

Usage Example

use igris_btree::prelude::*;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let llm = Arc::new(MockLlmProvider::with_navigation_plan());
    let mut context = BTreeContext::new().with_llm(llm);

    // Complete pipeline: Task → Plan → Execute
    let mut tree = Sequence::new("llm_pipeline")
        .add_child(Box::new(SetBlackboard::new(
            "set_task",
            "task",
            "Navigate to warehouse"
        )))
        .add_child(Box::new(LLMPlannerNode::new(
            "planner",
            "task",
            "plan"
        )))
        .add_child(Box::new(SubtreeLoader::new(
            "executor",
            "plan"
        )));

    // Execute complete pipeline
    let executor = BTreeExecutor::new();
    let result = executor.execute(&mut tree, &mut context).await?;

    println!("Pipeline status: {:?}", result.status);

    Ok(())
}

JSON Plan Format

{
  "type": "Sequence",
  "name": "Generated plan",
  "children": [
    {
      "type": "Action",
      "name": "Step 1",
      "tool": "tool_name",
      "args": {
        "param1": "value1",
        "param2": 42
      }
    },
    {
      "type": "Sequence",
      "name": "Nested sequence",
      "children": [
        {
          "type": "Action",
          "name": "Step 2",
          "tool": "another_tool",
          "args": {}
        }
      ]
    }
  ]
}

ReplanOnFailure Decorator

Automatically replans when child fails using LLM.

How It Works

1. Tick child normally
2. If child succeeds → Return Success
3. If child fails:
   a. Read task from blackboard
   b. Ask LLM for recovery plan
   c. Load and execute recovery plan
   d. If recovery succeeds → Return Success
   e. If recovery fails → Increment counter, repeat
4. If max replans reached → Return Failure

Usage Example

use igris_btree::prelude::*;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let llm = Arc::new(MockLlmProvider::with_recovery_plan());
    let mut context = BTreeContext::new().with_llm(llm);

    // Set task description
    context.blackboard.set(
        "navigation_task",
        "Navigate to warehouse avoiding obstacles"
    ).await;

    // Wrap action with adaptive recovery
    let mut adaptive = ReplanOnFailure::new(
        "adaptive_navigation",
        Box::new(ToolAction::new("navigate", "nav_to_warehouse", params)),
        "navigation_task",  // Task description key
        "recovery_plan",    // Recovery plan key
        3                   // Max 3 replans
    );

    // If primary action fails, LLM generates recovery
    let status = adaptive.tick(&mut context).await?;

    Ok(())
}

Replan Prompt

When child fails, the decorator generates a prompt like:

The following action failed:
Action: navigate
Parameters: {"destination": "warehouse", "speed": 1.0}

Task description: Navigate to warehouse avoiding obstacles

Generate a recovery plan as a JSON behavior tree to complete the task.
Consider alternative approaches and error recovery strategies.

Return ONLY the JSON tree, no explanation.

Configuration

ReplanOnFailure::new(
    name,
    child,                // Primary action
    task_key,             // Blackboard key for task description
    plan_key,             // Blackboard key for recovery plans
    max_replans           // Maximum replan attempts (1-3 recommended)
)

Best Practices

Set clear task descriptions: More context = better recovery plans
Limit replans: 1-3 replans is usually sufficient
Monitor costs: Each replan is an LLM call
Log replan events: Track when and why replanning occurs
Fallback safety: Combine with Selector for non-LLM fallback

Bounded Execution

LLM calls are bounded by igris-rt for real-time guarantees.

Default Timeout

LLM calls are limited to 5 seconds by default:

// In LLMPlannerNode and ReplanOnFailure:
// - Uses igris-rt executor if available
// - Falls back to unbounded tokio execution
// - Retries up to 3 times on timeout

Custom Timeout

Provide custom rt_executor in context:

// Custom bounded execution
let rt_executor = Arc::new(MyRtExecutor::new(
    Duration::from_secs(10)  // 10 second timeout
));

let context = BTreeContext::new()
    .with_llm(llm)
    .with_rt_executor(rt_executor);

Why Bounded?

Predictability: No indefinite waits
Real-time safety: Critical systems can't wait forever
Cost control: Prevents runaway LLM usage
Debugging: Easier to identify hanging calls

Testing with LLMs

Unit Testing

Use MockLlmProvider for deterministic tests:

#[tokio::test]
async fn test_llm_planning() {
    let llm = Arc::new(MockLlmProvider::new(vec![
        r#"{"type": "Sequence", "name": "test_plan", "children": []}"#.to_string()
    ]));

    let mut context = BTreeContext::new().with_llm(llm);
    context.blackboard.set("task", "Test task").await;

    let mut planner = LLMPlannerNode::new("planner", "task", "plan");
    let status = planner.tick(&mut context).await.unwrap();

    assert_eq!(status, NodeStatus::Success);
    assert!(context.blackboard.contains("plan").await);
}

Integration Testing

Test with real LLM but mock external tools:

#[tokio::test]
#[ignore]  // Run with --ignored for integration tests
async fn test_real_llm_planning() {
    let llm = Arc::new(OllamaProvider::new("http://localhost:11434", "phi3"));

    // Test real LLM generation
    let result = llm.generate("Generate a simple plan").await;
    assert!(result.is_ok());
}

Testing Patterns

Happy Path: Use MockLlmProvider with valid JSON
Error Cases: Test invalid JSON, missing provider, timeouts
Recovery: Test ReplanOnFailure with multiple plans
Performance: Measure LLM call latency
Integration: Test with real LLM in staging environment

Performance Optimization

Caching Strategies

use std::collections::HashMap;
use std::sync::RwLock;

pub struct CachedLlmProvider {
    inner: Arc<dyn LlmProvider>,
    cache: RwLock<HashMap<String, String>>,
}

impl CachedLlmProvider {
    pub fn new(inner: Arc<dyn LlmProvider>) -> Self {
        Self {
            inner,
            cache: RwLock::new(HashMap::new()),
        }
    }
}

#[async_trait]
impl LlmProvider for CachedLlmProvider {
    async fn generate(&self, prompt: &str) -> anyhow::Result<String> {
        // Check cache
        {
            let cache = self.cache.read().unwrap();
            if let Some(cached) = cache.get(prompt) {
                return Ok(cached.clone());
            }
        }

        // Generate and cache
        let response = self.inner.generate(prompt).await?;

        {
            let mut cache = self.cache.write().unwrap();
            cache.insert(prompt.to_string(), response.clone());
        }

        Ok(response)
    }

    fn name(&self) -> &str {
        "cached"
    }
}

Batch Planning

Generate multiple plans at once:

// Instead of multiple single plans:
LLMPlannerNode::new("plan1", "task1", "plan1")
LLMPlannerNode::new("plan2", "task2", "plan2")

// Batch into single call:
BatchPlannerNode::new("batch", vec![
    ("task1", "plan1"),
    ("task2", "plan2"),
])

Parallel LLM Calls

Use Parallel node for concurrent LLM calls:

Parallel::new("parallel_planning", ParallelPolicy::RequireAll)
    .add_child(Box::new(LLMPlannerNode::new("plan_a", "task_a", "result_a")))
    .add_child(Box::new(LLMPlannerNode::new("plan_b", "task_b", "result_b")))

Cost Management

Tracking LLM Calls

use std::sync::atomic::{AtomicU64, Ordering};

pub struct MeteredLlmProvider {
    inner: Arc<dyn LlmProvider>,
    call_count: AtomicU64,
    total_tokens: AtomicU64,
}

impl MeteredLlmProvider {
    pub fn new(inner: Arc<dyn LlmProvider>) -> Self {
        Self {
            inner,
            call_count: AtomicU64::new(0),
            total_tokens: AtomicU64::new(0),
        }
    }

    pub fn stats(&self) -> (u64, u64) {
        (
            self.call_count.load(Ordering::Relaxed),
            self.total_tokens.load(Ordering::Relaxed)
        )
    }
}

#[async_trait]
impl LlmProvider for MeteredLlmProvider {
    async fn generate(&self, prompt: &str) -> anyhow::Result<String> {
        self.call_count.fetch_add(1, Ordering::Relaxed);

        let response = self.inner.generate(prompt).await?;

        // Estimate tokens (rough)
        let tokens = (prompt.len() + response.len()) / 4;
        self.total_tokens.fetch_add(tokens as u64, Ordering::Relaxed);

        Ok(response)
    }

    fn name(&self) -> &str {
        "metered"
    }
}

Cost-Aware Strategies

Minimize calls: Use LLM only when deterministic fails
Cache aggressively: Cache similar prompts
Local first: Use local models for development
Cloud for production: Use cloud only when needed
Monitor usage: Track calls and costs

Advanced Patterns

Hybrid Planning

Combine deterministic and LLM planning:

Selector::new("hybrid_planning")
    // Try deterministic first (fast, free)
    .add_child(Box::new(Sequence::new("deterministic")
        .add_child(Box::new(CheckBlackboard::new("has_plan", "cached_plan", true)))
        .add_child(Box::new(SubtreeLoader::new("exec", "cached_plan")))
    ))
    // Fall back to LLM (slow, costs)
    .add_child(Box::new(Sequence::new("llm_fallback")
        .add_child(Box::new(LLMPlannerNode::new("planner", "task", "plan")))
        .add_child(Box::new(SubtreeLoader::new("exec", "plan")))
    ))

Multi-Stage Planning

Break complex tasks into stages:

Sequence::new("multi_stage")
    .add_child(Box::new(LLMPlannerNode::new("stage1", "task", "plan1")))
    .add_child(Box::new(SubtreeLoader::new("exec1", "plan1")))
    .add_child(Box::new(LLMPlannerNode::new("stage2", "task", "plan2")))
    .add_child(Box::new(SubtreeLoader::new("exec2", "plan2")))

Confidence-Based Planning

Use LLM confidence to decide whether to execute:

Sequence::new("confident_execution")
    .add_child(Box::new(LLMPlannerWithConfidence::new("planner", "task", "plan", "confidence")))
    .add_child(Box::new(CheckBlackboard::new("check_confidence", "confidence", 0.8)))
    .add_child(Box::new(SubtreeLoader::new("exec", "plan")))

Next Steps

Visualization - Monitor LLM calls in real-time
Examples - Complete LLM examples
API Reference - Full LlmProvider API
Testing Guide - Test LLM integration

Summary

Key takeaways:

BYOM: Bring your own model, no vendor lock-in
LlmProvider trait: Simple abstraction for any LLM
MockLlmProvider: Testing without real LLMs
LLMPlannerNode: Generate plans dynamically
SubtreeLoader: Execute generated plans
ReplanOnFailure: Automatic recovery with LLM
Bounded execution: 5s timeout for real-time safety
Cost management: Track and optimize LLM usage