Function createToolSelectionEvaluator

  • createToolSelectionEvaluator<
        RecordType extends
            Record<string, unknown> = ToolSelectionEvaluationRecord,
    >(
        args: ToolSelectionEvaluatorArgs<RecordType>,
    ): ClassificationEvaluator<RecordType>

    Creates a tool selection evaluator function.

    This function returns an evaluator that determines whether the correct tool was selected for a given context. Unlike the tool invocation evaluator which checks if the tool was called correctly with proper arguments, this evaluator focuses on whether the right tool was chosen in the first place.

    The evaluator checks for:

    • Whether the LLM chose the best available tool for the user query
    • Whether the tool name exists in the available tools list
    • Whether the correct number of tools were selected for the task
    • Whether the tool selection is safe and appropriate

    Type Parameters

    Parameters

    Returns ClassificationEvaluator<RecordType>

    An evaluator function that takes a ToolSelectionEvaluationRecord and returns a classification result indicating whether the tool selection is correct or incorrect.

    Example

    const evaluator = createToolSelectionEvaluator({ model: openai("gpt-4o-mini") });

    const result = await evaluator.evaluate({
      input: "User: What is the weather in San Francisco?",
      availableTools: `WeatherTool: Get the current weather for a location.
    NewsTool: Stay connected to global events with our up-to-date news around the world.
    MusicTool: Create playlists, search for music, and check the latest music trends.`,
      toolSelection: "WeatherTool"
    });
    console.log(result.label); // "correct" or "incorrect"

Settings

Member Visibility