The evaluator to bind (e.g., a hallucination evaluator)
The binding context containing the input mapping configuration
A new evaluator instance with the input mapping applied
Basic usage with simple field mapping:
import { bindEvaluator, createHallucinationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";
type MyData = {
question: string;
context: string;
answer: string;
};
const evaluator = bindEvaluator<MyData>(
createHallucinationEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
input: "question", // Evaluator expects "input", map from "question"
reference: "context", // Evaluator expects "reference", map from "context"
output: "answer", // Evaluator expects "output", map from "answer"
},
}
);
// Now you can evaluate with your data structure
const result = await evaluator.evaluate({
question: "What is AI?",
context: "AI is artificial intelligence...",
answer: "AI stands for artificial intelligence",
});
Using nested property access:
type ApiResponse = {
request: {
body: {
query: string;
context: string;
};
};
response: {
data: {
text: string;
};
};
};
const evaluator = bindEvaluator<ApiResponse>(
createHallucinationEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
input: "request.body.query",
reference: "request.body.context",
output: "response.data.text",
},
}
);
Using function-based mapping for data transformation:
type RawData = {
question: string;
contexts: string[]; // Array of context strings
answer: string;
};
const evaluator = bindEvaluator<RawData>(
createHallucinationEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
input: "question",
// Transform array to single string
reference: (data) => data.contexts.join("\n\n"),
output: "answer",
},
}
);
Using JSONPath for complex queries:
type ComplexData = {
conversation: {
messages: Array<{ role: string; content: string }>;
};
metadata: {
sources: string[];
};
};
const evaluator = bindEvaluator<ComplexData>(
createHallucinationEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
// Extract last user message
input: "$.conversation.messages[?(@.role=='user')].content[-1]",
// Extract all sources
reference: "$.metadata.sources[*]",
// Extract last assistant message
output: "$.conversation.messages[?(@.role=='assistant')].content[-1]",
},
}
);
Binding multiple evaluators with different mappings:
type EvaluationData = {
userQuery: string;
systemContext: string;
modelOutput: string;
expectedOutput?: string;
};
// Hallucination evaluator
const hallucinationEvaluator = bindEvaluator<EvaluationData>(
createHallucinationEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
input: "userQuery",
reference: "systemContext",
output: "modelOutput",
},
}
);
// Document relevancy evaluator (if it exists)
const relevancyEvaluator = bindEvaluator<EvaluationData>(
createDocumentRelevanceEvaluator({ model: openai("gpt-4") }),
{
inputMapping: {
query: "userQuery",
document: "systemContext",
output: "modelOutput",
},
}
);
Binds an evaluator to a specific data structure using input mapping.
This function creates a new evaluator instance that automatically transforms your data structure to match what the evaluator expects. This is particularly useful when your data schema doesn't match the evaluator's expected input format.