Interface Evaluator<ExampleType>

The Base Evaluator interface This is the interface that all evaluators must implement

interface Evaluator<ExampleType extends Record<string, unknown>> {
    evaluate: EvaluatorFn<ExampleType>;
    name: string;
    optimizationDirection?: OptimizationDirection;
    source: EvaluationSource;
}

Properties

The function that evaluates the example

name: string

The name of the evaluator / the metric that it measures

optimizationDirection?: OptimizationDirection

The direction to optimize the numeric evaluation score E.x. "MAXIMIZE" means that the higher the score, the better the evaluation

source: EvaluationSource

The source of the evaluation. Also known as the "kind" of evaluator.