Add MBPP evaluator

[natke]

Summary

Adds MBPP evaluator support for the ortgenai backend, enabling generative LM eval tasks (e.g., MBPP, HumanEval) via lm-eval. Key changes include:

• Add generate_until support to LMEvalORTGenAIEvaluator for token-by-token generation with stop-sequence handling
• Wire confirm_run_unsafe_code opt-in flag from CLI through to lm_eval.simple_evaluate for MBPP safety compliance
• Improve stop-sequence trimming to cut at the earliest match across all provided stop strings
• Improve generate_until with full-sequence decode for correct tokenizer whitespace/punctuation handling
• Harden gen_kwargs parsing: max tokens, temperature, and do_sample coercion for string/int inputs
• Add evaluator tests for unsafe-code flag propagation, multi-stop ordering, and token parsing edge cases

Example Olive config

The evaluator can be used standalone against a pre-existing ONNX model — no optimization passes are required. The passes field defaults to empty, and evaluate_input_model defaults to true, so Olive evaluates the input model directly:

{
    "input_model": {
        "type": "OnnxModel",
        "model_path": "models/Qwen2.5-Coder-0.5B-Instruct-cpu-int4"
    },
    "systems": {
        "local_system": {
            "type": "LocalSystem",
            "accelerators": [{ "device": "cpu", "execution_providers": ["CPUExecutionProvider"] }]
        }
    },
    "evaluators": {
        "lm_evaluator": {
            "type": "LMEvaluator",
            "tasks": ["mbpp"],
            "model_class": "ortgenai",
            "batch_size": 1,
            "max_length": 2048,
            "limit": 10,
            "confirm_run_unsafe_code": true
        }
    },
    "target": "local_system",
    "evaluator": "lm_evaluator"
}

Run with:

olive run -c config.json

Or via the CLI directly (no config file needed):

olive benchmark -m models/Qwen2.5-Coder-0.5B-Instruct-cpu-int4 \
    --tasks mbpp \
    --backend ortgenai \
    --confirm_run_unsafe_code

Note: MBPP executes generated Python code to check correctness. The confirm_run_unsafe_code flag (or --confirm_run_unsafe_code on the CLI) is required to acknowledge this.

Validation

• pytest -q test/evaluator/test_olive_evaluator.py -k "generate_until or confirm_run_unsafe_code or lmeval"
• pytest -q test/cli/test_cli.py -k "benchmark_command_onnxmodel_with_ortgenai_backend or benchmark_command_hfmodel"

[Copilot]

Pull request overview

This PR improves Olive’s lm-eval integration for the ORT GenAI backend by adding a working generate_until implementation with more robust stopping behavior, while also exposing lm-eval’s confirm_run_unsafe_code flag through Olive’s evaluator and CLI.

Changes:

• Implement generate_until for ortgenai with EOS and multi-stop trimming, and improve the unsupported-backend error for plain ort.
• Plumb confirm_run_unsafe_code through LMEvaluator and olive benchmark, with unit test coverage.
• Add evaluator tests covering multi-stop ordering, EOS handling, incremental decode behavior, and max-token parsing edge cases.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
olive/evaluator/lmeval_ort.py	Adds ORT-GenAI generate_until, supports multiple EOS IDs, and improves ort backend error messaging.
olive/evaluator/olive_evaluator.py	Adds confirm_run_unsafe_code to LMEvaluator and forwards it to lm_eval.simple_evaluate.
olive/cli/benchmark.py	Adds --confirm_run_unsafe_code CLI flag and writes it into the generated run config when enabled.
test/evaluator/test_olive_evaluator.py	Adds unit tests for confirm_run_unsafe_code forwarding and ORT-GenAI generate_until behaviors.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

test/evaluator/test_olive_evaluator.py:1

• The tests cover the max_gen_toks == 0 early-return path, but not the prompt_len >= self.max_length branch (which also triggers cache-hook behavior). Add a unit test where tokenizer.encode(...).tolist() returns a prompt length >= evaluator.max_length, and assert it returns \"\" and writes the empty partial to the cache hook.

# -------------------------------------------------------------------------

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.