Add MBPP evaluator#2439

Open

natke wants to merge 11 commits intomainfrom

feature/add-mbpp-evaluator

Commits on Apr 29, 2026

Add MBPP/code generation evaluator support via ortgenai backend

natke
and
Copilot
committed
Fix ORT GenAI stop handling and generation robustness
natke
committed

Commits on Apr 30, 2026

Address PR review comments in lmeval_ort generate_until

natke
and
Copilot
committed
Fix generate_until edge cases from PR review

natke
and
Copilot
committed
Fix tail-buffer and temperature coercion in generate_until

natke
and
Copilot
committed
Apply ruff formatting to fix lintrunner CI failure

natke
and
Copilot
committed
Fix batch_size in search_options and confirm_run_unsafe_code compat with older lm-eval

natke
and
Copilot
committed
Add test asserting batch_size is not passed to set_search_options

natke
and
Copilot
committed
Fix Copilot review batch 5: inspect.signature, full-seq decode, do_sample coercion

natke
and
Copilot
committed
Fix Copilot review batch 6: signature guard, null flag, tuple until, ort_genai stub

natke
and
Copilot
committed
Fix critical bugs from code review: do_sample flag, config clobber, tqdm, caching

natke
and
Copilot
committed