feat: update analyze CLI to use typed analyzer framework

[ryan-arman]

Summary

• Port the typed analyzer framework from the analyzer-refactor branch, replacing the old DataFrame-centric approach with Pydantic-based typed result models
• Align config naming convention with the API's pattern: AnalyzerConfig uses type/display_name (backward compat for id/instance_id), TestConfigYAML uses display_name (backward compat for title)
• Add new analyzers (Deduplication, LLM-as-judge with preset criteria), custom metric support, and extended test engine with PERCENTAGE/RANGE test types
• Preserve full API backward compatibility: create_analyzer_from_config, TestParams, BatchTestEngine all maintain original import paths and behavior

API Compatibility

The API worker uses oumi as a submodule. All existing import patterns are preserved:

• from oumi.analyze import AnalysisPipeline, create_analyzer_from_config
• from oumi.analyze.testing import TestParams as OumiTestParams
• from oumi.analyze.testing.batch_engine import BatchTestEngine
• analyzer.analyzer_id = display_name pattern continues to work

Config Changes

YAML configs now use type/display_name (matching API's AnalyzerConfigInput):

analyzers:
  - type: length
    display_name: Length
    params:
      tokenizer_name: cl100k_base

Old format with id/instance_id is still accepted for backward compatibility.

Test plan

• All 120 analyze unit tests pass
• All 7 CLI analyze tests pass
• API import compatibility verified
• End-to-end CLI test (oumi analyze -c config.yaml -o /tmp/test -f json)
• Backward compat: old-style YAML configs with id/instance_id/title still parse correctly
• Review against API's analyze_activity.py usage patterns

🤖 Generated with Claude Code

[gitar-bot]

Gitar is working

_Gitar

[ryan-arman]

Code review

Found 2 issues:

1. Custom metrics via CLI are completely broken — all three CLI call sites invoke TypedAnalyzeConfig.from_yaml() without passing allow_custom_code=True. The default (False) raises ValueError when any custom metric has a function field, making the feature non-functional. In the --list-metrics path, the error is silently swallowed by a bare except Exception: pass. Either the CLI should expose a --allow-custom-code flag, or pass True for trusted local invocations.

oumi/src/oumi/cli/analyze.py

Lines 118 to 120 in b71dcb7

	try:
	typed_config = TypedAnalyzeConfig.from_yaml(config)
	if typed_config.custom_metrics:

oumi/src/oumi/cli/analyze.py

Lines 168 to 170 in b71dcb7

	):
	typed_config = TypedAnalyzeConfig.from_yaml(config)

oumi/src/oumi/analyze/cli.py

Lines 468 to 470 in b71dcb7

	# Load config
	config = TypedAnalyzeConfig.from_yaml(config_path)

Security gate defined at:

oumi/src/oumi/analyze/config.py

Lines 397 to 406 in b71dcb7

	# Security check: reject custom code unless explicitly allowed
	if not allow_custom_code:
	metrics_with_code = [m.id for m in custom_metrics if m.function.strip()]
	if metrics_with_code:
	raise ValueError(
	f"Configuration contains custom metrics with executable code: "
	f"{metrics_with_code}. This is a security risk if loading from "
	f"untrusted sources. Set allow_custom_code=True to explicitly "
	f"allow code execution, or remove the 'function' fields."
	)

2. Same-class analyzers without explicit analyzer_id silently overwrite each other — _get_analyzer_name falls back to analyzer.__class__.__name__ when no analyzer_id is set. Two instances of the same analyzer class (e.g., LengthAnalyzer with different params) get the same key, and the second overwrites the first in self._results. No warning is emitted. Config-level validation for duplicate display_name exists in from_dict(), but the pipeline itself has no guard when constructed directly with a list of analyzers.

oumi/src/oumi/analyze/pipeline.py

Lines 468 to 483 in b71dcb7

	def _get_analyzer_name(self, analyzer: AnyAnalyzer) -> str:
	"""Get the name for an analyzer.
	Uses the class name by default, but can be overridden by
	setting an 'analyzer_id' attribute on the analyzer.
	Args:
	analyzer: The analyzer instance.
	Returns:
	Name string for the analyzer.
	"""
	analyzer_id = getattr(analyzer, "analyzer_id", None)
	if analyzer_id is not None:
	return str(analyzer_id)
	return analyzer.__class__.__name__

oumi/src/oumi/analyze/pipeline.py

Lines 361 to 363 in b71dcb7

	result = run_func(analyzer)
	self._results[name] = result
	if is_batch and isinstance(result, list):

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[ryan-arman]

Both findings have been addressed:

1. Custom metrics via CLI are completely broken
Not applicable — custom_metrics.py was deleted in this PR along with all CustomMetricConfig references, the allow_custom_code gate, and the custom metrics YAML section. The entire feature was removed.

2. Same-class analyzers without explicit analyzer_id silently overwrite each other
Fixed — AnalysisPipeline.__init__() now validates for duplicate analyzer names at construction time and raises ValueError with a clear message. Test: test_duplicate_analyzer_names_raises_error in test_pipeline.py. The config-level from_dict() validation for duplicate display_name values was already in place; this adds the equivalent guard for direct pipeline construction.