feat: concretize json type from query

[MichaelScofield]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

as title

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[gemini-code-assist]

Code Review

This pull request introduces a query-driven JSON type concretization mechanism to improve JSON data handling. A new optimizer rule, JsonTypeConcretizeRule, deduces expected JSON structures from json_get calls and passes these as hints to the storage engine via the ScanRequest. The FlatProjectionMapper and JsonArray have been updated to use these hints for aligning and casting JSON fields during scans, with JsonArray::try_align now discarding extra fields instead of failing. Feedback identifies a high-severity issue where the optimizer rule fails to trim the $ prefix from JSON paths, leading to incorrect type mapping. Additionally, there is a concern that removing explicit schema enforcement in the Parquet reader could negatively impact performance and pushdown capabilities.

[MichaelScofield]

@codex review
@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to deduce and concretize JSON types during query optimization to improve data alignment in the storage layer. It adds a new optimizer rule, JsonTypeConcretizeRule, which extracts expected JSON schemas from json_get calls and passes them as hints to the scan request. The storage engine uses these hints to align JSON arrays by casting fields or discarding extra data. Feedback identifies critical issues in the deduction logic, specifically regarding potential data loss when columns are accessed both directly and via json_get, the lack of support for the $ path prefix, and non-defensive error handling that could cause queries to fail. Additionally, the reviewer pointed out that modifying TableProvider state within an optimizer rule introduces side effects and concurrency risks.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 94535676fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Adds query-driven JSON type concretization so JSON extension columns can be read/converted using the types implied by query expressions (e.g., casts on json_get / j.a.b::Int64), improving correctness/consistency of JSON projections across read paths.

Changes:

• Introduces a DataFusion optimizer rule (JsonTypeConcretizeRule) to deduce JSON native types from query expressions and attach them as scan hints.
• Plumbs a json_type_hint through ScanRequest into mito2 scan/projection logic, and aligns JSON arrays against the concretized schema during conversion.
• Updates JSON type merge/Arrow-type conversion utilities and adjusts SQLness golden tests/output.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/cases/standalone/common/types/json/json2.sql	Adds result queries to validate JSON projections after concretization.
tests/cases/standalone/common/types/json/json2.result	Updates expected outputs for added JSON queries.
tests/cases/standalone/common/tql-explain-analyze/explain.result	Accounts for the new optimizer rule in verbose explain output.
src/store-api/src/storage/requests.rs	Adds json_type_hint to ScanRequest and prints it in Display.
src/query/src/query_engine/state.rs	Registers JsonTypeConcretizeRule in the logical optimizer pipeline.
src/query/src/optimizer/json_type_concretize.rs	New optimizer rule that deduces JSON types from json_get calls and sets scan hints.
src/query/src/optimizer.rs	Exposes the new optimizer module.
src/query/src/dummy_catalog.rs	Adds a setter on DummyTableProvider to store JSON type hints into the scan request.
src/mito2/src/sst/parquet/reader.rs	Adjusts Parquet Arrow reader options when JSON extension fields exist.
src/mito2/src/read/scan_region.rs	Applies query-driven JSON type hints by rewriting the projection output schema.
src/mito2/src/read/flat_projection.rs	Aligns JSON arrays to the (possibly concretized) JSON schema during conversion.
src/mito2/Cargo.toml	Adds arrow-schema dependency.
src/datatypes/src/vectors/json/array.rs	Changes JSON alignment to discard extra fields rather than error.
src/datatypes/src/types/json_type.rs	Adds JsonNativeType::merge and Arrow-type conversion; updates merge logic.
src/common/function/src/scalars/json/json_get.rs	Makes with_type non-optional internally (defaulting to Utf8View) and simplifies builders.
Cargo.lock	Records the new dependency resolution (arrow-schema).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MichaelScofield]

@sunng87 @fengys1996 PTAL

[fengys1996]

LGTM