Skip to content

[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format-based inference#27793

Open
raminqaf wants to merge 4 commits intoapache:masterfrom
raminqaf:FLINK-14925
Open

[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format-based inference#27793
raminqaf wants to merge 4 commits intoapache:masterfrom
raminqaf:FLINK-14925

Conversation

@raminqaf
Copy link
Contributor

What is the purpose of the change

This pull request makes the TO_TIMESTAMP function precision-aware when a format pattern is provided. Previously,
TO_TIMESTAMP always returned TIMESTAMP(3) regardless of the format pattern's fractional second precision, which forced users to lose sub-millisecond data. This is the TO_TIMESTAMP counterpart to the TO_TIMESTAMP_LTZ precision support added in FLINK-39244.

The output type for the 1-arg variant remains TIMESTAMP(3) for backward compatibility. For the 2-arg variant, precision is
inferred from the format pattern's trailing S count (e.g., SSSSSSTIMESTAMP(6)), with a minimum of 3.

As part of this change, TO_TIMESTAMP is migrated from the legacy Calcite-native function pattern (FlinkSqlOperatorTable + StringCallGen codegen) to the modern bridging function pattern (BuiltInFunctionDefinition + runtimeClass), matching how
TO_TIMESTAMP_LTZ is implemented. This was made possible by fixing the function name from camelCase "toTimestamp" to "TO_TIMESTAMP", which allows CoreModule to resolve it correctly for SQL queries without needing a separate FlinkSqlOperatorTable entry.

Brief change log

  • Type strategy (ToTimestampTypeStrategy): New output type strategy that returns TIMESTAMP(3) for the 1-arg variant
    and TIMESTAMP(max(sCount, 3)) for the 2-arg variant, where sCount is inferred from the format pattern's trailing S
    characters.
  • Runtime function (ToTimestampFunction): New runtime class with eval(StringData) and eval(StringData, StringData)
    methods. The 2-arg variant passes precisionFromFormat(format) to parseTimestampData for precision-aware parsing.
  • Function definition (BuiltInFunctionDefinitions): Changed name from "toTimestamp" to "TO_TIMESTAMP" (removing the need for explicit sqlName), added runtimeClass, and switched output type strategy to SpecificTypeStrategies.TO_TIMESTAMP.
  • Removed legacy plumbing: Removed FlinkSqlOperatorTable.TO_TIMESTAMP, DirectConvertRule mapping, StringCallGen
    cases, and BuiltInMethods.STRING_TO_TIMESTAMP / STRING_TO_TIMESTAMP_WITH_FORMAT — all superseded by the bridging function mechanism.
  • Documentation: Updated sql_functions.yml, sql_functions_zh.yml, and Python expressions.py / expression.py
    docstrings with precision-dependent output types and examples.

Verifying this change

This change added tests and can be verified as follows:

  • Added type strategy unit tests in ToTimestampTypeStrategyTest covering 1-arg default precision, 2-arg format-based
    precision (SSS/SSSSSS/SSSSSSSSS/no-S), invalid argument types, and argument count validation.
  • Added integration tests in TimeFunctionsITCase for 1-arg truncation to precision 3, 2-arg precision 6/9 from format, SSS format staying at precision 3, fewer input digits than format precision, unparsable string, and null input.
  • Removed redundant legacy tests from TemporalTypesTest.scala that are now covered by the new TimeFunctionsITCase tests.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no — the ToTimestampFunction.eval() methods call the same DateTimeUtils.parseTimestampData methods as before.
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs / JavaDocs / PyDocs

@snuyanzin snuyanzin marked this pull request as ready for review March 20, 2026 08:55
@flinkbot
Copy link
Collaborator

flinkbot commented Mar 20, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf raminqaf marked this pull request as draft March 20, 2026 08:58
@raminqaf raminqaf marked this pull request as ready for review March 20, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants