[FLINK-39185][table] Introduce BITMAP type for Table API/SQL#27778
[FLINK-39185][table] Introduce BITMAP type for Table API/SQL#27778dylanhz wants to merge 4 commits intoapache:masterfrom
Conversation
52de377 to
8eb53b0
Compare
lincoln-lil
left a comment
There was a problem hiding this comment.
@dylanhz Thanks for working on this! I've left some comments there, and for the tests, can you add cases covering cast call results, e.g., CAST(bitmap AS STRING) and CAST(bitmap AS VARBINARY).
Another question for the python part, should we adapt bitmap type in PythonTableUtils.converter()?
...-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/BitmapType.java
Show resolved
Hide resolved
@lincoln-lil Thanks for the feedback!
|
Ok, make sense to me. |
lincoln-lil
left a comment
There was a problem hiding this comment.
@dylanhz thanks for the updates!
What is the purpose of the change
This pull request adds Table API/SQL support for the BITMAP data type introduced in FLIP-556. It integrates BITMAP into Flink's type system, internal data format, planner, and code generation, enabling BITMAP columns to be used in SQL queries and Table API programs.
This is the third PR in the FLIP-556 series:
FLINK-39183): Parser supportFLINK-39184): DataStream API support (flink-core)FLINK-39185): Table API/SQL support (this PR)Brief change log
Suggested review order:
BitmapType,LogicalTypeRoot.BITMAP,LogicalTypeFamily.EXTENSION, visitor support, cast rules, and type parsingDataTypes.BITMAP(), registered type mappings inClassDataTypeConverter,TypeInfoDataTypeConverter, andValueDataTypeConverterRowData/ArrayDatawithgetBitmap(), implemented inBinaryRowData/BinaryArrayData/GenericRowData/GenericArrayData/NestedRowData; addedBinarySegmentUtils.readBitmap()andBinaryWriter.writeBitmap()BitmapRelDataType, integrated intoFlinkTypeFactory(bidirectional conversion betweenBitmapTypeandBitmapRelDataType), extendedCodeGenUtilsfor code generation, and updatedExpressionReducerBitmapToStringCastRuleandBitmapToBinaryCastRule(with trim/pad semantics); restrictedCAST(x AS BITMAP)inSqlCastFunctionBitmapBitmapConverter,DataFormatConverters.BitmapConverter, and JSON serialization/deserialization forBitmapTypeVerifying this change
This change added tests and can be verified as follows:
BitmapSemanticTest: End-to-end integration tests for BITMAP in SQL/Table API, covering source/sink roundtrip, projection, filtering, UDF invocation, and UDAF aggregationBinaryRowDataTest/BinaryArrayDataTest: Unit tests for BITMAP read/write in binary row and array formatsRowDataTest: Verifies BITMAP field access andFieldGetterinRowDataDataTypesTest: VerifiesDataTypes.BITMAP()resolution and class mappingLogicalTypesTest: TestsBitmapTypeproperties, serialization string, and cast compatibilityProjectionCodeGeneratorTest: Verifies BITMAP field projection in generated codeTypeInferenceExtractorTest: Tests type inference for UDFs that accept/return BITMAP, including rejection of custom Bitmap implementationsCastRulesTest: Tests cast rulesDoes this pull request potentially affect one of the following parts:
@Public(Evolving): yes (RowData,ArrayData,DataTypes,BinaryWriter)getBitmap/writeBitmapcode paths, but only activated for BITMAP type columns)Documentation