Open
Conversation
Rafferty97
approved these changes
Mar 13, 2026
Contributor
|
Have you run the benchmarks for this? |
Author
sorry, new here, could you tell me how do I run them? 😅 |
Contributor
I think "cargo bench -p arrow-cast" should be sufficient. |
Contributor
|
run benchmark cast_kernels |
Contributor
|
Thank you @Rafferty97 and @aryan-212 -- I kicked off some benchmark runs to verify the performance implications of this change |
alamb
reviewed
Mar 17, 2026
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you for this PR @aryan-212 and (especially) for the help reviwing @Rafferty97
| impl Parser for Float16Type { | ||
| fn parse(string: &str) -> Option<f16> { | ||
| lexical_core::parse(string.as_bytes()) | ||
| lexical_core::parse(string.trim().as_bytes()) |
Contributor
There was a problem hiding this comment.
I wonder if there will be any performance implications 🤔
| #[test] | ||
| fn test_parse_trimmed_whitespace() { | ||
| // Float types | ||
| assert_eq!(Float16Type::parse(" 1.5 "), Some(f16::from_f32(1.5))); |
Contributor
There was a problem hiding this comment.
Cn you please add some tests with only leading whitespace and some tests with only trailing whitespace?
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Author
I don't think there is any significant perf regression, isn't it? |
Contributor
My reading of this result is that this code is 16% slower when parsing strings to floating point. I will rerun the benchmarks to see if this is reproducable |
Contributor
|
run benchmark cast_kernels |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Contributor
|
🤔 same slowdown:
|
Author
|
hmmm, should I make changes in some other layer where the performance hit wouldn't be this much? Any pointers if possible? |
Contributor
Well we can use the existing DataFusion CLI v52.3.0
> select ' 4.5'::float;
Optimizer rule 'simplify_expressions' failed
caused by
Arrow error: Cast error: Cannot cast string ' 4.5' to value of Float32 type
> select trim(' 4.5')::float;
+----------------------+
| btrim(Utf8(" 4.5")) |
+----------------------+
| 4.5 |
+----------------------+
1 row(s) fetched.
Elapsed 0.022 seconds.Interestingly DuckDB does automatically trim andrewlamb@Andrews-MacBook-Pro-3:~/Software/influxdb_pro/ent$ duckdb
DuckDB v1.5.0 (Variegata)
Enter ".help" for usage hints.
memory D select ' 4.5'::float;
┌────────────────────────┐
│ CAST(' 4.5' AS FLOAT) │
│ float │
├────────────────────────┤
│ 4.5 │
└────────────────────────┘As does postgres 🤔 andrewlamb@Andrews-MacBook-Pro-3:~/Software/influxdb_pro/ent$ psql -h localhost -U postgres
psql (14.22 (Homebrew), server 11.16 (Debian 11.16-1.pgdg90+1))
Type "help" for help.
postgres=# select ' 4.5'::float;
float8
--------
4.5
(1 row) |
Contributor
|
Is it possible to improve the performance of trim? For example, trim just based on ascii whitespace rather than having to do utf8 checking? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The
Parser::parseimplementations for numeric types did not trim whitespace before parsing. This caused values like " 42 " or " 1.5 " to fail parsing and returnNone, even though they represent valid numbers.What changes are included in this PR?
.trim()calls before parsing in FloatType Parser implementations.string.trim()at the top of the parser_primitive! macro, which covers all integer and duration types.Are these changes tested?
Yes. Added test_parse_trimmed_whitespace covering:
Datafusion changes
For the following SQL :-
in datafusion we used to get
now after these changes, we get
this behaviour is now aligned with Databricks
Are there any user-facing changes?
Yes. Numeric parsing now accepts strings with leading/trailing whitespace. This is a relaxation of the previous behaviour (previously
None, nowSome(value)), so it is not a breaking change.