Skip to content

feat: support casting strings to durations - #11572

Open
miguelcorderopamphile wants to merge 3 commits into
apache:mainfrom
miguelcorderopamphile:feat/string-to-duration-cast
Open

miguelcorderopamphile wants to merge 3 commits into
apache:mainfrom
miguelcorderopamphile:feat/string-to-duration-cast

Conversation

@miguelcorderopamphile

Copy link
Copy Markdown
Contributor

Fixes #4555.

The cast kernels supported strings to intervals, but not to durations. This adds casts from Utf8, LargeUtf8 and Utf8View to Duration in all four units, reusing the existing Parser implementations for the Duration*Types (the same integer parsing the JSON reader uses, #6683).

  • can_cast_types accepts the string → duration combination
  • dispatch arms added for parse_string (Utf8/LargeUtf8) and parse_string_view (Utf8View)
  • tests: safe cast with nulls and invalid values, unsafe error message, large/view variants

cargo test -p arrow-cast --lib: 368 passed.

Add casts from Utf8/LargeUtf8/Utf8View to Duration for all four units,
reusing the existing Parser implementations for the Duration*Types
(same integer parsing as the JSON reader).
@github-actions github-actions Bot added arrow-kernels Crates that transform arrow data (e.g. compute, cast) enhancement New feature/addition labels Oct 15, 2026

#[test]
fn test_cast_string_to_duration() {
let array = StringArray::from(vec![Some("1"), Some("-2"), None, Some("invalid")]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we support only casting from numeric values? no units such as 1 hour 10 minutes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently numeric only: the cast reuses the existing Duration*Type parser impls (the same integer parsing the JSON reader uses).

I can extend it to unit strings like 1 hour 10 minutes by going through the existing string→interval parser and the Interval(MonthDayNano) → Duration cast, which keeps the current semantics (months/days are not convertible — cast_month_day_nano_to_duration nulls/errors them).

I could add it in this PR or keep this one numeric-only — your call.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be more useful if we could parse units

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ffa5e92 — non-integer strings now fall back to the interval parser, so "1 hour 10 minutes" casts to any Duration unit. Integer strings behave as before.

Months/days stay non-convertible to match the interval → duration cast (null when safe, error otherwise). I can change that if you'd rather treat "1 day" as 24h.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a good rule of thumb is to be able to roundtrip through the display representation:

let array = DurationSecondArray::from(vec![
1,
-1,
1000,
-1000,
45 * 60 * 60 * 24 + 14 * 60 * 60 + 2 * 60 + 34,
-45 * 60 * 60 * 24 - 14 * 60 * 60 - 2 * 60 - 34,
]);
let iso = format_array(&array, &iso_fmt);
let pretty = format_array(&array, &pretty_fmt);
assert_eq!(iso[0], "PT1S");
assert_eq!(pretty[0], "0 days 0 hours 0 mins 1 secs");
assert_eq!(iso[1], "-PT1S");
assert_eq!(pretty[1], "0 days 0 hours 0 mins -1 secs");
assert_eq!(iso[2], "PT1000S");
assert_eq!(pretty[2], "0 days 0 hours 16 mins 40 secs");
assert_eq!(iso[3], "-PT1000S");
assert_eq!(pretty[3], "0 days 0 hours -16 mins -40 secs");
assert_eq!(iso[4], "PT3938554S");
assert_eq!(pretty[4], "45 days 14 hours 2 mins 34 secs");
assert_eq!(iso[5], "-PT3938554S");
assert_eq!(pretty[5], "-45 days -14 hours -2 mins -34 secs");

can see how we display days here, so it would make sense to support parsing days

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d047bf0 — days now convert at 24h/day (weeks fold into days in the parser), so pretty display strings roundtrip: "45 days 14 hours 2 mins 34 secs" → 3938554s for Duration(s). Months stay non-convertible since the duration display never emits them.

Reuse the interval parser for non-integer strings (e.g. '1 hour 10 minutes').
Intervals with non-zero months or days remain non-convertible (null/error),
consistent with the interval to duration cast.
Convert days at 24h so pretty display strings roundtrip, e.g.
'45 days 14 hours 2 mins 34 secs'. Weeks fold into days in the
interval parser. Months stay non-convertible (never displayed).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-kernels Crates that transform arrow data (e.g. compute, cast) enhancement New feature/addition

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parse Durations From String

2 participants