Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions arrow-cast/src/cast/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,7 @@ pub fn can_cast_types(from_type: &DataType, to_type: &DataType) -> bool {
| Time64(Microsecond | Nanosecond)
| Timestamp(Second | Millisecond | Microsecond | Nanosecond, _)
| Interval(_)
| Duration(_)
| BinaryView,
) => true,
(Utf8 | LargeUtf8, Utf8View) => true,
Expand Down Expand Up @@ -1316,6 +1317,18 @@ pub fn cast_with_options(
Interval(IntervalUnit::MonthDayNano) => {
cast_string_to_month_day_nano_interval::<i32>(array, cast_options)
}
Duration(TimeUnit::Second) => {
cast_string_to_duration::<DurationSecondType, i32>(array, cast_options)
}
Duration(TimeUnit::Millisecond) => {
cast_string_to_duration::<DurationMillisecondType, i32>(array, cast_options)
}
Duration(TimeUnit::Microsecond) => {
cast_string_to_duration::<DurationMicrosecondType, i32>(array, cast_options)
}
Duration(TimeUnit::Nanosecond) => {
cast_string_to_duration::<DurationNanosecondType, i32>(array, cast_options)
}
_ => Err(ArrowError::CastError(format!(
"Casting from {from_type} to {to_type} not supported",
))),
Expand Down Expand Up @@ -1374,6 +1387,18 @@ pub fn cast_with_options(
Interval(IntervalUnit::MonthDayNano) => {
cast_view_to_month_day_nano_interval(array, cast_options)
}
Duration(TimeUnit::Second) => {
cast_view_to_duration::<DurationSecondType>(array, cast_options)
}
Duration(TimeUnit::Millisecond) => {
cast_view_to_duration::<DurationMillisecondType>(array, cast_options)
}
Duration(TimeUnit::Microsecond) => {
cast_view_to_duration::<DurationMicrosecondType>(array, cast_options)
}
Duration(TimeUnit::Nanosecond) => {
cast_view_to_duration::<DurationNanosecondType>(array, cast_options)
}
_ => Err(ArrowError::CastError(format!(
"Casting from {from_type} to {to_type} not supported",
))),
Expand Down Expand Up @@ -1449,6 +1474,18 @@ pub fn cast_with_options(
Interval(IntervalUnit::MonthDayNano) => {
cast_string_to_month_day_nano_interval::<i64>(array, cast_options)
}
Duration(TimeUnit::Second) => {
cast_string_to_duration::<DurationSecondType, i64>(array, cast_options)
}
Duration(TimeUnit::Millisecond) => {
cast_string_to_duration::<DurationMillisecondType, i64>(array, cast_options)
}
Duration(TimeUnit::Microsecond) => {
cast_string_to_duration::<DurationMicrosecondType, i64>(array, cast_options)
}
Duration(TimeUnit::Nanosecond) => {
cast_string_to_duration::<DurationNanosecondType, i64>(array, cast_options)
}
_ => Err(ArrowError::CastError(format!(
"Casting from {from_type} to {to_type} not supported",
))),
Expand Down Expand Up @@ -5526,6 +5563,88 @@ mod tests {
);
}

#[test]
fn test_cast_string_to_duration() {
let array = StringArray::from(vec![Some("1"), Some("-2"), None, Some("invalid")]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we support only casting from numeric values? no units such as 1 hour 10 minutes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently numeric only: the cast reuses the existing Duration*Type parser impls (the same integer parsing the JSON reader uses).

I can extend it to unit strings like 1 hour 10 minutes by going through the existing string→interval parser and the Interval(MonthDayNano) → Duration cast, which keeps the current semantics (months/days are not convertible — cast_month_day_nano_to_duration nulls/errors them).

I could add it in this PR or keep this one numeric-only — your call.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be more useful if we could parse units

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ffa5e92 — non-integer strings now fall back to the interval parser, so "1 hour 10 minutes" casts to any Duration unit. Integer strings behave as before.

Months/days stay non-convertible to match the interval → duration cast (null when safe, error otherwise). I can change that if you'd rather treat "1 day" as 24h.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a good rule of thumb is to be able to roundtrip through the display representation:

let array = DurationSecondArray::from(vec![
1,
-1,
1000,
-1000,
45 * 60 * 60 * 24 + 14 * 60 * 60 + 2 * 60 + 34,
-45 * 60 * 60 * 24 - 14 * 60 * 60 - 2 * 60 - 34,
]);
let iso = format_array(&array, &iso_fmt);
let pretty = format_array(&array, &pretty_fmt);
assert_eq!(iso[0], "PT1S");
assert_eq!(pretty[0], "0 days 0 hours 0 mins 1 secs");
assert_eq!(iso[1], "-PT1S");
assert_eq!(pretty[1], "0 days 0 hours 0 mins -1 secs");
assert_eq!(iso[2], "PT1000S");
assert_eq!(pretty[2], "0 days 0 hours 16 mins 40 secs");
assert_eq!(iso[3], "-PT1000S");
assert_eq!(pretty[3], "0 days 0 hours -16 mins -40 secs");
assert_eq!(iso[4], "PT3938554S");
assert_eq!(pretty[4], "45 days 14 hours 2 mins 34 secs");
assert_eq!(iso[5], "-PT3938554S");
assert_eq!(pretty[5], "-45 days -14 hours -2 mins -34 secs");

can see how we display days here, so it would make sense to support parsing days

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d047bf0 — days now convert at 24h/day (weeks fold into days in the parser), so pretty display strings roundtrip: "45 days 14 hours 2 mins 34 secs" → 3938554s for Duration(s). Months stay non-convertible since the duration display never emits them.

let to_type = DataType::Duration(TimeUnit::Second);
assert!(can_cast_types(array.data_type(), &to_type));

let casted = cast(&array, &to_type).unwrap();
assert_eq!(
casted.as_primitive::<DurationSecondType>(),
&DurationSecondArray::from(vec![Some(1), Some(-2), None, None])
);

let options = CastOptions {
safe: false,
format_options: FormatOptions::default(),
};
let err = cast_with_options(&array, &to_type, &options).unwrap_err();
assert_eq!(
err.to_string(),
"Cast error: Cannot cast string 'invalid' to value of Duration(s) type"
);
}

#[test]
fn test_cast_string_with_units_to_duration() {
let array = StringArray::from(vec![
Some("1 hour 10 minutes"),
Some("90 seconds"),
Some("0.5 hours"),
Some("-1 hour -30 minutes"),
None,
Some("2 days"),
]);
let to_type = DataType::Duration(TimeUnit::Second);

let casted = cast(&array, &to_type).unwrap();
assert_eq!(
casted.as_primitive::<DurationSecondType>(),
&DurationSecondArray::from(vec![
Some(4200),
Some(90),
Some(1800),
Some(-5400),
None,
None
])
);

let options = CastOptions {
safe: false,
format_options: FormatOptions::default(),
};
let err = cast_with_options(&array, &to_type, &options).unwrap_err();
assert_eq!(
err.to_string(),
"Compute error: Cannot convert interval containing non-zero months or days to duration"
);
}

#[test]
fn test_cast_large_string_and_view_to_duration() {
let to_type = DataType::Duration(TimeUnit::Nanosecond);
for array in [
Arc::new(LargeStringArray::from(vec![
Some("10"),
Some("1 minute"),
None,
])) as ArrayRef,
Arc::new(StringViewArray::from(vec![
Some("10"),
Some("1 minute"),
None,
])) as ArrayRef,
] {
let casted = cast(&array, &to_type).unwrap();
assert_eq!(
casted.as_primitive::<DurationNanosecondType>(),
&DurationNanosecondArray::from(vec![Some(10), Some(60_000_000_000), None])
);
}
}

#[test]
fn test_cast_binary_to_fixed_size_binary() {
let bytes_1 = b"Hiiii".as_slice();
Expand Down
102 changes: 102 additions & 0 deletions arrow-cast/src/cast/string.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
// under the License.

use crate::cast::*;
use crate::parse::{
IntervalParseConfig, IntervalUnit as IntervalParseUnit, parse_interval_month_day_nano_config,
};
use arrow_buffer::NullBuffer;

pub(crate) fn cast_bool_to_string<A>(array: &dyn Array) -> Result<ArrayRef, ArrowError>
Expand Down Expand Up @@ -132,6 +135,105 @@ fn parse_string_iter<
Ok(Arc::new(array) as ArrayRef)
}

/// Parse a string to a duration, accepting integer strings (interpreted
/// in the target unit) and interval strings with units, such as
/// `"1 hour 10 minutes"`.
fn parse_duration<D>(string: &str) -> Result<i64, ArrowError>
where
D: ArrowTemporalType<Native = i64> + Parser,
{
if let Some(duration) = D::parse(string) {
return Ok(duration);
}

let (unit, scale) = match D::DATA_TYPE {
DataType::Duration(TimeUnit::Second) => (IntervalParseUnit::Second, 1_000_000_000),
DataType::Duration(TimeUnit::Millisecond) => (IntervalParseUnit::Millisecond, 1_000_000),
DataType::Duration(TimeUnit::Microsecond) => (IntervalParseUnit::Microsecond, 1_000),
DataType::Duration(TimeUnit::Nanosecond) => (IntervalParseUnit::Nanosecond, 1),
_ => unreachable!(),
};

let interval = parse_interval_month_day_nano_config(string, IntervalParseConfig::new(unit))
.map_err(|_| {
ArrowError::CastError(format!(
"Cannot cast string '{string}' to value of {} type",
D::DATA_TYPE
))
})?;

if interval.months != 0 || interval.days != 0 {
return Err(ArrowError::ComputeError(
"Cannot convert interval containing non-zero months or days to duration".to_string(),
));
}

Ok(interval.nanoseconds / scale)
}

/// Casts generic string arrays to a Duration type
pub(crate) fn cast_string_to_duration<D, O>(
array: &dyn Array,
cast_options: &CastOptions,
) -> Result<ArrayRef, ArrowError>
where
D: ArrowTemporalType<Native = i64> + Parser,
O: OffsetSizeTrait,
{
let string_array = array.as_string::<O>();
parse_duration_iter::<D, _, _>(string_array.iter(), cast_options, || {
string_array.nulls().cloned()
})
}

/// Casts string view arrays to a Duration type
pub(crate) fn cast_view_to_duration<D>(
array: &dyn Array,
cast_options: &CastOptions,
) -> Result<ArrayRef, ArrowError>
where
D: ArrowTemporalType<Native = i64> + Parser,
{
let string_view_array = array.as_string_view();
parse_duration_iter::<D, _, _>(string_view_array.iter(), cast_options, || {
string_view_array.nulls().cloned()
})
}

fn parse_duration_iter<'a, D, I, F>(
iter: I,
cast_options: &CastOptions,
nulls: F,
) -> Result<ArrayRef, ArrowError>
where
D: ArrowTemporalType<Native = i64> + Parser,
I: Iterator<Item = Option<&'a str>>,
F: FnOnce() -> Option<NullBuffer>,
{
let array = if cast_options.safe {
let iter = iter.map(|x| {
let v = x?;
parse_duration::<D>(v).ok()
});

// Benefit:
// 20% performance improvement
// Soundness:
// The iterator is trustedLen because it comes from an `StringArray`.
unsafe { PrimitiveArray::<D>::from_trusted_len_iter(iter) }
} else {
let v = iter
.map(|x| match x {
Some(v) => parse_duration::<D>(v),
None => Ok(D::Native::default()),
})
.collect::<Result<Vec<_>, ArrowError>>()?;
PrimitiveArray::try_new(v.into(), nulls())?
};

Ok(Arc::new(array) as ArrayRef)
}

/// Casts generic string arrays to an ArrowTimestampType (TimeStampNanosecondArray, etc.)
pub(crate) fn cast_string_to_timestamp<O: OffsetSizeTrait, T: ArrowTimestampType>(
array: &dyn Array,
Expand Down
Loading