Skip to content

docs: document Unicode behavior - #80

Open
whitescreendev wants to merge 1 commit into
rapidfuzz:mainfrom
whitescreendev:docs/unicode-behavior
Open

whitescreendev wants to merge 1 commit into
rapidfuzz:mainfrom
whitescreendev:docs/unicode-behavior

Conversation

@whitescreendev

Copy link
Copy Markdown

This documentation update clarifies how string-based similarity functions handle Unicode text.

It documents that:

  • string-based functions operate on Rust char values rather than UTF-8 bytes;
  • canonically equivalent strings can have different scalar-value sequences;
  • Unicode normalization may be needed before comparison;
  • user-perceived characters can contain multiple Unicode scalar values;
  • grapheme-cluster segmentation may be appropriate when one visible character should count as one edit.

The implementation and API are unchanged. This is a documentation-only update.

An additional technical reference is included for Unicode normalization, grapheme segmentation, and text transformations in edit-distance processing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant