Skip to content

Fix String.reverse/1 grapheme ordering around invalid utf8 bytes #15645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
josevalim merged 1 commit into from
Jul 21, 2026
+2 −1
Merged

Fix String.reverse/1 grapheme ordering around invalid utf8 bytes #15645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion lib/elixir/lib/string.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1784,7 +1784,7 @@ defmodule String do
do: :unicode.characters_to_binary(acc)

defp do_reverse({:error, <<byte, rest::bits>>}, acc),
do: :unicode.characters_to_binary(acc) <> <<byte>> <> do_reverse(:unicode_util.gc(rest), [])
do: do_reverse(:unicode_util.gc(rest), []) <> <<byte>> <> :unicode.characters_to_binary(acc)

@josevalim josevalim Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move them to the accumulator instead?

Suggested change
do: do_reverse(:unicode_util.gc(rest), []) <> <<byte>> <> :unicode.characters_to_binary(acc)
do: do_reverse(:unicode_util.gc(rest), [<<byte>>, acc])

@AlexGx AlexGx Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[<<byte>>, acc]variant may produce a nested lists. And also this affects hot path (strings without invalid utf8), because needs rewrite to something like:

defp do_reverse([grapheme | rest], acc),
    # extra calls grapheme_to_binary, was: do_reverse(:unicode_util.gc(rest), [grapheme | acc])
    do: do_reverse(:unicode_util.gc(rest), [grapheme_to_binary(grapheme) | acc])

  defp do_reverse([], acc),
    # do: :unicode.characters_to_binary(acc)
    do: IO.iodata_to_binary(acc)

If this is what you mean, I will do benchmark...

@josevalim josevalim Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested lists are supported in Unicode characters to binary:

iex>  :unicode.characters_to_binary([[?j], ?o, <<?s>>, [[?é]]])
"josé"

But we should probably do this instead anyway:

Suggested change
do: do_reverse(:unicode_util.gc(rest), []) <> <<byte>> <> :unicode.characters_to_binary(acc)
do: do_reverse(:unicode_util.gc(rest), [<<byte>> | acc])

@AlexGx AlexGx Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im stuck, please help!

on OTP 29
iex> :unicode.characters_to_binary(["dc", [<<255>>, "b", "a"]])
{:error, "dc", [[<<255>>, "b", "a"]]}

@josevalim josevalim Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, of course, the operation will fail due to the invalid byte.


@doc """
Returns a string `subject` repeated `n` times.
Expand Down
1 change: 1 addition & 0 deletions lib/elixir/test/elixir/string_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,7 @@ defmodule StringTest do
assert String.reverse("Hello World") == "dlroW olleH"
assert String.reverse("Hello ∂og") == "go∂ olleH"
assert String.reverse("Ā̀stute") == "etutsĀ̀"
assert String.reverse("ab" <> <<254, 255>> <> "cd") == "dc" <> <<255, 254>> <> "ba"
assert String.reverse(String.reverse("Hello World")) == "Hello World"
assert String.reverse(String.reverse("Hello \r\n World")) == "Hello \r\n World"
end
Expand Down
Loading