From 9053733b6ed1d67849f53781496861b8f77e920a Mon Sep 17 00:00:00 2001 From: Percy Grunwald Date: Wed, 20 Feb 2019 15:49:58 +0800 Subject: [PATCH 1/6] Adds latest version of Elixir Parallel Letter Frequency Unicode post --- ...uency-elixir-part-1-unicode-match-regex.md | 160 ++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md diff --git a/posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md b/posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md new file mode 100644 index 0000000..b2106fd --- /dev/null +++ b/posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md @@ -0,0 +1,160 @@ +Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these synthetic problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real world problems more efficiently or in a more expressive way. + +[Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that unpacks a surprising number of interesting lessons. A central challenge in solving the exercise is handling letters from multiple languages, as one of the test cases is in German and contains characters outside the English alphabet. If you spend most of your time developing applications for English speakers, this may be the first time you've had to deal with a requirement like this. The learning from this exercise has clear benefits for anyone writing a multilingual/non-English application, but could also help in many other areas, such as more robust username and password validations. + +To solve the exercise successfully, you need to implement a `Frequency.frequency/2` function that determines letter frequency in a list of strings that could be in any language: + +```elixir +iex> Frequency.frequency(["Freude", "schöner", "Götterfunken"], workers) +%{ + "c" => 1, + "d" => 1, + "e" => 5, + ... + "ö" => 2 +} +``` + +Let's start with the fundamental problem this function needs to solve and work our way up to a full implementation. + +## Determine whether a character is a letter in Elixir + +How would you use Elixir to determine whether or not `"a"` is a letter? + +I think most people would apply a regular expression like `/[a-z]/`: + +```elixir +iex> String.match?("a", ~r/^[a-z]$/) +true +``` + +What about `"A"`? + +Adding the `i` [(caseless) modifier](https://hexdocs.pm/elixir/Regex.html#module-modifiers) would probably be the easiest way: + +```elixir +iex> String.match?("A", ~r/^[a-z]$/i) +true +``` + +Ok, now what about `"ö"`? + +When I first approached this problem, I wasn't sure of the best way --`/[a-z]/i` is definitely not going to work: + +```elixir +iex> String.match?("ö", ~r/^[a-z]$/i) +false +``` + +Determining whether or not `"ö"` is a letter is a core part of solving this Exercism problem, as one of the texts in the tests is in German: + +```elixir +# parallel_letter_frequency_test.exs +... +# Poem by Friedrich Schiller. The corresponding music is the European Anthem. +@ode_an_die_freude """ +Freude schöner Götterfunken +... +""" +``` + +Maybe you could use regular expression to check if a character **isn't** a special character, but it's likely to be long, inelegant and fragile. How confident can you be that you've covered every possible special character that might be passed as input to your function? I believe there's a better approach. + +## Unicode regular expressions in Elixir + +A better approach to this problem is using the [`u` modifier in Elixir's `Regex` module](https://hexdocs.pm/elixir/Regex.html): + +> unicode (`u`) - enables Unicode specific patterns like `\p` and change modifiers like `\w`, `\W`, `\s` and friends to also match on Unicode. + +It turns out that the `u` modifier -- and [specifically the `\p` pattern](https://www.regular-expressions.info/unicode.html) -- is a really elegant solution. The `\p` pattern lets you match a grapheme (another name for a single Unicode character) in any of the [Unicode character categories](https://en.wikipedia.org/wiki/Unicode_character_property#General_Category). This not only includes specific categories like `Ll` ([Letter, lowercase](https://www.compart.com/en/unicode/category/Ll)) and `Sc` ([Symbol, currency](https://www.compart.com/en/unicode/category/Sc)), but also the parent categories like `L` (Letter) and `S` (Symbol). + +You can match _any_ letter of _any_ case in _any_ [human language covered by Unicode](https://www.unicode.org/faq/basic_q.html) with the pattern `\p{L}`. This allows for some [pretty powerful matching](https://www.toptechskills.com/elixir-phoenix-tutorials-courses/how-to-match-any-unicode-letter-with-regex-elixir/#more-cool-stuff-you-can-match-with-unicode). + +Basic Latin characters from English work as usual: + +```elixir +iex> String.match?("a", ~r/^\p{L}$/u) +true +iex> String.match?("A", ~r/^\p{L}$/u) +true +``` + +Latin character variants with [umlauts](https://en.wikipedia.org/wiki/Umlaut_(linguistics)) and [acute accents](https://en.wikipedia.org/wiki/Acute_accent) are no problem either: + +```elixir +iex> String.match?("ö", ~r/^\p{L}$/u) +true +iex> String.match?("Á", ~r/^\p{L}$/u) +true +``` + +Let's make sure it's not just returning a match for any character. How about some characters that look like letters but aren't: + +```elixir +iex> String.match?("$", ~r/^\p{L}$/u) +false +iex> String.match?("@", ~r/^\p{L}$/u) +false +``` + +Very nice, but remember how I said _any_ language? No sweat: + +```elixir +# Chinese character for "you" +iex> String.match?("你", ~r/^\p{L}$/u) +true + +# Cyrillic capital letter "zhe" +iex> String.match?("Ж", ~r/^\p{L}$/u) +true +``` + +## Applying Unicode matching to the problem at hand + +Now that we have a tool that can help us determine whether or not a grapheme is a letter, we can apply it to solve the problem. An initial implementation of the `Frequency.frequency/2` function might look like this: + +```elixir +def frequency(texts, _workers) do + texts + |> get_all_graphemes() + |> count_letters() +end + +defp get_all_graphemes(texts) do + texts + |> Enum.join() + |> String.graphemes() +end +``` + +All `count_letters/1` would need to do is apply the `String.match?(grapheme, ~r/^\p{L}$/u)` pattern we identified above to increment the count of each letter in the list of `graphemes`. Here's an example implementation taken from [my solution to this Exercism problem]((https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency/solutions/cc80004beded4749bce81b5dc0820952).): + +```elixir +defp count_letters(graphemes) do + Enum.reduce(graphemes, %{}, fn grapheme, acc -> + if String.match?(grapheme, ~r/^\p{L}$/u) do + downcased_letter = String.downcase(grapheme) + Map.update(acc, downcased_letter, 1, fn count -> count + 1 end) + else + acc + end + end) +end +``` + +This function accepts a list of graphemes, e.g. `["a", "A", "ö", "$"]`, and returns a map that counts only the letters while ignoring the case -- `%{"a" => 2, "ö" => 1}`. Considering that this function can handle input from any language, I would say it's a pretty powerful 9 lines of code. + +## Conclusion + +It turns out that matching non-English letters becomes pretty simple when you know about Unicode matching, and luckily for us it's a core feature in Elixir's `Regex` module. Prior to solving this Exercism problem I barely knew about this feature, but I would now consider it an indispensable part of my Elixir toolbox. + +You could use this new tool in a number of ways, and a few that spring to my mind are more robust validation of passwords and usernames, or even for determining whether an input string is a valid currency string without needing to manually list [all possible currency symbols](https://www.compart.com/en/unicode/category/Sc): + +```elixir +iex> currency_string_regex = ~r/\p{Sc}\d+\.\d{2}/u +~r/\p{Sc}\d+\.\d{2}/u + +iex> ["$1.00", "£1.00", "¥1.00", "€1.00", "&1.00"] \ +...> |> Enum.filter(&String.match?(&1, currency_string_regex)) +["$1.00", "£1.00", "¥1.00", "€1.00"] +``` From aa3043b83b267120e47ed453464c7041a14c3082 Mon Sep 17 00:00:00 2001 From: Percy Grunwald Date: Thu, 21 Feb 2019 13:17:44 +0800 Subject: [PATCH 2/6] Renames Elixir parallel letter frequency unicode article file name and adds entry for it to blog.json --- blog.json | 13 +++++++++++++ ...lel-letter-frequency-unicode-matching-elixir.md} | 0 2 files changed, 13 insertions(+) rename posts/{solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md => parallel-letter-frequency-unicode-matching-elixir.md} (100%) diff --git a/blog.json b/blog.json index ab118e6..3074e31 100644 --- a/blog.json +++ b/blog.json @@ -137,5 +137,18 @@ "author_handle": "rpalo", "marketing_copy": "Ryan Palo explores the concepts of 'intentional coding' and 'design intent' in this excellent walkthrough of the Grains exercise in Bash. Thinking of ways to solve a problem is easy- designing one that communicates your thought process and goals to others is the hard part.", "image_url": "https://assets.exercism.io/blog/social/coding-intentionally-in-bash-grains.png" + }, + { + "uuid": "930d520f-1f94-4cf8-9aaf-6f5201470f80", + "slug": "parallel-letter-frequency-unicode-matching-elixir", + "category": "programming skills", + "language": "elixir", + "published_at": "2019-02-29 01:00", + "content_repository": "blog", + "content_filepath": "posts/parallel-letter-frequency-unicode-matching-elixir.md", + "title": "What Parallel Letter Frequency can teach you about Unicode matching in Elixir", + "author_handle": "percygrunwald", + "marketing_copy": "", + "image_url": "" } ] diff --git a/posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md b/posts/parallel-letter-frequency-unicode-matching-elixir.md similarity index 100% rename from posts/solving-parallel-letter-frequency-elixir-part-1-unicode-match-regex.md rename to posts/parallel-letter-frequency-unicode-matching-elixir.md From 5ac0cbd1b0ed659067c207e5b815e46c10d36bcc Mon Sep 17 00:00:00 2001 From: Katrina Owen Date: Thu, 21 Feb 2019 08:53:43 -0700 Subject: [PATCH 3/6] Make minor tweaks for grammar and flow --- ...allel-letter-frequency-unicode-matching-elixir.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/posts/parallel-letter-frequency-unicode-matching-elixir.md b/posts/parallel-letter-frequency-unicode-matching-elixir.md index b2106fd..4d05b59 100644 --- a/posts/parallel-letter-frequency-unicode-matching-elixir.md +++ b/posts/parallel-letter-frequency-unicode-matching-elixir.md @@ -1,8 +1,8 @@ -Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these synthetic problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real world problems more efficiently or in a more expressive way. +Exercises on Exercism are small, synthetic, and often seemingly trivial. It’s easy to imagine that experienced practitioners would have nothing to learn from them. However, solving these artificial problems can push you to learn and apply parts of your language that you may not have explored. This new learning can lead you to solve real-world problems more efficiently or more expressively. [Parallel Letter Frequency](https://exercism.io/tracks/elixir/exercises/parallel-letter-frequency) is a medium difficulty exercise on [Exercism's Elixir Track](https://exercism.io/tracks/elixir) that unpacks a surprising number of interesting lessons. A central challenge in solving the exercise is handling letters from multiple languages, as one of the test cases is in German and contains characters outside the English alphabet. If you spend most of your time developing applications for English speakers, this may be the first time you've had to deal with a requirement like this. The learning from this exercise has clear benefits for anyone writing a multilingual/non-English application, but could also help in many other areas, such as more robust username and password validations. -To solve the exercise successfully, you need to implement a `Frequency.frequency/2` function that determines letter frequency in a list of strings that could be in any language: +To solve the exercise successfully, you need to implement a function, `Frequency.frequency/2` , that determines letter frequency in a list of strings that could be in any language: ```elixir iex> Frequency.frequency(["Freude", "schöner", "Götterfunken"], workers) @@ -58,7 +58,7 @@ Freude schöner Götterfunken """ ``` -Maybe you could use regular expression to check if a character **isn't** a special character, but it's likely to be long, inelegant and fragile. How confident can you be that you've covered every possible special character that might be passed as input to your function? I believe there's a better approach. +Maybe you could use a regular expression to check if a character **isn't** a special character, but it's likely to be long, inelegant and fragile. How confident can you be that you've covered every possible special character that might be passed as input to your function? I believe there's a better approach. ## Unicode regular expressions in Elixir @@ -66,7 +66,7 @@ A better approach to this problem is using the [`u` modifier in Elixir's `Regex` > unicode (`u`) - enables Unicode specific patterns like `\p` and change modifiers like `\w`, `\W`, `\s` and friends to also match on Unicode. -It turns out that the `u` modifier -- and [specifically the `\p` pattern](https://www.regular-expressions.info/unicode.html) -- is a really elegant solution. The `\p` pattern lets you match a grapheme (another name for a single Unicode character) in any of the [Unicode character categories](https://en.wikipedia.org/wiki/Unicode_character_property#General_Category). This not only includes specific categories like `Ll` ([Letter, lowercase](https://www.compart.com/en/unicode/category/Ll)) and `Sc` ([Symbol, currency](https://www.compart.com/en/unicode/category/Sc)), but also the parent categories like `L` (Letter) and `S` (Symbol). +It turns out that the `u` modifier—and [specifically the `\p` pattern](https://www.regular-expressions.info/unicode.html)—is a really elegant solution. The `\p` pattern lets you match a grapheme (another name for a single Unicode character) in any of the [Unicode character categories](https://en.wikipedia.org/wiki/Unicode_character_property#General_Category). This not only includes specific categories like `Ll` ([Letter, lowercase](https://www.compart.com/en/unicode/category/Ll)) and `Sc` ([Symbol, currency](https://www.compart.com/en/unicode/category/Sc)), but also the parent categories like `L` (Letter) and `S` (Symbol). You can match _any_ letter of _any_ case in _any_ [human language covered by Unicode](https://www.unicode.org/faq/basic_q.html) with the pattern `\p{L}`. This allows for some [pretty powerful matching](https://www.toptechskills.com/elixir-phoenix-tutorials-courses/how-to-match-any-unicode-letter-with-regex-elixir/#more-cool-stuff-you-can-match-with-unicode). @@ -146,9 +146,9 @@ This function accepts a list of graphemes, e.g. `["a", "A", "ö", "$"]`, and ret ## Conclusion -It turns out that matching non-English letters becomes pretty simple when you know about Unicode matching, and luckily for us it's a core feature in Elixir's `Regex` module. Prior to solving this Exercism problem I barely knew about this feature, but I would now consider it an indispensable part of my Elixir toolbox. +It turns out that matching non-English letters becomes pretty simple when you know about Unicode matching, and luckily for us, it's a core feature in Elixir's `Regex` module. Before solving this Exercism problem I barely knew about this feature, but I would now consider it an indispensable part of my Elixir toolbox. -You could use this new tool in a number of ways, and a few that spring to my mind are more robust validation of passwords and usernames, or even for determining whether an input string is a valid currency string without needing to manually list [all possible currency symbols](https://www.compart.com/en/unicode/category/Sc): +You could use this new tool in many ways, and a few that spring to my mind are more robust validation of passwords and usernames, or even for determining whether an input string is a valid currency string without needing to manually list [all possible currency symbols](https://www.compart.com/en/unicode/category/Sc): ```elixir iex> currency_string_regex = ~r/\p{Sc}\d+\.\d{2}/u From 1d058e8ee969abd5f7712693a1badac643dd86ef Mon Sep 17 00:00:00 2001 From: Katrina Owen Date: Thu, 21 Feb 2019 08:56:02 -0700 Subject: [PATCH 4/6] Suggest marketing copy for Unicode article --- blog.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog.json b/blog.json index 3074e31..f7c7ebb 100644 --- a/blog.json +++ b/blog.json @@ -148,7 +148,7 @@ "content_filepath": "posts/parallel-letter-frequency-unicode-matching-elixir.md", "title": "What Parallel Letter Frequency can teach you about Unicode matching in Elixir", "author_handle": "percygrunwald", - "marketing_copy": "", + "marketing_copy": "Percy Grunwald shows how a trivial exercise can teach useful real-world lessons about handling Unicode in Elixir.", "image_url": "" } ] From 08d0399ec34f503f2f60892c652571ccf8d33c32 Mon Sep 17 00:00:00 2001 From: Jeremy Walker Date: Sun, 3 Mar 2019 20:03:36 +0000 Subject: [PATCH 5/6] Prepare unicode matching post for launch --- blog.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/blog.json b/blog.json index f7c7ebb..83c7bc7 100644 --- a/blog.json +++ b/blog.json @@ -128,7 +128,7 @@ { "uuid": "e6f3da5f-98b9-4cf7-be0b-447890aff367", "slug": "coding-intentionally-in-bash-grains", - "category": "programming skills", + "category": "programming-skills", "language": "bash", "published_at": "2019-02-14 01:00", "content_repository": "blog", @@ -141,14 +141,14 @@ { "uuid": "930d520f-1f94-4cf8-9aaf-6f5201470f80", "slug": "parallel-letter-frequency-unicode-matching-elixir", - "category": "programming skills", + "category": "programming-skills", "language": "elixir", - "published_at": "2019-02-29 01:00", + "published_at": "2019-03-03 01:00", "content_repository": "blog", "content_filepath": "posts/parallel-letter-frequency-unicode-matching-elixir.md", - "title": "What Parallel Letter Frequency can teach you about Unicode matching in Elixir", + "title": "Unicode matching in Elixir", "author_handle": "percygrunwald", - "marketing_copy": "Percy Grunwald shows how a trivial exercise can teach useful real-world lessons about handling Unicode in Elixir.", - "image_url": "" + "marketing_copy": "Percy Grunwald explores unicode matching in Elixir and shows how a trivial exercise can teach useful real-world lessons about handling Unicode in Elixir.", + "image_url": "https://assets.exercism.io/tracks/elixir-bordered-turquoise.png" } ] From f4b58adbe218a6bb020651fe152fbb6f00b45e6b Mon Sep 17 00:00:00 2001 From: Jeremy Walker Date: Sun, 3 Mar 2019 20:06:08 +0000 Subject: [PATCH 6/6] Rename for SEO --- blog.json | 4 ++-- ...icode-matching-elixir.md => unicode-matching-in-elixir.md} | 0 2 files changed, 2 insertions(+), 2 deletions(-) rename posts/{parallel-letter-frequency-unicode-matching-elixir.md => unicode-matching-in-elixir.md} (100%) diff --git a/blog.json b/blog.json index 83c7bc7..8f806a5 100644 --- a/blog.json +++ b/blog.json @@ -140,12 +140,12 @@ }, { "uuid": "930d520f-1f94-4cf8-9aaf-6f5201470f80", - "slug": "parallel-letter-frequency-unicode-matching-elixir", + "slug": "unicode-matching-in-elixir", "category": "programming-skills", "language": "elixir", "published_at": "2019-03-03 01:00", "content_repository": "blog", - "content_filepath": "posts/parallel-letter-frequency-unicode-matching-elixir.md", + "content_filepath": "posts/unicode-matching-in-elixir.md", "title": "Unicode matching in Elixir", "author_handle": "percygrunwald", "marketing_copy": "Percy Grunwald explores unicode matching in Elixir and shows how a trivial exercise can teach useful real-world lessons about handling Unicode in Elixir.", diff --git a/posts/parallel-letter-frequency-unicode-matching-elixir.md b/posts/unicode-matching-in-elixir.md similarity index 100% rename from posts/parallel-letter-frequency-unicode-matching-elixir.md rename to posts/unicode-matching-in-elixir.md