Add ascii fast path for unicode_word_indices and unicode_words

[PSeitz-dd]

This PR adds a fast path for unicode_word_indices and unicode_words.

• Add ascii version behind ascii check
• Add benchmark
• Add enum Iterator to wrap ascii and non-ascii into a single iterator
• Add proptest to make sure ascii and non-ascii methods behave exactly the same.

Performance for ASCII text is greatly improved, while everything else gets a slight performance hit.

unicode_word_indices/log
                        time:   [209.68 ns 212.35 ns 215.32 ns]
                        thrpt:  [761.82 MiB/s 772.45 MiB/s 782.29 MiB/s]
                 change:
                        time:   [-82.313% -82.117% -81.888%] (p = 0.00 < 0.05)
                        thrpt:  [+452.12% +459.18% +465.38%]
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
unicode_word_indices/english
                        time:   [426.95 µs 428.03 µs 429.18 µs]
                        thrpt:  [110.42 MiB/s 110.72 MiB/s 110.99 MiB/s]
                 change:
                        time:   [+2.0057% +2.3798% +2.7729%] (p = 0.00 < 0.05)
                        thrpt:  [-2.6981% -2.3245% -1.9662%]
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
unicode_word_indices/japanese
                        time:   [415.66 µs 416.25 µs 416.86 µs]
                        thrpt:  [116.02 MiB/s 116.18 MiB/s 116.35 MiB/s]
                 change:
                        time:   [+0.5766% +0.8184% +1.0398%] (p = 0.00 < 0.05)
                        thrpt:  [-1.0291% -0.8118% -0.5733%]

[Manishearth]

issue: removing these types is a breaking change

We deliberately have concrete types here so that they can be used inside other things conveniently.

[PSeitz]

I updated the PR to keep the Iterator structs
(personally I prefer Iterators though as the contract is clearer)

[Manishearth]

I don't want this to allocate either.

[PSeitz]

I replaced it with a custom enum iterator (it's also much faster :))

[Manishearth]

as it stands I think this is still a ways away from landing, for spec correctness reasons and for performance reasons. We should not be allocating a trait object here; you can get the same effect with a custom iterator that wraps an enum around the two iterators. (Either also works for this but I don't want to add a dependency to a somewhat bottom-of-the-deptree crate)

[PSeitz-dd]

as it stands I think this is still a ways away from landing, for spec correctness reasons and for performance reasons. We should not be allocating a trait object here; you can get the same effect with a custom iterator that wraps an enum around the two iterators. (Either also works for this but I don't want to add a dependency to a somewhat bottom-of-the-deptree crate)

I replaced the Box with a custom enum. It's faster, and the performance hit for non-ascii is quite low now.

[Manishearth]

This should not be pub, maybe pub(crate) if you need it to be

[Manishearth]

Thanks!!