Check valid encoding given an alphabet

[tcerqueira]

The problem

• I want to check the encoding is valid without decoding

A bit follow-up to this #191 .

Given an alphabet this seems a useful feature to me. Currently, I'm doing as suggested: decoding, checking for errors and discarding the result but I'm not sure the compiler can always optimize the allocations away.

This is not complex, right? We can check for the 4 kinds of error in DecodeError without decoding.

• Character not in alphabet
• Invalid length
• Invalid symbol before padding
• Invalid padding

Asking for some feedback, I can work on it.

[marshallpierce]

Yep, I think it's a reasonable feature if you need more speed than decode_slice can give you (e.g. don't want to have the space to write into, or don't want to generate the write bandwidth). If it doesn't autovectorize well (though it has a reasonable shot at doing so), it's pretty amenable to SIMD table lookup ala pshufb.

I think likely the natural home for this is another method on Engine.

For GeneralPurposeEngine, I'd tackle it like so:

• Partition off the input that's a multiple of 4 tokens in length, leaving the suffix (which might itself be 4 bytes). Then, you can do something like most_of_the_input.iter().all(|b| table_lookup(b) != INVALID_VALUE). This is what will hopefully be autovectorized.
• Since the suffix (1-4 tokens) is where all the complexity is, now you can do the relatively mundane but finicky work of checking the various padding cases per engine config, etc, without worrying about performance.
• Tests. Lots of tests. There should be plenty of decode tests with all sorts of invalid cases to crib from.