ACP: Document how escape_debug escapes characters.

[tmccombs]

Proposal

Problem statement

It is often desirable to escape a string in such a way that non-printable chracters, and some special characters like " and ' are escaped, but
most unicode printable characters can be printed as is.

Motivating examples or use cases

~~Consider https://github.com/sharkdp/fd/pull/1866/changes#diff-0d5151a0c02926abbc8a38d7c87533a96b33d24923034c0b0805d3539543b466R32-R36 .

In this case, I want to serialize some content for JSON. The json object is pretty simple, so pulling in a full serialization library, like serde-json is overkill. However, I do need to escape a string in a way that is usable in json strings.

escape_default would technically work, but it isn't ideal, because it would escape a lot of characters that I don't actually need escaped, and is thus less user-friendly if a human actually reads the resulting Json.

While this is useful for escaping in json, there are many other contexts where this kind of escaping is necessary, and could be useful.~~

Edit: this doesn't actually work for my intended use case. But I still think it would be useful to document it better.

Solution sketch

Add documentation to impl Debug for str, str::escape_debug and char::escape_debug that describes the escaping that is done by those functions, and commit to that escaping being stable going forward. This documentation may something to the effect of:

The following chars will be escaped in the following ways:

• \n, \t, \r, \0, are escaped using the escapes used to represent them here (ex "\\n" for a newline). That is, the same backslash escape that can be used in a rust string literal.
• ", ', and \ will be escaped by preceding the character with a "" (Which quote(s) is escaped currently depends on wither we are escaping a str or a char).
• non-printable (unicode) characters will be escaped using the unicode escape for that code point (as with char::escape_unicode).

Alternatives

• Instead of documenting the Debug implementation and the escape_debug functions. A new function could be added specifically to provide a stable interface. That way, the existing functions, and Debug implementation could be changed in the future, while still giving applications a garantee of the escaping behavior for a desirable escape function. However, this would create a more complicated API, and could be confusing on when you should use this new function vs. when you should use escape_debug.
• escape_debug could be more fully specified (for both str and char), but leave the impl Debug unspecified, and open to change in the future.
• Recommend using a third-party crate if stable and/or well specified behavior is desired. This allows flexibility in changing the output of these functions in the future. But it also seems wasteful to require adding a dependency when std already includes a suitable implementation that simply isn't guaranteed to remain suitable.
• Recommend using escape_default. As mentioned, this often escapes all non-ascii characters, and it is often desirable not to escape printable non-ascii characters.

I personally don't have a strong preference between the proposed solution and the first two alternatives, I think any of those options would be fine.

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[hanna-kruppe]

I don’t think any of the current escape functions, nor the new escaping rules you propose, nor any minor variation of it, would work for producing JSON.

• Rust uses \u{FFFF} syntax (possibly even with fewer hex digits) while JSON uses \uFFFF (always exactly four hex digits)
• When escaping non-BMP code points, Rust will escape the code point using more than four hex digits, while JSON requires splitting into two UTF-16 surrogates and escaping each half separately.
• Even for ASCII, the escape syntax of JSON (ant least RFC 8259) and Rust have some overlap but neither is a subset of the other. For example, JSON lacks Rust’s has \0 and \xFF while Rust lacks JSON’s \f and \b

Conversely, JSON string escaping that’s optimized for performance or wants to do as little escaping as possible will just print all non-ASCII characters unescaped since JSON only requires a small subset of ASCII to be escaped. Non-printable Unicode characters are allowed (just potentially confusing for human readers).

If your motivation is producing JSON without pulling in serde, there are several alternatives that are very small but actually implement the correct escaping, such as tinyjson, write-json, or (string escaping only) escape8259.

[joshtriplett]

We reviewed this ACP in today's libs-api meeting.

We'd like to see escape_debug documented. We were hesitant to commit to the exact set of characters that are escaped, but we would like to document how characters are escaped in general, and add a disclaimer that "currently, the set of characters that are escaped as unicode escapes are the characters Unicode designates as non-printable, but this function may change which characters are escaped in the future".

We generally felt that using one of the crates @hanna-kruppe recommended would be a better choice. Not least of which because dedicated json crates are also faster. But we do want to see the behavior of this method documented, and you can use it if it works for you.

Please close this after filing a PR to document escape_debug (including the disclaimer mentioned above).

[tgross35]

currently, the set of characters that are escaped as unicode escapes are the characters Unicode designates as non-printable, but this function may change which characters are escaped in the future

I was digging into this a little while ago because I had a similar need; is there an actual definition of non-printable in unicode? We have https://github.com/rust-lang/rust/blob/94a0cd15f5976fa35e5e6784e621c04e9f958e57/library/core/src/unicode/printable.py but it's not clear to me whether that's matching a specification somewhere or just our best-effort approach.

If there is such an official definition or if we can come up with one that is reasonable, I think it would be rather nice to have a char::is_unprintable method because that allows reimplementing the escape_debug pattern except for Grapheme_Extend which isn't always used.

I see escape_unprintable was brought up in the meeting; I think a boolean is_unprintable check would be better to allow for picking between \u{...} and other escape patterns.