frontend, libobs: Base work for localization and utf-8

[Dankirk]

Description

• Sets UTF-8 as active codepage for Windows in obs.manifest file.

  • This changes Windows APIs to use utf-8 for the A(nsi) functions instead of the language dependent ANSI codepage. The W(ide) functions are untouched and are still used by default since we use the UNICODE build flag. This also helps directing less encoding-aware 3rd party plugins and libraries towards a supported encoding.

  • Active codepage also encodes commandline arguments on Windows as utf-8, which allows for example ./obs64.exe --profile <name> to load profiles with special characters correctly.

• Change time formatting for logging to use ISO standard without localization to preserve logging format in different locales.

• Casts ctype function parameters from char to unsigned char to prevent non-ascii input from crashing them. Functions expect parameters to be in 0-255 range, which char using utf-8 casted to int might not be (signed char range is -128 to 127)

Motivation and Context

This is based on discussion PR #12624 and split from the main issue there.
This serves as ground work for enabling localization and handling utf-8 data with C APIs

How Has This Been Tested?

Tested active code page effect by naming a scene with some Japanese characters, selecting another scene so it's not opened by default on restart and then restarting the app with command line

./obs64.exe --scene <japanese characters>

This wasn't working in current version on Windows.

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

• Breaking change (fix or feature that would cause existing functionality to change)
  • 3rd party plugins etc, that have been using -A versions of Windows API have to expect them to give out/take in strings encoded in utf-8 instead of default ANSI code page.
    • It's still not broken if:
      • It's in ASCII range
      • If they needed wchar of it they used MultiByteToWideChar() with CP_ACP flag (using other flag may have been broken, but if it was CP_UTF8 it is now fixed)
      • Otherwise if the text was used as input to an obs api directly, it's now coincidentally in correct encoding.
    • It is breaking if:
      • Their source file is in their default ansi non-utf8 encoding, their build options reflected that, they have non-ascii hardcoded strings there and they used them for a Windows -A api, but not for an obs api
      • They saved the string, it was non-ascii, and they try to use it now for Windows A function again.
    • Otherwise it was probably already broken

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[Dankirk]

If the manifest utf-8 declaration is deemed too risky (detailed this in "breaking change" section of the description), we could do the commandline argument parsing with GetCommandLineW() or introducing wmain() for Windows and following with os_wcs_to_utf8().

[PatTheMav]

@RytoEX Was split off from the original PR. Changes to the time format (using fixed %F and %T) seem indeed more correct to me, and casting the character values to unsigned char to fit the expectations of the underlying functions is also correct to avoid triggering undefined behaviour.

The UTF-8 codepage change allows OBS' existing commandline argument parsing code to correctly handle arguments using non-ASCII characters on Windows, but will affect any call to an ANSI Windows API function (and indeed any call to a C library call) in turn.

[RytoEX]

Generally, I prefer the explicitness of %Y-%m-%d over the brevity of %F because I can immediately recognize the former while I have to look up the latter.

@PatTheMav Aside from the breaking items mentioned, is there anything else this would break?

[Dankirk]

Discovered a potential issue related to file operations using A functions with the codepage change.

On Windows the A apis largely support 260 chars/bytes as max path length (W apis support 260 wchars/utf-16 codepoints). On codepages like shift-JIS or Windows-932 Japanese characters use 1-2 bytes, while in UTF-8 most will be turned into 3 bytes consuming more space from the 260 bytes. That means some long paths may stop working for code that uses A functions.

This is not really a problem for OBS itself, we use W apis where it matters, but could be an issue with less prepared plugins.

[PatTheMav]

Discovered a potential issue related to file operations using A functions with the codepage change.

On Windows the A apis largely support 260 chars/bytes as max path length (W apis support 260 wchars/utf-16 codepoints). On codepages like shift-JIS or Windows-932 Japanese characters use 1-2 bytes, while in UTF-8 most will be turned into 3 bytes consuming more space from the 260 bytes. That means some long paths may stop working for code that uses A functions.

This is not really a problem for OBS itself, we use W apis where it matters, but could be an issue with less prepared plugins.

The unfortunate reality is that the UTF-8 codepage seems to be more of a crutch than anything and is still considered "beta" quality for these (and other) reasons. IIRC some ANSI-variants of Win32 APIs also will break in unexpected ways if the UTF-8 codepage is used.

The rub (at least as far as I can tell) is this - OBS itself can (and probably should) do the following:

• Replace all(!) uses of ANSI APIs with UNICODE APIs in all first party code. There are still quite a few instances of legacy Windows code in our code base that needs to be updated.
• Pass /ENTRY:wWinMainCRTStartup as a linker switch to enforce detection of OBS Studio as a UNICODE application. This is necessary because MSVC's linker cannot "see" the WinMain or wWinMain entry points provided by Qt as they exist in a static library only.
• Switch OBS to using the CP_UTF8 code page via manifest.
• This will force Qt's entry point code to convert the UTF-16 command line (passed to OBS Studio's wWinMain entry point enforced by the linker flag) to CP_ACP which should effectively be CP_UTF8.
• Optionally ignore argv on Windows and use CommandLineToArgvW directly. This might not be worth it, because Qt will always convert UTF-16 to CP_ACP and simply call main with it, so might as well use the conversion result and keep the same main for all platforms.

First party code would be unaffected by any bugs related to the UTF-8 codepage because no ANSI APIs are used as you mentioned.

But without thorough review we cannot say whether any of OBS Studio's direct dependencies (FFmpeg, et. al.) use ANSI APIs and whether those APIs (and the APIs they call transitively) are not affected by bugs in relation to the UTF-8 codepage.

And as you yourself have mentioned, any OBS Studio plugin that naively uses ANSI APIs might also break in unforeseen ways, which would mean that we'd have to announce the change in conjunction with a major version release and make the switch in the major version after to allow developers to migrate their code.

But just like with our recent DLL loading changes, we should not be deterred from doing "the right thing" in perpetuity just because of plugins.

[Dankirk]

Replace all(!) uses of ANSI APIs with UNICODE APIs in all first party code. There are still quite a few instances of legacy Windows code in our code base that needs to be updated.

Just started pr #13749 yesterday to address this from filename handling perspective. There's still a few cases of A usage left not directly related to files that need addressing, though admittedly their impact may be small (hardcoded things, things that shouldn't have non-ascii characters). I think I will be looking in to this soonish tough.