Special Characters Problem

[ts0unami]

The script works fine, however the output file converts special characters such as 中央区 and Athénée to gibberish. i.e.

中央区 -> 中央区

Athénée -> Athénée

The input file is a UTF-8 .CSV saved in MAC excel.

Could you please provide a revised script to handle special characters?

[shanealynn]

Hi Ash - this may be a problem with excel rather than with the python code - the encodings I find from Excel tend to get messed up with UTF8. :( I've had this problem before a few times. If you load the file into Pandas and then resave as CSV and just open again - do you get the same problem?

…

On Fri, Sep 21, 2018 at 7:54 PM Ash ***@***.***> wrote: The script works fine, however the output file converts special characters such as 中央区 and Athénée to gibberish. i.e. 中央区 -> 中央区 Athénée -> Athénée The input file is a UTF-8 .CSV saved in MAC excel. Could you please provide a revised script to handle special characters? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHyi3J2tYDRjacZkgdTkTyrsSqYPeFCkks5udTXegaJpZM4W0smY> .

[ts0unami]

Hi Shane,

I actually solved the problem by changing this line to utf-8-sig:

pd.DataFrame(results).to_csv(output_filename, encoding='utf-8-sig')
Maybe you can update your code for others, if you agree? Not sure why, but it works!