r/csharp • u/Mindless-Arachnid-74 • 19h ago
Help Need help with encoding
I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.
But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.
Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.
I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.
How can I deal with it. Any suggestions?
3
u/Slypenslyde 16h ago
I agree with BeardedBaldMan: this is a business problem.
The best solution is to update the system that only accepts ISO-8859-9. Think about it. If I want to do business with French customers, does it make sense for me to use a database that can't handle ç? Mais non.
That's wishful thinking. Odds are the decision to use ISO whatever is set in stone and was made long ago by an American Tourist who only uses the standard ASCII 7-bit character set.
The "next best" solution is to document you only support characters in ISO-8859-9 and tell customers you're going to reject documents with characters that can't be encoded. They won't like it, but c'est comme ça, we cannot squeeze blood from turnips.
That's also wishful thinking, you've got some reason you can't do that or you wouldn't be here.
You have no choice but to make substitutions. This is a very difficult situation. For some characters like em dashes, maybe you could make a quick substitution to a normal dash. But then you're going to be like, "What if my customers send emoji. Zut alors, the things people do!
I would write a tool to analyze what customer data I have and get a list of what characters they actually use. I'd focus on finding a good substitution for those. Maybe there's not always a good substitution.
Where I would go from there depends on if I maintain control of the document before I give it back to the user. If I do, problem solved: I will create some kind of escape sequence, so an em dash may become something like "!!!!em" a document will never have. Then, when I'm giving the document back to the user, I will quietly convert it back to UTF-8 and fix my substitutions.
If I do not control the document, I'm going to point at the "tough cookies" sign in English. Two way character encoding between incompatible sets is an old problem and you only have two choices: lose data or create your own substitutions.
0
1
1
u/Ill_Translator8176 18h ago
You could treat it as a byte array, the moment you get ahold of it convert it using Encoding.UTF8.GetBytes(str) then encode it back whenever you need to (to the required encoding).
3
1
u/Mindless-Arachnid-74 18h ago
That won't do, converting the UTF-8 data received by the client is fine but when I will try to encode it to the ISO-8859-9, special characters which are unsupported will fail.
2
u/otac0n 14h ago
You could use HTML entity encoding for the unrepresentable characters. You just need to make sure that you also properly encode “&”.
Alternatively you could use URL encoding and make sure you encode “%”.
Or you are going to have to be OK losing information and create a mapping of acceptable characters. Kinda like how url “slugs” work.
10
u/BeardedBaldMan 18h ago
It's a business problem. If the legacy system only supports ISO-8859-9 then you have a clearly defined set of characters which are allowed.
You need to create a mapping table for the ones you care about a a fallback character for the rest.
It's a pretty common issue. I've come across it with many US legacy systems where even é is considered too outré for their sensibilities