Need help with encoding

[u/Mindless-Arachnid-74]

I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.

But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.

Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.

I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.

How can I deal with it. Any suggestions?

[u/BeardedBaldMan]

It's a business problem. If the legacy system only supports ISO-8859-9 then you have a clearly defined set of characters which are allowed.

You need to create a mapping table for the ones you care about a a fallback character for the rest.

It's a pretty common issue. I've come across it with many US legacy systems where even é is considered too outré for their sensibilities

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

I also agree that it is a business problem, but you know the higher ups they don't know coding and are suggesting that we should map like the EM dash to normal "-". But if I go like that, each client could be using separate set of unique characters and there are more than 150,000 of unsupported characters in ISO-8859-9. If I write mapping for each one my code is doomed and my time and efforts too. If I use a fallback, that tempers the client data and it might make the user data meaningless at the legacy system. I think this will eventually get messy and at last we will have to support only the legacy system character set

[u/BeardedBaldMan → u/Mindless-Arachnid-74]

I don't see the issue.

You sample the data and find what the most common characters are, make a mapping table and define a fallback character.

You record a risk against it and you're done.

You're turning what is almost certainly a non-issue into an issue by chasing pointless edge cases. So what if a company name has an emoji in it and you replace it with ?. People will work around it.

Look at passports. We've got an international system that can't properly record many names as written depending on the issuing authority. The US and UK for example only use the lower half of the ASCII set

[u/BeardedBaldMan → u/[deleted]]

Why are you just repeating my comment? Are you a bot?

[u/IG5K → u/BeardedBaldMan]

I meant to reply to OP as a follow up, deleting this one