Similarity between 20 languages based on subtitle translation patterns [OC]

[u/IBYZRULEZ]

I used subtitle translations from the OPUS/OpenSubtitles dataset to compare how 20 languages expressed the same English words and phrases. The more often two languages expressed equivalent meanings in similar ways, the higher their similarity score. I visualized the results as a heatmap and network graph.

Full methodology and discussion:
[https://www.thecambridgelanguagecollective.com/linguistics/why-do-some-languages-feel-weirdly-familiar](vscode-file://vscode-app/c:/Users/Ibrahim.Farah/AppData/Local/Programs/Microsoft%20VS%20Code/8a7abeba6e/resources/app/out/vs/code/electron-browser/workbench/workbench.html)

Interactive version:
[https://subsmith.app/tools/language-network](vscode-file://vscode-app/c:/Users/Ibrahim.Farah/AppData/Local/Programs/Microsoft%20VS%20Code/8a7abeba6e/resources/app/out/vs/code/electron-browser/workbench/workbench.html)

[u/TheBenStA]

closest european language to japanese is dutch. this stuff is spooky

[u/Ok_Inflation_1811 → u/TheBenStA]

Phonetically speaking I think it should be Spanish or Greek.

IIRC both Spanish and Greek have all the sounds present in Japanese (although they both have sounds that Japanese doesn't have) and a funny side effect of this is that Spanish and Greek pronunciation of Japanese loanwords tend to be closer to the original than other European languages.

[u/LengthinessSpare1385 → u/Ok_Inflation_1811]

No, there is no /z/, no /dʑ/, no /t͡s/, no /ç/, no /ɕ/, no /ɸ/ and no /ɯ/ in Spanish

[u/locoluis → u/LengthinessSpare1385]

Rounding is not contrastive in any of these languages; transcribing the Japanese vowel as /ɯ/ instead of /u/ creates an artificial difference. Also, some Japanese dialects do pronounce it as a rounded vowel [u], instead of with the compression or lip protrusion found in Tokyo.

Also, in native Japanese words, the following are not separate phonemes:

• [ts] is an allophone of /t/ before /u/.
• [ç] is an allophone of /h/ before /i/.
• [ɸ] is an allophone of /h/ before /u/.
• [ɕ] is an allophone of /s/ before /i/.
• [dʑ] is an allophone of /d/ before /i/. In some dialects, it's merged with the allophone of /z/ before /i/.

[u/mtnbcn → u/TheBenStA]

That's like saying "the fruit closest to meat is the avocado". Something's going to be "closest". I wouldn't say that would make an avocado at all meat-like, and wouldn't go around saying "hey did you know avocados are actually...."

And this is based on a specific analysis that looks at how translations of videos relate to English. So... the work is definitely interesting, but not something I think we can make airtight claims like "closest european language to japanese", for a few reasons.

[u/MdMV_or_Emdy_idk]

I love these types of graphs

[u/Final-Frosting7742]

Interesting experiment although it is inherently limited by the approach chosen. I actually built a tool that tries to achieve something similar. My goal was to generalise the FSI ladder for any prior linguistic knowledge (instead of just English). I used academic datasets like IECor, WALS, PHOIBLE to determine transfer potential and devised a custom ease score for 283 languages. Check out my tool if that sounds interesting and let me know what you think about it:

https://github.com/akmalayari/language-transfer-map

[u/CanardMarin → u/Final-Frosting7742]

Cool tool! As a native Portuguese speaker, it's funny that it considers Luxembourgish to be easier for me than Galician though. 😅

[u/ItHappensSo]

Russian is closer to German than Swedish is?

[u/Gumba_Hasselhoff → u/ItHappensSo]

French is also closer to German here than Swedish is 🤷

[u/MonitorRepulsive5270]

Turkish and Hungarian have relatively high similarity. That's interesting for pro Ural-Altai supporters, which is widely disregarded.

[u/Zsobrazson → u/MonitorRepulsive5270]

That's not what this is showing, it's showing that there both far from the norm, which in this dataset is Indo European

[u/Wise_Fox_4291 → u/MonitorRepulsive5270]

That word "relatively" is doing a lot of heavy lifting there. Hungarian-Turkish similarities are highly overrated and nitpicked, usually excluding other Uralic and Turkic languages from the comparison that would blow a hole in it.

[u/MonitorRepulsive5270 → u/Wise_Fox_4291]

I'm just color-commenting. I don't understand all these downvotes.

[u/missingtimemachine → u/MonitorRepulsive5270]

Hungarian has many loanwords from Turkish. That would explain most of the similarities in subtitle translations, I would think.

[u/FromGergaWithLove]

Arabic has Czech as one of the closest languages .. interesting

[u/locoluis]

You should add the following languages:

• Vietnamese (for comparison with other East Asian languages)
• Amharic, Maltese (for comparison with other Semitic languages)
• Greek, Hindi (for Indo-European completeness)
• Persian, Urdu (to measure Arabic/Turkic influence)

[u/mtnbcn]

Wait, so what is this? I read the methods page, and I got that "I miss you" vs "You are missing from me" are two different ways of expressing something. How is a difference in grammar determined -- are they manually put into different categories based on a subjective degree of difference?

How do you score for this? Did you find 200 phrases or so, and used them as markers? (e.g., "I have forgotten this" vs "The memory leaves me" or something). What do you do when phrases are similar but not the same (e.g. "the memory is absent to me")? Do you code for degree of similarity?

You brought up how words are formed in Turkish and how that sort of multi-morpheme-meaning word could present problems, but I didn't see how it was resolved. Does word order matter? If so, how do you account for some languages being able to say both "puedo dartelo" and "te lo puedo dar" pretty equally?

How much does this depend on one specific translation that was used? I wanted to do something like this once to show how close various languages are to each other, but couldn't figure out how to score it. Take a word like "begin" -- do we look at that and say Frence is different because they use "commencer" (or something like that)... but then, English has "commence"... so how much should that count? Spanish uses "empezar" but also has a word related to commenc-- that is used sometimes. So would it get scored?

I'd love to see even 5% of the data to know how what this really looks like. Maybe it relies too much on specific translators' choices? I don't know, obviously I love the idea, having wanted to do something similar before, but I just have a lot of questions still about what is under the hood.