When I saw the latest in the Lord of the Rings trilogy of movies a short while ago, I wondered how Tolkien had invented the artificial languages of Middle Earth. In my previous article, I told of my desire to discover which real language had been the biggest influence on Tolkien for his invented ones. As a software developer, I wanted to discover this information algorithmically. My idea was to use my own string similarity algorithm to compare each word from a list of Tolkien words to words from 14 other real languages. For each Tolkien word, I would find and record the language with the word that is (lexically) most similar. The set of most-similar words and the languages from which they came would provide new insights into the influences on Tolkien.
Lord Of The Strings Part 2 - The Finnish Line (Page 6 of 7 )
If a language is selected as the best match for a Tolkien word, I call it a 'hit'. As you can see from the results of the query, Finnish scores the highest by a clear margin, with 100 hits. The languages with the next-most hits are English, Spanish and Japanese, with 64, 62 and 52 hits, respectively. At first sight, this seems to support the argument that Finnish is the language that had the biggest influence on Tolkien (as suggested by the Finnish speaker, Harri Perälä).
However, if you recall the sizes of the word lists, analyzed at the end of the Part 1 article, you will remember that the Finnish word list is also the largest word list of all the languages, comprising 21.4% of the words in the database. So perhaps the reason why Finnish scores so many hits is simply because I had so many Finnish words in the database?
Clearly, to conclude anything from the results, we need to take account of the relative sizes of the word lists for each of the languages. Perhaps the best way of doing this is to compute an expected number of hits for each language (based on the size of its word list), and compare that number to the actual number of hits that we found. The following pair of queries computes the number of expected hits for each of the languages, under the assumption that any of the languages is equally likely to contain the most similar word to any given Tolkien word (a so-called 'null hypothesis').
> select @total:=count(*) from words where lang<>'tolkien'; +------------------+ | @total:=count(*) | +------------------+ | 1342940 | +------------------+ 1 row in set (1.20 sec) mysql> select lang, round(count(*)*470/@total,1) as expected_hits from words where lang<>'tolkien' group by lang order by expected_hits desc; +-----------+---------------+ | lang | expected_hits | +-----------+---------------+ | FINNISH | 100.5 | | DUTCH | 62.4 | | GERMAN | 56.0 | | FRENCH | 48.4 | | JAPANESE | 40.3 | | POLISH | 38.3 | | SPANISH | 30.1 | | LATIN | 27.0 | | NORWEGIAN | 21.6 | | ENGLISH | 19.7 | | DANISH | 8.9 | | SWAHILI | 6.4 | | HUNGARIAN | 6.2 | | SWEDISH | 4.2 | +-----------+---------------+ 14 rows in set (1.06 sec)