Lord Of The Strings Part 2

If a language is selected as the best match for a Tolkien word, I call it a 'hit'. As you can see from the results of the query, Finnish scores the highest by a clear margin, with 100 hits. The languages with the next-most hits are English, Spanish and Japanese, with 64, 62 and 52 hits, respectively. At first sight, this seems to support the argument that Finnish is the language that had the biggest influence on Tolkien (as suggested by the Finnish speaker, Harri Perälä).

However, if you recall the sizes of the word lists, analyzed at the end of the Part 1 article, you will remember that the Finnish word list is also the largest word list of all the languages, comprising 21.4% of the words in the database. So perhaps the reason why Finnish scores so many hits is simply because I had so many Finnish words in the database?

Clearly, to conclude anything from the results, we need to take account of the relative sizes of the word lists for each of the languages. Perhaps the best way of doing this is to compute an expected number of hits for each language (based on the size of its word list), and compare that number to the actual number of hits that we found. The following pair of queries computes the number of expected hits for each of the languages, under the assumption that any of the languages is equally likely to contain the most similar word to any given Tolkien word (a so-called 'null hypothesis').


mysql> select @total:=count(*) from words where lang<>'tolkien';
+------------------+
| @total:=count(*) |
+------------------+
|          1342940 |
+------------------+
1 row in set (1.20 sec) 
mysql> select lang, round(count(*)*470/@total,1) as expected_hits 
       from words where lang<>'tolkien' 
       group by lang 
       order by expected_hits desc;
+-----------+---------------+
| lang      | expected_hits |
+-----------+---------------+
| FINNISH   |         100.5 |
| DUTCH     |          62.4 |
| GERMAN    |          56.0 |
| FRENCH    |          48.4 |
| JAPANESE  |          40.3 |
| POLISH    |          38.3 |
| SPANISH   |          30.1 |
| LATIN     |          27.0 |
| NORWEGIAN |          21.6 |
| ENGLISH   |          19.7 |
| DANISH    |           8.9 |
| SWAHILI   |           6.4 |
| HUNGARIAN |           6.2 |
| SWEDISH   |           4.2 |
+-----------+---------------+
14 rows in set (1.06 sec)