Too complicated words

[mat-teo]

Hi, I'm using your API for my small project (just a simple tipying speed test with random words). It's great but I think that there are many rare or "deprecated" words (at least regarding Italian which is my main language). I think it could be a cool option adding a way to choose the "difficulty" (or maybe the rarity/complexity) of the word to generate. Maybe using a separated JSON with only common or simple words (I can make it for you for the italian). Thanks

[RazorSh4rk]

I understand the issue, and I've seen it in the English corpus as well, but since I don't speak Italian, it would need 2-3 volunteers who go through the words and verify them, and that for every language. You can use the length filter, that can get rid of the longer words, but aside from that, this would be a tremendous effort. If you can come up with a way to repopulate the corpus for each language programmatically, I'd be more than happy ton incorporate that. I've also been thinking of crawling wikipedia to build up new corpuses (corpi?), but I'm in the middle of a move right now so that will come in the future unless someone makes a pr for it.

[RazorSh4rk]

Leaving this open in the meantime, because I agree with the idea, just would prefer not doing it manually

[mat-teo]

I have made a research and I've found a possible solution:
Adding another filter (?diff= or something like that) that goes from 1 to 3 according to the diffusion. So if someone doesn't care about the diffusion of the words can simply use the API without the filter like we're doing now, but I want to check the diffusion I use it and the API will check the diffusion of the word with Wikipedia API ("https://{lang}.wikipedia.org/w/api.php?action=query&list=search&srsearch={word}&format=json&origin=*") and by looking at the data.query.searchinfo.totalhits we can see the number of pages that contain that word, so according to the value of the parameter given by the request (1->rare, 2->not common, 3->common, or something like that) we decide if the word falls within it (maybe <500 rare, <1500 not rare and >1500 common). If you like the idea I will try to figure it out and make a pr.

[RazorSh4rk]

That's a really good resource, if you have some ideas on implementing this, I'd be open to it, but I'd prefer storing this information instead of having to query it on every request, the API gets about a thousand calls a minute, so querying it real time would be incredibly costly.
Anyway, if you have a spike ready, please link it in this thread :) And thank you!

[mat-teo]

I made a script in JavaScript that gets in input the JSON file with all the words and then recreate a new one identical but with the frequency added for every word!
The problem is that just the italian file has more than 270.000 words, so it would take like 20 hours to end.

[RazorSh4rk]

That's fine, I can port it to something multithreaded, if you make a PR with your script in a /scripts/ folder, I'd be willing to take a look at it.

(Sorry for taking so long to reply, I'm currently driving across the states haha)

[xeruf]

came here with a similar concern: want to make random daily prompts for learning a new language - in this case some friends who want to learn German - but the words are too complicated

as a German I find that German wordset quite weird, wonder where it is from?
Thinking about integrating the dict.cc database instead.

There are a lot of duplicates:

"Abdruck","abgedruckt","Abdrucke","Abdruck","vonframp","vonframps","ab","Ermutigung","Ausschreitungen

also around the word "normalisiert"
and a lot of declinations of the same words which seems a bit redundant
but worst of all a lot of really odd anglicisms I as a German have never heard of and that are not really proper German

[xeruf]

for illustration: I do not know 8 of these words, and 5 of them are not really german ("Gegentrade" sort of makes sense but seems a very niche colloquial use with "trade" borrowed from english)

[RazorSh4rk]

Ignore the typo in the commit message, but difficulty functionality has been added!