last day (14 days later) » 

1:09 PM
2
A: determination of human language from text:: system structure

Alex NevidomskyThat would work as a first approximation. The problem with fixed word lists for language detection, though, is that real texts (and especially short ones) may not provide enough hits in your list. A more reliable approach would collect parts of other language features (like statistics of letter ...

 
that's a good idea, are there lists of such functional words? __________________________________________________________________________ it's a learning excersize and I'm prohibited from using a library. __________________________________________________________ that's a very funny / good / creative example.
 
It helps to learn the language a little, then you can take them manually. But for a first approximation simply take short words with counts from some very large text.
Technically speaking, you can treat the problem as a purely statistical exercise: the count for a specific word or letter combination in a sufficiently large corpus of text can be seen as probability. Then, you are building a classifier that tries to assign maximum language probability (called likelihood) based on a number of observations.
 
the system I described in the OP is also sort of doing that isn't it? generating a probability based on observed words. in a sense.--------------------------------------------------------------------------‌​---------------------what do you have in mind? for instance, saving many documents from each language? how is that more efficient that just the words? because you also assign a weight (frequency count) to each word?----------------------- I think the nature of the text would pervert the count, such as specialized domains and the like. a pure list of common words seems better to me.
 
For short words you would need the counts, especially because they sometimes overlap across languages, and you need to know how reliable the specific word is. For a plain dictionary approach you are correct, it is probably not necessary.
 
actually i don't have any idea though. i think it's a good suggestion and I'm going to try to think about how to implement it. do you have any documentation or references describing it?
 
1:09 PM
I'm afraid most literature is quite technical, nothing comes to mind to suggest for an "entry level". Also, I don't quite understand the focus of the exercise: is it in coding or linguistics? You need to start with a piece of code that takes a number of texts (from files?) and converts them into feature sequences (calls a function on every feature found, such as word). Then you use it in two ways: an a number of texts of known language for training, and then on a number of texts of known language for testing.
 
the excercize was exactly this: "We would also like to ask you to implement a method to identify the language a document is written in. Please do not use an existing library to solve the task, but implement an approach on your own. If you re-implement an approach taken from the literature, please provide a reference. The suggested programming language is Java."
 
So it is a programming exercise in a course of computer science?
 
more or less yes
 
So start with a very generic piece of code.
 
I'm starting with this... hold on, let me just push to github what I've been worrking on
 
1:11 PM
Create some (fake) input files, data files, a program that reads them, tokenizes the text and calculates something.
And you can do the calculations later.
 
so far I have just this
i was thinking I might put them all to a hashmap and try to name the hashmap after the directory where they come from, and I was just thinking how to do that
by the way, i'm currently reading this book, which I think is quite good
 
1:42 PM
Well, it's a good start, but the details of reading directories can always be added at the very end. The first prototype can easily work out of string arrays. The key is to get something working as fast as possible.
 

last day (14 days later) »