« first day (1527 days earlier)      last day (3424 days later) » 
01:00 - 21:0021:00 - 00:00

abc
9:00 PM
10
Q: Is it good practice to yield from within a context manager?

sapiI recently wrote a method which returned a sequence of open files; in other words, something like this: # this is very much simplified, of course # the actual code returns file-like objects, not necessarily files def _iterdir(self, *path): dr = os.path.join(*path) paths = imap(lambda fn:...

 
so with that do you think i can apply some methods?
 
@ml_guy honestly, you've not provided any detail of what you want to do, so that question is pretty unanswerable.
 
Yes... that's what i said if i can put my question
here
maybe you can see my profile
 
user2555451
It sounds like you want to treat a bunch of files as one file, which means you will need to read them all in at once. This isn't a very efficient approach.
 
and look at the last question.
yes
icodez
What can be an efficent aproach?
 
9:03 PM
It depends on what you want to do, this could be a good approach or not.
Even with your question, we can't answer.
 
user2555451
Well, what do you want to do? If you need to count things, read through the files one at a time and use collections.Counter.
 
i would like to generate a feature matrix of the whole files
in the folder
 
@ml_guy please answer on one line, rather than multiple lines. You can edit your messages by pressing up.
 
user2555451
Then read them one-by-one and update the matrix for each file.
 
user2555451
loading everything at once is too much memory.
 
9:05 PM
Ok
 
whoa @ml_guy: are you trying to do some sort of topic/author detection, using a corpus of training data?
 
Yes
text classification
I would like to learn scikit learn and i started the second task of the tutorial
 
@ml_guy I'd suggest you do a Python tutorial as well.
 
what is this feature matrix that you would like to continually update with these files?
 
Sure... i just wanted to try an alternative way
i all ready read the files and that stuff...
the training data
just texts...
 
9:07 PM
what are your features?
 
words
i vectorize them with the hash trick
I'm vectorizing the whole text and looking what happens
 
My ML is a bit rusty. Please explain what matrix[i,j] represents
 
you say you "want to try an alternative way" but you don't say what is dissatisfying with your current way
2 messages moved to Trash
 
@ml_guy seriously. Please just write with one message on one line, otherwise you fill the chat with you saying 2-3 things.
 
please speak in complete, somewhat grammatically correct, sentences on one line
 
9:10 PM
 
otherwise, I will kick you
 
alright guys, I now know exactly what @ml_guy is trying to do, though I'm unsure of why he is dissatisfied with his current approach
 
yeah, that's what I've been trying to prompt him to tell us
 
he has multiple documents. Each document is made up of lines. Each line is made up of words. Each word is assigned an ID
 
I just want to know if there is a more pythonic way to do the this task...
 
9:13 PM
(the assignment of the ID is pretty arbitrary)
 
yes
 
from this collection of texts, he wants to create a matrix
 
what makes you think the code you have is un-pythonic?
 
the (i,j)th entry of the matrix contains a number - the number of times word i appears in document j
that is what he wants. I don't why he considers what he's doing as not-good, which I realize we're discussing right now
 
2 messages moved to Trash
 
9:14 PM
Please edit your messages!
He's basically done a bag-of-words?
 
Thats why i wanted to know if i can try an alternative procedure to do this, since i'll escalate with more text the above code...
I just want to do things easier......
 
@Ffisegydd @ml_guy we've had problems with him in the past: chat.stackoverflow.com/transcript/message/20347197#20347197
 
do your txt files really have parentheticized bigrams? or do they have flat text?
 
they are like bigrams
 
@davidism I know
 
9:17 PM
@ml_guy it looks like you didn't really learn how to interact with us better since the last time I kicked you
 
Actually both... bigrams and raw text
 
wait what?! how do you know which files have bigrams and which have raw text?
 
I have them in separate folders
 
bigrams need to be extracted (dealt with separately)
 
yes all ready do that
 
9:18 PM
@ml_guy rather than revealing this new information halfway through, try to form a coherent question before asking us
 
ok, let me see if I got this right: you have a second folder full of files that contain bigrams in raw text. Out of each of these files, the original text needs to be reconstructed, and the word counts computed. Correct?
 
I dont know if everybody here is related with machine learning....maybe if i go in details nobody will understand
at least inspector is understanding..
 
@ml_guy what we don't understand is why you aren't giving enough details to form a coherent question
 
we're smart enough to be able to google what we don't get, or at the very least ask you for clarification
I only understand because I study AI for a living
 
luckily @inspectorG4dget is patient and somehow can decode your little bits and pieces, but you're polluting the chat with it anyway
 
9:21 PM
Yes...
 
@ml_guy I understand exactly what you want. But you've lead us around a merry chase and messed us about. And it's starting to get annoying.
 
@ml_guy kicked, stop saying yes to everything
sorry @inspectorG4dget, I know you were trying to help but I couldn't take it anymore
 
@davidism: no sweat. I'm up all night anyways, and this is giving me something to do. Else, I'd be quite frustrated as well. Also, hopefully, I'm able to translate sufficiently, what mlguy is saying
alright, recap time. @ml_guy, @davidism, @Ffisegydd, please bear with me while I phrase the question that we seek
 
I think I'll end up doing a fair bit of bag-of-words for Nidaba.
 
There are two folders, each of which contain text files
 
9:24 PM
Need to develop some stop words based on Stack Overflow questions.
 
we want to create a matrix such that element (i,j) contains the number of occurrences of word j in file i
For this, we need to read all the words in all the files
However, there is a slight snag: one of the two folders contains text files in the format of bigrams: (word1, word2) as raw text. These bigrams need to be decoded/stitched into the original text that they were computed from, so that we can update the matrix with the word counts
(Note that as a matter of computational efficiency, we won't actually have to create the whole text)
 
Possibly. It's possible that the bigrams are formed from the raw text already.
 
I am done with the recap
@ml_guy is my recap accurate? (please answer 'yes' or 'no')
 
@inspectorG4dget he's gone.
 
facepalm
can I stop caring about this now?
 
9:29 PM
To be fair, he was booted from the room. But after n minutes he could rejoin, which he obviously hasn't.
 
possibly because he doesn't know he has to wait n minutes?
 
Possibly, yes. I don't know what the message says.
 
he may have tried to rejoin immediately, several times without success and given up
 
I'm pretty sure you get a message when you're kicked. But I'm not that hung up about it for some reason.
 
does booting affect the user's profile in any way? I'd volunteer to be booted just to see what happens; but only if there are no repercussions
 
9:31 PM
No it doesn't.
The first kick is 30s.
 
51
A: Impose a re-entry delay on users kicked out of a chat room

balphaThis has been implemented now. The short story is: room owners can kick abusive users, who will then be banned from re-entering the room for a certain time. Of course you want not just the short story but all the dirty details, so here they are: In the user popup that appears when you click on ...

first kick is 1 minute, penalties only start applying in a 24 hour window, they're not permanent
 
@Ffisegydd: please kick me. I'll try to re-enter immediately and report what I see. Please also save this message, in case it is needed later, to explain to mods
 
@inspectorG4dget kicked, you asked for it :)
 
SCIENCE!
 
wait, did we both kick him?
 
9:34 PM
Narp.
 
I'm back
 
cbg
 
trip report please
 
the page turns to "be respectful", etc
Then, all attempts to refresh the page give the same result. So do all attempts to access the python chatroom by navigating the list of chatrooms
There does appear to be a countdown (see screenshot) that tells me how much longer my timeout will last
 
9:38 PM
Wait, I have a more important question: How do you navigate? That browser has no controls!
cbg @corvid
 
abc
Rbrb guys
 
rbrb
 
The first tab is stackoverflow.com/questions/tagged/python*. I click on the stackexchange button and click on chat, to open in a new tab
rbrb @abc
btw, this is a stackoverflow "app", built with a site-specific-browser
 
oh, I thought that was a general web browser, that makes more sense
 
I'm on a mac, and my SSB of choice is fluid.app
well, it specifically is a sandbox'd safari
But I dislike having too many chrome tabs, so this is a useful alternative
Plus, having such an app allows me to do things like SO-specific growl notifications
@davidism: does that answer all questions?
 
9:47 PM
Not quite. How did getting kicked make you feel? Paint us a word-picture of your emotions.
 
well, it was a pain the @$$ (assuming that's where I got kicked)
It was mixed. I did feel bad that I got kicked out of my favorite place to be (though i know I asked for it, and it was for science, and there would be no repercussions)
 
Relax guys... I'm just a student, if i dont say much details about this is because maybe not everybody is related with machine learning... If i start talking with technical words maybe i can ofuscade the idea...
 
on the other hand, I had to collect data, to answer the questions we wanted to answer, as a result of my being kicked
 
even i'm just learning..
 
@ml_guy: trust that we're smart enough to be able to google what we don't understand, and ultimately will come to understand what you're asking, if you make a technical post.
 
9:50 PM
@ml_guy You continually ignored what people were asking of you. Which is why that you were booted.
 
Making a post that lacks required detail only adds to our difficulty
 
I know that all you guys are pretty smart... i just want to keep things simply
 
If it helps, I'm doing my PhD in AI and am quite familiar with ML techniques, so I'm happy to translate/explain any technical terms you use, that someone claims to not understand
 
I would like an example of how to use this scikit-learn.org/stable/modules/generated/… method
Man... i know you and everybody here is pretty smart
 
there is a difference between "simple" and "cryptic"
 
9:52 PM
that's why i came here
to learn
 
@ml_guy dude! One line! How is it so hard!?
 
I know... @inspectorG4dget, @davidism, @Ffisegydd are pretty intelligent.
 
@ml_guy: PSA: if you hit the up arrow key in the chat box, it lets you edit your last message. This way, you can complete incomplete sentences and correct accidentally hitting the ENTER key
 
Ok
 
;_;
 
9:56 PM
ok, you wanted an example of how to use that function. Here you go:

sklearn.datasets.load_files('path/to/folder/that/contains/the/two/folders/of/text/files')
That is all you need. The other options are irrelevant to you or are otherwise fine with the default values
 
Lol.. sorry @davidism , In fact i would like to use scikit-learn.org/stable/modules/generated/…, but the documentation does not provide an example
 
I have just provided you with the example you seek
 
What about load_files but merging the parsing i'm ready using.. for the bigram issue?
Yes, thank you...
 
@ml_guy: are you trying to feature hash bigrams, or unigrams (singles words), or both?
 
bigrams
 
10:01 PM
so your study does not use single words at all?
 
Well.. bigrams(training) vs raw text(testing)
 
even when you test, you'll have to compute bigram features from your test data, compute the class label from the classifier and determine whether that class label was correct. You cannot train a classifier with bigrams and feed it raw text to test. Does that make sense?
 
Why i can not train a classifier with bigrams?. As i said before i'm just looking what happens.
 
You can absolutely train a classifier with bigrams. However, once trained, such a classifier cannot be fed raw text. That would make no sense. You would have to extract the bigram features of your test documents and feed those bigrams to your classifier. Does this make sense?
@davidism @Ffisegydd: Please chime in if I'm being vague/unspecific/wrong/unhelpful
 
well.. vectorized bigrams (training) vs vectorized raw text. And then evaluate and see what happened.
 
10:06 PM
by vectorized, do you mean feature-hashed?
 
Yes... or with Bag or words, or tdidf
 
@ml_guy: pick ONE feature set for the moment (it's usually inadvisable to mix feature sets, unless you really know what you're doing)
 
In fact i all ready do that, now i would like to do it with more texts.
 
what do you already do?
 
10:10 PM
vectorized bigrams (training) vs vectorized raw text
 
BAH. It's too late to be working out this Python stuff. I hate databases and OOP and abstracting things.
 
I really doubt you'll get scientifically usable results if you train on bigrams and test with unigrams. Your features MUST be consistent
 
i see... i didnt know that i will change the aproach.
Why do you said that?...Why features must be consistent?
 
a classifier learns a function. If you train it with bigrams, it learns a function that takes two inputs. If you then give it only one input, it won't know what to do. Does that make sense?
 
ooh.. I see
 
10:15 PM
I'm getting dinner, so my responses will be slower
So now, pick the feature that you want to use
 
Sure thanks... I thought it could be better if i use the other aproach., as i said before im just learning.
 
what feature have you picked? Bigrams?
 
Im only 21
adjective and adverbs
 
'sall good. Just chat in complete, somewhat grammatically correct sentences
what do you mean "adjectives and adverbs"?
 
i catched all the ocurrences of that words.
 
10:22 PM
So you have a matrix where the (i,j)th entry is the number of occurrences of word i in document j, where word i is an adjective or an adverb. Correct?
 
Yes
 
This is a type of tf-idf approach. This is not a bigrams approach. Understood?
 
Ohh.... so a better way of vectorize the features is tf-idf?
 
you're already doing that by counting the occurrences
it's not necessarily a better way. It's simply another way; and it's the way you've chosen right now
 
I tried with the hash trick since i read about it and it interested me..
 
10:26 PM
Forget that for now. That's just a way of representing your data. It is also used in the example we're discussing right now
 
ermehgerd I've actually got some working code.
 
bring it on @Ffisegydd!
 
Am working on some database wrapper stuff for Nidaba
I can now query our DB by id
 
awesome!
@ml_guy: do you understand everything I've said so far?
 
Yes
thank you
 
10:28 PM
now, do you already have a list of all the adjectives and adverbs for each file in both your training set and your testing set?
 
hold on i will show you guys a little example.
pastebin.com/M8Fzr9v9
 
while we wait, allow me to play some jeopardy music
 
this is in a nutshel what I'm doing
 
why have I been unable to see pastebin links for the past few weeks?
@ml_guy: I am unable to open the pastebin link. Please paint me a word picture
 
huh
are you using firefox?
 
10:36 PM
@inspectorG4dget lol
 
I'm using chrome on a mac
@davidism I know I'm asking for it
 
@ml_guy you forgot to put http:// at the beginning of the url
 
I tried copy-pasting the url - nogo
 
10:40 PM
wow, you know what, that sentence was not English at all. Let's try it again.
I have implemented this Kesh code in a manner that I think matches what Jon suggested last week
 
alright all, I think I'm going to call it a night. It's past 4AM
Rhubarb all
 
rbrb dude
 
11:03 PM
hey I'm back! cbg all
 
yo, did you see the example?
 
pastebin not loading on my machine. Do you think you could manage an explanation (or paste the code here if it isn't too long)
 
from sklearn.feature_extraction.text import FeatureHasher
from sklearn.svm import SVC
from sklearn.cross_validation import train_test_split
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

training_data = [[('this', 'is'), ('a', 'opinion'),('positive', 'bla'), 'POS'],
[('bla', 'bla'), ('bad', 'opinion'), 'NEG'],
[('and', 'one'), ('neutral', 'bla'), 'NEU']]

feature_hasher_vect = FeatureHasher(input_type ='string')

X = feature_hasher_vect.transform(((' '.join(x) for x in sample) for sample in training_data))
 
First, we discussed that you wanted all adjectives and adverbs of a document as your features. Therefore, it does not make sense that your feature vector is a list of bigrams
 
well... assuming that training_data are the featores we discuss.
it's an example of what I¡m doing.
 
11:18 PM
Your example is of an idea that we have not discussed. As a result, I am unclear about what you are trying to do, and can therefore not offer any helpful feedback about your code
So, why are you showing us this code?
 
you told me
 
You and I discussed what you are trying to do, which you just said is not what the above code tries to do. So, you need to be very clear on what you are trying to achieve with the above code, so that I can comment on it (and no, it is not what we discussed)
 
I'm trying to do text classification. And the vectorized bigrams vs vectorized raw text aproach. You told me that i need to preserve the feature same feature space.
 
are you trying to determine whether counting word occurrences is a better feature than using bigrams (or vice versa)?
 
No im just learning how to use the diferent vectorization aproaches that scikit learn provide.
e.g. count vect, hash trick, tdidf, etc
 
11:23 PM
ahh, Ok. In that case, all you need to do is use the vecotorizer, and look at the output of the vectorization. That should teach you how to use each approach. As long as your code doesn't error, you're fine
 
But then i will have more questions, for example how can i get better accuracy with the different vectorization algorithms?. And the other issue about bigrams vs raw text.
Am I doing the feature selection right?... stuff like that.
 
your first question is "how can i get better accuracy with the different vectorization algorithms?". The answer is that it's experimental. it really depends on the degree of difference between the corpora you are using. It depends on whether you want a 1 vs 1 classifier or 1 vs many classifier. It depends on the language of each corpus (to some extent)
So, there's no ONE rule governing which vectorization algorithm is the best, overall
 
Thanks for the feedback guys
 
Your second question is "about bigrams vs raw text". The question is better phrased "what should my features be? raw text, or biggrams?". The answer is again, that it's experimental. If you're trying to detect law vs art history, a simple bag of words (raw text) might do. If you're trying to detect authorship between similar authors, you're better off with a bigram model
@ml_guy any other questions?
 
No thank you very much, im reading you.
 
11:31 PM
you're welcome
 
Hold on, what about sentiment what aproach you recommend? (i.e. polarity in an opinion)
 
sentiment? what do you mean?
@ml_guy: you there?
 
yes
I meant polarity in an opinion
 
ahh. What do you want to do with sentiment analysis?
cbg @MartijnPieters @jasonjonesutah
 
Im interested in different problems of text classification(sentiment, authorship, similarity)
well i mean i started reading about that.
 
11:38 PM
it's interesting you should ask, because I did a project on this once, so I know a little bit about this
Sentiment analysis is easily done by counting unigrams
You could also do it by counting bigrams
 
but bigrams will be counted as a unigram?
 
classifiers are a bit of an overkill, but can also be trained for sentiment analysis
bigrams cannot be counted as unigrams. Bigrams are tuples of words (i.e. (word1, word2)). Unigrams are singleton words (i.e. word1)
@ml_guy: My laptop's battery is at 20%. When it hits 15%, I'm going to call it a night and go to sleep (it's 5.11 AM now). So if you have any other questions, post them quickly
 
Thank you very much. I will think everything you told me
il will back
thanks guys
I will try to read more
 
Hang on, I might be able to point you to some papers of interest
 
please
 
11:46 PM
this is the paper I based my project on. It's pretty good for sentiment analysis and has some good references that you should be able to read also
and now, good night!
Rhubarb all
 
rbrb
 
hey @davidism: do I get a medal or what? lol
 
lol, you get a virtual high five
 
lol high five!
 
o/
 
11:48 PM
'\o
 
Thank you very much guys
 
night!
 
see you
 
01:00 - 21:0021:00 - 00:00

« first day (1527 days earlier)      last day (3424 days later) »