« first day (1528 days earlier)      last day (3411 days later) » 

9:01 PM
As i said before documents on the .txt folder look like this:('text', 'text'),...,('text', 'text'),'LABEL'
 
Yep. So it's going to return a tuple. And the final value will be 'LABEL'
 
with that how can i build the y that fit the tags?..
 
Ok so you've used load_files but what is your data? Is it in a list?
 
is not in a list... is in a directory...previously i have it in a list
now i'm loading it since i would like to escalate it
 
It's in a directory? Wat?
 
9:05 PM
the documents(train/test) are in separete folders..
 
How do you use the load_files function in your code at the moment?
 
wanna se the code?
 
In a pastebin, yeah ok.
 
hold on
 
I should have been a dentist.
 
Programmer isn't my first choice of profession, but my letter from Hogwarts hasn't arrived yet
War's over guys, get your paperwork in order
 
If you see... it's prediciting wrong
 
Actually I guess they don't accept American students. What's the US equivalent? Salem institute? Does anyone find it strange that there's only one wizarding school per country?
 
It has some comments on spanish....
 
@ml_guy who wrote the code for you? Or did you write it yourself?
 
9:10 PM
i do it
 
I don't understand why you've done a lot of it in certain ways.
 
I started it with little examples
 
Why join your bigrams together as strings? Why not just stick them as tuples of strings?
 
Mmm in fact i have the same question because they are not bigrams i guess they converted in unigrams
 
It might still work, not sure.
 
9:13 PM
I'm just watching what happen...
i would like to learn to use scikit learn and i decided to start the second task
 
Can you print training_corpus and show what it looks like?
 
Wait nvm it's ok
 
it looks as i told you...
well hold on
i'm in to different computers
 
@ml_guy no really it's fine. In any case, I think this will get the labels from the ends of the lines
training_labels = [x[-1] for x in load_files(raw_bigram_opions_folder, '*.txt')]
 
9:17 PM
Did you like the code?
 
I just caught up on the conversation that I missed - had to run to mom's side for some stuff
 
i will upload it to github once is finished... for everybody
 
It's not very readable, you should look up PEP8.
 
im new with python
and with scikit
 
9:19 PM
@inspectorG4dget yeah your address and phone number are on that
 
in fact
i have another question to inspector
since he's a phd from utoronto...
 
@Ffisegydd: I thought I busted that a long time ago. Fixing now
@ml_guy: Read the CV. My PhD is not from utoronto, though my homepage is
 
I was reading about deep learning could i use a RBM to increase the accuracy of a classifier for text classification?
 
"could I use a restricted boltzmann machine to improve the accuracy of my classifier?". Yes, if you're using a Naive Bayes classifier, almost definitely
 
ohh
 
9:22 PM
It really depends on language and the classification task at hand
 
with scikit?
 
... and the training and testing sets
When I say "language", I don't mean programming language. I mean the language in which the texts are written. How rich/complex the vocabulary is, etc
If in English, as it seems to be in your case, then I can almost guarantee an improvement from NB. However, that is still quite dependent on what the two classes are
I seem to recall explaining this to you about 24 hours ago
 
@Ffisegydd i tried with your solution and this happens: ValueError:cannot set an array element with a sequence
 
@ml_guy what's the full error message?
 
i understad you were not talking about the programming language
but scikit implementations are restrictly that's why i ask you with scikit
hold on
 
9:26 PM
@Ffisegydd: thanks for the heads-up. My CV's been updated
 
im moving the code to this computer
 
No problem, I re-did mine the other day. Let's see if I can find it.
 
@ml_guy has multiple computers. Sigh! #I'mAPoorKid
 
Stupid SO uploading as png, just a sec
 
what the fuck
in this computer it works...
0_0
 
9:32 PM
the proper way to swear in this room is "yam"
 
in fact i tried this: [x[-1]for x in load_files(raw_bigram_opions_folder,'*.txt')]
 
"yam" and "tomato" are bean words, not to be used in front of anyone's mother
 
but it printed the path of the folder @Ffisegydd
with spaces
now the problem is the following... I guess it's by the low quality of the features I'm using : the prediction is wrong
I wil try to do the RBM stuff @inspectorG4dget
as an exercise
 
alright @ml_guy. Let me ask you this: You have examples labeled POS (positive) and NEG (negative). What are the positive examples, examples of? What class do they actually belong to (law documents? documents written by John Grisham?)?
 
they are shitty examples i writed
lol
im just learning
 
9:38 PM
Did you just come up with random words and decide to write them into files?
 
yes
 
then no classifier is going to work. There's your problem
 
yes
low quality
 
You need specific corpora. Classifiers WILL NEVER learn the random function
 
i will do that inspector
 
9:39 PM
so your first task is to come up with better documents
 
i will crawl a corpora
but with little examples it was fine
 
PSA: "corpora" is the plural of "corpus". It is therefore impossible to crawl a corpora
 
ohhh i see
Could you guys explain me this: "What are the positive examples, examples of? What class do they actually belong to (law documents? documents written by John Grisham?)?"
 
if you were fine with little examples, I'm almost positive that there was some sample bias, and you therefore got VERY lucky. I wouldn't count on such luck lasting for bigger examples
@ml_guy: forget that for now. Get two distinct corpora first. That question has been somewhat answered by the fact that your texts are random
 
yes... as i said before i was just working with little lists of bigrams, in fact it has nice accuracy
Ok... guys i will get a corpus and see what happen
 
9:42 PM
It shouldn't. The only reason it did was because your SMALL samples were far too biased. This is a huge deal in ML
 
By now im interested in print some metrics...
just to see how they work
 
NO! Go get a proper dataset first. Then we'll talk about training and printing metrics
 
ok
I found one called tass
it's from twitter
 
@ml_guy: if you don't mind me asking, where do you study? Which university?
 
Why do you wanna know?
 
9:45 PM
You can get example data sets from scikit-learn
 
just curious, that's all. You don't have to answer at all, if you don't want to
 
i have no problem in say that... but i just dont want to burn the "reputation" of my university.. lol since i'm not a good student
 
nothing like that. I'd have looked at your machine learning course's curriculum, to be able to better explain stuff to you
 
Well In fact i never took a course of machine learning or nlp only data mining...but that was a very theorical course
Im learning here
 
very similar. Anyway, I'm off to dinner, See you in a bit
rbrb
 
9:49 PM
I've never done any courses either, am just learning by doing.
 
i'm from mexico
guys
thanks i will work more on this and i will let you know guys
 
once is finished i will upload it to github
 
cbg
@Ffisegydd was having a player earlier, what's the criteria for data being present in kesh?
 
@Jon what do you mean?
For now I've only done a Question object.
But tomorrow I might subclass that with Post and move its methods into Post as applicable.
 
10:01 PM
oh... yeah... was somewhat ambiguous there... ie, is there a criteria on tags or something when you did the upload?
 
Then it'll be easy to make Answer and Comment
Ah. Just a second.
My DB creation scripts are a royal mess but github.com/sopython/kesh/blob/master/kesh/_database/creation/… has a set of tags. At least one of the tags must be present in a question.
python_tags = {'python', 'python-2.7', 'python-3.x', 'python-2.x', 'python-3.3', 'python-2.6', 'python-3.4', 'python-2.5', 'python-3.2', 'python-2.4', }
 
ahh... that'd explain the counts then - that's fine
 
I didn't go into "Well matplotlib is a Python lib but if the Q only has matplotlib but no Python then we should add it anyway"
Because that would be a mess getting all the various tags.
 
It wouldn't be a mess - as it could be uploaded, but just ignored :)
 
The issue is that we also need all the answers and post histories.
It's the histories that are the biggest.
But yeah, it's something we could think about adding later.
 
10:05 PM
true, but that's why we have that server size
 
I don't know if you've looked at the api stuff I've put together. It's pretty rough for now.
But it allowed me to actually begin doing a bit of exploratory analysis.
 
'fraid not - got an angry French man yelling at me... not quite sure if he doesn't yell in normal speech - difficult to tell
 
What is kesh? What are you guys building?
 
cbg @bigblind
 
Kesh is a database api thing that allows us to access Stack Overflow data for doing data analysis
 
10:07 PM
@bigblind the ultimate goal being sopython.com/pages/nidaba
 
What kind of data analisys do you do?
 
@ml_guy we're doing a machine learning project, detailed in the link Jon posted.
 
^ perfectly clear
 
Nice...
you are using scikit learn
 
I'm planning on doing some more flow charts actually :D
@ml_guy possibly. Probably actually. At least for bits.
 
10:10 PM
Weka or scikit?
 
scikit-learn or vowpal wabbit. Or both.
 
Kesh is the datbase and communication between our other projects, such as our chatbot, dupe analysis, and website.
 
Why no weka?..
 
Because this is the Python room.
 
Blame @Ffisegydd for Kesh as a name cough
 
10:11 PM
why weka?
 
Weka is Java. If I wrote it in Weka, I'd have to learn Java :P
 
although - you can't say we discriminate against project names - they're at the least interesting :p
 
I would like to prove why scikit is a better option against weka
 
They're different libraries and while they both do ML, Python and Java are different beasts and have different uses.
We use Python because we're a Python community and because Python is clearly superior to Java.
 
I'm back. cbg all!
 
10:16 PM
I found an interesting discussion:quora.com/…
How does Scikit Learn compare to Weka ( in terms of Speed, convenience and power )?
 
Note: there is a pyweka (Python API to weka), which I tried using once, and immediately regretted
 
its a wrapper?
 
It's that plastic thing that chocolates are wrapped in
 
Ohh why?
 
A wrapper is something that wraps around something else. So if you have a function: def f(x): return x+1
 
10:21 PM
@ml_guy please don't start the "which ml library is better" discussion again, it ended poorly last time
 
Sorry... but i had a teacher that was clunged with weka
 
clunged is not a word
 
and you want to write a function g that wraps around f, takes two arguments, adds them up, adds one and prints the result, returning nothing, then you could write g like this:
def g(x,y):
    tot = x + f(y)
    print(tot)
therefore, g is a wrapper around f
 
amm clung? @davidism
I see @inspectorG4dget
 
I have no idea what you're saying.
 
10:24 PM
Your teacher clung to Weka. As in "cling".
 
ah, ok, "was attached to" is probably a better phrase
 
"clinged to" or "clung to"...
 
yes, clung is a word, I get that, but "clunged" is not
 
well this guy is sticked to weka
i couldnt convice him why scikit learn is a better tool
 
10:27 PM
that'd be "is stuck on" :p
 
It isn't necessarily better.
 
good old English language... lucky it's my 1st language else I'd be screwed
 
why? @Ffisegydd
 
We have had this discussion before.
 
huh... thabks guys i will go to eat something..
thabks for your advices
thanks*
 
see you
i will check that @Ffisegydd
 
It was for everyone in general
 
thanks
see you guys
 
rhubarb @ml_guy
so.. @Ffisegydd, @JonClements, @davidism, what's new with you guys?
 
@inspectorG4dget not much really mate. Just having a snowball and watching QI.
 
10:44 PM
@inspectorG4dget just trying to work out what the hell to do when I have a few days off :)
 
Having a snowball? As in, the Hostess confection? A mixed drink? A literal ball of snow?
 
The mixed drink.
 
I'm on vacation for 2 weeks, going sailing in the Caribbean in a few days.
 
@Ffisegydd oh really? Not the "literal ball of snow"? :p
@davidism yes, yes... rub it in :p
@davidism mind you - was reading up on US politics... the rules appear to be somewhat relaxed on Cuba now
wb @tristan
 
user559633
thanks @JonClements :)
 
10:55 PM
Good to hear!
@davidism: Cancun is quite lovely if you've never been
 
What happens the second time?
@Ffisegydd Hadn't heard of that one before. Something kind of like egg nog, plus something lemony? Hmm... sounds odd.
 
@inspectorG4dget I quite liked Martinique
 
I've never been, but I like what Google is showing me
 
user559633
Related thoughts or have you googled another thing entirely?
 
Anyone seen walle cyrill ?
 
11:01 PM
rephrase: I like what Google is showing me about Martinique
 
user559633
alright, well, i'm off. take care all
 
@tristan rbrb then?
 
user559633
heading home :) i'll be on tomorrow
 
I have no idea what this user is really trying to do: stackoverflow.com/questions/27611216/…
 
@tristan I hope so... we need our MTFL present!
 
11:06 PM
Rhubarb @tristan
 
@inspectorG4dget was there for 2 days I think - but that was what seems a life time ago :(
 
awww @JonClements. Do you need a hug?
FWIW, my books are in the other room, and my data is in the other directory. Yet with all that's happening around me, they all seem worlds away
 
awww - didn't need a hug myself, but massive hugs back at ya for what you're going through with ya mum mate
 
"If there was a bug, yo I'll squash it. Check out my code why the debugger renders it!"
Thank you Jon. I'm super exhausted. I haven't dealt with much non-work stress before, and I'm not quite sure how to handle it
 
just be you and follow your heart
"just" isn't the right word there... but I'll stick by it, because whatever, it'll end up defining you as a person
 
11:20 PM
Is CPython going to garbage collect the old me, then? :P
 
nah, you get an additional reference counter and end up being stronger :p
 
I like this!
 
Wow - nearly Tuesday - time's running away
 
It's 5AM Tuesday here
 
playing some music set to random... listening to a great track
 
11:31 PM
which one?
 
one sec - see if I can find you an online version
 
oh! that's a good song, indeed
 
@inspectorG4dget probably biased.. .but a huge fan of Billy Joel, have every album, single, LP that's ever been realised and acquired some unrealised stuff...blah blah :p
 
ahh. I'm not THAT much of a fan, but I do like some of his music. I guess I'm a casual listener
 
@inspectorG4dget his earlier work is fantastic... his later stuff - not great
 
11:46 PM
@JonClements it's quite relevant to me right now. I like it
this has to be my favorite of his songs, though
 
The Stranger is a great album. I remember listening to it a lot when I was a kid, before I had a CD player. My choices were the radio, or my mom's LPs...
Fortunately she had good taste in music. IMO, at least.
 
@inspectorG4dget classic... let me find some earlier of his work - you might just appreciate it
 
@AirThomas: were you picked on in school for that reason? It happened to me for listening to Simon and Garfunkel
@JonClements Please, just not Uptown Girl
 
There were more than enough reasons for kids to pick on me in school at that age. I don't think musical tastes were discussed.
 
11:50 PM
you too, huh. I was the only one who had a working CD player with spare batteries on the bus, so they pretty much HAD to be nice to me then
 
In first and second grade I liked to wear tube socks pulled up to my knees at all times.
 
and an all time favourite: youtube.com/watch?v=sCiW161e_YI
 
Enough said, right?
 
@MartijnPieters that example sentence didn't sway you? ;-)
 
fair enough
I was the scrawny kid, who became one of the best endurance runners and swimmers by running away from classmates who made fun of my dreams of becoming a fighter pilot
@JonClements: that's a really good song - very telling of my mood at 3AM every night
 
11:55 PM
IIRC, Billy Joel is the only artist to have had #1 hits in 4 (or 5?) consecutive decades
 
well, I got the "Best Sportmanship" award at primary school? Does that count!? :p
 
I find that these days, my coding music seems to come from Amaranthe
 
I've probably misremembered the actual statistic.
 
@JonClements I vote yes
 
My working music lately has been... Hmm. let me consult Grooveshark instead of my memory, which is not legend...
Oh yeah, Squarepusher.
 
11:58 PM
I've been rocking Les Mis and Phantom Of The Opera.
 
Before that it was Telefon Tel Aviv, Jonathan Coulton, and a bunch of soundtrack music. In reverse chronological order.
 

« first day (1528 days earlier)      last day (3411 days later) »