« first day (1646 days earlier)      last day (3296 days later) » 

9:00 AM
which people often refer to when using constructs like try/except
 
@StephanKetterer it's a bit more than that.
 
yeah i am sure :) was just my first impression
 
Watch this presentation if you want to see what refactoring code to make it Pythonic can mean.
 
cbg @Martijn
 
@JonClements cbg
 
9:02 AM
@AvinashRaj I realised in my early 20s that I knew next to nothing about India, even though it has such a rich diverse and ancient culture (and 20% of the planet's population), so I decided to do something about it. Luckily, I was living in an area with quite a few people of Indian heritage (and a Vaishnava temple). Also, several of my friends of European heritage became interested in Vaishnavism, so it was fairly easy to learn & discuss stuff.
 
Martijn, what IDE do you use?
 
@PascalvKooten Sublime Text
 
Is there any automatic/magical refactoring for python anyway?
Since it is not strict :/
 
Do you mean formatting?
 
@thefourtheye Technically, ST isn't an IDE
 
9:04 AM
No I mean like, making a class out of a function or something like that
 
cbg
 
@HamZa True, its just an Editor. But what qualifies a software as an IDE?
 
@HamZa we could run python programs on SublimeText itself.
 
@thefourtheye As the name suggests
@AvinashRaj you mean because ST supports python (as in plugins)?
 
@Jon i can i show you how i tried to solve that problem that DictReader gives me just strings, and consequently i never get int as datatype
 
9:07 AM
i think it's not a full-fledged IDE.
 
@Stephan sure - if you want... I'm curious to see :)
 
I expect an IDE to contain (almost) everything what an editor has plus debugger and build tools
also intelligent code completion
 
@AvinashRaj Do you mean that you can extend Sublime with functionality from Python?
 
the indentations are wrong somehow again, but i did it right
 
I'm using Emacs and I just couldn't imagine a world without all that customisability... but if Sublime offers extensibility through Python... now that would change something
 
9:09 AM
I don't see why you would choose ST over notepad++ apart from the fancy colors
@PascalvKooten you could extend ST with py
 
now when i give the function something like "5" it returns me int :)
 
lunch time. bye. rhubarb..
 
Damn, is it weird I don't have a linkedin account?
 
It's all about PyCharm.
 
@Ffisegydd I like it but I don't have enough experience to gain the maximum out of it.
Got a student license btw
 
9:12 AM
I extended emacs to alllow using Control+2+Enter to send to python2, C+3+Enter to send to python3, and C+4+Enter to PyPy
and everything I execute automatically gets a timing
And Control+Enter is smart to eval smart parts of code, e.g. in the middel of a class it will eval the whole class
I don't think this would be possible with Sublime / PyCharm
 
Probably would be
 
@PascalvKooten mandatory reference xkcd.com/378
 
Butterfly reference? Or the learning curve one?
Heh, the butterfly one :P
 
Bah... not a good time for broadband to be mucking about... on mobile now...
 
is there an easy construct with try/except.. but with several tries?
that would be awesome :)
 
9:15 AM
wut?
 
You typically have one try and then multiple excepts
 
@PascalvKooten You don't have permission to access /blog/wp-content/uploads/2006/08/curves.jpg on this server.
 
@PascalvKooten forbidden
 
can there be normal code in excepts?
 
9:15 AM
Hmmm, it shows for me
couldnt find the xkcd ref
 
I used google image reverse search and I found it
 
How to post it in here?
 
168
A: What are the pros and cons of Vim and Emacs?

Michael MrozekI'll post what I think are the main benefits of each: Emacs has considerably more extensions to let you do tasks that are only vaguely text-editor related, like browsing the filesystem or messing with version control, and extensions that are in no way text-editor related, like reading RSS feed...

 
yea haha
Also, I have smart parentheses in Emacs, when I hit ( it can wrap around logically when pressing it again (with python language knowledge)
 
is that something that can be done : try:
return (type(int(value)))
except (type(float(value)))=="float"):
return (type(float(value)))
except ValueError:
return (type("bla"))
 
9:19 AM
@StephanKetterer hint: try it
 
Stephan, you would generally not use try...except for these things, you can just test using if...elif...else
 
i tried it
and it does not work
int works
but if i try 4.3 as value
i get str as return
 
btw, "float" (WITHOUT quotes) is a type
 
oh got it
and already corrected it
 
you can also do like type(1.3) == float
or isinstance(1.3, float)
 
9:21 AM
but my code is just getting worse and worse.. because it comes from a DictReader
so all the inputs are strings
 
Use the second
isinstance is the best choice
 
Yea, I know
I was just trying to transition from his type to a better form of type and then to the better way :P
 
sitting on it for a week now.. not moving along at all :)
 
It's fine with a dict reader... look at the gist I posted... :-D
 
Stephan, how is the tutorial getting along! :P?
Like I said, the problem you're trying to solve is not an easy one
When you haven't finished a tutorial
 
9:24 AM
good ok. this is going nowhere, whats the best tutorial for me to do right now ?
gonna start with code academy python course, its 13 hours it says
 
You don't have to do it all in one sitting
lol, you seem to be serious about python though
so it's for sure going to be worth it
do it until your bored with it, try your problem again, continue with tutorial etc
@Ffisegydd I was wondering if we can change BeautifulSoup to lxml in nidaba ?
 
will just try hard and finish the codeacaemdy thing in 2 days or so :)
hoefully
 
@StephanKetterer good luck :)
 
omg.. it has levels :) and awards..exactly the thing that motivates a degenerate gambler
 
umm... looks like I'm tethering for a little bit then...
(must remember to not load anything large!)
 
9:32 AM
@StephanKetterer Exactly, we're still stuck in python world getting our daily fix
 
9:45 AM
Cbg :)
 
Right! Finally home!
@Pascal I'm happy to change from bs4 to lxml if there is a good reason for it. If we were to change then the time would be now when we have a smaller code base that would need to be modified.
 
@Ffisegydd Yea I did a grep and there's barely any "soup"
The reason would be that lxml is going to be 10x faster. If we're talking 300k documents then that's easy gain.
 
yea, and in the test
 
9:53 AM
Ah yeah, of course.
Yeah I'm happy to change it up though if there's a big speed increase, as long as the usage isn't 100x more difficult.
 
Ahhh huh.... broad band doesn't work when the cable for some reason isn't plugged into the filter anymore... strange that
 
If people know a bit of xpath it's not going to be an issue at all
If we write things in lxml then people can just copy the logic and change it
 
I actually thought BS used lxml internally anyway
 
I did timings recently and while choosing lxml as backend for bs4 will be a bit faster, it's still way slower
('lxml', 7.070770025253296)
('soup lxml', 46.9744770526886)
('soup default', 71.4960401058197)
 
@PascalvKooten did you ever time using lxml.objectify as well?
 
9:59 AM
Is it much more difficult to use then? As, for instance, getting all the code in a question is just:
    def _get_code(self):
        """
        Extract code without markup tags from a given html content.
        :param html: String
        :return List of code strings in the given content
        """

        return [i.get_text() for i in self.soup.find_all('code')]
 
@JonClements Nope...
 
@Pascal because then you'd be closer to introducing the Python object overhead that BS introduces
 
[i.text_content() for i in self.tree.('.//code')]
self.tree.xpath('.//code')
 
@Pascal umm.... self.tree.xpath('.//code/text()') ?
 
lol Jon, yea you're right
 
10:01 AM
re-cbg
 
I'm just a bit worried about optimising at the expense of ease-to-usage. Especially as we don't know if HTML-parsing will be such a big time aspect.
 
@Ffisegydd Well, I don't know it either. I just know everything has to be parsed at least once
@Ffisegydd I'm still not sure how I should work with nidaba
 
Just a sec just finishing my lunch.
Okay, let me go over what the plan is and what we have so far then.
 
Yes please :)
 
I presume you've read sopython.com/pages/nidaba and so have an overview.
 
10:06 AM
@Ffisegydd I've never actually used lxml, but I have read the docs a couple of times. IMHO, it's pretty impressive, and quite Pythonic for something that's essentially a wrapper around the C libraries that do the bulk of the work. It's best used on clean XML or HTML - it doesn't cope well with malformed documents, but you can get it to call Beautiful Soup when you encounter broken stuff.
 
So for the moment we're trying to build the feature extraction code.
The way we've thought about doing it is such that we have Question, Answer, Comment, etc objects that we can apply functions to and return features.
 
Yea, I've read it.
 
Also, I guess the learning curve for effective use of lxml is less for people who are already familiar with xpath (which I'm not).
 
To help with the higher-level "object" functions we've been writing lower-level utility functions.
 
I'm porting my old bs4 code to lxml lately
 
10:09 AM
Once we've got some nice features we can actually worry about building models (and maybe have to build more features, etc)
 
Yea, I was just wondering how to connect to data, other than the test objects?
I'd like to connect perhaps a sample of 1000 of each or something
 
Getting the data together is a different project called Kesh.
 
Yea that's what I figured
Though I thought you want to have a short circle there no?
 
@Ffisegydd where's that diagram gone? :p
 
where you test is where you have the data
Do you load kesh next to nidaba?
 
10:13 AM
No Kesh is a different process.
Kesh will load new data from SO and pass it to a message broker.
Nidaba can listen to that message broker for the new content.
And then pass its results to a 2nd message broker.
And then anything else can listen to the 2nd.
Let me see if I can find the diagram.
 
still using the phone no?
 
Nah I'm home now.
 
consistent titlecase on the first word --> usually phone :D
 
So we read data from SO using websockets, using the Kesh.WS process. This data is passed to an Apache Kafka message broker.
Kesh.Process takes this message and gets any extra data we need from the SE API then passes it into another Kafka message broker.
Nidaba then takes that data to do it's magic.
After the magic, Nidaba passes its results to a different Kafka message broker, various things can listen to it from then on.
 
Right, because the end goal is to use the results from nidaba to influence SO
directly
 
10:19 AM
Well the end goal is that we can get recommendations for closing stuff.
So it'll influence SO through us.
 
Hmmm, yea
what is so great is here is that we basically have a labelled dataset here
 
Exactly. It's not perfect though because content is deleted.
So any old questions that have been deleted are lost.
 
Hmmm, they are not archived or something then
 
SO may still have a copy but they're not released in the data dump.
Of course we can keep track of content that is deleted from now on into the future.
 
Yea, we should start that immediately :)
I mean, it's so valuable as data
 
10:23 AM
That's actually what I've been working on recently. Been working on Kesh to sort out a postgresql database and sorting out the websockets so we constantly update the data.
Even if it takes us a while to sort out Nidaba we can still keep track of new data.
 
Yea, I agree
If we'd have the state of a question & everything else right before the delete action, then that's labelled
 
I'm going to be busy for the next week or so (at least) as I'm preparing for interviews, will still be around to discuss it (just won't be able to write much code)
 
Cool, yea I just feel like I'm missing a lot of information before getting started
but this has already cleared up some questions
 
Well ask away now if you want :P
I won't start my prep for today for another hour or so.
 
:D
Well, I don't see the connection yet in nidaba?
 
10:28 AM
What do you mean?
 
Particularly your comment of "
Nidaba then takes that data to do it's magic."
It cannot do that just yet right?
 
No.
And the internal logic may change.
That's just a generic idea for now.
 
user4183195
Ssup!
 
user4183195
im back
 
user4183195
so
 
10:29 AM
Well, I'd just assume that while writing things in nidaba, I'd keep real data close?
Or are you saying to just make testcases and work with those?
 
Yeah that's probably the best. Get some real data and develop with it.
 
I feel like that'd be just a fake feeling of comfort, since you wouldn't know how it works on 300k docs
If you'd have 1000 samples for each object, then you'd be more condifent
confident*
 
PS It's more like 400k docs now :P
 
hah okay :P
 
In fact, it's only 400k questions, not including all the answers :P
 
10:31 AM
wb @Switch :)
 
user4183195
is _Learn Python the Hard Way_s HTML version complete?
 
user4183195
I mean does it have everything that the author has written
 
@Pascal I'd estimate we have 1,000,000+ if you include questions and answers.
 
user4183195
or is it like a demo with only a sample of it
 
Not sure how much use Answers will be as features though.
 
user4183195
10:33 AM
@JonClements hai
 
As new content is unlikely (by definition) to have any answers.
 
I cannot wait to get scikit on that data :P
CountVectorizer for starters
 
user4183195
so?
 
user4183195
anybody?
 
@Ffisegydd Your goal is to close it asap then?
 
10:34 AM
@Switch don't use LPTHW.
 
user4183195
@Ffisegydd why not?
 
@Pascal yeah that's the plan.
@Switch I've heard various bad things about it.
 
Probably at first that the system will flag it for deletion or something?
 
@Ffisegydd Well... if users in good standing are answering, then it's likely the question isn't that bad kind of thing...
 
user4183195
@Ffisegydd like?
 
10:36 AM
bad things
 
I'm personally really interested in the duplicate question detection
 
user4183195
@Ffisegydd what about Python for Dummies?
 
But I think closing should also be possible
 
@Pascal yeah that's obviously a different problem (clustering rather than supervised learning)
To be honest we could start with either, as I think they will use similar features.
@Switch no idea, sorry. I don't read Python tuts. I just know LPTHW has a bad rep.
 
Yea, though indeed closing should be easier due to supervised
 
user4183195
10:38 AM
@Ffisegydd citation needed please =)
 
@Switch I have better things to do than find you a citation :) I gave you some friendly advice, take it or leave it.
 
@Ffisegydd In the end we imagine some kind of flat dataset right, to train/test with?
from there we get some kind of weights for each feature
which we can then use to classify questions real-time
Not sure how that fits into the structure of the Nidaba package?
 
@Switch I'm fairly sure it's complete. What you don't get is access to the PDF version, the video tutorials, or email access to the author for help. But I agree with Fizzy: avoid LPTHW. It may be popular, but it's not as good as the author thinks it is, judging from the nature of the questions we get on SO from new programmers who are working through LPTHW.
 
@Pascal that's the plan yes, though we haven't gotten that far in advance thinking about models or whatever.
 
Actual analysis is a really small step compared to the rest
But I think it's good to get a closed loop
regardless of the quality of the individual components
But that might just be me though
 
10:46 AM
Originally I figured it made sense to get feature extraction sorted first, then we could experiment with model building.
Once we do the model building we can use the full data set to get weighting etc.
 
Maybe I'm just impatient that I can't wait to test features
hehe
Though I have to say I'd prefer to have the data in my hands while thinking of features
 
Yeah I agree.
 
Ofcourse not all is needed
 
I've been using the data to try and think about features.
 
Using kesh then?
 
10:49 AM
That's a schema of the datadump/api data.
But then if I need to look at actual numbers, yeah just grabbed some random data from our database.
 
How did you do the random grabbing?
 
I asked people in chat if they knew of any decent questions with lots of different code and stuff :P
 
So I could test the html parsing mostly
 
I'd suggest we add a component in nidaba that can grab 1000 random objects of each type?
or probably predefined 1000 is even better
I'm still not really sure how you think I could contribute P
:P*
 
10:55 AM
I'd say the pre-defined is better. As by grabbing 1000 randomly then you'd end up getting confused between Kesh and Nidaba.
Contribute however you think you can mate.
 
And also, it's nice when people would work with the same 1000 examples
Ah cool
Well, unless you think it's not a good idea
 
No it makes sense.
 
I'd go with adding the example data, and use the existing features to make a training/test set
 
Not sure how you'd store it.
 
since we dont have clear labels
 
10:57 AM
We don't have enough features really.
 
probably just training for now lol
well, you're having hierarchical objects
I'd just look for a way to make a flat structure
 
What do you mean?
 
you extract things like
number of comments as attribute of e.g. a question
and number of comments as attribute of e.g. an answer to a question
in the training set you could have these next to each other for example
question_num_comments and question_answer1_num_comments
also question_title
question_bodyText
question_code
actually, you will probably instead have some kind of word counter object
but that's all not as important
at this point
I'd just try to consider the step from features to training set
@Ffisegydd Have to go soon. Nice talking to you. I'll contribute over the next couple of days :)
 
Yeah cool @Pascal. Hopefully once my interview stuff has quietened down we can sort some stuff out.
 
@Ffisegydd Yea, good luck with it!
See you all, bye
 
11:41 AM
I found a bug in the Python/C API, which is also a note to my future self:
> If you want to set ml_flags which represents a staticmethod, which also accepts keyword arguments, not only positional ones, you have to use METH_VARARGS | METH_KEYWORDS | METH_STATIC, instead of METH_KEYWORDS | METH_STATIC otherwise you will get: Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!
@AnttiHaapala && @QuestionC FYI ^
 
Does someone who knows about file buffering in Windows want to look at this: stackoverflow.com/q/29729082/4014959
 
12:03 PM
Hey up @Antti
 
12:22 PM
I got nerd sniped by MikhailTal's question from yesterday about extracting sublists from a string of ints, where each sublist is preceded by its length.
 
Is the following the wrong syntax for running a python script from a windows command line? python C:\Users\aaa\Desktop\test\hist.py
 
My best answer is
data = [int(u) for u in data.split()]
gen = iter(data)
ilist = [[next(gen) for _ in range(u)] for u in gen]
 
when I simply run: python hist.py when in the directory it works fine
 
I also tried ilist = [list(islice(gen, u)) for u in gen], but it's slower.
 
but when I do the above I get errors relating to the python script such as "image not found"
anyone have any ideas please?
 
12:28 PM
@SurajKhosla When you launch any program it inherits the current directory from the shell you launch it in.
 
so there is no simple way of doing sort of the following: python C:\Users\aaa\Desktop\test\hist.py
ie, calling it from another directory?
 
Sure, you can call it from another directory, but if the script contains relative file names it won't be able to find those files.
 
ahh ok, so if I explicitly define the directory path rather than "./" in my python script I should be ok?
 
So you need to either 1) use absolute file names, or 2) tell your script to change its current working directory, which you can do with os.chdir
 
hey hey
I wasn't online @Ffisegydd
@PeterVaro 2 or 3?
 
12:36 PM
3
 
wtf?
meth_oldargs == 0
you sure you did use | and not & :D
 
Hey. Is there anything else than "cleaner code" difference in these two?

a = lambda x: x*2

and

def b(x):
return x * 2
 
@AnttiHaapala r u serious? :P
 
I mean, METH_OLDARGS is exactly 0
 
and is there some way to format spaces correctly in chat room? :D
 
12:39 PM
I know.. still, try it for yourself, as I mentioned => it is a bug
 
I am thinking it is something else, UB?
which version
switch (PyCFunction_GET_FLAGS(func) & ~(METH_CLASS | METH_STATIC | METH_COEXIST)) {
case METH_VARARGS | METH_KEYWORDS:
it is there
the problem is the default is worded:
    default:
        PyErr_SetString(PyExc_SystemError, "Bad call flags in "
                        "PyCFunction_Call. METH_OLDARGS is no "
                        "longer supported!");
which is wrong
it should test for 0, and if not 0, then give "unknown flags"
 
@PM2Ring Thank you, works perfectly :)
 
Cool! FWIW, you might be able to get the full path used to start your script from sys.argv[0], but that depends on your OS. I'm not very familiar with Windows. But you can experiment. :) Put a print sys.argv[0] in your program and start it via python C:\Users\aaa\Desktop\test\hist.py and see what it prints.
 
@AnttiHaapala told you ;)
 
@makallio85 Yes: Use Ctrl-k
 
12:43 PM
@PeterVaro this is python 3
 
Or select the text and a Fixed Font button will appear.
 
@PM2Ring thanks
 
@AnttiHaapala so?
 
@PeterVaro I asked which version of python 3 you are using
 
And you can
enter multiple
lines by using Shift-Enter
 
12:44 PM
(I said I'm using the Python 3 C API)
 
there is no bug in the code
as far as I can see
this is hg tip
 
8 mins ago, by Peter Varo
3
 
seriously?!?
I asked which version of Python 3.
 
ahh... 3.4.3
 
12:46 PM
thanks!
 
will you report it?
 
I need to reread the Python 3 C-api documentation
 
all righty
 
Should this question be edited to add pandas to the title, question body and tags? Or should it just die the death it deserves? :)
 
12:50 PM
apparently google appengine still does not support python 3
 
@PeterVaro you might be right
 
@AnttiHaapala as always ;)
 
METH lol
 
the docs are really bad I guess
here
@PeterVaro added comment
@PeterVaro too late for 3.5 I guess :D:D:d
 
12:59 PM
</thumbs-up>
 
anw that is shit
 
So stackoverflow chats are licensed under CC-BY-SA-2.5?
 
@AnttiHaapala I have all the time, I can wait, I already added a /* BUG */ tagged comment in my code
 

« first day (1646 days earlier)      last day (3296 days later) »