« first day (1836 days earlier)      last day (3113 days later) » 

8:01 PM
@DSM I forgot about that because I never use yield's return value unless I'm doing weird coroutine tricks.
 
I'm dealing with ~90GB of event data, maybe 150million json objects. The internet tells me it's not enough data for spark/hadoop, but the execution times of these queries tells me Postgres is not ideal - Has anyone had success with getting postgres to query tables this large in minutes and not hours?
 
mmm... facebook earthquake safety reports are pretty nice...
but I am like "wtf is my friend doing in Afghanistan"
 
DSM
I think it was the throw/send stuff which won the day for yield from.
 
(and cbg)
also hating: timezones
 
Hmm, looking at that code again, I see a logic error... iter_ways(11,1) should have no results, but it yields [11] instead.
 
8:03 PM
also hating: points in time...
think I will be storing ISO 8601 in my db... :D
 
@Kevin: so the return after yield... what is its purpose?
 
@rp372 what are you querying?
 
is that because python moves on to the next line after it has exhausted the yield?
 
@rp372 you need to create appropriate indexes
 
DSM
@inspectorG4dget: was that aimed at Kevin?
 
8:06 PM
Not that it made a difference in my answer, since the incorrectly yielded values would get filtered away by valid anyway, but I'll fix it anyway
 
damn! yes. I always confuse your two DPs
 
@inspectorG4dget just look at the starboard side man
or strikethrough :D
 
eh? me no understand
 
as it seems kevin is not there
 
@inspectorG4dget If you're not a fan of multiple points of return, it works just as well if you do
def iter_ways(x, nums):
    if x == 0:
        pass
    elif nums == 1:
        if 1 <= x<= 9:
            yield [x]
    else:
        for i in range(1,10):
            if i <= x:
                for rest in iter_ways(x-i, nums-1):
                    yield [i] + rest
It's important that the for loop not execute if either of the first conditionals are met
 
8:09 PM
it's not that I dislike multiple points of return. I seem to recall something about tossing in multiple sequential yield statements in the same scope (if block, etc)
 
@AnttiHaapala The queries are pretty varied, I'm trying to use jsonb and GIN indexes because the schema of the JSON isn't very stable, so unpacking it to columns is tough
 
We want a clean separation between the base cases ("there are zero ways to do this", "there are either zero or one ways to do this") and the recursive case ("there are possibly very many ways to do this")
 
so the indexes have to be very specific
 
is it important that the for loop not execute for the sake of the logic of the program, or because of the way yield works?
 
you can also do functional partial indexes if possible
@rp372 also you are using JSONB are you?
 
8:11 PM
@inspectorG4dget For the logic of the program. Python would happily do it, but then the answer wouldn't be correct.
 
@AnttiHaapala Aye, using jsonb
 
SQLAlchemy?
 
ahh, I get it. But isn't there also the case where you could do something like this:
def myfunc(inputs):
    # code
    if condition:
        yield from mygen()
        yield from myOtherGen()
 
Sure. You might see something like that in a tree iterator.
def infix_iter(self):
    yield self.value
    if self.left: yield from self.left.infix_iter()
    if self.right: yield from self.right.infix_iter()
 
@AnttiHaapala What does SQLAlchemy do that helps with large datasets?
 
8:14 PM
it doesn't, it just makes SQL slightly more bearable :D
 
well I do like that
the main place I'm running into time constraints is generating session ids, which requires window functions and aggregations on a big table - and I don't know how to index to help that out
 
ahh. That clears up some things
I think I get it now. Thanks a lot @Kevin
 
generating session ids requires window functions and aggregations?
haha I guess I am lost already :D
 
Here is the official documentation for yield expressions, although it's a bit cryptic
I imagine the linked PEPs might be more comprehendible
 
oh cool! thank you :)
 
8:20 PM
This is going to sound weird, but if I have a dictionary and a key, can I get the key as it is in the dictionary?
 
wat?
 
Now I'm interested myself in seeing an example where x = yield from something() would be used
 
I have a custom object that only checks equality of certain attributes.
Hang on, I'll draw up a quick example.
 
@AnttiHaapala Maybe it doesn't necessarily - I say two events are part of the same session if they happen within 30 seconds of each other
so I'm using lag() to determine that
 
Do I buy a Blizzcon virtual ticket? :/
 
8:23 PM
I have something like this:
class Foo:
    def __init__(self, x, y=0, z=0):
        self.x = x
        self.y = y
        self.z = z

    def __hash__(self):
        return hash(self.x)

    def __eq__(self, other):
        return self.x == other.x


foo_dict = {Foo(1, 2, 3): 'bar'}
foo_item = foo_dict[Foo(1)]
 
I was going to say hell yes, but then the edit...
 
And I want to get the y and z of the Foo object that's in the dict.
 
Interesting. print((yield from troz(x))) is legal syntax, but print(yield from troz(x)) isn't.
 
I suppose I could just loop through the dict, but I was wondering if there was another way.
 
Might come in handy during a trivia challenge. "Name a Python expression where thing((other_thing)) is legal and thing(other_thing) is not"
 
8:32 PM
@MorganThrapp do a FooCapturing object that will store the other if it compares equal to self :D
 
@AnttiHaapala Oooo, that might just work.
 
it is the ugliest solution that I could think of
 
I ended up just doing a loop. :P It seemed not too bad.
 
i wanted to install a package from the branch through pip
But whenever I get the clone link, it gives me github.com/itdxer/neupy.git
Not sure this would install from the branch github.com/itdxer/neupy/tree/bug-27-oja-mem-leakage
 
pip install https://github.com/itdxer/neupy/tarball/bug-27-oja-mem-leakage
use the generated archive for the branch, rather than trying to clone and checkout the repo
If you want to use git, see the docs: pip.pypa.io/en/latest/reference/pip_install/#git
 
8:50 PM
Is there an easy way to pad a list to a certain length? Eg, I have a list that I need to always have 4 elements, but I don't care what the data in that list is.
 
Thanks davidism!!!
 
Does yield from.. work in Python2.7?
 
Kind like zip, but one dimensional.
 
@MorganThrapp (my_data + [None] * max_len)[:max_len]
 
@davidism Sweet, thanks.
 
8:52 PM
@Augusta no, it was implemented in 3.3
 
Aw.
Thanks
...if I'm supposed to ignore air resistance, and I supposed to ignore the fact that the plane has no practical way of actually flying as well? =_=;
 
@Augusta Are you ignoring gravity as well?
 
I don't remember being told to.
 
Better go ahead and do that, and assume a spherical plane to be safe
 
I mean, it's possible that the plane is aimed upwards and that its engines' thrust is greater than its mass.
 
9:00 PM
lack of air resistance doesn't mean the plane can't fly
 
Assume a spherical plane in a vacuum.
 
@tzaman I would assume that the air would move out from under the wings, rather than accumulate the pressure needed to support it.
 
I suppose if it has rocket engines... Jet/props won't work too well sans air resistance
 
frictionless air molecules being redirected by the wings will still impart force
 
So the wing would be a scoop instead of a foil?
@rp372 I was thinking that the force of the engine exhaust would drive the plane upwards, but now that you mention it, any engine that uses a fan or a turbine would be useless.
It would just turn into a metal jacket full of exploding gas.
...Maybe I was happier with the phasing sphere plane. -_-
 
9:06 PM
I say include air resistance
non-linear differentials are more fun anyway
 
Maybe this was actually some kind of subversive exercise designed to keep my imagination in good shape. If I view the question that way, it's much more satisfying. As well as easier to answer.
 
If you ignore drag, you just have Kerbal physics, which ain't too bad.
Unless they added drag while I wasn't looking.
 
They must have drag because bits can be ripped off if they aren't attached sufficiently.
 
I hope they have drag, I'm using Kerbal to prototype my home rocket
 
I think Kerbal has drag, because ships behave differently when depending on their interface angles, etc..
It's just not a super-sophisticated model.
 
9:11 PM
I think things falling off Kerbal rockets is just caused by structural forces.
And it certainly does have lift.
 
Oh wait no, Kerbal certainly does have drag of -some- persuasion, because it has parachutes. Parachutes which are useless in over-thin/under-dense atmospheres.
The parachutes function regardless of whether or not they clip into and overlap with each other, mind you, but still. Drag is drag. (Even when it isn't.)
 
It has drag of a sort. Simplistic but at least it scales with v**2.
 
DSM
I'm having unpleasant flashbacks to having to consult fluid dynamics tables to estimate the drag on different types of planetesimal in different regimes.
 
I never did fluid dynamics (thankfully)
 
I'm seeing they added drag, but it's relatively new.
Aka: People complaining about drag all of a sudden in May
 
9:27 PM
Yeah, they came up with a new aerodynamics model that included heat accumulation from atmospheric friction.
And all of a sudden, a lot of spaceships couldn't come back from space. <3
"S'all right, as long as I keep the narrow-end forward, everyone'll be fi-- WOAH WHAT EVERYTHING IS EXPLODING I AM IN A WORLD OF WIND AND DEATH WHAT NO NO NO"
And that is my tale.
They added customizable fairings, and that should have been a big clue. Oh well.
 
ooh are we talking about KSP?
I love KSP
 
Air
9:45 PM
Oh no, fluid dynamics D:
I suddenly regret rejoining the room
Oh wait, I remember why I'm here - I was going to tell @Kevin that I'm a few dozen pages into Blindsight and digging it.
 
10:12 PM
 
Wow, that got del-hammered quick (or removed by questioner?)
 
quick question: how to use slice in a 'foreach'? I.e.: for i, d in enumerate(list) like in: for i in list[:2]
 
I don't understand the question
Could be an XY problem with a sprinkling of PHP thrown in. I might be allergic.
 
for i in lst[:2] is perfectly OK, what's the issue?
it does make an extra copy of the list
you can use itertools.islice if you want to avoid that
 
just to know if is possible =D itertools.islice did the trick, thanks!
 
10:27 PM
I often answer those with highly cryptic, but correct "there's no way a student wrote that" answers. Then I realize that others might come to that question in the course of researching their problem, and my answer would be totally useless. So I leave my "useless" answer in the comments
 
@inspectorG4dget Context is everything or nothing. If you're referring to my recent comment, it's about the question immediately above it here, not anything on the main site :-)
 
it's somewhat related to the question you referred to just now. I forgot to linkback to your comment
 
Ah. The question on the main site got del-hammered so quickly I didn't even see it, so my subsequent comment was on the chat question.
 
DSM
10:42 PM
@PatrickMaupin: self-deleted, FYI.
 
That makes more sense.
 
Ugh, pygame. Why are your versions such mush? =_=;
 
I prefer stories where someone is smited, though. (And my wife prefers stories where someone is smitten.)
 
What about stories where someone is smelt?
 
Don't have any small children any more.
 
10:44 PM
@Kevin's list.append(Synecdoche, New York) // I want to hear your interpretation on this one
 
I see.
 
Grandchildren of the correct age may show up eventually, but I'd turn blue if I held my breath.
 
Alternatively, consider that scene from Terminator 2(?) where the villain falls into a pool of molten steel.
 
Yeah, I like things like that. And I know that's the correct usage. But "He who smelt it dealt it" is one of those things that probably hangs on into the most advanced stages of Alzheimer's.
 
Even when my memory deteriorates into nothingness, and my cognitive abilities rival that of krill, the thought and dream of your manhandling of my native tongue will live on. <3
"Reading, 'Riting, 'Rithmatic."
 
10:54 PM
Unfortunately, these days I think it's "Standardized tests and CS."
Where CS == some really watered down non-CS crap that may or may not help with an entry-level IT job.
 
just used re.sub with a repl function for the first time
 
After fiddling with pygame for a day and a half, the memory leak seems to be solved, and my framerate is... halved. I don't even know what order I installed the updates in anymore, and the version isn't being reported accurately.
I guess this is the point where I should pull everything out and reinstall it. -_-
 
That doesn't sound happy.
 
I'm in hell.
 
My hell is purely physical -- still recovering from a cold.
 
10:58 PM
the call it STEM now
 
Pygame's website is a nightmare. The "most recent download" points to their FTP, which offers 1.9.2pre, which is an older and less functional file than 1.9.1release, which is newer and corrects a B-list bug that I needed it to, but is missing parts.
There is a newer 1.9.2a0 on their Bitbucket thing, but it's kind of bits and parts and I don't think any of them are wholly complete.
 
It seems like a cool project, but somebody needs to take the lead, I guess.
@JGreenwell Well, STEM needs CS people, just like architecture needs carpenters and bricklayers.
 
That's the thing, their lead seems like a pretty decent guy, but I have no idea whether there is a proper release newer than 2012.
 
Science, Tech, Engineering, Math - no reading/writing at all anymore and the focus is more on Java topics then actual useful CS/IT learning more often then not (well from a developer's perspective)
you beat me by a second Patrick
 
(And, of course, by "CS people" I mean "people learning what passes for CS these days", not people who really understand CS.)
@Augusta So is it a lack of a binary Windows release, or a lack of anything that will build on any architecture?
 
11:02 PM
It's more like there are a series of msis in various versions and vintages (which are not at all consistent), and a raft of zips which I do not know how to handle.
Or rather, the consistency follows some arcane calculus I haven't broken yet.
 
Maybe it's just epicycles.
 
I would argue with that (beyond "hey! I try!" ;) but I did just have to explain how a stack/queue works to someone with a BS in software development
granted, it is from an "online only we only focus on the practical skills" school
 
@JGreenwell Sorry, we devolved from "Reading, 'Riting, 'Rithmatic." to "Standardized tests and CS" so from my viewpoint we weren't discussing post-high-school things...
 
oh, high schools vary massively by area here in the US but I have noticed a tendency to skip "straight to coding" without covering any basic, core concepts (like data structures and algorithms).....which gets people involved but at the cost of so many script kiddies
 
Air
@JGreenwell Wait, explain how? You mean they didn't know that stacks are LIFO and queues are FIFO, or something lower-level?
I'd love to be in charge of picking the analogy used in a 101 class to introduce that subject, by the way. This semester, stacks will be laundry hampers and queues will be pinball machines launchers.
 
11:14 PM
LIFO or FIFO; or forgetting terminology they did not know how to code a simple stack/queue
 
Air
Extra credit: Code a Jenga data structure.
 
Doing LIFO/FIFO linked lists is often one of my interview questions...
 
Air
every time you access a value it gets popped and appended
 
<- might be bitter cause that person got a job over him
 
Air
Space is allocated in 3-element-large blocks and you're not allowed to access the last block
 
11:17 PM
yep, guy knew how to "manipulate an array" (except he should have said list) but "oh, I never learned what a stack queue is - the professors said that 'that type' of terminology wasn't important"
 
@JGreenwell You can be bitter, but don't act bitter. It makes it harder for your brilliance to shine through.
 
@JGreenwell I'm in favor of this approach. You can't get a kid interested in programming by showing him a nicely implemented bubble sort.
 
Question: I've recently restructured my program to run in multiple processes. The processes connect to a website, take the HTML, parse it, put it in a dictionary, and put the dictionary to a queue (there are some operations done on the HTML; searches, loops, and the like)
I'm running 6 processes, but things seem to be crawling
 
Air
Guys, I'm going to be rich
The Jenga data structure needs to have a metric for fragmentation that it recalculates after each element access, and if it exceeds some threshold, it destroys all references to itself and is garbage collected
 
(crawling ha!)
 
11:19 PM
Does any of what the program does sound I/O bound?
 
eh, I've gotten jobs and I've gotten passed over - shrug, move on (this just happened today and was for a really good job offer soooo I'm betting it is still a little fresh.....esp. after he completely missed the stack/queue question)
 
@Air Sounds good to me. But needs some sort of stochastic mechanism to simulate sneezes, ceiling fans, and unlevel building surfaces, as well.
 
@JGreenwell What's the difference between a stack and a queue? In my head I visualize a stack as being LIFO and a queue as being FIFO, but I don't actually know if that's the case.
 
Queue is a more general purpose term in some cases.
Hence "FIFO queue"
 
@AmagicalFishy Are you serializing? How long are the network calls taking? Those would be your only two points of I/O
 
11:21 PM
Although Wikipedia claims that queues are always FIFO. Maybe terminology has coalesced.
 
Oh, yeah. It's specifically the multiprocessing.JoinableQueue() class (not just a general queue)
I'm not serializing (not explicitly at least), but haven't measured the network calls!
 
Oops, no wikipedia says "principal (sometimes only) operation" -- not "only operation"
 
@AdamSmith I was just pointing out an issue I noticed about this type of policy - I really do not know much about the K-12 grade levels - it only really bothers me at the college level. For the record, I learned by building a BASIC game where a monkey threw bananas at people
 
Though, should they be much slower than my just clicking on the website manually?
(which is pretty fast)
 
Air
@PatrickMaupin I think I agree that stacks could be considered a special case of queues
 
11:23 PM
@AmagicalFishy Should be much faster than that, though parsing the HTML is possibly much slower. Are you using BeautifulSoup or etc?
 
If those're the only I/O bound actions, though—why aren't my cores being used up very well? It looks like each core's usage is like 14% maximum
BeautifulSoup, yes~
 
Air
I guess either is a special case of a deque, though
 
I implemented my first BFS the other day. I feel like a big boy programmer now!
 
(BFS?)
 
As to stack/queue - I meant that as stack or queue: ie. LIFO/FIFO problems the only difference I know is a queue is ordered a stack may not be
 
11:24 PM
(Breadth-first search)
 
(Oh!)
 
(germane to the queue topic happening elsewhere in the room :P)
 
The ordering of a stack is implicit, but it depends on both queue and dequeue operations.
 
(I thought "BFS... BeautiFulSoup? No, no, can't be")
 
Air
@JGreenwell How can a stack be unordered?
 
11:25 PM
I think he means ordered strictly by enqueuing.
 
@AmagicalFishy I'm not sure why you're not seeing larger CPU usage. Multiprocessing is a strange and magical beast. Parallelism in Python in general is something that is usually just done by wizardry as far as I'm concerned~
 
Hm, damn. I'm explicitly defining processes via the multiprocessing module—and things definitely move faster than they would w/o my having done so (in fact, a lot faster)
 
I'd recommend profiling the code to see what's taking the longest. Step back from the parallel processes for a minute and just try to get the tightest algorithm you can
 
I'm just anal and want my cores to be at like 75% >:D
 
yes, @PatrickMaupin thank you
 
11:28 PM
usually people ask about the other problem -- why is helloworld.py taking 80% CPU? :P
 
Alright, I will do that, first~
Haha, I think my asking about the opposite (I need more CPU!) is indicative of my programming skills growing!
 
Air
Are you talking about how, e.g., if I add A, B, C to a queue, nothing else that I add will change the fact that they'll come out in that sequence - but with a stack, I could pop one of them and then add something, changing the sequence?
 
if you're going to go async, though, I strongly recommend you have a process that makes the async request and hands it off to another worker process. That at least gives you some degrees of separation between I/O and CPU.
 
Can you explain that a bit more? (I've just started dealing w/ multi-processing/threading a couple of days ago). Right now, all the processes draw from the same multiprocessing.Queue() (which is process-safe)
specifically, what do you mean by the "async request" ?
 
main process hands a URL to the HTML process. HTML process makes the network call and hands the result to the Parser process. Parser process parses and does stuff and hands the result back to the main process
 
Air
11:32 PM
"Asynchronous HTTP request" I think (maybe general request though)
 
I wouldn't have one worker make the network call AND operate on the data.
yeah async is wrong there.
 
Ah! In part, that's what I'm doing now (the one worker making the network call and operating on data), only because I thought that storing all that HTML in the queue would take up a lot of memory
(since it ends up w/ about 10,000 pages) -- or is that something I'd just have to time correctly? Like... if the queue is of a certain size, stop the URL -> HTML process until there's space
 
no more than storing it in RAM
it's stored either way :P
but yes if there's a possibility of you running out of memory, you'll have to limit the size of your queues
 
alright. so here's my plan so far: 1) profile everything (i should have done this in the first place)
2) further separate HTML + parsing processes
... I've already gathered all the data overnight yesterday, and honestly don't have much of a need to keep doing it (not for another month or something, at least). BUT IT NEEDS TO GO FASTER >:(
 
Does pip work out of the box with python 2.7.10?
 
11:37 PM
I think so, @Augusta
 
Then I must be using it the wrong way. -_-; Hm.
 
it started coming in python 2.7.9
according to this
 
The actual question btw. was the standard how to evaluate a string equation mantaining the order of operations. With a follow up about reverse Polish notation. Quick version of my answer: loop, build stack and output remembering to evaluate the order of operation based on last and take into account parenthesizes. His answer was "use eval"
 
I've read that, to install a whl, all I should need to do is type pip install <filepath>. But I guess I have to import pip first. Okay, cool. Still nothing.
 
sorry, in-laws are here rbrb
 
11:39 PM
Cbg.
 
I dunno what OS you're using
but when I use <code>pip</code>, I'm not using it from within a python interpreter
i'm just using it from the terminal
 
Air
@AmagicalFishy Backticks in chat.
 
oh
yeah, I just tried that, Fis. That did it, ish
 
Yeah, you should be using the terminal.
 
pip
Thanks @Air :D
 
11:41 PM
It doesn't like the wheel now. Nice. >:I
 
Air
Hey, it just hit me... did people decide to call them "wheels" because they come from the cheeseshop? :D
 
Oh, I see. Windows is being an ass. Right.
 
what's... a wheel?
 
trying to think up an idiomatic way to get the subsections in a sudoku board defined as a 2D list
grouper comes to mind but I'm not sure how I'd get more than one
grouper(grouper(row, 3), 3)? Does that even work?
no that's dumb
hrm
 
so like... each subsection has its own coordinate system?
 
11:47 PM
board = [
  [7,8,4, 1,5,9, 3,2,6],
  [5,3,9, 6,7,2, 8,4,1],
  [6,1,2, 4,3,8, 7,5,9],

  [9,2,8, 7,1,5, 4,6,3],
  [3,5,7, 8,4,6, 1,9,2],
  [4,6,1, 9,2,3, 5,8,7],

  [8,7,6, 3,9,4, 2,1,5],
  [2,4,3, 5,6,1, 9,7,8],
  [1,9,5, 2,8,7, 6,3,4]
]
it's a sudoku validator for a programming puzzly thingy
 
are you doing codewars @AdamSmith
 
I could do it with nested for loops but it seems ugly
yeah
 
:)
I know that one
I have it starred to go back to it
 
maybe [zip(*grouper(row, 3) for row in grouper(board, 3)] ? What does that look like...
ipython away!
 
Air
11:51 PM
@AmagicalFishy Wheel is the current format for distributing Python packages, unless the wind has changed today
 
@AmagicalFishy A wheel is a-- What air said. -_-
Maybe the lag is from not having restarted lately.
Guess I'll try that.
 
close. [grouper(zip(*row), 3) for row in grouper(board, 3)] gives something kinda like what I need
[itertools.chain.from_iterable(grouper(zip(*row), 3)) for row in grouper(board, 3)] should do it
 

« first day (1836 days earlier)      last day (3113 days later) »