« first day (1834 days earlier)      last day (3123 days later) » 
00:00 - 17:0018:00 - 00:00

6:01 PM
@PeterVaro It's quite interesting. I've been trying to figure out how it's made. It's not a polar function because it's impossible to do that tiny loop maneuver near the center. I thought it might be a hypotrochoid but I couldn't find the right parameters, if any exist.
I'm currently kicking around the idea of a "double hypotrochoid" which would have a circle inside a circle inside a circle. It's a long shot that's what the original author used, but I should end up with something interesting regardless.
I wish I could find the original tumblr page where this image was hosted, but you can't find the page given the image URL, and google's reverse-image search only searches by the first frame of the gif, so I only get white rectangles
@davidism Yeah, I think so. I might gravitate away from PIL soon as I'm dissatisfied with its poor antialiasing support.
I'll have to take a look to see if Gizeh does any better
 
cbg
 
@Kevin it has to do something with the Fermet's spiral
and with the phibonacci sequence IMO
once I did something very similar to that, let me search for it
*Fermat
this is two fermat's spiral on top of each other
I used it for a logo/icon I've designed:
 
6:19 PM
The hard part is keeping the two central lobes from touching. Ordinary polar graphs can't loop like that unless they pass through the origin (AFAIK)
 
as you can see, the texture of the huge B letter:
is a voronoi tesselation on top of fermat's spiral intersection points :)
 
Nice :-)
 
Hey guys, anyone ever use spyder (or maybe its autocomplete)? I'm declaring __all__ in a subpackage and importing * in the __init__ but the autocomplete doesn't pick up on it.
 
Can't say I have
 
@AaronHall "autocomplete"?
isn't that something your IDE/Editor will provide? ;)
 
6:26 PM
I think it's using rope
 
What's funny is looking up Python Spyder in Google comes up with a Google ad for PyCharm
 
I've used rope in emacs before, I think.
Might be using jedi...
 
user559633
"Why is it annoying to use tests? Oh, because the Flask object is literally api.api.api.api on this project"
 
figured it out, thought I had cat'd the import * into the __ini__.py but I hadn't.
now it works
plus I have pylint installed due to my fussing with settings, so that's good.
 
@tristan unittest woes?
 
user559633
6:40 PM
@idjaw Yes, but for Stealth Startup (tm)
 
ah ok
 
user559633
aka "the dream" aka "let's pretend i'll someday be passionate about the 'day job'"
 
user559633
i ended up just doing aliases on the api.api.api.api thing (e.g. at api.api, branched off a _api = api)
 
I've had issues like that...most cases I wasn't using the mock properly.
but, I also have been having some funny issues with Flask too...
 
user559633
oh, yeah, this was just a structuring issue -- it came out of splitting up the site and the api, then the api being split again
 
user559633
6:45 PM
doing this so i can spin up N API workers separately from the web workers -- they currently also listen to a queue for directions
 
On the topic of unittests, I'm trying to figure this question out:

https://stackoverflow.com/questions/33321296/how-to-access-testcase-instance-method-from-mock-instance-method
 
user559633
from reading his tl;dr that sounds like why class setUp exists
 
@Kevin are you familiar with nanoblocks?
I truly want to buy some sets :)
 
Is that what they could be implying? Reading OP's examples, I got a bit lost trying to figure out the exact goal of what they are trying to do. Because it seems like they are trying to keep track of certain counts of method calls, which is being implemented correctly.
btw, on what you just mentioned about your splitting of API and workers. We just did that ourselves actually. Was well worth the refactoring effort.
 
user559633
@idjaw i just sort of skimmed it. it looks like he's trying to recreate singletons or class methods
 
user559633
6:50 PM
@idjaw oh yeah, totally -- especially because i can use autoscaling groups or watch a moving average to scale up on demand without changing my web-working pool
 
yes, definitely. Our structure went something like client -> LB -> API(s) -> worker(s)
 
user559633
the workers are current N-proc python+flask+api workers, so when they're sitting idle, they become data miners/scrapers, but when the frontend needs work from them, they can prioritize that
 
that's really neat!
 
user559633
yeah -- i'm doing
               API------queue
client-LB<  |-----------DB
               Web
 
user559633
bleh whatever. the LB can talk to API or web (depending on subdomain), the web talks to the API, which listens to the queue for work, but also handles all DB work
 
6:53 PM
Right. What's your queue implementation like? Is it a standard threading queue?
 
user559633
So when a user asks for $really_expensive_call, it gets dropped onto the queue, the user gets a "we're hackin on it" back immediately, a free worker picks up the request for the call, does the deed, then drops it off to the DB and drops a receipt into the session DB, which the web server polls
 
ah..ok..I understand.
 
user559633
using redis, rabbitmq, and postgres. redis for sessions and some dirty-no-good-hacks, rabbitmq for worker queues, postgres for persistent/long-term
 
Ah yes, nice selection. Recently used redis for distributed locking using redlock-py
 
user559633
It's pretty solid.
 
6:58 PM
was a really fun project to work on
out of curiosity, what's your deployment process like for this?
 
user559633
It's still heavy in dev, but some basic tests, merge to master, githook on master to servers to pull
 
user559633
then i just run an ansible script to do some sanity checking/host targeting. nothing too fancy and will be looking into buildbot in the short term, building what i need into YAMS in the longer term
 
user559633
although i can mostly avoid tested or complicated deployments because i write perfect code coughs (please ignore all evidence to the contrary, including above mention of api.api.api.api)
 
user559633
but yeah, i want to get it to the point in which pushing to the master branch runs tests (including synthetic transactions, selenium, android&ios tests), sends a notification to YAMS to clone and make a debian package, which then tells amazon to spin up a fresh host, install, run user tests, then on success of that, start rolling new hosts into the load balancer
 
ah, cool. We use vagrant, puppet and maven
shoot...sorry to cut it short...gotta run....yard work awaits.
rbrb
 
user559633
7:10 PM
cool, good enough setup. i don't really get excited about build stuff, but where Steam Startup(tm) is 3 people, it's pretty huge to be able to deploy without thinking about it
 
user559633
*stealth
 
7:48 PM
Well, I feel more like an accomplished adult now doing all that leaf raking.
 
Hey, guys. I just started working with multiple threads/processes—but it seems way more difficult to debug. When something hangs, I have no clue where it's hanging or what it's hanging on
basically, I'm grabbing HTML from different webpages and parsing it w/ Beautiful Soup on each process. One hangs, but I don't know which one
Might you guys have any tips? GDB isn't helping ;(
 
Use the logging module.
 
hey Aaron! :)
 
@AaronHall I'll give that a look into. Thanks!
 
Hey idjaw!
 
8:00 PM
how goes your saturday
 
user559633
re: logging and multi thread/proc -- it's not as easy as just using the standard lib logging module
 
damn, man.
making this multiprocessing is almost seeming not worth it :l
 
user559633
iirc, you need to look into the handler lock acquisition and release. you could also get away with this by writing to a pipe or an FD and selecting and writing into a file
 
user559633
it's totally worth it, it's just one of those areas in which if you're used to writing python, you forget that there's this whole sea of shit right under the interpreter
 
you have successfully motivated me
 
user559633
8:06 PM
what you can do is log out to something like syslog instead of using a file
 
user559633
that way you're just dumping the log intent into the OS and letting it mostly handle it for you -- just drop the thread id into your log format and drop the message there.
 
Builds character
 
alright. i will do that—can i find everything i need to do that in the multiprocessing documentation?
(this is my first time dealing w/ multiprocessing—the function itself works fine, but trying to put it into multiple processes has it hanging somewhere)
 
user559633
to actually try to motivate you -- it's not that complex of an issue, but keep in mind that threads aren't analogous with processes. multithreading -- the GIL continues to be your guardian angel, keeping you from touching hot stovetops. multiprocessing -- the GIL can try, but you're basically telling your guardian angel "i got this"
 
user559633
@AmagicalFishy multiprocess or multithread? if you're doing a scraper and multi-thread, i'd suggest that it's hanging on a slow HTTP req.
 
user559633
8:09 PM
if you're doing multi-proc and multi-thread, same, but you have made yourself a little hydra of debugging.
 
ah! maybe i should learn more about the basics of what i'm doing before trying to implement this code, haha. i'm pretty sure i'm using multiple processes and multiple threads
 
user559633
keep in mind that the GIL is engaged for all file-like objects and will hang whenever a synchronous request (like fire req and wait for resp) occurs (are you using the requests modue?)
 
using Pool from the multiprocessing module, w/ 25 threads in each process (which, in my case, ends up w/ 200 threads)
 
user559633
You have 8 cores?
 
user559633
8:11 PM
Python 3?
 
(I was surprised, too. ;v)
2.7
 
user559633
 
so, have I just dug myself a super deep hole by using multiple threads AND processes?
 
user559633
No, you just have some stuff to learn now
 
user559633
If this is a weekend/outside-of-work project, then you just did something incredibly good for yourself
 
8:14 PM
it totally is—and that's pretty encouraging, haha. excellent!
 
user559633
I'd say read the document for threading and multiprocessing and probably the select one for above
 
(the threading link you posted links to the select one)
 
user559633
IIRC, the python 2 logging documentation also issues a stern warning about thread safety to file objects
 
user559633
Yeah, read threading and then select
 
user559633
I wish I knew of a good resource for the GIL. the popular ones on google are mostly geared towards "so you already understand this and you want to talk about how to get rid of it, here's why you can't, oh sweet summer child"
 
8:17 PM
haha
awesome. reading threading, select, and multiprocessing.
 
user559633
in other words, welcome to the land of dragons. there are many of them here, but you won't get burned if you know how to avoid or tame them
 
excellent. soon i will be a better-than-novice dragon tamer
 
user559633
That's probably a better order than what I had in mind
 
thanks for all your help, btw, @tristan — I was pretty completely lost :D
 
user559633
@AmagicalFishy lol, everyone is for a long time after going into threading/multiproc
 
user559633
8:19 PM
keep in mind what CPython is -- a set of rules in a virtual machine sitting on top of C
 
user559633
absolutely nothing about it is magic and it carries all the same limitations and challenges as C
 
user559633
To help you out more specifically, part of the challenge of using worker pools is that you assign them a function and some arguments, and then wait on them to return something
 
indeed! a lot of it is a matter of learning what's available for use (i.e. - this threading thing is awesome; i've just read of the ability to leave one thread dormant until it's called upon by another thread)
 
user559633
Yeah -- and select is cool too. This is a really good foundation to have before jumping up to event-driven stuff like signals or async in python 3.4+
 
user559633
And with the GIL in place, let's say that you have a process and 20 threads, but the GIL says that only one thread can do work at a time. One of those threads is like "hey, i'm doing work! let's lock down" and then goes at a turtle's pace (slow network connection/other end is throttling you/site waits 30 seconds to respond with useful bytes) everyone else in the proc is waiting
 
user559633
8:23 PM
Same applies when you have 4x of those processes, or 8x, or 2000x
 
right now i had a bunch of code parsing a bunch of URLs, then following another URL from the first, and putting some data into a dictionary to write to a csv. I basically put all of that code in a function which took URLs for the argument, got a .csv of all the URLs, and threw that function into Pool.map()

this looks like it's going to give me the opportunity to fine-tune things a lot more than just throwing them into pool and waiting for it to do its thing
 
user559633
Is that CSV file shared by all the threads?
 
@tristan I don't think so. the output .csv isnt' written until all the threads are finished (they each return a dictionary, which is put into a list after they're all done, and said dictionaries are written into the .csv as a row afterward)
 
user559633
Anyway, I think you'll very soon understand why I asked that.
 
the input .csv might be: i'm not sure how the multiprocessing module uses it
the code for that looks like this:
with open(fileName, 'rb') as urlFile:
resultsMeta = pool.map(GetMatchStatistics, urlFile)
 
user559633
8:27 PM
before the thread pool happens?
 
user559633
In any case, I have total confidence that you'll figure out where it's hanging and how to log after reading those resources.
 
yeah. i think [think] that's what instantiates the thread pool
i think so, too
 
user559633
I'd suggest that you do a write-up and do a self-answer on stackoverflow after figuring it out -- I'm sure it will help a lot of people.
 
thanks so much for your help :D
 
user559633
No problem at all, I'm here almost all the time, so let me know where you end up.
 
user559633
8:28 PM
Also, welcome
 
very excellent :D
(i will do that; is that something that I tag as community wiki?)
(i'm always asking questions. it'd be nice to contribute more often)
 
user559633
You can, I guess. I'm not really sure why or what community wiki is, so I'd just say make it a normal question / answer.
 
user559633
StackOverflow internet points are as useless as any other, but there are some "privileges" that make using the site more fun up to 10k.
 
user559633
I have a certain reputation to defend in being gruff/sarcastic, but asking questions is contributing as it gives others practice in how to teach.
 
indeed! i'd like to teach every now and again, though. ;)
 
user559633
8:33 PM
Self-answer. Anyway, thanks for coming in with that question, it was a good one.
 
yours was a good answer~
 
user559633
It's one of those cases in which using the module (multiproc) is easy, but logging/inspection is hard
 
user559633
@AaronHall do you still give regular talks/presentations at meetups?
 
user559633
[sorry, topic that doesn't exactly follow what we've been talking about]
 
user559633
I'm curious if you've given/been to a good one on the python 3.5 async def/native coroutines stuff
 
8:58 PM
oh, man.
ok, so i think i'm going to forgo the use of multiprocessing.pool
and re-write my function to manually manage processes and threads
where each main webpage's HTML will be fetched and parsed by one thread, other threads created to follow, fetch, and parse the 50 links on that page--- and then clone this into several different processes, which each do that on their own main webpages
i don't know how viable this is, but if it is, it'd make logging easier and, i think, make everything move faster
i will name the threads according to the order in which they're created so there's no weird shifting in case one thread finishes before another
THIS WILL BE GREAT
 
@AmagicalFishy The safest strategy is to never start a subprocess if there is more than one thread running. The easiest way to enforce this is to always start all your subprocesses (or just start a single subprocess that will start others for you) before you start any threads.
 
@tristan I led a discussion today on best practices for module creation, but it wasn't exactly a "talk" per se. Those require full approval in advance by corporate communications, so I only do talks at conventions. And I'm only volunteering at PyData since it's so much work to put together a full fledged talk, with the approvals and everything. But I'm hoping PyCon accepts a talk from me next time.
 
@PatrickMaupin so it would be advisable to do something like: 1) open process 2) create 51 threads within said process [1 to get main html, 50 to fetch/parse other html links on that main page, which all wait for the first to finish]
then do this for however many processes i want (one per core, i think?)
that is, all threads are bound to a process (i don't know if i'm making any sense, haha)
 
I don't think you need to create threads up-front. And you don't even need to create processes up-front if you have a proxy process to create them for you. AFAICT, the root of the problem is that when you fork, only the forking thread is copied over into the new process properly, and if anything is happening "under the hood" with the other threads and the GIL during the time of the fork, bad things could result.
But yeah, if you only created threads from sub-processes which themselves never created sub-sub-processes, that should solve the problem.
You have to be careful when using things like multiprocessing, though, because I think it uses threads behind the curtain to manage some of the higher level interfaces.
 
9:13 PM
oh, crap. that is good advice.
say i didn't use multiprocessing at all, and only used the threading module—then i wouldn't be able to make use of multiple cores, right?
 
9:36 PM
Not very well in pure Python, no. If you are writing extensions that release the GIL, yes.
 
wim
#SOreadytodownvote
 
Not sure that's the slogan they're looking for.
 
Anonymous
10:32 PM
Is there some sort of conventional directory structure for a python app, even when you are not following the MVC, and any design pattern?
 
Anonymous
I have just a simple set of files each containing a function, but work together for one task, and I have no idea how to structure the files.
 
@Kevin's list.append(Frequencies) // quite original, I gave it a strong 8.5/10
 
Anonymous
10:52 PM
 
@samayo do not ever -- and I mean NEVER EVER NEVER EVER NEVER EVER -- read anything from that guy
LPTHW is a useless crap, just as many other books he wrote
 
Anonymous
@PeterVaro Are you serious? I really thought it was the best place to learn python properly, given its popularity and how well it is written.
 
> We strongly recommend you don’t use Learn Python The Hard Way unless you’re very careful. It contains some “interesting” decisions, and its structure tends to lead people into asking premature questions.
 
Anonymous
eh, I hate it when this happens
 
Anonymous
But since I am a noob, I doubt it may cause more harm than good.
 
10:58 PM
indeed!
if you want to learn Python The Good Way -- I suggest you to stick with its brilliant and whole official documentation
 
Anonymous
Hm, the official is for python 3, I just spent 3 weeks learning 2.7 and to make matters even worse, I am on CentOs and python 3 is not yet available
 
Anonymous
Any guides for 2.7?
 
@samayo it is available everywhere => you should use your package manager to search for and download python3
if you are not forced in some ways to stick with python 2.x, then for the sake of sanity, please use python 3.x
(if you are forced (you have dependencies, legacy code, etc.) then switch that little box on top left to 2.7)
(but please don't, I'm begging you, for your own good :))
 
Anonymous
Okay, I will waste some unplaned time to upgrade :). Do you know any good reads/guides though? As the official doc is more like a manual than a guide anyway.
 
8 mins ago, by Peter Varo
http://sopython.com/wiki/What_tutorial_should_I_read%3F:
 
Anonymous
11:07 PM
thanks
 
So pygame.surfarray.pixels2d() seems to be eating up a bit of memory every time I call it, and doesn't release it until the program and shell close or restart. Anyone know how to fix it?
Also, does anyone else have this problem, or is it something local?
If I assign it to anything, or apply a change to it in-place, python bloats a little bit more.
Just calling pygame.surfarray.pixels2d(testsurf), without assignment to or by another value, is safe. I don't need the array to look at, though; I want to do things with it, and that causes this weird problem. -_-
 
Anonymous
[root@localhost /]# python -V
Python 3.3.2
 
Anonymous
@PeterVaro I hope you are happy now :D
 
11:26 PM
Also, is the new pygame.org website really awful, or is it just me? =_=;
 
@Augusta hey, what's up?
 
Trying to figure out this horrible thing in pygame, to little effect.
You?
 
I'm tweaking my docstrings in my new library based on spyder's treatment of them. I need to go find some docs on it, I only started tweaking them on the subway when I couldn't search...
I suppose if I do it Numpy style I'm safe, right?
 
11:44 PM
@Augusta The main page made me cringe, but fortunately the docs are unchanged.
 
I was afraid it might be the page itself. -_- Thanks for confirming.
Also, yeah, thank god they left the docs alone.
 
I like how they call the new thing "hifi.html."
 
Yeah, that's what made me think it was me. I can see it being something better that doesn't work, but it's hard to imagine it being deliberate.
 
They basically put the entire website in a single page, and you move the viewing window by scrolling vertically or clicking a fancy little horizontal scroll button...
 
I'd feel like it would be fancier if it was 1998. =_=
 
11:49 PM
If this revamp had taken place 10-15 years ago, it would've been an unskippable Flash video intro instead.
 
One with audio.
 
Nowadays it's all smooth JS controls.
Yeah, loud, low-quality audio.
 
Low-quality audio that, for some reason, comes in massive audio files.
 
I don't think there's anything shameful or amateurish about a basic layout that doesn't hog CPU unless it's doing something important, but everything's gotta be whiz-bang.
 
I think there's something shameful in replacing a functional but visually-unattractive layout with a broken and visually-unattractive one, though. =_=;
A text-only screen full of data and hyperlinks is fine as long as it's navigable.
But man, this new thing is a wreck. (So sayeth I.)
 
11:58 PM
oh wow..that IS a terrible site
I was just on it a few days ago and it wasn't that
 
00:00 - 17:0018:00 - 00:00

« first day (1834 days earlier)      last day (3123 days later) »