« first day (2286 days earlier)      last day (2661 days later) » 

12:00 AM
numpy mostly allows you to do fun(i,j,k) or fun((i,j,k)), doesn't it?
which would be equivalent to expecting *args, and looking at its length when needed
>>> def foo(*args):
...     print(args)
...
>>> foo(3)
(3,)
>>> foo(3,4,5)
(3, 4, 5)
>>> foo(*(3,4,5))
(3, 4, 5)
 
[2,3] * [[3,4],[5,6]] = [[2,3],[2,3]] * [[3,4],[5,6]] = [[6,8],[15,18]] - that is the basics but it works "for all functions", not only operators. (So also for data analysis, like linearization of nd arrays, or passing functions to matplotlib for plotting).
 
if you have a few values, pass them as arguments; if you have an iterable, pass it with a splat
@paul23 oh you mean array broadcasting
 
oh that's the correct term yes.
But also 3 = [3] for everything in numpy
 
ummm....
I'd have to think about that
yeah, no
 
Well a single value is also broadcasted to any dimension you need.
 
12:05 AM
right, it's scalar
but [3] is not
btw there's numpy.broadcast() that explicitly mimics array broadcasting for you
 |  Examples
 |  --------
 |  Manually adding two vectors, using broadcasting:
 |
 |  >>> x = np.array([[1], [2], [3]])
 |  >>> y = np.array([4, 5, 6])
 |  >>> b = np.broadcast(x, y)
 |
 |  >>> out = np.empty(b.shape)
 |  >>> out.flat = [u+v for (u,v) in b]
 |  >>> out
 |  array([[ 5.,  6.,  7.],
 |         [ 6.,  7.,  8.],
 |         [ 7.,  8.,  9.]])
although I've never used it, I tend to use numpy built-ins
 
I didn't even know it was there :P
 
some further fun:
>>> np.array([3]).shape
(1,)
>>> np.array(3).shape
()
the latter one is a proper numpy scalar
it's a weird type that, if I recall correctly, is mostly around for historical reasons (not sure)
>>> np.array([3]).ndim
1
>>> np.array(3).ndim
0
 
 
2 hours later…
1:58 AM
@paul23 Side-effects in a comprehension is considered bad style. However, you could do this instead of modifying that method to return its arg. [formatit(x.mutablething(i) or i) for i in range(10)]
 
DSM
2:53 AM
@MarcusS: thanks for that Norse link the other day -- I went through a saga-obsessed period about a decade ago and that guy's videos are pretty interesting.
 
How to exclude multiple classes or better yet all of them with scrapy?
response.xpath('//span[not(contains(@class, "result", "win-txt"))]/text()').extract()
 
3:55 AM
@davidism Why they are talking about a different game here, I think the logic still applies: youtube.com/watch?v=TS9zD_LPhw0
*While
 
4:49 AM
Cabbage :-)
Did you people see BossSensor?
 
5:33 AM
don't use the cv-pls script (you can tell because it adds [tag:python])
@MarcusS "different game"
There was a similar question in the AMA. I hear that it was actually a pretty good game though, so maybe they're actually serious.
 
5:55 AM
I think he was actually serious... for the most part, anyway -- but either way, I think his answers are still relevant for HL3
 
6:12 AM
@MichaelHCameron You said: "is there a more preferential means of updating output on the console?". It depends on the terminal. In *nix, you can do all sorts of interesting things with ANSI/VT100 Terminal Control Escape Sequences. There are libraries that take care of the messy details, but for your program it should be easy enough to do it by hand. See this answer of mine for a demo.
 
6:56 AM
Global Britain <3
 
cbg @AnttiHaapala
 
cbg
@IljaEverilä loving my nespresso
 
How goes life at EvilCorp
 
which one of them
 
The former IljaCorp :P
You've gone 100% telecommuting since you got your nespresso?
 
6:59 AM
something like that
so, I guess the honeymoon with NewIljaCorp is over :D
time for 6 :D
 
Nah, endless honeymoon here ;P
What with having to handle client's broken VPN solution that has little to no linux support and sends some broken s*it so that one has to patch client software in order to be able to communicate
 
7:39 AM
at least you're not using "Skype" for "Business"
 
Well guess again
 
ok, you're using "Skype" for "Business"
 
Don't post answers here. You will get downvoted :D stackoverflow.com/questions/41713154/…
 
8:20 AM
 
@BhargavRao Makes sense.
 
@MYGz I always wonder why people bother about downvoting. It's almost like reputation is money. Back in the day when Google Groups was a big thing someone went around systematically downvoting everything that I and Fredrik Lundh (the guy who wrote the Python Image Library and much of the stdlib re code) systematically. I'm still alive and still happy. It's only ones and zeroes ...
Which I guess is to say cabbage, all
 
@holdenweb Agree. cabbage :)
 
Cabbage holden
 
9:07 AM
cabbage , does anyone know how to filter specific instance without use of lambda, for example
filter(isint,listelements)
 
[el for el in listelements if type(el) == int] ?
 
thanks, but i was thinking about 'filter' function, and type.obj == type is almost the same as isinstance.
and still would need 2 variables if i made it into a function, varaible and desired type.
 
9:22 AM
Can you create a MCVE?
 
Cabbage!
 
Im trying that now, and i was wondering if someone smarter than me has already came up with solution
 
Cbg!
 
a bit of intro : i made a dump script, where any file you input can be converted to bin,hex,oct etc etc, but since normal numbers have 1 byte, chars 2 bytes and pixels 8 bytes then i need to differentiate between those types and pack them to stream/integer for better performance , then bin shifting to retrieve specific parts. So maybe someone of y'all has allready tackled this problem.
also cabbage
 
9:41 AM
@Danilo
>>> int.__instancecheck__(5)
True
>>> int.__instancecheck__(5.5)
False
or... if you need exact type check, you can't do it without a function
 
Awesome! Thanks, but what do you mean with exact type check
 
subclasses will pass that test
>>> class MyInt(int):
...     pass
...
>>> int.__instancecheck__(MyInt(42))
True
 
Also,
>>> list(filter(int.__instancecheck__, [1,2,'3',4,5.5,True]))
[1, 2, 4, True]
>>> list(filter(bool.__instancecheck__, [False,0,1,2,True]))
[False, True]
@Danilo "normal numbers have 1 byte, chars 2 bytes and pixels 8 bytes " I'm not sure what that means.
 
well minimal requirements to describe 1 small number is 1 byte, but for ,chharacter is 2 since readable ( caps, normal and punctuation ) characters populate 2 bytes since control characters populate range (0,0xf)
thanks :D will read it and test it :D
 
cabbage
 
9:55 AM
Also, for my own better understanding of file structure i want to display control characters only with '--cc' argument and 'DATA' where data is. I want to learn how to use them, and if it is possible for me to write a file ( eg. png ) from scratch by hand (keyboard)
cabbage
 
....
make sense you do not :D
 
make sense that i don't understand the current topic, or that i am like socrates and that i know nothing ?
 
@Danilo Ah. You didn't say that was the minimal size, which is why I was confused. Python integers can be huge. I guess you're displaying pixel as a triplet of hex values with a separator, eg ff 77 00.
 
Question: How to register Postgres bitmap index in SqlAlchemy Model.
Answer: "Go to Signup and fill up the fields to register Postgres bitmap index in SqlAlchemy Model and then log in your account."
@MartijnPieters why do you remove this answer: stackoverflow.com/a/41694454/918959
as per SO guidelines, that is clearly an answer, and to the question :P Technical inaccuracies aside, it is a perfectly valid answer :D
 
:)
@Danilo Creating a valid PNG file is a complex process. Even doing it with libpng is scary. Doing it totally from scratch is frankly insane.
 
10:02 AM
perhaps time to post to meta and start complaining that users <10k have no way of dealing with such answers, besides downvote.
 
yeah :D but that is the fun of it :D think about the things i would learn if i manage that :D @PM2Ring
also yeah, but first i would need to figure out where pixels are in those jumble of binary code
 
PNG is a chunk-structured format, so at the outermost level it's well organised. However, the image data is compressed, and PNG supports various compression schemes with a variety of options, which leads to many hundreds of possible ways to compress a given image.
Virtually all PNG writers make a stab at choosing a compression scheme that's appropriate for the current pixel data, but there are PNG optimisers that will do a more thorough search to find the scheme that yields good compression on the given data.
 
hmm, but still there is a guy/gal that coded the open png into a script and managed somehow to find the way to read and manipulate the data. My assumption is that first you need to figure out logic behind it and then implement it. Someone on this earth knows how to do it.
 
10:19 AM
Sure. If you're really keen take a look at the official PNG Documentation. And if you can read C, study the libpng source code: it's well-written, and well-commented. But it's still scary. :)
FWIW, Mark Adler, who was a major contributor to zlib and gzip, and also made significant contributions to developing the PNG format, is a SO member. He still occasionally answers questions.
 
Awesome! thanks @PM2Ring
 
Just don't go annoying him by making random comments on his posts. ;)
Mark also posts on PCG, eg codegolf.stackexchange.com/a/78901/46655
 
well to be honest i probably would not message him or comment on his posts at all, i will search for papers and publications and learn from there. i dont want to bug people unless it is necessary
i mean coding is a hobby and a way to increase my own knowledge, i am an architect by profession. it don't seem nice from me to bug other people to satisfy my own curiosity. The ecstasy is in discovery :D
And thank you all for helping me with material and examples :D
 
10:38 AM
@MooingRawr That are no Kanji those are hiragana/katakana :P
and yes it is cheese bread
 
@Danilo see these >
This post is packed with so much history and information that I feel like some citations need be added incase people try to reference this post as an information source. Though if this information is reflected somewhere with citations like Wikipedia, a link to such similar cited work would be appreciated. — ThorSummoner Oct 16 '15 at 16:29
I am the reference, having been part of all of that. This post could be cited in Wikipedia as an original source. — Mark Adler Oct 16 '15 at 16:38
 
10:58 AM
Why does scrapy become slow after the first few hundred pages?
 
PEBKAC error?
or rate limiting from the server
 
cbg
 
11:06 AM
@thefourtheye I sent this to my boss
 
@AnttiHaapala yeah looks like that's it
Will using a different proxy every so often bypass that
 
Working on improving my own internal library for code snippets I re-use all over the place. What's the best DRY way to check a directory exists, whilst avoiding a race condition? Currently, I'm using:
    def save_trigger_settings(self, filename):

        destination = os.path.join(self.folder, filename)

        try:
            self._actually_save(destination)
        except FileNotFoundError as fnfe:
            if fnfe.errno != errno.ENOENT:
                raise
            else:
                os.makedirs(os.path.dirname(destination))
                self._actually_save(destination)
Which doesn't avoid the scenario of some basket case process waiting for me to create the folder and then immediately delete it, but it's better than just trying to open the file and failing.
 
it is better though?
I kinda like user gives a existing director approach more if there can be race conditions
 
What if os.makedirs fails due to not having permission to create the dir(s)?
 
11:15 AM
@khajvah - This won't be something that will be running with end users in mind, but might share a system with other programs that I haven't developed.
@PM2Ring - then it should quite rightly fail.
 
Ok.
 
@IntrepidBrit I don't think you can guard against, for example root messing around with your program data
 
You're right that I can't protect against a PITA rooted process, but it's entirely valid that another process (say, something like puppet) should have access to some of the configuration files and might accidentally cause a race condition
 
@IntrepidBrit run your process by a special user and chown the directory
like android does(I think?)
 
Then puppet wouldn't be able to update/configure some of the settings of the program, which would be a worse condition.
 
11:25 AM
Put puppet and your special user in a group that owns the dir
 
I don't want to get hung up on the specifics of the origin of a race condition (unless it's directly pertinent), but if I have thousands of these programs running over the course of 10 or so years - a race condition is something that could crop up from any number of different sources
 
What I like about that is that you have an unlimited maintenance job adding more stuff to that group in the future
Think of it like a pension
 
A bad pension?
 
Mornin' all
 
cbg("JRS")
 
11:31 AM
Sounds like a very bad pension.
 
But I guess ultimately because creating a folder then a file isn't a single atomic operation at the OS level, there's no surefire way to avoid a race condition.
 
12:01 PM
This OP is getting hostile. I guess we need a diamond mod to step in. stackoverflow.com/questions/41718699/…
 
@IntrepidBrit actually: nooo...
 
@PM2Ring cabbage
 
instead you should just open the files from the directory
@IntrepidBrit which OS btw?
 
@BhargavRao Thanks! The OP has improved their question, and clarified that they meant "open" rather than "attach". But after that bad behaviour I don't think they deserve an answer.
 
Bah, I'll let the reopen review queue handle it. I'm sure that there's a dupe for that. :/
 
12:20 PM
Well, there are a zillion CSV questions, but without more details from the OP it's hard to choose an appropriate one. Hopefully they'll respond to my last comment with some solid details, or ask a new question... or go away, never to be seen again.
 
@AnttiHaapala The library is currently used on various flavours of linux and windaz, but I don't have a problem with splitting them into platform specific versions (have to do that with some of my raspberry pi gpio helper code). Why?
 
12:37 PM
I need a library for 3-way JSON merge :(
unfortunately the jsonmerge available in PyPI just does 2-way
@IntrepidBrit linux has openat which is what you should be using.
supported by recent pythons
windoze I don't know
and
this is the only sane way of doing it... :d
>>> import os
>>> dir_fd = os.open('somedir', os.O_RDONLY)
>>> def opener(path, flags):
...     return os.open(path, flags, dir_fd=dir_fd)
...
>>> with open('spamspam.txt', 'w', opener=opener) as f:
...     print('This will be written to somedir/spamspam.txt', file=f)
...
>>> os.close(dir_fd)  # don't leak a file descriptor
if that works in Windows, you're fine :P
sorry
Currently dir_fd parameters only work on Unix platforms; none of them work on Windows.
so, you're screwed :D
wowo
3way json merge
 
12:58 PM
How can I extract the text "DAMAGED" from ======= Suspect \nItem: \n\nFrequency \nVia \n \u2610 NR \u2610 Unk \n\nID # \n \u2610 NR \uf078 Unk \n\nStart \n \u2610 NR \u2610 Unk \n\nStop \n \u2610 NR \u2610 Unk \n\nDAMAGED Initial \n2 packs per days \n\n \nIV A \n\n \n12-Jan-2017 \n\n \nN/A \n\nReport:`
Using regex that is
 
@AnttiHaapala Perfect. Thank you for pointing me towards openat.
 
@IntrepidBrit doesn't work in windoze
 
Aye, I saw. But I'll split that improved function out into a linux specific handler.
Looking at NtCreateFile just now to see if there's something I could put together for windaz later
 
@vinita: for that kind of problem I prefer to avoid regex when I can. Is the need specific?
 
Im sorry but instead of "DAMAGED" the word could be "WORKING" or it could be blank " ". The regex should match these three cases
@holdenweb Actually it should match these three cases
 
1:09 PM
For example, if your data is held in variable data then data.splitlines()[16] gives you 'DAMAGED Initial'. This is much easier to work with. So divide and conquer
That's why I asked whether you were actually required to use regexen
 
quick question
i'm progressing slowly on my python learning
and yesterday
i looked
at the exception part
is there any good explanation about it, where you can use that in a function for example?
 
@AndyK Exceptions are a way of coping with exceptional circumstances. The beautiful thing about them is that they can "bubble up". So if an exception happens in some function it has a choice of dealing with the exception itself, or letting it bubble up to the function that called it. And if that function doesn't know what to do, it can let the exception bubble up to the next level, etc. If nothing catches the exception then it'll get reported to the user and the program will halt.
This is in accordance with the ancient proverb: "Never test for an error condition that you don't know how to handle."
 
1:25 PM
It takes some getting used to changing from "look before you leap" to "it's easier to ask for forgiveness than permission"
 
multiprocessing is evil
doubly so in Python 2.7
in Python 2.7, using multiprocessing on Unix, you'd just want to kill yourself.
 
cabbage
@holdenweb how official is "regexen" as a plural of regex? I love it:)
 
Dunno. It certainly isn't original to me
 
DSM
@AnttiHaapala: what makes you say that? Admittedly my mental fortitude is known far and wide, but I've managed to use multiprocessing without any temptation toward self-harm.
 
@AndyK Case in point:
def average(sum, count):
    return sum / count

if __name__ == "__main__":
    try:
        average(3, 0)
    except ZeroDivisionError:
        print("Learn to count, you muppet.")
 
1:29 PM
I've used multiprocessing for trivially parallelized loops with success; but I suspect Antti's grievances root in more complex problems
 
@DSM if you use threads, forking is not safe. If you use sockets, forking is not safe. If you use X, Y or Z, forking is not safe. If you use macosx + dns, forking is not safe.
 
And if the exception actually isn't all that exceptional but tends to happen fairly frequently, it's more efficient (time-wise) to "look before you leap"; OTOH, once you're used to them, using exceptions makes the code a lot easier to read, rather than cluttering it up with a bunch of ifs and passing error codes around, so it can be worthwhile using the "it's easier to ask for forgiveness than permission" strategy at the expense of slower execution.
 
In Pythons <3.4, multiprocessing is forking-only, so, it is not safe at all
just debugging this one problem that is caused by using sqlalchemy in a single-threaded app with multiprocessing.
 
It doesn't make sense to catch the ZeroDivisionError in the average function, because you don't know what the calling program is doing in that context. Instead of berating the user, you may want it to go off and work collect the count data instead.
 
crash, crash, crash, crash, crash, crash - basically it is impossible to get right in a complex program.
 
1:31 PM
cbg
 
My example is a bit contrived.
 
@PM2Ring not to mention the few cases where looking before you leap can make you fall in a moving pit; but that's another discussion:)
 
basically: you've got a constructor, and destructor/finalization/whatever.
you fork.
now your destructor/finalizer/whatever will run twice.
 
DSM
@AnttiHaapala: I always suspected I was capable of doing many impossible things before breakfast! The third-party validation, while unnecessary, is refreshing.
 
1:33 PM
say, you're using named temporary directory. you're using multiprocessing. now you're cleaning your directory twice.
 
Looks like Windows demonstrates its superiority again by not using fork in multiprocessing B-)
 
exactly
 
Take a seat Linux you might learn something from the pros
 
hmm
at least the file isn't removed
I guess globals aren't cleaned.
 
@AnttiHaapala isn't there a switch/lock/whatever for doing these things only once?
I'm only a physicist, but our parallel code works such that only one node at a time is writing to stdout :P
 
1:39 PM
@AndrasDeak no, the thing is you don't want it to happen even once...
 
Cleaning your directory? Then don't do it; problem solved.
 
@AndrasDeak if you wouldn't wipe your ass after going to toilet...
 
I'm not following:P
 
ok let's phrase it this way:
...
 
1:46 PM
#consider the code:
create_temporary_directory()
fork()
do_stuff_to_directory_that_takes_an_indeterminate_amount_of_time()
delete_temporary_directory_if_it_exists()
#if thread A takes five seconds to execute `do_stuff` and thread B takes fifty seconds, then thread A will delete the temporary directory forty-five seconds before B would normally be done using it. This is bad.
 
is theres a way to properly modify str built-in type? I've tried to replace __builtins__.str with my ModifiedStr class, but it only affects str() function, while creating string via ' '/" " creates original string with 'str' type.
 
@MaxLunar why would you do that?
you generally don't want to meddle with built-ins
 
You're never going to get a "can't delete directory that doesn't exist" error on the delete_directory line, but that's not the problem
 
I want to store some additional properties in string without use of dicts
 
@MaxLunar don't
subclass str and explicitly use that?
 
1:48 PM
x = StringWithProperties("hello, world!", reticulated_splines_count=23)
 
You've quickly gone a long way from "how do I do exec with globals()" :|
I suggest that you turn back.
 
forbiddenfruit library is doing work well, but I want to work with std lib instead
 
@Kevin so why not stop_doing_that_forky_thing(); delete_tmpdir_finally()?
 
Myself, I'd suggest using join before allowing only the main thread to delete the directory, if multiprocessing has such a thing. idk I only use it once a year
 
more confusion
 
1:51 PM
No problem; my code is always confusing.
 
@Kevin store stuff in a global variable in child process - it never gets cleaned
import multiprocessing
import tempfile

def f():
    x = tempfile.NamedTemporaryFile(delete=True)
    print(x.name)

p = multiprocessing.Process(target=f)
p.start()
p.join()
named temporary file is deleted correctly
add "global x" into f -> it is never deleted.
however if you add del x again there, it is :D
 
@AnttiHaapala brief political segue - my friendship group is watching Finland with interest with respect to Universal credit.
 
@IntrepidBrit it is a trap
 
"The news agencies will kidnap them and force them to work in a Finnish labour camp to pay for Universal credit" - kind of trap?
 
not going to happen. what they've doing now is a politically-motivated botched research with faulty premises that is going to produce unneeded results.
 
1:57 PM
I'm willing to tolerate a library that has its share of pitfalls. But the critical point is this: are they predictable? Can you reliably write a multiprocessing program that doesn't exhibit unusual behavior, just by carefully reading your source and thinking, "no, this will double-destruct, better work around it..."? Or does it require several frustrating iterations of run-crash-diagnose?
 
@Kevin a program of any complexity...
the main point of multiprocessing is that you share data.
but when you share data, you will get this as the result if you say drop any references to global variables.
 
cbg
 

« first day (2286 days earlier)      last day (2661 days later) »