« first day (3588 days earlier)      last day (1345 days later) » 

9:01 AM
how is it possible that I can't get a sim card with a public ip address? I mean I would imagine ip6 addresses to be dirt cheap, why aren't providers handing out ip6 addresses?
 
9:40 AM
switching on IPv6 opens the gates to hell, and the void, and the far beyond, and purgatory.
There are so many assumptions that break with IPv6, it's not funny anymore.
Middleware Programmer: "Use an interface which is public." Me after the network folks switch on IPv6: "Waaaaaaaargh!!!!!"
 
Sarcasm over the internet is hard. I strongly assume and hope you are sarcastic?
 
9:57 AM
No, not at all. Using IPv6 is a lot more complicated than handling colons instead of dots.
Of course, there's also always the option that your provide just won't give you anything for free.
 
 
1 hour later…
11:18 AM
Hi! I tried to ask on IRC, but got no answer, so I'll try my luck here :)

I am trying to POST an image to a server using requests, but the server receives a malformed file. The image itself is not stored in a file, but as a string. When writing this string to a local file, it is a valid PNG. On the server, it is malformed. The paste: https://bpa.st/GQFQ
 
You're writing a valid PNG file in text mode? O.o
 
Yeah, both file and Gnome image viewer can recognize and open it fine
Though that part is only to debug that the data is correct
 
Does files={'file': ('icon.png', data)} work?
 
Nope, same error
 
I'm decently sure images should be read as bytes, not str.
 
11:31 AM
@MisterMiyagi Wrapping data in bytes() does not help either
 
Yeah, it's probably just a matter of your file system encoding being different from the server's
 
@Janman wrapping the data in bytes should plain throw an error.
To be more precise than my initial statement: data should never have been str in the first place.
 
I know, but that is what I have at the moment. It does not come from a filesystem file.
I suspect I need to convert data to some kind of byte stream? I have no idea how though
 
So where are you getting your png in text format from?
 
11:35 AM
Its from some kind of Python HTTP object store developed inhouse. Its an abstraction to reading files from various places (S3, filesystem, memory, etc )
 
And said system only supports text files?
 
No, any kind of file I guess
Again, I know that bytes should not be handled as `str` as such, but that is what I have to work with.

Surely, if writing the `data` to the local filesystem works, it must be possible to interpret the bytes and send them off to the server?
 
I mean, it can only give you the file contents as text? Not as bytes?
 
It is not really defined. In the comment by the original dev, it simply says str
 
Well, if there's really no way to get it as bytes, you'll have to resort to data = data.encode(insert_encoding_here)
 
11:45 AM
Okay, I'll try
Encodings are my nemesis, but I'll see what I can do. Thanks, @Aran-Fey
 
12:22 PM
how do you guys keep track where you stopped in smbc and xkcd comics?
xkcd atleast has numbers, smbc only has titles, which sucks
 
12:56 PM
I actually have a self-made comic reader app that's been in desperate need of a rewrite for at least 7 years
 
@Janman Rarely a good idea to treat images as string data:
Are you, by chance, using Python 2?
 
I know, but I cannot get around the framework used by the codebase.

Yeah, I am using Python 2
 
Python 2's str type is roughly Python 3's bytes type.
Emphasis on roughly.
That explains why things don't blow up immediately, but only later on.
 
Yeeeaaahhhh, this would be messy enough in python 3, but in python 2 I wouldn't want to touch this with a 10 foot pole
The fact that writing a png file in text mode worked suggests that your framework isn't handling strings/bytes properly either
 
You're sending form data. Have you checked that the server is expecting multipart/form data?
 
1:13 PM
Let me see
Not sure, but with my Postman debug request, it accepts "Content-Type: image/png"
 
That's not what your Python code sends. It sends a Content-type: multipart/form-data POST request, which the server must be able to handle if it's going to make sense of what it receives.
 
I changed the Python code too, but no cigar.

(The server accepts a file nonetheless, it is just weirdly mangled)
 
    r = requests.post(url,
                      headers = {'Content-Type': 'image/png')},
                      data=data)
 
user13415013
Hi guys
 
user13415013
Is there any computer vision expert here
 
1:27 PM
Might work?
 
@nerd there are none
 
user13415013
atleast 1 would be , i hope
 
user13415013
If you guys know any, Please tell me
 
24 mins is a bit soon (see the room rules).
 
@holdenweb Yup, thats got it!
Thanks a lot!
 
1:36 PM
Cool!
 
@BikramjeetSingh please see what holdenweb said; we ask not to ask for help here with fresh questions on the main site
 
Hey everyone,
I have a django app which will store users and authenticate them when provided with credentials. I'm using django default authentication module for that. but when i call authenticate(usernmae="uname", password="pass"), this method is returning string instead of user object.. please help me with this peoblem.
 
What's the output of print(authenticate.__module__, authenticate.__qualname__)?
 
2:18 PM
@Aran-Fey Output: django.contrib.auth authenticate
 
welp, then I'm out of ideas
 
:50191241 class UserProfile(models.Model):
    Roles = (
      ('ADMIN', "Admin"),
      ('MANAGER', "Manager"),
      ('AUTHOR', "Author"),
      ('STUDENT', "Student"),
    )
    user = models.OneToOneField(User, on_delete=models.CASCADE, default=None, null=True)
    username = models.CharField(max_length=50, default=None)
    role = models.CharField(max_length=50, choices=Roles, default='Student')

    def _str_(self):
        return self.username
def user_login(request):
if request.method == 'POST':
user_data = json.loads(request.body)
mail = user_data['name']
password = user_data['value']
user = authenticate(request, username=mail, password=password)
print(authenticate._module, authenticate.qualname_)
print(user)
 
And print(type(user)) outputs <class 'str'>?
 
@Aran-Fey <class 'django.contrib.auth.models.User'>
 
2:41 PM
Well, so what's the problem then?
 
3:01 PM
@Aran-Fey i cant access role from that object!
 
Well, I don't know django, but that doesn't surprise me, because django gives you a django User instance, not a UserProfile instance
If you want to know how to get the corresponding UserProfile object, I can't help with that, sorry
 
3:23 PM
Hello all. Should I flag for moderation attention if a user comments "Can you upvote my answer?"
As I see it, it's vote fraud.
 
@NicolasGervais no
@NicolasGervais also no
flag it as no longer needed
It's noise and annoying but not against the rules (in the needs-mod-action sense) unless they insist on it and badger users multiple times in order to force them to upvote
 
It's okay, I said I don't think it's appropriate to ask that and he removed it.
 
That's pretty much all you can do
 
Thanks for your input
 
although engaging these users in comments might just lead to revenge downvotes
that's your risk
 
3:29 PM
Well I didn't downvote so there's little chance of that happening (and it didn't)
 
3:39 PM
Pedantry corner: independent of whether it's against the rules or not, I wouldn't call it "fraud" because fraud requires deception
 
Maybe not vote fraud but some kind of mixture between serial voting and vote fraud. In the same sense that asking your friend to upvote your answers is against the rules.
 
@NicolasGervais no, asking is not against the rules
it's ultimately the voter's responsibility to vote correctly
if you offer money for an upvote, that's against the rules
 
I think it would be nice if nobody did it, but I can't muster enough ire to get mad at people that currently do
To paraphrase meta.stackoverflow.com/questions/262345/…, "don't ask your friends for votes, because if it significantly skews the system, that's going to be a Problem"
 
I sometimes post a comment like this:
Note that it's considered rude on Stack Exchange to ask people for upvotes. Besides, the OP doesn't have enough rep to vote. OTOH, it is ok to explain to new OPs about accepting an answer. — PM 2Ring 2 days ago
 
Seems like it's "not recommended" rather than flat-out disallowed because a complete ban would clog the report system with small time first offenders
Easier to let the community enforce soft norms and let the voting ring detector do the heavy lifting if things get out of hand
 
3:56 PM
@PM2Ring I don't even like to tell askers to accept. Often happens with multiple answers...
 
If the upvote request is in the answer body, you could edit it out, with an appropriate diplomatic edit comment. If it's in a comment, you can be anonymous & just flag it as "too chatty".
 
I'll nudge for an accept it if the OP replies "thanks! This solved my problem" and no accept seems to be forthcoming
 
@PM2Ring "too chatty" - when's the last time you tried flagging a comment? :p
 
Well, I consider it poor etiquette to flat-out ask for an accept. OTOH, if the OP is newish it's ok to inform that accepting is a thing, and to ask them to please consider accepting your answer, if you feel confident that your answer actually deserves it.
 
@JonClements that reminds me: I was told by a mod that the automatic "move this conversation to chat" feature on main triggers that the OP will be able to talk in the room, even with <20 rep. Which covers a lot of instances of your feature-request
 
4:00 PM
FWIW, I've posted comments about accepting on other people's answers, and on the question itself, when the OP is new. Or newish, with a poor track record for accepting.
 
@AndrasDeak well... if someone just spent 5 minutes fixing the actual bug in chat that ROs can't grant write access to everyone as the text implies you should be able to - I'd be happier :)
 
@AndrasDeak If only we could trigger that process without posting half a dozen back & forth comments... I think if a 3rd party joins the conversation it takes even longer for the auto-chat to be triggered.
 
@JonClements Same...
 
@JonClements A few hours ago. Ok, I know that reason is now obsolete, but I forget the new ones. :)
 
@PM2Ring yeah, I rarely see the feature these days
 
4:04 PM
mods can grant write access to users with < 20 rep just fine... so whatever code does that should be if granter is mod or RO... I can't imagine it's that big a change... but - can't rush these things :)
 
Yeah, I'm aware..
it came up with a more ambitious FR
 
I can imagine a few less-than-ideal system designs that would make this kind of change hard
Mostly any design where chat and the main site are developed by separate teams that don't talk to one another much
 
Here's a comment-chat (about QM) I got involved in earlier today: chat.stackexchange.com/rooms/111711/… It looks like it got triggered after 6 comments. I assume that's the same on all 3 chat networks.
 
"Yes, mods do have the can_grant_write_access property set to True... No, we can't do the same for ROs, because that class inherits from ChatUser, not SiteUser, and only the latter has the property"
ChatUser and SiteUser have no common base class other than object, naturally. Why would it be otherwise?
 
Im pretty sure there was some minor change to chat announced recently. I think I saw it on MSE, or maybe the MSE Tavern. But I don't remember the details, and I don't think it'd be easy to search for.
 
4:12 PM
I'm not sure if it's done yet or still in progress but I know there's something regarding access control and the Teachers Lounge...
 
There was a discussion in the Tavern a few days back about how you can ping people who haven't been in the room for ages, by replying to one of their ancient messages in the transcript. Some people felt that was a bad thing because it could be abused. And now there's a MSE question about it, posted by a U&L mod: meta.stackexchange.com/q/235322/334566
I'm pretty sure a dev or CM did mention a minor change to the chat software. But maybe I'm imagining things. :)
Oops. That's an old question, with a fresh bounty from terdon.
 
Theory: any chat feature requiring changes to the UI are impossible because they accidentally lost the pre-minified pre-obfuscated version of the client JS
 
4:41 PM
Hello!
Is there any builtin way of doing more methods inside a class that does the same thing but with different names?
let's say I have self.counter and I want 3 getters : getCount, getIndex, getNumber. Is there any shortcuts?
 
@CătălinaSîrbu Hi. Do you want those 3 getters to do exactly the same thing?
 
yes
just return the self.counter. I always forgot the method name and I have to check the name. This way the chance to forget the function is lowered:))
and now I forgot I use python and that I can acces the member without any methods... oh..
 
@CătălinaSîrbu :)
 
but I would like to see how would you best implement what I asked for earlier
if you can still answer
 
I was just about to ask if you just wanted to fetch the attribute, or if the getter does some processing.
 
4:47 PM
the answer is that I only wanted to get the attribute value, yes
 
class Thing:
    def __init__(self):
        self.counter = 0
    def getCount(self):
        return self.counter
    getIndex = getCount
    getNumber = getCount

x = Thing()
print(x.getCount())
print(x.getIndex())
print(x.getNumber())
 
yayayay thanks :) It looks amazing
 
@CătălinaSîrbu Sure, that's easy. You can set up synonyms like that as class attributes. OTOH, it's probably not a great idea, since it adds complexity to your class without much real benefit. But anyway... Ah, I see Kevin beat me to it.
 
@Kevin shouln't there be def getIndex or def getNumber ?
 
No need
 
4:50 PM
@PM2Ring thank you !
 
@CătălinaSîrbu You could create new methods with def that call getCount, but that's less efficient.
 
ok so I think that I just assign a pointer to that method (already existent) instead
 
@CătălinaSîrbu Basically. Although Python has no concept of pointers (unless you mess around with ctypes). Just think of it as binding 3 names to the one method.
 
No one was going to say: getCount = getIndex = getNumber = lambda self: self.counter then? :p
 
@JonClements please no :))
 
4:54 PM
:Slaps puppy with rolled up newspaper:
 
@PM2Ring yes, sure
 
Ironically my thoughts having 3 different get methods that do the same thing :p
@PM2Ring meep meep...
 
@CătălinaSîrbu BTW, here's a great article on how names work in Python. nedbatchelder.com/text/names.html IMHO, it's essential reading for everyone using Python, especially if they're coming from another language (like the C family) that doesn't use Python's data model.
 
Thank you!
 
No worries.
 
5:02 PM
class Something:
    def __init__(self, value):
        self._counter = value

    @property
    def counter(self):
        return self._counter
Might also want to consider a property and remember the name of it :)
/me waits for the newspaper.... :p
 
5:26 PM
i am afraid I do not know the @property stuff @JonClements
 
5:48 PM
@CătălinaSîrbu if you google for "Python properties" the first several hits should be helpful
 
@Wayne oh hey.... been a while - how's things?
 
6:07 PM
Hey @JonClements o/ cbg!
Doing well - life kind of took a bit of a detour, heh.
How's the Clements Corner of the world?
 
@WayneWerner as weird as ever :)
 
Heh. Yeah - it's a bit on the interesting side because we had intended on selling our house (which we did) and getting in an RV and touring around the country. Since I work remote, it was going to be great, cause we could see the country, I could attend/speak at different conferences...
Apparently the timeline had different plans.
instead of having close living spaces, and just going out and experiencing all the things, instead we have close living spaces because there's nothing else to do (:
We decided that we really enjoyed the process of cleaning up our old house for sale, so we decided to get a new house that needed to be fixed up. So, that's the current state of the (not so) Wandering Werners
 
yeah... 2020 is putting ruin to many plans for many :(
 
6:28 PM
[re-cbg] Hi both. It has been a weird year so far, hasn't it?
 
cbg @holdenweb

True story
I saw a recent thing about waterfalls flowing up in Australia? Because it's 2020 and of course they are.
 
@holdenweb heh - just a little :p
 
6:44 PM
@PM2Ring that's the question due to which the RO write access thing came up
 
@WayneWerner I would question this...but usually when I hear something absurd about Australia, it's only half true, and the truth is crazier.
 
whenever I hear "waterfalls flowing up" it usually means torrential wind that whips up against the waterfall
torrential might not be the best word but you know what I mean
I hope they've accounted for the fact that in Oz everything flows upside down
 
@AndrasDeak I connect torrential with water, but since such events probably involve precipitation it sounds right to me.
 
7:01 PM
Weird: thought MacOS had gone crazy, but apparently somehow my When switching to an application, switch to a Space with open windows for the application setting got reset. ???
 
@toonarmycaptain yeah, and although the wiktionary talks about "flow" I suspect most people don't think of wind as a flow of a fluid :)
 
@AndrasDeak True.
 
The town of Mt Gambier in South Australia is situated on an extinct volcano. It gets its water from Blue Lake (a crater lake), which turns extremely blue in summer. If you fill a small white hand basin with water, you can see the blue tint. Allegedly, it's harmless. :)
 
@PM2Ring Heh. Allegedly. Wait...isn't SA the geologically inert area they've suggested/actually decided to store radioactive things? An "extinct" volcano doesn't sound as low risk as I thought the area was.
 
7:16 PM
@toonarmycaptain Mt Gambier's right on the eastern border, quite a distance from the region proposed for radioactive waste storage. Mt Gambier last erupted several thousand years ago, they aren't quite sure when.
 
@PM2Ring I mean, I hear you. I just understood the "inert" statements to mean much more than "eh it's been a few thousand years since this volcano erupted" - half life of plutonium, 24500 years?
I mean, WA had that alleged nuclear incident in '96, but this is different. Besides...wine country and nuclear storage already has it's concerns.
 
@PM2Ring I'm strangely reminded of something I read a while back... a local council ended up dyeing an area of water black because despite it being left over from an old mining place/something like that (I forgot the specifics) and the signs saying "This water is not actually water - it's very alkaline heavy and liable to cause injuries" - people still did so as it looked really nice being the blue it was...
 
Speaking of radioactive stuff, I've just been chatting with a German guy who decommissions nuclear reactors.
in The h Bar on The Stack Exchange Network Chat, 2 hours ago, by Faded Giant
In only partly related news: We pulled out our old core barrel and lower core support yesterday.
 
dyeing to prevent dying...
 
7:28 PM
and people taking a roll of the di(c)e about it...
 
@JonClements Oh dear. I guess black's a good choice.
 
Now they have to keep all the goths away...
 
On the South Australian salt flats, along the coast, there are shallow pools that turn bright pink, I think from bromide & iodide. The pools aren't deep enough to swim in, and the salt content is extremely high.
 
@PM2Ring And they don't taste good..
 
We have lakes that are rather dark brown, from the high tannin content due to the surrounding melaleuca trees. Those lakes are safe to swim in, though. The common name for melaleuca is Tea Tree, and Tea Tree oil is sold for its antiseptic properties.
 
7:39 PM
I am trying to insert the age values as a condition tensor for the first layer of the generator for the cDCGAN. I am following this (https://arxiv.org/pdf/1702.01983.pdf) and this (https://arxiv.org/pdf/1611.06355.pdf).

how to concatenate the latent vector `z` of `torch.Size([64, 128, 1, 1])` and age values as one-hot vector of `torch.Size([64, 6])` in the first dimension and give to the first layer of G?
sorry for such a simple query, I am learning PyTorch and simultaneously learning GAN.
 
When the covid-19 thing started, it became very difficult to buy Tea Tree oil, because so many people wanted it to make antiseptic hand wash.
 
@AnilSarode I'm not a ML person. Does it make sense to think of the one-hot vector as something that has shape (64, 6, 1, 1)?
in that case I could imagine stacking them to get a (64, 134, 1, 1)-shaped tensor
in any other case I'd cautiously suspect that you can't combine those two tensors, but I don't have the domain knowledge to be certain
 
@AndrasDeak Yes, I know torch.cat () will not work as it required the same dimension for tensor
The one-hot vector represent 64 binary values for six categories.
 
I understand what one-hot encoding is, what I don't understand is what the trailing singleton dimensions of the z vector represent
 
z is random noise noise = torch.randn(batch_size, nz, 1, 1, device=device)
batch_size=64
nz=128
I created the tensor of shape (64 ,6, 1 ,1) and torch.cat() also works. But now the question is does it is a correct way to combine and feed to the Network? I will check. Thanks for your attention.
oh to create (64 ,6, 1 ,1) I used .view()
 
8:03 PM
@AnilSarode that's the part I can't help with without knowing what those two size-1 dimensions represent. From what I've seen here and there about neural networks it might make as much sense as anything else
 
 
1 hour later…
9:30 PM
@PM2Ring I swam in a river like that when I went kayaking in the US on a conference. It was cool, up to the point that they told us the river was full of snapping turtles after we were encouraged to jump in... and then you could see them :P
Conferences were kinda fun, minus the water demons. I don't know whether programming conferences are similar to the Chem Eng ones, but they put on some great days out for networking.
 
:50194705 it was actually meant as good humored and slightly ironic remark.
 
it didn't come across as such
 
@AndrasDeak no honestly, that form of remark inviting to critical self-reflection is a big classic in Spanish and Portuguese literature. You'll hardly read anything devoid of such passages.
 
hi
 
9:50 PM
@Speedy hello
 
i'm portuguese and we do have good literature
 
user13869189
hii i want to convert '2018-09-05T12:00:00.005000Z' to 20180905T12000000Z how can i do it
 
@Speedy Very True!!!
 
@seismo for non-standard formats, you could use dateutil.parser.parse. As for the formatting to a new string, the datetime.strftime should suffice.
You may have to manipulate the parsed datetime if the removal of milliseconds is intentional.
 
10:09 PM
yo guys
my algorithm runs multiple non-linear scipy optimizations, and it takes 99,76% of my runtime. any tips on how to make it faster?
I've seen some stuff about cython and multiprocessing etc, but I'm not sure it applies because I think scipy is already making most of the calculations using C (I may be mistaken though). Also I'm processing each optimization at a time, not all of them at once.
 
@PedroSpinola profile your code properly. If it's really scipy functions running most of the time there's not much you can do except use solvers that are faster but less accurate/less general etc
 
I did profile it. Actually I wasn't accurate enough. Around 20% of runtime is spent on my solver function (inside my algorithm, not differential_Evolution.py)
 
diff. evolution should also be able to run on multiple cores; can't it?
 
How did you profile it?
 
using cProfile and pstats
 
10:17 PM
Diff evolution has the workers keyword argument that handles multiprocessing for you. Not much else you can do except optimize your function.
 
tbh I know nothing about cores, and processing, etc.. will read about it. but I'm trying to run the test with all cores, by scipy doc
yeah workers = -1
 
That's all you can do. Differential evolution races a bunch of optimizers against one another, so it trivially parallelizes.
make sure you really need a global optimizer
 
workers -1 gives me an error and the script halts
but I suppose I'll try this later on on linux
 
nevermind, found it
@PedroSpinola "an error" is never enough information, we've told you that already
> Requires that func be pickleable.
probably that
 
exactly that. and probably because some of my constraint functions take one other argument apart from x
 
10:24 PM
@PedroSpinola or you have lambdas or something. I've had a similar problem and I could find a solution in 5 minutes of googling (I had to define a helper class)
google your error message with site:stackoverflow.com
 
good tip! I'll take a look at it
 
# class instead of an unpicklable lambda for use in multiprocessing.Pool.map:
# TODO: see if functools.partial would work instead; clearer
hmm, good idea, I really should check that after 3 years :D
 
:D
yo btw.. is there any way to tell my differential ev call that I don't care about precision after 3 decimal (0,000), to maybe make it faster?
I tried messing with some parameters like tol and atol but doesn't seem to be it
 
those should be what you need for that
19 mins ago, by Andras Deak
make sure you really need a global optimizer
 
10:40 PM
yeah I checked with my partner, who is the science guy, and he said we do, unfortunately
 
I've never used pstats but is cProfile telling you that almost all of your time really is in a single function call?
 
Did you experiment with the available methods etc.? There's not much more you can do I think, other than finding another solver.
 
around 20% of time is on a single function which is a constraint, than 79,24% is inside diff_ev.py
 
@roganjosh that's prefectly plausible if it's a number-crunching code that does 10k function evaluations
 
10:41 PM
10k? it does millions
 
But is diff_env actually part of the library or your own function that calls the library?
 
per iteration :P
 
@AndrasDeak sure, I'm just trying to rule out the fact that they may have some for loop in the same function that happens to call the solver
 
I used shortname josh. Mean the scipy library itself
 
Because, if it's not that, there aren't many options :(
 
10:43 PM
When I optimized a Monte Carlo code of mine I stopped when 95%+ was spent in code that I squeezed through numba (or somebody else has optimized already)
 
I'm still working on optimizing this specific function that takes 20% time, but it looks like this for the moment:
def efic(x, mm):
	shortcut = pos['m'][mm]
	loga = np.log
	e = [0] * pos['qt'][mm]
	for i, her in enumerate(shortcut):
		sob = shortcut[her]['sob']
		e[i] = (sob + (1 - sob) / (1 + exp(shortcut[her]['curv'] * (loga(x[pos['h'][her]['i']]) - loga(shortcut[her]['d50'])))))
	return (1 - np.prod(e))
 
It will depend on how you've formulated the problem, too. You could try Google ORTools but there's really nothing to know whether you formulated the problem correctly in the first place
 
@PedroSpinola How long is shortcut?
 
I considered vectorizing this dictionary info too, to work on it directly with numpy
inside this func
 
So you do have a for loop
 
10:45 PM
what u mean Andras? It's just a shortcut for a dictionary reference
 
@PedroSpinola yes, do that if it's plausible. At worst you tried something that's slower
@PedroSpinola you are looping over it. How long is the loop? 2 iterations or 2000?
 
varies from 1-7
actually this is the function upon which I build the constraints
 
I don't really use cProfile, but line_profiler would show the number of function calls too. If the function is reported to take ~93% of the runtime, is that the aggregate of every function call?
 
because the number of constraints vary each time
 
So often finding myself writing

    count = 0
    def callback:
        nonlocal count
        #something
        count=count+1

I wonder if there is some way to avoid the nonlocal keyword?
 
10:47 PM
@Mikhail by not using nonlocals/globals
 
lol
 
if it's a callback then probably not
@PedroSpinola then the loop is fine
 
it is cumulative yes rogan.
20% runtime, that funct
 
you can micro-optimize by putting loga=np.log into the function's signature
 
In which case, your vectorization thoughts might be along the right lines (without using np.vectorize. Leave that well alone)
 
10:48 PM
Would be cool if you could do something like parent.count, we should write a PEP
 
I'd expect the lookups to be of comparable slowness, so you can try reducing their number
@Mikhail you can, if you have access to a parent
if you don't have explicit access to a parent, then nonlocal is exactly what does what you want to do
 
I hate that stuff aswell @Mikhail
 
Yep I know, but there should be some magic keyword that drops you into nonlocal space
maybe nonlocal.count
 
Good, I won't use np.vectorize then, I appreciate :)
 
@Mikhail ugh, no
@Mikhail that's not how python works
 
10:51 PM
But it could be how python works :-)
 
good Andras, I appreciate :)
 
Hot Meta Posts are back, am I dreaming?
 
@Mikhail It seems Python have a very strict scope policy by design
 
@PedroSpinola it's not so much about making sure you don't use crap helper methods with misleading names as much as it is being able to abstract the problem to just arrays
 
Except that they have the nonlocal keyword, so clearly not
 
10:53 PM
A well designed language by definition has elements that allow you to violate it's design policies, since it's well designed
 
yeah, I have worked with easier languages regarding scope and name assignment in the past (I'm still a beginner though)
you're telling me to not use helper methods @roganjosh?
 
@PedroSpinola he meant that np.vectorize is misleading crap
 
ohh I see
I know
that's what I meant with "vectorize"
 
@PedroSpinola By the time my problem gets to the solver loop, there are no custom classes. Just array operations that mimic attribute lookups
By that point, everything is a custom dictionary of constraints represented as arrays, or just arrays
 
And if you have arrays then e[i] = (sob + (1 - sob) / (1 + exp(shortcut[her]['curv'] * (loga(x[pos['h'][her]['i']]) - loga(shortcut[her]['d50']))))) can be passed through numexpr that might lead to considerable speedup (and memory efficiency)
but you have to banish all those dicts for that first, not trivial to do so keep using your version control
 
10:59 PM
I suppose I could keep some classes, since I could just convert all their attributes to arrays or dicts of arrays, but that confuses me. It looks very little like the problem that the user defined
 
Yeah I have a specific dictionary for each time the solver runs already in place. So should be easy to transform each constraint into a matrix (not an array, because it's multidimensional), and then use each matrix for each particular constraint
numexpr(), awesome!
 
@PedroSpinola you have it the wrong way around. Array is more general than matrix. Matrices are 2d and the numpy matrix class is deprecated.
@PedroSpinola it's numexpr.evaluate(), but yeah
 
ohhh alright, I didn't know that. I can build a matrix using an array of course
 
@PedroSpinola when you say "matrix", do you actually mean a numpy.matrix instance?
 
I'll tell you guys in a few days how much speed reduction% I was able to achieve, after 'vectorization', and all cores activated
 
11:02 PM
@PedroSpinola depends what you're trying to solve. Depends what a row represents. Depends how many things-represented-by-rows need to change in each iteration of the solver
 
I have it figured out already, should be pretty simple tbh. The problem ain't very complex.
 
I guess we'll never know
 
The complexity stems more from the fact that each time it runs, the parameters are different (and the number of constraints too)
but inside the solver things are pretty straightforward
Wanna see it?
 
>>> a, b, c = np.random.rand(3, 100000000)
... assert np.array_equal(a + b + c, numexpr.evaluate('a + b + c'))
... %timeit a + b + c
... %timeit numexpr.evaluate('a + b + c')
363 ms ± 5.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
309 ms ± 7.54 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
 
@PedroSpinola Well, if you figure out a way of abstracting the problem, as I suggested, it wouldn't matter too much
 
11:04 PM
that's as much numexpr can do, and it's mostly faster for large arrays or many of them
 
Nice, pretty significant increase!
 
I'm not suggesting that it's easy. It took some months for me to be able to abstract my current problem away to arrays that could reliably handle all the constraints I could envisage. I don't sleep well.
 
I'm very good with abstractions
But I suppose my problem is way simpler than yours lool
 
One last try: do you actually have numpy.matrix instances?
 
no no Andras
nothing with matrix
 
11:07 PM
OK
 
I was just considering it for turning all dictionary lookup into numerical info directly, to be able to use numexpr.e()
 
oh yeah, and numexpr can also do reductions (one outermost reduction, to be precise)
>>> a, b, c = np.random.rand(3, 100000000)
... assert np.array_equal((a + b + c).prod(), numexpr.evaluate('prod(a + b + c)'))
... %timeit (a + b + c).prod()
... %timeit numexpr.evaluate('prod(a + b + c)')
440 ms ± 4.14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
281 ms ± 3.02 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
@PedroSpinola you don't need matrices for that. An array can do almost everything a matrix can, and more (and what it can't do is not what you seem to be needing now).
Rule of thumb: don't ever use numpy.matrix instances.
 
good to know. I hate matrix, this is why I didn't implement this earlier (didn't consider an array could serve just as well - or better)
 
Shameless self-plug: stackoverflow.com/questions/53254738/… . Matrices might seem tempting for linear algebra, but they're not worth it.
 
my partner was asking me to use a matrix as data structure for my algorithm lol
since the beginning
because he thought it would be faster
 
11:12 PM
tell them I ordered you to use a multidimensional array instead
 
no, he wanted me to store all info in matrixes
would be faster, of course
but would be VERY difficult to work with
 
You can do the same thing in 2d arrays. It won't be slower than matrices.
 
dictionary keys save my day
yeah I get that now :D
 
You or your partner sound a bit confused. The argument seems to be "matrix faster than dict", whereas it should be "numpy array-like faster than dict", and that's also a statement that's only true under certain circumstances
but I'm done now, I think I've made my point :P
 
either way I'll be working with dictionaries unless I really need to optimize some function
he knows nothing about dictionaries or python
he's a science guy lol
 
11:14 PM
@PedroSpinola that's a good approach (except your last hour here sounded like you really needed to optimize a function :P)
@PedroSpinola so am I
 
@PedroSpinola Yeah, they have some wacky ideas :P
 
or should I say, so are we
 
he thought my script was gonna be vry slow. neither did he now that a single function amongst 40 functions would consume more than 99% runtime lool
and that's the one I'm optimizing now :D
I'm not so much because I don't work with science on a day-to-day basis, so I forgot a lot of fundamentals :P
sry
 
For what it's worth most of what general "The Science" brings to the function optimization table is an analytical approach and attention to detail, nothing more.
 
in this case, he has the maths part of it. the formulae and also he did select the solver method
I don't even know what a global minimum is, and I don't care lol :P
 
11:18 PM
If he doesn't know python are you sure you can rely on his judgement?
 
Not knowing what a global minimum is, is likely to be trouble for you, btw
 
he did some research, we tested some methods together. he was using an excel solver to do it, before
 
I'd definitely cross-check various solvers under test conditions to see if something else can be used. Differential evolution is a laser cannon. Sometimes you need to remove a mountain, but often what you really need is a laser pointer.
@PedroSpinola that excel bit only further proves my point
 
But we couldn't help with this beyond such generalities anyway, so that's it
 
11:20 PM
A global minimum should be some common-ground between you, or you really don't understand what each party is trying to do
 
I'm gonna have to trust him on this one, also he's dooing masters so he got help from his orienter (professor)
 
famous last words
But it's all your call :)
 
I appreciate the coonsideration, anyhow :P
 
Indeed; best of luck on the project
 
we science guys tend to approach problems with "trust but verify", which is why we're being so problematic here
 
11:22 PM
I know it has to do with the graph of the function and stuff, and local minima vs global minima etc
I'm pretty sure he knows what he's doing.
But of course we'll doouble-check later on
 
sure thing
 

« first day (3588 days earlier)      last day (1345 days later) »