« first day (3942 days earlier)      last day (989 days later) » 

1:04 AM
@roganjosh just curious, are you the only dev who does all the coding in your org or at least for the beatroute package?
 
 
3 hours later…
3:51 AM
how can i peek at the 2nd highest element in a heap?
 
 
3 hours later…
6:36 AM
@python_user I suppose I'm the lead developer. I brought vehicle routing experience with me when I joined the company and we're still expanding rapidly so every customer project was basically custom code in every case. I decided to try build some tooling, so I built the solver server and interface, and the engineers deploy parts of the stack on kubernetes. People on other projects that now use it will contribute code when they need new features or I build them in for them
 
that makes sense, cool job though ngl
 
I also had customer work for 4 customers myself, involving things like inventory management and forecasting, so I don't get as much time as I would like to work on tooling. I'm trying to reposition myself to get less of that and focus on beatroute and also a machine scheduling package (yet to be started)
 
your hands seem to be full for the next couple months
 
Years I imagine :P We're growing 100% per year so we're all fully loaded. I think the Data Science team has doubled since I joined to something like 55 now and I've not even been here a year
 
are you hiring from India? ;) (only half joking)
 
6:42 AM
We are
@python_user Open positions are at the bottom here
 
it is kinda early for me for a job change but thanks I will keep this at my watch
you guys also use bamboo hr :D
 
 
3 hours later…
9:38 AM
@Diego Hello. Please see our room rules specifically in regard to waiting 48 hours before linking to a question on main
 
 
2 hours later…
12:06 PM
Hi guys, I am having some problems with JAX packages in python, is someone more expert than me with respect to Automatic Differentiation?
 
@John_maddon please don't ask for help here with fresh questions on the main site as per our rules
 
@AndrasDeak You are absolutely right, sorry.. it's just I am in trouble with this from 2 weeks.. :(
 
No worries. Traffic is low on weekends anyway. If you're still stuck in 2 days we'll gladly look at it.
 
12:22 PM
@AndrasDeak Okok many thanks, I really appreciate this forum, hope to have some answers :), thanks anyway!!
 
good luck
 
1:00 PM
@roganjosh ugh... got distracted this morning... enjoy your BBQ today and we can work out some play time next week or something... sighs
 
1:25 PM
@JonClements oh how the tables have turned...! (I really have no right saying that given that it was a gift :P )
 
just being nice in delaying your humiliation is all... :p
 
I couldn't get tomorrow morning off as my boss is away so I'm not travelling to the 50th. I don't fancy a 7am drive back. I'm gonna head to the pub shortly. A remarkably productive morning, though; 5 issues/enhancements closed. I think I did a good
 
sounds like an achievement worthy of a pub visit :p
 
 
1 hour later…
2:46 PM
hello fellas, how you doing? :) long time no see!
 
3:02 PM
howdy
 
 
4 hours later…
6:59 PM
@ParitoshSingh pretty nice, although trying to answer questions again can be daunting because I've lost the knack :P
 
hey guys, would someone try and give me any ideas to optimize this cython function? I'm a little bit lost with the self. stuff (which are common python lists)
 
Have you set annotate=True in your cythonize function?
 
josh! :)
 
Hola
How have you been? :)
 
quite a few hurdles in private life the few last months, but at least my project is slowly but surely going forward :D
what about you?
 
7:15 PM
Ah, I hope all is ok. At least you're still moving forwards :) Yeah, not much to report here tbh. Plodding on but all good!
 
Boring pandemics, right?
Ohh I didn't know about annotate. Taking a look at the html right now
 
Indeed. And as it happens, I've been doing a lot of reading around cython recently while I plan out a package so I'm curious to see the result if you can link to a pic on imgur or something
There's a few things in that where I think it's calling into Python but I'll see whether I've guessed them right :P
 
there you go
I tried to cdef self.dataPoints list but it throws an error
 
7:32 PM
Almost all of it is in Python, then :P Can you reduce the code down to a standalone example? You haven't typed y for a start. Once I have something to run self-contained, I'm in the mood to play around a bit on this as practice
I can see it's in a class, too, but I don't know what exactly those attributes are
Even if it's just a pure Python example. That'll be a good starting basis
 
this function is being used by a class method that's being invoked to create a NonLinear constraint function to be added to constraints list and then run differential_evolution (scipy).
all those self.Stuff are common python lists, and they contain some data I need in the computing
y is a subsection (not sure if this is the right word to describe it) of x, where x is [x] on diff evolution
 
Fair enough, but it's still a lot easier if you just gave a MCVE with code I can copy/paste and run
 
it containts from 2 to 7 float numbers
it's so messy right now lmao
but I will build it
only it will take a while, not sure I'll finish today
 
It shouldn't be many more lines than you've written. I only need stand-in objects that represent what you're actually working with. But if it isn't going to be tonight then just ping me when it's done. I'm only going to play with this one function, not cythonize your entire code :P
 
the problem is all this data comes from sql in the first place
 
7:42 PM
Then give a list of tuples that looks something like what you get back
 
so I'll have to hardcode enough of it, to make a working example
@roganjosh back from what, exactly?
 
The MCVE only needs to represent the spirit of what you're trying to do. The values can be garbage, as long as they're the right type as what you get
 
alright, I'll do it.
 
@PedroSpinola Back from an SQL query. That would be (usually) a list of tuples. Unless you're using an ORM. But this is probably too much detail for an MCVE anyway. Just make the values up, but give them the correct type
 
I was using numpy before, but it was very slow, so all the data structures are being converted .tolist()
 
7:47 PM
That statement in itself raises alarm bells. Numpy is not slow; it's only if you misuse it that it becomes slow
If you're cythonizing stuff to get around numpy slowness then you're on an XY trajectory
 
8:17 PM
It can work in some cases, though. Looping a huge numpy array is slow; doing the same in compiled code will be fast again :P
 
True. That's also the reason that I'm looking at cython. But ditching numpy in favour of cythonizing a list comp here raises alarm bells for me
 
since we're overengineering solutions, we could also just use numba!
that way you dont have to cythonize it, and you still get to iterate and gain time
 
Depends whether it's all supported by numba :) I must admit that my research is showing that the single decorator does an awful lot of fancy stuff, though
 
8:41 PM
oh yeah, as long as the dtypes are figured out, almost everything should be a numba friendly. i will say though, there's a quite real possibility that iteration still falls short of vectorization if that's feasible. (i just dont have the mental capacity to figure out what the code is doing or what not. in my defense, it's the end of my weekend) :P
 
I tried jit, njit and jitclass for numba, but didn't seem to help. This is obviously my first time doing such things so I'm experimenting a lot still. I tried without list comprehensions, tried the non-logarithmic function, etc. I was probably using numpy wrong to be honest, but because it's never too many data inside each np.array, some people told me I shouldn't be using it.
I also tried PyPy.
I have an almost-working MCVE that will run only 1 time when in real code this would run thousand's of times... Also I'm not being able to get the solver to return a success
I tried to reduce it as much as I could, but it's still big @roganjosh
the eficN class method in this version is not cythonized
(sorry for my ugly code)
 
You haven't posted it. But as I said, we're only looking at one function. I don't care what function generated the output
 
additional info: there's never more than 7 elements in a mixture (or submixture)
 
Fair enough, though, it does run
 
I can add some more context if it's needed
 
8:50 PM
I only wanted to try cythonize one function. You could have stripped the entire context away and just given a representative input to that particular method and an expected output.
1 hour ago, by roganjosh
It shouldn't be many more lines than you've written. I only need stand-in objects that represent what you're actually working with. But if it isn't going to be tonight then just ping me when it's done. I'm only going to play with this one function, not cythonize your entire code :P
 
@roganjosh yeah, you sometimes have to replace things like arr.reshape with np.reshape or arr[:, None] with an explicit reshape, but it's normally very easy to numbify decent numerical code
 
I wish I had decent numerical code LMAO
I see josh, I'm sorry.
 
No need to be sorry. I'm just not going to plough through all of that code, so I'm sorry
 
@PedroSpinola small arrays are not necessarily a reason not to use numpy. Numpy still offers a lot of functions and methods, as well as powerful vectorisation capabilities. If you're not using any of that either then yes, you probably don't need numpy.
 
Yeah of course, it's a lot. If I provided the data structure printed, an example input, and an example output, would it suffice josh?
 
8:55 PM
@MisterMiyagi I came across this old comment from BDEVIL
> IIRC we debated a similar issues when descriptors were added and decided that it's a perverse style that we just won't support. If you really have this you can add # type: ignore.
Context is instance.prop = obj where type(obj) and type(instance.prop) are different. Brilliant.
 
@AndrasDeak I see. I have ditched it only for now, but after I do some serious studying I'll try to use it again. It's hard for me because of all the math and matrixes I'm not very used to, and there's also a lot of funny stuff with data types :/
 
If there's math and matrices then numpy is probably the right tool
and if you don't know linear algebra then matrices alone can be scary, even without adding a python framework into the picture
 
Maybe, but I need to get that vectorization completely right first, and it's hard I feel lost with current knowledge and practice (shouldn't have tried to do my first vectorization with such a complex formula, but that's what life threw at me)
yeah. matrices alone are scary tbh. but I'll overcome it eventually when I get enough time and patience to study it
 
I haven't looked at your code yet
 
I was just experimenting with optimizing what I got now so I learn some cool stuff before looking at vectorization again. Original test script was taking 1h40 min to run and now only 22 mins, so I'm quite happy already :)
 
9:02 PM
Did you profile the code to see where bottlenecks are?
 
@PedroSpinola What I'm trying to do is break the problem down. We were looking at a single function originally, right? I don't care how it fits into your problem as long as I know what general rules I have to abide by
 
yes! and 98.2% of time is spent on eficN function which runs more than 70 million times
@roganjosh alright! I'll prepare a new paste
 
This is more complex than I'd like, but it's the first example to hand. Here's an MCVE I passed to a colleague for them to play with. The data is nonsense but it is representative
That's only a subset of what we were trying to solve. I just wanted to give him something to break
But the point is, the data that feeds the query is entirely removed from the actual problem because I just made it up. He wasn't sure about the SQL so I just gave my colleague something representative. It doesn't matter how it fits into the actual program, it just illustrates one part in isolation
 
yeah now I do understand. I thought you needed the entire class because of the self. stuff
but here you go, I think it should suffice
un-cythonized:
 
9:18 PM
Much better, thanks. I just flipped the definitions to be before the function and had to deal with annoying line break issues that dpaste introduces. But I can run this now, thanks. It might be tomorrow before I can get back to you as it's getting a bit late now
 
great :) just remember those lists originally are class objects
np my friend, I have to go aswell
tomorrow we talk
cya ;)
 
rbrb
 
9:57 PM
@roganjosh to give your Cython a run for its money, here's a pythran version. It brings down the runtime from 17 μs to 1.2 μs for the small example, which is probably not representative of the large problem. cc @PedroSpinola
the same vectorized function runs in 16 μs or so
too many small temporary arrays
 
Challenge accepted... but not tonight :P
 
10:18 PM
And just to mess with you: the emitted C++ code is 206 lines (13827 characters) :P
 
10:36 PM
Actually, I messed up the original timing by converting the inputs to arrays. It's 3.9 μs with native python.
 
11:09 PM
@Rahul2304 hello. Please don't ask for help here with fresh questions on the main site as per our rules. And two links to the same post would always be unnecessary.
 
Hello Andreas, my apologies I am new to stackoverflow
 
@Rahul2304 it's alright, these are our local rules. Welcome :) You should add the generic tag to your question for visibility.
 

« first day (3942 days earlier)      last day (989 days later) »