« first day (3933 days earlier)      last day (1001 days later) » 

3:28 AM
cbg
 
4:04 AM
Hi everyone!
-2
Q: Python Mysql stuck on cursor.execute()

Robertas KirkaI am executing sql select statement with python and sometimes execution get stuck. I can see on mysql server that query is in state "Writing to net" after some time on mysql server query disappears, but python is still on execution. What can be wrong? Can it be problems with DB or python configur...

I am having exactly the same problem. Can someone please take a look and maybe have any ideas why it's happening
I am using oracle db, when running multiple queries saved in a query, some runs, sometimes some gets stuck, on pressing ctrl+c, it'd give some results instead of terminating
dict_query = {'query1': 'Sql query goes here', 'query2': 'another query is here', ........}

and so on, then I use a for loop to run those queries and fetch the results in the same dictionary
 
you might want to verify the server's actually not overloaded, see: stackoverflow.com/questions/20583082/…
 
my_conn = conn_db(host, uname, pwd, srv, port)
cur = my_conn.cursor()
prev_tbl=''
total_cnt=0
for key in query_dict:
s =time.time()
if(query_dict[key]['table'] != prev_tbl) :
cur.execute(f"SELECT query count statement")
total_cnt=cur.fetchall()[0][0]

cur.execute(query_dict[key]['query']) #in key as a dictionary, query statement is stored in 'query' key
query_res=cur.fetchall()[0][0]
e = time.time()
query_dict[key]['query_tcount']=total_cnt # query count result to dictionary
query_dict[key]['error_result']=query_res #query result added to dictionary
@metatoaster in that solution, it says the server might be overloaded and it rejected some queries, but when I press ctrl+c when it's stuck for a long time, it shows the result for the next 3-4 queries (and the time to execute & fetch was also about 10-11 secs).
So, if the server is overloaded, and it rejects the queries, should I be getting the result on ctrl+c? and why doesn't ctrl+c terminates, instead it gives result for the stuck queries. No keyboard interrup error
 
4:23 AM
ORM and SQL are two tools available that web developers can use in database management. When comparing them, SQL has a higher hands-on management than ORM. Because ORM has a higher level of abstraction and more complexity than SQL, less hands-on management is required; this makes data management more efficient
 
so, when you say ORM, you mean something like sqlalchemy? @blessed
should I use sqlalchemy to establish my connections and not cx_oracle?
 
Exactly or peewee
Peewee is lighter
 
Okay, I'll try that
 
 
2 hours later…
6:35 AM
cbg
 
7:07 AM
Last year I had an assignement that was making a compiler from scratch using Java. I was thinking about making it again but in Python. Does it make sense to use an interpreted language to make a compiler ?
It did not build an exe though, the last step was generating MIPS code.
 
yeah, why not. also, sidenote, interpreted/compiled is a property of the "implementation", not of the language.
so, CPython is interpreted but other implementations of python may not be such as PyPy. (and actually, CPython does a compilation as well. in that sense to be honest this runs very similar to how the usual Java implementation runs, but for some reason we call Java compiled and python interpreted.)
 
Okay I definitively need to learn more about CPython. Are this what those .pyc are about? Is it like Java bytecode?
 
yep, the pyc files are python's compiled bytecode. CPython is the implementation of python that's installed when you just install "python", it's the default implementation
and as you guessed correctly, it compiles to bytecode which is then run in an interpreter loop.
 
Thanks for the clarification!
 
7:25 AM
@ParitoshSingh This interpreter is running C?
 
7:59 AM
@blessed This is just a soundbite that doesn't have any relation to the problem they are asking about. How do you propose that an ORM is going to fix a hanging query? The ORM still is SQL being executed an a connection. You've set them off on a tangent
 
@AlexandreMarcq Sort of, but there are important differences between what Java & CPython do with their bytecodes. Modern Java systems have JIT compilation, which dynamically compiles bytecode down to actual machine code. CPython doesn't do that. I got into a discussion about this topic a month or so ago in this comment thread on a HNQ: stackoverflow.com/questions/67988828/…
> The Python source is compiled to bytecode. That bytecode is then executed on a virtual machine. That is, the bytecode is the machine language of the virtual machine.
 
8:15 AM
@PM2Ring Thanks for the details. I think I need to go take a look back at the course I had on compilers, those comments went a bit over my head TBH.
 
@blessed your previous message was a link to a site, the content of which was useless garbage. I asked you if you had written it (because if you had, I wasn't going to call it useless garbage). I got no answer. Now you post this general blurb with no precursor. Can you stop doing this?
 
@AlexandreMarcq Fair enough. It takes a while to get your head around this stuff. en.wikipedia.org/wiki/Interpreter_(computing) may help to explain the differences between the different interpreter strategies.
 
@AndrasDeak What's wrong with the site? (Not that I'm defending it, I'm genuinely curious about which statements are wrong.)
 
I didn't say it's wrong.
Read it and let me know what it tells you about python.
I mean the URL has both "java" and "oops" in it so it's hardly surprising.
 
It allows us to develop applications using an Object-Oriented approach.
Sounds like it's all what's Python is about.
 
8:33 AM
@AndrasDeak There is some validity to mathematical theories of harmony, but the reality is a bit messy. :) If we just compare pure tones, most people (not including "tone-deaf" people) will agree that 2 notes whose frequencies are in a ratio involving small whole numbers sound good together. So virtually every culture agrees that unison (1/1), the octave (2/1), and the fifth (3/2) sound good.
However, human frequency analysis systems aren't mathematically perfect, even when it comes to octaves. So the highest notes on a piano have to be tuned slightly off the exact power of 2 (down, IIRC) in order to sound the best.
 
I also know there are forbidden frequencies that sound really bad, which is partly what defines how fretbars are positioned on a guitar
 
And of course, we rarely listen to pure sine waves. They get pretty boring very quickly. :)
 
at least that's what I was taught during undergrad
 
@AndrasDeak Modern Western music uses an equally tempered scale. So the frequency ratio between adjacent notes is identical (apart from that minute adjustments of the highest notes that I mentioned previously). There are 12 semitones in an octave, so the ratio between adjacent semitones is the twelfth root of 2.
 
I'm listening to microtonal music, I do not feel at ease.
 
8:48 AM
The most basic chords, the major & minor, consist of the root note and its fifth, with either a minor or major third in between. In the simple just intonation system, a fifth has the ratio of 3/2, the minor 3rd is 6/5, the major 3rd is 5/4. All those intervals (ratios) sound good, but you run into numerical problems when you try to construct a complete scale from them.
The Pythagorean scale tries to construct a scale just using 3/2 and 2/1 (and their reciprocals). It almost works, but you end up with some very ugly sounds, no matter how you juggle it. The history of Western scale development is basically trying various compromises to try & keep close to the small number harmonies while working around the numerical problems and trying to somehow avoid producing ugly intervals.
The core mathematical problem is that there's no positive integer solution to 2^x = 3^y
(I'm using ^ for exponentiation).
 
No matter what we do we can't get away from math.
@PM2Ring Do you have a source for this?
 
However, 2^8=256 ~= 3^5 = 243, which leads to pentatonic scales. And more importantly, 2^19=524288 ~= 3^12=531441. That's the basis of our modern 12 (semi)tone equally tempered system.
 
@AlexandreMarcq I'm sure the theorem has a name and a proof, but if you take the binary logarithm you get x = y log2(3) = y ln(3/2) which is quite suspicious
ln(3/2) doesn't sound very rational
 
So (3/2)^12 ~= 2^7. The error between those two ratios is called the comma of Pythagoras. I
 
9:05 AM
@AndrasDeak I meant why does it comes down to this equation.
 
@AlexandreMarcq Well, any power of 3 is odd, and any power of 2 is even. :) But there are various interesting approximate solutions, which can be found from the continued fraction of ln(3)/ln(2). One solution that has been explored musically uses 53 subdivisions of the octave, i.e 2^(1/53)
 
ah
@PM2Ring good point
 
@AlexandreMarcq Ah, right. Obviously, we want to build a scale where all octaves are perfect. The next step is to give our scale fifths. And we'd like all of those fifths to be the same size. ;) That means we try to make all the other notes just multiplying (or dividing) by 2 or 3/2.
The modern solution is to simply flatten the perfect 5th slightly, from 1.5 to 2^(7/12) ~= 1.49830707687668
Pythagorean tuning is what you get when you try to follow the rule of using only 2 & 3/2 to construct the scale. en.wikipedia.org/wiki/Pythagorean_tuning
FWIW, I wrote a Math.SE answer about this stuff earlier this year: math.stackexchange.com/a/3983124/207316
And there are quite a few posts on the Music Theory stack about various temperaments, including that 53 tone system, aka 53-TET en.wikipedia.org/wiki/53_equal_temperament
 
9:21 AM
@PM2Ring I'll try to dive into this once I get some free time.
 
53-TET gives you beautiful fifth. Unfortunately, its minor & major thirds aren't much of an improvement over 12-TET, which is a big reason why 53-TET hasn't had much of an impact. Of course, in earlier times, it wasn't easy to experiment with this stuff, but it's pretty easy to do it digitally.
 
This and compilers, this weekend is going to be headache inducing laurel.
 
:)
 
9:36 AM
Don't spend too much time on 53-TET. It can be a massive time sink. ;) But it's instructive to see the comparison between the Just intonation and 12-TET intervals. Eg, 3 major 3rds & 4 minor 3rds are perfect octaves in 12-TET, but (5/4)^3 = 125/64<2 and (6/5)^4 = 1296/625>2
 
I think I'll first look at the ratios and the meaning of "making" notes first.
 
Hello
 
10:04 AM
cabbage
 
user16280768
10:47 AM
Hai, How to extract feature from multiple videos and stored into individual H5 file
 
1:24 PM
I need some help with the quickselect algorithm. I wrote this to get me the kth smallest element. What can I change so that it returns the kth largest element?
 
@Dave hello. Please see our room rules about posting large blocks of code off-site and linking back here. Also, for future reference, we have a formatting guide for chat because it sadly doesn't work like on main (backticks for blocks won't work)
 
@roganjosh thanks for that. I was wondering about the formatiing. Any way to delete text?
Here is the algorithm again.
 
Messages can be formatted for up to two minutes (if you hover over the message, you'll see a dropdown arrow appear on the left). After that time, Room Owners can move it to a different room, but only Moderators can delete stuff permanently
 
@roganjosh Thank you.
 
Clearly you didn't read the room rules. Given that you did sort the formatting and I don't find it overly obtrusive, I'll leave it (other ROs may feel differently). In future, please do post it off-site and link back
 
1:35 PM
@Dave You can get more people interested in your issue if you already provide some example case where the algorithm misbehaves.
 
@roganjosh I did read the rules. I don't get what you mean by offsite though. Please explain.
@MisterMiyagi thanks for that. Here's my scenario. I wrote this algorithm to find the Kth smallest element. Like the 2nd or 3rd smallest element. This same algorithm can be changed to find the Kth largest element. The algorithm is called quick select and it's just a deviation of quick sort. My help here was that I'm having trouble with the logic to get the largest element.
I generally struggle with these types of questions
 
The point in the rules: 'If your code is longer than about 12 lines, use an external paste tool such as dpaste.com.'
It's just because things get difficult to follow when there's big blocks of code. Especially on my little 13" screen. It's just a lot easier to maintain the flow of conversation as people scroll up and down the transcript
 
Thank you for that. If you could delete the message I'll use dpaste
 
@Dave Thanks for being understanding :)
 
@roganjosh Sorry for switching subject but your VRP solver is really impressive !
 
1:46 PM
Here is the dpaste link for anyone who would like to help me.
https://dpaste.com/CBGTSMDFM
 
Thank you, Alexandre. Always fine to switch topic for a bit of ego massaging! :P
Is it a space you work in?
 
It's exactly what I've been working on for the past three weeks
 
Oh, neat. What's the context?
 
My end of study report has to have a scientific part, so I chose to do a research on VRP solving (has to have some use as well for the company I'm in).
 
How many vehicles/jobs are you looking at?
 
1:51 PM
I didn't have much info so I eyeballed at about 15 trucks.
 
i came across a question where one user provided an answer, i posted my answer. and noticed a downvote. the thing is that scenario always happens when i provide an answer to a post where the same user is there!
stackoverflow.com/a/68499653/7658985 do i need to flag such behavior ?
 
Nice. That's in the realm of what you could reasonably solve heuristically in Python with a completely custom algorithm (which I assume is the point of the project). If it reduces down to a problem that OR Tools can handle, even better, you'll have scalability but once you pile enough constraints in, linear programming falls down
 
I was aiming for linear programming at first to have more mathematics included but my teacher was not available so I decided to have a bit of "fun" and went with a genetic algorithm.
 
If there's one thing for sure, you can definitely cram mathematics into VRP. The number of papers that give an entire page of mathematical notation and you realise "right, so you guessed about this parameter, calculated that, then added to your cost function"
 
Exactly, and cherry on top I had a run of 30+ hours to generate a lot of statistics regarding eight different algorithms!
Unfortunately I did not take into account the time constraints because I did not have much time to work on the project. And without thinking too much about it I programmed it with the depot at the center of the map so it may have biased the results.. At least I'll have things to say about the improvements.
 
2:43 PM
cbg, I have a python / docker related question, if I create a docker image as shown in this stackoverflow.com/questions/56287088/… can I now delete my python source code and run it only using the docker image?
I am asking this because, when I try what is attempted in the answer, it seems to run the python code from my local not the one that is in the image
I know it is running from the local one, because I added some print to my local, these print were not in the original source code when the docker image was built
 
@Dave Are you aware that this code reorders the input array? That's terrible design
I'm 99% sure all you have to do is swap out the < for a > btw
 
2:58 PM
@python_user Can you provide your Dockerfile?
 
it is the same as seen in the answer, same file name, same everything
 
What happens if you rename your folder and try to run the container again ?
 
I am guessing this is not a python question anymore, mind joining me in a new room, so as to not clutter room 6?
 
3:14 PM
@Ar
@Aran-Fey I swapped the comparison operators in the pivot side, but that did not work. I think I'm doing something really worng.
 
I mean, it works fine with my test inputs
 
Lazy solution: def kth_largest_element(seq, k): return -kth_smallest_element([-x for x in seq], k)
Hard mode: make it work for types that define a total ordering, but not multiplication by -1. For example, strings.
 
Kevin the lazy won't work for me. I'm preparing for interviews and it won't be accepted sadly. @Aran-Fey what does this input give you?

Input: nums = [3,2,3,1,2,4,5,5,6], k = 4
The answer should be 4
 
I don't know why it wouldn't be accepted. I solved it in one line, they should hire me on the spot.
 
Haha. Well interviews are broken.
 
3:22 PM
Umm, the answer should be 3, no?
 
Optimal in terms of Big O as well, since by definition you can't do better than O(N)
 
Here's the leetcode link
@Kevin Yea I really wish interviews accepted this. Sadly they want you to write long and crazy code.
 
I apparently have solved this question and I used heapq
 
So your k starts at 1, not at 0? Pretty sure the algorithm isn't designed for that
 
@Dave When you say "what does this input give you?", do you mean for kth_smallest_element, or kth_largest_element?
 
3:25 PM
Kth largest. And I'll like to use a max heap to solve this. Seems to be less trouble
 
^^^^ Yeah heapq is pretty much ideal here if you're not trying to reinvent the wheel
 
@Aran-Fey which algorithms starts at 1?
Got to learn about heapq first
 
The k in the leetcode problem does. But the code you wrote doesn't.
 
At the risk of being overly cynical, I'd say don't bother learning about heapq, because you'll just end up with a super efficient and concise solution that won't be long or crazy enough to impress an interviewer.
 
Oh. I didn't look at the constraints. That makes sense. Do you think this condition in the main quick select algorithm would work? If start == end: return array[start]
I basically have to make the algorithms start from a
1
 
3:29 PM
Writing your own heap implementation on the other hand... Now there's a nightmare minefield of off-by-one errors and opaque index arithmetic that will wow any interviewer
 
@Dave Why not just subtract 1 from the index?
 
I won't ever do that haha. Man I hate trees first of all
Not to talk about writing a full tree code
 
heaps are like... Half of trees.
Since you can implement them with a flat array, so there isn't necessarily a nodes-and-branches structure
 
Which index are you talking about Aran?
Well the full implementation uses a linked structure
So you have the node, etc
 
I'm talking about k
...and maybe also lowerbound and upperbound
 
3:35 PM
k works. After changing the pivot comparison sign. As I said before, I'm not good in these types of logic. So the question is, why subtract 1?
 
........
Because the task is to write an algorithm that works with 1-based indices, and your algorithm works with 0-based indices?
 
Yes. But since it's 0 won't you need to add one rather?
0 + 1 = 1
 
No.
 
@Dave Depends on what you mean by "full implementation". I believe the heapq module uses a flat array, and it still produces the same results as a design that uses full-fledged Node class definitions etc etc.
 
3:50 PM
"Leet" code, where there's a python 2 option and there's mandatory class Solution. Not sure why people keep using that site.
I mean I'm reasonably sure, but it's still sad.
 
@Aran-Fey I looked at it again and yes. Because it starts from 0, if k == 2 it would actually be 3 in my array. So I need to remove 1. Thanks
@AndrasDeak I really hate the class sloution.
@Kevin I mean you can also use a sorted doubly linked list to do it haha. The only reason we can use an array is because it's a complete binary tree.
 
If you're saying "I don't want to use an array, because what if the input doesn't form a complete binary tree when heapified?", worry not. The result of heapification is always a complete binary tree.
Specifically the definition of "complete" that allows the bottom row to be only partially filled in. Contrast to "perfect binary tree"
 
 
2 hours later…
5:58 PM
@AlexandreMarcq if you could take into account "soft" time windows and actually apply it, that'd be super-cool
So many companies work on the basis that it either gets delivered in a specific timeslot or gets unassigned entirely... even if it's only going to be 1 minute late
It's not too hard on small problems, you just need a non-linear penalty function of missing the time slot. I don't know if you have any time left to add to your solution, but it'd be really neat if you could do it. Generally, your approach probably wouldn't hold for larger problems, but 15 vehicles... maybe it wouldn't be too complex
 
mas
6:18 PM
cbg all
 
cbg
 
mas
6:37 PM
Do you ever shoot yourself in the foot with a library and spend hours and hours debugging only to finally see you were the one pointing the gun?
I just found I kept putting empty data in a database even though all my validation checks kept working and not executing db.session.all. Turns out I was establishing a relationship before any of the checks passed and that added it to the database the whole time.
Afterward I swore SQLAlchemy was stupid for doing that, and then realized I actually like that feature sometimes.
 
The tricky part about computers is that they do what you tell them to. ;)
 
when something goes wrong, challenge your own assumptions. Often you're the one to blame
It's one of the most important lessons to learn, they happen sooner or later*
 
@mas I'm Josh and I've gone 3 hours without a major brain fart. Welcome to the fold
 
@roganjosh I'll try my best. What I did for now was assign an arrival time to each destination and take into account the time taken to go from A to B. To each solution I assign a "total lateness" variable. So far it usually takes about 10k generations to have less than 150 minutes for the best individual..
Also didn't really implement the number of trucks, it's mostly "unlimited" for now but it never goes more than 12 or so.
 
How long does 10k iterations take?
 
6:49 PM
I didn't time it, but I'd say around 2 minutes ?
 
Not too shabby! :)
 
Yeah it could be worse!
 
Just don't dig too deep. Next thing you know, you've got warehouses in different dimensions, drivers delivering their breaks, and incomprehensible array operations. You've been given fair warning :P
 
Yeah I'm basically done with the research part, that's what is the most important to me. Application is just a bonus.
At one point I was supposed to take driving regulations into account as well so yeah didn't believe in it much laurel.
 
Fun fact; we screwed up with our maps at the start of a project. We were routing 32 tonne trucks at 70 mph on the wrong side of the motorway. I mean, our efficiency gains were fantastic but I think the public might have an issue with that
3
 
7:02 PM
"Trust me, those IT guys said it would work better."
 
Who said bugs couldn't be fun? :P
 
For statistics purposes, I had a plot of the fitness for each algorithm. I would have 20 run of n generations to have some average. Everything worked as expected until I forgot to remove the plotting part when I had 200 runs of 100 to 1000 generations.
I mean, it's colourful.
 
there's certainly data in there
 
data is power. I think we just need to fit a trendline
I'm going linear regression.... but maybe there's something more in that
 
Rectangle regression ?
 
7:14 PM
sanity regression.
 
Oxymoron.
 
This is the kind of result I would get with the best algorithm, after 1000 generations. Fitness here is only based on standard deviation.
 
So it's capacited?
 
Oh yes forgot to mention that.
 
And presumably Haversine distances?
Quick win: use folium to neaten up your output. It'd be 10 lines of code (though you'd need some understanding of the library)
Then it would actually be plotted on a real map
real-road distances are more complex. I think I'll stop with that suggestion
 
7:22 PM
Where we're going we don't need roads!
 
My brain has to work in multiple dimensions and in odd ways with VRP. I'm dangerously close to falling into the Physics trap
That's not a place I wanna be. I just wanna have a box in location X at time Y :'(
 
the cow is only spherical in the ways it really matters
 
@roganjosh Don't ban me, I used Euclidean distance
 
found the flat Earther
 
But I have a good excuse
 
7:28 PM
Small town?
 
I'm too benevolent to ban you. I just will use ridicule instead!
(I can't ban you :( )
 
All I had at first were post codes and city names
In an excel
 
But, serious point, there really is no excuse to not use Haversine here
 
Does that really matter that much? Is the difference between road vs Haversine not far greater than Haversine vs Euclidean?
I certainly understand the purity appeal, just wondering about the practical implications
 
road vs Haversine is huge
 
7:31 PM
To be honest I didn't even know about that. All that mattered to me was for it to kind of work. For me this was just a bunch of points in a 2D plane, hence the shortcut.
Thank you for the tip, I'll write it down in my report.
 
@roganjosh that's what I'd expect
 
Euclidean vs Haversine - try it
 
@roganjosh nice try trying to get me to write a routing engine
 
If it's hyper-localised, I'll do it myself, but over any reasonable scale, it is just terrible
 
Drivers must stop at stops, so solving the routing problem amounts to solving the halting problem, so it can't be done. There. #physicisted
 
7:34 PM
Is France small enough to be considered hyper-localised?
 
No
Different VRPs work on different timescales
 
At least I tried.
 
If you're talking long-haul trucks, you can get away with a bit of wiggle-room without knowing the road network
Haversine won't cut it, either way. You'd need OSRM (if you don't want to buy an extortionate Goole Maps license)
 
I just know that some clients are accross the country whereas some are about 30 km away
 
So Euclidean distances... err, I think it answers itself
30km is enough for Haversine to break down totally
 
7:41 PM
This needs lat and long right ?
 
I think he means that at 30 km even Haversine won't cut it
 
It does. But if you don't consider the road network then you're well wrong. It'll probably have a motorway involved, then some A-road, then some residential area
^ UK-centric but there are different road types
 
I think you've over-estimated my capacities for this project. I don't think I'll be implementing all of this in my reasearch algorithm but I will certainly talk about all those misconceptions errors in my paper.
 
If you're routing on a spherical cow, Haversine is perfect
 
And throw in some TODO's for the one who'll be taking care of this (if the project is not just thrown away when I leave).
 
7:46 PM
well you did say "research"...
 
True
I should have said that I had around a month to do that.
 
ending nowadays?
 
Yes, I have to focus on writing my report now the deadline is in less than a month.
I think I'll put the code on GitHub and try to improve it in the future.
 
@AlexandreMarcq i want to reverse a bit
I think I've been too harsh. For the purposes of your project, I think you've done a great job
You suggested it needed to be applied. I'm picking you up on that side
If you get stuck with the practical application then I'm happy for you to open another chat and I'll have a go (when I have time). In terms of your project itself, the soft time windows was a suggestion that I think might help you stand out
 
8:22 PM
@roganjosh I'll try my best to add that when my report is done. Thanks a lot anyway for your input. By the way, what kind of heuristics did you use?
 
Simulated Annealing
That's my basis, but my solvers go a little bit futher
 
into pocket dimensions
 
I'll invent dimensions as, and when, I need them to make things work.... oh god.
 
that you are
 
I thought I was just physics
 
8:35 PM
That algorithm is really interesting.
 
My message was just awful. I'm not physics but the edit menu on mobile vie was off-screen so I couldn't fix what I was trying to say :,(
 
you need an uparrow
 

« first day (3933 days earlier)      last day (1001 days later) »