« first day (4238 days earlier)      last day (693 days later) » 

12:07 AM
What's a good reference I can link people for Python's behaviour when creating classes (i.e. the way that it executes code in a separate namespace and passes the results to type)?
I guess there's docs.python.org/3/reference/… , might not be able to do better
i.e. this stackoverflow.com/questions/72355834/… is ordinarily closed as a typo, but I can imagine a "why does Python behave like this" version of the question
 
12:21 AM
Hello, would like to ask more of a linguistic question (English not being my first, nor second language), what do we call a structure that, given an input, would compute the next step to take and returns the next step to take in processing? Closest I've found is a state machine (states/transitions, each state when given an input transitions to another state), is there anything like that?
 
Yeah, state machine is the closest thing I'd have. Maybe a finite state machine: en.wikipedia.org/wiki/Finite-state_machine
 
12:50 AM
that's exactly what I'd call it too, yes
 
generators are a somewhat related concept to state machines. They perform a bit of calculation, then send the result to their owner and pause themselves. Later, the owner can unpause the generator and have it do more work. The owner can send input to the generator, although in practice that's fairly rare
I see next() get used a lot more often than send()
 
 
2 hours later…
2:58 AM
Has anybody used Andriod ADB from Ptyhon? I googled on stackoverflow, I found pyadb seems a solution, but I have do some tests, it always fails now and then.
 
 
4 hours later…
6:53 AM
@KarlKnechtel There's types.new_class if you need something concise. For a more in-depth description, the data model on metaclasses and following sections is my own go-to reference.
 
 
6 hours later…
12:26 PM
@John If you can get Android ADB working by running it manually from the command line, then you should be able to run the same commands with Python's subprocess module.
That's probably what pyadb is doing.
 
 
3 hours later…
2:58 PM
3 hours of refactoring, and I've almost got the code doing what it was already doing just fine before
I'm sure the boss will be pleased to hear I'm standing triumphantly on square 1
 
3:08 PM
Vague database design question: I want to start a website that sells fish and bicycles. All fish and all bicycles have a brand name and a price and a weight. fish have a "requires salt water" boolean, and bicycles do not. Bicycles have a "number of spokes" integer, and fish do not. How should I structure my database tables?
 
I'd look into using "model inheritance" (in ORM terms) with one parent table for the common columns and two separate tables for fish- and bicycle-specific columns
 
Here is some pseudo-code outlining the approaches I was pondering. @vaultah, sounds much like my Design A.
Hard mode: I wish to expand my business and sell e-books about quasars. All e-books have a brand name and a price, but they don't have a weight. I also want to provide mints at the cash register, which have a brand name and weight, but no price.
 
I'd just make the weight and price columns nullable
:p
 
3:32 PM
A bit more in the style of Design B, then. A fairly common approach, if I'm reading en.wikipedia.org/wiki/Single_Table_Inheritance correctly
 
What I meant was Design A with nullable Product.price and Product.weight and without Product.product_type and Product.fish_or_bike_id
Or something else entirely
Table Product
    int product_id (PK)
    string brand_name
    number price (NULL)
    number weight (NULL)

Table Fish
    int product_id (FK Product)
    bool requires_salt_water

Table Bicycle
    int product_id (FK Product)
    int number_of_spokes
 
I like that design
 
I'd ask myself: Is the weight of fishes and the weight of bicycles really conceptually the same thing? Physically it's the same thing, sure - but are you ever gonna query/sort/filter fishes AND bicycles at the same time?
In other words: I'd just move the weight to the child-tables
And free mints doesn't sound like "no price" to me - it sounds like price=0 to me
Products with a nullable price just feels WRONG.
 
Actually yeah
We could maybe make NULL mean "not available" or something similar
 
In the general case, I do expect to query/sort/filter fishes and bicycles quite often. For example, when I'm printing out a comprehensive price catalog, I'll want every product I carry. Or when I'm going over the sale records for the day, I need to know the prices of everything I sold.
To use a weight-specific example, I might need to know the total weight of all the products I plan to deliver in a day, to make sure my truck can legally cross over "1 ton or less only" bridges
 
3:46 PM
Doesn't seem like "bikes as products" and "fishes as products" vs "fishes as fishes" actually overlap for these use-cases.
 
I would like the catalog to also show the saltwaterness of fishes and the spokeness of bicycles, if that changes your mind
 
Depends on whether I can get you interested into this nice bridge to NoSQL...
 
I don't mind giving mints a price of 0 and e-books a weight of 0... Unless things somehow get out of hand and 90% of my disk space is taken up by values that mean "this kind of object does not have a meaningful value for this attribute"
 
I can well see putting a 0 price into the products table, since that's actually still a price.
 
Hello
 
4:00 PM
Faking weights to 0 is somewhat fishy to me. You're not supposed to literally package e-books, so having the package'able is just asking for trouble.
You probably also should differentiate between shipping weight and product weight. Shipping a bike takes more weight than the bike itself, for example.
@OlauPla Hi there!
 
@MisterMiyagi no, 0 number of spokes is fishy
 
See? Tripped me up already!
 
Spokes are bikey
 
I imagine that shipping a fish also takes more weight than the fish itself. Unless I'm just hucking them into the back of the truck, no packaging or nothing
 
conveyor belt or airdrop are the only acceptable means to transport fish
 
4:10 PM
Perhaps if I specialize in fish that are known to travel out of the water for moderate periods of time... Kevin's Bicycles and Mudskippers and Octopus, Inc
 
How did octopodes make that list?
 
That "Octopus" there is singular because I only have one. Every time I sell him, he breaks out and comes back to me.
I've seen a video or two online of an octopus crawling over dry sand near the shore, or climbing up out of their tank in an aquarium. Here's an example.
 
yeah, but I thought you were talking about fish
 
Does an Octopus have spokes?
 
(Hmm, I better not tell Andras I have dolphins in my fish table as well)
 
And spiders in my insects table! Chaos reigns!
 
@Kevin Reminds me of xkcd.com/349
 
That's usually how this scenario turns out for me, yeah.
But much to my surprise, I find myself standing on square two with nary a shark bite
 
@Kevin I assume pycnogonids are present both in your fish and your insect table...
 
Yep, right below the platypus
 
4:47 PM
Sadly, I've never seen a platypus in the wild, only at the zoo. Our 20 cent coin features a platypus.
 
 
1 hour later…
5:59 PM
When I was a kid I thought platipus was hyper mythical
like, there's tales of it
like bigfoot
which makes sense
because it's just such an over-realistic animal.
 
Reminds me of the coelacanth, a fish thought to be extinct for 40 million years, until a live one was spotted in 1938
I don't know how much academic support there is for this theory, but I've heard that ancient eyewitness accounts of unicorns may have been rhinos
 
ya ok but are they a mix of 101 distinc animal parts
and the only mammal giving birth with eggs
also something about no stomach so need to always eat
like what
thats not real
 
6:39 PM
Today I learned AWS has like 4 times the market share of GCP
and that makes me feel some type of way
 
 
3 hours later…
9:11 PM
cabbage
 
What is the quickest way to flag duplicates? I don't want to remove them but flag which one I want to use with a "1" and the rest with "0". I have spatial data so X,Y,Z, Value and normally if X=X and Y=Y and Z=Z, one of the values is missing (-99 for this dataset) and the other has the value so easily would flag the -99 as 0 and the actual value would flag as 1.
Currently I'm using a dictionary and a set of > and <= for the value. Any other ideas that could make that go faster? I started it last night at 850pm and its been running ever since, now at 2:20pm the next day (1.3M rows).
 
"dict" and 1.3M rows sounds like a bad combination. Do you mean your data is also in dicts, or just the flags?
I'd use a (1.3M, 3)-shaped array named points for the data, and (points != -99).any(-1) is then a (1.3M,)-shaped array of bools.
@CelesteWilson that's a bit too much code for chat, could you please put that in a pastebin or similar instead?
 
whats a pastebin
 
It's also way more than what I have to know, and still missing data to be an MCVE...
@CelesteWilson pastebin.com, gist.github.com etc.
instead of posting all that code, just try telling me in exact terms what your data is structured like
 
comma delimited
 
dhid (ids), midx(Easting), midy (Northing), midz (Elevation), value, dtype (drill type)
 
What you seem to be saying is that your data comes from a CSV file, right?
 
Is your only specification that you want to end up with another CSV file with one more column containing the 0 or 1 flag?
 
yes so same exact data as in CSV but with a KEEPVA column with flag 0 or 1
 
9:27 PM
OK
do you have to use python for this?
 
I don't have to... but the entire workflow is python. I'm willing to try something else
hahahahha
Excel is a no go for sure
1.3M rows
 
sounds like a very simple awk script would do this for you
Are the "x,y,z" values actually the midx/midy/midz values? And you really want to check if any of those is equal to -99?
 
if (X1=X2,Y1=Y2,Z1=Z2) then put a flag of 1 on the row that has max(Value1,Value2)
so if its in the same location spatially, choose the value that is higher
 
Come again? You mean there are multiple rows with the same point coordinates?
 
yes
they drill fans
 
9:32 PM
That sounds like an important bit of information for your problem
So what's "max(Value1,Value2)" then?
I only see one value column
 
that is it just 1 and 2 are the rows
 
Sure would be nice if we had some sample data to look at, wouldn't it
 
@CelesteWilson ah, OK, this is another problem
 
so if I have a value overlapping another value at the exact same point
 
You want to remove rows with -99 in coordinates. And you want to keep the highest value among equivalent points.
One of those is a lot harder than the other.
 
9:34 PM
I want to flag the rows with -99 that share the same location as ones that have an actual value, and if both values are -99 then it doesn't matter which one is flagged
I don't want to remove anything
just flag with a 1 so that the next software I use knows a 1 = use, 0 = don't use
 
@CelesteWilson I don't understand that
You don't want to use rows with any -99... unless?
And you are talking about both "values" of -99 and point coordinates?
 
You've tried to explain this 5 different ways and we still don't get it, so maybe consider giving us a visual representation of the data instead
 
one sec let me see if I can give an example
that's what I'm doing
I just can't share the real data because its not public
so I'll share a fake one
 
$ cat tmp.csv
dhid (ids), midx(Easting), midy (Northing), midz (Elevation), value, dtype (drill type)
0, 1, 2, 3, 4, 5
1, -99, 6, 7, 8, 9
2, 10, -99, -99, 11, 12
3, 13, 14, 15, -99, 16
$ awk -F, 'NR==1{suffix="KEEPVA"} NR>1{keepva=1; for(i=2; i<=4; ++i){if($i == -99){keepva=0; break}}; suffix=keepva} {print $0,",",suffix}' tmp.csv
dhid (ids), midx(Easting), midy (Northing), midz (Elevation), value, dtype (drill type) , KEEPVA
0, 1, 2, 3, 4, 5 , 1
1, -99, 6, 7, 8, 9 , 0
2, 10, -99, -99, 11, 12 , 0
this ^ dirty awk script handles setting the flag any row with -99 to 0
anything that has to consider equal coordinates will need python and lots of runtime/memory to find pairs, at least you'd have to sort the rows by coordinates to group equal points
 
@AndrasDeak--СлаваУкраїні I think -99 is the null "value" field value. Don't look for it in an X, Y, Z coordinate.
 
9:42 PM
@PM2Ring aaah, that would make a lot of sense, thanks
 
We should get into the stock market, considering how much we always speculate
 
the problem specification is clear once you know what the actual problem is :P
so the whole premise is that -99 only appears in values when the coordinates overlap
So the only issue is when all equivalent points have a value of -99 because it would seem that in this case a random row would have to be kept with a flag of 1
 
So we just need to sort on (x, y, z, value), and flag the last row in each group of rows with identical (x, y, z) with 1, and the previous rows in that group with 0.
 
In ten minutes: "but sometimes we have x=1.01 and sometimes x=1.0998 and these are still the same point" :P
 
maybe this is not the best place to ask, but I would like to know how to get a single frame from an external camera with opencv all the codes I have seen about it capture an infinity of frames and I don't know where to start because well I am not a programmer.
 
9:47 PM
My guess is if all values in a group are -99, then that whole group is a dud, and they all should be flagged 0. But if multiples in a group have values that aren't -99, it doesn't matter which one gets flagged 1, but only one row can get the 1 flag.
 
import pandas as pd
data = {'dhid':['R85','R85A','R85','R85A'],
        'midx':[300.8,300.8,290.4,290.4],
        'midy' : [455.6,455.6,450.2,450.2],
        'midz' : [56.2,56.2,48.7,48.7],
        'dtype': ['RC','Core','RC','Core'],
        'value' : [-99,0.04,-99,-99],
        'keepva': [0,1,0,0]
        }

df = pd.DataFrame(data)
df[['dhid', 'midx', 'midy', 'midz', 'dtype','value','keepva']]
 
@OsmarHD Well, then step 1 is to learn programming
 
here is a fake dataset
 
@OsmarHD if you're using opencv in python, this is the place. cv2.VideoCapture(device_id) should have a .read() method that reads a frame. What that exactly means is left as an exercise to the reader.
You can use any solution that reads an infinity number of frames, just stop after one frame
 
Oops. The max value should be flagged 1
18 mins ago, by Celeste Wilson
if (X1=X2,Y1=Y2,Z1=Z2) then put a flag of 1 on the row that has max(Value1,Value2)
 
9:49 PM
@CelesteWilson And what's the output you want? Is keepva the column you want to generate?
 
yes
I just put it in the example so you can see what it does
I wish my dataset was that small
 
Is it correct that your last two points are the same, both have a value of -99 and you are keeping neither?
 
yeah I mean I could just keep one
but the next software ignores -99
 
COULD you, or DO YOU HAVE TO?
 
if they both are -99
I don't have to, if I wanted to be extra cautious I would
 
9:52 PM
DO you want to be extra cautious?
 
currently the dictionary method I'm using just picks one and puts 1
hahahahha
yes
I should
it goes to the shareholders
lol
 
I'm asking all this because if you don't HAVE to keep any of these rows, then all this about duplicate points is a red herring. "Flag rows with -99 value", done.
 
what if I had another one that that had real values for both
 
@AndrasDeak--СлаваУкраїні yeah im using opencv on python, maybe I'm explaining myself wrong, I want to do the function of a camera with a button to capture the frame I'm interested in and not a random one.
 
this is also the difference between an O(N) solution and an O(N*log(N)) one
 
9:53 PM
like 0.04 and 0.05
 
@CelesteWilson come again?
Are you asking me about your specifications?
 
@AndrasDeak--СлаваУкраїні Well, that code you sent to the Knives has # Distance tolerance for duplicates removal dtol = 0.1 which does make things slightly tricky. Especially if we have to use the Euclidean metric. But maybe we can use 3D Manhattan, and do the grouping with rounding.
 
import pandas as pd
data = {'dhid':['R85','R85A','R85','R85A','R85','R85A'],
        'midx':[300.8,300.8,290.4,290.4, 270.1,270.1],
        'midy' : [455.6,455.6,450.2,450.2,445.2,445.2],
        'midz' : [56.2,56.2,48.7,48.7,39.3,39.3],
        'dtype': ['RC','Core','RC','Core','RC','Core'],
        'value' : [-99,0.04,-99,-99,0.05,0.01],
        'keepva': [0,1,0,0,1,0]
        }

df = pd.DataFrame(data)
df[['dhid', 'midx', 'midy', 'midz', 'dtype','value','keepva']]
added one more that has real value for both
 
9:57 PM
I have lots of fun topics. Do any of you do machine learning
lol
 
10:08 PM
I know exactly what I want to do, but I can't figure out how to translate it into pandas...
 
shoot
if it's close enough to numpy I can probably help
then again I have to assemble another mid-term for tomorrow still...
 
max_values = df.groupby(['midx', 'midy', 'midz']).max()
max_values_dict = dict(zip(max_values.index, max_values['value']))
df['keepva'] = [
    1 if row.value != -99 and max_values_dict[(row.midx, row.midy, row.midz)] == row.value else 0
    for row in df.itertuples()
]
Can you spot where I gave up on pandas?
 
those parens in max_values_dict[] are superfluous
 
I think it's more intuitive with parens tbh
 
yeah, because you're not used to multidimensional arrays ;)
but fair enough
 
10:18 PM
Creating columns in pandas is cheap enough btw
 
@Aran-Fey that worked on a smaller dataset, let me try it on the big one now. thank you for all the help!
 
1 if row.value != -99 could just become a column (obviously with the condition completed)
 
that's pretty much just row.value != -99
well, df.value != -99
 
But even if I store that in a column I still need the listcomp, right? So what does it achieve?
 
@AndrasDeak--СлаваУкраїні Sure, it's a simple case, I just wanted to make the point that you can have state between operations and not have to cram everything into a 1-liner
 
10:21 PM
@Aran-Fey you'd want to remove the list comp
 
Well, that's the part I couldn't figure out how to do
 
you can probably .agg the groupby with the max of the value column... somehow
Or is it transform?
 
I haven't been paying attention and I have an early drive tomorrow. max_values_dict[(row.midx, row.midy, row.midz)] looks like it could be a groupby followed by a descending sort and first()
But I could be way off without reading back, sorry
 
In a fast language, you'd just use a regular loop and it would be fine.
 
No comment. Except <- that one
FWIW I got that optimiser working with numpy about moving incoming stock around and I think I can can cut Customer X's stock holding by 2 million units. And it takes 7 secs, tops, to solve it. So, I can live with that
rbrb
 
10:53 PM
It seems that when using an REPL that uses pygments, if a blocking function is executed, if Ctrl+C is pressed, Python exits, but any typing doesn't show up unless you type reset or tset.
I have realized this is due to PDM
 
11:45 PM
It seems to be due to the fact that PDM uses subproccess to call programs --- using os.system does not exhibit this behavior.
Is there a way to call subprocess.call to prevent this behavior?
 
@DrownedSuccess see if shell=True changes this
 

« first day (4238 days earlier)      last day (693 days later) »