« first day (4049 days earlier)      last day (1121 days later) » 

00:04
@Kevin @Aran-Fey I don't want to wait until the end as the intermediate logging has a lot of information regarding some edge cases and what not that might go wrong
 
7 hours later…
07:31
@lutz500 here is the result of the code using the belowZero:
08:01
@LarrytheLlama it's better to post text as text, not images
@LarrytheLlama note that this user hasn't done anything on the main site since october, and with 1 rep they can't chat, and you can't ping them if they aren't here
08:30
Thanks @AndrasDeak, I completely forgot. Secondly, I know it is the norm, but they already have the code so I am not too worried about amending the image-to-text.
09:09
@LarrytheLlama There are people here other than the two of you :P
@AndrasDeak ugh... my ninja skills fail again... sighs
did you really think this would work?
09:24
umm... I'm writing a complaint email to cutestdogever.com - the site should only contain pictures of me... it's blatantly false advertising otherwise :p
 
3 hours later…
12:22
How does packaging work if you want to compile stuff on the user's PC? I.e. not shipping a pre-compiled dll
The only thing I'm somewhat sure about is that pyproject.toml can't do this and you need a setup.py
Something something source package?
Is this what an sdist is for?
As opposed to wheels
Each time I try to google about pyproject.toml and best practices I get lost
13:01
@Aran-Fey You can find some guidance in the ticket for actually documenting that. This part from the docs should still be relevant. The Cython guide is probably a good starting point as well.
You can still ship wheels even when you have extension modules. That requires building separate wheels for the major architectures and CPython ABI versions, though.
The good part is that if you are fine with source distributions and only need Cython, things work out of the box with setuptools/setup.py.
Was afk, so sorry for the late reply, but thanks for all the reading material
Hey, if reading all that makes you find out how to do it in practice, let me know. :P
13:30
@MisterMiyagi isn't that just "shipping a pre-compiled dll", just multiple ones?
@AndrasDeak It is. Not sure exactly what Aran intends to do, but I hope it's something that can be fixed by wiggling the rear view mirror.
It's actually my brother's project, so I don't know the details either. As far as I know, he wants to ship some c or rust source files and compile them during installation
I found that the cryptography module has rust extensions, so that might serve as a useful reference
If it's pure Rust with just Python bindings, PyO3 is allegedly super simple.
You're basically just writing Rust and a cargo toml, the rest happens via macro magic.
I see unable to find vcvarsall.bat a lot among Windows users that can't figure out how to pip-install a project that contains non-Python code. Therefore I speculate that vcvarsall.bat is an important part of the compiling-on-the-users-PC toolchain
Now instead of having to compile12ish different dlls, you can let each user compile their own. Easy! Unless they don't have Visual Studio 2017 installed already, in which case they'll have to decipher the scary red error message, google a couple of things, find a download page on Microsoft's dead link labyrinth, and do battle with the Installation Wizard
A wise man once said... "not my problem"
13:45
Yeah, the project maintainer has been sipping margaritas on a white sandy beach since sentence 1 of that scenario
Opening up my gaming laptop after a few years was enough to need 6 hours to work through the mess of OS and VS updates. That was just for running binaries. I've got no idea how you folks can build anything on that platform. 🤷‍♂️
Only 6 hours? Mine takes 3 hours every 2 weeks
Hey all, I was wondering if anyone would be willing to give some perspective on this question about Cluster Analysis? stackoverflow.com/questions/69983613/…
My Windows 7 computer abruptly stopped getting updates a while back, so that's nice. My Windows 10 computer installs updates and reboots without my permission every few days, but it must do it at 2 AM or something, because I've never seen it happen
Updates are the reason why a free, not activated, version of Windows is better than a paid-for, activated, version of Windows. Thank you for coming to my TED talk.
13:54
You know something's fundamentally broken when updates are a bad thing
@Aran-Fey Well, I stopped looking at it after 2 hours, so that's just an estimate. ;)
@DataScienceAcolyte You might want to take the tour and crawl through the help-center, but the TLDR is that such questions are off-topic on SO. Another SE site might be appropriate though.
On the rare occasion that I'm actually present while updates are installing, they typically take about fifteen minutes. I've had a couple hour-plus monsters in truly exceptional circumstances.
@MisterMiyagi someone downvoted the question, which I thought it was perfectly legitimate to ask about the implementation of K-Medians. So, I think I will consider another site.
I thought it was a well-structured question, FWIW.
Perhaps the downvoter has a trained pavlonian reaction to the phrase "step by step example", since it is often used by question-askers that haven't put a good-faith effort towards learning the material themselves. (I'm not accusing you of that or anything)
14:10
It's not a bad question per se, but SO is for specific programming questions. The question seems rather to require explaining a broad range of clustering concepts.
Hm, I'm honestly surprised that scipy seems not to have that algorithm.
Would Cross Validated be recommended then? I did look at a lot of different questions on that site about K-Medians, but most of the answers were largely theoretical. I understand that K-Medians is based on Manhattan Distance rather than Euclidean distance, but I am looking to understand the mechanics of how to use a specific clustering package. If I cannot figure out the PyClustering package then I will probably just write it myself. But I guess that is where I am coming from here.
Regarding question 1, "how do you find the medians?". If k-medians is like k-means, then your initial medians only need to be a reasonable guess. en.wikipedia.org/wiki/K-means_clustering#Initialization_methods discusses some approaches for k-means.
"create K empty clusters and put each point in a random one" seems relatively easy
14:46
@Kevin thank you. So, I think I understand now what I need to do. The medians supplied have to be based on the dimensionality of the dataset. and so if I want to test for 2 clusters with a dataset with 3 columns, my medians supplied have to have 2 entries with 3 values for each entry.
15:09
Yeah I think so
15:25
I'm stumped by typing again.
miyagi: Stump, problem solved
My understanding was that TypeVars default to invariance. But if I use something like def f1(a: T, b: T) -> bool:, MyPy happily accepts different subtypes for a and b via their common base class.
Even if the typevar is both the in and output type, like def f2(a: T, b: T) -> T:, MyPy has no problem accepting subtypes.
Me: Hmm, I prototyped this project on top of an HTTP server, but I could do more cool stuff if I had full duplex communication. Should I integrate websockets, or just make the clients poll the server every second?
Mozilla docs: websockets are easy and fun! Before you begin, you need to know how to implement 32 bit XOR decryption. Djikstra defines a 'bit' as... [etc]
Me: ... I think I'll just do polling.
15:41
Some of us are still waiting to read Dijkstra's definition of a 'bit' , you know.
It is a marvelous definition, but too long to fit within these margins
16:06
I just realized that I still don't understand variance
Has mypy always thrown an error when you try to set covariant=False or contravariant=False?
I assume it did. You can try older versions in the playground.
Yes, 0.620 already throws that error.
Alright, maybe I got it mixed up with python
16:23
Hi Team ,
I need some help with pyspark I am importing csv with dataframe which contains blank values and observed blank values are replaced with null
so I am using this to filter null values
df = df.withColumn("status", F.when(F.col('category').isNull(),1).otherwise(0))
but this is also getting me values which were originally blank.
I need only those values which were originally null not the blanks which got popullated as null
Can someone please help me with this ......I am stuck in bad way
@Aran-Fey can you please look into this
Sure. The conclusion I've reached is that I don't know anything about pyspark.
ohhh my bad any refrences
16:46
You have to know things about pyspark in order to know any references for pyspark ;-)
For future reference, try not to ping other users unless the other person had previously contributed to the conversation
I was stuck in issue so was looking for some help
0
Q: mark null rows excluding blank rows pyspark

user3749031Suppose you have the following data stored in the some_people.csv file: |first_name|age| |:luisa|:23| |""|:45| |bill|""| |bill1|:null| +----------+----+ After importing csv it replaces blank values with null values |first_name| age| | luisa| 23| | null| 45| | bill|null| | bill...

@MisterMiyagi I think I figured it out. The problem is that variance only affects subtype relationships between generic types (i.e. "Is Foo[A] a subtype of Foo[B]?"). By declaring a function like def f(a: T, b: T) -> T, all you're really doing is annotating it as f: Callable[[T, T], T].
You never ask the question "Is f a subtype of <something else>?" - so variance is not relevant here. The question you're asking is "Can f be called with the arguments Sub1() and Sub2()?" and the answer to that is "yes it can". If you set T = Base, that is a valid way to call the function.
@user3749031 Is this problem inextricably related to pyspark? If it only involves .csv files and dataframes, you might get more views if you tag the question with pandas and maybe csv.
Actually "Is f a subtype of <something else>?" is incorrect - the correct question would be "Is Callable[[T, T], T] a subtype of <something else>?"
And of course an MCVE is always a good idea
16:53
18 minutes, nice
Consider: Even though lists are invariant, you can still do my_list: List[int] = [True]. You're not asking if List[bool] is a subtype of List[int], you're asking if a boolean can be stored in a list of ints. And the answer is "yes".
Ooh, markdown rendered the csv data as a table. That's fun. (albeit not super easy to copy-paste)
@Kevin Thanks updated
@user3749031 How are you importing* the csv data? I think there are a couple ways to do it, and they might have different ways of handling empty strings and nulls.
16:57
I am facing same issue while I am importing the table into spark memory from snowflake
(*terminology note: in Python-related conversations, "import" is usually reserved exclusively for the import moduleName statement. A csv file is not a module, so we might call it "loading" when you put its data from the file into an object)
or "reading" a csv works well too
@Aran-Fey I think I see what you are getting at. It could likewise take T as the Union of both argument types, correct?
For example pandas.pydata.org/docs/reference/api/pandas.read_csv.html's na_filter parameter controls how empty strings should be treated
If you're saying that snowflake automatically loads data from the csv file, without you having any control over its configuration, then that's trickier
docs.snowflake.com/en/sql-reference/sql/… looks like it might be helpful
Keeping in mind that I had never heard of Snowflake before the start of this conversation
17:33
@MisterMiyagi Hmm, that is an interesting question. I don't think so. That would defeat the point of the TypeVar, wouldn't it? The idea behind using a TypeVar is that you can give multiple arguments (or the return value) the same type. If you use a Union, that's pointless
Actually, it's not completely pointless because it would make the return type into a Union. Hmm...
TL;DR: I'm leaning towards "no", but I'm not sure
17:47
uh oh... my machine and two servers (in two different data centres) are all throwing (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1122)'))) stuff...
oh... the ecommerce system they're all connecting to is having issues... never mind...
18:07
Why does mypy complain about def f(a: T) -> T: return "foobar" when I call f(23)? Shouldn't it recognize that setting T to object satisfies both types? mypy-play.net/…
This is 30% a tongue-in-cheek question because TypeVars wouldn't be very useful if they worked this way. The other 70% is confusion over why Miyagi's code passes and mine doesn't.
Maybe it's because I wrote a function body instead of "...", so it's being more critical of me.
18:27
... is used for functions in stub files, so that's probably why it doesn't complain about MM's code, yeah
18:58
@Kevin IIRC mypy does not inspect the function body to infer the signature. That makes functions much simpler to reason about, since most of the logic only needs to be validated.
19:48
keycallable, optional
Apply the key function to the values before sorting. This is similar to the key argument in the builtin sorted() function, with the notable difference that this key function should be vectorized. It should expect a Series and return a Series with the same shape as the input. It will be applied to each column in by independently.
trying to work out how to use that elegantly (without a lambda - although not fussed if that's the only way)
I get the impression it would be easier if they'd let you pass in a Series instead of a callable that returns a Series
my test case here is that I know the column is always a string and I want to sort case-insensitive, so I'd in native python use sorted(whatever, key=str.lower) and fine
I tried (although I knew it wouldn't work but took a punt), of trying key=pd.Series.str.lower
20:03
Has anyone tried + succeeded in getting scipy installed on a Macbook with an M1 processor, without resorting to conda? I had a call with a colleague earlier and the problem dissolved during the screen share, so I don't have any real understanding of what was going on... but I guess this will be the new future for us
@roganjosh spec up a decent non-Mac machine and install linux? :p
All our company machines are macs. We have no choice on that one!
weird... last company I worked full time for offered the choice of what you wanted... the network was a mix of Windows, Mac and Linux all working together just fine
I can find threads about M1 processors causing problems, but they're a bit scatter-gun and from different dates and, in the absence of now being able to test on this machine I miraculously fixed by just being in the hangout, I don't really know what the state of play is
@JonClements The justification is that the VPN only works on Macs, which feels a bit weak. <shrug>. This is what we have
I don't follow Mac stuff... but from what I recall... Nick Craver of SO fame (ex-developer on this network) - had trouble with docker and a few other bits with the M1
@roganjosh " a bit weak"? How about just absolute yam?
20:13
I mean, I got a MacbookPro for free, so I'm gonna pick my fights :P
"You mean I have to be furnished with a 16GB RAM, i5 for free because of your VPN? Outrageous!"
Thankfully, mine doesn't have an M1 processor, but it looks like they're swapping the policy on the models they're buying, so I can see this issue becoming very prevalent in future given our hiring rate, but I don't have a grip on what was actually blowing up with scipy :/
When a lot of stuff was Windows based for a company I worked for... and we had computer racks of stuff... and Cisco routers and VPN and all that... it used to be different
you needed a Windows client to do whatever
these days... you really don't... setting up a VPN is either free (if you want to maintain it yourself) or cheap if you want a 3rd party to do it (and they do the infrastructure - but still have no access to the data across it)
Oh, the VPN thing is almost certainly bull, but it's probably a get-out-easy clause for DevOps to pass on to the recruitment team that won't know any of this stuff but have to field "can I pick an different OS for my laptop?" questions from new hires
heh... just got a random playlist going on and youtube.com/watch?v=T-XjUHr3hzU came up... I'm fairly sure if I hadn't of noticed the speed camera on the motorway while boogying and getting a bit foot heavy on the accelerator - I'd have been in big yam for that one
might have been doing slightly more than 70mph (cough cough cough)
20:28
Once upon a time the discussion was about raising the speed limit to 80mph. Now it's how we can spend billions in "Smart" motorways. I'd rather they just replaced the 40 screens with a film so I had something to watch while at a standstill. Probably cheaper too
"smart" motorways are stupid
Without a doubt. A monumental waste of money
I still can't get my head around why they're still pushing on with it... unless they can definitely put all the tech. in, definitely put all the "safe places" should you break down and pull over etc... it's just... it's just... huh? Why?
They don't work anyway. Their selective speed limits at each checkpoint are all messed up (whoever wrote that algorithm needs sacking). It's just another tech company rinsing the govt., though, so what do we expect?
a lane on a motorway that might be a lane, but then suddenly not be a lane but become what was the hard-shoulder very soon... it's just rubbish.
20:38
My mum knew someone in one of the companies fitting them and (apparently) if you change lanes between the checkpoints of average speed checks, they lose track of your car. That's so outrageously bad in design that I couldn't believe it, but that ended in a full argument. This was years ago, but certainly not at the dawn of the tech. I've never been brave enough to test it
even if the tech worked and it was near enough real time - people aren't going to notice it necessarily - and even then, if everyone did, they're going to all try to move from the left lane, potentially at the same time, which is potentially going to cause other incidents
Smart motorways don't just open hard-shoulders. They're supposed to reduce you to 50mph, then delimit you back to 70mph at the next checkpoint so they can try control flow
Traffic tends to work like a wave, so they basically put baffles in to break up the wave pattern. But they can't even tune them properly, so it all just grinds to a halt anyway
just glad I can work remote and get a train if I need to these days - I use to enjoy driving - but meh...
one route I use to take was tortuous... single lane residential area - fine... it's 20, few hundred metres on - it's the same road - the scenario is exactly the same but it's 30... then you're back to 20... you end up spending more time looking for speed cameras and road signs than you do at the road... it's just borked
Well, the answer to that is to move up North :P Then you only have such issues on motorways
(and you have dual carriage ways that are normally 60 but for tiny bits of 'em (with speed cameras) they apparently should be 50))
@roganjosh if what I had left of family wasn't in the southeast mate - I would be.
20:50
Is it bad design to have a property that can raise an error?
What error? I've been having fun with null pointers in Java today which might sway my opinion
@Aran-Fey yes and no ? Could you be more explicit?
Basically "the property you're trying to access doesn't exist yet". The class is Task and the properties in question are result and error - depending on what happened, one or both of them may not be available (yet)
what's the desired behaviour then?
Personal opinion - the attribute should be in __init__ so that I can run dir() on it and at least know the name exists
20:54
@JonClements It'll throw a StateError
dir() will return properties also, won't it?
I should probably just mimic asyncio.Task's interface, to be honest
That's basically how asyncio.Future and concurrent.futures.Future work indeed.
Or maybe not? My Tasks are different in some ways, so being too similar might be confusing for some people. Hmm
Actually, I didn't know this. A.a doesn't exist.. hmm
class A:

    def __init__(self):
        self.a = 1

    def make_b(self):
        self.b = 2

inst = A()
print(dir(A))
I feel like I might have carried some incorrect info around in my head for many years about initialising variables in __init__
Oh yeah, if you don't call it on an instance then you won't see instance attributes
Actually, no, it was a typo here. print(dir(inst)) does what I expect
21:00
stackoverflow.com/a/42276221 just came across this, clicked "delete" and "yes" to add my delete vote, turns out you can retract delete votes now and the ui looks almost identical
@roganjosh seems a lot of people forget you can use the builtin vars(something) :)
Related question: If the task was cancelled, should the exception (property, or getter, or whatever I decide to use) return or raise CancelledError?
@JonClements That still fails:
class A:

    def __init__(self):
        self.a = 1

    def make_b(self):
        self.b = 2

inst = A()
print(vars(inst))
The point I'm making is that it's handy to know that A can have a b attribute before actually using it (IMO). Then I can search for it in the docs
In my case it will either be a property or a getter function, both of which are class attributes, so no need to do anything in __init__
@roganjosh that's rather presumptuous though - you can never know what a class/instance of a class might/might not have
21:05
I don't think it's presumptuous, though I'm not going to argue that it's watertight
For example (lifted right from my own work library) this
(and the one linked in comments)
@Aran-Fey you mean task.exception?
voted, I'll just have to retrain years of muscle memory not to blindly click yes on the dialog for now
@roganjosh could have? How is anything run-time based ever going to figure that one out?
This is what we've been training for all our lives: not to read banners
@AndrasDeak yep
21:11
@JonClements I don't follow, unless you're classing the programmer in "anything"?
despite reading all the objections - how does everyone feel about the new granular "last seen" stuff on user profiles?
@JonClements useless
Related to the question that "answer" was on, turns out socket.fromfd has worked on Windows for quite some time, we just never updated the compat code that used the less direct workaround that was addressing that issue. I've been removing so much deprecated and leftover code for all the next Pallets versions.
@roganjosh no, no... perhaps getting tired and failing to express myself properly
(long day as usual)
@Aran-Fey note that I still don't do (nor understand) anything async. But I would expect a task.exception to be an exception rather than raise one.
21:13
from beatroute.core import Problem

problem = Problem('2021-01-01 00:00:01')

print(dir(problem))
@JonClements This is my only rebuttal of doing it this way anyway:
it's easy to raise it if you need to, but the other way around you'd need a try-except around it just to figure out if there was an exception and what it was
Hmm, internet posted that the wrong way around
@AndrasDeak That aligns with my expectations as well... but it's different from what asyncio.Task does. Which is why I'm conflicted about this
yesterday, by Aran-Fey
I swear the standard library is the most unpythonic thing I've ever seen
haha, good point
21:15
[ '_driver_name_map', '_end_time', '_is_built', '_is_solved', '_job_map', '_max_distance_between_locations', '_max_time_between_locations', '_meta_map', '_payload', '_plottable_locations', '_plottable_start_ends', '_send_time', '_url', '_vehicle_type_map', 'add_consecutive_jobs', 'add_job', 'add_location_incentive', ...]`
but yeah, if you want your task to be a duck async task then your hands are tied
@roganjosh I'd rather not rely on the default dir... put a custom method that makes it more easy
My router did a real number on me there, posting things out of sync. But the point is that I only needed to instantiate Problem() and I immediately know all the methods and attributes before I use it
I'm not really trying to be an asyncio.Task... the class is just conceptually very similar. It does have some key differences though, so it's probably a good idea to aim for a noticeably different interface
yeah, if the choice is either sane or asyncio API then go with the former :P
(someone, somewhere, 10 years from now: "damn it, why does aransyncio.Task have to be slightly different in every regard!")
21:21
@JonClements what custom method would you recommend?
Hahaha, I'm totally stealing that name for one of my future projects :D
my gift to mankind
@roganjosh ummm... without "in" knowledge - my best guess is i'd probably do a dataclass of what you have and reference that inside an object that works with that data.
that way you abstract the data and the methods
That object does deal with the data already. It'll handle the defaults of None for all the variables I initialise. I'm not sure why anyone should create a Problem() instance with a single bit of data and then delegate all the defaults to some other object
(internally in the library)
21:33
My last message was nonsense, sorry. But I'm also going through Flask's code to see how dir() should work. It'd be easier if you gave an example for "i'd probably do a dataclass of what you have and reference that inside an object that works with that data." for:
class A:

    def __init__(self):
        self.a = 1

    def make_b(self):
        self.b = 2

inst = A()
print(dir(inst))
Instead of:
class A:

    def __init__(self):
        self.a = 1
        self.b = None

    def make_b(self):
        self.b = 2

inst = A()
print(dir(inst))
okay... I need to sleep mate... still got long-covid sh*t and I get tired so easily
catch up tomorrow
To be clear, I'm not arguing with you, I just don't know if I'm breaking convention. I hope you're feeling better tomorrow, rest up mate
21:51
I have a feeling you two were talking about different things
...I would elaborate if I could, but that's all I've got
I mean that's only fair, since the whole discussion probably stemmed from a misunderstanding of your problem :)
22:09
I'm yet to try @property just to confuse things more :P
But I'll probably keep that one back :)
 
1 hour later…
23:19
"What attributes could this object possibly have under any set of circumstances?" is halting-problem-hard, but perhaps you could apply some reasonable-ish constraints to turn it into something solvable at compile time. Like, "no direct manipulation of __dict__ allowed", and "all classes have globally unique names", and "use the name self for each method's first parameter, and don't reassign it"
Then you can scan over a class definition, look for self.thing = ... statements, look for @property, recurse into the class' parents, etc. With a fair amount of confidence that you won't get any false positives or negatives unless the code's author is intentionally trying to foil you
"is halting-problem-hard" I can't disagree because you can tag anything on. But I can initialise all the variables that I intend it to use so that you can ask "well, I have this object. What can I do with it?"
In the abstract, it might show me that frobnicate is in dir() and so that would give me a lead on what to search, because the name sounds about right
Although frobnicate would probably be a method vs. just an attribute

« first day (4049 days earlier)      last day (1121 days later) »