« first day (4231 days earlier)      last day (700 days later) » 

12:07 AM
Well, fortunately I solved the problem myself!! I found that I had a bizarre column called "Unnamed: 0" that had the indexes of the rows sorted in descending order. I just had to delete this column.
I did not know how that happened
Thanks anyway Andras and Karl
Oh no, it was not solved... in my original dataframe the problem remains... But I will check it
This new bizarre column appeared after I loaded the dataframe from the csv
And in this dataframe the index column is ok
 
I have this issue that only happens on MacOS; on Arch Linux, no problems. When I call subprocess.communicate() in a thread, it hangs, even after the process ends. It was talked about here: stackoverflow.com/questions/984941/…, but OP never said how he fixed it.
 
Well, now I've definitely resolved it!!
Just doing it:
weird_df = weird_df.reset_index(drop=True)
SO reference that helped me: stackoverflow.com/questions/33165734/…
 
12:23 AM
@DUOLabs duckduckgo.com/?q=subprocess.communicate+hangs ? it's hard to be more specific without an MRE
what overall problem are you trying to solve?
 
Re: regarding an MRE, I can't replicate it outside the code in my application, but it's so embedded, that the only way to test it is by running the entire script (I know it's a problem with subprocess, as if I remove the call, there's no more blocking). In my program, I have some commands that run in a thread. However, only the first command can execute, then after that, nothing runs.
 
12:49 AM
what do the "commands" entail?
and what do you mean "nothing runs"? does the thread terminate? is it hanging? in some kind of zombie state?
 
1:18 AM
Just things like running grep and ls. The group of commands run in a function, which is the target of the thread. The function loops every 10 seconds.
 
1:33 AM
After the first command in the function runs, none of the other commands run.
 
Avv
1:44 AM
Hello Guys,

Do you have a good resource where I can start learning information retrieval in Python please?
 
2:03 AM
What kind of retrieval?
Also, I found that a lot of things in my scripts do not work on Darwin: killing processes result in a ProcessLookupError, and SIGTERM doesn't execute the function assigned to it.
 
 
2 hours later…
4:19 AM
@DUOLabs I hope you aren't trying to parse the output of ls. That's not a good idea. Please see unix.stackexchange.com/q/128985/88378 & mywiki.wooledge.org/ParsingLs If you want info from a filesystem, use the appropriate tools in the Python stdlib, don't call shell commands with subprocess.
 
 
2 hours later…
6:35 AM
@Marco it's not a bizarre column, it's just unnamed, as the name suggests. If you read a CSV the top row will be interpreted as the column names. If there's a blank entry in that row then it will get that name by default. Any number of columns can get the default name
 
@MisterMiyagi just to settle my question.. i ended up writing a custom sleeper for sched.scheduler having one conditional variable waiting either the required amount of time or a notice from another thread triggered by wake up from suspension (i use dbus since i couldn't catch the SIGSTOP)
thanks again yall
 
6:56 AM
@Marco I created a brief list of Python podcasts and some additional pointers in a gist gist.github.com/tripleee/e551469fdb8d24a99880ff5e4527d480
 
Hello All... Is it possible for someone to look at a problem i'm trying to solve using pandas/python
 
I don't remember offhand which guests were on which podcasts but Brett Cannon has been a frequent guest on several of these shows I believe; Barry Warsaw and Pablo Galindo Salgado were on several too I believe, and of course Lukasz Langa regarding his role as Developer in Residence
@NorwegianSalmon please review the room rules; your question should be 48 hours old before you bring it up here
 
@Michele Nice, that seems like a good solution!
 
7:23 AM
@tripleee apologies, shall wait for the prescribed time if no solutions. thank you
 
 
2 hours later…
8:57 AM
cbg
 
hello guys
url = "https://maps.googleapis.com/maps/api/place/textsearch/json?"
I am using this url to search for places in google maps, and getting results capped 20
is it possible to get more than that?
url = "https://maps.googleapis.com/maps/api/place/textsearch/json?"
query = 'ATM new delhi'

r = requests.get(url + 'query=' + query +'&key=' + api_key)
 
> Returns up to 20 results from a previously run search. Setting a pagetoken parameter will execute a search with the same parameters used previously — all parameters other than pagetoken will be ignored.
 
9:28 AM
I've seen something strange right now. Is there a way to execute arbitrary code on LOAD_NAME of an object? e.g.
x = MyMagicObject()
x  # can I make this line do something? I always assumed this will always be a no op
 
Well, it'll retrieve x from the dict that represents the local scope. So if you can set that dict, you can do anything you want
class MyDict(dict):
    def __getitem__(self, name):
        print('hi there')

exec('x', MyDict())
 
@Arne probably irrelevant, but for completeness' sake: just x in a REPL will call __repr__()
 
@Arne Under normal conditions, that should be a no op.
sees Aran's FrameDict on main "normal" conditions
 
9:54 AM
Hey it's still a no-op, even if some weirdo injected new variables into your scope
 
 
3 hours later…
1:01 PM
@PM2Ring No, it's just an example.
 
1:55 PM
I would use multiprocessing, but then it says it can't pickle my function.
 
That sounds easy to fix. Just use a global function, and not a lambda
 
It is a function, not a lambda. Also, my function has to be local, for different reasons.
The reason I say this is because there's a signal.pause() right after the thread, so it's possible that for some reason, Python is stopping on the pause on the main thread, instead of switching back to the subthread.
 
I don't see what signals have to do with function scopes
 
signals+threads is evil, though.
 
No, I mean, I start a thread like this threading.Thread(target=func,daemon=True).start(), then after that, I do signal.pause().
 
2:06 PM
If you're having problems with process and threads while there are signals around, I blame signals.
 
So, Python may be stuck on the pause, instead of switching back to the thread. That doesn't explain why it doesn't work on MacOS.
Is there another way to have indefinite sleep (similar to sleep infinity)
 
Umm, I'm talking about multiprocessing here
 
@DUOLabs time.sleep(float("inf"))?
Events?
 
For time, I get OverflowError: timestamp too large to convert to C _PyTime_t
 
Just to be clear we are solving the same problem: When should the main thread wake up again?
 
2:11 PM
@MisterMiyagi It works!
Basically, func is a list of subprocesses, then a time.sleep in a while True loop.
Thank you so much.
It also fixed a lot of other problems concerning handing signal.SIGTERM
 
2:29 PM
Don't know why it only happens on Darwin, though.
 
Signal handling is different on BSD versus Linux systems.
So are threads and subprocesses, come to think of it...
 
2:59 PM
Hello
I got a problem , i dont want answer but recommendation.
Using Asyncio , what project would you recommend me , i can build?
 
3:20 PM
I just wrote my first python script today since last year, it felt weird laurel
 
3:47 PM
I used loop.run_in_executor to make a synchronous function async, but now I'm wondering 1) why asyncio even has an executor and 2) how likely it is that run_in_executor tries to pickle my function
Or I guess to make the 2nd question more general: Is run_in_executor preferred over manually starting a thread that does the work?
Oh and I should mention that I need a way to communicate with the function. Right now I'm just assuming it's executed in a thread, so I just used a dict to store all my state in.
 
4:22 PM
If you're passing ThreadPoolExecutor to run_in_executor, I expect that it won't try to pickle anything
 
Oh, right. No, I'm passing None. Not much of a point to make a threadpool for 1 thread, at that point I'd just start the thread manually
 
5:07 PM
@Aran-Fey the default executor is guaranteed to use threads, so you're safe about pickling.
 
@Becauseihatemyself the simplest thing you can build with Async IO is a web scraper.
 
Using the executor or even to_thread (or whatever it is called) safes you much hassle of bridging threads to asyncio wrt signaling when things are done. It's not rocket science though,and if you don't await the thread there is no advantage.
 
building a google trends based python package popularity index could be useful.
something like this index, but for package not for languages:
https://pypl.github.io/PYPL.html
 
5:23 PM
@MisterMiyagi Apparently to_thread just calls loop.run_in_executor(None, so I guess it's safe to assume that the executor will always be thread-based
Installing a ProcessPoolExecutor might make for some cool fireworks, maybe I'll try that sometime
 
TypeError: Loop.run_in_executor expected 2 arguments, got 1 :P
 
Better a type than a syntax :P
 
5:45 PM
@Aran-Fey at least for the default executor there is a type check that it's a thread pool. So for the default you are always safe.
 
Oh, great. Nothing to worry about then
 
TBH I think it's a bit shaky design to have something called to_thread use a pool of limited size. But it's probably large enough for all practical problems.
Until it isn't and weird stuff happens, of course.
 
better design is to rename it two_thread and restrict it to 2 PEs?
 
I remember being severely disappointed when I first saw the implementation of to_thread
Just run_in_executor(None, ...)? Come on
 
6:02 PM
What's wrong with that?
 
I'd prefer a different executor or just more interesting/low-level stuff
The current implementation doesn't need to exist in the standard library IMO
 
It's convenient and more self-documenting than run_in_executor, so I think it's a nice addition
 
Core devs rejected a lot of proposed additions to the standard library because "an average Python programmer would be able to implement them easily and correctly", meaning that standard library should contain implementations of things that are hard to get right
loop = events.get_running_loop()
ctx = contextvars.copy_context()
func_call = functools.partial(ctx.run, func, *args, **kwargs)
return await loop.run_in_executor(None, func_call)
There's nothing tricky about this implementation, and you'll probably need to supply your own executor at some point
Besides, how often do you think you need to copy_context?
It's O(1), but still redundant in many cases
 
6:18 PM
To me, you're describing perfect python. Batteries included, self-documenting, and does everything, even though you don't always need everything
 
Doesn't it bother you that it implicitly decides which executor to use (the default one!) and that you have no control over it?
(apart from being able to set the global default executor, of course)
 
Meh, I feel to_thread is perfectly fine because it is what 99% of people want. The run_in_executor API is mostly old cruft, or at least pretty low-level.
Better to have a shiny and shitty way of doing things than just a shitty way.
 
What he said. All I really care about is that I can call a function without blocking the rest of my code
 
Argghh, the peer pressure
 
Come to think of it, threading might do well having a similar function. :P
 
6:33 PM
Hmm, would that return a Future? Or just call a function in a thread without any way to access its return value and/or exception?
 
I'd prefer a Future or at least something similar. Having no return value for threads/processes is super annoying.
Sure, I like to use them for side-effects as much as the next mad hermit, but deep down I know that proper code is functional'ish and returns stuff.
 
The threading module feels pretty low-level, so getting a Future from it feels... weird
 
Well, there's concurrent waiting to be filled...
 
7:01 PM
Thanks for the input on my question, had to log off right after posting that. I'll spend some time to debug a file then where a coworker told me "it stops working if I remove that name lookup"
 
ask for an MCVE ;)
or if they are using a jupyter notebook
 
7:25 PM
@Aran-Fey thank you
 
7:46 PM
@KarlKnechtel @tripleee I see was thinking there was a general practice like enclosing very long strings with some character to prevent any possibility of strings breaking
By the way how do you backtrack and find previous chat made here? Trying to remember the solution to a problem from last year
 
8:32 PM
@roganjosh Okay, but I said that the column is bizarre because this column didn't exist, I don't know how it appeared. This only occurred when I converted my original dataframe to csv and then loaded it. In my original dataframe this did not happen. But none of that matters, what matters is that the solution in relation to my original dataframe was with that method I mentioned that resets row index.
 
I know exactly why it appeared, and it's a mega pain. When you did .to_csv() you didn't pass index=False as an argument
 
@tripleee Awesome, I had a look and then I'll see more calmly... thank you very much!!
 
It drives me nuts because I dump stuff to CSVs all the time and, in my not so humble opinion, the default should be to not include the index, which contains no header
If you happen to dump to CSV then read it back in and dump it again somewhere else, whilst forgetting index=False you'll just continue to accumulate Unnamed: N columns. Because that's user-friendly
 
@Pherdindy The search page, mostly. chat.stackoverflow.com/search?q=&room=6
It's not especially good
For example I can't even find what problem you were discussing with KarlKnechtel and tripleee
 
Roganjosh: nice to know this (although I don't quite understand why it is necessary to set index = False)... But there's a detail, in addition to that bizarre index row column (with descending ordering), the normal index row column (with ascending ordering) was also there...
 
8:39 PM
My advice for ensuring there won't be any accidental escape characters, is to sit down and formally prove that your serialization and deserialization algorithms are 1-to-1 inverses of one another.
 
@Marco At best I would be speculating. You might have a multi-index in your df. I kinda just trained myself to always use index=False. Separately, at the risk of sending mixed messages (sorry), you're fine to ping me if there are multiple convos happening at the same time (as now). It only bugs me when it's just us two chatting and it's obvious who you're replying to
 
And yes, I am aware of the irony of suggesting this, while roganjosh and Marco lament the non 1-to-1 invertability of .to_csv
 
That's, like, the opposite of irony
 
There are at least three kinds of irony, so I bet I fall under at least one of them
 
@roganjosh Ok, thanks again
 
8:44 PM
I think we just need Alanis Morissette to come in and settle this once and for all. The irony of her song is that there's no irony in the examples, which is itself irony.... RecursionError
 
@Kevin Wow, I think this is a little complex to me, but no problem :)
 
I could have maintained Dramatic Irony if I pretended that I was completely unaware of roganjosh's and Marco's chat. But the idea of doing so, did not spark joy.
 
hahaha, ok, just kidding, so
 
@Marco Well if you boil it down to the basic principle, you just have to think real hard about corner cases where serialize(deserialize(x)) != x, until you're satisfied there aren't any
 
@Kevin thanks found the answer I was looking for it was made July 14th of last year
 
8:48 PM
I still need to study serialization :(
I just try to learn a little bit of various things
So I always know too little of all
 
json is good for serializing when memory is not a constraint. struct is good for serializing when developer time and sanity are not constraints
 
And the .npy ones?
 
if you already have numpy arrays then npy is fine
or npz, the compressed version
 
At least to me this is always an option
 
I don't know what npy is, but the first page of google seems to have a positive opinion of it. I approve.
 
8:52 PM
It's a format to export *drumroll* numpy arrays with
you can technically put arbitrary objects into numpy arrays, and you can technically save them to npy, but then you could just be using pickle without the middleman
 
I had a suspicion it was numpy related, but I was skimming too fast to make any firm deductions
 
Maybe json = npy > csv > pickle?
In terms of good options
 
yes, csv is at least twice as good as pickle, maybe even 2.5
 
"Article has clean white background, no images of burning garbage fires or malefactors in balaclavas. Sentiment analysis: positive"
 
@AndrasDeak--СлаваУкраїні I did not know that
@AndrasDeak--СлаваУкраїні Right
 
8:55 PM
allow_pickle=True is what puts the "um" back in .npy
 
So it's possible to use pickle with allow_pickle = false?
 
@Marco well, you can have both compressed and uncompressed npz files, see savez vs savez_compressed. The difference is like that between .tar and .tar.gz.
@Marco what?
 
@Marco It's a rock-paper-scissors situation
 
I thought that was related with "using pickle without the middleman"
 
@Marco numpy is the middleman
 
8:57 PM
Oh, ok
Sorry
I wrote without thinking straight
 
You said "at least to me [npy] is always an option". You can technically save a dict to a .npy file, but only with allow_pickle=True because you can only put a dict into an object-dtype array, which should never* be used, and using npy like that will be equivalent to using pickle, which is insecure and not portable. So don't use .npy files for anything other than proper numpy arrays. It's not "always an option".
 
Ok, nice
And what is the proper way to save pandas dataframes?
 
probably with one of the dozen to_*() methods
 
None preferable?
 
Related to the topic. My current dataframe has 578 rows of data. When I export it using df.to_csv() it becomes 580 rows with 2 rows of broken strings. But when I do df.read_csv() it is fixed again and back at 578 rows. I have no clue what escape character broke it
 
9:04 PM
What a mistery
 
Wait wait wait, df.to_csv() modifies df? O.o
 
@Pherdindy probably two \n inside strings, which is shown as 2 rows with "broken strings" because you're not using a csv parser
@Aran-Fey nah, just opening the file with notepad, I'm sure
 
I'm using microsoft excel
 
9:06 PM
you can see what row gets mangled so it's not that hard to figure out what string breaks it
 
Excel's encoding problem probably
 
Yeah the string before it was the broken one. Although it broke a list which has no escape characters
 
Excel's microsoft problem. Probably implementing a known bug to be compliant with lotus from '95 or something.
@Pherdindy what is "a list" in a csv?
 
@AndrasDeak--СлаваУкраїні I think this is the perfect answer
 
My data looks like [datetime.datetime(2020, 11, 5, 0, 0), datetime.datetime(2021, 9, 15, 0, 0), datetime.datetime(2020, 2, 13, 0, 0), datetime.datetime(2021, 9, 23, 0, 0)] and it only contains datetime.datetime objects
 
9:11 PM
last time I checked datetime.datetime was not a valid type in a csv
 
But it starts splitting somewhere in the middle of this list for both the broken rows
 
all those commas in the function calls are interpreted as field separators, which makes me wonder what you're actually doing
since it's datetime.datetime and not a pandas datetime it might be something wonky that pandas does, instead of raising an error
pandas has some questionable fallbacks, I could imagine that it switches to saving the repr for unknown objects
I'd use stdlib csv.reader to look at the few surrounding lines to see exactly what they read back as, assuming it's even valid csv
I'm also curious why only one or two of your rows contains a list of datetimes when the others don't, assuming that's the case. It would be a huge design code smell.
 
All rows have it just that the broken row kinds of makes it own columns
Yeah I suppose it has something to do with the commas then
 
OK. Perhaps that one row is missing the quotes around the list for some reason. Like I said: look at it with a decent text editor or a decent csv parser. Maybe both.
 
When it comes to dates I have always wondered what format I should export it as
Right will take a look at those options
 
9:23 PM
Wouldn't it be a good idea to do that?
DataFrame.to_excel(excel_writer[, ...])
If the biggest intention is just to read in excel
By chance I was looking at a pandas doc's page (pandas.pydata.org/docs/reference/io.html#) and I found this possibility
 
I will try using .to_excel() but my main purpose is just for data storage I don't do any work in excel since I really don't have any knowledge on databases like SQL
I didn't want to bother finding a different medium
 
Ok
 
Since learning SQL will take more time lol sadly
 
So you could just try to convert to JSON
I think it could be a better idea
 
Yeah i'll rethink how I structured this one
 
9:26 PM
DataFrame.to_json([path_or_buf, orient, ...])
 
I am actually searching an ID from each data set into an API to make some time series analysis
 
Ok
 
So I thought I could just stuff a list of dates
 
Converting to JSON and converting to CSV are fundamentally different. You can't just iterate JSON without loading the entire thing in memory
 
Good point that I didn't know
 
9:36 PM
A major bugbear of mine, which hasn't come up for a couple of years, granted, is people building backend storage in JSON because it's "easier" than going with sqlite3. This is just wrong
If there's any hint of scope creep, drop your JSON ambitions. It will end in pain
 
So completing: SQL > CSV > NPY > JSON > PICKLE
IMO
 
I don't have anything against JSON, and I use it all the time. But you'll hit a ceiling if you're using it as a backend storage and you'll have to rip everything apart if that data store can continue to grow through any means
 
Nice to know these things
 
@Marco 3 > 4j > potato > blue >
 
WTF
 
9:40 PM
Andras is making a comment about how those formats aren't comparable
 
Yeah, I know that each case is a case, but I think this works in general
 
Flip it around; how did you come up with that ordering?
 
I am not saying that in any situation you should opt for SQL, per example, just that it may be the most appropriate option for storing and retrieving data.
 
But you just ordered something, so there must be a reasoning behind it
 
Data storage options
 
9:45 PM
I think you're missing the point. I didn't ask you about data storage, I asked you on what basis you formed that ranking of preference for data storage
"for data formats"*. I wasn't just thinking about storage, but you've proffered a suggested hierarchy for everyone and I want to know what underpins it
 
Well, PICKLE is insecure, etc

JSON has these advantages that you cited

From what I understand from Andras' explanation that you should save in npy only when working with numpy arrays, so if you're working with dataframes it wouldn't be a good option

CSV is a storage format that does not have the disadvantages of the others mentioned.

And about SQL I think that does not have disadvantages over CSV and have the advantages to be in a RDBMS, but I think would be better adjust to SQL = CSV
 
I don't think I listed any advantages of JSON?
 
I think no
Oops
 
"JSON has these advantages that you cited"
 
I mean disadvantages
sorry
 
9:55 PM
CSV is kind of a trap. Most people seem to forget that CSV doesn't even have a concept of data types. You have no idea if a value was a string, or a number, or something else
 
I agree
 
@Aran-Fey aren't numbers valid fields as opposed to strings?
otherwise quotes around strings would be redundant
 
But even over these JSON's disadvantages that roganjosh cited, JSON > CSV?
 
CSV can differentiate between numbers (int/float) and strings. You've got me doubting myself
 
@AndrasDeak--СлаваУкраїні well not necessarily, because you'd still have to protect field separators inside strings
 
9:58 PM
Quotes just instruct the parser to ignore commas until the quote ends. But you don't have to put quotes around strings if they don't contain a comma
 
can confirm at least with the default (excel...) dialect that stdlib csv.reader returns strings even for number-looking fields
 
Well, I suppose another reason why it's a trap is because there isn't really a CSV specification. It's possible that some parsers differentiate between strings and numbers based on the presence of quotes, but generally speaking, CSV doesn't do that
 
@Marco let me tie up my train of thought. You're espousing ideas on what people should or should not do, or ordering storage formats/query formats etc. as though it's an informed opinion, but we're seeing that it isn't. If you don't know, you don't know. I respect that far more than things being presented as fact that others might take away
 
csv.QUOTE_NONNUMERIC

    Instructs writer objects to quote all non-numeric fields.

    Instructs the reader to convert all non-quoted fields to type float.
I guess that's how you can tell csv.reader/writer to automate that for you
 
10:23 PM
44 messages moved to MetaPython
please continue the meta discussion in the meta room
1 message moved to MetaPython
>>> with open('foo.csv') as f:
...     next(csv.reader(f, quoting=csv.QUOTE_NONNUMERIC))
...
[1.0, 2.0, 3.0]
it works too
1 message moved to MetaPython
 
11:21 PM
@Aran-Fey Nice.
 

« first day (4231 days earlier)      last day (700 days later) »