« first day (4255 days earlier)      last day (915 days later) » 

00:01
My concern is actually something else. I don't really care why .env files are so popular; they're an adequate solution. If you want to use them, go for it. But they're not necessarily the best solution, so when I see someone struggle to define envvars in PyCharm, I want them to consider other options
That's fair enough. We perhaps debated this in more detail than necessary (triggered by me)
I do think it's mostly just convention, but there could well be benefits that I'm not aware of. I've never deployed a web server. Or a database. Or anything, really.
I think it does come into prominence with the AWS tenants but I can't speak for whether that's the best practice. If someone shoots it down then I'll raise it, which is why I wanted to probe this a bit further
One day I was annoyed because I couldn't figure out how to set my PYTHONPATH on linux, so I started thinking about why env vars even exist, and now we're here
That's my one redeeming quality, I think about stuff a lot
I still don't know why PYTHONPATH is an env var and not an entry in a config file though
00:30
because PATH has always been a thing and Python, back in the early 90s, thought it would be cute to emulate that
01:16
Any cv enthusiasts here? :)
 
2 hours later…
03:41
Definitely! Lots of close-vote enthusiasts around.
04:22
sometimes I wonder if I close too many questions...
04:39
Seems like we've still got plenty.
 
2 hours later…
07:08
@roganjosh To put this into perspective: Using .env for security is – IMO – perfectly fine when a malicious actor doesn't have access to it. That would be the case for a website which I assume you run on a single-user system without any untrusted subprocesses.
If someone breaches that, then it doesn't matter where your passwords are stored. They'll have access anyways.
 
3 hours later…
I'm stuck in speeding up a for loop
I need to apply a function on every row of dataframe (row count ~ 100k)
I have tried with basic functionality in pandas as well as multiprocessing
The issue with multiprocessing is I am not able to retrieve output at every step
Is there a way out?
Is the problem that the function is slow or that you have a lot of data?
I guess with 100k rows it's probably the former
What does the function do? You should look to vectorization before multiprocessing
@MisterMiyagi Cool, thanks. As long as I wasn't off-the-ball on something
10:48
Yeah function has nested loops for different conditions
+1 for vectorization. basically, get rid of the for loop, do operations along the entire column if possible. multiprocessing itself isnt a silver bullet, and can sometimes introduce more headaches than its worth. you first should work on optimizing the code itself
sidenote, you may not know this, but iterating on dataframes is one of the worst things you can do performance wise. you want to keep for loops as low as possible. (also, apply internally is very similar to a for loop as well, so by extension, if you can avoid apply, you should)
So... what exactly does the function do?
And is it required to work on a dataframe or could you for example trivially parallelise the data reading and processing?
It checks for 3-4 conditions and return a boolean value
Input also takes couple of dataframes
11:04
why does it need nested loops?
It would be much easier for us to help you if we could see the function in a minimal example
and why are there two separate dataframe inputs? are they similarly sized?
Well, step 1 is to replace those | and & with boolean operators
With some luck, you won't even need a step 2
(upc in upc2Cat3BV["UPC"].values) & ((upc2Cat3BV[upc2Cat3BV == upc] this is basically the same operation twice
That code doesn't strike me as "pandas". If some of those names are pandas/numpy objects, binary ops are necessary
11:14
They are pandas objects
some are dicts
Yeah, that's kind of a huge difference.
I don't get the point of the final processBrandVariants check. That block seems identical to what was already done before.
We can set it to false later depending on the need
But the same code already runs as part of the first else:else: path.
Good catch
11:21
@Aran-Fey To do this, we have to check for the value in column and look for same value in another dict in order to confirm
Hm, I am guessing you can make this a lot faster by first doing a bulk/vectorized operation for the first if case, then do a second pass processing only the remaining false'y entries.
Do you know which of the cases are hit most frequently?
E.g. how many actually need the expensive brandDao or canonicalBrandDao handling?
Actually, I don't understand what upc2Cat3BV[upc2Cat3BV == upc].values[0] is doing. Doing df[df == x] returns a df where almost everything is set to NaN, doesn't it? So .values[0] is almost always an array of NaNs?
>>> df
   UPC  CAT3
0    0   100
1    1   101
>>> df[df == 1]
   UPC  CAT3
0  NaN   NaN
1  1.0   NaN
>>> df[df == 1].values[0]
array([nan, nan])
@Aran-Fey It is getting the value of "CAT3" from upc2Cat3BV where upc column value == upc which we pass as arg
@MisterMiyagi almost 55-60% goes to condition 1
Oh right, I forgot about the ['CAT3']. But that just means it returns a single NaN instead of an array of NaNs
@Aran-Fey This reminds me of a pattern I sometimes see in non-pandas code: if key in the_dict and the_dict[key] == value:. Since that can be simplified to if the_dict.get(key) == value:,* I wonder if this pandas code can also be simplified in a similar way.
(*assuming that value is guaranteed not to be None)
11:31
@Aran-Fey Oh I forgot to put "UPC" column name as well in the code.
Here's my attempt at a rewrite
Oh, and like Miyagi said, the if processBrandVariants: (and everything inside of it) can be removed
Alright. I'll give it a try and let you guys know. Thanks a lot.
I did try multiprocessing as well with async, but every output is a generator and to decipher that is another for loop
multiprocessing and async are different things.
I meant both
Also, catching up on conversation now, is this function going to be used along with a .apply on the dataframe, or or is just a function that's being passed a dataframe?
11:45
Async doesn't make your code faster, so that's pointless here
You should stop throwing random technologies at the problem, and instead optimize your code. Because you used | and & instead of or and and, your code does a lot of unnecessary computation. And then it does it twice. And it doesn't even do it correctly
upc in upc2Cat3BV["UPC"].values seems to be missing parens?
it's also not clear what this does have to do with pandas... what things are dataframes here?
the ones where you think .values are missing parens are probably all dataframes
Hello, how can I group data per minute if I have thousands of values in my time column as of: 2021-08-30T00:00:00Z Form?
What type is the column?
datetime64[ns, UTC]
11:53
(or is "dtype" the correct term? Assume I used whichever word that makes more sense)
Ok, that answers my question, whatever my question was
@Aran-Fey thanks
derive a column from the time data to get a column that can be used for grouping, i guess? theres probably some method to extract just the minute and so on
But how can I group for a minute?
I suspect groupby will be useful. It seems pretty versatile, so maybe you can get it to group by minute without deriving any columns beforehand
My first search result for "pandas group by minute" is a SO question named "How to groupby time series by 10 minutes using pandas". So close :(
But wait, the 2nd sentence in the question is "I know how to group by 1 minute"
11:59
my search was "how to group data by the hour python" (because i presumed grouping by hour is more common) and i ended up with this which funnily enough has an hour and minute aggregation, so...yay me i guess?
the approach in the answer would have to be tweaked to account for the date also, but otherwise it seems like a fairly reasonable starting point. I will say i was imagining something even simpler, preferrably only a single column with the time converted to the nearest minute
You may find more approaches if you research "binning" in addition to "grouping by"
ooh, dt.floor looks like what i had in mind
@Aran-Fey I don't see why the specific time interval would make a meaningful difference. Or was there a bit of irony in there :)
12:04
Maybe I can ask differentyl
Trades.resample('S', on='Time').Volume.sum()
This works, grouping per second
Trades.resample('H', on='Time').Volume.sum()
This works, grouping per hour
But how to group per minute? M is month, m doesn't work, mm doesn't work..
ahh. i see. cases like these usually you want to start with documentation
if you already know a function or method that works, but dont know what to pass it, i'd start by going to it's documentation first, then i'd start digging
pandas.pydata.org/docs/reference/api/… doesn't directly describe what "S" and "H" mean for its first argument, but if you follow some of the links, you can find a table of valid arguments. And there's one for minute :-)
So, if i start here i see two things: one is a mention under Notes for another page, and another is an example that uses 3T for 3 minutes. I presume it's safe to speculate that you can use 1T for 1 minute, but i also went on the link mentioned in the notes which also showed examples with '5Min' for 5 minutes.
Seems like theres a few ways it accepts things
Thanks to all!
I'm curious if "60S" would also work. ("But what about leap seconds?", asks the devil in the details. Quiet, you.)
12:11
i would be fairly confident 60S would work, and also fairly confident leap seconds can just go in their corner and stay there
@KarlKnechtel I'd prefer not to answer that (:
@Kevin
Min works
It was just a question of wording/ syntax
As is often the case
Maybe you can help me with one further question. Do you know if it is possible to group by Price.VWAP() (volume weighted average price) instead of Price.mean(). Meaning, do you know if you can define your own formular to put instead of mean(); sum(); min(), etc?
https://stackoverflow.com/questions/72574126

I feel like there is the core of a useful reference question here, but I can't tell how the question would be phrased in order to introduce the topic properly. There are already canonicals for how `**` works both in parameter lists and in call arguments, but it might not be clear how those two things interact.
(oh, OP deleted it after my last comment. I guess the problem was solved, but.)
12:56
I'm sorry, I'm stuck again. This doesn't really save the problem
Trades = Trades.groupby(Trades.Time, group_keys = False).apply(vwap). (Time = columns with times as of form: 2021-08-31 17:29:29.082000+00:00 ). I still (after testing all the suggestions) don't know how to group the Trades per minute here. can someone help?
So you decided not to use Resample? Was it giving you trouble?
Because with.resample, I can not apply my custom made vwap function
Trades = Trades.resample('S', on='Time').apply(vwap). doesn't work, that's why I tried now groupby
13:14
Oh, I'm surprised, I thought that would work. Did you get an error message? If so, what does it say?
Yes, error message. Content: AttributeError: 'Series' object has no attribute 'Volume'
Volume is one attribute used to calulcate the vwap..
a column in the Trades df
I guess that means that vwap only has access to the time column, and not any other column
mh I have to think about that, not sure that I understand what that implicates..
Hello, i am trying to use exec for a string statament in python.
When the string statement is builded (+) with a string variable, when the string variable has " character for example: window.MP3jPLAYERS[0]["list"][0].mp3 then there is a problem.
One solution i thought is to replace the " char with ' char, but i don't know if this catch all the possible cases.

For example the string variable may have and " char and ' char...
Any help?
The implication might be that resample is not the right tool for this job. I must ponder this.
13:22
@ChrisP Don't manually add quotes, just repr the string
>>> text = 'foo["bar"]'
>>> print(f'a = {text!r}')
a = 'foo["bar"]'
Well this statement: self.playlist["items"][0]["javascript_code"] = "window.MP3jPLAYERS[0]["list"][0].mp3" fails with syntaxError.
Obligatory warning: you should almost never use exec in serious code.
and this: self.playlist["items"][0]["javascript_code"] = "window.MP3jPLAYERS[0]['list'][0].mp3" succeeds.
Right. I understand the problem
I don't :>
13:25
Ok, upon closer inspection of the working code, I'm no longer sure if I really understand the problem
You say the code fails when you build the string with "+". But the code you've shared so far doesn't have a "+" anywhere.
The only diferrence is: "list" --> 'list'
Look, just use exec(f'self.playlist["items"][0]["javascript_code"] = {text!r}')
where text variable will have those data: window.MP3jPLAYERS[0]["list"][0].mp3 ?
Do I have to answer that?
13:30
I will try and post the results.
Thanks anyway.
(What !r means?)
I expect the real problem has slightly more complex logic, because there's no reason to write exec(f"a = {b!r}") when you could simply write a = b
@ChrisP Essentially, "call repr on text before putting it in the string"
Pf, i am bit confused.
Me too :-)
I have to go in church.
But i will try when i arrive back to home.
Thanks anyway.
OK, I'll be happy to discuss this later, if you like
Even though I think it should never be used in serious code, I think exec is pretty fun to use in non-serious code
13:35
I think dataclasses actually use exec
I suppose that's OK, since it's not much of an attack vector for bad guys. They'd need to already have control over the code in order to gain control over the code.
It's a bit weird to have <string> show up in your tracebacks though
@dataclasses.dataclass(frozen=True)
class Foo:
    x: int

foo = Foo(3)
foo.x = 17

# Traceback (most recent call last):
#   File "untitled.py", line 9, in <module>
#     foo.x = 17
#   File "<string>", line 4, in __setattr__
# dataclasses.FrozenInstanceError: cannot assign to field 'x'
huh, thats odd
Gonna make you do a double-take the first time you see it for sure
The implication might be that resample is not the right tool for this job. I must ponder this.
--> yes thats why I chose groupby, but then again it doesn't work :/
13:42
Sensible.
Did you give the approach of creating a new column for grouping a try? Because that should work for sure, at least going by how you've described the issue.
perhaps if you can prepare an MCVE i'll take a look
 
1 hour later…
15:06
Math terminology Q. In a game I'm playing, it's displaying a formula, (3.02e161 / 1.00e21) dilate 0.5 = 7.12e11. What does "dilate" mean?
Google takes me to en.wikipedia.org/wiki/Dilation_(metric_space), which looks like a very fancy way of saying "multiplication"
@Kevin that's a map that expands space
It multiplies distances
@Kevin "game"
3.02e140 dilate 0.5" -> 7.12e11... hmm...
Seems arbitrary to me
15:09
what's throwing me off is what looks like an innocent division that would be in the scale of e140 or something, coming down to a value in e11
The specific game is alemaninc.github.io/Exotic-Matter-Dimensions. And you're correct to put "game" in quotes, since idle games are more like spreadsheets than games.
I opened the url and i realised that im too dumb to even comprehend this game...thanks Kevin
The leftmost number, 3.02e161, is a measure of the game's primary currency, "exotic matter". The rightmost number, 7.12e11, is a measurement of how much secondary currency, "stardust", I could receive if I cashed in all my exotic matter. 1e21 and 0.5 appear to be constants.
The function appears to be pretty slow-growing. If I multiply my exotic matter count by 10, I might expect my potential stardust to increase by 1%.
is it possible they essentially treat them like two different units then? ie. maybe theres a constant multiplier that explains the e140 difference, and then dilate is essentially like saying * 0.5 * constant c
Two different units is possible. The formulas on the statistics page don't show units, just numbers and operators
15:19
i guess if we leave speculation aside, dialation seems to be explained well here. we can only hope their dilate is related to this dialation.
https://stackoverflow.com/questions/72576539 is there a better/more general duplicate than "list of lists changes reflected across sublists unexpectedly?"

the broader problem is creating a nested structure that aliases contents. It happens with more than just lists, and in more ways than just list multiplication.
dupe with a comment sounds perfectly reasonable here. i think most of such cases just go to that list dupe
Normally my prime suspicion would be that the dev invented his own personal function and named it "dilate", and I won't find it defined anywhere but the source code. But other parts of the game have references to somewhat obscure math concepts, such as tetration and Knuth's Up Arrow notation. So it's possible that the dev is well-read enough to include super obscure math concepts that are hard to google, and maybe dilate() is one of them.
stackoverflow.com/questions/72576024 duplicate, messed up tag edits and couldn't hammer, haven't done that in a while
I don't see why this is a duplicate.
that is: the config options that OP describes, seem compatible with what is recommended in the answer on the candidate duplicate.
15:57
my understanding is that the OP is right, as well as davidism,because* this is how it "would have been" solved up until very recently, and i think google has just hammered away this approach into oblivion i believe? so...guess that makes them the hammerers now.
user19284560
16:55
Hey y'all
Howdy
 
1 hour later…
18:08
Yo yo yo
has anybody used VS Code's ""python.envFile", I have added a bunch of variables but I can't seem to come up with a way to test if that file is actually used
I'm guessing that the envFile loads environment variables into the project's virtual environment. You should be able to check if those environment variables are there by importing os and then doing os.environ["variable name goes here"]. If it crashes with a KeyError, your envFile probably isn't working.
Yeah it does, thats what I've been checking
but its kinda weird I mean I have a "python.envFile": "${workspaceFolder}/.env" but it still doesn't work
but also no obvious error message like "file not found"
https://code.visualstudio.com/docs/python/environments
"The PYTHONPATH environment variable specifies additional locations where the Python interpreter should look for modules. In VS Code, PYTHONPATH can be set through the terminal settings (terminal.integrated.env.*) and/or within an .env file.

"
Or maybe you're asking something like: "a friend of mine told me that if I set an environment variable GOFASTER to TRUE, then my Python programs will speed up. Is there any way to tell whether the GOFASTER environment variable is actually being checked somewhere?". I don't think it's possible to do that.
No I'm not asking that
Ok.
18:21
So terminal.integrated.env. works as expected but no dice on setting PYTHONPATH from the .env file. And without the PYTHONPATH set pylance's gui doesn't seem to detect external packages.
What's the filename of your envFile?
I used

"python.envFile": "${workspaceFolder}/.env",

So there is file called ".env"
Ok. I thought that might be the case, but I wanted to double check.
So code navigation works to the external directory but not pylance
Are you on Windows? Sometimes If you create a file in Windows and give it an unusual name, Windows will try to be helpful and "correct" it to something else. For example it might change .env to .env.txt
18:27
naw, I'm doing linux with remote ssh
Ok.
If you change your setting to an obviously wrong path, does it say the file isn't found? For example, "python.envFile": "${workspaceFolder}/kevin/coconut/foobar/.env". Assuming you do not have a "kevin" directory in your project.
If it does say the file isn't found, that means that it was able to locate the ${workspaceFolder}/.env file, and the problem lies elsewhere. If it doesn't say the file isn't found, then we don't know one way or the other whether it could find the ${workspaceFolder}/.env file.
yeah no error message with a wrong file, but maybe i'm looking for the error message in the right place?
Maybe you just have to restart pycharm / reopen the project / re-select the interpreter after changing the env file?
This might be relevant: "When PYTHONPATH is set using an .env file, it will affect anything the extension does on your behalf and actions performed by the debugger, but it will not affect tools run in the terminal". So perhaps the .env file is being read, but its PYTHONPATH value is being intentionally ignored for terminal-related actions.
I don't know if pylance counts as "terminal related"
19:19
what does the file currently contain? What observable effect are you expecting as a result of the file contents?
@Kevin So I actually have the opposite problem. The terminal related stuff works (which is controlled with terminal.integrated.env) but not the pylint stuff.
Its just a few things
`
TEST=BAZ
PYTHONPATH=/bunch/of/absolute/paths:`
So the same PYTHONPATH has the expected outcome when I set it as part of `terminal.integrated.env` (aka the python interpreter inside the mini-terminal can access the python paths)
@Kevin no, pylance isn't terminal related
 
2 hours later…
21:04
Considering the evening discussion about exec and eval i have created a question: stackoverflow.com/questions/72579813/…
21:16
@ChrisP in don't understand the question
So you have an external sub process call and you need to serialize the data into python?
21:51
> if i do something like [...] the self.playlist variable will not be updated.
How did you arrive at that conclusion? Did you even try it?
22:30
Hey, funny question: how can I make my script sleep indefinitely?
Basically something like while True: time.sleep(10)
Like that. Why are you looking for a different solution?
I don't know. Looks ugly. ;)
Is the question about how to make the script wake up?
@rattlesnake threading.Event().wait()
Wake up? No, quite the opposite
I want to simulate a "frozen" script
22:35
while True:
    time.sleep(1)
but the meaning of frozen can be different in different contexts
for example you can call a C-API module that goes into an infinite loop, which will prevent the python interpreter from processing messages, and breaking that kind of loop requires different signals
@Mikhail Yeah that's what I currently have (see above)
@vaultah Yeah that's not as ugly, I think. A little bit more explanatory to wait for an Event that never happens. Thanks :)
@Mikhail Ah. No, that's way too deep. I just want to simulate a frozen generator.
in the spirit of useless bike shedding
A C-API module that goes into a for loop wont' have the event queue serviced by windows and will show up as crashed (after some number of seconds) while the while True. time.sleep(1) case will never show up as crashed
Yeah. My generator doesn't crash. It just waits for input that never arrives.
23:01
On an unrelated note has anybody tried to generate stubs for cython modules?

« first day (4255 days earlier)      last day (915 days later) »