« first day (4192 days earlier)      last day (760 days later) » 

4:06 AM
A small question about pandas, can u take a look?
 
@XJTLUmedia please don't ask for help with fresh questions on the main site as per our rules
 
 
2 hours later…
5:54 AM
@roganjosh For me it's useful to see that two objects are (not) the same, even if they don't have a nice repr.
Oh my, seven-hour-Kevin'd
 
 
2 hours later…
8:04 AM
I have a question. Some of my colleagues are obsessed about typing and started to add annotation to old code. The question is could adding type annotations theoretically slow down execution? (yes, it's already slow cause it's python, but)
 
theoretically: yes. practically: no.
 
8:59 AM
typing will almost (teehee, gotta save yourself) never be the bottleneck causing the slowdown, the actual program will.
 
9:13 AM
FWIW, if you are worried about performance then use from __future__ import annotations.
With it annotations are only parsed, not evaluated. Since Python caches that result across runs, it means even the tiny cost is only paid the first time a file is imported.
 
9:56 AM
Being worried about performance and using python is a weird combination :D I was just seeking arguments to stop them (colleagues) annotate whole project and do something useful instead.
 
That's how you get type check pass requirements in CI, after which you'll also be forced to annotate your code :P
 
Actually, they're currently annotating code I've written long time ago, so no action required from my side
 
that's how they get you
 
If you need to dig for an excuse to stop them, I recommend not to stop them.
 
will no one think of the pitchtorches?
 
10:11 AM
I've got a side business in pitchtyping these days. :P
Righteous smiting of async is 50% off today, though.
 
10:39 AM
Hi all, what's considered the standard design pattern for storing instance state, undoing? (if this is pickleable, so much the better). I have a (Wordle solver) class with 5 state variables (1 string, 1 list, 1 dict, 2 pandas dataframes). It has 3 methods that modify state (filter pandas dataframe of candidate words). I searched but didn't find a decent solution.
 
10:50 AM
@Kevin @rb3652 Was that a programming problem, or a math problem? (Assuming the former, cos if the latter, I doubt they let you use linalg solver). What were the expressions you used as an upper-bound on the terms y_1, ..., y_4?
...like do you consider the square of each of those distances y_i? but how to make sure they're underestimates
 
Hi All,
Is there a better way to write the below code
s = pd.Series([1,1,2,3,3])
for i ,j in s.iteritems():
try:
if s[i+1] == j:
s[i] = 'X'
else:
s[i] = 'O'
except:
s[i] = 'O'
basically I am trying to assign for the last element of the group. Meaning in the series mentioned the expected o/p is : [X,O,O,X,O]
 
s.diff() == 0 gets you close but you would have to move the first False to the last position. What numpy.roll would do, except for Series, and I can't seem to find a pandas implementation for that. So perhaps a different approach is better.
 
1:05 PM
"yield from" should be used as context manager expression is a weird way of saying "you need async with instead of with"
 
 
1 hour later…
2:30 PM
you could use shift
import pandas as pd
import numpy as np
s = pd.Series([1,1,2,3,3])

comparison = s.shift(-1) == s # is next item equal to current item
result = np.where(comparison, "X", "O") #apologies to those who hate np.where, feel free to use something else :P
 
2:56 PM
It's all good Paritosh, we did some timings a while back and np.where() outperformed the other options :)
 
3:10 PM
hi all, I have been stuck in an issue for a week. I am new to python and webscraping.
It's basically on ways to rotate the user-agent.
-1
Q: How to rotate User-agents as to not get blocked?

Shruthi RavishankarWhen I am trying to run my code, I am getting the error saying connection refused. So, I tried rotating the user agent and the method has some errors. I have tried various methods to get requests by rotating user-agents. This is the following program. def parse(self, response): l...

 
3:36 PM
@ShruthiRavishankar If they are blocking you, that's a good sign that you aren't meant to roast their servers by scraping in such a volume. Apparently, they have an API as well.
 
Can you really avoid getting blocked just by changing your user-agent?
 
Not if they're doing IP blocking...
 
@Aran-Fey yes, it's working. You will be blocked anyway if you're planning to spam website with mad amount of requests, but if your task is to scrape some data switching user-agent (and some other HTTP headers) is a working method.
 
* website statistics show an sudden increase in people using Internet Explorer :p
 
*taking notes* What other HTTP headers are we talking about? I usually don't even send any when I'm scraping
 
4:01 PM
Accept, Accept-Language, User-Agent
Just sniff 3 different devices opening same website and check differences in request headers.
 
@ParitoshSingh ah, shift before comparing, good idea
 
yep yep, that little trick has been useful for me a couple times now.
 
 
3 hours later…
6:52 PM
@Stramzik Not that this is magically better, but ...
pd.Series('O', s.index).mask(s.eq(s.shift(-1)), 'X')
 
Avv
Hello Guys,

I am trying to use Python RegExp to parse the following data to CSV:

2021-02-09 13:11:16.750 OO-OO 6 Tx 28C 1 10 xx | yy | zz
Each line has varying spaces, so not sure how to capture that in Python. So there is no predefined structure for the file. Any hint what to do in this scenario please? Can Python work this out?
 
You have to also know where the commas are supposed to go
 
Avv
Pardon me?
 
@Avv Also, if there is no predefined structure then all is lost. The trick is finding the structure.
 
Yeah, you can't just dump a line of text here and expect us to know how that's supposed to be converted to CSV
 
Avv
6:58 PM
I know :(
Some lines has more spaces between columns than others. Not sure what to do in this case.
there is a semi structure
I can not post images here?
 
@Avv What's the expected result of parsing this?
 
Avv
I just want to be able to find time differences between request made via and response time
 
What you should do is put more lines of this into a pastebin or something. Then when you give us a link, also include how it should end up. That way, we aren't guessing what you want.
 
And why call it a csv when there isn't a clearly defined separator?
 
Avv
It's a log txt file that I want to convert to csv as it's easy to process I assume?
 
7:03 PM
I'm sure it is. but this is more an exercise in explaining how you should ask a question more than explaining how to parse text.
 
Avv
The above is a sample of the data
As you see the space varies between columns
 
Looks like every column has a well-defined fixed width?
 
Avv
@Aran-Fey. Not all. For example, look at 00 FF 0F 10 FF 7F 7F FE FE 01 FF 0E 01 0E 7F FF
 
@Aran-Fey first data row is offset with respect to the other rows
 
7:05 PM
Is this tab separated by any chance?
 
quite possible ^
the best strategy is to go back to the software that printed this and change it
 
If it's not tab separated, just regex split on "2 or more spaces"
 
@AndrasDeak--СлаваУкраїні "Test case verdict passed" tells me that's easier said than class Solution:
 
the tabstops don't seem equal though, so maybe not tabs
 
Avv
@Aran-Fey. Thanks for the hint!
Probably that will solve the issue to regexp on 2 and more spaces
 
7:07 PM
We stlll don't know what you want to end up with, so we can assume it will definitely solve something.
 
Avv
I just want to find difference between timestamps
 
Why parse anything else then?
 
Avv
2021-02-09 13:11:20.981
2021-02-09 13:11:20.982
2021-02-09 13:11:20.989
.
.
.
 
The timestamps seem pretty regular.
 
Avv
This will help me find if there is DoS attack
any DoS time that is within 100ms of the verified time an accurate result
 
7:09 PM
@Avv why would you not start with that? Even when others have explicitly asked you what you're doing?
oh well
 
Avv
@AndrasDeak--СлаваУкраїні. I added above chat.stackoverflow.com/transcript/message/54328164#54328164
 
@Avv thanks, my bad then, sorry. I definitely glossed over that message of yours.
 
so you just want the time part?
 
Although "request made via and response time" sounds like you need more than the timestamps.
So I'm back to "oh well".
 
Avv
@AndrasDeak--СлаваУкраїні. I should have elaborated more. Sorry for that.
@MisterMiyagi. I don't see why the following test failed requirement that, "any DoS time that is within 100ms of the verified time an accurate result"
2021-02-09 13:11:20.981
2021-02-09 13:11:20.982
2021-02-09 13:11:20.989

But the following passes:
2021-02-09 13:11:21.082
2021-02-09 13:11:21.082
2021-02-09 13:11:21.090
So I am not sure if this will help me to solve the issue or not but I will try.
 
7:14 PM
As @MisterMiyagi also said, time is usually or infact mostly a fixed length string which I guess you already know so just open the log.txt file using with open and go by each line and slice every line for upto that number which holds the timestamp value in every line and use that at your discretion. datetime.datetime will also let you convert timestamps to whatever value you want. But something tells me you might have already tried this full approach.
 
Avv
I would assume that if request time - response time > 100ms (fail DoS test). For example, 13:11:16.750 corresponds to HOUR:MINUTE:SECONDS as I see.
So I convert seconds to mseconds and then check > 100ms or not.
This is my approach to solve it.
 
@Avv Which is kind of correct, however I was talking about parsing the timestamp from your text file rather than what you do with time which is what you are asking I guess?
 
Rule #-∞ of asking for help: Don't describe your code, show your code.
 
Avv
Sure. Thanks.
 
The way you've described it, there's at least one bug. Maybe 2.
 
7:22 PM
@Avv This sounds like your criteria is not correct. Reread the problem statement if there are any additional rules.
 
 
2 hours later…
9:16 PM
Cabbages
Is there any way we can "Defend" the opening of a question? In particular this one:
https://stackoverflow.com/questions/62429677/how-to-use-str-replace-to-replace-multiple-pairs-at-once/62429824#62429824
Because both answers flagged as duplicates do not suffice, since the first one is for python, and OP asks about pandas (where the function replace does work for multiple pairs) and the second answer that could answer it, also doesn't fully address OP's issue
 
9:50 PM
@CeliusStingher there's no "answers flagged as duplicates". The dupe targets are questions. The pandas dupe target has 6 answers. Is the second top voted one not exactly what your answer does?
 

« first day (4192 days earlier)      last day (760 days later) »