« first day (3561 days earlier)      last day (1603 days later) » 
00:00 - 19:0019:00 - 00:00

19:18
I swear I'm putting actual effort into this, but I've had to take breaks
Why isn't my regex working?
I figured I could use BeautifulSoup, but it seemed that re would be simpler for this:
The output is "12"
Go to this page: view-source:discounttire.com/tires/winter-catalog
If you ctrl-F "/buy-tires," you get 30 results
The 1st or 2nd element of this is clearly wrong; it's just a big glob
The rest are right
Not 1st or 2nd, 1st AND 2nd
my output is actually 0. Seems like the whole page is spawned by javascript
regex101.com
You should split the problem in half and determine whether it's the requests part that's broken, or the regex part
The ideal MCVE will use exactly one of the two of them
The first part is okay, I've already tested
FWIW if you have infinite character wildcards in your regex string, weird things can happen if they can be nested. for instance matching anything that starts with A and ends with Z, you will get more than three matches for AZAZAZ
19:27
Ok, so make us a dpaste that runs the regex on a static string. No point in making us consume bandwidth going to a tire website if you already have a test case
because there's also the matches for index pairs (0, 4), (0, 6), (2, 6) on top of the (0, 1), (2, 3), (4, 5)
Thats probably why you're getting big blobs.
Is there any way to intercept a future.cancel() call? I am writing a blocking function that will take awhile to complete, so the best bet is for a user to throw it into a ThreadPoolExecutor and just get a future and go on about their business. However, if this function is cancelled or hits an exception I have to do some cleanup action. I wish you could just cancel the future and that would raise an exception in the function or something that I could catch. What's the right way to design this?
Hmm, what's the "\" before the question mark for? What happens if you remove it?
>>> import re
>>> s = "foobar/buy-tiresbazqux"
>>> re.findall(r'\/buy-tires.*\?', s)
[]
>>> re.findall(r'\/buy-tires.*?', s)
['/buy-tires']
@Kevin the slash is an escape char, because ? has a particular meaning in regex, so presumably they want all strings that start with /buy-tires and ends with a ?
\/buy-tires\/[a-zA-Z0-9\-]*\? worked
I wanted the literal ?
Thanks for the sugg alkasm
I wonder what re.findall(r'\/buy-tires.*?\?', s) would do, then...
If you get unusually large matches, sometimes that can be fixed by marking your asterisks or plusses as nongreedy, using a question mark
>>> re.findall(r'\/buy-tires.*\?', "foobar/buy-tires?hello?bazqux")
['/buy-tires?hello?']
>>> re.findall(r'\/buy-tires.*?\?', "foobar/buy-tires?hello?bazqux")
['/buy-tires?']
19:35
Oh good call
@roganjosh :49940948 Really it's that I've a couple of db backends that I want to run the same tests for. Most of those tests require some state (data already in there, etc). I can rely on my functions (eg create_user_in_db, add_data_to_user_in_db *not actual names) to add that...but that relies on those functions being correct. The alternative is to do that via direct sql statements...but those then have to change in every test if the implementation alters slightly.
Sorry if I'm being cranky, I got off a conference call a while ago and I'm still shifting out of combat mode
@RE60K If there is no culture for code quality, expect things to get difficult. You'll want to get some backing by senior engineers that see value in keeping code health high. Your best bet is probably to try and collect cases where bad code quality hurt the company goal, i.e. delayed/prevented product delivery.
Re-asking this since I posted it in the middle of the regex discussion: chat.stackoverflow.com/transcript/message/49946637#49946637
Hmm, canceling futures seems like something that ought to be possible...
19:45
not quite sure which part is a question and which one is a statement. Note that you cannot cancel a thread in the general case.
You can use black vodoo magic (aka the CPythoh C-API) to raise an exception in an arbitrary thread.
Oops, I misread the question.
Predictable cancellations are the strong suit of the async world.
FWIW I can design it cooperatively, this is my code. The task I'm cancelling is a function I wrote---not some arbitrary thing. So it's easy to like, periodically check a variable like a threading event.
But I don't even know how to set the threading event, because I don't think you can detect if a future has been cancelled (inside the task that the future represents)
Futures are basically generators and generators have the throw method, so [insert handwaving here] therefore it is possible with futures too
You could set the future's add_done_callback to the Event.set method.
@Kevin Erm, I don't think that futures are generators.
19:51
I was just thinking that while typing above. If I cancel does the done callback get called?
...I'll go RTFD
According to the docs, yes.
Oh important note, I'm talking about concurrent.futures.Future, not asyncio.Future.
I think the done callback is all I need. It takes the future as an arg so I can check inside the done callback whether it was cancelled or if it's done. Thanks!
Hmm, my mental map must have a hole in it. Futures are awaitable, and awaitables are async defs, and async defs are syntactcal sugar for functions with yield in them... Or is this one of those "all cats are animals, therefore all animals are cats" situations
In this case, it's a dog, not a cat. Shares the same name, though.
asyncio has some historical cruft. It often uses classes where modern people would use a coroutine.
By "historical cruft" I of course mean "it comes from a more civilised time". Back when gentlemen would synchronously tip their hats.
It's all Greek to me I'm afraid
20:03
@Kevin in general, generators and awaitables are both coroutines (functions that suspend), but also I specified concurrent.futures.Future after you said that since that Future is like a...thread task promise rather than a coro.
@Kevin asyncio.Future is to coroutines what range is to generators.
Hmm interesting
Give or take some tender whacking with a hammer.
@MisterMiyagi somehow that makes it worse
Imagine four balls on the edge of a cliff...
20:11
I do not understand that reference
It is dialogue from the vaporware game Zybourne Clock where the protagonist attempts to explain time with the most confusing metaphor possible
> Imagine four balls on the edge of a cliff. Say a direct copy of the ball nearest the cliff is sent to the back of the line of balls and takes the place of the first ball. The formerly first ball becomes the second, the second becomes the third, and the fourth falls off the cliff. Time works the same way
Hmmm, unfortunately this doesn't work quite how I need it to. future.cancel() does not cancel the future because the task is currently running...so the done callback doesn't get hit.
A solution that I could do is return the threading.Event to the user and then they can cancel the job with .set() but this is a strange API. I feel like I'm missing some useful abstraction here or something.
20:33
@AndrasDeak If you say that because you think "range is nothing like a generator", then welcome to the path to enlightenment.
I also have this bridge to enlightenment to sell...
I am starting to get frustrated with this problem. Freaking spending all my time and it just will not work...gees...wish i could swear in here
20:57
@MisterMiyagi do you mind helping me out please
@Starter You could help us help you by answering the question I posed earlier.
I did answer your question
It's pretty unclear what you are trying to do, and what your problem is with doing it.
all I am doing is just compare multiple columns in 2 files. i.e file1 and file2, if there is a match take a field from another column of the matching line in file1 and append it to the matching line in file2 if not append NA. I am not sure why my tiny brain can't get a handle on this. I feel dumb sometimes
the problem is that at the end, the fields get rewritten with NA's which, I think, suggest that the lines are getting reread and compared again. doing so will lead to no match hence NA
Neither of this nor your earlier reply to me adds any detail to your initial question. What is your criteria for skipping lines? What problem do you have skipping lines?
21:07
@Starter please don't ping people with your problems
@AndrasDeak sorry about that
I only pinged @MisterMiyagi cos we were discussing earlier
Yes, but you didn't really respond to his queries so bugging him without using his earlier help is not nice
@Andreas I am not sure but I said sorry. Also he did not complain
The regulars who are bombarded here with various questions are way past "complain". If we were to complain about askers not responding when they are asking for help we'd be a lot more tired and frustrated all the time. If you ask for help and ignore what others tell you you're only causing a problem to yourself. It only becomes our problem when you later ask the same thing again.
But let's set this aside and start over.
@Starter reread the chat transcript and see what questions you were asked, and try to see if you carefully answered them. If you think you did, let's discuss why you think that so we can figure out what you're asking for help with.
And by the way my name is not Andreas
@AndrasDeak noted. thanks
@MisterMiyagi The reason for me wanting to skip a line after it has been processed is so that the values in the column doesn't get overridden with NA. So its going to be one line at a time. I did not know i could click reply on the message earlier
21:24
@Sam finally collected my thoughts here: stackoverflow.com/questions/62943622/…, if interesed :)
that's quite a wall of a question
Yeah
I was hesitant to post it as well like that
@Starter I still don't really understand your issue. Since you have working code that produces unwanted results ("NA"s, whatever that really is), could you put together an MCVE? It could make it clear what you're trying to do, or at least what you are doing.
couldn't figure out a shorter way while keeping the question reproducible & not just posting a bunch of code without explanation (both of them seems annoying - I thought maybe more annoying than a long q)
@zabop yeah, out of these three options the long question is the smallest evil.
I won't even try to decide if it's answerable/on-topic though, because I'm not familiar with the domain
21:26
@AndrasDeak thanks, thats reassuring
@AndrasDeak I can do that
@AndrasDeak alright, no worries. Hopefully there will be some reaction to it by the morning
@Starter OK. I suspect it will be more than around a dozen lines of code, so please put it into pastebin or something similar and link that here
@AndrasDeak exactly what I am about to do
I thought so (since you've clearly read the rules), but I figured it can't hurt to be explicit
21:36
@toonarmycaptain test tests of tests... of tests? :P I think you really only need to come up with a standard sqlite db to play around with. At some point, I think you have to call it day
@AndrasDeak here is the link pastebin.com/JMiT7C3X
thanks
I'm not a big tester, but you knew that already. And you also know that Sqlite doesn't even enforce types now, so you'd also get a hole in your coverage there but it's super lightweight for shoving examples together
@Starter What kind of files are your inputs?
And what should the code do?
@AndrasDeak They are tab delimited text files. The code should compare 3 columns in file2 with 3 columns in file1, if a match is found in the lines, the matched line infile1 should be updated with 2 other columns from file2 else it should remain as 'NA'
21:44
Can you put together two small example files to show what should happen? This is needed to make it an MCVE. You can also use io.StringIO to create a fake file in memory from a string literal.
@AndrasDeak Can I add files here?
No, I suggest either adding them as comments in the same pastebin, or as I said using a StringIO. We probably don't need a lot more than, say, 5 lines per file to demonstrate what you're looking for
can some help me with the bc function from the docs.scipy.org/doc/scipy/reference/generated/…? I can't understand what is ya, yb.
@VisheshMangla since those go into the boundary conditions, they are probably the value of the solution y(x) at the boundaries: y(a) and y(b).
@AndrasDeak okay let me try that.
21:50
@zabop Unfortunately this might get closed since it has many questions instead of a single, focused question. My experience has been that SO is a poor format for these types of questions, where something like Reddit or another discussion-based forum is often better. Though some times you can luck out.
Oh yes, thanks for the clarification @AndrasDeak
@LinkBerest I've given you edit access so you can do so at your leisure
@zabop I cast the third (final) close vote on the question. It's one of the most in-depth and prepared questions I've seen in a long time, and (for someone with domain knowledge) pretty easy to follow along, so congrats on that. However, I agree with @alkasm - it's just too broad and opinion-based for this particular site, as it's inviting a dialogue, not a single discrete "Do this and your code will work" answer. Good luck finding a solution, though!
Also the docs say that the output of the bc function must be of dim n or n+k. I can't understand this too. If you see the first example given there consider if a condition like y'(0) = 1 is also given, then instead of 2 there w'd be three boundary conditions.
Where did you read that "n or n+k"? I could only find "For the problem to be determined, there must be n + k boundary conditions, i.e., bc must be an (n + k)-D function." but admittedly I didn't peruse the page
22:04
bc: callable

Function evaluating residuals of the boundary conditions. The calling signature is bc(ya, yb), or bc(ya, yb, p) if parameters are present. All arguments are ndarray: ya and yb with shape (n,), and p with shape (k,). The return value must be an array with shape (n + k,).
I don't see any choices there
My p is empty therefore k = 0
So in your case n + k == n, but that's not "n or n+k"
yep
my apologies
@alkasm Yes, you are right, it get closed!
@MattDMo Alright, thanks for the compliments! :) Will try on other platforms, maybe I'll find somewhere it is suited
22:08
can I get answer to the question then , why this constraint on output of bc to be "n+k"?
I got more boundary cnds in the problem that I m solving.
@AndrasDeak I created a new paste with sample data here pastebin.com/3rUvdxV3
again its just sample as close as possible to my files. definitely not enough columns
@VisheshMangla sounds like the reason is mathematics. If you have fewer boundary conditions then the problem is underspecified. If you have more the problem is overspecified.
For instance a given ODE might have a unique solution when you set y(0) and y(2) to 0, but if you also try to set y(1) to 0 you don't get any solutions. So the overnumerous conditions are either contradictory with the first n+k ones (in which case there's a problem) or they are redundant (in which case you don't need more than n+k)
@Starter that's what an MCVE should be like, so no problem there. I'll try to take a look soon
@AndrasDeak Now I know better. Thanks
I think that'll work as an example, thanks
thanks for the explanation @AndrasDeak .
22:23
@Starter stuff like data2[26] = data1[12] makes me a bit anxious. That's awfully specific and pretty fragile once you change things. Is pandas a possibility?
I know you said you're avoiding it, but I'm not sure exactly on what grounds
@roganjosh The size of my file makes it difficult for pandas. Runs into memory issues but maybe you have a way
400K lines gives you memory issues?
@roganjosh thats the smallest of my files
you can iterate over dataframe in chunks if you have memory issues
@Starter I haven't had a chance to run it yet but if tuple(data1_row) in [tuple(vep_data1_row)] also looks like something unintended. As it currently is it should probably be wquivalent to if tuple(data1_row) == tuple(vep_data1_row) which in turn is the same as if data1_row == vep_data1_row assuming they are both lists.
22:27
@roganjosh K, thanks. I'm finishing up that list for my classes now (two are Java but there's some crossover with CS general stuff). I'll see if I can condense it to just the Python stuff next month and add it
@AndrasDeak yes I think thats true too
@Starter and the largest?
@VisheshMangla True, how does that work for millions of lines
It doesn't
say about 9 gigs will be my estimate
22:31
I did a machine learning and data science course last year and I used iter_in_chunks to prevent my pc from freezing. Datasets are huge 10 20 gbs. If time is an issue you need to use dask and other better frameworks.
I regularly used pandas with 10+ Gb files - you need to either: read in columns as categories & use use_cols when reading the csv or use chunck_size and process it in chunks. (you could also use dask but I've not needed to to that....or munge the data better....or a bunch of other possibilities)
@VisheshMangla I just need to get it working for medium size data first and then think of how to handle the largest data
I mean now I clean everything by making chunks into SQLite Databases and then processing all those together (into Spark Dataframes) but my file size is way above 10 now & not all text :)
I'm kinda leaning towards SQLite here tbh
These are pretty big files to be doing everything in memory
I could just use Pandas DFs for this with similar results (I only use Spark cause there is a lot of Java & Scala that has to use it too)
22:36
@LinkBerest how will you use pandas for this problem? say given the 400k lines?
......in exactly the way I just said:
4 mins ago, by LinkBerest
I regularly used pandas with 10+ Gb files - you need to either: read in columns as categories & use use_cols when reading the csv or use chunck_size and process it in chunks. (you could also use dask but I've not needed to to that....or munge the data better....or a bunch of other possibilities)
I remember jpp wrote a good post about it somewhere...... yep, good ol'jpp
There's more than just reading the files. They want to look up rows in one file in the other.
but that's just to the general problem - you wrote a lot and I'm still reading the transcripts on the specific problem (so far I have the same questions about it everyone else has already asked)
30 mins ago, by Starter
@AndrasDeak I created a new paste with sample data here https://pastebin.com/3rUvdxV3
@Starter Okay - caught up: I would clean out the unwanted data using SQLite (or any database - I just like this one for that - or even general text parsing) then use pandas
there are a lot of ifs in their for skips - lot of skips = not clean from my experience
22:44
@LinkBerest yeah ikr. some of those were borne out of frustration but I had to try to get it going somehow
If I have to use pandas I would use the method I said first - load it as columns and then run compares and write results back to disk (avoids loading everything into memory all at once) ... I also could be misreading your use-case though
^ the first link in jpp's answer shows how to do that btw
@AndrasDeak you are absolutely right
@LinkBerest taking a look at that too
he recommends dask though rather than pandas especially for the larger files
he makes 3 recommendations there (only one of which is dask)
and considering your question seems to boil down to: "how to I match an item for category 1 with an item in category 2 then add these matches to category 3" - his first suggestion would make the most sense. If your data is such that you cannot even load it into a good data structure it probably needs more pre-processing before being analyzed (your code is trying to do both)
also cbg all - and condolences to anyone who had to sit on conference calls all day like myself (and Kevin it seems)
00:00 - 19:0019:00 - 00:00

« first day (3561 days earlier)      last day (1603 days later) »