« first day (3011 days earlier)      last day (2162 days later) » 
01:00 - 21:0021:00 - 23:00

01:21
@jpp why ^ ?
hello
Is there a way in regex I can ignore the first for chars?
like doing this in python some_string[4:]
jpp
jpp
01:37
@AndrasDeak, Why not? The error's pretty clear. I mean, if you like you can edit the question with an arbitrary version number.
Or just downvote it all :)
@Vader why not do the python version?
Something like '^.{4}' at the start might work...
Because that would be too easy
I feel like I should know hot to do it in regex
Shouldn't be too hard to find out
Then again what I wrote only works for patterns matched at the start
You might do something with lookbehinds... 4 character positive lookbehind
Or just do it with python...
01:51
yeah the positive/negative lookahead/behind is confusing
Python is probably both faster and clearer
@Vader now you can choose between too easy and confusing
Rhubarb
02:29
I am going through some older posts of mine and editing them to be more useful. Absolutely facepalming at how badly some of the question titles are.
"Apply function to dataframe" (actually a groupby problem)
"pandas transpose sample" (nothing to do with transposition)
"pandas series key filter" (looks like someone threw the kitchen sink at the keyboard)
03:03
My regex is finding matches in an online debugger but not in python
import re

regex =  r'(?P<wx_intensity>(-)|(\+)|())((?P<vicinity_or_not>(?<!O)VC)|())(?P<description>(BC|BL|DR|FZ|MI|PR|SH|TS)|())((?P<precipitation>DZ|GR|GS|IC|PL|RA|SG|SN|UP)|())(?P<obscuration>BR|DU|FG|FU|HZ|PY|SA|VA|())(?P<other>(DS|FC|PO|SQ|SS)|())'

ob_str = 'KRAP 130152Z 18006KT 1/2SM FZFG VV002 M03/M04 A3023 RMK AO2 SLP277 I1001 T10281039 RVRNO'
weather = re.match(regex, ob_str[4:])  # [4:] is to exclude the first 4 chars because the might accidently be matched as weather
print(weather)
the match is ''
it should be 'FZFG'
should match from group description and precipitation
 
8 hours later…
10:39
@Vader make sure your online debugger uses python flavour
Is that METAR?
You might want to drop the first word rather than the first four chars: ob_str.partition(' ')[-1]. Might be able to drop the space as well
What are all those empty groups?
Are you aware of the ? quantifier?
Hi, I have a code (paste.ofcode.org/cyCQiuYM7f7Yvn8S4MPm5H) where it reads a group of files, strip the last line (1RowxnColumns), delete a word "Data" at the beginning of that line, convert the line to (nRowsx3Columns) coma seperated, and saves each file as CSV with the original file name. The code works excpet for the last 2 lines. The resulted CSVs are (nRowsx1Column) means each charecter on a line, seperated by ",".
@AndrasDeak any insight?
the output should be
43.6432,-81.6804,8.5000
but i am getting
4
3
.
6
4
3
2
","
-
8
1
.
6
8
0
4
","
8
.
5
0
0
0
"
"
@Arne, any insight?
11:11
@user2153702 don't ping users without relevant previous discussion.
sure, new here
11:30
Cbg
@user2153702 that's the base code I gave you yesterday. How did you get around the row size limit for CSV?
@roganjosh, morning
yes, it is
I added the limit (first line) and also changed the 2nd WITH OPEN to wb
as you can see, it writes every charecter on a row and puts the comma as ","
Hey if I want to overfit my data to achieve zero loss, I can't have any random operations in the graph, like dropout right?
hope somebody does ml here :P
Bizarre that it's as simple as altering the module row size limit, I'm not sure what the initial limit is useful for
I'll fire my laptop up in a couple of mins and finish what I started :)
@roganjosh, thanks! looking forward to it :)
I found out yesterday that I'm getting a new half sister so that kinda took priority over finishing the code sorry :)
11:37
Congrats
Thanks :) Right, laptop up. for n in range(len(last_line)//3):You're trying to convert the last line into individual rows?
nRows x 3 Columns
43.6432,-81.6804,8.5000
43.6432,-81.6675,8.5000
lat, long, elevation?
yes
elev=water
I'm tempted to use a numpy reshape but it can be avoided. Do you have issues with using numpy on this?
I don't know whether you have a restriction on dependencies
11:43
no, unless it complicates things down the road for me. I have few more calculations to perform on each csv
let's try it
@roganjosh half or step?
@Hakaishin half
My dad is remarried and appears not to have given up just yet. I already have one that just turned 1 :) By the time I can take them for a pint I'll be nearly 50 :O
@roganjosh wow, congrats
@AndrasDeak my dad's wife is Polish and you wouldn't believe how hard I'm pushing for them to register dual nationality :)
We might just manage to keep our foot in the door yet :P
11:48
Hehe
FYI "Polak, Węgier, dwa bratanki"
Huh, I never knew that
I'd guess it's an inside thing :D
The only thing you really need from that is the reshape operation.
@AndrasDeak where abouts are you based? At some point I will visit Poland, possibly this year, so may hop over the border to harass you since I also haven't been to Hungary :)
Heh. Budapest.
You'll have to hop more than one border ;)
Sounding super-enthusiastic there :P
Fewer from Poland than from the UK
There's a massive block of Europe that I just haven't seen. At some point I really do need to fix that.
@JasminParent It looks like it's a good fit for your requirements then?
12:05
@roganjosh, it worked but adds the word outputs at the beginning of the file name
I tried to remove the '\outputs' from the 2nd WITH OPNE but it didn't work
@roganjosh, when removing the '\outputs
sorry. When removing the entire '\outputs' the code excute with no errors but no csvs appear in the directory!
That's the directory path, and it's /outputs/ so it's not possible for the code I sent you to add it to the file name. It would throw an error if the directory didn't exist but, unedited from what I sent, it wouldn't change the file name
for me to make the code run yesterday, I had to change the following: '/outputs/' to '\outputs'
but yesterday, it didn't write the word outputs at the beginning of the filename!!!
Also as I said, the only thing you needed to take from it was the reshape operation. Keep in mind that the code is for my local setup to test, you'd already made the modifications to make the last code work locally for you, so there's no reason to trash your own setup by copy/pasting the entire code I sent you
@user2153702 and that's exactly the change that's causing the problem. That's not code I wrote
You need an extra / at the end of the the string to make it a directory and not part of the file name
changed '/outputs/' to '//' and it worked. thanks!
<wipes brow> Have we finally killed this one off? :P
12:15
YESSSSSS, and thanks to you
At least I learned about the CSV row limit, still not sure what the purpose of it is though
Haha, no worries :)
what is weird, in the new code, I didn't add the file size limits and it doesn't give me errors!!!!!
Well it won't, because we reshaped the data
Now there are only 3 items per row, so you don't exceed the row limit. Previously we wrote the whole thing to a single row
now, i want to read the csvs (batch) as pandas to add the header and perform calculations and then change switch the places of (lat, lon) to (lon, lat). Can i do that within the last WITH OPEN?
lol
"[following a 36-hour open heart surgery] OK, now that we're done we can finally do that heart transplant"
12:23
^
@user2153702 you wouldn't have shortened the life span of roganjosh by at least two months if you had started with wanting to end up with pandas
also that
What is the point of the interim CSVs at this stage?
which is, you know, the tool for these kind of jobs assuming "the job" is clear ;) We don't usually recommend pandas for handling csvs because that's overkill
but if you want to manipulate the data in nontrivial (and collective) ways, and you know you want to end up with pandas...you should mention that
1)convert and manupliate files to csv, 2) add a new column to each csv (calculation from 3rd column), 4) re-arrange the columns from (lat, lon, water) to (lon, lat, water, calc)
part 1 is done
Well part 1 is not in isolation. What's the point in generating the CSVs to then change them?
12:28
i will create a new loop to read each csv and perform rest of the tasks.
In all seriousness I was debating making the extra changes but I think you'll need to make some progress on your own here because working like that is not productive and I think it's stretching beyond what you can reasonably expect one person to help with
I am working on it, @rog
pro-tip: using pandas from step 1 and only using csv for IO will probably save you a lot of trouble
I wasn't asking you to do it, rather asking for advice if i can make a new for loop or add to the existing one
you also won't do calculations using the math module
@user2153702 you said you wanted pandas. Well this is a great time to get familiar with it :)
I didn't read what you want to do with your data but it's almost certain you can get rid of all the loops
12:32
you can do that, I have to use the loops
Why do you have to use loops?
don't know otherwise
So we go back to Andras' point; now is a great time to learn
I already have the work done for each file alone. I just need to batch it
and i am working on it. Thanks for your help
you have to loop over files, I only meant not looping within each file
12:40
@AndrasDeak In fairness, I don't think I could have started the task completely in pandas. The CSV was a convenience to me to drop the first column of what looks like some kind of specification sheet
First column as in first csv column?
It's just a text file, but I treated it as a whitespace delimited CSV
if it can be handled as a csv it's a csv
oh you mean it's proper csv plus a whitespace delimited row header
But it would have been messy in pandas because the second column was then comma delimited. I'd rather no faff trying to get the first column of the last row as a split list using Pandas
setting both comma and space as regex delimiters would've probably worked
12:44
True, I didn't think of that. Simple enough to split by index. It was only 2 lines of regular python code
for just csv work I wouldn't think to use pandas for this. But with the rest of the reordering and dicing business...
Oh for sure, they will need a DF now or have a somewhat hellish task. I think I still would do the pre-processing in the csv module though tbh
But then you loop over a potentially large file twice, creating a duplicate potentially large file with almost the same size. Right?
that is true
Sorry, I'm not suggesting the interim file that I wrote (without knowing the following task) is in any way necessary
I meant that I would still read the base file in the same way. Rather than write to an interim CSV, I would take the data after the numpy reshape and convert to a DF there
12:48
ah, OK
yeah, that would probably be more or less equivalent to pandas, with a bit more elbow grease
13:04
A bit of a general question; is anyone aware of any soft limits to a Flask project size? From what I can tell, it can grow without bound and only limited to memory, but I'm wondering if adding more blueprints and url endpoints eventually starts to degrade performance?
I know the development server is limited to how many HTML templates it can cache but that job will be taken over by the production server
13:27
@roganjosh define "eventually"
From what I've seen of its code, which is limited, there don't seem to be any artificial limits to that sort of thing
The HTML template caching of the development server is an "eventually", for example, because there is a definite limit on size. The rest of the code seems to scale in O(1), since url endpoints etc can just be stored in a dict. I was wondering if there was something I was overlooking
14:10
"grope elements of a long list in python based on the element name" best title misspelling this year :)
14:29
Am I right in thinking that question names things in Latin?
Yup, sounds like binominal nomenclature
Just conjured up memories of the Cambridge Latin Course. Those stories were epic. "multos sanguis fluebat". People being speared and eaten by crocodiles everywhere then patched up again with spider web and mouse guts
15:26
The Catholic Church is finally catching up with the times and learning to code
@ChuckIvan not gonna lie, I also immediately thought it was Vatican coding :P
I know right
Jesus gets re-instantiated after a bit of time.sleep()
"So I have this manuscripts...er, 'source code'"
lol
sleep(sys.maxsize)
I wonder what an IT shop at the vatican would be like
probably the same as an IT shop anywhere else
15:35
Right. Until you pull that certain bookshelf from the wall...
book from the bookshelf* Please don't pull entire bookshelves from walls it's dangerous
There's a certain kind of dignity in someone registering their revenge downvote on my highest upvoted answer. Well played to that person, I appreciate the consideration.
I probably hate the answer more than them anyway
15:58
Maybe it's the Vatican punishing our heretical comments
I'm done with these jokes I don't mean to offend anyone in this chat
Nah, I was engaged in a debate with someone who posted an answer and then defended their answer by saying, in their own words, that the question was unclear.
Foot meet bullet, but the answer went and the downvote came within the same minute, so I'm pretty convinced what happened but meh
I haven't answered any question important enough for that to happen yet. Sounds exciting
It's never on important questions
When comments completely swamp the question

https://stackoverflow.com/questions/54170501/access-first-key-value-of-each-dictionary-within-list-in-python
It's not important questions. It's on the rubbish questions that need to be closed as dupes or unclear, but people flock to answer them anyway
At least it sparked a good comment debate
Also unclear
@ChuckIvan I got used to seeing this very debate whenever the order of dictionaries is mentioned
There also rarely is a good comment debate. The whole notion of "there's no such thing as a bad question" goes out of the window these days with what is being dumped on SO
"In Python 3.6 and newer dictionaries are ordered"
"Uh-uh. Actually..."
I fear Jörg W Mittag is too nice and patient not to respond to that too
Surprise 300k user with not much python in their profile, by the way
Please don't pile on the comments and the answer
Using dicts without knowing what a hash function is
@coldspeed this means you as well ^
16:21
@AndrasDeak with the greatest respect, that's not your purview? It's outside of chat
@coldspeed Agreed
@roganjosh it is if users in chat start piling on users like a mob following a link here
Fair enough
Sorry for the pings
Phone is on silent, "ping" is meaningless :P
 
1 hour later…
17:35
cbg!
cbg
@amanb please don't do that
17:48
You're correct, that's cheap
thanks for letting me know, I won't do that!
Hey, I didn't deserve a downvote on that answer atleast
I didn't downvote but you brought it to everyone's attention. Maybe they will explain but there are consequences to what you did
That chat room might be silent for long periods but plenty of people are watching, I've come to learn that :)
insert ominous music here
I never thought it through..maybe its a lesson..I'll be a sport and keep working on good answers and wait till I deserve rep..you earn what you deserve
It'll be fine :) Just focus on producing good content. The rest will follow.
17:59
yep, dont worry about it. Heck, you'll eventually realise that honestly rep is just a meaningless number after a point. But its just nice to learn and help others at the same time.
..and it makes me appreciate SO even more.
protip: If an OP does not have enough rep, look through their past QnAs and see if there's anything worth upvoting
I'll be cheap and just say "all of the above" :P
@ParitoshSingh, I completely agree..I love working with good questions(there have been many bad ones lately) and helping ppl as much as I can..I realize how daunting it can be when a dev is stuck with a problem that is part of a time-bound project.
Folks, what is the more idiomatic way of checking whether an object is a function? I know of two: inspect.isfunction, and callable.
Gotchas wise, I know callable returns true for classes, and objects defining __call__. Wondering if there is anything else?
18:04
theres also the types library. i actually remember answering a question on it recently.
@coldspeed it probably also depends on what you want to do with the information
I would expect most applications to depend on callability
Didn't expect a banner. that took away half the screen haha
hmm, fair point—guessing the right answer always is "it depends"
@ParitoshSingh didn't know about types.FunctionType, thanks!
anytime! (and until recently, i didn't either haha)
@AndrasDeak
Yes it is a metar
I will drop the 1st word instead
The ? quantifier is only a label for the groups
because I dont need any of the groups they all have empty alternatives, I guess i could also do something like this
(watherever_regex)*
The star means zero or more times
18:17
I know, now check what ? means
I had written a script that queries the Github API and extracts issues, milestones and lot of other data and writes to a csv file. Later I also added the Zenhub API queries to do the same thing. The average time to generate a full csv report is about 15 minutes. I am trying to find a way to improve the performance.
Can anyone here help me? How can I improve my questions to lift the question ban on my account? I can't delete them as the downvotes are still counted.
The API calls are too many and I am trying to achieve concurrency in API calls & R/W to csv. I'd like to get this report in say 5 mins. I explored asyncio and now into Cython.Any good suggestions how I can achieve this?
Hey guys, how do I detect a key press (arrow keys) within the OpenAiGym environment?
@bhatnaushad the general suggestions are here. Bottom line is usually "try to make your posts better, make good use of the free question opportunity you get every 6 months" or something
18:20
looks like ? is also a quantifier
(?<name>...) This capturing group can be referred to using the given name instead of a number.
no way :P
ugh, that's not it
..I'm still absorbing asyncio, and its taking a while to see any considerable difference. someone suggested I should look into Cython to improve the performance.
@Vader I wouldn't call that a quantifier
try to find the first occurence of the special char ? in the docs
quantifier usually means something that tells how much/many things there are of something
have you checked out some kind of profiling yet @amanb ? you may want to see whats taking the longest time
@AndrasDeak repetition quantifiers
but not in the way I used it right, I am following using it for naming.
18:23
Unable to start Spyder after conda update/install packages? installation debugging help, irreproducible, user solved own issue
yes, you're using question marks in a different context, part of different syntax
?
    Causes the resulting RE to match 0 or 1 repetitions of the preceding RE. ab? will match either ‘a’ or ‘ab’.
what I've been trying to tell you ^
@smci none of those are close reasons
oh, you mean "typo/no repro"?
@AndrasDeak I see what you mean
@smci I guess no repro works, sorry
I though that because I have the <> after it is not used as such
inside (?<...>) it isn't. But you could use it as a quantifier instead of the |() parts
18:26
@AndrasDeak :) to borrow your favorite line, I'm not idly standing by watching you argue with yourself... :-) Happy New Year cbg
happy new year
Anyone here any opinion, positive or negative, on library Box? Box: Python dictionaries with recursive dot notation access, Box github homeoage
@AndrasDeak like such?
((?P<wx_intensity>(-)|(\+))?)(((?P<vicinity_or_not>(?<!O)VC))?)(?P<description>(BC|BL|DR|FZ|MI|PR|SH|TS)?)((?P<precipitation>DZ|GR|GS|IC|PL|RA|SG|SN|UP)?)((?P<obscuration>BR|DU|FG|FU|HZ|PY|SA|VA|)?)((?P<other>(DS|FC|PO|SQ|SS)?))
@ParitoshSingh, I'll look into cProfile. but I'm very sure the API calls take the most time. For any piece of information that goes into the csv, a Requests session authenticates. So while the script is running I see many Warnings about using an SSL certificate which confirms the API calls are several.
@Vader maybe, although you probably have to put the quantifier outside the group, as in (?P<description>(BC|BL|DR|FZ|MI|PR|SH|TS))?
18:29
you might not be able to do a whole lot in that case
otherwise you'll get empty matches all over the place
also oh god, regexes are scary
this is aviation abbreviation regex, not exactly entry level ;)
meh I've seen worse
it's not so bad without the labeled group syntax
I did do that @AndrasDeak so many parentheses
@Vader do what? What I posted has the parentheses elsewhere compared to what you posted
Now, here's a good question about regexes: stackoverflow.com/questions/800813/…
What a confusing read coming back into chat :)
@Vader I think you can drop some of those parentheses, unless you also want to match what's inside the name groups (you probably don't)
18:35
@roganjosh, I've been pondering over a problem with concurrency and performance with an old script. I described it earlier in the chatroom..do you have any good suggestions?
I've just come back and there's multiple convos going on. I'll have a look at your code shortly but I promise nothing
@Vader something like this:
(?P<wx_intensity>[-+])?
(?P<vicinity_or_not>(?<!O)VC)?
(?P<description>BC|BL|DR|FZ|MI|PR|SH|TS)?
(?P<precipitation>DZ|GR|GS|IC|PL|RA|SG|SN|UP)?
(?P<obscuration>BR|DU|FG|FU|HZ|PY|SA|VA)?
(?P<other>DS|FC|PO|SQ|SS)?
(just on the same line unless you allow verbose regexen or something)
@roganjosh, thanks, however, there's no code..just looking for advice
@AndrasDeak thanks. I will give it a shit. ALso how did you know it was a metar
* a shot :P
18:42
@bhatnaushad no, for the python parts, I don't think they can be improved tbh. I have such questions myself. I don't think you should have been banned though
well, too late to edit it now, haha
@Vader I saw METAR somewhere a week ago
@Vader, good one!
exciting stuff
Does METAR have fixed order? I hope so
18:44
yes
but not everyone follows it
:/
Undefined order will mess with regex
and sometimes they are incomplete. when you see an M instead of something it means missing
@AndrasDeak thats why I search for it
You might have to consider parsing in python
How can I do regex in python?
Without regex
Loop over words -> parse the word
You're also not matching the numeric values and whitespace with your regex...
18:48
I've already gotten the other METAR stuff regexed
the weather group is that hardest one so far
that and clouds
@amanb I actually don't have a clue what you were asking my opinion on. Did I miss something?
@roganjosh, the problem is: I had written a script that queries the Github API and extracts issues, milestones and lot of other data and writes to a csv file. Later I also added the Zenhub API queries to do the same thing. The average time to generate a full csv report is about 15 minutes. I am trying to find a way to improve the performance.
contd..
The API calls are too many and I am trying to achieve concurrency in API calls & R/W to csv. I'd like to get this report in say 5 mins. I explored asyncio and now into Cython.Any good suggestions how I can achieve this?
@Vader oh, you're matching a single word now, I missed that, sorry
I can be. But it doesnt have to
I meant the above regex
FZFG for instance
18:53
Oh in that case I am
@amanb requests-futures?
but it should also work one something like -TSRA BR
@AndrasDeak I just tried out your version and it gives me a million empty groups
Then you need to add optional whitespace \s* between blocks
That's async, super easy to use, and you can add callbacks
@AndrasDeak I did that and now there is even more matches
18:56
yeah, that will do that
@Vader my version doesn't have any *, yours does
((?P<wx_intensity>(-)|(\+))*) <-- that's very much not what I wrote
Sorry, you're right. I have so many debuggers open
I know
[cv-pls] debugging help/ typo New to Python - operator //
@AndrasDeak Alright I tried it again and it's finding them now. But I is still returning the empty ones too. I guess I can use python to ignore those
@smci closed
@roganjosh, yes, that's an interesting library. I'll be looking to tie in ThreadPoolExecutor with FuturesSession. One ques: Is it advisable to use threads for API calls and file R/W or should I use processes for that? Or a mix of both?
19:00
@Vader well each group has a ? which means that the pattern will also match an empty string
if you know there's a mandatory part for the weather block you should remove the ? from there
If there is no weather going on
it will just not include anything for the weather group
KDEN 131853Z 21003KT 10SM FEW100 FEW220 00/M06 A3022 RMK AO2 SLP262 T00001061 $
I know, but if there is a weather group, is there any part that is mandatory?
yes
either group precipitation or obscuration
hmm, that's a toughie
"either of these or both but not neither" is not very regular
Let me rephrase that, one of of the two is REQUIRED
19:03
but both can appear, right?
but it can have both in it and still be valid
correct
yeah, that's what you said originally, alas
@amanb well, I/O isn't bound by the GIL so much. So maybe threads are the best way. But I'm gonna defer for someone else on whether that's the best approach and I don't think we have enough info anyway
if only either could be used you could have ((?P<precipitation>DZ|GR|GS|IC|PL|RA|SG|SN|UP)|(?P<obscuration>BR|DU|FG|FU|HZ|P‌​Y|SA|VA)), but then allowing both is a mess
@roganjosh, I think I'll work on simple examples and find it out!
19:08
@AndrasDeak thanks for your help so far. I will see what I can do as far as filtering the reuslts.
The script is a mess
@Vader no problem, good luck
you can ignore zero-length matches for starters
Is was going to just ingore '' matches
same thing right?
Yeah. Regex match objects have attributes that tell you the length
 
1 hour later…
20:12
I require help
so I am trying
is anyone there?
20:41
Phew, just managed to update the files on an mp3 player using mtp in the command line on debian. I feel like a real hacker...
NAME
       mtp-tools - tools for communicating with Media Transfer Protocol (MTP) devices
SYNOPSIS
       [...]
       mtp-delfile [args]
DESCRIPTION
       This manual page documents briefly the commands distributed with the libmtp package, collectively called mtp-tools.
       This manual page was written for the Debian distribution because the original programs do not have manual pages. Some of the commands bundled with mtp-tools may provide a concise synopsis if were run with -h.
And...that's about it. Well there's also a link to the sourceforge page
@RayWu some people are always around but they will only help if they see a specific problem. That's why our rules page suggests that you just ask your question and then see if anybody can and is willing to help
01:00 - 21:0021:00 - 23:00

« first day (3011 days earlier)      last day (2162 days later) »