« first day (2726 days earlier)      last day (2447 days later) » 
00:00 - 15:0015:00 - 00:00

00:29
Just finished my new canonical. Off to airport. rbrb for now folks.
props to you for writing all of that. I didn't even have enough patience to scroll all the way down
Lol thx. That was a ton of work. And I’ve already earned a downvote 😊
wim
wim
00:49
people hate on self-answers
I didn't downvote, but you might want to make the question a little less contrived
looks like a blog post currently
Note that the last canonical piRSquared wrote was in september and is almost at a 100 upvotes
And that's one behemoth of a Q&A
rbrb. Thanks for the help @wim @Aran-Fey
We just need something as a target to close RTFM questions because people apparently don't know How To Google ™
Next on the list would be merge/join questions. There'd need to be a canon on normalising JSON, and a primer on the hundreds of available options in pd.read_csv as well.
@wim yup
wim
wim
00:52
I'm surprised that there are 0 downvotes on it, given that there are 9 (!) questions in there
(okay, november, not september... I stand corrected)
wim
wim
if the docs are lacking then why not make PR to official docs
you can help a wider audience this way
Sadly, questions cannot be closed as a duplicate of documentation. And while I agree that the docs need to be extended, Q&As like this certainly must be welcomed. Ever since november, the number of pivot questions being asked/answered has reduced like anything
And, those that are asked are speedily closed as duplicate by anyone with has a gold badge.
@cᴏʟᴅsᴘᴇᴇᴅ yes. lotta questions on json<->csv<->dict
01:10
@wim OK, could you give me a little bit more information a bout how to do that properly?
the correctly permissioned file, that is.
I mean, do you make it where it's group controlled, and the app operates in that group, so it has RO access to it?
wim
wim
yep
or you can control by user, if you don't need the group otherwise
you set the minimum mode bits on the file necessary so that the user your app runs as can read in the secrets
it's discouraged to use environment variables because 1) environment is inherited by subprocesses, who have no business knowing your secrets, and 2) the environment can be accidentally leaked by crashes and/or logging
gotcha
Thanks for help, dude. I owe you a beer.
or libation of your choice.
I don't judge.
01:36
@Aran-Fey "To improve your understanding on how to not write python code... see [...]'s solution"
01:58
Hi everybody
ok, i need an example of a process that doesn't benefit from multithreading because there is no way to evenly divide up the work....
@VictorA. cbg
    In Flask using this:

    @app.route('/user/<username>')
    def show_user(username):
        user = User.query.filter_by(username=username).first_or_404()
        return render_template('show_user.html', user=user)

how need to be a href button to get /user/<username_in_actual_session> ?
 
5 hours later…
06:51
Hi,
Whats the best way to read large JSON files (in my case 6GB) in python?
I tried pandas read_json with chunks. It gives up by 'ValueError: Trailing Data'
Is the entire contents of the file one big json object? or can it be broken up?
There are multiple objects
@piRSquared
Its a mongoexport to be specific
Is the entire file a valid json object? Or is it several json objects saved to the same file... oh ok. Not familiar, one sec
@MayurBhangale Just to be sure, did you pass lines=True to read_json()?
06:58
@MayurBhangale If you have a 6GB JSON file, you've already done something wrong.
@IljaEverilä Yes I did, It still crashes
@JGrindal I am taking a mongoexport in JSON format. Any alternative to consume data like this for machine learning models?
@MayurBhangale mongoexport --type csv will export it as CSV, which is much more easily handled than a JSON file.
I have zero experience doing this but if push came to shove, I'd write a simple parsing tool that kept taking characters until I got to the closing bracket/brace I started with. Process that and move to the next thing. I'd be left hoping that each object wasn't too big on its own. And that's all I got.
@JGrindal
I agree, but parsing it further is difficult since I have fields like addresses which themselves have commas
@MayurBhangale You'd usually want to feed a model line by line. What JGrindal wrote about CSV being easier for that approach is true, and a good start
You can also use h5py.org if you still run into problem with memory
@MayurBhangale Commas can be escaped, and a good non-buggy CSV parser will do that automatically
07:05
Depending on your dataset size, loading it as a CSV may STILL be problematic (I've had issues with 2GB csv's loading into a pd df), so you may want to process it in chunks, but the first thing you need to do is NOT have 6GB JSON file.
@Arne Thanks! I'll try this approach.
@piRSquared That's a good idea, until one of your datachunks has too many rows and has the same memory issue as parsing the whole file.
Not that I speak from experience.....
<.<
>.>
<.<
@JGrindal I was splitting it after, my issue was loading it in first place
@MayurBhangale Yeah, I think you'll find that dealing with CSV's is always preferrable to JSON objects if you can get them. The main way that JSON comes in handy is if it's being served via RESTful API. If you have direct access to the database, there are almost always superior ways to have your data served to you.
@JGrindal Makes sense. Thanks!
07:11
Also, @MayurBhangale I'm taking a guess that you starred those two posts. We tend to reserve starring posts for other things. See room rules https://sopython.com/chatroom

Don't star posts to reward users for answering your question. Your gratitude is reward enough. If your question meets Stack Overflow's standards, you may post your question there and invite the user to post an answer which you may accept.
@Arne h5py is great if all you have is numerical data. I've had issues with string-based data like addresses and the whatnot in HDF5, especially since numpy doesn't like to play nice with null-terminated strings
That said, I'm far from a power user, so I may have just made a dumb that's easily avoidable by people smarter or more detail oriented than me.
Not using it is an effective way to avoid that problem.
That has been my solution since then.
Can confirm csv isn't helping either. Pandas throws ParserError.
huh
maybe it doesn't escape comas
07:16
@MayurBhangale can you post a traceback?
I wasn't prepared to debug so early on a monday
@JGrindal Sure
@Arne it's tuesday.
D=
well, we had easter yesterday, so today feels like monday
Just keeps gettin' worse, doesn't it?
07:16
@Arne laurel... You got the early part right
Traceback
https://gist.github.com/mayurbhangale/e378b32d9106486ed17f5dd2bcf0905e
@MayurBhangale OK, so pandas is having an issue tokenizing your data. This could be for a few reasons, but is almost assuredly because there are a few lines that are screwed up in formatting (someone has a comma in their number or something).
in your pd.read_csv add error_bad_lines=False and see if it can read your data.
It's skipping almost everything after error_bad_lines=False.
Any other way in which I can split JSON in parts?
@MayurBhangale check your CSV and make sure your first line is formatted correctly (pandas infers the number of columns from the first row, so if its screwed up, it tries to impose that on the future lines)
@JGrindal mongoexport is dumping JSON in CSV which seems weird to me
07:26
@MayurBhangale Here's the thing - JSON files are not designed to handle machine-learning-training-dataset sized chunks of data. So you can either limit the size of the JSON file you're getting served by narrowing your data query to get less data and then concatenating later (this sounds like a lot of work), you can parse the JSON bit by bit (which is also a lot of work and is error-prone), or you can deal with a more efficent means of data storage like CSV.
@MayurBhangale that's very odd - I've never seen that kind of issue with a mongoexport before....
@JGrindal I'll try to deal with CSV. Thanks.
Sorry I can't be of more help, it's difficult to troubleshoot some data-specific issues without the data, haha.
@JGrindal Haha, can understand. Appreciate your help.
07:40
It might be a good idea to write up a canonical "data processing for ML"-post one day..
@Arne Step 1: YOU NO USE JSON. JSON BAD.
Step2: something something pandas
Step3: ??
@JGrindal Nice, the title's already there
Step 4: Profit
Step 1.5: Learn pandas
Still stuck on that one, I continue writing everything in plain ol python
like a caveman
Out of curiosity... I'd like to know which syntax would one find most convenient.

def max_1(x, y):
if x >= y:
return x
return y

def max_2(x, y):
if x >= y:
return x
else:
return y

def max_3(x, y):
return x if x >= y else y
07:50
last one is most readable imo
second one will be highlighted by some linters with 'redundant "else" clause'
def max_b(x, y):
    return [x, y][int(x < y)]
oh god my eyes
'fourth one will cause headache when reading it'
from operator import lt
max_b = lambda *t: t[int(lt(*t))]
can you do one with itertools for bonus points?
laurel
And no... I have standards
07:56
I know one that's having fun. ;D
If you know C, you gotta be good at the [International Obfuscated C Code Contest](https://www.ioccc.org/).
For how long have you been using python piRSquared ?
4 years
I'm not a developer. Not really, it's complicated.
Team py27 or py36 ? ;D
(I see the py35 coming)
3.6 f-strings ftw
You're unaware of .format() in py27 ?
I'm aware. (-: f-strings win on coolness
08:01
ha
who introduced f-strings?
^ googling now
Author of python.org/dev/peps/pep-0498 is Eric Smith
Don't know if that means introduced by
Question about ftplib. I'm seeing a lot of stuff using the context:

data = []
data = ftp.dir(data.append)

but when I try to do this, data is just left as `None`....
Formatting hates me today.
You are a ware of This method returns None.?
08:14
@Arne "If the last argument is a function, it is used as a callback function as for retrlines()"
oh
I'm an idiot.
quack
08:26
https://github.com/python/cpython/blob/2.7/Lib/ftplib.py#L533

You can just see in the source code that there's no `return`. :P
Yes, it was a holdover, the data.append was the callback during the run of the function, but all of that got overriden at the end when it set data= (return value), which was None.
I forgot to remove the data=
08:50
Ugh, why is there no standardization around how directory listings are returned from an FTP server.
I blame the Mongol Horde.
09:01
I'm trying to come up with a regex pattern, can anybody help?
Depends
In this string: "render -u user -p pass -h 127.0.0.1" (including quotes), I want to match -h 127.0.0.1
-h \S+ this almost works, except it includes the quotation mark at the end
([\d\.]+) will match a sequence of digits and dots
the rest depends on how you apply the regex
(-h \S+).$
09:05
but it needs to also work if it's an alpha word, like localhost
@JGrindal I need to get rid of the quotation mark
@ElliottB Group 1 of that regex will be sans the quotation mark
@ElliottB if you don't want to use groups -h \S+[^"]
inp = 'render -u user -p pass -h 127.0.0.1'
inp = np.array(inp.split(' ')[1:]).reshape(-1, 2)
print inp
[['-u' 'user']
 ['-p' 'pass']
 ['-h' '127.0.0.1']]
I personally would have done this to parse your string.
@IMCoins That's not a regex.
"I need to parse a string"
"OK, FIRST IMPORT NUMPY"
:thinking_face:
09:12
>>> import re

>>> command = '"render -u user -p pass -h 127.0.0.1"'
>>> ip_matcher = re.compile(r'(?<=-h )([\d\.]+|localhost)')
>>> ip_matcher.search(command).group(1)
127.0.0.1
I just thought that regex was way too overkill for such a pattern. :)
@JGrindal thanks. That works, except if the string is rearranged like this "render -p pass -h 127.0.0.1 -u user" then it includes a space at the end
@IMCoins "Regex is overkill, better go to numpy" was your thought process? #DataScientistAlert
@ElliottB Then add whitespace to the list of omitted chars -h \S+[^"\W]
.. or whitelist the exact two patterns you want to match
09:16
@Arne what do you mean by whitelist?
cbg
@AndyK cbg!
@ElliottB If you want to match something you can either go about it by accepting broadly, like with \W, and then exclude everything you don't want to match in a blacklist. Or, if you have a very limited amount of things you want to match, you just match exactly those and nothing else, which would be a whitelist.
oh, gotcha
09:20
Cabbage
cbg
If I'm using a set as a hash table, is there a pythonic way of checking the max value of the key?
I mean, I know I can iterate through the set and find it, but that's so..... unsexy.
@Cleb FYI, it's considered bad form to post your new question in chat, please read the room rules sopython.com/chatroom
@JGrindal: Sorry for that; will delete the post then; just kinda desperate by now... ;)
No worries, I understand the feeling. I would help, but not familiar enough with how flask does its routing.
Yes, also rather a beginner. Usually routing works fine, but now I basically need two routes, one for the json and then one for the output and I don't know how to do this but must be simple...
09:31
@JGrindal You can inherit from set and add a current_max member that gets updated in the .add method
I will not comment on the pythonicity of that approach though
Yeah, that doesn't solve my problem, haha
Well, if you want the maximum of an array... you need to, as @Arne suggested keep a current_max to save you the iteration each and every time you want it. If you don't do this, how would you know what's the max, without checking every element ? :p
@IMCoins Because it's not an array, it's a set.
Does that change its behavior regarding max?
@Arne yes
09:41
In how far?
Figured it out, using a dict instead of a set allows for it to work the way I intended.
Good to hear =)
set is implemented as a dict with only key values, so when you have a "2D set" it's doing it as a tuple
if you do it via dict, then max automatically searches the key value.
Interesting. TIL.
Now I also regret not making a joke about not using a dict as a first reaction to your question
@Arne it always pays off to be a smartass.
=)
09:46
I thought a set() was an iterable that looked like a list more than a dict.
@IMCoins Negatory. set and dict are both built as hashtables, but a set only implements the key part of it.
What did your set() looked like ? I'm curious
{(datetime1, value1), (datetime2, value2), ... (datetimen, valuen)}
And so, what was not pythonic using k = max(your_set) to get the max value, and then using your_dict[k] ?
Because the set was a set of tuples, and max doesn't know how to deal with tuples. With a dict, it gets the max key.
10:00
If you want to check the max of your set() that contains tuples, you can do : max(your_set, key = lambda x:x[:][your_index])
My formulation is not very eloquent, hope you got what I meant haha
using a dict does sound like the overall better solution for the problem though
and it handles max better as a bonus
I agree, but I'm just very curious, and exploring the above behavior that was unknown to me, I found this.
@IMCoins Yes you can do that, but essentially it's just converting it to a dict anyway, so you may as well just store the whole thing as a dict to begin with. Then the whole problem is clean and tidy =)
10:34
@JGrindal what do you mean?
>>> max({(0,'a'), (1,'b'), (1,'c')})
(1, 'c')
@AndrasDeak it was giving an error with a (datetime, string) tuple.¯\_(ツ)_/¯
ordering on tuples works lexicographically just fine
This is a Python room
10:46
lexicographically will be my word of the day
... :D
@kanishktanwar as vaultah has said, this is a Python room. If you have a question related to Python then please read our room rules at sopython.com/chatroom and then ask if appropriate.
10:55
thumbs up
jjj
jjj
11:18
cabbage all o/
11:46
cbg o/
12:43
cbg
@kanishktanwar Try this room: chat.stackoverflow.com/rooms/29074/html-css-webdesign they might be able to help there.
13:01
How exciting, I'm actually using a db for a personal project
I'm tracking the health of my network connection and I need to store the results of the pings I'm running every thirty seconds. Don't really want to store 8*60*2 records per day in a text file.
@Kevin if you're just checking health, only store the changes?
If you're saying "only add new records if the health changes", I expect it to change every single time, since ping measures packet time at a millisecond resolution
If you're saying "store the delta of the current packet time minus the previous packet time", I think that would take up as much space as just storing the current packet time
@Kevin if you're only checking health, presumably, you only want to know if ping works or not
Additional complication: I also want to store information about pings that can't find the host, and pings that can find the host but can't send packets. Neither of those have any numeric data to do math on
If "health" means only "did ping work, yes or no?" then I guess health isn't just what I'm looking for
@Kevin how are you going to look at the data afterwards?
in other words, can you aggregate the data every 15 mins or something like that?
13:13
I'm thinking a graph of time of day vs. average round trip time, and maybe incorporating information about packet loss via background colors or something
@Kevin otherwise, it's less than 2kB/day, so after a year you'd still have less than 1MB
so just keep it that way.
I figure it makes sense to capture more data than necessary now, and I can always cut it down later
@Kevin honestly, it's just numbers, space won't be an issue. If it ever is, you'll probably gain a lot by just figuring out a nice data layout on disk. (e.g. compression does wonders.)
\o cbg
@MooingRawr cbg!
13:24
o/
Yeah I suspect even the most clownshoes database schema will be better than appending to a text file. It's hard to bumble into O(N^2) runtime if you're just doing regular inserts
34
A: database vs. flat files

voyagerThis is an answer I've already given some time ago: It depends entirely on the domain-specific application needs. A lot of times direct text file/binary files access can be extremely fast, efficient, as well as providing you all the file access capabilities of your OS's file syste...

Well, according to some posts on SO, it is best to just have a textfile database for you.
Morning testing out the flask tutorial cabbage
Maybe thirty minutes ago that was true, but consider which is simpler: the textfile implementation that has not been written, and the db implementation that has. The latter requires zero lines of code and zero seconds of development time
In other words, I'm already in this hole, may as well keep digging
Well, anyway, as a collegue once told me : "if you want full optimization, you shouldn't choose python in the first place".
13:34
I need to process two records a minute. I could do that with Python running on a speak-n-spell.
However, just because you're using Python doesn't mean you have to code without regard to time complexity
Aaaaand my monitoring loop has crashed. Ah, it's my first lost packet of the day. I couldn't write a parser for these because I hadn't seen one yet.
The above message took fifteen seconds to make it to the server, so I'm guessing there are many lost packets in my future. Hooray data!
I love when we can celebrate failure!
cbg everyone
13:41
Hmm, what should I name my boolean that indicates whether a packet was lost or not? I would prefer True to indicate a success condition, so I can't call it lost. I need the antonym for "lost". But "found" isn't a great fit.
I guess I'll go with connected
@Kevin Could you? I feel like there should be a record of this a la kernelmag.dailydot.com/issue-sections/features-issue-sections/…
@Kevin How about, acquired?
Although connected is somewhat generic, it maintains a consistent interface because I also have a connected attribute higher up in the data structure that indicates whether ping was able to resolve an ip address from the given host name. If I have ping.connected and ping.packets[0].connected, then the user only needs to remember one name, rather than thinking "which one was the boolean for the packets, again? Was that connected, or acquired?"
@Kevin arrived
cbg
I have a problem and I don't even know how to ask... I don't know what's happening but something's not working for some reason
I don't know if there's a single word that encapsulates the action of sending a thing, having the thing make it to its destination, and then having the destination send you a thing which makes it to your point of origin
13:47
Oooh, I have it! Call it "condition_x"
packet_was_sent_to_the_destination_and_the_destination_received_it_and_they_sen‌​t_you_a_second_packet_and_you_received_that_packet = True
Of course I'm joking. It's much better to abbreviate that to pwsttdatdriatsyaspayrtp = True
@Kevin ack?
That is murder for pep 8 and max 79 chars python.org/dev/peps/pep-0008/#id19
@Neoares Happens to me all the time. Do your best :-)
@Kevin Hooo-ah!
13:57
Command 'ping www.google.com' returned non-zero exit status 1. I didn't even know it could do that.
@Kevin I'll...
despite I'm not good at explaining problems shortly
@Kevin resolved_both, resolved_up, resolved_down
it's related to sqlalchemy
my_model = model.Artifact(**some_properties)
ret = db.Artifact.save(my_model)
this is not working
but if I go into the project through the terminal, it works :D
and it's not giving me any error
so I just want hints / tips from more experienced people
because I just don't know how to handle this
Do you have a logger?
yep
my_model gives me the correct object
I'm getting the same things if I run it from terminal that from the real project
14:02
If it's something going wrong in sqalch you can set up the project to forward all log messages to a file or something. That's where I would start
but one doesn't store the element in the database :P
Rule of thumb: if your explanation includes "not working", you haven't investigated far enough. Determine exactly what you want the thing to do, and what the thing is actually doing, and the difference between them
IN PYCHAARMMM!!!!!!
I am stuck big time, there is this one HTML file and one CSS file(both in the same folder)
so if I run HTML directly it works as expected, but pycharm can't find the directory for css file ,and did not change the appearance of the page
"GET /styles.css HTTP/1.1" 404 - this is the error
14:06
you know pycharm has nothing to do with python, right?
3 hours ago, by vaultah
This is a Python room
well it took me 3 hours to ask again
You will be surprised
but this is still a python room
9
okay
i will be back
14:07
i am back
so in python
it remembers me to the small guy from "papers, please"
coming every 3-4 days giving you fake IDs and stuff like that
you reject him and he always says "no problem, I'll come back in a few days"
I have a problem with this library called flask
flask giving me error "GET /styles.css HTTP/1.1" 404 -
now is this python related? or its some module related and i need to refer their faq?
well, did you follow Simon's tip?
1 hour ago, by Simon
@kanishktanwar Try this room: https://chat.stackoverflow.com/rooms/29074/html-css-webdesign they might be able to help there.
thank you for not redirecting the guy to the JS one
@kanishktanwar It's a standard 404, which means either your URL is wrong or your settings for that is wrong. It should be relatively simple to find out.
14:13
@kanishktanwar remove the /
you said they're in the same path
go for styles.css or ./styles.css
cbg all
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Fancy blog</title>
<link rel="stylesheet" href="styles.css"/>
</head>
<body>
<form id="login-form">
<input type="text" id="email">
<input type="password" id="passsword">
<input type="submit" id="submit-button">
</form>

</body> </html>
this is that HTML file
This is a python room, please ask over in the HTML room about this type of thing
I am leaving anyway, will find myself thanks
Bye.
14:20
close the door
@kanishktanwar Your doubts are basic, and the info you are providing don't match correctly with what might be wrong (either the relative path is wrong or your static file serving is structured incorrectly). Please refer to a basic static files tutorial in flask.
rbrb for a bit
@kanishktanwar I think the problem here exists between keyboard and chair
cbg
@WayneWerner I just had a mental image of person sitting backwards in chair with keyboard resting back, facing a TV/large monitor, who is very confused at your statement.
14:34
lol
cbg all
14:46
does anyone have any sources on how to mount disk volumes?
man mount
jjj
jjj
(sryy bad joke)
I meant the hardware aspect
Hardware aspect of "mount[ing] disk volumes"? Beyond "plug in your HDD/SSD"?
@Neoares no need to be rude
Anyone have a good cheat sheet for rtf values? Trying to search for a good one that includes all of the characters. Id like to be able to just cntr+f and find "\fs18" not have to google each command.
14:51
@AndrasDeak I'm not rude, I was getting cold :P
@Neoares right
I come in between tasks, and it seems I can't read english properly.
00:00 - 15:0015:00 - 00:00

« first day (2726 days earlier)      last day (2447 days later) »