Next on the list would be merge/join questions. There'd need to be a canon on normalising JSON, and a primer on the hundreds of available options in pd.read_csv as well.
Sadly, questions cannot be closed as a duplicate of documentation. And while I agree that the docs need to be extended, Q&As like this certainly must be welcomed. Ever since november, the number of pivot questions being asked/answered has reduced like anything
And, those that are asked are speedily closed as duplicate by anyone with has a gold badge.
or you can control by user, if you don't need the group otherwise
you set the minimum mode bits on the file necessary so that the user your app runs as can read in the secrets
it's discouraged to use environment variables because 1) environment is inherited by subprocesses, who have no business knowing your secrets, and 2) the environment can be accidentally leaked by crashes and/or logging
In Flask using this:
@app.route('/user/<username>')
def show_user(username):
user = User.query.filter_by(username=username).first_or_404()
return render_template('show_user.html', user=user)
how need to be a href button to get /user/<username_in_actual_session> ?
Hi, Whats the best way to read large JSON files (in my case 6GB) in python? I tried pandas read_json with chunks. It gives up by 'ValueError: Trailing Data'
I have zero experience doing this but if push came to shove, I'd write a simple parsing tool that kept taking characters until I got to the closing bracket/brace I started with. Process that and move to the next thing. I'd be left hoping that each object wasn't too big on its own. And that's all I got.
Depending on your dataset size, loading it as a CSV may STILL be problematic (I've had issues with 2GB csv's loading into a pd df), so you may want to process it in chunks, but the first thing you need to do is NOT have 6GB JSON file.
@MayurBhangale Yeah, I think you'll find that dealing with CSV's is always preferrable to JSON objects if you can get them. The main way that JSON comes in handy is if it's being served via RESTful API. If you have direct access to the database, there are almost always superior ways to have your data served to you.
Also, @MayurBhangale I'm taking a guess that you starred those two posts. We tend to reserve starring posts for other things. See room rules https://sopython.com/chatroom
Don't star posts to reward users for answering your question. Your gratitude is reward enough. If your question meets Stack Overflow's standards, you may post your question there and invite the user to post an answer which you may accept.
@Arne h5py is great if all you have is numerical data. I've had issues with string-based data like addresses and the whatnot in HDF5, especially since numpy doesn't like to play nice with null-terminated strings
That said, I'm far from a power user, so I may have just made a dumb that's easily avoidable by people smarter or more detail oriented than me.
@MayurBhangale OK, so pandas is having an issue tokenizing your data. This could be for a few reasons, but is almost assuredly because there are a few lines that are screwed up in formatting (someone has a comma in their number or something).
in your pd.read_csv add error_bad_lines=False and see if it can read your data.
@MayurBhangale check your CSV and make sure your first line is formatted correctly (pandas infers the number of columns from the first row, so if its screwed up, it tries to impose that on the future lines)
@MayurBhangale Here's the thing - JSON files are not designed to handle machine-learning-training-dataset sized chunks of data. So you can either limit the size of the JSON file you're getting served by narrowing your data query to get less data and then concatenating later (this sounds like a lot of work), you can parse the JSON bit by bit (which is also a lot of work and is error-prone), or you can deal with a more efficent means of data storage like CSV.
@MayurBhangale that's very odd - I've never seen that kind of issue with a mongoexport before....
Yes, it was a holdover, the data.append was the callback during the run of the function, but all of that got overriden at the end when it set data= (return value), which was None.
@ElliottB If you want to match something you can either go about it by accepting broadly, like with \W, and then exclude everything you don't want to match in a blacklist. Or, if you have a very limited amount of things you want to match, you just match exactly those and nothing else, which would be a whitelist.
Yes, also rather a beginner. Usually routing works fine, but now I basically need two routes, one for the json and then one for the output and I don't know how to do this but must be simple...
Well, if you want the maximum of an array... you need to, as @Arne suggested keep a current_max to save you the iteration each and every time you want it. If you don't do this, how would you know what's the max, without checking every element ? :p
@IMCoins Yes you can do that, but essentially it's just converting it to a dict anyway, so you may as well just store the whole thing as a dict to begin with. Then the whole problem is clean and tidy =)
@kanishktanwar as vaultah has said, this is a Python room. If you have a question related to Python then please read our room rules at sopython.com/chatroom and then ask if appropriate.
How exciting, I'm actually using a db for a personal project
I'm tracking the health of my network connection and I need to store the results of the pings I'm running every thirty seconds. Don't really want to store 8*60*2 records per day in a text file.
If you're saying "only add new records if the health changes", I expect it to change every single time, since ping measures packet time at a millisecond resolution
If you're saying "store the delta of the current packet time minus the previous packet time", I think that would take up as much space as just storing the current packet time
Additional complication: I also want to store information about pings that can't find the host, and pings that can find the host but can't send packets. Neither of those have any numeric data to do math on
If "health" means only "did ping work, yes or no?" then I guess health isn't just what I'm looking for
I'm thinking a graph of time of day vs. average round trip time, and maybe incorporating information about packet loss via background colors or something
@Kevin honestly, it's just numbers, space won't be an issue. If it ever is, you'll probably gain a lot by just figuring out a nice data layout on disk. (e.g. compression does wonders.)
Yeah I suspect even the most clownshoes database schema will be better than appending to a text file. It's hard to bumble into O(N^2) runtime if you're just doing regular inserts
This is an answer I've already given some time ago:
It depends entirely on the
domain-specific application needs. A
lot of times direct text file/binary
files access can be extremely fast,
efficient, as well as providing you
all the file access capabilities of
your OS's file syste...
Well, according to some posts on SO, it is best to just have a textfile database for you.
Maybe thirty minutes ago that was true, but consider which is simpler: the textfile implementation that has not been written, and the db implementation that has. The latter requires zero lines of code and zero seconds of development time
In other words, I'm already in this hole, may as well keep digging
Hmm, what should I name my boolean that indicates whether a packet was lost or not? I would prefer True to indicate a success condition, so I can't call it lost. I need the antonym for "lost". But "found" isn't a great fit.
Although connected is somewhat generic, it maintains a consistent interface because I also have a connected attribute higher up in the data structure that indicates whether ping was able to resolve an ip address from the given host name. If I have ping.connected and ping.packets[0].connected, then the user only needs to remember one name, rather than thinking "which one was the boolean for the packets, again? Was that connected, or acquired?"
I don't know if there's a single word that encapsulates the action of sending a thing, having the thing make it to its destination, and then having the destination send you a thing which makes it to your point of origin
Rule of thumb: if your explanation includes "not working", you haven't investigated far enough. Determine exactly what you want the thing to do, and what the thing is actually doing, and the difference between them
I am stuck big time, there is this one HTML file and one CSS file(both in the same folder) so if I run HTML directly it works as expected, but pycharm can't find the directory for css file ,and did not change the appearance of the page
@kanishktanwar It's a standard 404, which means either your URL is wrong or your settings for that is wrong. It should be relatively simple to find out.
@kanishktanwar Your doubts are basic, and the info you are providing don't match correctly with what might be wrong (either the relative path is wrong or your static file serving is structured incorrectly). Please refer to a basic static files tutorial in flask.
@WayneWerner I just had a mental image of person sitting backwards in chair with keyboard resting back, facing a TV/large monitor, who is very confused at your statement.
Anyone have a good cheat sheet for rtf values? Trying to search for a good one that includes all of the characters. Id like to be able to just cntr+f and find "\fs18" not have to google each command.