Basic question: can someone explain to me what this syntax means? codecs.getreader("utf-8")(sys.stdin) Is this codecs.getreader returning a function and I'm then calling it with argument sys.stdin?
Your original guess would have been correct if you had said "codecs.getreader returning a callable and I'm then calling it with argument sys.stdin". One word off isn't bad :-)
Even the official docs play fast and loose with the distinction between functions and classes. For example, built-in functions mentions bool, dict, float, int, type, etc. None of those are functions.
Hello! I will need a lot of data for a ML project but I do not have a lot of money for storage. Im looking at this and I will need storage for 5-15 GBs per day for at least a couple of weeks-months.
The original format is JSON. I dont know much about compression and I was wondering if the best option was simply to compress that on the file system until I can download it locally from the server. Is there a better way? Should I look into databases rather than files?
You might have luck composing your own data format. For example, If you know the data is a list with a million elements, and every element is a list of exactly eight integers that all lie within 0 and 65,536, then you can eliminate 100% of the brackets and commas and just stick the whole thing in one giant binary table
Cause I got offer a double in pay, but I'm stuck with that scenario and I don't know what to do, on the plus side, double pay - on the down side.... Downtown commute is so off putting lol
on the up side it's Python, downside Django, on the plus side I'm moving in life for more experience, on the down side, the location is god awful, on the plus side it's full stack, on the down side that means front end too :\
I've never compressed one, though, so I don't know how much you might save with that but it's super simple to test with just generating large numpy arrays of random values
Even if compression works poorly on arrays of random values, it might work better on actual real-world data, for the same reason that a picture of a bunny compresses better than television static.
I have a question, not wanting a full deep and thought through answer right now, just a few keyword to do some research on my own because after a few hours or researching I'm still not sure where to actually start.
Basically, I would like to implement a small encryption and/or authentification-protocol between my Python based webserver and a external application with a custom scripting language. Connection is over http requests, nothing else possible because of limitations of the external application. My Python server is quite beefy, however the target application is not so I would like to do as much heavy calculations on my server side as possible.
Additionally the target application would have the same script running on multiple instances that need to have a seperate "key", so I can't just hard code something into that, I would need some sort of lightweight key generator and key exchange protocol
Any broad ideas of how I could approach this problem?
That's a good question, actually. I am not so sure. Encryption is not my main problem, because I've designed the interface in a way that it's accepting control commands, some identification about the agent sending the request is send over with the http header, but that could very easily be faked. My main concern is about making sure that my server actually know who it is talking to.
No sensitive data is stored on the server, encryption would more be some sort of nice to have to write onto an advertisement, but that's about it.
What I would do is have a two-factor authentification thingy so I can expect the first communication to be from whom I think I'm talking to. (Basically first create an account on my side, then link it with the external application agent.)
Then I was thinking about pretending there is no middle man and just exchanging some key. It's not perfectly safe, but it's a lot better then it is right now
Public key cryptography ensures authenticity, to the extent of "the person that sent this message is the same person that gave me this public key". Of course, you're doomed if you have no way of verifying where the public key came from.
Yeah, my question is, what sort of protocol should I use for that? I know about the principle of diffie hellman so that should probably work, but I've also read that's not really something I should use because it's not safe enough.
Well, they would have to offer https. In the days of letsencrypt, or, if you control both ends, just self-signing your own stuff, there's no reason they shouldn't be able to
Uhm, kind of, well, a Second-Life kind of system. (This is probably the point where people will kick me out of the room, so have a good day everyone xD)
recbg - I'm pretty excited - managed to mock open() and a function call plus a couple of variables, check what they were called with and have it all working!
I control a build in script that allows me to send http requests and receive the response. I can't go deeper, though. I give it a URL, and get a HTML in plain text as response.
No, I mean that's it. There is literally one function that allows me to interact with the internet, and that's it. HTTPRequest(URL) -> HTTPReceivedEvent(Header, Body)
what capabilities do you have on that server? Like... can you upload scripts? Run arbitrary programs? Or are you limited to whatever kind of access they give you?
Okay, for sake of simplicity, let's say "A" is my python server and "B" is the client accessing it. On "A" I can do everything as long as it doesn't catch fire doing so. On "B" I can write scripts in the build in scripting language.
That scripting language has as far as I can tell only one single HTTP-Request function and raises an HTTP-Response event once the HTTP-Request got a response from the server.
Can I ask what "B" is? It sounds more like the pieces of kit we have the factory that come packed with their own language which is completely horrific to work with and lacking lots of basic capabilities
Someone is giving me money, I'm trying to at least make it a bit secure :D
I can't test it right now since I'm not at my home PC, but I will test it later. What happens if I try to access an HTTPS sever with a client that does not support HTTPS?
Do I just get some authentification HTTP error code or something like that in response? Because if so, that would probably be quite easy to implement, the server I'm programing on already has an SSL certificate. (Hope I'm not mixing terms up here, I'm new to web stuff ^^°)
So what should I google to get all the insight on how to implement that? If it's based on HTTPS that means I have a direct connection to whom I'm talking to, right? That would mean the wollowing would be an acceptable approach:
User creates an account on "A" -> User gets an one-time use key from "A" -> User inputs the key to "B" which sends it together with a unique-key request to "A" -> "A" returns the new private key for "B", and because It's HTTPS it's secure -> With every request I can do diffie helmann magic to check what user "B" currently corresponds with.
Just probably formulated in a more intelligent and gramatically correct way
Or, probably I don't even need the diffie helmann at that point, I can just send the private identifying key of the instance on "B" to "A" in plain text, since it's still always HTTPS
I wouldn't just give them the API key in plain text. Just some two-factor authentification key which the SL script takes and sends together with a request for an API key to the server and gets a API key back that's stored in it's script memory (which is a thing.)
Yeah, so a 1-time use API key to get a n-times use key
Not attack, just don't want to have the actual key visible to the user that might be doing some Let's Play or Lifestream or screenshots or whatever. Just a small extra step to make sure users can't accidentally share their API key :D
I don't know whether this perspective will be relevant, but maybe. I'm doing a lot of work on Flask atm and layering in more and more authentication as the system grows. Thankfully it's internal, but authentication is really tough for me to feel like "I know enough". Are you sure that piling in more and more complex systems on something you're just learning isn't actually opening up more vulnerabilities?
This is probably way too much overkill anyway. I'm just looking at this as an personal excersice to get a bit into the idea behind authentification methods.
Like, it always seems to me that these things all have to slot nicely in together or it's kinda just a mashed up system that I don't properly understand, and that gives me the shakes
Basically this way I am completely counting on HTTPS to make sure I have a secured and encrypted connection. My API-Key is just a unique identifier that gets send over this already secure connection for me to be able to link a SL user to a user on my backend.
Thanks for the help everyone, now I have a idea of how to achive a basic layer of protection! I'll look a bit more into how HTTPS actually works because it seems like I don't know enough about it considering the fact that I didn't even think about that option.
I think even though it's a bit overkill in my example it's always a good practise to think about security and encryption when your program is somehow communicating with something external :D
@Uebertreiberman that wasn't the point I was making. For sure it's good to think about authentication. I just won't allow myself to implement multiple different approaches of increasing complexity at the same time while I'm learning all of them. I'd rather go with the base (but industry acceptable) level that I can focus on, and improve as I learn more
@AndrasDeak yep, I used to graphitize wood at Uni with a single-mode microwave. That reaction is... scary. Just seems odd that hundreds of votes were thrown about but the accepted answer had 0 and was posted yesterday. The epic Physicist politics that I was imaging have been destroyed now, though :/
Throw enough microwaves at it to get to 550 deg. C as fast as possible from the moisture content and kaboom, thermal runaway
pandas is driving me a little bonkers, i imported this csv and want to make its date the only index, but even after I set the index there is this list of integers it seems is still an index
Oh yeah, but it still throws it right to the top for a lot of the 17k viewers, and there's plenty of voting going on, so I was just surprised to see that there was not a single vote on that particular post
@AndrasDeak my big loss from that project, actually, was not taking more note on the chapter of one of the books I had about mashed potato in a domestic microwave. Plenty of water content, but it never heats properly :'(
Considering I usually just throw reset_index() at my problems when I get to the point of punching the screen with Pandas, and hope for the best, I'm probably not the person to explain it. But my first comment should fix your issue.
And why shouldn't it? You've just described your own approach, it would be unreasonable for the library to guess (I'm not sure it even could) what you wanted from that setup