« first day (3009 days earlier)      last day (2163 days later) » 
00:00 - 20:0020:00 - 23:00

20:01
^ huh, look at that guy quoting bobby tables, what an xkcd normie
Basic question: can someone explain to me what this syntax means? codecs.getreader("utf-8")(sys.stdin) Is this codecs.getreader returning a function and I'm then calling it with argument sys.stdin?
it would seem like it
encodings.utf_8.StreamReader is a function?
That's what codecs.getreader("utf-8") returns...
Have you given this a read, it returns a streamReader class or factory function
@flow2k probably a class, but you instantiate classes by calling them
20:11
Oh I see - so MyClass(bluh) is the syntax for instantiation...
new to Python of course...just trying get someone else's code working...
If you plan to use it beyond making that one code work, you'll probably benefit from reading a good tutorial :)
True...though finite time is always the enemy here...
Thanks though
finite time is a friend not an enemy, as I wouldn't wish to have infinite time with no way out :P
Your original guess would have been correct if you had said "codecs.getreader returning a callable and I'm then calling it with argument sys.stdin". One word off isn't bad :-)
20:18
Even the official docs play fast and loose with the distinction between functions and classes. For example, built-in functions mentions bool, dict, float, int, type, etc. None of those are functions.
huh. good to know
everything quacks :P
Yes, it is known.
panta quackei
Hello!
I will need a lot of data for a ML project but I do not have a lot of money for storage.
Im looking at this and I will need storage for 5-15 GBs per day for at least a couple of weeks-months.

The original format is JSON. I dont know much about compression and I was wondering if the best option was simply to compress that on the file system until I can download it locally from the server.
Is there a better way? Should I look into databases rather than files?
20:26
I know Oracle can do compression and I bet other database vendors have similar abilities.
files are a type of primitive database, anyways.... you could look into if there's a stream of ML data available or something (off topic)
I wonder if JSON compresses better than other kinds of data because it's got so many curly brackets.
@Kevin Yea thats what im looking at but for someone whos looking to save as much space as possible I dont think DBs do a better job.
For the curly braces I will change the format the remove all of that and get more of csv-like
It depends on the type of data in the JSON
is only numbers
20:29
Straight-up removing all the braces is... A daring approach.
Also, incrementally adding to a single JSON object will eventually get something that won't fit in memory to be able to search through anyway
@Kevin Yea the data is super simplistic thats why I can
how much of a raise % would it take for you to switch job and commute for an hour (plus or minus 15 minutes), into a spot that you hate? (off topic)
an hour more?
@Kevin it's ascii, that's more than enough usually
20:30
if ur current job was 10 minutes drive away
so, 50 minutes more
no just a flat hour Wayne,
or, 6 times your current commute
oh and I forgot current u get a car, new job forces public transit ! dum dum dummmm
I'd require at least 50%, I think. But then again, I'm very comfortable at my current position
I <3 public transit (mostly), so that's actually a bonus for me
20:31
Does the new work interest you more? That should count for a lot
but still, probably 50%
@roganjosh thanks, I dont care about read performance
similar work , just commute and pay is different
But presumably you do care about actually being able to parse the data
Yeah, I mean... it would have to be something pretty slick to draw me
20:32
You might have luck composing your own data format. For example, If you know the data is a list with a million elements, and every element is a list of exactly eight integers that all lie within 0 and 65,536, then you can eliminate 100% of the brackets and commas and just stick the whole thing in one giant binary table
called a numpy array :>
Cause I got offer a double in pay, but I'm stuck with that scenario and I don't know what to do, on the plus side, double pay - on the down side.... Downtown commute is so off putting lol
double pay doesn't sound all that bad
yea itll be an array basically :)
HDF5 may actually be less space if you don't have heavily nested objects
20:33
on the up side it's Python, downside Django, on the plus side I'm moving in life for more experience, on the down side, the location is god awful, on the plus side it's full stack, on the down side that means front end too :\
How can full stack be both plus side and downside? :)
HDF5 also supports compression
cuz you have both ups and downs :D anyways, I think I might pass as they look like they have trouble holding people based on reviews
@roganjosh hmm interesting, you just format the filesystem in HDF5 ?
I think he just means a single hdf5 file
20:36
See here. You'd parse your JSON arrays to numpy arrays and then store them
@roganjosh Thanks thats sounds neat!
I've never compressed one, though, so I don't know how much you might save with that but it's super simple to test with just generating large numpy arrays of random values
Don't forget to set a non-default integer type if applicable. As Kevin said you might not need to have int64s.
range of values is about 100k-0.00000001
Even if compression works poorly on arrays of random values, it might work better on actual real-world data, for the same reason that a picture of a bunny compresses better than television static.
20:40
ah, then you probably need default doubles (float64)
... compressed bunnies :/
Hello dear programming companions!
hello
I will squeeze him and compress him and name him George
Allright thanks yall, Ill look into that!
20:41
@JasminParent I would be quite curious if you go with it and can report back how it went :)
I have a question, not wanting a full deep and thought through answer right now, just a few keyword to do some research on my own because after a few hours or researching I'm still not sure where to actually start.
@roganjosh Sure, I will send you what I got :)
Basically, I would like to implement a small encryption and/or authentification-protocol between my Python based webserver and a external application with a custom scripting language. Connection is over http requests, nothing else possible because of limitations of the external application. My Python server is quite beefy, however the target application is not so I would like to do as much heavy calculations on my server side as possible.
Additionally the target application would have the same script running on multiple instances that need to have a seperate "key", so I can't just hard code something into that, I would need some sort of lightweight key generator and key exchange protocol
Any broad ideas of how I could approach this problem?
Tis not my area, sorry. People will chime in if they have ideas; don't worry, your question will have been read :)
I hope so a lot ^^ Looking forward to suggestions and thanking everyone in advance.
20:48
@Uebertreiberman not even https?
because, eww
That's a good question, actually. I am not so sure. Encryption is not my main problem, because I've designed the interface in a way that it's accepting control commands, some identification about the agent sending the request is send over with the http header, but that could very easily be faked. My main concern is about making sure that my server actually know who it is talking to.
No sensitive data is stored on the server, encryption would more be some sort of nice to have to write onto an advertisement, but that's about it.
If you don't have encryption then you don't have any guarantees that you're talking to the person that you think you're talking to
That's my question, how can I throw some sort of authentification layer into there?
As an example, suppose your boss hands me a piece of paper and tells me to give it to you
I can add or take away whatever I want from this paper and you have no clue whether it was really from him or not
that's where encryption comes into play
@Fabien I think this chat will be interesting for you
20:53
it allows you a way to guarantee that you and your boss can communicate with each other
I happen to be a very skilled graphologist. I'd be right on to you, Wayne!
and know that they're talking to the one they expect
@LoudghiriAhmed Thanks
What I would do is have a two-factor authentification thingy so I can expect the first communication to be from whom I think I'm talking to. (Basically first create an account on my side, then link it with the external application agent.)
@roganjosh what if I'm even better at photoshop? ;)
20:54
Then I was thinking about pretending there is no middle man and just exchanging some key. It's not perfectly safe, but it's a lot better then it is right now
@Uebertreiberman that's encryption
> exchanging some key
Public key cryptography ensures authenticity, to the extent of "the person that sent this message is the same person that gave me this public key". Of course, you're doomed if you have no way of verifying where the public key came from.
Yeah, my question is, what sort of protocol should I use for that?
I know about the principle of diffie hellman so that should probably work, but I've also read that's not really something I should use because it's not safe enough.
^what kevin said
@Uebertreiberman lol. I'm curious who said that because that's what like... everything uses? the DH
What people keep suggesting is "don't roll your own crypto, use a ready-made solution". Might or might not be very helpful guidance in this situation.
20:56
but you don't need to reinvent the wheel - https does exactly what you're asking
well, half of it anyway :)
what kind of system is this remote system? Like... is it a linux-y thing? Windows-y thing? Embedded-y thing?
I'm not sure if the target application actually uses https on their side, would it be enough to just tell it to use https on my server?
Well, they would have to offer https. In the days of letsencrypt, or, if you control both ends, just self-signing your own stuff, there's no reason they shouldn't be able to
Uhm, kind of, well, a Second-Life kind of system.
(This is probably the point where people will kick me out of the room, so have a good day everyone xD)
I prefer public key exchanges to occur by sitting on adjacent benches in the park and subtly switching identical looking briefcases
@Kevin you like to put the "public" in public key exchange, eh?
20:58
recbg - I'm pretty excited - managed to mock open() and a function call plus a couple of variables, check what they were called with and have it all working!
@toonarmycaptain congrats :)
It's not public if you're dressed like Spy Vs Spy, everyone knows that
Remembering how to do it tomorrow with another test is a different story, but...practice and perfection...
That would be a good implementation, just would probably time my server out :D
20:59
Oversized briefcases completely padded with foam, with just a little memory stick in the middle
@Uebertreiberman https is way less intensive than you think. If your server is not capable of doing https then it's not capable of doing security
My server is, but I'm just not sure if I can actually make use of it on the other side, I need both sides to know what to do with https, right?
I cannot imagine being able to write anything that's more secure than plaintext that's less intensive than https
@Uebertreiberman Yeah. Do you control the other side?
Just checked in to celebrate, rhubarb!
I control a build in script that allows me to send http requests and receive the response. I can't go deeper, though. I give it a URL, and get a HTML in plain text as response.
21:03
You mean into more detail?
No, I mean that's it. There is literally one function that allows me to interact with the internet, and that's it. HTTPRequest(URL) -> HTTPReceivedEvent(Header, Body)
on the server?
On my server I can do everything. That's the clients view of how to interact with my server.
okay, I guess I mean on the remote system
My server is a Linux server running Python running Flask running my website.
So, the remote system, so to speak
21:06
okay, so Flask can serve HTTPS just fine
(Sorry, I'm already a bit tired and no native speaker)
the other system(s) invovled
can they make https requests?
your language is fine
As I said, I don't think so.
or is this the on that you're telling us only has HTTPRequest/HTTPReceived?
21:07
That's the one
what capabilities do you have on that server? Like... can you upload scripts? Run arbitrary programs? Or are you limited to whatever kind of access they give you?
you mentioned a "custom scripting language"
Is that why stuff are limited?
sorry, I have no idea what you guys are talking about, just curious
Okay, for sake of simplicity, let's say "A" is my python server and "B" is the client accessing it. On "A" I can do everything as long as it doesn't catch fire doing so. On "B" I can write scripts in the build in scripting language.
and B involves that single send/get request function?
That scripting language has as far as I can tell only one single HTTP-Request function and raises an HTTP-Response event once the HTTP-Request got a response from the server.
Yes
21:10
OK, thanks
Much clearer now :)
Sorry I was not clear before ^^
See if you can provide https://google.com to that function
Can I ask what "B" is? It sounds more like the pieces of kit we have the factory that come packed with their own language which is completely horrific to work with and lacking lots of basic capabilities
because sometimes tools say "http" but they still support https
21:12
As I said, "B" is literally the game Second Life which is 15 years old by now xD
Ah, sorry, I hadn't put 2-and-2 together
Okay, so it's actually second life
I thought you said it was something like Second Life
Collective mental jigsaw... done. :)
21:13
Someone is giving me money, I'm trying to at least make it a bit secure :D
I can't test it right now since I'm not at my home PC, but I will test it later. What happens if I try to access an HTTPS sever with a client that does not support HTTPS?
Do I just get some authentification HTTP error code or something like that in response? Because if so, that would probably be quite easy to implement, the server I'm programing on already has an SSL certificate. (Hope I'm not mixing terms up here, I'm new to web stuff ^^°)
@Uebertreiberman according to my quick search, LSL should be supporting https: wiki.secondlife.com/wiki/HTTP_VERIFY_CERT
Ah, okay!
So what should I google to get all the insight on how to implement that?
If it's based on HTTPS that means I have a direct connection to whom I'm talking to, right? That would mean the wollowing would be an acceptable approach:
User creates an account on "A" -> User gets an one-time use key from "A" -> User inputs the key to "B" which sends it together with a unique-key request to "A" -> "A" returns the new private key for "B", and because It's HTTPS it's secure -> With every request I can do diffie helmann magic to check what user "B" currently corresponds with.
Just probably formulated in a more intelligent and gramatically correct way
Or, probably I don't even need the diffie helmann at that point, I can just send the private identifying key of the instance on "B" to "A" in plain text, since it's still always HTTPS
Yeah, assuming that there's a way to keep a session key around, that's basically what you're doing
So if your user comes on A, they would get something (typically called an API key) that allows them to call A from... well, wherever they want :)
but in this case, you're going to assume it's from Second Life
(if that's not an assumption that's safe to make, you have some other questions to answer)
and as long as they provide the API key with their request, you'll treat it as valid
because they're connecting to you over HTTPS though, that's how you can assume that nobody else is using their key
Of course, like a real key, if someone else copies it then they can get access and do anything that the other person could do
It's good to give them the ability to regenerate their API key/invalidate old ones, for that reason
Then someone will come along and ask a question about your server on Stack Overflow and post their API key :P
^that
or post it in a gist on GitHub
21:28
I wouldn't just give them the API key in plain text. Just some two-factor authentification key which the SL script takes and sends together with a request for an API key to the server and gets a API key back that's stored in it's script memory (which is a thing.)
This seems like overkill..
that's effectively an API key - it's just a 1-time use API key
what attack are you trying to mitigate against with that approach?
Yeah, so a 1-time use API key to get a n-times use key
Not attack, just don't want to have the actual key visible to the user that might be doing some Let's Play or Lifestream or screenshots or whatever. Just a small extra step to make sure users can't accidentally share their API key :D
I don't know whether this perspective will be relevant, but maybe. I'm doing a lot of work on Flask atm and layering in more and more authentication as the system grows. Thankfully it's internal, but authentication is really tough for me to feel like "I know enough". Are you sure that piling in more and more complex systems on something you're just learning isn't actually opening up more vulnerabilities?
This is probably way too much overkill anyway. I'm just looking at this as an personal excersice to get a bit into the idea behind authentification methods.
21:34
Like, it always seems to me that these things all have to slot nicely in together or it's kinda just a mashed up system that I don't properly understand, and that gives me the shakes
Basically this way I am completely counting on HTTPS to make sure I have a secured and encrypted connection. My API-Key is just a unique identifier that gets send over this already secure connection for me to be able to link a SL user to a user on my backend.
But you're already planning a base system with two-factor authentication
I don't expect there to ever even be an attempt to break into my system becauses there is literally no reason to do so :D
Anyway, my own perspective might be completely unfounded in the first place, it just haunts me :P
any reason why, on windows, os.chdir passes without error but os.curdir keeps returning the same thing?
21:39
Thanks for the help everyone, now I have a idea of how to achive a basic layer of protection! I'll look a bit more into how HTTPS actually works because it seems like I don't know enough about it considering the fact that I didn't even think about that option.
Thankfully I don't have to worry about GDPR for now... until the machines I talk to get rights etc. At that point I need a new job.
by then your company won't be operating in the EU ;)
... ..., well, .... You're mean.
I just realized that the new "code fence" code blocks on SO also support language specification, like on github (allegedly)
I always struggle to get the syntax right
I think even though it's a bit overkill in my example it's always a good practise to think about security and encryption when your program is somehow communicating with something external :D
21:44
@Uebertreiberman that wasn't the point I was making. For sure it's good to think about authentication. I just won't allow myself to implement multiple different approaches of increasing complexity at the same time while I'm learning all of them. I'd rather go with the base (but industry acceptable) level that I can focus on, and improve as I learn more
21:56
Is the answer to If visible light has more energy than microwaves, why isn't visible light dangerous? one of the most controversial from HNQ's history or was something done to the upvote/downvote count on the accepted answer?
I don't have any rep so I can't see if there's some vote-balancing battle going on and there are no comments
Huh, now it has two upvotes. I've swayed the imaginary battle in my head, commanding the charge blindly
water is a very important aspect of a microwave, sad that the first answer that mentions this is this one
@roganjosh timeline: +2/-0
cbg
@AndrasDeak yep, I used to graphitize wood at Uni with a single-mode microwave. That reaction is... scary. Just seems odd that hundreds of votes were thrown about but the accepted answer had 0 and was posted yesterday. The epic Physicist politics that I was imaging have been destroyed now, though :/
Throw enough microwaves at it to get to 550 deg. C as fast as possible from the moisture content and kaboom, thermal runaway
pandas is driving me a little bonkers, i imported this csv and want to make its date the only index, but even after I set the index there is this list of integers it seems is still an index
@roganjosh that's why accepts are irrelevant. They're down to the sole judgement of askers.
22:10
Oh yeah, but it still throws it right to the top for a lot of the 17k viewers, and there's plenty of voting going on, so I was just surprised to see that there was not a single vote on that particular post
@Skyler do you have an MCVE?
import pandas as pd

df_raw = pd.read_csv('GOOGL.csv')
df_raw.reset_index()
df_raw.set_index('Date')
print(df_raw.head())
@AndrasDeak my big loss from that project, actually, was not taking more note on the chapter of one of the books I had about mashed potato in a domestic microwave. Plenty of water content, but it never heats properly :'(
hmm, why isnt the backtick working...
GOOGL.csv

Date,Open,High,Low,Close,Adj Close,Volume
2009-05-22,198.528534,199.524521,196.196198,196.946945,196.946945,3433700
Because you don't need it for multi-line code blocks. Use ctrl + k with it all highlighted.
oh ok
This gets printed out
         Date        Open        High   ...          Close   Adj Close   Volume
0  2009-05-22  198.528534  199.524521   ...     196.946945  196.946945  3433700
the first column is an index of some kind that I'm trying to replace with Date
22:20
set_index doesn't work in place. Use df_raw.set_index('Date', inplace=True)
I also don't think you need df_raw.reset_index() at all
yea, reset was an attempt to get rid of that other index to no avail, I guess I'm a bit confused what it does in general
Considering I usually just throw reset_index() at my problems when I get to the point of punching the screen with Pandas, and hope for the best, I'm probably not the person to explain it. But my first comment should fix your issue.
yep
                  Open        High   ...      Adj Close   Volume
Date                                 ...
2009-05-22  198.528534  199.524521   ...     196.946945  3433700
ty
wim
wim
@piRSquared surprisingly legit
@Skyler welcome. Tonnes of Pandas methods work like this, make it a habit of scanning the documented arguments for inplace if you're ever stuck :)
22:27
the weird thing about the output from the previous method i tried is it doesnt even look like it made the date an index at all
was i adding the date as a second index or just outright doing something wrong and it ignored me?
It did, in a completely new object, then threw that object away
lolwat
This is not unique to pandas
so it made a full copy of the object (so that I coudl assign it to a new variable), since i didnt it tossed it
22:29
makes sense, I guess I sort of saw this when I started seeing warnings in Jupyter about chained assignments
And why shouldn't it? You've just described your own approach, it would be unreasonable for the library to guess (I'm not sure it even could) what you wanted from that setup
exactly, the tutorial i used often did a case where that'd be preferred
`df = df.dostuff() #not inline`
ok i thought at least that one line would be formatted for code
code for inline right
oh ok, that being said it doesn't look like you can have code snippets and written text in a multiline message, right?
yes
and when in doubt, practice in the sandbox
22:37
aye aye
and thanks for the guide
00:00 - 20:0020:00 - 23:00

« first day (3009 days earlier)      last day (2163 days later) »