« first day (3141 days earlier)      last day (1802 days later) » 
01:00 - 15:0015:00 - 00:00

1:55 AM
I've figured out how to post off topic comments without having killjoys flag them for deletion: also post something helpful/useful in the same breath. Well, should've done that to begin with :P
@piRSquared Thought of any canon ideas lately? I've seen a lot of people be like "Yeh I can do xyz in SQL but new to pandas"... wondering whether a canon mapping some basic SQL constructs to their equivalent pandas syntax wouldn't be worth writing...
 
2:31 AM
Hi @cs95 saw the cool tshirt and mug on your SO profile? How do you get one of those?
 
you basically slog to 100k, pissing off a horde of angry, irate people in the process, then wait 6-8 weeks to get mugged.
but the key is 100k
 
Aah thanks, like the pun mugged :)
 
 
1 hour later…
3:59 AM
Table describing arguments to df.to_csv in the context of situations they're useful... yay or nay?
 
@cs95 what a sceptical point of view... :)
 
@JonClements the goal of that description was to explain the process and discourage potential FGITWers at the same time :P
 
4:22 AM
@cs95 umm... most of that fine... probably the most important thing anyone would ever want is index=False...
 
I listed every argument I've seen a question on here
there are more that I've ignored, but these are the ones I thought most important
 
it's all good :)
I've seen people write df[['col1', 'col2']].to_csv(...) for instance
 
4:39 AM
A kitten dies without fail everytime someone writes a hack
 
@JonClements Thank you for answering on my question. I have upvoted your answer. But want to give the New contributor a chance of accepted answer, as an encouragement. Please pardon me if I offended you.
 
Nice try, but you'll have to do more to offend this pupperino
P.S.: that was not meant to be encouragement ^ :P
 
@cs95 Is ya messaging me?
 
@JafferWilson errr? Thanks?
@cs95 "pupperino"? :p
 
A pupperino is the male version of a pupperina, also known as a pupper, it is the formal name given to a floofin woofer or a mleming potat
 
4:45 AM
So... it's like a doggo?
 
Here comes an A M G E R Y boi AKA @thefourtheye
 
omg... my evil twin... :)
 
@JennaSloan No, a doggo is a boofer. A pupper is, well, less boofy and more goofy
 
@cs95 I think the only thing I could add to what you're trying to put into that document is I'd add a column saying "why" you'd want to use it
a kind of "you'd use this option because...." thing
 
Hmm, it's more a matter of "use this if you want to achieve this", rather than "I recommend using this because..."
do you see any place where adding a "why" would make sense? I can only think of compression...
 
4:48 AM
yeah.... why wouldn't I want an index column written to a CSV file for instance?
 
here's the context, for your ref
whoops, wrong link. fixed
@JonClements The only situation you'd not want to write an index is if your DataFrame doesn't have an index (i.e., it has the default RangeIndex), do you mean that?
 
yes - that kind of thing... don't forget that in the bigger world - you/I are just processing data and needs to present it to a system further along the line or something...
that means, if it's expecting certain columns named in someway, having the writer, write its index is going to mess things up
 
@cs95 A wild cute dog appears...
 
@thefourtheye woof :)
 
You're giving me some nice ideas... but a table column is severely restricting, so I'll add these as footnotes below.
 
4:53 AM
woof woof woof, woof?
 
Bow wow bow bow wow...
 
boof woof, bow bow, woof woof bark
@cs95 if you're going to try and make it useful... all I'm saying is try to explain why and in what cases you'd want to use those options... otherwise, they're just in the documentation anyhow
 
Thank my stars, I'm working with markdown syntax so I can always add superscripts and enumerate
 
Its election result day in India :-)
 
@JonClements fair point. I'm only wondering if I should add a single line for the "why" part or make an Aaron Hall answer underneath the table :P
 
4:57 AM
@cs95 there's a middle ground :)
 
I've become bad at finding it but I think I know what I need to do B)
Really appreciate the help Jon :)
 
@cs95 take a step back from it, then read it as though you didn't know anything about it... then with your experience, take each row as a "I can use this" but I'd want to do that because
@thefourtheye better luck for you then we're having here :p
 
sage advice, will do
 
@JonClements Looks like there is going to be no change here :-/
Entire office is glued to the TV... Its a rare sight :D
 
@cs95 sorry... didn't mean to knock your effort or anything... but you asked for input... you're doing great... thank you
 
5:03 AM
not at all, you've given me exactly what I needed and more
Stack Overflow will have a better answer now thanks to you :P
 
anyway... gotta run for a bit... by "run", I mean actually get some sleep for once... bbiab
 
(two better answers, I'm going to do the same thing for "Read from a CSV")
 
@thefourtheye rbrb woof
 
5:51 AM
@JonClements I've more or less fleshed out the why/because for the arguments I thought needed it. Finished result. I think this is good but let me know if something is amiss.
 
6:27 AM
@JonClements rbrb Puppy. Sleep well...
 
 
1 hour later…
7:40 AM
salt = uuid.uuid4().hex
hashed_password = hashlib.sha512("1234".encode('utf-8') + salt).hexdigest()
I am getting this error:
TypeError: can't concat str to bytes
In hashed_password = hashlib.sha512("1234".encode('utf-8') + salt).hexdigest()
 
I'm running cmd commands using the subprocess module.
Codes;
# -*- coding: utf-8 -*-
import subprocess
result=subprocess.check_output("dir",shell=True)
print(type(result))
print(result)
when I use the `dir` command, it gives error because of the Turkish file names.
Type of output data: `run.std_output`
Error:
https://notepad.pw/share/3yq0q8jzv





How do I resolve the issue of Turkish character in output?
 
8:07 AM
@QuicoLlinaresLlorens .hex gives you a string. Use .bytes instead.
 
I am doing this now
bcrypt.hashpw(password.encode(), bcrypt.gensalt(10))
but it returns a binary
like binary('hashedPassword')
 
I've never used the bcrypt module, so I can't help you with that
 
@HibritUsta That traceback looks suspicious... what the heck is C:\Users\Mustafa\AppData\Local\Programs\Python\Python37-32\lib\site-packages\run\__init__.py and why is subprocess using it?
 
cbg
@PM2Ring Well, just saying so he knows how to do it if he isn't searching, because it's not quite clear, maybe he wanted sub...
@PaulMcG I used that several times before... regex101
 
8:53 AM
@HibritUsta this is a very common error, you need to run it in a place where your system is configured to support the character set you are trying to use
what exactly that means depends on your precise platform
your error message looks like Windows, I like to recommend to get rid of that if you can but a quick workaround might be chcp 65001 in the CMD window where you are running Python from
except your code page is apparently 1254 so maybe try that instead
 
Hi everyone, I'm trying to send emails using the library smtplib. But I struggle with the argument to put in smtplib.SMTP(), I tried many things. I want to do it from my professional outlook,is it possible that somehow it's blocked?
 
@Mez13 your question doesn't really make sense, how do you mean "do it from my professional outlook"? SMTP is very commonly blocked from consumer IP addresses but you can submit email to a provider you have a relationship with and of course provided you log in with a valid password etc
 
I'm not sure to get your answer. I just say that I'm not using a public domain like Gmail
 
do you have a server name, port number, user name, and password which you know to be working otherwise? or are you randomly hoping to find an SMTP server which will relay traffic for you, because you don't have an ISP who provides you with one?
 
@tripleee Perhaps Outlook in Office Professional
 
9:07 AM
@roganjosh I know what Outlook is but the OP is clearly attempting to send email from Python
 
@tripleee Of course I do
 
so can you show us what you tried and how it failed, with pertinent details like the password blotted out?
 
I used the "ping mail.domainename.com" to get it
 
and you have an account at "domainename.com"?
guessing the server name is definitely the wrong approach, consult their documentation for what to use
 
I got an error here smtplib.SMTP(SERVER), where SERVER is what I got from the cmd line I just sent you
 
9:10 AM
the fact that "mail" occurs in the server name does not by itself imply that the host is running an SMTP server for outgoing email, you have to know which server to use (mail might be solely for external email coming into the organization, or a number of other arrangements)
and there is no need to ping anything to obtain the IP address (ping is the wrong tool for that anyway, though it sometimes works) -- you can just use the server name
 
Yeah, I also tried a number. Actually I tried many other arguments according to the methods found on internet, non works
 
again, the server owner needs to tell you what to use exactly, we can't guess any better than you can
for example, here is the documentation for fastmail.com fastmail.com/help/technical/servernamesandports.html
you should be able to find something similar for the service you are using, perhaps inside their private intranet or something if it's a closed organization
there are two common approaches -- one is to be inside the firewall and use standard SMTP on port 25, possibly without any authentication, but more and more organizations are moving to managed email solutions where you can connect to the outgoing email server from anywhere in the world, but then of course you need to authenticate successfully
port 25 is generally blocked outside of safe environment like a company intranet
so typically you want port 465 or 587 for Google/Outlook 365-like services
is this making sense to you?
 
@tripleee Thanks. I'm gonna investigate on it
 
9:26 AM
python 3.8 assexp strikes again :) stackoverflow.com/questions/56271912/…
 
rbrb
 
Another related question. Is it possible to send emails without using the smtp protocol?
 
9:42 AM
@Mez13 i was struggling with this few days back, turned out the host and the port was not default, do you have the host and port from your local IT? (Sorry if i have missed anythhing)
Also instead of the hostname I gave the ip from socket.getaddrinfo() and it worked :)
 
cabbage
I wanted to know how the python default "sum()" method works , it returns the sum of an iterable . so I went to the Builtins.py file where this function is located and to my surprise this function has a "Pass" line inside. How does this function work?
There is no logic defined at the function definition
 
I doubt it's defined in builtins.py, whatever that file is
 
That entire builtins.py fie has no logical code whatsoever
How does one trace to the original function definition of python's built in functions
 
@anky_91 You mean that you use the results of socket.getaddrinfo() as an input of smtplib.SMTP()?
 
9:57 AM
@Mez13 yes, not the entire info, just the ip you get
use that as host
and the port as provided by your IT
i use outlook too bdw
 
@vaultah Okay.. so the actual code is in C?
 
Yes
 
Is python just a wrapper over C?
 
Python is implemented in C
 
@anky_91 Actually, I get something quite blurred starting by "AddressFamily.AF_INET" and finishing by the input I gave. So not sure about what to use
 
10:00 AM
okay..
 
@Anarach If you can read C, take a look at github.com/python/cpython/blob/…
@Anarach Definitely not. But the standard Python interpreter is implemented in C.
 
@Mez13 exactly, use the one after that (one which is inside the tuple)
 
But what there is inside the tuple is exactly what I gave to the function. What's the point?
 
[(<AddressFamily.AF_INET: x>, x, x, '', ('856.56.2.14', 45))]
'856.56.2.14' <- this one
this is a dummy
 
Yeah. '856.56.2.14' was already my input
 
10:08 AM
hmm, any errors?
 
@anky_91 [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
 
interesting i did it like smtplib.SMTP('856.56.2.14', 45,timeout=120) and it works for me. i initially gave a wrong domain, its better consult the smtp team for a proper hostname
 
@anky_91 Thank you very much. You're right, it probably comes from the domain name, I will doucle check it
 
@Mez13 some services allow you to use e.g. a REST API to submit messages, including Gmail and mailchimp
 
Do you know if there is a way to send emails without smtp protocol ?
 
10:13 AM
@anky_91 there's no need, SMTP will do the getaddrinfo() if you pass in a host name
@Mez13 that's what I just answered in the comment above yours
 
@tripleee okay. :) for some reason it didnt work out back then. May be i didnt try putting the hostname later. (Y)
 
of course the REST API they offer is just a convenience wrapper, it ends up using SMTP to send the messages eventually
 
@tripleee Ok thanks. Actually, I want to send emails from my outlook mailbox, but I think there is no smtp. We use microsoft exchange
 
@Mez13 there is a proprietary Microsoft protocol called MAPI
not sure what you mean by "from your Outlook mailbox", you can basically save a copy of the outgoing message there but there is no way for a mailbox to send anything
 
And im sure MAPI stands for PITA :P
 
10:16 AM
@ParitoshSingh yeah, when you read "Microsoft priorietary" you should translate to "run fast, don't look back"
 
I don't know enough stuff about this to follow you guys
 
haha sorry, ignore what i said. it was a tongue in cheek joke :P
 
@PM2Ring Yeah I saw that thanks to vaultah
 
The package mailR, in R, looked much easier, haha
 
@Mez13 you can make Python call any code you want, there is SMTP for submitting a message to a mail server or HTTP to call a web server and give it a message to pass to an SMTP server on your behalf, or some Microsoft abomination which more or less amounts to the same thing
 
10:19 AM
And you can usually find libraries that abstract away most of the headaches associated with the process
 
rdocumentation.org/packages/mailR/versions/0.4.1 that seems to require basically the same information, the example has smtp = list(host.name = "aspmx.l.google.com", port = 25),
 
@tripleee Yeah sure, in theory. The fact is that something is going wrong there
@tripleee Yes, and it works. That's a big difference, haha
 
@ParitoshSingh there's a package called yagmail which probably looks more like the R one
@Mez13 if you can't connect to the server from Python then you can't from R either
 
@tripleee yagmail just supports gmail i guess
:)
 
@anky_91 no, not at all, Pascal regularly posts answers on Stack Overflow to regular email questions recommending his package
(I tried to remember the name of the package here last week IIRC, now I have tried to memorize it properly :-)
 
10:22 AM
@tripleee i see, the docs say "yagmail is a GMAIL/SMTP client that aims to make it as simple as possible to send emails" so i didnt explore much
:-)
 
yeah, that's slightly obscure
 
Aha, thanks for letting me know. :)
 
@DeveshKumarSingh four close votes for "primarily opinion-based", what am I missing?
 
@Anarach True, but my link is to the line number where sum is defined. ;) But yeah, I didn't see vaultah's link when I posted mine.
 
10:56 AM
cbg
So I'm looping over a big list, building a dictionary and then appending it to a list.
After ~1mn iterations, list is unable to fit in RAM.
I was thinking of appending this to something like pickle file which will be on disk rather than on memory.
Suggestions?
 
What are you planning to do with this list? What are you creating it for?
 
So, the end result is a list of many dictionaries?
 
@ParitoshSingh Yep
@Aran-Fey To push in Kafka after merging it with another dataset
 
i'd Address Aran's question in case there's a way you could avoid making that much data. If not, then sure, make a bunch of dictionaries, then just write to a file, as say json. wouldn't even bother with pickle at that point.
 
@MayurBhangale What's in the dictionary?
 
11:01 AM
Unless you need to store more complex python variables themselves, a json should serve your purposes just fine.
 
How do I continously dump dictionary in file?
 
I don't know Kafka, but it seems to be some kind of pipeline thing... So maybe you could use a generator instead of a list? You don't need the whole thing in memory, do you?
 
Also, csv's are another option if all of the dictionaries are the same keys as each other
 
If Kafka operates on files, then sure, go ahead and dump the thing to a file. But if Kafka doesn't operate on files, then saving the list to disk doesn't solve your problem (because you still have to read it into memory to work with it).
 
I have to do merge operations later (for which I'm considering using Dask)
 
11:05 AM
CSV or jsonlines which is becoming more prevalent. But I'm still curious what's in the dictionaries because it may not be appropriate if it's not, e.g. primatives
 
Today's thematically relevant puzzle is rated dangerous / 10:
# Provide a chunk of data that sets the global `winner` variable
# to True when unpickled

import pickle

winner = False

data = b'...  # your data here'

pickle.loads(data)

assert winner, "You didn't toggle the variable"
print('You win!')
 
@MayurBhangale Does each dict in the list have the same keys?
 
Yes
I'm going ahead with generators as suggested by @Aran-Fey because I obviously can't read it back into memory later
 
@MayurBhangale Ok. Then it will be much more compact to save it as a CSV. And you may be able to fit the whole thing in RAM if you make a list of tuples, rather than a list of dicts.
Each dict uses about 300 bytes more than a tuple, plus extra RAM to save the pointers to all those repeated keys.
 
Fair point
 
11:16 AM
@MayurBhangale If you can make your code work with a generator of dicts, that may be simpler than my suggestion.
Roughly how many items are in each dict?
 
4 keys with varying sub objects.
I'm lazily appending it to dask dataframe and then merging
And its much faster. Way too fast.
 
df[df['A']>0] will return a dataframe with rows where 'A' is greater than 0. How do I do it if I want to have 2 criteria (e.g. 'A' and 'B' has to be greater than 0)?
 
@Pherdindy What have you tried?
 
I tried df[df.loc[:,['A','B']]>0]
But returns a dataframe with NaN values popping out, but not exactly what I want
 
You want to go along with the way that you would specify multiple conditions in regular python, I don't think loc can do something fancy like that
 
11:26 AM
 
In other words, you need to specify the condition for each column separately, and the problem should be easily searchable
 
I'll come up with 2 series of True and False values
On both 'A' and 'B'
How do I combine them into 1 series though
 
NaN are nothing but less then or equal to zero, what do you want to do with them?
 
@anky_91 More of like include rows of the dataframe where the conditions are met only (e.g. df['A']>0 and df['B']>0)
 
@Pherdindy you're getting closer
 
11:32 AM
@Pherdindy assume if A meets criteria, and B doesnt, what happens to that row?
 
It will not show up
 
@anky_91 I'm not sure what you're suggesting because it doesn't fit with the definition of how nan compares to other values
 
@roganjosh i am trying to figure out what is desired when 1 column is True and other is false. :)
 
df[df['A']>0 == df['B']>0] ?
That should work right?
 
I'm not sure why you haven't searched for the solution yet. I'm not trying to be awkward (well, maybe I am) but pointing out that I'm pretty sure you could get a solution to this problem without just turning to us
 
11:37 AM
I just thought it would be easy for you guys :P lol my bad force of habit
 
@roganjosh i see what you mean now, i was meaning something else(bad way of writing the sentence may be) :P i meant if you implement( df[df[['A','B']]>0] , you would get nans where the cond is false)
 
Ah ok, I wasn't aware you were referring specifically to that (hoped) functionality sorry. Crossed wires are untangled :)
 
yep. :)
 
It doesn't work I get an error ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
array_1 = df['A']>0
array_2 = df['B']>0
array_3 = array_1 == array_2
df[array_3]
This is the one that works for me anyway. Thanks
 
I don't understand how you came to that
df = pd.DataFrame({'a': [1, -1, 0], 'b': [1, 1, 1]})

df = df[(df['a'] > 0) & (df['b'] > 0)]
 
11:49 AM
Ohh you need parenthesis. Sorry my python isn't so good also
Yea I usually forget that bit
 
cbg
 
cbg
 
12:13 PM
@Aran-Fey well, i learnt something today im not entirely sure i should have :P
 
You're looking at it the wrong way... it's a bad thing, but it's good that you learned about it :)
 
whats amusing is that i spent half an hour trying to figure out eval before finding out there's something called exec, and the two things are different
 
It can be done with eval, but exec is much more convenient, yeah
 
hm, now im curious
spoiler Wanna peek over ?
okay, i got it. apparently no, i did it with an eval then :P
 
12:29 PM
I roughly understand why this works but I'm not clear on why the protocol allows it
 
TL;DR: The protocol doesn't give a yam
> Those pickles are very dangerous pickles. I literally can’t begin to tell you how really dangerous they are. You have to trust me on that. It’s important, Ok?
^ lol
 
Pickle was designed to allow a program to persist its own state info between runs. It was never intended as a general way to pass data around, and it has minimal security safeguards.
 
And, to nobody's surprise, people started using it for all the wrong things
 
To be clear, I'm not saying "given the choice between a feature-rich module and a secure module, we should choose security". I'm saying "I don't know what feature we're getting in exchange for giving up security"
 
yeah slight tangent, i remember ned wrote a blog post on the subject. There's definitely atleast some steps like disabling access to builtins that could be done
At the end of it all, apparently there was no good way to keep all exploits out. link
 
12:37 PM
Maybe an illustrative example would clear up my confusion. What kind of object would you need to pickle in order to get b"cos\nsystem\n(S'echo hello world'\ntR."? Or any other bytes that contains a builtin function call.
It's not like pickle.dumps(math.sin(0)) will give you b"cmath\nsin\n(0\ntR. Or whatever
 
class Evil:
   def __reduce__(self):
       return (os.system, ('echo hello world',))

data = pickle.dumps(Evil())
 
oh, note that this is a more "Readable" version of the bytestring
til about reduce, pretty nifty
@Aran-Fey would this work out of the box? more specifically, curious about the lack of os import call
 
No, you do have to import os. Pickle is dangerous, but it's not magic :P
 
oh, i think i know an issue in your solution
can you take your pickle generated strings and test them in a fresh session?
 
No, as I found out
 
12:45 PM
@Kevin The "why it's hard to implement a safe deserializer" problem summed up:
*) You can't deserialize code for obvious reasons
*) Because you can't deserialize code you have to allow references to existing classes and/or functions
*) Because an attacker can reference existing code, there's no telling what they might do with it
 
I realised from this puzzle that my understanding of pickle is so out of whack that it's a little embarrassing
 
Allowing a reference to an existing class doesn't seem all that dangerous. If my loads call returns a reference to os.system, that's fine. If my loads call calls os.system, that's bad.
 
Python 3.7.3 (default, Mar 26 2019, 21:43:19)
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.loads(b'\x80\x03cposix\nsystem\nq\x00X\x10\x00\x00\x00echo hello worldq\x01\x85q\x02Rq\x03.')
hello world
0
^ works fine
 
@roganjosh fwiw, i think "aah run away taboo avoid!" is good enough to get by :P I got to learn some pretty cool things from this exercise though
ah i see, thanks.
 
I tentatively accept the justification of "__reduce__ can make loads execute arbitrary code, and this is necessary because sometimes this is the most efficient option or the only option for serializing legitimately useful classes"
I notice that the pickle docs say "We will show, however, cases where using __reduce__() is the only option or leads to more efficient pickling or both." But, uh, I can't find the section where they actually do this
 
12:49 PM
Aran mentioned that its not the only interface that you can use. I ended up utilizing something called setstate without realizing it.
And so, reduce is not the only vulnerability here.
 
@Aran-Fey I am so confused by this
For a setup:
class MyClass:

    def hello(self):
        print('hello')

a = MyClass()
print("The instance:", pickle.dumps(a))
print("The class:", pickle.dumps(MyClass))

with open('some_file.pkl', 'wb') as outfile:
    pickle.dump(a, outfile)

with open('some_file.pkl', 'rb') as infile:
    rebuilt = pickle.load(infile)
    rebuilt.hello()
If I restart the kernel and do:
import pickle

with open('some_file.pkl', 'rb') as infile:
    rebuilt = pickle.load(infile)
    rebuilt.hello()
I get AttributeError: Can't get attribute 'MyClass' on <module '__main__' from 'C:/Users/Josh/Desktop/github/untitled3.py'>
 
Is there a good style link or something to see when x or __def or something is used? Right now mine looks like....well...cabbage.
 
Am I being exceptionally dumb here ('yes' is perfectly acceptable btw)?
 
@Kevin True, but it's not like the attacker has to run dangerous code during the deserialization process. They might just as well make you deserialize a FluffyTeddyBear that has its cuddle method shadowed by Dragon.breathe_fire. If you're not careful, it's still a risk
 
@roganjosh The latter half of the section docs.python.org/3/library/… talks about this a little bit
"when class instances are pickled, their class’s code and data are not pickled along with them."
 
12:53 PM
Oh man, what an oversight on my part. I've only ever thought of pickle in the context of classes because they're not easily serialized into JSON
@Kevin thanks for connecting the dots for me
 
On a related note, I have had people pickle their pyparsing grammar, finding that unpickling their parser is faster than defining it explicitly. The hitch was when parser expressions had attached parse actions (parse-time callbacks). This was "resolved" by using dill instead of pickle.
 
@roganjosh The difference is that MyClass doesn't implement a pickle protocol, so pickle falls back to pickling the class and the __dict__ of your object. When you try to deserialize it and the class doesn't exist, it fails. My Evil object on the other hand never pickles itself, it only pickles a reference to os.system and an argument tuple.
 
@Aran-Fey True. I wonder if this is possible without exec or eval, since function objects don't get directly serialized.
 
@Aran-Fey Hopefully I've given you a perfect anecdote on why the misuse comes about because I've made way too many assumptions about how it works, even knowing the security issues
 
More formally, how would you construct a bytes object that does this:
import pickle
data = b'...'
obj = pickle.loads(b)   #this does nothing bad
obj.frobnicate()        #this executes `os.system("echo Hello world")` or a similar sinister payload
 
1:03 PM
Like that basically
 
:( dpaste is blocked for me
 
TL;DR:
teddy = FluffyTeddyBear()
teddy.cuddle = Dragon().breathe_fire
data = pickle.dumps(teddy)
 
While it is a potential security issue that the attacker can construct a FrankenObject out of the classes' methods defined in your program, it doesn't give them a whole lot of leeway, since they'll just get AttributeError: Can't get attribute 'FluffyTeddyBear' on <module '__main__' (built-in)> if they try to use a method that your environment doesn't have
 
i can pickle a class itself though, right
 
@biggi_ I don't know about a style guide, but did you see the section about Private variables in the tutorial? There are a few related questions on the main site. You should never make up your own names of the form __mything__ that begin & end with double underscore, they're reserved for the language implementers.
The general rule of thumb is to use names starting with a single underscore if you don't want users of your module to mess with them. Leading double underscore is only needed to stop name collisions between a parent class & its child classes.
 
1:09 PM
nvm, realised my mistake
 
@Kevin Passing arguments to the method is easy too:
teddy = FluffyTeddyBear()
teddy.cuddle = functools.partial(exec, 'print("I\'m running arbitrary code")')
data = pickle.dumps(teddy)
The attacker really just needs to find a pickleable object that you call a method on
 
@PM2Ring thank you! A lib I was working with has all their defs as __defname and I wasn't sure why
 
oh, sorry missed your msg in that mess somehow.
 
@Aran-Fey Ok, now we're getting somewhere. If you define a class whose __reduce__ is return (type, ("Payload", (object,), {"frobnicate": functools.partial(exec, 'print("I\'m running arbitrary code")')})), then we get something that executes the payload when you run frobnicate.
 
@biggi_ take a look at this too
 
1:13 PM
Now let's try to do it without exec.
 
Without exec, eval, compile, os.system, os.popen, subprocess.Popen and types.FunctionType, you mean?
 
huh, compile? o.o
oh wow, til.
 
I suppose compile is the least dangerous of the bunch... because you still need to do something with the resulting code object
 
@biggi_ That can be a valid thing to do if those classes are designed to be inherited from. OTOH, the writers of that code may be misguided. ;)
 
Hmm, I don't think there is an interesting formulation of my problem that doesn't boil down to "achieve arbitrary code execution without using any method that gives you arbitrary code execution"
 
1:20 PM
@ParitoshSingh See Antti's mammoth answer here: stackoverflow.com/questions/2220699/…
 
@PM2Ring it's the PyLidar3 library. So I think it might be the first case, but I don't know much about inheritance tbh
 
if I check the dtype of a dataframe column, and get back object. That means string? Not integer right?
 
so it seems compile itself does not execute code then, just preps it?
 
yeah
it's not really dangerous unless you manage to somehow assign the code object to a function's .__code__ attribute or something
 
@Violatic no, it means it was something complex. Usually at that point, i'd say access a single value and check its type
 
1:23 PM
@PM2Ring I like this concept of "avoiding name collisions" better than "privacy" for __ variables. They are just such a pain in the neck when using pdb.
 
@Paritosh Singh I checked the values for the entire column, and they are all integers
 
usually you get objects if you insert containers in a column, such as lists or dicts.
 
So try df['some_column'] = df['some_column'].astype(np.int64)
If that fails then you know why Pandas fell back to the object datatype
 
yeah that worked fine, now my conditions are all okay too
I think it has something to do with the fact I pulled the data from a SQL DB
 
now im curious what it was doing before
ah
 
1:25 PM
and pypyodbc keeps the datatypes from the DB?
 
@Kevin For the record, I do believe it's possible to implement a safe deserializer... You just need to have a whitelist/blacklist of safe/unsafe callables (i.e. functions and classes), let the deserializer know that you're expecting to get a FluffyTeddyBear and not something else, and verify that the deserialized object has no attributes that a FluffyTeddyBear shouldn't have. Easy, right?
 
@PM2Ring that was an awesome read, thank you
 
docs.python.org/3/library/pickle.html#restricting-globals gives some guidance on whitelisting callables
 
@ParitoshSingh Don't thank me, thank Antti. :)
 
Aye, ^^ thanks Antti. He's got my upvote already :)
 
1:30 PM
Blacklisting looks a little more complicated since you have to override Unpickler.find_class(), and anyway trying to forbid only and every arbitrary code executor in the standard libs is rather like plugging all the holes in a block of swiss cheese
Just forbidding eval will probably stymie 99% of script kiddies, at least.
 
eval and exec together probably cover every "simple" way to get arbitrary code to run i'd imagine really. though Aran did mention os calls, and those can be a major pain too.
 
I've actually set out to write a safer alternative to pickle a while ago, and what I did there was to have a whitelist for functions (iter, ...) and a blacklist for classes (subprocess.Popen, ...). The assumption was that calling a class generally won't do something dangerous.
 
Whenever I need to write a custom serializer, the final design almost always includes "... And then, pass that data structure of built-in types to json.dumps..."
 
@PM2Ring looks like they use the _ for class methods github.com/lakshmanmallidi/PyLidar3/blob/master/PyLidar3/…
 
@cs95 the sql2pandas might be interesting. Only thing I have up my sleeve is a rehash of something I did for defunct SO Docs but it is specific to my chosen occupation. Basically, I want to show how to efficiently construct some commonly used analytics. It wouldn't be a high flyer but it would be helpful.
 
1:44 PM
@Kevin Can't blame you. Writing a (de-)serializer was waaay more complicated than I initially thought.
Once you start trying to correctly preserve references between serialized objects, you enter a pretty deep rabbit hole
 
I've gone down to the bit level and used struct once or twice, when I needed a memory-efficient serialization protocol. But I can never remember how endianness works so it's a fraught process
I could probably do circular references, but I won't do it for free.
 
Does it matter endianness it works? You just need to be consistent with it, no?
 
Maybe, but I've got to blame my broken prototypes on something, and endianness is a nice high-visibility scapegoat
 
If you dump something in little-endian, you need to load it with little-endian... it's like when you dump something with pickle, you have to load it with pickle and not with json. You don't need to know how pickle works
Ah, fair enough then :D
 
@biggi_ Ok. At a quick glance, that code is using single _ correctly, and it doesn't use leading __ at all. So when you use that module you shouldn't normally call any method that starts with a single _, or read or write any attribute that starts with a single _.
In C, the endian thing is an issue because using native endian is always faster, but in Python the extra overhead of converting between Python objects & C integers probably swamps that time difference.
 
1:59 PM
And then network latency swamps that, and the green grass grows all around
 
2:18 PM
cabbage
 
2:37 PM
Hey there!
If I define a function (someFunction (f)) that takes a lamda function as argument (lamda x,y: ...) can I read the parameters of the lambda function in my other function?
If so, how?
 
Define "read the parameters".
 
@piRSquared I found this while doing a little research... seems like it's already there. Going through it now but this page is more or less structured the way I imagined how I'd have written it
 
Well look at that. I should remember that to link to.
 
Can I
1) Count the number of arguments
2) Select the x,y variable (applied to my sample given)
 
def someFunction(f):
    parameter_names = f.__code__.co_varnames
    #co_varnames also contains the local variables of a code object,
    #but lambdas can't create their own local variables (... I think),
    #so this should only capture its parameters.
    print("This lambda has the parameters:", parameter_names)


someFunction(lambda a,b,c=1: a+b+c)
#result:
#This lambda has the parameters: ('a', 'b', 'c')
Possibly you are now thinking "great! Now I have the number of arguments. But how do I get the values of a and b?". But this is an incoherent question because the names a and b are not bound to any values in any existing scope.
 
2:46 PM
I see the problem, I'll try another approach then, thank you!
 
Parameters are only bound to values when you call the function. You haven't called f, so a and b have no values. If you are planning to call f, then you already know what value a and b will have, because you're calling f with those values.
 
this is an interesting question: stackoverflow.com/questions/56277404/…
why does recursion stack fills up here?
 
cbg fellow pythoners. qq. Suppose I have a df like:

import pandas as pd
tmp = pd.DataFrame({"EUR": {"A": 1, "B": 2}, "GBP: {"A": 3, "B": 4}})
assign_this = {"A": 3}

Now I want to assign the latter to EUR row say:

`tmp.loc['EUR'] = assign_this`

This will set the column `B` to `NaN`. What is a good way to assign a dict without looping? Where keys are column names. And the dict is variable size. Without loops :)
 
01:00 - 15:0015:00 - 00:00

« first day (3141 days earlier)      last day (1802 days later) »