« first day (3706 days earlier)      last day (1241 days later) » 

12:06 AM
Given a long list of json objects and a key-value pair to search for, what's the most efficient way of search through json in Python?
 
1:01 AM
Is there some trick to minimize the number of __eq__ or at least write fewer? I think in general you could compare the slots and dictionary of the object, that way you don't need to explicitly enumerate the members?
 
1:59 AM
@dataclass @Mikhail?
cbg
 
 
6 hours later…
7:47 AM
any1 knows scrapy
 
8:12 AM
@alkasm Yep. You can even have __prepare__ return a custom mapping and the class body is evaluated with its assignment rules. Allows for some nifty things such as duplicate attributes, when the mapping maps each name to a list of values instead of just one value.
 
yay, just noticed I've made exactly 5,500 answers here!
6
 
šŸ‘
 
8:30 AM
So another wacky question: I got the *** AttributeError: Can't pickle local object 'MyClass.__init__.<locals>.<lambda>'. This almost makes sense but the class doesn't have a dictionary and ` slots ` is set to empty. Heck I didn't even call __init__. Is there any way to figure out which .<locals> the python is complaining about?
 
You defined a function in __init__ and assigned it as an attribute. Don't do that.
If you just want to bind some parameters, use functools.partial.
 
9:08 AM
cbg
 
 
2 hours later…
user13415013
11:17 AM
:)
 
12:16 PM
buff = io.BytesIO(base64.standard_b64decode(item))
with gzip.GzipFile(fileobj=buff) as gz:
    decompressed_data = gz.read()
    decompressed_prod_details = decompressed_data.decode("utf-8")
How can I compress it back using above steps?
 
12:41 PM
do you know how to reverse each individual step?
 
1:00 PM
If you get rid of the GzipFile and use gzip.decompress instead, it becomes super trivial to reverse
 
 
1 hour later…
2:17 PM
morning cabbages, folks!
 
2:31 PM
@inspectorG4dget Good morning!
 
potato?
 
Banana
and potato?
 
banana, melon
still trying to tomato wake up. This coffee is defective, I think. I wonder if the company would be willing to stock cocaine in the office
 
I hear that is bad for the profit margin but, for the right person, they might
 
bahaha. I had to overhaul one of our core systems and now I'm fighting all the "growing pains" fires
 
2:42 PM
Ah, the old "now I see why they haven't overhauled this core system"
At least the fires will keep you warm
 
actually, the overhaul is what the system should have actually been since day 1. Basically, we've been running everything on duct tape, paper clips, and kerosene. All I did was build the F35 we've always had our eyes on. But now, we need to ETL that glider made of duct tape into that F35, all while providing air support to the systems that relied on the paperclips but are now forced to pivot their interface for a fighterjet
 
3:21 PM
Hmm, I might need to reinvent the struct module from scratch so I can unpack this three byte integer
 
@Kevin You've encountered a problem the developers of that module never considered?
 
perhaps this is something they're actively working on and will ultimately get Kevin'd on
 
Possibly they considered whether to natively support three byte ints, and thought, "people will need that only rarely -- they can just unpack the raw data and derive the value themselves"
 
signed 3 bytes?
 
Unsigned, big-endian
It's not all that difficult to slice out the three bytes and call int.from_bytes, but it doesn't play too nicely with my existing architecture
 
3:30 PM
is there anything else in the struct?
otherwise, just pad it and read it as 4 bytes.
 
Preceded and followed by a uchar, alas
 
In that case, find out who wrote the 3 byte int producing code, come up with an elaborate scheme to gaslight them, and hope the next dude maintaining it does it right.
 
>>> import struct
>>> x=b"hello"
>>> a,b,c = struct.unpack("!B3sB", x)
>>> b = int.from_bytes(b, "big")
>>> (a,b,c)
(104, 6646892, 111)
Not too bad, but my real code has about three layers of abstractions between where unpack is called and the first convenient place I can put from_bytes
Rather violates good encapsulation principles unless I get real creative
Challenge: come up with a sensible name for an integer type whose size is larger than "int" and smaller than "long".
 
3:47 PM
"smallong"
"medium"
By the way, how about just inserting a null byte before the 3byte data?
 
(Oops, int and long have the same size in the struct module. I meant "short" and "long")
 
... so that's just "medium" then, isn't it
 
inserting a null byte, as in struct.unpack("!BLB", x[:1] + b"\x00" + x[1:]), is about two layers of abstraction away from being convenient.
 
Streaming Advent of Code day 8: twitch.tv/davidism
 
4:04 PM
I kinda want to ask why on earth you have 3 abstraction layers dealing with binary data, but at the same time I really don't
 
Let's go up an XY level. C is quite capable of unpacking binary data into data structures of arbitrary complexity. You define whatever structs you need, and cast your char array into the top level struct. I would like to be able to do this in Python.
I want to be able to unpack bytes into a data structure that supports named attributes that may either be primitive types or nested data structures. The attributes ought to be ordered, although the order does not necessarily need to be accessible from outside the data structure class at runtime.
Perhaps the syntax would look something like data = fancy_unpack(the_bytes, fmt=named_struct(eth_src=ulong, eth_dst=ulong, ipv4=named_struct(ip_src=umedium,ip_dst=umedium,payload_size=ulong, payload=str)))
(Pretend that the ipv4 format uses three byte integers for its source and destination ip fields, even though it doesn't. Think of it as ipv3.)
Then I'll be able to do data.ipv4.payload and get the inner data of the ipv3 packet
 
auto_struct kind of seems promising? It doesn't seem to have 3-byte ints, but if you look a bit under the hood, I think you can pretty easily implement them yourself
 
4:19 PM
I'm having this error while trying to effect db migrations in the API service I wrote via Flask
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1049, "Unknown database 'rent-service'")
 
Open question: how to make the payload attribute consume exactly payload_size bytes from the source, given that we can't do something like payload=SizedString(size=payload_size) because payload_size is not bound to a name in the calling context
 
This happens anytime I call -

python manage.py db migrate --message 'initial database migration'
I already created the schema in MySQL so I was wondering why I would be gettting unknown database
In my config file where I set db connections
 
I could define a PascalString type that expects a four byte size header followed by string data, but in my real code, the size header does not always immediately precede the string.
 
class DevelopmentConfig(Config):
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'mysql://root:my_password@localhost/rent-service'
SQLALCHEMY_TRACK_MODIFICATIONS = False
In my init.py file -
 
Are mysql database names allowed to contain hyphens? I'm guessing yes, but it's worth checking
 
4:25 PM
Sorry about the code snippet I posted
Not sure how can I indent here
 
Eh, close enough
 
But in my config.py
https://pastebin.com/KT81mE7P
 
Consult tinyurl.com/urnzp7k for future reference
 
Thanks @Kevin
I have connected via sqllite and had no issues
But this is my first time connecting to MySQL
 
4:38 PM
umm,help?
 
Hmm, I don't see anything obviously wrong with your code
Then again I'm not familiar with mysql myself
 
4:59 PM
*sigh
 
Relatable
 
@MisterMiyagi Yeah, I kinda understand the error, except that I can't seem to find where, and heck, I'm using slots instead of dictionaries. After work I'll need to try to see if there is some automated way to find unpicklable attributes.
 
>>> import pickle
...
... class Nopickle:
...     __slots__ = 'potato',
...     def __init__(self):
...         self.potato = lambda x: x
...
... pickle.dumps(Nopickle())
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-320-846bae535b14> in <module>
      6         self.potato = lambda x: x
      7
----> 8 pickle.dumps(Nopickle())

AttributeError: Can't pickle local object 'Nopickle.__init__.<locals>.<lambda>'
I don't see why slots would make your class immune to the error
 
At some point it just becomes easier to just take a snapshot of the entire Python process memory
 
you could, you know, check each name in the slots before pickling and check which one is a lambda
 
5:06 PM
I assume that's a thing that virtual machines can do
 
Yeah, in a VM you can freeze the whole OS complete with RAM. Like hybernation.
 
Is... Is this what docker is
 
I mean, I can't find any member that is a lambda, but I do have a circular reference (aka each child holds the parent which holds a dictionary to the children).
 
Huh, I thought pickle could handle circular refs. I wonder if weakref would help?
 
Yeah I thought it could too. But yeah I can't find the member that is a lambda, but pickling the circular reference (the child's parent member) causes the error.
I did learn something new, apparently you need __getnewargs__ to make objects that implement __new__ pickable
 
5:12 PM
Maybe baseline pickle has caching magic that can detect cycles, but the magic goes away when you start defining your own serialization logic
 
Yeah I also thought that, but I didn't defined my own
 
Much like how __repr__ magically prevents infinitely long strings when a list contains itself
 
@Mikhail did you just look at the code or did you check the actual instance?
I.e. what does "can't find" entail?
 
Both, I tried saving stuff member by member, and looked at the code and the keyword lambda appears zero times in the whole massive project...
 
What are the contents of MyClass.__init__?
 
5:24 PM
@Mikhail I am failing to see how any of what you mention is relevant. Neither __slots__ nor dict nor cycles are to blame here. The lambda defined inside MyClass.__init__ is.
The name of the lambda is taken from the source code, so you can just look at where it tells you to.
 
Yeah, except I can't seem any lambda's the whole source... (not to mention that init)
 
Possibly the lambda is being generated by a builtin function or third party library. Then you could have a lambda object without ever typing the word lambda
 
Inspect the __file__ of the module that defines MyClass. Make sure you actually look at the code you are running.
 
Hmm, don't generator objects use lambda under the hood...? I forget
 
generated code usually uses its own names.
 
5:27 PM
Either way, pickle refuses to pickle a generator and gives a non-cryptic error, so I guess it's not specifically that. I still think a secret lambda could be lurking somewhere though.
 
The issue with generators is that their state is in drumroll <locals>.
CPython's pickle can't handle <locals>.
PyPy can, for example.
 
I want Python to be able to serialize arbitrary program state so we can steal call/cc from Scheme
I learned about reification a week ago and now I want to apply this hammer to every nail
 
Just dress it up nicely as pattern matching.
"Use the Kevin operator to match the current program state."
 
That's the kind of twisty thinking you need to effectively use call/cc :-)
 
So far, I only managed the other kinds of twisty thinking. :/
 
5:34 PM
I thought about pretzels yesterday. Does that count?
 
I hereby propose the kevpression operator :K
@inspectorG4dget Depends. Did you think about twisting them yourself?
 
let's say yes
 
That's definitely twisted!
 
in reality, I was thinking about eating a pretzel, while doing some yoga... so I'm not sure exactly how that works out
 
I'm more of a "sit on the couch and think about potatoes" kind of guy
 
5:38 PM
gets excited at the possibility of the salad crossover
 
Does a twisted mind count?
 
depends on the level of numeracy, I suppose
 
Hi, Can anyone help me solve a problem in pandas?
 
I think you might need a veterinarian
3
just ask your question, bud. If anyone knows, they'll speak up :)
 
user13415013
6:17 PM
:)
 
@gbade_ Have you created the tables in a specific schema?
 
@holdenSaysNoPhonies hello
 
6:42 PM
Huh, TIL that Grotesk in font names is just a reference to them being sans-serif. I can't help but feel the inhabitants of the 19th century might have taken things a little too seriously
 
7:01 PM
sans-serif was the Rock ā€™nā€™ Roll of that time, seducing youngsters to mischief with its daunting style.
 
7:24 PM
What a time to be alive :) Imagine showing them some Comic Sans!
Sorry, Comic Grotesk
 
7:50 PM
this channel is the most active python channel i've found so far!
wooo
 
@MisterMiyagi, I have a follow-up question from your comment on my question. stackoverflow.com/questions/65204185/… It seemed a little long for a comment. So it makes sense that importing from my submodule makes it an attribute of my parent package. Since that is the case, should I then be prefacing the submodule with a leading underscore, if I don't want the contents of that module to appear to be a public part of the api?
 
@dshanahan is your question a duplicate of stackoverflow.com/questions/21866036/import-a-b-also-imports-a?
In any case you might be better off with a manual, opt-in __all__ that contains string literals
you can also del names in your __init__.py that you don't want to have imported; I also vaguely remember that __all__ mostly affects star imports. So it won't necessarily affect what names are visible when importing your package.
 
8:09 PM
Hi
 
8:23 PM
can someone tell me the diff between running pytest --cov-report xml to coverage xml ?
 
@AndrasDeak not a duplicate but similar logic applies, thanks. I would really rather this be dynamic (scipy does it this, for example) but it does look like I will have to put a little more effort in. I believe things like Jupyter tab completion are based on __all__ as well so that is why I am taking an interest in what it identifies.`
 
I mean coverage run -m pytest and then coverage xml
 
@arielma pretty sure there is no difference
 
really? as the xml structure I'm getting for each one looks a bit different
 
@dshanahan right, that also rings a bell
 
8:44 PM
@Aran-Fey I raised question in the pyteset-dev githab page
 
8:57 PM
@dshanahan Yes, preface all internals with underscore. Doing that thoroughly means you can usually skip __all__ as well.
 
9:24 PM
Ok thanks @MisterMiyagi
 
The only diff I saw is written here:
https://coverage.readthedocs.io/en/coverage-5.3/

"Many people choose to use the pytest-cov plugin, but for most purposes, it is unnecessary."
 
10:05 PM
@MisterMiyagi This is exactly how I saw it being used, via Beazley's SLY lexer/parser.
 

« first day (3706 days earlier)      last day (1241 days later) »