« first day (4426 days earlier)      last day (746 days later) » 

10:14
what would you call a package that supports using pickle + deque and pickle + queue, but in multiple different configuration/ways? I wanted to call it PickleQueue but then it might be confusing as to how it work (since it would also support Deque too)
I guess I could call it PickleGalore but don't know if it fits since I'm not using Pickle with more than two stuff (aside from the dozen of different configurations it supports)
TBH even having followed this topic here I'm not entirely sure what it's actually supposed to mean,
You pickle stuff and then... what? put it in a deque?
@MisterMiyagi basically I put either a queue or a deque inside a pickle, but done so in different configurations. I didn't thought of doing the reverse though
the class for it is mostly done, but I was just wondering about a name, since it supports more than just Queue...although I guess Deque is a type of queue, but yeah
What does the module do, exactly? If pickling a queue is all it does, it seems pretty useless - pickle can already do that.
@NordineLotfi What do you mean by "put […] a deque inside a pickle"? As in, pickle.dumps(some_deque)?
Technical confusion notwithstanding, PickleGalore sounds awesome.
@Aran-Fey I did say multiple configurations, but I guess this needs to be known in order to choose a name hmm. Basically, I made this as a possible alternative (albeit slower) to the socket module. It's basically mostly for communicating between processes/threads
@MisterMiyagi Thanks, I thought it looked better than the other one too :)
I thought of other names like PickleDB or PickleMulti but those probably don't fit here
I'm implementing different schemes/ways to handle temporary pickle files, persistent files, locking, lock delay/timing, etc. That's what I mean by different configurations.
10:31
There's an awful lack of pickled things in that list. :P
That's true actually. I mean most of the "customizations" are still working toward/around Queue/Deque + Pickle, but I see what you mean
It sounds like the primary purpose of the module is interprocess communication, in which case it would probably be a good idea to put that in the name. Who cares that the module internally uses pickle to send data from one process to another? That sounds like an implementation detail
I also thought of supporting stdout/stdin for processes, so maybe PickleSomething isn't really the right mindset.
@Aran-Fey you're right
"mpqueue" maybe
oh, that's nice yeah. But mine also supports threads/subprocess though, do you think it won't be confusing?
10:36
Could be
hmm, naming things is hard
FWIW, deque implies to me that it is in-memory of a single process.
I see, didn't think about that. I do plan on trying to add support for in-memory stuff too (since I mostly did this to work with files instead of in-memory) but that's for later maybe
but naming things is really hard
I did look at github.com/noxdafox/pebble and github.com/BrianPugh/lox which does tell me I don't necessarily have to settle for a verbatim/self-explanatory name
Thanks for both of your input, I'll try and take inspiration from it :)
10:52
I have an OCR engine that outputs "runs" of text + bounding boxes. I need to merge all of these into one single string. Essentially the same problem as this. Any ideas? Surely there must be established algorithms for this, but I can't find anything useful
That's awfully close to particle track finding...
You probably don't want to use a Kalman Filter, do you?
I'm guessing some clustering algorithms may also be suitable. Effectively, you could cluster the boxes by alignment/orientation and then pick close pairs to form strings.
take a look at this: stackoverflow.com/questions/55654142/… if you do this on each bounding boxes, you should be able to make it work
you could also train tesseract on a text dataset (eg: of english words, etc) but using different rotation of said text in the training.
there this too, but you'll need to edit the way it handles rotation I think: gist.github.com/jarodsmk/4d3c0f19fba9c386cfec292513e946b4
@MisterMiyagi I don't even understand how to use that or what to use it for
there also this but I don't think this work for every angle/rotation: stackoverflow.com/a/66234564/12349101
if you had an example image with some text, maybe I/someone could make a MRE for you
final link: pypi.org/project/deskew might be the best option, since it work by using the alignment of the text to get either an horizontal or vertical image, inside of other angle/rotation
I think you misunderstood the problem, I don't want to rotate anything, I just want to merge the text
Here are some examples/test cases with basic axis-aligned bounding rectangles
@MisterMiyagi That and particle tracking sound like useful leads, thanks
11:12
You might also want to look into general collision detection, as done in games. Your task is basically to find things that collide with some wiggle room.
But that would probably be very close to the clustering approach in practice.
@Aran-Fey ah, got you
so basically the same problem as the post you linked but without any rotation/different angle?
@Aran-Fey tested it on Pycharm and 3.8, merge is undefined :/
Actually with rotations and bounding boxes that aren't even rectangular. I just simplified it for the sake of the example
merge is the function I need to implement
ah, make sense then
but that's also why I mentioned rotation on the image/correcting the alignment, because by doing so on each bounding boxes, you'll be able to use OCR on it.
I have already used OCR on it, that's where I got the text and the bounding box from
 
1 hour later…
12:22
@Aran-Fey would you be fine if my MRE is using tkinter + PIL? (I used PIL to create image out of strings since the angle feature of Tk.canvas kind of make the text looks weird/pixelated when rotated)
I don't know why you need a GUI, but sure
wait, so your bounding box aren't displayed on a GUI? (it's all in ASCII?)
There's no display at all as far as I understood. Just the raw data.
Exactly. I get a list of Runs as input, and have to produce a string as output
but that raw data, it's just images right? which contains one or more bounding box with text inside?
12:31
Just the text and geometry information, I think.
It was an image before I passed it into the OCR engine, now it's just a list of Runs
OH, so you just use the result from the OCR you did, which is the geometry and text. You don't use any images/etc, just that? and you want to be able to reproduce the correct string from that only?
Yeah
now that make more sense. I guess I don't even need PIL then :D
Tricky problem. In all three test cases, we want "Hello" to appear first in the output. But sometimes "hello" has a lower Y coordinate than "world" in the input, and sometimes it has a higher one. Hard to define an objective criteria for firstness.
12:43
@Aran-Fey do you have any rules though for reconstructing the string? I mean, there many different ways to use the given dimensions to recreate a string using N text/element. eg: nearest neighbor with a limit/range, if they are on the same line/alignment, if the bounding/dimensions are "touching"/overlapping each other boundaries, etc
@Kevin exactly (got kevin'd since this is slightly what I mentioned above)
@Kevin It's the x coordinate that counts, doesn't it?
Perhaps "sort the runs using the key run.x + run.y" would get you consistent ordering for all three test cases, but I imagine it could give strange results when runs are more densely packed
I thought the same too Miyagi, but I can imagine one or two cases with more than 2 element/text where it would be hard to know the "priority" of the element to form a string
if you have only hello and world in an imaginary space and want to form them as string, then x coordinate is probably the only thing you'll use, but try that with more element, and you'll see what I mean quickly (unless you have a decent foundation/rules to form a string from coordinate)
@MisterMiyagi In the first two test cases, yeah. But in the final test case, "Hello" has a greater X coordinate than "World"
@NordineLotfi Nope, no hard rules. The goal is that if you show a human the input image and the text, their reaction should be "Looks reasonable" and not "WTF"
12:49
   +---------------------------+
   |Lorem ipsum dolor sit amet,|
+--+-------------------------+-+
|consectetur adipiscing elit.|
+-----------------------+----+
|Praesent sed diam arcu.|
+-----------------------+
@Aran-Fey but if there are more than two elements (eg: more than just "hello" and "world") it requires the knowledge and understanding of context. If you know how a string should look when formed correctly and I show you a bunch of boxes misaligned/rotated with text with varying distances between each others, then you'll be able to easily do it, because you're human and know what context is.
Consider this example paragraph, which might get OCR'd as three runs, illustrated here as rectangles.
The run beginning with "Lorem" has the greatest X coordinate of all the runs, but we want it to appear first in the output.
Not an insurmountable problem, I feel. But it makes things a smidge harder
@NordineLotfi I think that also applies if there are only 2 elements
The context is "text is written from left to right, and top to bottom"
Re: context. If you show me a paragraph of text written in a foreign language*, I could probably still tell you the order the characters should be read. This is fortunate -- if we want to teach a machine that kind of context, it doesn't have to learn what words mean. It only needs to know how they're typically arranged.
(* and also tell me "text is written from left to right, and top to bottom")
I mean, if you never heard or seen Chinese, (and didn't ever have access to books or the internet or heard rumors/facts about it from other human beings) you probably would have a hard time in knowing the order in which a foreign language should be read.
12:58
Let us hope Aran's input does not include any boustrophedon
Wow, yeah, I know the language and thus direction of the text, but I don't support that
@NordineLotfi Valid. I have very little experience with Chinese characters, and I can't confidently tell you what their directional rules are.
It would be nice if the OCR system could detect U+200F RIGHT-TO-LEFT MARK
@Kevin to do that, it would need to disassociate the part of the text that has to be read in that particular order, from the rest of the text (unless your whole input has to be read this way, then you can kind of cheat by already using your knowledge of it being so). I guess you could do it if you can make an OCR system that detect languages when mixed together (really hard to do if they all share the same alphabet though)
although, it's not necessarily hard for humans to do that, even if I don't know Arabic/Hebrew, I can still recognize at least one or two chunks of text in that alphabet, and thus know that it has to be read this way. If the different languages all share the same alphabet but are all different, it requires more knowledge of the respective languages and understanding.
OCR system developers, please implement
there probably some existing paper or github repo that does the above, but since it would probably require a lot of time to install or huge computational power (eg: more than one GPU, lots of ram, etc) I sadly can't really test any of them
13:15
Coincidentally, I've just come upon a paper with that very topic pdfdrive.com/optical-character-recognition-e53691108.html
> Spitz [Spi97] presented language identification by classifying scripts into two classes, Han-based and Latin-based. The classification was based on the spatial relationship of features related to the upward concavities in character structures. Language identification within the Han script class (Chinese, Japanese, Korean) was performed by the analysis of optical density distribution in text images.
From page 57
Interesting. Also, I found this: stackoverflow.com/a/67362839/12349101 might work with Tesseract by just using an additional model
stackoverflow.com/questions/74599505 this is either unclear, NMF or else a duplicate of some kind of string formatting canonical
In computer vision or natural language processing, document layout analysis is the process of identifying and categorizing the regions of interest in the scanned image of a text document. A reading system requires the segmentation of text zones from non-textual ones and the arrangement in their correct reading order. Detection and labeling of the different zones (or blocks) as text body, illustrations, math symbols, and tables embedded in a document is called geometric layout analysis. But text zones play different logical roles inside the document (titles, captions, footnotes, etc.) and this kind...
I often find it encouraging to know that there is a name for the problem I'm trying to solve
same, but sadly, finding the right name for a concept or something you thought of but didn't know existed/if it already exist is really hard and time consuming
it's like hand-waving but through text, until you find the nearest word for what you mean, or even better, the exact one
"time consuming". Case in point: it took me an hour to find this.
13:28
@NordineLotfi except google, it always finds the right thing. I often wish that any search box would be as intelligent as google ones
@Kevin Yep, would have taken me more than an hour knowing me
@Hakaishin as long as you use the right search term, and it's something that have been correctly indexed
more often than not, I find myself needing to use more than a dozen different synonyms or nearest meanings to find what I want. if it's really popular or I'm "feeling lucky", then I usually find it
I find that using a lot of different search engine has more chance to yield more results for the things I'm searching (since they don't always index the same things, or have the same results). But when I'm lazy, I just use duckduckgo and google
14:02
Hi, getting this exception while working with sqlalchemy: (psycopg2.errors.DatatypeMismatch) column "createdat" cannot be cast automatically to type timestamp without time zone
op.alter_column(
'table', 'createdat', nullable=True, type_=sa.TIMESTAMP)
14:35
I'm not sure what's unclear
You also haven't asked a question. You're just stating you have an error
15:05
when I define functions inside a class and use those as methods for my class, I always see pycharm labeling them as weak warnings (eg: "Method X may be static). I know what this means (eg: define the function outside of the class instead) but does that actually indicate that it's bad practice to do that?
I saw a couples of modules on pip that do this, so not sure which is best practice
@MisterMiyagi Thanks to you for saying that, I just noticed that Pickle actually doesn't support that many python objects :/ (eg: generators, etc). There are ways to emulate support for those in pure python, but...I guess it's another entry on my todo list
If your method never makes use of self, but you define it as an ordinary method anyway, this may be inconvenient to future users of your class. They may want to use your method without first creating an instance of the class. Best case, it's a minor annoyance for them; worst case, they won't be able to create it at all due to some requirement they're lacking. For example, if your __init__ requires a database cursor as an argument, and the user doesn't have a database.
"define the function outside of the class instead" is one possible design. Consider also the @staticmethod and @classmethod decorators.
@Kevin I see :o can you show me a couple of examples (eg: using SO post, or snippet) of those minor annoyances? Also, I do support arguments on my class, but I try to details the default arguments using docstrings, and comments
didn't thought of using those decorators, nice
Extensive documentation is nice to have, certainly. But if you extensively document that the user must provide a database cursor, and the user doesn't have a database cursor, the beautiful docstrings will not help them out of that jam
that's true
By minor annoyance, I mean something like: the user would prefer to write x = Frobnicator.reticulate(y), but instead they must write z = Frobnicator(a,b,c); x = z.reticulate(y)
15:16
ah, yeah I see what you mean
@NordineLotfi You may want to look at the dill package.
yeah, already know about it :) it's pretty good
but I wanted to do it myself for other projects, especially since there a lot of unresolved bugs I came upon on their github so...you know
oh dill sounds super cool, what I at first thought pickle is
I know what you're gonna say, but honestly, I feel like this is gonna be a nice feedback loop: on one hand I'll do it myself, so I guess I'll pick up some stuff as I go, and on the other hand, what I do might help in figuring out some solution for the issues on their github
@NordineLotfi My life has become much better since I stopped trying to invent my own rules and doing-everything-I-can-do. Classes are method turf, no functions allowed.
15:23
@NordineLotfi yes, it does. See Kevins explanation
@MisterMiyagi so you just define those functions outside of the class then? do you have an example template to take inspiration or something :o
Meanwhile I am still trying to use classes like namespaces (which are one honking great idea), so my staticmethods are abundant
@NordineLotfi Not sure what template you need there. It's pretty simple, actually. No functions inside classes.
@Kevin I'm still confused/conflicted if the line about namespaces is sarcastic. I honestly always read it as sarcastic, since I hated them coming from c++, but I'm not sure if in python they are unironically a great idea :shrug:
but then how can you call whatever those functions do inside your class? or do you just build your class to do what your functions do?
15:24
@Kevin I'm selling these nice Metaclasses. Buy now and get a bridge for free!
@NordineLotfi you just call them
I'm sorry if this sounds weird, I think I said once I don't always use classes and prefer just using functions (and sometimes one or two decorators)
@Hakaishin ah, I see
Well if the bridge is free...
15:36
@MisterMiyagi I just want to make sure I understand: does that mean that things like this are a no-go for you? I did try to find an example on your github, but don't think this applies here (you're using def inside of a class, so doesn't this count as functions inside of a class?)
@Hakaishin I already know that, see my first question earlier which mentions/hint that I already know: chat.stackoverflow.com/transcript/message/55599821#55599821
There is a real problem, that colloquially speaking people call methods functions, but strictly speaking one should differentiate between methods and functions, especially when teaching the subject or when it is not clear what is meant from context. Then later on one can revert again to calling everything a function
@NordineLotfi this question indicates to me that you do not know, otherwise you wouldn't have asked it
@Hakaishin so what Miyagi meant was that methods inside classes are OK but function classes are bad practice?
please read the tutorial I linked you
15:46
@Hakaishin I'm asking even though I already know functions defined inside classes are called method and such. I'm just unsure of what's best practice
then what are function classes?
16:06
Hello,

Anyone with some experience with openpyxl?
@Mez13 some know about it here, feel free to ask your question
Alright.
Have a very specific issue. Trying to build Excel LineChart using openpyxl then, everything works fine but the coloring. I try to color the lines in the charts using what 'everyone' recommends : graphicalProperties.solidFill and graphicalProperties.line.solidFill. But when using that the output Excel file appears corrupted, I can't even open it
Did that every happen to anyone?
@Mez13 what is the filetype/extensions you're saving it as?
.xlsx
16:35
@Hakaishin I could link the python official docs, but I don't think you want me to do that. Based on the previous replies, it sounds like you're only judging based on how I say things vs what I actually say (eg: when you said "colloquially", etc). I'm aware I may not always use the correct term sometimes when talking, so I admit it can be frustrating when you and others only ever always use the correct term.
we could argue the semantics of how certain things in programming concepts such as classes are just that, concepts, but I don't want to go into that, because that's beside the point. I just wanted to know the best practice, as I already know how to make a class and what a class is. Me not using the correct term sometimes isn't a valid/fair point I think.
17:07
@NordineLotfi Not sure what you mean. Those are all methods...
@Kevin Your sacrifice is appreciated
Kudos for looking at that compatibility layer, by the way. That one is a re-implementation of an existing API, though, so it's not exactly how I would implement things myself.
@MisterMiyagi ah, yeah that's what I meant earlier: chat.stackoverflow.com/transcript/message/55599821#55599821 I guess I forgot to mention that I said "defined functions inside a class" to say "methods inside a class", since I moved all of my functions inside of a class, thus why I asked for best practice on how to do that (Kevin gave some possible solutions, but yeah)
@MisterMiyagi Thanks, I had to look at many files until I found a decent candidate :)
@NordineLotfi If you have a def inside a class then it's not a function, it's a method. PyCharm should not warn about that. Can you perhaps give an example of what you mean?
I only get a weak warning about the error I mentioned for the put, get, queue, and size functions inside the class. I know, I'm not using self yet, but I planned on using it, which is why you see it everywhere
17:21
@Hakaishin I personally think namespaces can be a great idea, if the language gives them strong support. But I don't think Python puts any serious effort towards supporting them, so the import this author might have sarcastic intent for that line.
@Aran-Fey It was fun :-)
The kind of namespace support I'd like to see in Python would look like:
namespace foo:
    x = 23
    namespace bar:
        y = 42

print(foo.x) #23
print(foo.bar.y) #42
@Kevin might be possible to emulate with a context manager? Not sure
@Mez13 try and see if this example from the official docs works: openpyxl.readthedocs.io/en/latest/charts/line.html (the first one I mean)
I believe I once tried making a namespace block by using a context manager. It required some frame inspection trickery and a couple of peeks at locals(). A bit more dark magic than I'd have liked.
@NordineLotfi As a simple first step, make the filename an instance attribute instead of a class attribute. That way you automatically avoid writing incomplete methods.
@MisterMiyagi that's actually a good idea :D Thanks
@Kevin Interesting. I was just guessing but I did thought it would be more complicated than that
maybe you could do it by rewriting the AST, but might be even more complicated
@Kevin erm, modules? It's pretty hard not to have namespaces in Python.
17:31
when you said "first step", does that mean you have other things in mind that I could improve (on the link I posted at least)
@MisterMiyagi Modules are nice, but I want more :-)
@Kevin MetaModules? :P
@NordineLotfi Well, I don't get what default_type is there for. You may want to implement that properly. :P
95% of my projects each occupy a single file, so the namespaceability of modules is of little use to me
FWIW, I don't think your size implementation is correct. Pickles are streams, you would have to read everything and get the size of that.
Unless you dump the entire queue everytime, which I would not recommend.
@MisterMiyagi hmm, that's a good point
@Kevin sounds like me
@MisterMiyagi you're right. I did think/doubted whether mine was good enough. Don't know any ways to improve it, so this is mostly a possible implementation (albeit bad). Do you know of any ways to improve it?
@MisterMiyagi it's not necessarily made for speed, but I agree it'll be pretty bad once you have too many elements in the queue/deque.
17:48
Depends a bit on whether the use case is a 1:1, possibly reader writer, relationship or more of a broadcast where any process can read/write.
I'd probably store the length separately and use the pickle as an actual pickle stream where you directly dump/load objects.
I recreated my namespace context manager code. As before, it has more dark magic than I'd like. gist.githubusercontent.com/kms70847/…
@MisterMiyagi hmm, I'll ponder about this more. Thank you for your input :D
@Kevin that's nice. I was about to say earlier, since I already know you can query the lines of code using inspect, you can then just take any lines after the keyword namespace and execute that separately (eg: like an external module would)
Sounds hard. Particularly, making sure that all global and nonlocal variables still point to the right locations.
 
2 hours later…
19:28
@Kevin looking at KevinScript for inspiration, but I noticed you use @staticmethod here but not here? Just curious
I'm interpreting that question as, "why do you have functions inside your ObjectFactory class that don't have a self parameter? Shouldn't those be staticmethods?". Those functions are effectively "private" functions, because they're defined inside other functions. For example, connect is defined inside __init__. One of the reasons they aren't staticmethods is because I don't want anyone else to call them.
I probably went overboard with the nested functions, but oh well
I see! this does explain a lot why some people do that. Thanks for answering
I always had this feeling that using classes was more "professional", even though I don't ever use them (except when I was obliged to use them a couple times, but those don't count). I'm tempted to just use my current style, which is to just throw working functions inside a single file, or maybe use modular programming instead of OO...
There is too much cognitive load when using classes, (for me at least) to be honest. I don't know how Aran and everyone else do it.
19:49
Python is a multi-paradigm language, so you should feel free to ignore the OO paradigm whenever it suits you
8
Even if it's for making a package for people to use? (instead of just something I use for myself I mean).
I never saw any packages on pip or github that had 100% functions like what I'm currently doing
If the package is understandable and user-friendly without OO, I don't see any need to add it in
My philosophy is "right tool for the right job"
I see :) I feared I might get bad feedback from people who prefer OO, but I guess it might not be as bad as I thought
Thanks, I guess I can concentrate on the logical part of what I'm doing instead now
although, I guess it might bite me back when a job require me to work with classes exclusively, but that's a story for later, maybe
I usually prefer just writing functions first too. So I do have a personal tip for when i see that i need to use classes: it's when i find myself passing the same dictionary or object to multiple functions. if 5 of my functions all accept the same argument, it might be worth considering whether those functions belong together with the data in a single bundle.
@ParitoshSingh Interesting, that's a good tip. Thanks! (also it's been a while)
19:58
Aye, been a while, i haven't even been able to lurk properly lately! Got a lot going irl, been busy.
I know what that feels like. Welcome back :)
20:12
@Mez13 did the example from the doc help? I mentioned it since I tried it with a different rgb code, and it looks like it works?

« first day (4426 days earlier)      last day (746 days later) »