« first day (4549 days earlier)      last day (385 days later) » 

2:35 AM
I messed up a bit while trying to figure out what's a duplicate of which. I would like to close stackoverflow.com/questions/14083111 now as a duplicate of stackoverflow.com/questions/728891 but I reopened instead of editing the dupe link
I eventually figured out there are essentially two canonicals for the topic: that one (about whether the encoding declaration will be necessary, and how it should be written), and the more important one stackoverflow.com/questions/10589620 which is about the corresponding error message, why there is a problem, and giving the advice to add an encoding declaration.
 
 
8 hours later…
10:53 AM
I've been thinking; two abstractions that seriously need to catch on are files and file systems. Imagine if there was a well-defined, universally accepted interface for both of those things. Accessing files on dropbox or google drive or a github repo or a web server or an ftp server or in a zip file could be (almost) as easy as accessing local files.
 
you mean, like, URIs?
or do you mean like, some kind of OS integration for browsing e.g. github directory trees?
because in the latter case, how do you communicate to the OS that "hey, github dot com slash username slash projectname is a webpage that in some way conceptually represents a directory tree"?
(and then furthermore how it does so?)
 
OS integration would be optimal, but I'd be happy enough with just a well-defined python interface
fs = GitFilesystem('/path/to/repo', branch='main')
for file in fs.iterdir():
    print(file.name, file.read_text())
Gimme something like that ^
 
The main problem with doing something like that is supporting different API for accessing/decoding/encoding files. Things that do not really change or need changing (eg: zip, tar, etc support) are fine, but then you have things that are relatively constantly changing, such as gdrive, github, etc
two tool I know that try to do this as much as possible, but still have quite a bit of bugs spread around would be git-annex and rclone. Most of what they do is offset the "multi support" to the community, so it's extremely fragmented (since not everyone would have the same level of dedication to maintain and fix different extension/plugin so it supports different API)
 
11:10 AM
String paths should be outlawed, every piece of code should use Path objects and it would just... automatically work with any kind of file system
Imagine if you could just shutil.copy(dropbox / 'foo.txt', onedrive / 'foo.txt')
 
the trick is defining those file systems
in particular, the logic needed for working with them
 
People have already written that logic a dozen times over, just look at all the modules on pypi
Each of them with its own unique interface
 
fundamentally, the dropbox and onedrive filesystems are going to require a certain amount of custom logic
because they'll translate "paths" to URIs differently
that said, extending Path sounds like a really elegant interface
but then you also need to redo shutil to work with that... somehow?
 
It would work for code that uses path.open() instead of open(path)
 
yeah, but for the shutil case in particular
you'd lose the optimization for "move file" cases that don't require copy-and-delete
perhaps if there were Path.move, but then the Path would also need some concept of the "volume" it's on
(type-checking isn't enough; you need to distinguish the case of moving within one person's Dropbox vs. between two different ones)
 
11:27 AM
@Aran-Fey As in POSIX?
 
I'd probably just make a special case for if src_path.file_system is dest_path.file_system: tbh
@MisterMiyagi I have no idea. What does POSIX have that's similar to this? Virtual file systems?
 
It's actually pretty simple to turn arbitrary things into filesystems using FUSE.
The main problem is satisfying the expectations on a filesystem, such as consistency and promptness.
 
People expect promptness from a file system? O.o
 
@Aran-Fey on POSIX, there this thinking of "everything is a file", so I guess it has it's own way of merging the abstraction of files and file systems.
 
Accessing files is, like, the slowest thing you can make a PC do
 
11:32 AM
@MisterMiyagi yeah, FUSE can be really slow
 
@Aran-Fey Well... not if we're talking about the slowest thing you can make a PC do with other PCs.
 
@Aran-Fey I mean, it depends on size. If you take as much time accessing a 1MB file as you would for a 1GB file, there might be a problem somewhere. Smaller file size -> Faster speed of transfer
and if you use chunks and other tricks, you don't really notice too much slowdown until a specific size (then the main problem will be network speed, but that one is beside the point)
 
Right, so what about... ls? Or cd?
People expect these to be fast.
If you've got a reasonably large background service, they won't be.
 
cd is easy since you don't really have to really do it if it's a remote filesystem (you just switch the parent directory). for ls you would probably have to cache the directory tree/index if it's too big though
 
I hate it when a cluster takes half a minute to ls
 
11:36 AM
@NordineLotfi How would you switch the directory without actually checking if the directory exists?
Or that it is a directory?
 
@MisterMiyagi I don't see the problem? All I'm saying is there should be a universal interface. I don't think this proposal has any significant impact on the speed with which you can access your files
 
@MisterMiyagi ah, I admit I'm not too sure on that one. It might depend on the API or filesystem you're accessing (eg: if it's a third party API from some storage company, etc)
 
The problem is that if you provide people a filesystem-like interface, they use it like a filesystem.
And that doesn't work outside of trivial cases.
@AndrasDeak--СлаваУкраїні I hate it when users ls on a cluster. D:
 
I don't really see why not. A slow filesystem is still a filesystem
 
11:39 AM
Is it really that much of a burden to do ls on a cluster?
It's not something that I could imagine hogging tonnes of resource even if it's slow running
 
Depends on the filesystem
 
@Aran-Fey Is it really? If my local filesystem takes a second to respond, I trash the system and get a new one. Most software isn't written with "slow filesystem" in mind, and most users can't stomach the idea either.
 
But arguably you need ls to do things
 
@roganjosh Yeah, having to "stat" tons of files takes notable time on some distributed file systems.
 
I'm absolutely the worst then because I wouldn't have thought twice about it
 
11:41 AM
The problem is that metadata usually is required to be consistent (assumptions again...) so operations on them scale badly.
 
@MisterMiyagi Would you trash your dropbox account if it took 1 second to respond?
 
@Aran-Fey No, because I know it's not a filesystem. ;)
 
Meaning what exactly? If it's used like open('foo.txt') then it must be fast, but if it's used like dropbox.Dropbox().download_file('foo.txt') then it can be slow?
 
Don't get me wrong, I too would like to have such an abstraction. But I know I wouldn't use it right at 2AM.
@Aran-Fey Well... yes? Code that does open('foo.txt') expects to run on all filesystems, and will have expectations matching most filesystems.
 
@MisterMiyagi Technically it's a filesystem, but not in the POSIX sense , eg: you don't directly access it (except if you see things from the company's access perspective instead of the end-user perspective)
 
11:44 AM
Pretty sure dropbox is an object store, not a filesystem.
Which is probably nitpicky to most people, admittedly, but that it's nitpicky is actually the point.
 
I'm sorry, but I don't get the point
 
@MisterMiyagi hmm, I mean yeah, it's similar to S3 in that regard (eg: it being a bucket/glorified dict). But when you know entries on that object store/bucket/etc are files/folders, and the company-side probably have it distributed on a virtual filesystem over multiple servers, I guess you could say it's a filesystem? just not in the most direct access sense
@MisterMiyagi yeah, I do think it's fair to only think of whatever you can directly access as you would with ssh as filesystem. Maybe I was being pedantic :P
 
@Aran-Fey "people aren't sensitive to the difference between storage systems and only care about them if it gets thrown into their face."
 
@NordineLotfi Well, from this discussion I'm starting to see why boto3 might limit search results to the first 1000 values, which baffled me in this day in age
 
Now I'm more confused than before :D Isn't my idea to minimize the differences between storage systems?
 
11:50 AM
Then again, it chooses to recurse directories by default, which really confused me at first. Given that it's not a filesystem it seems like its default behaviour would actually ramp up the complexity of what it's doing. A default footgun
 
@roganjosh sounds bad yeah :/ I'm guessing there might be some way to mitigate that, but I never used boto3
 
You're lucky. You can set the max number of results to whatever you want, but if it's over 1000, you're getting 1000 and that's it. Not that it tells you that when it truncates your search
 
when you say search, do you mean that for content inside files or just the filenames/hashes?
 
Basically what you would assume to be os.listdir(), but it's not. And if it finds a directory and starts recursing that, you gobble up your 1000 result limit very quickly
It's still by far my least favourite big python library. It's totally bloated but still somehow a thin wrapper around JSON. It won't tell you if there's any exception, you have to result.get('Exception') just to see if it worked or not, so it silently fails unless you know this kind of stuff
 
I guess that's what you meant by using IsTruncated last time: stackoverflow.com/a/54314628/12349101
 
12:05 PM
Similarly with the truncation. I can't remember the exact code but it's something like result.get('IsTruncated'). Basically you have to actively search for whatever bonkers thing it might have done
 
12:59 PM
@Aran-Fey Yeah, and it's a good idea if you're dealing with consenting adults.
 
 
6 hours later…
7:28 PM
I didn't think it was possible to create a documentation software worse than sphinx, and yet here we are, with mkdocs not providing syntax highlighting out-of-the-box, and this stupid extension requiring you to prefix every single code block with py
I'm ready to give up programming and become a hermit now
 
8:06 PM
I remember having a similar problem but with Emacs: stackoverflow.com/questions/61723481/…
in Org mode, you have to prefix or wrap a code block with its corresponding syntax identifier (eg: if you want syntax highlighting or being able to execute it)
 
@Aran-Fey is there no conf.py-compatible alternative?
 
What does that mean? An alternative documentation generator that can read sphinx's conf.py?
 
Is that just for sphinx? Nevermind then
 
 
1 hour later…
9:32 PM
@Aran-Fey I just found a relevant feature-re...nevermind
 
Must be really new! There wasn't one when I looked! :D
 
(That's a lie, I didn't look)
 
I mean, I googled. But I didn't use the github search bar
 
9:39 PM
Hello. I have a Flask application. Is it possible to detect what WSGI server is used using Python?
 
there's this for "inline", don't know if it generalizes (issue is weird, question and answer both in first comment) github.com/facelessuser/pymdown-extensions/issues/910
 
I don't think so; I had a quick look at the code and I didn't see it access any config values other than guess_lang
 

« first day (4549 days earlier)      last day (385 days later) »