last day (14 days later) » 

3:07 PM
0
Q: Does Python have a versioned dependency management solution?

Josh StoneDoes Python have anything similar to apt or Maven where a single repository can house different versions of a library as opposed to just the current version? For example: My site-packages folder does not group libraries by version. So instead of: /Library/Python/2.7/site-packages/tox/1_2_3 We...

 
You're trolling a bit.
 
I mean this question genuinely - not trying to troll. I'd love to help build something like this, but I'm wondering if it's possible why it hasn't already been done.
 
Also, there's easy_install and pip.
 
But even with pip I still have software installed that suffers from version conflicts.
 
You don't really need 12 virtual environments unless you actually need to use specific versions of libraries, usually for testing purposes. For the average user, installing libraries with pip or easy_install installs the latest version and that's all that's needed.
 
3:07 PM
I do need specific versions of libraries due to different software depending on specific versions. Otherwise I have breakages. Simply depending on the newest version of something only works if every change to that library forever is backwards compatible. This is pretty rare.
 
plus, I'm pretty sure that Maven isn't actually that more sane than either of these tools -- it's just that Java programs often ship with their (sometimes slimmed down) version of the libraries they need in their own source tree. Which is a bad thing for open source dev, IMHO. Thus, conflicts rarer appear.
 
Python (or more specifically pip) has requirements.txt, which might help - pip.readthedocs.org/en/1.1/requirements.html
 
Maven is not sane in many ways, but having a single directory on a machine for all dependencies (.m2/repository), sorted by version, seems much saner than dumping all libraries in a directory and hoping there won't be version conflicts (usr/local/lib) or creating a separate copy of libraries for each piece of software (virtualenvs). Am I wrong?
 
Python does have a single directory for all dependencies; it's site-packages. It seems what you are really asking is "why does Python not allow you to install multiple versions of the same library at the same time". That is an interesting idea, but it would require every program to specify not just what library it wants, but what version it wants. Right now this version specification is offloaded into pip. I think your question is likely to be closed as "opinion-based". It might be better to reframe your question to describe specific problems you're having and ask how to solve them.
 
@BrenBarn If I install via pip, don't I still suffer from possible version conflicts with other libraries?
@BrenBarn Updated the question to try and be less opinionated.
 
3:07 PM
@JoshStone: Possibly, yes. I don't know enough about Maven, but I would be surprised if Maven in itself solves all these problems. I suspect the real difference is differing practices in the Python and Java worls about when and how libraries are released, meaning that interpackage backwards incompatibilities are more likely to crop up in practice.
I think your edited question is much better and I'd be interested to see an answer. As I said, though, the problem with doing this is, if you write a program that wants to use tox, how do you do it? Do you have to write import tox_1.2.3? If so, you force users to install an old version even if your program doesn't specifically require it. Otherwise you have to start specifying things like import tox>=1.2.3, which I think is just culturally a level of detail that Python programmers prefer not to have in the actual program source code.
 
I agree that the versioning problem is a level above Python. Perhaps Python just needs some build/packaging tooling that selects the appropriate version of a dependency to run/build/package against, as Maven based build tools (in various languages, mind you) do. Pip seems to be close to this, but in installing all dependencies in a common site-packages directory you still end up with version conflicts.
 
@BrenBarn Plus you still have no idea if that's even a valid thing to do, since a version after 1.2.3 might not be backwards compatible. setup.py allows for some limited specification of version requirements upon installation, but it definitely isn't a version selector.
 
Do the packages need to be installed globally?
 
@ThorSummoner Globally installing packages gives you some resource sharing (without having multiple copies of something, as with virtual environments), but you still need to store them by version to avoid conflicts.
 
@JoshStone Well that didn't answer my question; though My motive for asking was that, if your project can live with local versions and doesn't require them to be global installed, you could include the version inside your project and use non-absolute imports so the legacy code will pickup the nearest install rather than grab the global package.
Part of the magic in my eyes of python, is that you can import namespaces as other namespaces, so in your legacy code you can replace something like import tox with import tox_legacy as tox
 
3:07 PM
@ThorSummoner But then we move away from the common repository idea, which is very useful. We just need the common repository to be version specific.
 
I agree it would be pretty cool if there were an "on-the-fly pip" that read the dependencies of everything and did behind-the-scenes magic so that import libraryx automatically imported the right version without having to create a whole virtualenv for it. I'm not sure how achievable that is overall, though.
 
@JoshStone You can rename the site package module (Well you might have to edit the legacy code's setup.py and reinstall it under a new name to do it cleanly) from which your dependent code can then import the old version in place of the current one. This moves away from pip though and relies on slightly more manual installation procedures.
 
@BrenBarn I think Python would just need a way to know where to find all of its runtime dependencies. For example, with Java's classpath we tell the runtime where to find all of the dependencies, including specific versions of all the libraries we depend on: usr/local/lib/python27/site-packages/tox/1.2.3, ....
 
Oh I also should note that dots in a module name are syntaxically invalid as dots are used as namespace separators. So you'd have to name it like tox/1_2_3/
 
@JoshStone: I think the problem is, though, that to make that work through-and-through it would have to parse the library dynamically on every load to see what it's importing. Python is much more dynamic than Java so you can't easily precompute dependencies. For a complex dependency structure that could add non-negligible overhead to imports.
 
3:07 PM
@BrenBarn hell in Python I can import a module in the middle of a function call if I want to save time on imports for large libraries so that users only have to pay the import costs on methods they actually use. And this import statement could in theory be based on a textual string that the user entered at runtime or that I loaded from a database. It's basically impossible to precompute what a Python module is going to import.
 
@BrenBarn I imagined things would work the way they currently do, with the addition of being able to influence where Python looks for dependencies when executing an import statement. But I take it this is not possible?
 
@JoshStone: Heh, you keep saying that, but I don't think it's really what your question means. Python doesn't look for "dependencies" per se when executing an import statement; it looks for modules and packages. You already can influence where it looks for them, by modifying sys,path, using .PTH files, etc. The issue is that you need it to compute the dependency structure at the time of the import in order to know which version to import, but the information you need to do that isn't part of the import statement, because the import statement doesn't specify the version.
 
Another thing to think about; Let's assume you could let modules specify what version of a module to import. What happens when two modules import different versions inside the same runtime? Given the aforementioned ability of Python to import a module at any time, even form a REPL source, defeating the ability to pre-compute import conflicts, what would you expect to happen?
 
The import statement shouldn't specify the version - that should live elsewhere - such as in a .pth file or in a set of arguments passed to the interpreter. @aruisdante Modules shouldn't be able to specify what version they import - they merely import and assume it's the correct version (as it stands today). Ensuring the correct version is imported is the job of whoever sets up the runtime environment - via .pth files (I suppose) or whatever else.
 
@aruisdante I thought that any given scope would import the version of the lib it needed, to my knowledge you can import two entirely different modules that have the same name so long as the scope and import precedence are considered, Eg one per scope per import precedent or many per scope with explicit and aliased imports.
 
3:07 PM
@BrenBarn Do you think it's possible to utilize .pth files to specify a set of specific modules (identified by name/version) per python application? Something like, one .pth file per application that I install. Doable?
 
@JoshStone the second question you just edited in is too broad. The answer is of course "yes, with enough work".
 
I created a chat room if you want to discuss the idea further, since this comment thread is clearly getting too long: chat.stackexchange.com/rooms/21446/on-the-fly-pip
 

last day (14 days later) »