Disk space and memory are relatively cheap, so I don’t see having duplicate libraries as particularly problematic, especially when most applications are dominated by assets (e.g. icons and other images).
My larger concern is security. There are two major ways of thinking here. One is the shared library model where everything uses the same handful of libraries, so if a vulnerability is found, that handful of libraries can be updated and everything is secure again. However, if a vulnerability is discovered, every application that links to it is potentially vulnerable, so it’s kind of like putting all your eggs in one basket.
The alternative solution is containerization. Basically, put all of an application’s dependencies in a container and restrict what access the application has to the outside world. This way, if the application has a vulnerability, the attacker has to also break out of the container, and there’s not as high of a chance that the rest of the applications have a similar vulnerability if they’re using different versions of the library.
I personally believe containerization is on net more secure than having all applications use the same version of a dependency, provided the interface between the container and the rest of the system can be limited.
game can be lost
This is also fixed with containerization. I can still play old Windows games through WINE because WINE essentially packages those dependencies. For Steam games, Steam ships a directory of libraries for Steam games to use, because many Steam games don’t ship every dependency.
Steam could take this a step further and run Steam games within a container to limit access to the rest of the system. You could also run Steam within a container and limit its access yourself.
tools to use up lots of the machine
But how much is “lots”?
My system uses well under 10GB for libraries, and many of those are only used by one or two applications. I could duplicate the entire system 3-4 times before I start to notice the disk usage. If I converted everything to FlatPak, I’m guessing I’d still be well under 50GB, which is my personal limit for considering the disk usage to be a problem. Many packages use a similar base, so many would share the same base image layer.
If your priority is security, you’d probably be better if e containerizing everything and managing access of each app to the rest of the system than trying to secure a handful of shared libraries. If your priority is preservation, again, containerizing is better because you can freeze dependencies as needed. If your priority is space, then you won’t want containerization. But given how cheap storage, that doesn’t seem like it should be the target.
This debate is older that Docker/Flatpack/Snaps/etc.
It’s basically the same as static linking everything vs dynamic linking. Or even, App folders of RISCOS of my youth.
Disk and RAM disk usage does add up, reference: Windows and WinSxS directory. Also, having all those old interfaces hanging around doesn’t scale. RISCOS taught me the value of centralized libs as compared to every app having it’s own (which was RISCOS’s norm).
WINE does an amazing job, trying to match Windows “bug for bug” so stuff runs, but it’s a thankless task that doesn’t scale either. Unfortunately WINE gets the blame when stuff doesn’t run, not the concept of closed software, or the vendor of that software, or Microsoft for the platform design. I’m glad WINE exists (and it’s code is well written) but it’s fighting a war from a weak position.
I want up to date libs, and some containerization on things exposed. Containers doesn’t have to mean duplicate old libs. We can have the best of both. If it’s open.
If it’s closed, it all gets bloated and stale and crusty.
If taken to either extreme it can become problematic. On one hand you try to dynamically link everything and you can end up with huge issues when you try to upgrade one of those common dependencies. On the other hand if you statically link everything (or have separate copies of dynamically linked libraries) you have a lot more disk and memory usage.
So the sensible thing is to take the middle road. Dynamically link all of your system packages, like your desktop environment, core utilities, etc, and containerize the rest of your apps. That way all of your riskier applications (closed source, or stuff with a big stack surface like a browser) can have a layer of security between it and the rest of the OS and also have a separate set of libraries that the vendor ships with. You’ll pay a small penalty for duplicate libraries, but you should only have a handful of them.
I think every containerized application should have duplicate libs. You want this exposed applications to have whatever the vendor has vetted, and you want to make sure it’s only interacting with other containerized libs.
This is basically solved for decades with package management. Dependences are in a database and things are rebuilt accordingly.
Somethings are already run in a container. Without lib duplication. I agree that there is an argument for more network facing things to be run like that.
This isn’t just static vs dynamic, but the whole app folder things again.
Far too often this whole thing is just an excuse to avoid packaging properly. Instead they gift wrap their environment of old libs. Closed stuff has no choice, so always champions ways of shipping with old libs. Really pisses me of when open stuff does it so they can avoid the work of porting to the current version of say, Python. When it’s Docker, it’s often most of some hacked old Debian/Ubuntu. It’s the exact opposite of “reproducable builds” and means that software will never make it into things like BuildRoot or Yocto. Never mind Debian/etc proper.
Closed source could document system dependencies, it just can’t be rebuilt on demand to target a different set of libraries. So it’s usually easier to give it a separate set of libraries instead of expecting the system to accommodate it.
porting to the current version of say, Python
It kinda goes both ways. To properly work with package management, it needs to support the oldest and newest versions of a library in popular distros. So for Python, that may mean Python 2.x and 3.x around the launch of Python 3. Supporting both is possible, it’s just more work for the developers for what could be considered a pretty minor benefit.
This isn’t just a Python or an interpreted language thing, it also happens for compiled shared libraries. If you need a feature from the latest libc, Debian can’t ship your package until it ships a new enough libc, but maybe it’ll ship with Ubuntu or Arch.
That said, most “system” packages are willing to go through this effort, so I expect things like KDE, git, GNU utils, etc to all use the same set of shared libraries. I think browsers are special enough that they should be containerized, if only because of the large attack surface that you should use the exact libraries they recommend instead of perhaps an older or newer one that happens to work.
Shipping your own dependencies should absolutely be the exception, not the rule, but I think it’s a good thing for some types of applications. Bloating an install from something like 30MB to 300MB is fine if it’s only for a handful of applications that tend to use a ton of resources anyway, like a browser, video game, or web service.
No, things support Python 3 or Python 2. Not seen stuff packaged with support for both. In the package dependency information is a window of versions required. 3.9 - 3.11 kind of thing.
If you need the new features in the latest libs (rarely really case) get involved with getting the latest lib into the new stable release. Yes, in the mean time, maybe ship your own version, but it’s makes a right old mess if that is done all the time for all dependencies.
If it’s for a handfull of apps, maybe that’s acceptable for a short period of time. Closed apps have no choice but to be messy.
Me, I’m in little rush for latest and greatest. Rolling with Debian Testing is plenty new enough for my desktop. For servers, I want old and stable anyway, so Debian Stable.
I meant that maybe one target doesn’t support Python 3 (e.g. older Debian) and another doesn’t support Python 2 (i.e. a bleeding edge distro). There were tons of libraries that supported both at the time, though it wasn’t due to distro compatibility. I used it as an example because the transition was so rocky and most people were aware of the issue.
But the same idea can happen when using newer compiler features or something. If I need a newer version of something than a distro supports, my options are:
maintain an older version of the tool that works with whatever the distro supports
campaign to get that version upgraded
not ship on that distro
ship with my special dependencies (e.g. use a FlatPak)
Of those options, the last is a lot more attractive and requires the least work. Ideally these are rare exceptions, but it should absolutely be an option.
The Python 2 -> 3 transission was a mess. Same with GTK2 -> GTK3. Lots of young developers think they can just ditch legacy and do all new and shiny. Learning it is a mistake to disregard legacy is part of maturing as a developer.
Debian has packaged libs as python-libname and python3-libname. There is a few python2-libname but python2 is on the path to removal.
Right, but they didn’t support Python 3 for a while. And with 5-year release cycles, it’s entirely possible that you’ll have a situation where some distros don’t support Python 3 and some don’t support Python 2. Or any other platform (e.g. maybe you need a shiny new C++ feature, but certain distros don’t support that version yet).
So those distros can either not include those packages (seems to be the current approach), add a feature mid-cycle (pretty rare, though people do make repos that do so), or allow some method of bundling it with its unsupported dependencies.
Disk space and memory are relatively cheap, so I don’t see having duplicate libraries as particularly problematic, especially when most applications are dominated by assets (e.g. icons and other images).
My larger concern is security. There are two major ways of thinking here. One is the shared library model where everything uses the same handful of libraries, so if a vulnerability is found, that handful of libraries can be updated and everything is secure again. However, if a vulnerability is discovered, every application that links to it is potentially vulnerable, so it’s kind of like putting all your eggs in one basket.
The alternative solution is containerization. Basically, put all of an application’s dependencies in a container and restrict what access the application has to the outside world. This way, if the application has a vulnerability, the attacker has to also break out of the container, and there’s not as high of a chance that the rest of the applications have a similar vulnerability if they’re using different versions of the library.
I personally believe containerization is on net more secure than having all applications use the same version of a dependency, provided the interface between the container and the rest of the system can be limited.
This is also fixed with containerization. I can still play old Windows games through WINE because WINE essentially packages those dependencies. For Steam games, Steam ships a directory of libraries for Steam games to use, because many Steam games don’t ship every dependency.
Steam could take this a step further and run Steam games within a container to limit access to the rest of the system. You could also run Steam within a container and limit its access yourself.
But how much is “lots”?
My system uses well under 10GB for libraries, and many of those are only used by one or two applications. I could duplicate the entire system 3-4 times before I start to notice the disk usage. If I converted everything to FlatPak, I’m guessing I’d still be well under 50GB, which is my personal limit for considering the disk usage to be a problem. Many packages use a similar base, so many would share the same base image layer.
If your priority is security, you’d probably be better if e containerizing everything and managing access of each app to the rest of the system than trying to secure a handful of shared libraries. If your priority is preservation, again, containerizing is better because you can freeze dependencies as needed. If your priority is space, then you won’t want containerization. But given how cheap storage, that doesn’t seem like it should be the target.
This debate is older that Docker/Flatpack/Snaps/etc.
It’s basically the same as static linking everything vs dynamic linking. Or even, App folders of RISCOS of my youth.
Disk and RAM disk usage does add up, reference: Windows and WinSxS directory. Also, having all those old interfaces hanging around doesn’t scale. RISCOS taught me the value of centralized libs as compared to every app having it’s own (which was RISCOS’s norm).
WINE does an amazing job, trying to match Windows “bug for bug” so stuff runs, but it’s a thankless task that doesn’t scale either. Unfortunately WINE gets the blame when stuff doesn’t run, not the concept of closed software, or the vendor of that software, or Microsoft for the platform design. I’m glad WINE exists (and it’s code is well written) but it’s fighting a war from a weak position.
I want up to date libs, and some containerization on things exposed. Containers doesn’t have to mean duplicate old libs. We can have the best of both. If it’s open.
If it’s closed, it all gets bloated and stale and crusty.
If taken to either extreme it can become problematic. On one hand you try to dynamically link everything and you can end up with huge issues when you try to upgrade one of those common dependencies. On the other hand if you statically link everything (or have separate copies of dynamically linked libraries) you have a lot more disk and memory usage.
So the sensible thing is to take the middle road. Dynamically link all of your system packages, like your desktop environment, core utilities, etc, and containerize the rest of your apps. That way all of your riskier applications (closed source, or stuff with a big stack surface like a browser) can have a layer of security between it and the rest of the OS and also have a separate set of libraries that the vendor ships with. You’ll pay a small penalty for duplicate libraries, but you should only have a handful of them.
I think every containerized application should have duplicate libs. You want this exposed applications to have whatever the vendor has vetted, and you want to make sure it’s only interacting with other containerized libs.
This is basically solved for decades with package management. Dependences are in a database and things are rebuilt accordingly.
Somethings are already run in a container. Without lib duplication. I agree that there is an argument for more network facing things to be run like that.
This isn’t just static vs dynamic, but the whole app folder things again.
Far too often this whole thing is just an excuse to avoid packaging properly. Instead they gift wrap their environment of old libs. Closed stuff has no choice, so always champions ways of shipping with old libs. Really pisses me of when open stuff does it so they can avoid the work of porting to the current version of say, Python. When it’s Docker, it’s often most of some hacked old Debian/Ubuntu. It’s the exact opposite of “reproducable builds” and means that software will never make it into things like BuildRoot or Yocto. Never mind Debian/etc proper.
Closed source could document system dependencies, it just can’t be rebuilt on demand to target a different set of libraries. So it’s usually easier to give it a separate set of libraries instead of expecting the system to accommodate it.
It kinda goes both ways. To properly work with package management, it needs to support the oldest and newest versions of a library in popular distros. So for Python, that may mean Python 2.x and 3.x around the launch of Python 3. Supporting both is possible, it’s just more work for the developers for what could be considered a pretty minor benefit.
This isn’t just a Python or an interpreted language thing, it also happens for compiled shared libraries. If you need a feature from the latest libc, Debian can’t ship your package until it ships a new enough libc, but maybe it’ll ship with Ubuntu or Arch.
That said, most “system” packages are willing to go through this effort, so I expect things like KDE, git, GNU utils, etc to all use the same set of shared libraries. I think browsers are special enough that they should be containerized, if only because of the large attack surface that you should use the exact libraries they recommend instead of perhaps an older or newer one that happens to work.
Shipping your own dependencies should absolutely be the exception, not the rule, but I think it’s a good thing for some types of applications. Bloating an install from something like 30MB to 300MB is fine if it’s only for a handful of applications that tend to use a ton of resources anyway, like a browser, video game, or web service.
No, things support Python 3 or Python 2. Not seen stuff packaged with support for both. In the package dependency information is a window of versions required. 3.9 - 3.11 kind of thing.
If you need the new features in the latest libs (rarely really case) get involved with getting the latest lib into the new stable release. Yes, in the mean time, maybe ship your own version, but it’s makes a right old mess if that is done all the time for all dependencies.
If it’s for a handfull of apps, maybe that’s acceptable for a short period of time. Closed apps have no choice but to be messy.
Me, I’m in little rush for latest and greatest. Rolling with Debian Testing is plenty new enough for my desktop. For servers, I want old and stable anyway, so Debian Stable.
I meant that maybe one target doesn’t support Python 3 (e.g. older Debian) and another doesn’t support Python 2 (i.e. a bleeding edge distro). There were tons of libraries that supported both at the time, though it wasn’t due to distro compatibility. I used it as an example because the transition was so rocky and most people were aware of the issue.
But the same idea can happen when using newer compiler features or something. If I need a newer version of something than a distro supports, my options are:
Of those options, the last is a lot more attractive and requires the least work. Ideally these are rare exceptions, but it should absolutely be an option.
The Python 2 -> 3 transission was a mess. Same with GTK2 -> GTK3. Lots of young developers think they can just ditch legacy and do all new and shiny. Learning it is a mistake to disregard legacy is part of maturing as a developer.
Debian has packaged libs as python-libname and python3-libname. There is a few python2-libname but python2 is on the path to removal.
Right, but they didn’t support Python 3 for a while. And with 5-year release cycles, it’s entirely possible that you’ll have a situation where some distros don’t support Python 3 and some don’t support Python 2. Or any other platform (e.g. maybe you need a shiny new C++ feature, but certain distros don’t support that version yet).
So those distros can either not include those packages (seems to be the current approach), add a feature mid-cycle (pretty rare, though people do make repos that do so), or allow some method of bundling it with its unsupported dependencies.