Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vendoring dependencies? #301

Closed
rochacbruno opened this issue Feb 27, 2020 · 12 comments
Closed

Vendoring dependencies? #301

rochacbruno opened this issue Feb 27, 2020 · 12 comments
Labels
Milestone

Comments

@rochacbruno
Copy link
Member

rochacbruno commented Feb 27, 2020

Recently python-box broke and also we had to pin python-dotenv, so given the way dynaconf is embedded on libraries and applications looks like a good idea to vendor 3 libs.

  • toml
  • python-box
  • python-dotenv
  • click
@rochacbruno
Copy link
Member Author

maybe replace box with a simpler implementation.

@rochacbruno rochacbruno added this to the 3.0.0 milestone Mar 10, 2020
@rochacbruno rochacbruno added the HIGH High Priority label Mar 10, 2020
rochacbruno added a commit that referenced this issue May 22, 2020
rochacbruno added a commit that referenced this issue May 22, 2020
rochacbruno added a commit that referenced this issue May 22, 2020
rochacbruno added a commit that referenced this issue May 22, 2020
rochacbruno added a commit that referenced this issue Jun 22, 2020
Shortlog of commits since last release:

    Bernardo Gomes (2):
          Adding f string (#319)
          Added little information about how dev into this project. (#321)

    Bruno Rocha (18):
          Release version 3.0.0rc1
          Better exception handling on env_loader (#316)
          Add support for config aliases (#332)
          Add ENVLESS_MODE (#337)
          Fix #272 allow access of lowercase keys (#338)
          Fix #298 allow auto complete for editors and console (#339)
          Vendoring dependencies Fix #301 (#345)
          Clean tox installation for local testing (#346)
          Validator improvements on conditions (#353)
          Add note about quoting in env vars (#347)
          DEPRECATED global settings object.
          DEPRECATED global settings object. (#356)
          Lowecase read allowed by default (#357)
          Merge branch 'master' of github.com:rochacbruno/dynaconf
          envless by default - breaking change ⚠️ (#358)
          dotenv is no more loaded by default (#360)
          No more loading of `settings.*` by default (#361)
          NO more logger and debug messages (#362)

    Douglas Maciel d'Auriol Souza (1):
          Insert news validator conditions: (len_eq, len_ne, len_min, len_max, contd) (#328)

    Jeff Wayne (1):
          s/DYNACONF_ENV/ENV_FOR_DYNACONF (#335)

    Marcos Benevides (1):
          Fix minor typo in Flask extension docs (#318)

    Nicholas Nadeau, Ph.D., P.Eng (1):
          Fixed comma typo (#334)

    sfunkhouser (1):
          Add option to override default mount_point for vault (#349)
@pavel-kalmykov
Copy link

Hello.

I stumbled upon your project and wondered: why would you rather have your dependencies "vendored" (that is, to keep a full copy of them inside this repo) instead of using a dependency management tool? If you have some time to answer this please let me know because I am curious about the reasoning to do so.

@rochacbruno
Copy link
Member Author

Hi @pavel-razgovorov

That decision was made because dynaconf started being packaged for many O.S including debian, RHEL, Arch.

To enable dynaconf to be a .rpm package for example it would require also the packaging of every dependency individually, so we decided to vendor it (the same way other projects does ex. https://github.com/pypa/pipenv/tree/master/pipenv/vendor for the same reason)

@rochacbruno
Copy link
Member Author

rochacbruno commented Oct 5, 2021

NOTE: The vendor_src is where the full copy is kept

The vendor contains a minified version to save disk space, the build process includes only the minified vendor

@Apteryks
Copy link

Apteryks commented Apr 19, 2022

On some system such as Debian and GNU Guix, vendoring or bundling is against the packaging policy, so this won't help there. No matter how you look at it, bundled dependencies are almost always a bad idea and is akin to technical debt (what happens the day a bundled dependency is no longer supported by the current Python? You are either forced to do a large maintenance sprint to update them, or do development on the old bundled copies to bring them to compatibility).

@rochacbruno
Copy link
Member Author

@Apteryks once Python gets builtin TOML and YAML I am open to write a builtin implemention of Box and then remove all other dependencies.

@rochacbruno
Copy link
Member Author

@Apteryks vendoring is common on python projects, pip for example has some libraries vendored. As long as the licenses permits it is allowed.

@gotmax23
Copy link
Contributor

Hi!, @rochacbruno! Would you be willing to reconsider this decision? I've responded inline to clarify my opinion. Thanks.

@rochacbruno:

@Apteryks vendoring is common on python projects,

This does not align with my experience in the Python ecosystem.

pip for example has some libraries vendored. As long as the licenses permits it is allowed.

This is a false equivalency. Pip has very specific and justified reasons for vendoring its dependencies. This is needed for bootstrap reasons and for safety reasons (other packages in the environment should not break the package installer). Even so, pip provides a way to unbundle the dependencies. Dynaconf does not have any of these problems, and the vendoring is not optional without patching out a bunch of hardcoded import paths, etc.

@Apteryks:

On some system such as Debian and GNU Guix, vendoring or bundling is against the packaging policy, so this won't help there.

This affects Fedora as well. Fedora discourages vendored dependencies. In cases where bundling is warranted (this is not one IMO), there's extra metadata that needs to be added and licenses that need to be accounted for. The vendored deps make it harder for us to package your software.

I came across this when submitting a PR to update Fedora's python3-dynaconf package and figured I'd engage with upstream on this.

No matter how you look at it, bundled dependencies are almost always a bad idea and is akin to technical debt (what happens the day a bundled dependency is no longer supported by the current Python? You are either forced to do a large maintenance sprint to update them, or do development on the old bundled copies to bring them to compatibility).

Yes, vendored dependencies can lead to technical debt. They also need to be babysat for security vulnerabilities, critical bugs, etc. They encourage bad development practice (e.g. just vendor an ancient dependency version and forget about it until a problem pops up and you need to scramble to fix a bunch of incompatabilities to get a security fix). This is less of a problem with normal Python dependency management. I almost always prefer to use packaged versions of dependencies instead of hidden vendored copies.

@gotmax23
Copy link
Contributor

once Python gets builtin TOML and YAML I am open to write a builtin implemention of Box and then remove all other dependencies.

Python 3.11 and above have tomllib. I don't see the problem with external dependencies on tomli or pyyaml/ruamel.yaml.

@rochacbruno
Copy link
Member Author

@gotmax23 this is going to change on the release 4.0, not sure when, but 4.0 will be a metapackage pip install dynaconf will bring [dynaconf-core, dynaconf-toml, dynaconf-yaml, dynaconf-jinja, etc..`] so there will be some plugins and each plugin will bring its own deps.

@gotmax23
Copy link
Contributor

each plugin will bring its own deps.

What do you mean by "bring its own deps?" Will those be vendored?

@rochacbruno
Copy link
Member Author

@gotmax23 for the plugins under the dynaconf/ org it will no more vendor anything, for external plugins it will depend on plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants