Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a recommended approach to caching? #898

Open
indigoviolet opened this issue Feb 19, 2022 · 5 comments
Open

Is there a recommended approach to caching? #898

indigoviolet opened this issue Feb 19, 2022 · 5 comments
Labels

Comments

@indigoviolet
Copy link

I would like direnv loads to be even faster.

Most of my envrc files are layout_poetry (setting up virtual environments for Python packages); I use https://github.com/purcell/envrc in Emacs to handle these loads and sometimes restarting Emacs with many projects/buffers open causes many direnv loads to happen.

Describe the solution you'd like
I have seen discussion like #173, https://dev.to/allenap/some-direnv-best-practices-actually-just-one-4864#fn4, typically in the context of nix. Many primitives supporting caching also appear to exist dump, show_dump, apply_dump, $DIRENV_DIFF etc.

I could imagine using them to put together a simple cache that is invalidated either explicitly (just delete the cache file?) or perhaps once a day.

Does such a thing already exist? Has this direction been abandoned except in the case of nix-related usage?

@indigoviolet
Copy link
Author

I ended up putting together an implementation: https://github.com/indigoviolet/direnv-cache .

I would love any feedback. I also came up with some suggestions for additions to direnv that would help with such a feature.

@indigoviolet
Copy link
Author

@zimbatm apologies for mentioning, but i wanted to make you aware of ^

@zimbatm
Copy link
Member

zimbatm commented Apr 11, 2022

Thanks, I had a quick look. My first impression is that you really understand how direnv works, it's nice.

One of the dangers of caching is that if it's not precise enough, the user cannot trust the result and ends up invalidating the cache all the time, just in case. Or you have to be aware of the inner workings of the cache and remember to manually invalidate it when touching such-and-such files.

But, I recognise the need to have everything fast, it's nice :)

There are other projects like https://github.com/nix-community/nix-direnv that try to do similar things in a language-specific manner.

I'm open to add direnv export dotenv, I think it can be useful generally.

Potentially, it might be possible to find the path to the allow file and compare the mtime. If it's younger than the cache file, then you know that the user typed direnv reload and can invalidate the cache.

@jankatins
Copy link

I ended up caching the result of the poetry venv call. But something more generic like a

local envdir=$(cache_output --expect-dir --depend-on=pyproject.toml --ignore-stderr -- poetry env info --path)

would be nice...

layout_poetry() {
  if [[ ! -f pyproject.toml ]]; then
    log_error 'No pyproject.toml found.  Use `poetry new` or `poetry init` to create one first.'
    exit 2
  fi
  
  cache_dir=${HOME}/.cache/direnv_layout_poetry
  pyproject_hash=$(sha1sum pyproject.toml | awk '{print $1}')
  mkdir -p "${cache_dir}"
  cachefile=${cache_dir}/${pyproject_hash}

  local envdir=""

  if [[ -f "${cachefile}" ]]; then
    local envdir_cached=$(< "${cachefile}")
    if [[ -d ${envdir_cached} ]] ; then
      local envdir_candidate=${envdir_cached};
      if [[ ! -f ${envdir_candidate}/bin/activate ]] ; then
        rm "${cachefile}"
        log_error "Cache of poetry env location (${envdir_candidate}) is invalid, deleted cache."
      else
        envdir=${envdir_candidate}
        #log_status "Using cached poetry venv path from '${cachefile}': '${envdir}'"
      fi
    fi
  fi

  if [[ "x${envdir}" == "x" ]] ; then
    envdir=$(poetry env info --path 2>/dev/null);
    log_status "Caching poetry venv path '${envdir}' to '${cachefile}'"
    printf '%s' "${envdir}" > "${cachefile}"
  fi

  if [[ -f ${envdir}/bin/activate ]] ; then
    source "${envdir}/bin/activate"
    export POETRY_ACTIVE=1
    log_status "Using poetry venv '${envdir}'."
  else
    log_error "Poetry venv does not yet exist, use 'poetry install' to create it."
  fi
}

damncabbage added a commit to damncabbage/dotfiles that referenced this issue Oct 13, 2023
@jankatins
Copy link

Here are two more solutions to caching:

  • https://github.com/dimo414/bkt (native binary, downloadable from gihthub release) will cache a cli command output which then can be sourced as usual. Supports all the features I need to refresh the values only when needed (file mod times, time based,...)
  • https://github.com/direnv/direnv/wiki/Cache-command-output, based on the above idea, my original solution to cache output, in pure bash, supports invalidation on hash changes of a file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

3 participants