Skip to content

Commit

Permalink
Renaming: kxgit -> snow
Browse files Browse the repository at this point in the history 
`snowdrop` is an alias as well.
Also change repository directory naming: .git/ -> .snow/
  • Loading branch information
@rcoup
rcoup committed Jul 2, 2019
1 parent b13b442 commit 24bd75c
Show file tree
Hide file tree
Showing 10 changed files with 94 additions and 94 deletions.
4 changes: 1 addition & 3 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
*.egg-info

*.pyc
*.pyo
*.py?
/data
/venv
/vendor
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ USER snowdrop
WORKDIR /data

# smoke test
RUN kxgit --version
RUN snow --version

ENTRYPOINT ["/venv/bin/tini", "--"]
CMD ["/venv/bin/kxgit"]
CMD ["/venv/bin/snow"]
71 changes: 35 additions & 36 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
SnowDrop
--------
Project Snowdrop
----------------
```
@@@
@
@
@
@@
@@@@
@@@@@
@@@@@@@@@
@@@@@@@@@@@@
@@@@@@@@@@@@@@
@@@@@@@@@@@ @@@@
@@@@@@@@@@ @@@@
@@@@@ @@@@@ @@@@
@@@@ @@@@@@ @@@@
@@@ @@@@@@ @@@@
Expand All @@ -23,7 +21,7 @@ SnowDrop
@
```

### Koordinates git-like Sync Client (experiments)
### Distributed version-control for datasets

## Installing

Expand All @@ -34,11 +32,11 @@ You need [Homebrew](https://brew.sh/) installed.
#### For just general running/updating
```console
$ brew tap --force-auto-update koordinates/snowdrop git@github.com:koordinates/snowdrop.git
$ brew install --HEAD kxgit
$ brew install --HEAD snowdrop

# check it's working (these should be the version numbers)
$ kxgit --version
kxgit proof of concept
$ snow --version
Project Snowdrop
GDAL v2.4.1
PyGit2 v0.28.2; Libgit2 v0.28.0
```
Expand All @@ -47,7 +45,7 @@ PyGit2 v0.28.2; Libgit2 v0.28.0
```
$ git clone git@github.com:koordinates/snowdrop.git --branch=gitlike-2019
$ cd snowdrop
$ brew install --only-dependencies --HEAD HomebrewFormula/kxgit.rb
$ brew install --only-dependencies --HEAD HomebrewFormula/snowdrop.rb
# create our virtualenv
$ virtualenv --python=python3.7 ./venv
Expand Down Expand Up @@ -77,17 +75,18 @@ $ pip install pygdal=="$(gdal-config --version).*"
$ pip install -r requirements-dev.txt
$ rm venv/lib/python*/site-packages/no-global-site-packages.txt
# install kxgit
# install snowdrop
$ pip install -e .
# make kxgit globally accessible
$ ln -sf $(pwd)/venv/bin/kxgit /usr/local/bin/kxgit
# make snowdrop globally accessible
$ ln -sf $(pwd)/venv/bin/snow /usr/local/bin/snow
$ ln -sf $(pwd)/venv/bin/snowdrop /usr/local/bin/snowdrop
# quit the virtualenv
$ deactivate
# check it's working (these should be the version numbers)
$ kxgit --version
kxgit proof of concept
$ snow --version
Project Snowdrop
GDAL v2.4.1
PyGit2 v0.28.2; Libgit2 v0.28.0
```
Expand All @@ -100,7 +99,7 @@ Sources:
```console
$ docker build -t snowdrop .
# in repository/data directory
$ /path/to/snowdrop/kxgit-docker.sh kxgit ...
$ /path/to/snowdrop/snowdrop-docker.sh snow ...
```

## Usage
Expand All @@ -109,43 +108,43 @@ $ /path/to/snowdrop/kxgit-docker.sh kxgit ...
* With a single vector layer
* Which has a primary key
* Get the whole layer
2. Import the GeoPackage (eg. `kx-foo-layer.gpkg`) into a kxgit repository.
2. Import the GeoPackage (eg. `kx-foo-layer.gpkg`) into a Snowdrop repository.
```console
# find the table name in the GeoPackage
$ kxgit import-gpkg --list-tables kx-foo-layer.gpkg
$ snow import-gpkg --list-tables kx-foo-layer.gpkg
# import the layer (`foo_layer`) — the repo directory will be created:
$ kxgit --repo=/path/to/kx-foo-layer.git import-gpkg kx-foo-layer.gpkg foo_layer
$ cd /path/to/kx-foo-layer.git
$ snow --repo=/path/to/kx-foo-layer.snow import-gpkg kx-foo-layer.gpkg foo_layer
$ cd /path/to/kx-foo-layer.snow
```
This will create a _bare_ git repository at `/path/to/kx-foo-layer.git`. Normal git tools & commands will work in there (mostly).
This will create a _bare_ git repository at `/path/to/kx-foo-layer.snow`.

Use this repository as the directory to run all the other commands in.
3. Checkout a working copy to edit in eg. QGIS
```console
# find/check the table name in the geopackage
$ kxgit checkout --layer=foo_layer --working-copy=/path/to/foo.gpkg
$ snow checkout --layer=foo_layer --working-copy=/path/to/foo.gpkg
```
4. Editing in QGIS/etc:
* will track changes in the `__kxg_*` tables
* additions/edits/deletes of features are supported
* changing feature PKs is supported
* schema changes should be detected, but aren't supported yet (will error).
* Use F5 to refresh your QGIS map after changing the underlying working-copy data using `kxgit`.
6. With your working copy, `kxgit` commands should work if run from the `kx-foo-layer.git/` folder. Check `--help` for options, the most important ones are supported, and in some cases options are passed straight through to the underlying git command:
* `kxgit diff` diff the working copy against the repository (no index!)
* `kxgit commit -m {message}` commit outstanding changes from the working copy
* `kxgit log` review commit history
* `kxgit branch` & `kxgit checkout -b` branch management
* `kxgit fetch` fetch upstream changes.
* `kxgit merge` merge. Supports `--ff`/`--no-ff`/`--ff-only` from one merge source.
* `kxgit push`
7. Other git commands will _probably_ work if run from the `kx-foo-layer.git/` folder. eg:
* `git remote ...`. Remember simple remotes can just be another local directory.
* `git reset --soft {commitish}`
* `git tag ...`
* Use F5 to refresh your QGIS map after changing the underlying working-copy data using `snow`.
6. With your working copy, `snow` commands should work if run from the `kx-foo-layer.snow/` folder. Check `--help` for options, the most important ones are supported. In some cases options are passed straight through to an underlying git command:
* `snow diff` diff the working copy against the repository (no index!)
* `snow commit -m {message}` commit outstanding changes from the working copy
* `snow log` review commit history
* `snow branch` & `snow checkout -b` branch management
* `snow fetch` fetch upstream changes.
* `snow merge` merge. Supports `--ff`/`--no-ff`/`--ff-only` from one merge source.
* `snow tag ...`
* `snow remote ...`. Remember simple remotes can just be another local directory.
* `snow push` / `snow pull`
7. Other git commands will _probably_ work if run from the `kx-foo-layer.snow/` folder. eg:
* `snow reset --soft {commitish}`
8. If you need a remote, head to https://kxgit-gitea.kx.gd and create a repository. Add it as a remote via:
```console
$ git remote add origin https://kxgit-gitea.kx.gd/myuser/myrepo.git
# enter your gitea username/password when prompted
$ kxgit push --all --set-upstream origin
$ snow push --all --set-upstream origin
```
8 changes: 4 additions & 4 deletions docs/sqlite3-ondisk.txt
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
sqlite3 -> on-disk

feature -> blob
feature -> blob
encode as (ordered) JSON now, msgpack/protobufs/avro/thrift/etc later

tree -> UUIDs:features
generate or supply UUIDs (v4?)
we supply for Kx exports, kxgit client supplies for new features
we supply for Kx exports, snowdrop client supplies for new features

subtrees are more efficient
split on UUID parts?
ULIDs: lexigraphically sorted: https://github.com/ulid/spec <- allows deriving add-time from ID
https://blog.2ndquadrant.com/sequential-uuid-generators/ <- rolls over every X days to provide some isolation. C/Java code are both compact. Implement in Python?
Lots of other implementations of pseudo-UUIDs.
Lots of other implementations of pseudo-UUIDs.
Keeping the 10MSB of a 32bit timestamp would have ~48 day precision, 11MSB~=24 days. Conversely, rollover based on the LSB to mask? 19bits ~=6 days.

sqlite has internal rowids for tables, though:
Expand All @@ -35,7 +35,7 @@ repo layout
my-layer-name.gpkg ignore'd
my-layer-name/
meta/
version KxGit version
version Snowdrop version
schema.proto Protobuf schema
source.json Kx source information (sorted JSON)
gpkg_*.json dump of gpkg_* tables (sorted JSON)
Expand Down
5 changes: 3 additions & 2 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
setup(
name='snowdrop',
version='0.1',
description='Koordinates gitlike experiments',
description='Distributed version-control for datasets',
url='http://github.com/koordinates/snowdrop',
author='Koordinates Limited',
author_email='support@koordinates.com',
Expand All @@ -12,7 +12,8 @@
zip_safe=False,
entry_points={
'console_scripts': [
'kxgit = snowdrop.cli:cli',
'snow = snowdrop.cli:cli',
'snowdrop = snowdrop.cli:cli',
],
},
)
38 changes: 20 additions & 18 deletions snowdrop/cli.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,10 @@ def print_version(ctx, param, value):
return

import osgeo
import pkg_resources # part of setuptools
version = pkg_resources.require("snowdrop")[0].version

click.echo("kxgit proof of concept")
click.echo(f"Project Snowdrop v{version}")
click.echo(f"GDAL v{osgeo._gdal.__version__}")
click.echo(f"PyGit2 v{pygit2.__version__}; Libgit2 v{pygit2.LIBGIT2_VERSION}")
ctx.exit()
Expand Down Expand Up @@ -213,16 +215,16 @@ def import_gpkg(ctx, geopackage, table, list_tables):
repo_dir = ctx.obj["repo_dir"]
if os.path.exists(repo_dir):
repo = pygit2.Repository(repo_dir)
assert repo.is_bare, "Not a bare repository?!"
assert repo.is_bare, "Not a valid repository"

if not repo.is_empty:
raise click.ClickException(
"Looks like you already have commits in this repository"
)
else:
if not repo_dir.endswith(".git"):
if not repo_dir.endswith(".snow"):
raise click.BadParameter(
"Path should end in .git for now", param_hint="--repo"
"Path should end in .snow", param_hint="--repo"
)
repo = pygit2.init_repository(repo_dir, bare=True)

Expand Down Expand Up @@ -342,7 +344,7 @@ def checkout(ctx, branch, refish, working_copy, layer, force, fmt):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

# refish could be:
Expand Down Expand Up @@ -1311,11 +1313,11 @@ def diff(ctx):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

working_copy = _get_working_copy(repo)
assert working_copy, f"No working copy? Try `kxgit checkout`"
assert working_copy, f"No working copy? Try `snow checkout`"

db = _get_db(working_copy.path, isolation_level="DEFERRED")
with db:
Expand Down Expand Up @@ -1399,7 +1401,7 @@ def commit(ctx, message):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)
commit = repo.head.peel(pygit2.Commit)
tree = commit.tree
Expand Down Expand Up @@ -1628,7 +1630,7 @@ def fsck(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

click.echo("Checking repository integrity...")
Expand Down Expand Up @@ -1861,7 +1863,7 @@ def pull(ctx, ff, ff_only, repository, refspecs):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

if repository is None:
Expand Down Expand Up @@ -1909,7 +1911,7 @@ def log(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "log"] + list(args))
Expand All @@ -1924,7 +1926,7 @@ def push(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "push"] + list(args))
Expand All @@ -1939,7 +1941,7 @@ def fetch(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "fetch"] + list(args))
Expand All @@ -1954,7 +1956,7 @@ def branch(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "branch"] + list(args))
Expand All @@ -1969,7 +1971,7 @@ def remote(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "remote"] + list(args))
Expand All @@ -1984,7 +1986,7 @@ def tag(ctx, args):
repo = pygit2.Repository(repo_dir)
if not repo or not repo.is_bare:
raise click.BadParameter(
"Not an existing bare repository?", param_hint="--repo"
"Not an existing repository", param_hint="--repo"
)

_execvp("git", ["git", "-C", repo_dir, "tag"] + list(args))
Expand All @@ -1996,8 +1998,8 @@ def tag(ctx, args):
def clone(repository, directory):
""" Clone a repository into a new directory """
repo_dir = directory or os.path.split(repository)[1]
if not repo_dir.endswith(".git") or len(repo_dir) == 4:
raise click.BadParameter("Repository should be foo.git")
if not repo_dir.endswith(".snow") or len(repo_dir) == 4:
raise click.BadParameter("Repository should be myproject.snow")

subprocess.check_call(["git", "clone", "--bare", repository, repo_dir])
subprocess.check_call(
Expand Down
6 changes: 3 additions & 3 deletions tests/conftest.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,16 +99,16 @@ def _data_archive(name):
def data_working_copy(data_archive, tmp_path, cli_runner):
"""
Extract a repo archive with a working copy geopackage
If the geopackage isn't in the archive, create it via `kxgit checkout`
If the geopackage isn't in the archive, create it via `snow checkout`
Context-manager produces a 2-tuple: (repository_path, working_copy_path)
"""

@contextlib.contextmanager
def _data_working_copy(name, force_new=False):
with data_archive(name) as repo_dir:
if name.endswith(".git"):
name = name[:-4]
if name.endswith(".snow"):
name = name[:-5]

wc_path = repo_dir / f"{name}.gpkg"
if wc_path.exists():
Expand Down
Binary file removed tests/data/points.git.tgz
View file
Binary file not shown.
Binary file added tests/data/points.snow.tgz
View file
Binary file not shown.
Loading

0 comments on commit 24bd75c

Please sign in to comment.