Added a simple Python version of the Atlas #140

lucaspcram · 2018-06-18T21:34:34Z

See the readme under pyatlas/README.md for more information.

This PR modifies the Travis release process to publish a pyatlas distribution to PyPI.
Currently, the distribution is not GPG signed. It is possible to change this, if we feel it is necessary. However I do not believe PyPI requires it.

test function.

…k with pydoc

matthieun

Really nice addition! A few questions included. Also Is the pyatlas/pyatlas/autogen/__init__.py empty file included on purpose?
Also: Maybe add a short description + link in the main readme?

matthieun · 2018-06-19T16:06:45Z

pyatlas/package.sh

+# GNU and BSD sed have different "in-place" flag syntax
+if [ "$(uname)" == "Darwin" ];
+then
+    sed -i "" "s/version=.*/version=\"$atlas_version\",/" "$pyatlas_root_dir/setup.py"


A bit of a hack but it works! Maintaining it is my problem anyway :)

matthieun · 2018-06-19T16:14:30Z

pyatlas/pyatlas/atlas.py

+        load_method_to_call()
+
+
+class _PackedTagStore(object):


Are those included in the same file to make them "private"?

It's not the fact that they are included in the same package, rather the leading _ character is what makes them private. You can see in the geometry module that there are many classes defined there, none of which are private.

The reason I placed _PackedTagStore and _IntegerDictionary in the atlas module is because they are only called from within that module, and they don't need to be exposed to the user for import or documentation.

matthieun · 2018-06-19T16:20:34Z

pyatlas/pyatlas/atlas_entities.py

+        return frozenset(relation_set)
+
+
+class Point(AtlasEntity):


Pretty cool flyweight implementation!

I take no credit for this design. Copied straight from your PackedAtlas :)

matthieun · 2018-06-19T16:28:35Z

pyatlas/pyatlas/geometry.py

+
+def _urshift32(to_shift, shift_amount):
+    """
+    Perform a 32 bit unsigned right shift (drag in leading 0s).


matthieun · 2018-06-19T16:30:16Z

pyatlas/pyatlas/spatial_index.py

+        immutable, so this method forces a rebuild of the entire tree.
+        """
+        self.contents.append(entity)
+        self._construct_tree_from_contents()


Would it make sense to have two methods, one called "add" and another one called "finishedAdding"? That way you do not build the rtree at every call?

I suppose I could change the name to something different, just to make it more clear. I included an implementation of add in RTree to make the interface consistent across tree implementations, since the SpatialIndex.add method simply calls add on whatever its backing tree is. In the case that you have a SpatialIndex backed by a QuadTree, there is no forced tree rebuild.

The reason I implemented the RTree this way (with the forced rebuild) is because the STR packed R-tree provided by GEOS (thru Shapely) is effectively immutable once it has been "built" (you can see what I mean by "built" by checking out this GEOS class, "building" occurs as a side effect of queries to the tree, see the query method in the linked code ).
So effectively, the only way to add elements to the tree once it has been "built" is to rebuild the tree from scratch. However, this limitation does not apply when constructing the tree for the first time, which is why I provided the initial_entities parameter in the constructor of the RTree class.

Ideally, users will not be building their own RTrees anyway. The Atlas lazily constructs the spatial indices it needs at query time. However, I made the SpatialIndex interface public in case someone really wants to extend the spatial capabilities of an Atlas without necessarily modifying the Atlas class itself. Using the provided interface, one could attach a custom SpatialIndex with a custom subset of features to an arbitrary Atlas.

It would be awesome to add all this as a comment in that method!

Haha sounds good, I'll clean this up to be a bit more concise and to the point.

matthieun · 2018-06-19T16:32:04Z

pyatlas/unit_tests/test_atlas.py

+from pyatlas.geometry import Rectangle
+
+
+class AtlasTest(unittest.TestCase):


If those tests fail, are they build breaking?

Yes, failing unit tests cause the build to abort with a diagnostic detailing the failed test and the cause of the failure. It's very similar to the JUnit setup we have in Java Atlas.

lucaspcram · 2018-06-19T16:46:40Z

Glad to get this thing started!

Yes the __init__.py is a pythonism that allows the interpreter to detect a directory as a python package. Here is some info on the subject

matthieun · 2018-06-19T16:49:45Z

I was wondering if it was ok to have it in the autogen folder. Thanks!

lucaspcram · 2018-06-19T17:37:37Z

You have to, since the autogen folder is included as a subpackage by other modules in the main pyatlas package (eg. You can see at the top of atlas.py that it has lots of import autogen.FOO_pb2 statements). Hope that clears everything up! :-)

matthieun · 2018-06-19T17:50:48Z

README.md

@@ -160,6 +160,10 @@ final Atlas atlas2;
 new PackedAtlasCloner().cloneFrom(new MultiAtlas(atlas1, atlas2)).save(new File("/path/to/file.atlas"));
 ```

+# PyAtlas
+
+`Atlas` features a lightweight Python version of the core functionality. The `pyatlas` is available as a Python package from PyPI, but can also be built and installed locally. Check out the `pyatlas` subfolder and the `pyatlas/README.md` file for more information!


You could link it directly here using ... [Readme file](pyatlas/README.md) ...

Ah ok let me add that. I did not realize you could do relative URLs.

jwpgage

A couple questions, but looks really good! Thanks for the thorough tests.

jwpgage · 2018-06-19T23:20:36Z

pyatlas/pyatlas/pyatlas_globalfunc.py

+    """
+    Print a welcome message!
+    """
+    print "Hello Atlas!"


Is this just a default global function to be replaced in future?

Good question :) It's pretty much just there for the Getting Started section of the README, which has the user call it to test that the package built correctly. In the future this module may have additional function definitions.

jwpgage · 2018-06-19T23:21:40Z

pyatlas/package.sh

+#!/usr/bin/env bash
+
+# THIS ENTIRE SCRIPT IS A MASSIVE HACK
+# THIS SHOULD REALLY BE DONE WITH GRADLE


I really need to remove this comment... :P

jwpgage · 2018-06-19T23:24:30Z

pyatlas/unit_tests/test_atlas.py

+        self.assertEquals({atlas.point(3)}, test_results)
+
+        test_results = atlas.points_at(geometry.location_with_degrees(38, -118))
+        self.assertEquals({atlas.point(1)}, test_results)


Have these been tested and verified against the Java version?

There is no automatic verification, but I have tested it manually against the Java version with the help of a script not included in this PR. Based on my tests the query results are consistent. But more tests are always welcome!

MikeGost

Really nice work!

MikeGost · 2018-06-20T00:38:13Z

pyatlas/pyatlas/atlas.py

+        return len(self.word_to_index)
+
+    def word(self, index):
+        # TODO this could throw a KeyError, how to handle?


Something to address or ignore for now?

I changed the way that class is used internally, so I believe that comment is outdated now. Good catch.

lucaspcram added 30 commits May 25, 2018 09:24

Initial folder and README

d1dbb70

Added build script and build gradle target for pyatlas. Added a simple

9875e62

test function.

Working on build proto integration

6d2f193

Added a clean script

dc59474

buildPyatlas target now compiles the proto files into Python metaclasses

b818fbf

Updated README

2a9ba0f

pyatlas module now autoinstalls dependencies

66538eb

Added test function to print zipentries

306ea90

Modifications to build and clean scripts

adc262b

Restructured autogen location. Added atlas class.

4211919

Fixed typos in README

3d0d9ee

Basic support for loading atlas metadata

43b8cf9

Updated travis build script

551e87d

Fix for protoc not found in travis build test.

7f94093

More travis testing

58f345c

Update to build script.

609bd80

Fixed downloaded protoc binary to be 64 bit

27380c6

Build script now aborts on any failure

0a04ac8

Implemented flow for formatting and unit testing.

62abda1

Added shellcheck to travis

e14434e

Test shellcheck on travis

f7cb31e

Changed script back

d010a88

Change to abort travis build if shellcheck fails

91d4da4

One more travis test

12626e0

Fixed script

2305398

Added print message to python formatter

f44aff4

Refactored format checking. Should fail travis.

8710ac8

Fixed format error.

753d80b

Updated format script

b1d01e5

Added a wrapper metadata class

d8600c0

lucaspcram added 11 commits June 18, 2018 08:31

Lots of updates to docs

bfbc08d

Updated license copy process

87da4d0

Setup travis to publish pyatlas

f44a0e0

Fixed a shellcheck issue.

7a2812a

Made pyatlas version dynamic

e0d8969

Added check for OS to choose correct sed syntax

9b773cb

Fixed annoying issue with find exec and pydoc. find exec will not wor…

317d064

…k with pydoc

README update

003abf4

README update

9b2cd3a

Checked doc into repo for convenience

7905405

Removed docs from repo.

384799b

lucaspcram requested review from jwpgage, MikeGost and matthieun June 18, 2018 22:15

matthieun added the New Feature label Jun 19, 2018

matthieun reviewed Jun 19, 2018

View reviewed changes

Updated main README with a note about Pyatlas

97ffa80

matthieun reviewed Jun 19, 2018

View reviewed changes

lucaspcram added 2 commits June 19, 2018 10:54

Added links to README

032c0bb

Updated doc comments in _RTree

fe1967d

jwpgage approved these changes Jun 19, 2018

View reviewed changes

MikeGost approved these changes Jun 20, 2018

View reviewed changes

jwpgage merged commit 9529b5d into osmlab:dev Jun 20, 2018

lucaspcram deleted the pyatlas branch June 20, 2018 15:30

matthieun modified the milestone: 5.1.3 Jul 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a simple Python version of the Atlas #140

Added a simple Python version of the Atlas #140

lucaspcram commented Jun 18, 2018 •

edited

Loading

matthieun left a comment •

edited

Loading

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

matthieun Jun 19, 2018

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018 •

edited

Loading

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

lucaspcram commented Jun 19, 2018

matthieun commented Jun 19, 2018

lucaspcram commented Jun 19, 2018 •

edited

Loading

matthieun Jun 19, 2018

lucaspcram Jun 19, 2018

jwpgage left a comment

jwpgage Jun 19, 2018

lucaspcram Jun 20, 2018

jwpgage Jun 19, 2018

lucaspcram Jun 20, 2018

jwpgage Jun 19, 2018

lucaspcram Jun 20, 2018

MikeGost left a comment

MikeGost Jun 20, 2018

lucaspcram Jun 20, 2018

		from pyatlas.geometry import Rectangle


		class AtlasTest(unittest.TestCase):

Added a simple Python version of the Atlas #140

Added a simple Python version of the Atlas #140

Conversation

lucaspcram commented Jun 18, 2018 • edited Loading

matthieun left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucaspcram Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucaspcram commented Jun 19, 2018

matthieun commented Jun 19, 2018

lucaspcram commented Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwpgage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MikeGost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucaspcram commented Jun 18, 2018 •

edited

Loading

matthieun left a comment •

edited

Loading

lucaspcram Jun 19, 2018 •

edited

Loading

lucaspcram commented Jun 19, 2018 •

edited

Loading