Skip to content

Commit

Permalink
Merge branch 'master' into d3fc_plugin
Browse files Browse the repository at this point in the history
  • Loading branch information
redbearsam authored Feb 19, 2019
2 parents 1070b79 + 867f89d commit 0474191
Show file tree
Hide file tree
Showing 281 changed files with 7,953 additions and 3,079 deletions.
5 changes: 3 additions & 2 deletions .appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ max_jobs: 8

image:
- Visual Studio 2015
- Visual Studio 2017
# - Visual Studio 2017

configuration:
- Release
- Debug
# - Debug

platform:
- x64
Expand All @@ -30,6 +30,7 @@ init:

before_build:
- cmd: |-
cd cpp\perspective
mkdir build
cd build
cmake --version
Expand Down
10 changes: 9 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ Gemfile.lock
Vagrantfile
/packages/sigma
/packages/fin-hypergrid
coverage

.DS_Store

Expand All @@ -124,4 +125,11 @@ website/static/css/material.dark.css
obj
packages/*/cjs
cppbuild
.coverage
docsbuild
.coverage

# docs generated
docs/_build
docs/modules.rst
docs/perspective.*.rst
docs/python
8 changes: 7 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ node_js: "8"
sudo: required

matrix:
fast_finish: true
include:
- node_js: "8"
language: node_js
Expand All @@ -24,6 +25,11 @@ matrix:
env: TEST=CPP_OSX
os: osx

allow_failures:
- language: c++
env: TEST=CPP_OSX
os: osx

cache:
yarn: true
directories:
Expand All @@ -45,7 +51,7 @@ install:
- yarn

script:
- if [ "$TEST" = "WASM" ]; then yarn lint; fi
- if [ "$TEST" = "WASM" ]; then PSP_DOCKER=1 yarn lint; fi
- if [ "$TEST" = "WASM" ]; then PSP_DOCKER=1 yarn build; fi
- if [ "$TEST" = "WASM" ]; then PSP_DOCKER=1 yarn test --quiet; fi
- if [ "$TEST" = "CPP" ]; then PSP_DOCKER=1 yarn build_cpp; fi
Expand Down
46 changes: 46 additions & 0 deletions API.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Perspective API design notes

### Goals
- Make the core C++ library an extensible base for developers to build on using different languages.
- Port features previously segregated to Javascript into C++, thereby removing dependency on JS for the core library.
- Move business logic into C++, thus allowing the purpose of the JS library to act as a bound wrapper rather than a dependent extension.

This document aims to elaborate on the design of the core library API, as well as information designed to aid the core developer in creating new features/extending the library into new language bindings.

### Components
- Table: C++ class that will implement the features previously found in the `perspective.js` table, including all prototype methods and class members.
- t_table: C++ class that manages the underlying operations for `Table`.
- View: C++ class that will implement all features and members previously found in the `perspective.js` view.
- pool: A collection of `gnode`(s) managed in C++.
- gnode: A "graph node" that contains any number of `t_table`s, although in implementation we only assign one `t_table` to one `gnode`.

### Steps for porting
1. Transliterate JS to C++: liberal use of `embind` and `val` in order to start porting features previously segregated to Javascript into C++.
2. Make C++ functions portable: start removing the use of all JS-dependent data structures, members, and methods in order to allow for C++ portability.
3. Add the translation layer: now that C++ functions expect generic C++ parameters (and not a `val`, on which any number of arbitrary JS-only operations may be called), add the translator layer that converts JS input into what the portable C++ API expects, as well as converts the output into something suitable for JS.
4. Convert the JS library into simple function calls: though the `perspective.js` API will not change, the underlying implementation for methods will simply return the output of the C++ method, with translation where necessary.

### C++
`View`: class which will implement all methods from the `perspective.js` version of view.
- Expects C++ data structure input and returns native data structures.
- Should be as portable and extensible as possible.
- Should abstract away concepts like pool and gnode.

`Table`: class which implements all methods from the `perspective.js` table.
- Expects C++ data structure input and returns native data structures.
- Should be as portable and extensible as possible.

### Translation/Helper layer
`make_view`: helper function in `emscripten.cpp` that returns a `std::shared_ptr<View<CONTEXT_T>>`.
- Should construct the underlying `View` object.
- Should expect native C++ data structures as input.

*TBD: how should we implement the translation layer?*
- As separate methods that are called, which makes `make_view` extremely dependent on side-effects?
- As a layer in `perspective.js`, which makes it harder to create C++ native data structures?
- As a new binding in the C++ `src` folder, allowing for a place for developers to add their own language bindings?

### Javascript
`View`: the view class on the Javascript side should be a wrapper around calls to the Emscripten `__MODULE__`.
- Should maintain an internal reference to the C++ `View` object created on instantiation.
- Methods should call the corresponding method on the C++ `View`, and return output that is ready for JS consumption.
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,24 @@
# Changelog

## [0.2.16] - 2019-02-19
### Added
* [#431](https://github.com/jpmorganchase/perspective/pull/431) Added `clear()` and `replace()` APIs to `perspective` and `<perspective-viewer>`.
* [#435](https://github.com/jpmorganchase/perspective/pull/435) Added `to_arrow()` method to `view()` for writing Apache Arrow `ArrayBuffer`s.
* [#436](https://github.com/jpmorganchase/perspective/pull/436) New module `perspective-phosphor`, which adds bindings for the Phosphor.js framework.

### Fixes
* [#434](https://github.com/jpmorganchase/perspective/pull/434) Deprecated `[column]` sort syntax for `perspective` and `<perspective-viewer>`.

### Internal
* [#426](https://github.com/jpmorganchase/perspective/pull/426) Refactored C++ projects into separate repo structure.
* [#413](https://github.com/jpmorganchase/perspective/pull/413) Moved structure of `view()` to C++.

## [0.2.15] - 2019-02-07
### Fixes
* [#416](https://github.com/jpmorganchase/perspective/pull/416) Fixed highcharts bug which caused `null` groups to not render.
* [#419](https://github.com/jpmorganchase/perspective/pull/419) Fixed regression in cross-origin loading.
* [#421](https://github.com/jpmorganchase/perspective/pull/421) Fixed JSON/CSV loading when columns contain mixed numeric/string values.

## [0.2.14] - 2019-02-04
### Added
* [#408](https://github.com/jpmorganchase/perspective/pull/408) Added `flush()` method to `<perspective-viewer>`
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Perspective

[![Build Status](https://travis-ci.org/jpmorganchase/perspective.svg?branch=master)](https://travis-ci.org/jpmorganchase/perspective)
![https://ci.appveyor.com/api/projects/status/github/jpmorganchase/perspective?svg=true](https://ci.appveyor.com/api/projects/status/github/jpmorganchase/perspective?svg=true)
[![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/jpmorganchase/perspective)
[![npm](https://img.shields.io/npm/v/@jpmorganchase/perspective.svg?style=flat-square)](https://www.npmjs.com/package/@jpmorganchase/perspective)
[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)
Expand Down
47 changes: 38 additions & 9 deletions CMakeLists.txt → cpp/perspective/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
cmake_minimum_required (VERSION 2.8.11)
project (psp)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/modules/" ${CMAKE_MODULE_PATH} )
set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/../../cmake/modules/" ${CMAKE_MODULE_PATH} )

if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
set(WIN32 ON)
Expand Down Expand Up @@ -51,6 +51,7 @@ option(PSP_CPP_BUILD "Build the C++ Project" OFF)
option(PSP_CPP_BUILD_TESTS "Build the C++ Tests" OFF)
option(PSP_PYTHON_BUILD "Build the Python Bindings" OFF)
option(PSP_CPP_BUILD_STRICT "Build the C++ with strict warnings" OFF)
option(PSP_BUILD_DOCS "Build the Perspective documentation" OFF)


if (NOT DEFINED PSP_WASM_BUILD)
Expand Down Expand Up @@ -128,6 +129,12 @@ if(CMAKE_BUILD_TYPE_LOWER STREQUAL debug)
else()
message(WARNING "${Cyan}Building RELEASE${ColorReset}")
endif()

if(PSP_BUILD_DOCS)
message(WARNING "${Cyan}Building Perspective Documentation${ColorReset}")
else()
message(WARNING "${Cyan}Skipping Perspective Documentation${ColorReset}")
endif()
#######################
include_directories("${CMAKE_SOURCE_DIR}/src/include")

Expand Down Expand Up @@ -179,7 +186,7 @@ if (PSP_WASM_BUILD)
set(SYNC_MODE_FLAGS "-s ENVIRONMENT=node -s BINARYEN_ASYNC_COMPILATION=0 -s BINARYEN_METHOD='\"native-wasm\"' -s WASM=1")
set(ASYNC_MODE_FLAGS "-s ENVIRONMENT=worker -s BINARYEN_ASYNC_COMPILATION=1 -s BINARYEN_METHOD='\"native-wasm\"' -s WASM=1")
set(ASMJS_MODE_FLAGS "-s ENVIRONMENT=worker -s LEGACY_VM_SUPPORT=1 -s BINARYEN_ASYNC_COMPILATION=0 -s BINARYEN_METHOD='\"asmjs\"' -Wno-almost-asm -s WASM=0")
else()
elseif(PSP_CPP_BUILD OR PSP_PYTHON_BUILD)
#####################
# VANILLA CPP BUILD #
#####################
Expand Down Expand Up @@ -209,7 +216,7 @@ else()
find_package(TBB)
if(NOT TBB_FOUND)
message("${Red}TBB could not be located${ColorReset}")
psp_build_dep("tbb" "cmake/TBB.txt.in")
psp_build_dep("tbb" "../../cmake/TBB.txt.in")
else()
message("${Cyan}Found tbb in ${TBB_INCLUDE_DIRS} ${TBB_LIBRARY_DIRS} ${ColorReset}")
include_directories( ${TBB_INCLUDE_DIRS} )
Expand All @@ -228,7 +235,7 @@ else()
# PYTHON BINDINGS BUILD #
#########################
include_directories("/usr/local/include/python3.7m") # FIXME
include_directories("${CMAKE_SOURCE_DIR}/python/perspective/include")
include_directories("${CMAKE_SOURCE_DIR}/../../python/perspective/include")
find_package( PythonInterp 3.7 REQUIRED )
find_package( PythonLibs 3.7 REQUIRED )
include_directories( ${PYTHON_INCLUDE_DIRS} )
Expand Down Expand Up @@ -325,6 +332,7 @@ set (SOURCE_FILES
src/cpp/tree_context_common.cpp
src/cpp/utils.cpp
src/cpp/update_task.cpp
src/cpp/view.cpp
src/cpp/vocab.cpp
)

Expand Down Expand Up @@ -363,7 +371,7 @@ if (PSP_WASM_BUILD)
set_target_properties(perspective.asm PROPERTIES OUTPUT_NAME "psp.asmjs")
add_dependencies(perspective.asm perspective.sync)
endif()
else()
elseif(PSP_CPP_BUILD OR PSP_PYTHON_BUILD)
if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
set(CMAKE_SHARED_LIBRARY_SUFFIX .dylib)
endif()
Expand All @@ -372,14 +380,14 @@ else()
########################
# Python extra targets #
########################
add_library(psp SHARED ${SOURCE_FILES} ${CMAKE_SOURCE_DIR}/python/perspective/src/numpy.cpp ${HEADER_FILES})
add_library(psp SHARED ${SOURCE_FILES} ${CMAKE_SOURCE_DIR}/../../python/perspective/src/numpy.cpp ${HEADER_FILES})
target_compile_definitions(psp PRIVATE PSP_ENABLE_PYTHON=1)
target_link_libraries(psp ${BOOST_PYTHON})
target_link_libraries(psp ${BOOST_NUMPY})
target_link_libraries(psp ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
set(CMAKE_SHARED_LIBRARY_SUFFIX .so)

add_library(binding SHARED ${CMAKE_SOURCE_DIR}/python/perspective/src/python.cpp)
add_library(binding SHARED ${CMAKE_SOURCE_DIR}/../../python/perspective/src/python.cpp)
target_compile_definitions(binding PRIVATE PSP_ENABLE_PYTHON=1)
target_link_libraries(binding psp)
target_link_libraries(binding tbb)
Expand Down Expand Up @@ -414,7 +422,7 @@ if (PSP_CPP_BUILD_TESTS)
if (WIN32)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
endif()
psp_build_dep("googletest" "cmake/GTest.txt.in")
psp_build_dep("googletest" "../../cmake/GTest.txt.in")

add_subdirectory(test)
add_custom_target(gcov
Expand All @@ -429,3 +437,24 @@ if (PSP_CPP_BUILD_TESTS)
)
endif()
#############

########
# Docs #
########
if(PSP_BUILD_DOCS)
add_custom_target(doxygen)
add_custom_command(TARGET doxygen
COMMAND doxygen doxygen.conf
WORKING_DIRECTORY ../docs
COMMENT "Build doxygen xml files used by sphinx/breathe."
)

add_custom_target(docs-html ALL)
add_custom_command(TARGET docs-html
COMMAND sphinx-build -b html . _build/html
WORKING_DIRECTORY ../docs
COMMENT "Build html documentation"
)
add_dependencies(docs-html doxygen)
endif()
##########
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ t_col_name_type::t_col_name_type(const std::string& name, t_dtype type)
: m_name(name)
, m_type(type) {}

t_aggspec::t_aggspec(){}
t_aggspec::t_aggspec() {}

t_aggspec::t_aggspec(const t_aggspec_recipe& v) {
m_name = v.m_name;
Expand Down
File renamed without changes.
6 changes: 3 additions & 3 deletions src/cpp/base.cpp → cpp/perspective/src/cpp/base.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@

namespace perspective {

void psp_abort() {
void
psp_abort() {
std::cerr << "abort()" << std::endl;
std::raise(SIGINT);
}

Expand Down Expand Up @@ -315,14 +317,12 @@ is_internal_colname(const std::string& c) {
return c.compare(std::string("psp_")) == 0;
}


template <typename T>
t_dtype
type_to_dtype() {
return DTYPE_NONE;
}


template <>
t_dtype
type_to_dtype<std::int64_t>() {
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
9 changes: 2 additions & 7 deletions src/cpp/binding.cpp → cpp/perspective/src/cpp/binding.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,5 @@
#include <perspective/binding.h>
using namespace perspective;
namespace perspective {
namespace binding {




}
}
namespace binding {}
} // namespace perspective
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@
namespace perspective {

t_ctx_grouped_pkey::t_ctx_grouped_pkey()
: m_depth(0),
m_depth_set(false) {}
: m_depth(0)
, m_depth_set(false) {}

t_ctx_grouped_pkey::t_ctx_grouped_pkey(t_schema schema, t_config config)
: m_depth(0),
m_depth_set(false) {
PSP_COMPLAIN_AND_ABORT("Not Implemented");
: m_depth(0)
, m_depth_set(false) {
PSP_COMPLAIN_AND_ABORT("Not Implemented");
}

t_ctx_grouped_pkey::~t_ctx_grouped_pkey() {}
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -16,24 +16,18 @@
namespace perspective {

// Milliseconds & timezones are not currently handled
const std::string t_date_parser::VALID_FORMATS[12] = {
"%Y%m%dT%H%M%S", // ISO "%Y%m%dT%H%M%S%F%q"
"%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S", // ISO extended
"%A, %d %b %Y %H:%M:%S", // RFC 0822
"%Y-%m-%d\\%H:%M:%S"
"%m-%d-%Y",
"%m/%d/%Y",
"%m-%d-%Y",
"%m %d %Y",
"%m/%d/%Y",
"%m/%d/%y",
"%d %m %Y"
};
const std::string t_date_parser::VALID_FORMATS[12]
= {"%Y%m%dT%H%M%S", // ISO "%Y%m%dT%H%M%S%F%q"
"%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S", // ISO extended
"%A, %d %b %Y %H:%M:%S", // RFC 0822
"%Y-%m-%d\\%H:%M:%S"
"%m-%d-%Y",
"%m/%d/%Y", "%m-%d-%Y", "%m %d %Y", "%m/%d/%Y", "%m/%d/%y", "%d %m %Y"};

t_date_parser::t_date_parser() {}

bool
bool
t_date_parser::is_valid(std::string const& datestring) {
for (const std::string& fmt : VALID_FORMATS) {
if (fmt != "") {
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 0474191

Please sign in to comment.