cloudfoundry · KlapTrap · Feb 19, 2019 · Jan 25, 2019 · Jan 25, 2019 · Jan 25, 2019
diff --git a/docs/cf-api-v2-usage.md b/docs/cf-api-v2-usage.md
@@ -31,8 +31,8 @@ Stratos exercises many features of the Cloud Foundry v2 API, in order to underst
 
 ### Entity Relations & Validation
 - Cloud Foundry entities (organisation, route, service plan, etc) are stored client side in an entity store
-- The console provides methods to fetch an entity or collection of entities from the store. At this time the console states if any of the
-  entity's relations are required, for instance when fetching an organisation the spaces and services properties are required
+- The console provides methods to fetch an entity or collection of entities from the store. At this time Stratos states if any of the
+  entity's relations are required, for instance when fetching an organisation the spaces and services properties may be required
 - The console checks if the required entity/s exist in the store.
   - If they're missing they're then fetched via the API, including any required relationships
   - If they exist in the store their relations are checked. Any missing relations are fetched using the `<relation name>_url` property
@@ -41,7 +41,7 @@ Stratos exercises many features of the Cloud Foundry v2 API, in order to underst
 - Example 1
   - To filter the applications in the applications list by cf/org/space we fetch a list of organisations and state each org must have it's
     spaces property
-      - The resulting org entity will only have the spaces relation, each space will have no other relation
+      - The resulting org entity will only have the space's relation, each space will have no other relation
   - The user navigates to a space summary screen
   - The space summary screen states the space should contain applications, service instances, space quota, service binding, etc
   - Entity validation will fetch the missing relations using the `_url` of the relation
@@ -105,21 +105,20 @@ the console but also how we have to fetch data.
 - We provide a way for users of all types to view a list of users (at the cf, organisation and space levels) and their roles
 - We have to fetch this information differently between CF administrators with certain scopes and non-cf administrators
 - CF Admins can list users via `/users`
-- Non-CF admins have to fetch users via each org endpoint `/organization/<guid>/users`.
-  - We hide this feature if there are too many organisations.
-  - We have to extract user roles from the `organization` response rather than the user's relations, as the `<x>_url` referenced by the
-    entities in `/users` is only allowed for the connected user. Calls to fetch `<x>_url` for entities other than the connected user result
-    in a api permission error.
+- Non-CF admins cannot access `/users` list (aside from themselves)
+  - At the CF level we do not show a list of all users
+  - At the org level we show all users via `organizations/${guid}/users`
+  - At the space level we show all users via `spaces/${guid}/user_roles`
 
 #### Permissions
 
 - Permissions are dictated by the admin/non-admin state of the connected user, cf feature flags and all of the users organisation and space roles
-- For CF admins we skip fetching roles, as there admin state overrides roles
-- For non-admins we hit each `users/<non-admin's guid>/<role>` endpoint, there's a lot of duplicated data here
+- For CF admins we skip fetching roles, as their admin state overrides roles
+- For non-admins we hit each `users/<non-admin's guid>/<role>` endpoint, there's a lot of duplicated data (orgs and spaces) here
 
 ## API Parameters
 
-### Relations (`inline-relations-depth`, `include-relations`)
+### Relations (`include-relations`, `inline-relations-depth`)
 
 In order to support the entity validation process described above we make heavy use of the generic `inline-relations-depth` and `include-relations` parameters
 
@@ -179,6 +178,14 @@ missing in the api response. If the missing application is not already in the st
   - A route is created in a space, however cannot filter routes via space
   - Service brokers can be filtered by space, however not by organisation
 
+## Scaling
+
+We've recently started updating a lot of features to be sensitive of CFs with large collections of entities. This involves...
+- Local Lists will check the total number of results before fetching them all. If too high we refuse to show the table/cards.
+- Counts of entities are done via `results-per-page` of 1 and `total_results` instead of fetching all entities
+- Inline relations of one to many, and the count might be too high, are removed and the functionality that requires them reworked
+  - For instance space --> routes
+  - Some will remain the same, for instance service --> service plans
 
 ## API Examples
 
@@ -701,9 +708,9 @@ Endpoint | Sorted Locally | Sorted via API | Filtered Locally | Filtered via API
 `apps/${guid}/service_bindings` | `created_at`
 `apps` | __*`name`*__, __*`instances`*__, __*`disk_quota`*__, __*`memory`*__, `created_at` | - | `name`, `organisation`, `space` | `organisation`, `space`
 `organizations/${orgGuid}/spaces` | __*`name`*__, `created_at` | - | `name` | -
-`organizations`| __*`name`*__, `created_at` | - | `name` | -
+`organizations`| `name`, `created_at` | - | `name` | -
 `organizations/${guid}/users` | __*`username`*__ | - | __*`username`*__
-`spaces` | __*`name`*__
+`spaces` | `name`
 `spaces/${spaceGuid}/routes` | - | `created_at` | - | - | Yes | __*Host*__
 `spaces/${spaceGuid}/apps` | - | `created_at` | - | - | Yes | __*Name*__, __*Status*__
 `spaces/${spaceGuid}/services` | __*`name`*__ |  `created_at` | - | - |
@@ -727,6 +734,8 @@ __*``*__
 ### Inline Relations
 Below is a list of all the inline relations we use for each of the more fleshed out v3 endpoints of apps, spaces and organisations.
 
+> Note - Some of the one to many relations will be removed in the near future to support scaling, see [Scaling](#Scaling)
+
 #### Application
 Endpoint | Inline Relations
 --- | ---
@@ -784,8 +793,9 @@ In terms of api requests we also make use of...
 
 ### Cloud Foundry, Organisation and Space Summary Information
 In the `Cloud Foundry` section of Stratos we show summary pages at the CF, Org and Space levels. These consist of information from the core
-cf/org/space but also statistic such as number of users or memory used within that cf/org/sapce. The only way to fetch these stats is via
-requesting a large amount of additional data, either inline with the org/space or as separate requests.
+cf/org/space but also statistics such as number of users or memory used within that cf/org/space. 
+Some of these can be fetched by `results-per-page` of 1 and `total_results`, however others are derived and require fetching a large amount
+of additional data, either inline with the org/space or as separate requests (`Memory usage` is the count of memory per instance per app).
 Below is a table showing the stats at each level. The counts are respective of that level (number of routes in that specific cf, org or space)
 
 Stat/Info | CF Summary | Org Summary | Space Summary
@@ -798,13 +808,12 @@ Memory Usage | *yep* | *yep* | *yep*
 No. of Routes | | *yep*| *yep*
 No. of Private Domains | |*yep*
 No. of Service Instances | | *yep*| *yep*
-Last 10 updated apps | *yep* | *yep* | *yep*
 
 ## V2 Summary
 
 ### Current v2 Issues
 
-- Most lists fetch all entities up front to provide a reasonable level of sorting and filtering
+- Most lists will fetch all entities up front to provide a reasonable level of sorting and filtering
   - API provides limited sorting and filtering capabilities
   - Sorting is mostly just on `creation_date`
   - Filtering sometimes contains org, space and name, but not all
@@ -813,14 +822,15 @@ Last 10 updated apps | *yep* | *yep* | *yep*
   organisation, number of users etc) requires fetching all entities of a certain type. This can be quite a costly set of requests.
   > Desired - `counts` API which, given a filter, could sum up a selection of numerical properties of collection of entities. Some of this
     may already be possible by making individual requests with `results-per-page=1` & `total_results` and `q` filters, however a neat way to
-    combine these into a single request would be awesome
+    combine these into a single request would be awesome. For instance in a single request, for an org, fetch total number of apps, app
+    instances, routes, etc)
 - Determining an informative application state requires an additional request to the `application/<guid>/stats` endpoint.
   > Desired - It would massively improve Stratos performance if the APIs to list applications and retrieve a specific application could
     return the app stats for the application(s). If this is not possibly, an app stats call that can return stats for all running
     applications would help.
 - Fetching a list of users for non-admins can be expensive if there are many organizations.
   > Desired - In an ideal world non-admins would be able to hit `/users` and get the same response as hitting all if their visible
-    `organization/guid/users` endpoints.
+    `organization/${guid}/users` endpoints.
 - The user entities that are returned by the `/users` request contain a lot of duplicated organisation and space entities.
   > Desired - Entities are listed in a common bucket and referenced by guid inline. Think this might be how v3 works
 - The `<role>_url` in a user entity can only be called by an administrator or by the same user

diff --git a/docs/cf-api-v3.md b/docs/cf-api-v3.md
@@ -8,14 +8,27 @@ See [Cloud Foundry API v2 Feature Usage](cf-api-v2-usage.md) for v2 information
 
 ## Comparing v2 features to v3
 
+### V3 Docs
+- http://v3-apidocs.cloudfoundry.org/
+- https://github.com/cloudfoundry/cc-api-v3-style-guide
+
 ### Entity Relations... `include-relations` --> `include`
-- Like `include-relations`, `include` is used to request entity types in full in the response.
-- These are included in a bucket at the end, rather than including inline (avoid the case where same `space` entity is repeated in `apps` request)
-- Very limited implementation
+Previously...
+- When fetching an entity, any referenced child entity or list of entities were omitted. To have them included the property name was
+  provided in an `include-relations` parameter. The covered direct child entities and children of that child entity
+- Lists of entities that were bigger than 50 were simply omitted.
+Now, from my understanding, ...
+- Child entities (single or lists) are referenced by guid in the parent's `relationships` section
+  - Pagination of lists is only proposed - https://github.com/cloudfoundry/cc-api-v3-style-guide#proposal-pagination-of-related-resources
+- Child entities are also listed, with by url to fetch (like the old `<property>_url` property), in the `links` section.
+- An `include` parameter can be supplied which will add an `included` section to the entity
+- An entity in the `included` section is in the same format as the parent
+- `include` has very limited implementation
   - `/apps` only supports `space` include
   - `/organization` doesn't support any includes
   - `/space` doesn't support any includes
-- Unclear if `include` will cover chained entities aka `inline-relations-depth` from v2
+  - (see [Cloud Foundry API v2 Feature Usage - Inline-Relations](cf-api-v2-usage.md#Inline-Relations) for required)
+~- Unclear if `include` will cover chained entities aka `inline-relations-depth` from v2~
 
 ### Collections - Pagination
 - This is covered just fine (fetch a specific page, total page count, total result count, etc)
@@ -43,12 +56,11 @@ See [Cloud Foundry API v2 Feature Usage](cf-api-v2-usage.md) for v2 information
 
 ## V3 Availability
 - Stratos needs to support cloud foundry's with different api versions from many different providers and epochs
-- Currently, it looks like neither SCF (2.84.0), IBM Cloud (2.106.0) or PCFDev (2.82.0) support v3 with `includes`. PWS (2.125.0) and
- SAP (2.120.0) however do.
-
-> Note - cf-dev I haven't tested as it's unsupported on linux (major regression from PCFDev there)
-
-> Note - Couldn't find an easy way to determine the version of v3
+- Update, many common CFs we use support some kind of v3 version.
+> Note - cf-dev I haven't tested as it's unsupported on linux - see https://github.com/cloudfoundry-incubator/cfdev/issues/18 (major regression from PCFDev there)
+- ~~Currently, it looks like neither SCF (2.84.0), IBM Cloud (2.106.0) or PCFDev (2.82.0) support v3 with `includes`. PWS (2.125.0) and
+ SAP (2.120.0) however do.~~
+> Note - ~~Couldn't find an easy way to determine the version of v3~~
 
 ## Stratos Adoption of v3
 
@@ -61,33 +73,40 @@ Then Stratos should either ..
 - Support both versions at runtime, switching each endpoint from v2 to v3 at determined cut off dates
 
 ### Blocking Issues
-- `include` replacement does not have parity with v2.
-  - Not supported on all endpoints and does not cover enough `links`/relations (see [Cloud Foundry API v2 Feature Usage - Inline-Relations](cf-api-v2-usage.md#Inline-Relations))
-  - New v3 entities are not `links` (`/apps` `package`, `processes`, `route_mappings`, `environment_variables`, `droplets`, `tasks`)
-  - Not supported by common CFs used to develop with (SCF, PCFDev).
-- Cannot determine if a CF supports v3 or when it does support v3 which endpoints it has
-  - Getting the v2 version is simple, I don't know if there's correlation to v3 version
-  - Having a v3/info which published which endpoints are supported would help a lot. Including whether `include` is supported
-- Chained `links`/relations don't appear to be supported (fetch app-->space-->org all in one request)
-  - Might be wrong on this one due to the lack of current `include` integration
-- `/organizations` and `/spaces` endpoints are not completed and contain only guid, create/updated date and name (space additionally has experimental `organization`)
+- Coverage of `include` is not on par with v2 `include-relations`
+  - To meet parity it should support all entities in `relationships` and `links`, whether one to one or one to many
+  - Currently not supported on all endpoints and does not cover enough `relationships`/`links` (see [Cloud Foundry API v2 Feature Usage - Inline-Relations](cf-api-v2-usage.md#Inline-Relations))
+  - For instance new v3 entities are not `include`s (`/apps` - `package`, `processes`, `route_mappings`, `environment_variables`, `droplets`, `tasks`)
+  - ~~Not supported by common CFs used to develop with (SCF, PCFDev)~~.
+- Entities do not contain all properties that were in v2 (where functionality has not changed)
+  - Covers simple values and entities (one to one and one to many) 
+  - For instance `/organizations` and `/spaces` endpoints are not completed and contain only guid, create/updated date and name (space additionally has experimental `organization`)  
+- ~~Cannot determine if a CF supports v3 or when it does support v3 which endpoints it has~~
+  - v3 version and supported endpoints can be determined by response to `<cf api url>` and `<cf api url>/v3`
+  - ~~Getting the v2 version is simple, I don't know if there's correlation to v3 version~~
+  - ~~Having a v3/info which published which endpoints are supported would help a lot. Including whether `include` is supported~~
+- ~~Chained `links`/relations don't appear to be supported (fetch app-->space-->org all in one request)~~
+  - It looks like this is achievable by notation, for example `/v3/apps?include=space,space.organization`
+  - See https://github.com/cloudfoundry/cc-api-v3-style-guide#including-related-resources
+  - ~~Might be wrong on this one due to the lack of current `include` integration~~
 
 ### Frustrating Issues
+- Fetching a list of users as a non-cf admin involves making a request to every organisation (`organization/${guid}/users`). The response
+  of all of these calls contains a lot of overlapping data
+  - Ideally making a request to `/users` as a user with 0:M org roles would return a list of users in those organisations
+  - Have asked this question in the V3 Users proposal - https://docs.google.com/document/d/1EA65UN3Xsi0EuX-3YfbFNqtJGseFr6FGBt2SR9c4Aqk/edit#heading=h.tyy5zdgqnnt0
+- Not all v2 endpoints exist in v3, for instance no `domains`, `events`, `route`, org/space quota definitions, etc
+  - This would make our entity validation much more complex
 - Cannot utilise v3 pagination due to limited sorting and filtering functionality (see [Cloud Foundry API v2 Feature Usage - Sorting/Filtering](cf-api-v2-usage.md#sortingFiltering) for missing fields)
   - This is currently on par with v2, but causes us a lot of headaches for large data sets
-- The Stratos method of calculating application state is much harder
+- The Stratos method of calculating application state has become much harder
   - Additional requests to app `/packages` are required. This would be resolved if applications were `link`ed to package
   - Would love a flag in `/apps` to also return the `processes` data. This would mean a longer request time, but we're making that request
     in the frontend anyway.
 - No easy way to fetch organisation or space summary information
   - Stratos show summary information such as number of users (see [Cloud Foundry API v2 Feature Usage V2 Specifics - Cloud Foundry, Organisation and Space Summary Information](cf-api-v2-usage.md#cloud-foundry-organisation-and-space-summary-information) for specific stats)
   - In an ideal world we could hit one endpoint that would give us the counts for all of these details. Filters could then limit this to
     an organisation or space
-- Fetching a list of users as a non-cf admin involves making a request to every organisation (`organization/${guid}/users`). The response
-  of all of these calls contains a lot of overlapping data
-  - Ideally making a request to `/users` as a user with 0:M org roles would return a list of users in those organisations
-- Not all v2 endpoints exist in v3, for instance no `domains`, `events`, `route`, org/space quota definitions, etc
-  - This would make our entity validation much more complex
 
 ### Stratos Tasks
 - Depending on adoption approach, Stratos needs to support v2 and v3 endpoints concurrently
@@ -104,5 +123,17 @@ Then Stratos should either ..
   - https://github.com/cloudfoundry-incubator/stratos/issues/3149 (Container issue for related v3 api process changes)
 
 ### Questions
-- Will `include` cover children of children? For instance `app` --> `route` --> `domain`
-  - How will lists be covered? For instance `organization` --> `space` --> `service instances`
+- ~~Will `include` cover children of children? For instance `app` --> `route` --> `domain`~~
+  - ~~How will lists be covered? For instance `organization` --> `space` --> `service instances`~~
+-  How will the deprecation of v2 endpoints happen?
+  - One by one?
+  - All together once v2 parity is reached?
+- Will duplicated `include`ed entities only appear once in a top level (entity or pagination) `included`? For example..conceptually..
+  - Fetch an application, the application's space, the application's routes and application routes spaces in a single request
+  - If the application space appeared in the route's space, would it only appear once in the application `included`... or appear twice (once 
+    in application `included` and again in route `included`)?
+- The style guide references a way to fetch one to many relationships as `/v3/apps/:app_guid/relationships/routes` (https://github.com/cloudfoundry/cc-api-v3-style-guide#viewing-1)
+  - This doesn't seem to work (404), is it yet to be implemented?
+  - `/v3/apps/:app_guid/routes` also does not work (404)
+  - `/v3/apps/:app_guid/route_mappings` works, but there doesn't seem to be a way to `include` the `route` such that it appears in the response
+- Which version is the `include=space,space.organization` notation supported in?
diff --git a/src/jetstream/default.config.properties b/src/jetstream/default.config.properties
@@ -18,4 +18,13 @@ ENCRYPTION_KEY=B374A26A71490437AA024E4FADD5B497FDFF1A8EA6FF12F6FB65AF2720B59CCF
 #VCAP_APPLICATION={"cf_api": "https://api.10.4.21.240.nip.io:8443"}
 # Kepe the sql lite database file
 SQLITE_KEEP_DB=true
-UI_PATH=../../dist
+UI_PATH=../../dist
+
+
+# User Invites
+SMTP_FROM_ADDRESS=Stratos<test@test.com>
+SMTP_HOST=127.0.0.1
+SMTP_PASSWORD=
+SMTP_PORT=1025
+SMTP_USER=
+TEMPLATE_DIR=./templates