Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve change stream UI #2178

Merged
merged 1 commit into from
Aug 5, 2024
Merged

Improve change stream UI #2178

merged 1 commit into from
Aug 5, 2024

Conversation

Karakatiza666
Copy link
Contributor

start paused to follow the stream from the beginning

Enable following multiple streams

Is this a user-visible change (yes/no): yes

…beginning

Enable following multiple streams

Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
@Karakatiza666 Karakatiza666 added enhancement Web Console Related to the browser based UI User-facing For PRs that lead to Feldera-user visible changes labels Aug 4, 2024
@lalithsuresh
Copy link
Collaborator

lalithsuresh commented Aug 4, 2024

@Karakatiza666 There is a strange flicker I see sometimes when I click the play button (I haven't tested on this branch). The UI flickers, status remains in "Ready to run", and then switches to Running afterwards. I suspect its a race where you miss the "Starting up" state.

@ryzhyk
Copy link
Contributor

ryzhyk commented Aug 4, 2024

I tested this using the following program:

-- Credit card holders.
CREATE TABLE CUSTOMER (
    cc_num BIGINT NOT NULL PRIMARY KEY, -- Credit card number
    name varchar,                       -- Customer name
    lat DOUBLE,                         -- Customer home address latitude
    long DOUBLE                         -- Customer home address longitude
) WITH (
    'materialized' = 'true',
    -- Configure the random data generator to generate 100000 customer records.
    -- (see https://www.feldera.com/docs/connectors/sources/datagen)
    'connectors' = '[{
      "transport": {
        "name": "datagen",
        "config": {
          "plan": [{
            "limit": 100000,
            "fields": {
              "name": { "strategy": "name" },
              "cc_num": { "range": [ 100000000000000, 100000000100000 ] },
              "lat": { "strategy": "uniform", "range": [ 25, 50 ] },
              "long": { "strategy": "uniform", "range": [ -126, -67 ] }
            }
          }]
        }
      }
    }]'
);

-- Credit card transactions.
CREATE TABLE TRANSACTION (
    ts TIMESTAMP LATENESS INTERVAL 10 MINUTES, -- Transaction time
    amt DOUBLE,                                -- Transaction amount
    cc_num BIGINT NOT NULL,                    -- Credit card number
    shipping_lat DOUBLE,                       -- Shipping address latitude
    shipping_long DOUBLE,                      -- Shipping address longitude
    FOREIGN KEY (cc_num) REFERENCES CUSTOMER(cc_num)
) WITH (
    'materialized' = 'true',
    -- Configure the random data generator to generate 1M transactions at the rate of 1000 transactions/s.
    'connectors' = '[{
      "transport": {
      "name": "datagen",
        "config": {
          "plan": [{
            "limit": 1000000,
            "rate": 1000,
            "fields": {
              "ts": { "strategy": "increment", "scale": 1000, "range": [1722063600000,2226985200000] },
              "amt": { "strategy": "zipf", "range": [ 1, 10000 ] },
              "cc_num": { "strategy": "uniform", "range": [ 100000000000000, 100000000100000 ] },
              "shipping_lat": { "strategy": "uniform", "range": [ 25, 50 ] },
              "shipping_long": { "strategy": "uniform", "range": [ -126, -67 ] }
            }
          }]
        }
      }
    }]'
);

-- Data enrichment query: left-join the TRANSACTION table with the CUSTOMER table to compute the
-- distance between the shipping address and the customer's home address for each transaction.
CREATE VIEW TRANSACTION_WITH_DISTANCE AS
    SELECT
        t.*,
        ST_DISTANCE(ST_POINT(shipping_long, shipping_lat), ST_POINT(long,lat)) AS distance
    FROM
        TRANSACTION as t
        LEFT JOIN CUSTOMER as c
        ON t.cc_num = c.cc_num;

-- Compute two rolling aggregates over a 1-day time window for each transaction:
-- 1. Average spend per transaction.
-- 2. The number of transactions whose shipping address is more than 50,000 meters away from
--    the card holder's home address.
CREATE VIEW TRANSACTION_WITH_AGGREGATES AS
SELECT
   *,
   AVG(amt) OVER window_1_day as avg_1day,
   SUM(case when distance > 50000 then 1 else 0 end) OVER window_1_day as count_1day
FROM
  TRANSACTION_WITH_DISTANCE
WINDOW window_1_day AS (PARTITION BY cc_num ORDER BY ts RANGE BETWEEN INTERVAL 1 DAYS PRECEDING AND CURRENT ROW);
  • It's quite unreliable. I see some logs some of the time, but but not for all enabled streams. Often times I don't get any logs at all. Deselecting and re-selecting a tables sometimes helps, but often it doesn't. Overall, this needs more testing
  • Restarting the pipeline doesn't seem to clear the log (it's ok to show the old log after shutting down the pipeline, but when you start it again, the log should clear)
  • There is no way to follow the tail of the log. I think the standard behavior is for the window to track the tail of the log by default. The user can scroll to the middle of the log, at which point the window stops tracking the tail. Scrolling to the end of the log re-enables tracking. [CORRECTION: I realized the log still grows up, not down, so the tail is at the top, but I think we should reverse the direction].
  • Long lines shouldn't wrap around (see below)
image

@ryzhyk
Copy link
Contributor

ryzhyk commented Aug 4, 2024

Did some more experimentation. The behavior depends on the rate of changes. When I configure the generator to produce 1 or 100 transactions/s, it seems pretty robust. But at 1000 records/s it becomes very random. Looks like the webconsole doesn't like either large chunks of changes or a high rate of changes.

@Karakatiza666 Karakatiza666 merged commit fa97938 into main Aug 5, 2024
5 checks passed
@Karakatiza666 Karakatiza666 deleted the improve-stream-ui branch August 5, 2024 08:44
@Karakatiza666
Copy link
Contributor Author

I'll address all feedback in a follow-up PR. Merging as-is

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
User-facing For PRs that lead to Feldera-user visible changes Web Console Related to the browser based UI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants