Great work, some feedback from giving it a first try:

• Serve the new UI under / and remove the build and serving the old UI, also delete all of web-console/

• left navigvation should be open by default at first:

• Needs some text how about: "Start a New Pipeline"

• This should have some more spacing between heading and description
  

• Clicking this doesn't do anything yet? Best case can we configure it to fetch a file from github and populate the pipeline editor with that?

• it seems odd that an empty SQL program is ready to run?

• It's odd that we call this a "PROGRAM ERR" now maybe the state for a failed rust compilation is called something else? Otherwise maybe we should say "SYNTAX ERROR"

• It's a bit confusing that the pipeline only shows up with a delay when creating a new one

• The whole space should be clickable to go to the pipeline not just the "test'

• There are these ERROR in the log that probably shouldn't be ERROR

2024-07-30 20:27:51 ERROR [manager] [HTTP error response] PipelineNotRunningOrPaused: Pipeline 0191054a-5ae2-7362-9993-c8c097f5a56b is not currently running or paused.

• Sometimes clicking the play button doesn't have any effect

• remove ad-hoc query and query-plan tabs for now

• The errors only show up seconds after the program failed to compile here
  Screencast from 07-30-2024 01:31:16 PM.webm

• if I use the mini-view in the text editor to slide up in the text and let the mouse go on the slack button it sends me to the community slack (note I just let the mouse button go I didn't click the community slack link or anything -- this is on firefox)

-if autosave is off and turned back on, if the state has changed the default action should be to save
Screencast from 07-30-2024 01:35:16 PM.webm

• having this set to null seems wrong?

• If there is an error in the json, we should grey out the save button and add a tooltip to it to make sure to inform the user that there is a json syntax error
  Screencast from 07-30-2024 01:37:17 PM.webm

Thanks a ton for the feedback, a few replies already:

There are these ERROR in the log that probably shouldn't be ERROR

The return code should be an error (issuing an action which is not valid in a certain status).

having this set to null seems wrong?

These are the current defaults, I agree it would be good to change to None, though for this PR not planning on changing that part of the API.

The return code should be an error (issuing an action which is not valid in a certain status).

I think for logging we should only log error! if something goes really wrong, returning an error status code to the user that's not due to an unexpected thing happening in the manager should probably just be warn! or info!

(I think there were cases even before this where we logged error! in the manager when we shouldn't have so maybe it's still the same issue)

Yeah I noticed this too after coming back to the pipeline manager after a while. We shouldn't be spamming error! logs when the user isn't doing anything wrong. So either these are not legitimate errors, and shouldn't be logged, or the UI shouldn't be doing whatever it was that's causing these error logs.

We could distinguish that error-level log messages should only occur when there was truly an error at the api-server.

In this case, the UI is issuing an invalid request (we can diagnose why), so it should get back an HTTP error though (maybe log level should just be info of it though, or even debug).

In this case, the UI is issuing an invalid request (we can diagnose why),

For a user it's confusing to see errors in a log if they aren't really errors for him -- for debugging our own code a lower log-level should suffice. Sending HTTP errors back is totally fine, just doesn't need to be logged as an error.

Rebased on latest main

@Karakatiza666 the pipeline settings no longer shows after click for me on latest commit

@snkas Some bugs to fix.

When following the REST API tutorial, I noticed a few things. I created the following file:

➜  /tmp cat program.json
{
    "name": "sc-program",
    "description": "Supply Chain program",
    "runtime_config": {"workers": 4},
    "program_config": {},
    "program_code": "
CREATE TABLE vendor (                                    \n
    id BIGINT NOT NULL PRIMARY KEY,                      \n
    name VARCHAR,                                        \n
    address VARCHAR                                      \n
);                                                       \n
                                                         \n
CREATE TABLE part (                                      \n
    id bigint NOT NULL PRIMARY KEY,                      \n
    name VARCHAR                                         \n
);                                                       \n
                                                         \n
CREATE TABLE price (                                     \n
    part BIGINT NOT NULL,                                \n
    vendor BIGINT NOT NULL,                              \n
    price DECIMAL                                        \n
);                                                       \n
                                                         \n
CREATE VIEW low_price                                    \n
    (part, price)                                        \n
    AS                                                   \n
    SELECT                                               \n
        price.part AS part,                              \n
        MIN(price.price) AS price                        \n
    FROM price                                           \n
    GROUP BY price.part;                                 \n
                                                         \n
CREATE MATERIALIZED VIEW preferred_vendor                \n
    (part_id, part_name, vendor_id, vendor_name, price)  \n
    AS                                                   \n
    SELECT                                               \n
        part.id AS part_id,                              \n
        part.name AS part_name,                          \n
        vendor.id AS vendor_id,                          \n
        vendor.name AS vendor_name,                      \n
        price.price AS price                             \n
    FROM                                                 \n
        price, part, vendor, low_price                   \n
    WHERE                                                \n
        price.price = low_price.price AND                \n
        price.part = low_price.part AND                  \n
        part.id = price.part AND                         \n
        vendor.id = price.vendor;                        \n
"
}

• I tried to use PUT /v0/pipelines/{name}. I started without "name", or the two config fields. I was forced to add each of them by consecutive error messages that asked for the missing fields. The config fields should be optional.

• Minor: After that however, I get the following error for PUT, which was confusing (note the names is off in the URL vs the program which I fixed afterwards). PUT doesn't usually signal "rename" to a user, so perhaps just give a better error message here:

➜  /tmp curl -i -X PUT http://localhost:8080/v0/pipelines/sc-pipeline \
-H 'Content-Type: application/json' \
-d @program.json
HTTP/1.1 400 Bad Request
content-length: 119
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
content-type: application/json
access-control-allow-credentials: true
date: Fri, 02 Aug 2024 00:32:47 GMT

{"message":"Cannot rename a pipeline which does not exist","error_code":"CannotRenameNonExistingPipeline","details":{}}%      

• Source position error when there is a connector misconfiguration. I kept going where the editor directed me to (line 6) but what the error message actually means is line 6 within the create table statement on line 13.

The config fields should be optional.
@lalithsuresh

The biggest downside to making them optional is that any typos will go unnoticed as the property is now optional (e.g., runtiem_config: { workers: 10 } would just be ignored and instead the default used). We could address this by adding disallowing unknown fields, but that would be inconsistent behavior with all the other serialized types in the nesting of runtime_config and program_config, and as well might affect compatibility in future.

The primary form factor where this is an inconvenience is doing direct API calls, which now that the CURL tutorial is rewritten (thank you :) ) people will use as basis in any case. I'm leaning towards keeping it explicit for now ({} = set to default) and see, what do you think?

error for PUT, which was confusing
@lalithsuresh

Replaced with the error:

The pipeline name in the request body does not match the one provided in the URL path. This is not allowed when no pipeline with the name provided in the URL path exists.

Source position error when there is a connector misconfiguration
@lalithsuresh

I've added the relation name now to the error and added that the position is within the string, for example:

 ConnectorGenerationError:
relation 'example': property 'connectors' has value '[{
        "transport2": {
         "name": "datagen",
         "config": {
            "plan": [ { "rate": 1000 } ]
         }
      }
    }]' which is invalid: deserialization failed: missing field `transport` at line 8 column 5 (position is within the string itself)

Exact positional indication in the SQL program will require additional information from the SQL compiler.

The compiler can provide the starting point in the source sql program for the connectors string. If that's enough I can add a property to the Metadata json

Ideally it would be just for each property such that in the future if we have new or different property expectations, it line/column number can be used for those as well.

We can for now have a dedicated one for the connectors key, where in the ProgramSchema is Metadata located though? I cannot seem to find it?

I call everything metadata including the properties themselves