[Streams 🌊] Add processors validation and simulation gate #206566
Conversation
tonyghiani
changed the title
tonyghiani
added
release_note:skip
|
Tangential to this PR, but I just noticed we also don't validate whether the mapped field type will work: Here I'm picking double as the field type which will break during ingest because 
Also we should show a more aggressive warning when the user disabled "Ignore failures for this processor" and there are unmatched sample documents - while unmatched docs are benign as long as failures are ignored, they are a pretty big problem once they lead to failure for ingesting the doc. |
Make sense, the current simulation is just simulating the pipeline, not a data ingestion, updating the simulate API to try an ingest simulation should also handle this. I'll open a follow up PR for this specifically.
Sounds good, @patpscal can we add to design a variant that handles this case? Not sure what feedback we could provide here, something like a bold warning icon+text next to the disabled flag seems fine to me.
Mmm I see what's happening, to validate that the updated processor configuration is correct, I'm running a new simulation on submit retrieving the data sample, but it's possible the user won't have data ingested for the time range used. Actually, this might be limiting also when running the validation for creating the processor. I'll take a look at how we can handle this, but I'm not aware of any JS API available in Kibana to validate a processor config without simulating ingestion/pipeline or updating a real pipeline, I believe that syntax-check lives in Elasticsearch only. |
I think we can't safely do this, because existing already processed documents might fail with the new configuration even if it would work with raw documents coming from the source (e.g. they will have the extracted field defined already and it would look like it updated it, but actually raw documents wouldn't have the extracted field set). I think we just shouldn't do the validation for the editing case, only for the new processor case. Annoying, but all coming back to the same root problem that we need original documents to test edits to processing, which we can't get. We need some logic to only run this simulation check when the only change to the processing pipeline is adding new processors to the end of the pipeline, not reordering existing ones or changing existing ones. |
|
@flash1293 I addressed your suggestions, this is ready for another look. @patpscal I made this to provide a warning caption once the Screen.Recording.2025-01-15.at.09.13.07.mov |
...ity/plugins/streams_app/public/components/stream_detail_enrichment/flyout/ignore_toggles.tsx
Show resolved
Hide resolved
There was a problem hiding this comment.
LGTM
Beyond the stuff we already noted down as follow ups - for the "unmatched" filter it doesn't really make sense to keep the columns for the fields that were not extracted:
This will give the message itself more space so it's easier to see what the problem is.
Addressed with refactor(streams): hide detected fields columns for unmatched filter Screen.Recording.2025-01-15.at.11.04.04.movI'll get this merged when it goes 💚 again and take on the next steps. |
💚 Build Succeeded
Metrics [docs]Module Count
Async chunks
Unknown metric groupsESLint disabled in files
Total ESLint disabled count
History
|
|
Starting backport for target branches: 8.x |
…6566) ## 📓 Summary Closes elastic/streams-program#66 This work adds changes to prevent invalid processors from being submitted. The main rule is that a simulation is performed before any add/edit submission to guarantee that the processor config is valid. This work also updates the simulation API to detect whether there is a non-additive change in any simulated document. @patpscal error reporting UI for add/edit is different since the simulator is not visible for edit, I used a callout but we can easily update this once there is a final design in place. ### Form validation + simulation https://github.com/user-attachments/assets/f7fc351b-6efc-4500-8490-b7f1c85139bf ### Non-additive processors https://github.com/user-attachments/assets/47b5b739-c2cf-4a74-93a8-6ef43521c7d4 (cherry picked from commit 6429c53)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
) (#206803) # Backport This will backport the following commits from `main` to `8.x`: - [[Streams 🌊] Add processors validation and simulation gate (#206566)](#206566) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Marco Antonio Ghiani","email":"marcoantonio.ghiani01@gmail.com"},"sourceCommit":{"committedDate":"2025-01-15T15:09:02Z","message":"[Streams 🌊] Add processors validation and simulation gate (#206566)\n\n## 📓 Summary\r\n\r\nCloses https://github.com/elastic/streams-program/issues/66\r\n\r\nThis work adds changes to prevent invalid processors from being\r\nsubmitted.\r\nThe main rule is that a simulation is performed before any add/edit\r\nsubmission to guarantee that the processor config is valid.\r\n\r\nThis work also updates the simulation API to detect whether there is a\r\nnon-additive change in any simulated document.\r\n\r\n@patpscal error reporting UI for add/edit is different since the\r\nsimulator is not visible for edit, I used a callout but we can easily\r\nupdate this once there is a final design in place.\r\n\r\n### Form validation + simulation\r\n\r\n\r\nhttps://github.com/user-attachments/assets/f7fc351b-6efc-4500-8490-b7f1c85139bf\r\n\r\n### Non-additive processors\r\n\r\n\r\nhttps://github.com/user-attachments/assets/47b5b739-c2cf-4a74-93a8-6ef43521c7d4","sha":"6429c53597b203d0f4435909333131b00c01a996","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","Feature:Streams"],"title":"[Streams 🌊] Add processors validation and simulation gate","number":206566,"url":"https://github.com/elastic/kibana/pull/206566","mergeCommit":{"message":"[Streams 🌊] Add processors validation and simulation gate (#206566)\n\n## 📓 Summary\r\n\r\nCloses https://github.com/elastic/streams-program/issues/66\r\n\r\nThis work adds changes to prevent invalid processors from being\r\nsubmitted.\r\nThe main rule is that a simulation is performed before any add/edit\r\nsubmission to guarantee that the processor config is valid.\r\n\r\nThis work also updates the simulation API to detect whether there is a\r\nnon-additive change in any simulated document.\r\n\r\n@patpscal error reporting UI for add/edit is different since the\r\nsimulator is not visible for edit, I used a callout but we can easily\r\nupdate this once there is a final design in place.\r\n\r\n### Form validation + simulation\r\n\r\n\r\nhttps://github.com/user-attachments/assets/f7fc351b-6efc-4500-8490-b7f1c85139bf\r\n\r\n### Non-additive processors\r\n\r\n\r\nhttps://github.com/user-attachments/assets/47b5b739-c2cf-4a74-93a8-6ef43521c7d4","sha":"6429c53597b203d0f4435909333131b00c01a996"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/206566","number":206566,"mergeCommit":{"message":"[Streams 🌊] Add processors validation and simulation gate (#206566)\n\n## 📓 Summary\r\n\r\nCloses https://github.com/elastic/streams-program/issues/66\r\n\r\nThis work adds changes to prevent invalid processors from being\r\nsubmitted.\r\nThe main rule is that a simulation is performed before any add/edit\r\nsubmission to guarantee that the processor config is valid.\r\n\r\nThis work also updates the simulation API to detect whether there is a\r\nnon-additive change in any simulated document.\r\n\r\n@patpscal error reporting UI for add/edit is different since the\r\nsimulator is not visible for edit, I used a callout but we can easily\r\nupdate this once there is a final design in place.\r\n\r\n### Form validation + simulation\r\n\r\n\r\nhttps://github.com/user-attachments/assets/f7fc351b-6efc-4500-8490-b7f1c85139bf\r\n\r\n### Non-additive processors\r\n\r\n\r\nhttps://github.com/user-attachments/assets/47b5b739-c2cf-4a74-93a8-6ef43521c7d4","sha":"6429c53597b203d0f4435909333131b00c01a996"}}]}] BACKPORT--> Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani01@gmail.com>
📓 Summary
Closes https://github.com/elastic/streams-program/issues/66
This work adds changes to prevent invalid processors from being submitted.
The main rule is that a simulation is performed before any add/edit submission to guarantee that the processor config is valid.
This work also updates the simulation API to detect whether there is a non-additive change in any simulated document.
@patpscal error reporting UI for add/edit is different since the simulator is not visible for edit, I used a callout but we can easily update this once there is a final design in place.
Form validation + simulation
demo.mov
Non-additive processors
Screen.Recording.2025-01-14.at.14.15.18.mov