I think the datasource should be part of the plugin as well. The variable definition can be ( or all the time ) related to the type of the Datasource.

And on Grafana, the plugin is "just" the panel. I assume it's probably because it's super hard or impossible to load at runtime new golang code : https://groups.google.com/g/golang-nuts/c/c269TCEyjWQ

I'm looking at the CRDs because it's an interesting way to add custom object in an API. But in that case there is no interaction with the other object. But it means that we could add more variables definitions dynamically, but the dashboard data definition won't take in consideration these new types.

Variables: currently in the UI, I'm thinking of this as "how do I get the available values for a variable so a user can make a selection?"

I assume you are talking about different kind of type of variable. But if you are really talking about the actual value of the variables, well it will depend of the kind of the datasource and if it's dynamic ... GOTO beginning of my comment.

Otherwise currently you can do like explain in the doc

Sooooo two solutions from my point of view:

1. if we want to create real plugin with server side, where the plugin is a Golang code like fluent-bit, we will have to change the language used to implement the backend. Perhaps Rust would be a good candidate (Would like to avoid C, it's a bit too hardcore for me). Quick google search returns interesting result when talking about plugin.
   I want to say it's opening some new horizons and we will really have something different than Grafana. Providing a real alternative.

2. Or we do something similar than Grafana:

   • Only Panel can be extendible
   • The backend is providing a sort of proxy that exposed more or less directly the API exposed by the datasource, like /proxy/datasources/<datasource_kind>/<datasource_name>/<datasource_api like prometheus>. With that, the logic about how to query the datasource is entirely on the front-end side. (Talking about variables and data for panels). The backend is just modifying the request by injecting any things described in the datasource configuration (like secret, certificat ...etc.)
   • But the dashboard definition will be typeless. I fear we will ended up to have the same kind of dashboard datamodel than Grafana where you don't really get what you are manipulating.
   • Looks like a copy of Grafana ?

Yeah, I neither see us using native Go plugins, nor switching to a different backend language than Go, in my strong opinion. Not only do we all know Go best, but it's also pretty much the standard in the space of people + infra we are working in, and thus we would also be losing a lot of other potential talent and existing library reusability.

So the reality will likely be that only the frontend will be fully pluggable, and while there can be datasource-specific backend code as well, it either has to be statically part of the backend or run as a separate process. So likely a dumb backend datasource proxy makes sense (for people who only want to load new frontend code), at least in addition to a possibly higher abstracted backend proxying method for more well-established and compiled-in datasource plugins in the backend.

yeah datasource as a plugin could be seen as a webhook like AlertManager. No idea exactly how it could work yet and probably it won't matter so much for the moment.

mmm if we assert it's impossible or too complicate to do a datasource as a plugin (not even sure it's reallistic to see the datasource through a webhook), probably having the panels plugable doesn't worth it too ?

I mean I get the point of supporting multiple different datasources, but multiple different panels ? The feedback from the SRE is more they wish less panel, because the fancy one are well just fancy but not really necessary. So I'm really wondering where the need of having the panel as a plugin is coming from. Also currently the different panels are typed, so the backend is able to check if the different mandatory fields are properly filled. I think it's important to keep these kind of checks as they will guarantee that your dashboard will be properly displayed.

For sure it can interesting in term of design / architecture to treate the panel as a plugin so it will be easier to add more. But that doesn't necessary mean to open it up for the reason above.

I also get the point on we prefer that the front-end can be able to talk directly to the datasource, For this particular point (which will solve the point 2 and 3), I already have more or less the code to do this kind of thing. I basically just have to transfer it and adapt it a bit. So it should be quick.

Probably the code managing the build order of the variables can be easily transferable to the frontend as well.

I think at this point of the discussion we can assert that the backend is here to:

• Store the different data like the dashboard, the datsource configuration ...etc.
• A bit later, protect the access to these data using authentication and authorization layer (that will be defined later)
• A proxy to the frontend when it wants to access to the different datasources through /proxy/datasources/..... The backend in this case will inject in the request everything required to be able to contact the server (like the certificate / the secret / later finding through the discovery where is the server to contact)

Just to elaborate a little bit on why I'm initially thinking this way on two things:

Keeping plugins fine-grained (at least initially)

One thing that I haven't loved while working in the Grafana plugin API is that it's so coarse-grained (i.e. you either get a Panel or a DataSource), that you end up with an API that you have to implement whether you want to or not. So, you need a "DataSource" that implements a time-series data querying API before you can even think about adding something like a new variable type. Is it possible that a user might want to just implement a new type of variable? Does a variable always need a "Data Source" to fetch data from (e.g. what if the option values are static and just stored in the config)? Are there going to be variables that work in the context of more than one "Data Source" (e.g. does an "Interval" variable make sense in more than just Prometheus)?

Grafana also tries to reuse plugin APIs for multiple things and as a result, the parameters that get passed into them can be pretty gnarly and confusing. For example, the main API for a DataSource plugin (the .query() method) is used to run time series data queries for both Dashboard Panels and their Metrics Explorer feature. The options object that gets passed into that contains all sorts of properties to try and cover both use cases, but many times some options don't make sense in one context or the other (e.g. what should the panelId be from a Metrics Explorer query or why do you even need it?). To me, it makes sense to try and start fine-grained and use the type system in Typescript to our advantage and to help plugin authors (including us!).

If we ever end up with a similar "Metrics Explorer" type feature, I can imagine that being a separate plugin type (i.e. something like a MetricsExplorerQuery plugin vs a ChartQuery plugin). There we can define exactly the API that makes sense for that feature and avoid trying to over-generalize and create some perfect abstraction for both (different) use cases. There's nothing stopping plugin authors from reusing core functionality (e.g. how to run a Prom range query, how to do variable templating, etc.) in both plugin implementations. But at least they get an accurate picture (via the type system) of exactly what they need to do to implement a specific Perses feature.

Front-end plugins are just made up of React Hooks + Components

This one I'm still not so sure on for a number of reasons, but using React primitives as the building blocks of front-end plugins seems like a reasonable starting point. For components, I'm not really sure we have much of an option since Perses is going to be a React application, so the logical/lowest friction way to let plugin authors control how the UI looks is via custom React components. On the hooks side for implementing state/logic, I'm less certain if it's better to let plugins implement a hooks-based API, or just implement plain old functions of some kind. I went with hooks initially because of two main reasons:

1. They give plugin authors a lot of freedom and allow them use of all the React primitives they're used to in their own React applications.
2. We don't have to try and come up with abstractions for a function-based API, particularly around what to pass in as parameters.

I can see number two in particular being an issue. For example, in a Variable plugin where you (the plugin author) have to give me a list of options to show, what information does Perses need to pass in as a function parameter to your getVariableOptions implementation? Do you need the value of other variables? Do you need the available options of other variables? Do you need the currently selected Dashboard time range? You can imagine scenarios where sometimes you do (e.g. a Prom label names query) and sometimes you don't (e.g. a list of static Intervals to select from). So does that API end up with a bloated list of parameters to cover every possible use case, whether the plugin implementation needs it or not?

I think a hooks-based API solves this problem in a slightly better way in that plugin authors can just use other hooks to get access to the things they need, rather than having to pass "all the things" in via parameters. So if you need dashboard variable values, just useDashboardVariables() in your plugin hook and there you have them. If you need the current time range for the dashboard, just useDashboardTimeRange() and you're set. In some ways this is kinda getting around the issue by taking advantage of React Context (or Redux, or wherever that state is stored) to get data that's contextual to the hook. The downside to this approach is that I can see it being confusing for a plugin author to know exactly what Perses hooks they can use in a given context.

OK, sorry for the long-winded explanation, just wanted to get some of this out there so you can know where the initial approach came from.