👍 besides comment.

Switch back to a single remote writer, as we were only ever meant to have one

Have only just spotted this! I had intended this to be a list, so you could send the samples to multiple downstream services - for instance, we want to allow people to send samples to two cluster (for HA), and we want to be able to send samples from our dev to our dev & prod cluster a vice versa.

The alternative (for us) is to run multiple Prometheus instances, one pushing to each cluster. Problem with this is managing the config (as k8s config maps) - we're ending up with multiple copies of the same config and its a pain to ensure they're in sync.

Would you be open to me revising this? Alternatively, would you be open to Prometheus loading multiple configs and defining some kind of config merge?

Why don't you mount the same ConfigMap instead of syncing multiple if they are the same?

Why don't you mount the same ConfigMap instead of syncing multiple if they are the same?

They need to differ very slightly (the remote storage url & credentials). And I'm not aware of a way of doing this with k8s other than a blind copy.

Right now the answer is to have a receiver acting as a tee. Anything beyond that needs to wait at the least until we've removed the legacy remote storage systems.

Ok that makes sense. IIRC there is a proposal out there for templating in kubernetes but I don't remember the scope of it and am also not able to find it right now.

Right now the answer is to have a receiver acting as a tee. Anything beyond that needs to wait at the least until we've removed the legacy remote storage systems.

Hmmm, thats a bit gross (having a tee). Why does this need to wait on removing the old legacy remote storage? What am I missing?

Hmmm, thats a bit gross (having a tee).

The consensus when this was first discussed was that there was to be only one.

Why does this need to wait on removing the old legacy remote storage? What am I missing?

Think about how the config file would have to look without removing that.

The consensus when this was first discussed was that there was to be only one.

I don't specifically recall this - can you remind me about the tradeoff behind this decision?

Either way I don't think we should hold it as set in stone - we've modified previous decisions taken during that discussion (cf gRPC), and if we have good reasons to reevaluate it we should.

Think about how the config file would have to look without removing that.

The original PR adding generic remote storage to the config file (#1957) had multiple and it looked fine to me... what do you see as the problem?

I don't specifically recall this - can you remind me about the tradeoff behind this decision?

It was in the big long initial thread about adding this support.

The original PR adding generic remote storage to the config file (#1957) had multiple and it looked fine to me... what do you see as the problem?

The semantics of relabelling there had not been considered, as having multiple hadn't been on the table.

The semantics of relabelling there had not been considered, as having multiple hadn't been on the table.

The relabelling config is part of the RemoteWriteConfig struct, as can be repeated too. Whats the problem with this?

The relabelling config is part of the RemoteWriteConfig struct, as can be repeated too. Whats the problem with this?

How would that work for legacy remote storage?

Do we want the relabelling to be global or per receiver?

As I indicated back on the commit where this was added, by adding support for multiple receivers you completely blindsided me (as this support was looking to be ~2 years out) so none of this stuff had been figured out in advance and was blocking more important work. This stuff still hasn't been figured out.

I agree that multiple remote writers would be useful for Cortex, but also for any other case where you want to send data to multiple systems (like to a long-term storage and a stream processor at the same time). Running yet another process outside of Prometheus for teeing gets a bit ridiculous, given that you already have to run external adapter processes to convert Prometheus's remote write format to whatever system you're sending samples to.

How would that work for legacy remote storage?

Given that legacy remote storage is deprecated and soon-to-be removed, I don't think we need to support multiple of them anymore (that would also require moving them from flags to the config).

Do we want the relabelling to be global or per receiver?

Per receiver seems to make most sense, given that you may want to send different subsets of samples to different targets. The downside is having to repeat common relabelings, but that's better than not being able to send different subsets, IMO.

If we can agree on this, I think it makes sense to re-add support for multiple remote writers.

Since there's no reply here yet, I'm going to work on a PR that re-adds multiple remote writers, so we can discuss further there.

We need to kill off the legacy remote storage before that can be considered, otherwise we'd be making more breaking changes than is necessary.

@brian-brazil I think now I finally understand. I wrongly assumed that the remote write relabeling added here only applied to the new generic remote writes, not all the legacy ones. That would have made intuitive sense for me, since its part of the generic remote write configuration section.

Ok, I would love to get rid of the legacy ones (and provide external reference implemenations for them if needed), but will check the priority of that with my Weavy colleagues.

External legacy example support out in #2381