Thanks for the reply, but this is not solving my use case.

So the issue here is not assigning replicas to nodes, but assigning pods (preference) to nodes that have replicas on them, to avoid rebuilding the replicas on other nodes if not necessary.

Please let me know if I have not 'mmade myself clear with my explanation above and I will rephrase the description.

I hope that makes my point more clear? Or should I describe it Better?

They only provide node-affinity/taint mechanism for you to place the pods on the nodes you want

these are techniques i know of, but i am not sure how k8s handles pod <-> volume stickiness contraint (affinity?) for regular volumes, think hostPath, or local provisioner in k3s, etc. I was hoping there was some advanced hidden mechanism that could be used to force pods on the same nodes as a volume

Again, Longhorn cant control users pods and it shouldn't.

i understand, but cloud providers charge big money for moving data around between data nodes, so this would make quite an impact on budgets (and performance too)

Adding labels to the node and adding node-selector or node-affinity to your pods is the most standard solution to this and is encouraged by the K8s official docs.

i see ... is there any pod in the longhorn-system namespace that one could make use of in order to create a podAffinity for the workloads ? Is there any pod, like the instanceManager for example that can be used to trigger the desired mechanism? I have not checked what happens if a node goes down whose volume was attached to the workload. is the instanceManager spawning on a new node BEFORE the workload is scheduled or does it happen AFTER ? if it happens before, then perhaps one could use the instanceMAnager as the affinity source

In short, "k8s handles pod <-> volume stickiness contraint (affinity?)" doesn't exist in K8s

Pod is pod, managed by K8s and you
Volume is volume, managed by the third party, e.g. Longhorn, Ceph, GPC, AWS,...

As far as I know, hostPath only allows you to mount the path in the host to the pod on the same node. But it doesn't include forcing your pod to the node where you are going to provide the host path.
You still need to use node-selector to force the pod to the node you want, am I right?

For local-path-provisioner, it also don't include the user pods scheduling.
It only allows you to pre-defined the path in the host that you want it to be a hostPath
Then when the pod using that PVC is scheduled to one of the node, it will mount the pre-defined hostPath.
For example, if you have this config in the local-path-provisioner

  config.json: |-
        {
                "nodePathMap":[
                {
                        "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
                        "paths":["/opt/local-path-provisioner"]
                },
                {
                        "node":"node1",
                        "paths":["/node1", "/data1"]
                },
                {
                        "node":"node2",
                        "paths":["/node2", "data2"]
                }
                ]
        }

If one node is listed on the nodePathMap, the specified paths in paths will be used for provisioning.
If one node is listed but with paths set to [], the provisioner will refuse to provision on this node.
If more than one path was specified, the path would be chosen randomly when provisioning.

If your data is on node2 "/node2", you still need to specify the pod to be scheduled to the node2, so it can access the data in the path "/node2" on node2.

Instance-Manager is like a daemon-set which will be created on every nodes to manage the replica/engine processes on it.
When the node goes down, the instance-manager on that node won't be created on another node since it is 1:1 mapping.
In this case, replica on that node(instance-manager) goes down, the volume still works because it use network protocal and can access another replica data on another node(instance-manager).
If the node also contains YOUR pod, then your pod will be scheduled to another node BY K8S (if yours is a deployment) and Longhorn will remount the volume to the new recreated pod and connect to other replicas.
That is the fundamental difference between using hostPath or local-path-provisioner. The whole IO system built on top of the network protocol allows the next-level resilience.

Again, volume scheduling is managed by Longhorn.
But pod scheduling is managed by K8s and You.

Moreover, volume is just a concept
For Longhorn, it is an iscsi initiator and target between engine and replicas on different node.
The IO from your pod goes through the engine and is dispatched to all the replicas and even wait until all replicas ack to ensure data consistency.
You cant say you want you pod to be as close as the volume to shorten the data path as long as you have extra replica on other nodes.

For OpenStack, the volume is a logical volume on different node as your VM, and it also uses the iscsi to dispatch the IO.
For Ceph, the volume is a on the RADOS layer developed by themselves which is a distributed storage system, your volume is divided by multiple blocks and stored by sharding on different nodes in the cluster. There is of course no pod-volume affinity in this case.

So, if IO performance is really a key metric for your application.
I would suggest you should looks more into the storage solution you are going to use, see how the IO works.
And do the real benchmark to see which one you can tolerate.

Resilience, replication and fault-tolerance is a tradeoff of the IO performance because you need to add more layer into the path.

@dberardo-com If you can wait a little while I can build something for you

we should possibly make the node-labeling behavior optional for every volume, as when one has lot of volumes and replicas this might end up filling up nodes with lot of labels, which might not be a best practice

There's no need to modify or label any resources. This can be done on the fly with the mutation webhook

@dberardo-com Did you still need this? I found an even simpler way of doing this with new additions to Kubernetes since 1.29 and custom scheduler

Sounds great, sure, I still wish to test some solutions within the next weeks so this could be helpful

New method is pretty much:

• On pod definition name a custom scheduler schedulerName: pvScheduler.
• That scheduler needs read access to PV bindings.
• It can tell what PVC correlates to which PV because of the pod definition and information in the PV longhorn creates

It will leave the pod in pending state until the PVs are created. When they are finish it knows which nodes they were bound to. The scheduler then files a binding resource to bind pod to one of the nodes with PV.

Everytime the pod needs to be rescheduled it will use this scheduler. It will be able to do this stateless

i see, i am not familiar with this concept either. do we need to create a custom scheduler ? is this a global or a namespaced resource? and how would the definition look like ?

and other then that: would then suffice to add the schedulerName attribute to all deployments/statefulSets requiring and thats it ? no need to modify also storageclasses or PVC ?

Also: why is k8s v 1.29 needed for that ? which feature is the new one ?