Ok, I think I understand what's going on there. I had expected the C API's UpdateCUDAProviderOptions function to be something I could use to append options to a CUDA options struct, but it looks like what it actually does is delete the old one and only set the options specified in the update call (https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/cuda/cuda_provider_factory.cc#L236). The Java code calls UpdateCUDAProviderOptions each time the add method is called, so the new option overwrites all the old ones. That's pretty annoying, but I can fix it on the Java side. I'll put a fix together next week.

Thanks for the clarification!
Regardingthe 2nd problem, does the call to UpdateCUDAProviderOptions allocate some memory on device 0 (maybe for initialization) before setting main device to 6 and proceed with main computation? And is it possible to pass multiple options at once to avoid calling the add method several times?

It is possible in the native code to pass multiple options at once, but not how I've written the Java binding to that native code. The Java object tracks all the options that are set, so I need to modify the SessionOptions.addCUDA call to call a new method on OrtCUDAProviderOptions which calls update once with the aggregated parameters before it's passed in.

WRT the memory allocation on GPU zero, that might be an artifact of how CUDA & ORT works, I think the primary GPU tends to end up with some driver & code related stuff in general, but someone with more CUDA expertise might be able to help there.

I did additional tests regarding the memory allocation problem on gpu 0, where I simply inserted the following code using cuda api before calling anything that is ORT related:

    int status = cudart.cudaSetDevice(6);      // set cuda device
    checkCuda(cudart.CUDA_SUCCESS, status, "cudaSetDevice");      // check for exception

and ORT code still allocates on deviceId=6:

OrtCUDAProviderOptions cudaOptions = new OrtCUDAProviderOptions();
cudaOptions.add("device_id", String.valueOf(6));
options.addCUDA(cudaOptions);

and the problem above is solved, where nothing is allocated on gpu 0 anymore. However, the same memory size is still allocated on gpu 6 --> 1545MiB. I would have expected that the memory now should be aggregated with that which has been allocated on gpu 0 to become maybe smthn like ~2000MiB if it is to be some cuda driver related stuff.

In other words, I assume there is also a bug where ORT starts allocating on gpu 0 for main computation before detecting the deviceId set by the user?

Additionally, without the cuda code:

    int status = cudart.cudaSetDevice(6);      // set cuda device
    checkCuda(cudart.CUDA_SUCCESS, status, "cudaSetDevice");      // check for exception

When gpu 0 is free, 545MiB is being allocated on it, however, when it is allocated by another process and not much space is left, a portion of that memory is allocated instead ~100MiB