Describe the bug

This issue was first reported by @edeandrea on this zulip thread: https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/gRPC.20server.20oTel.20errors/near/398274468

On the initial report there was scope.close warnings on the logs:

07:57:59 WARN  [io.qu.op.ru.QuarkusContextStorage] (vert.x-eventloop-thread-3) Context in storage not the expected context, Scope.close was not called correctly. Details: OTel context before: {spanId=6ab0c0488d656975, traceId=11cf803891be9354f0e1e5e7b434cda6, sampled=true, parentId=b59e2af98fc4046d}. OTel context toAttach: {spanId=b59e2af98fc4046d, traceId=11cf803891be9354f0e1e5e7b434cda6, sampled=true}
07:57:59 WARN  [io.qu.op.ru.QuarkusContextStorage] (vert.x-eventloop-thread-3) Context in storage not the expected context, Scope.close was not called correctly. Details: OTel context before: {}. OTel context toAttach: {spanId=6ab0c0488d656975, traceId=11cf803891be9354f0e1e5e7b434cda6, sampled=true, parentId=b59e2af98fc4046d}

Subsequent analysis revealed that the spans for the grpc server were being duplicated because 2 different OTel instrumenters were in the request pipeline:

• HttpInstrumenterVertxTracer
• GrpcTracingServerInterceptor

Each instrumenter will create a span and sometimes they end the spans in different order, therefore triggering the Scope close warning messages.

It turns out we have 2 different of grpc servers, the legacy separate and the internal using Vert.x. This can be switched by using this property: quarkus.grpc.server.use-separate-server=false

The problem only happens if we use the internal Vert.x based server.

Expected behavior

When using the new grpc server we want a unique span for the grpc sever with no warning messages.

Actual behavior

2 different spans are created and ocasional messages in the logs warning about the Scope not being properly closed.

How to Reproduce?

Run the GrpcHelloServiceTest on this PR: #37139 with property quarkus.grpc.server.use-separate-server=false at application.properties

Output of uname -a or ver

Darwin Kernel Version 22.6.0: Thu Nov 2 07:43:57 PDT 2023; root:xnu-8796.141.3.701.17~6/RELEASE_ARM64_T6000

Output of java -version

Java version: 17.0.7, vendor: Eclipse Adoptium

Quarkus version or git rev

Quarkus 3.5.0 at least

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546) Java version: 17.0.7, vendor: Eclipse Adoptium, Default locale: en_PT, platform encoding: UTF-8 OS name: "mac os x", version: "13.6.2", arch: "aarch64", family: "mac"

Additional information

No response