Open Policy Agent
Bundle fails to load #650
Replies: 1 comment 2 replies
-
|
I've restarted the service, and changed log level to "info", and found out that the bundle is not refreshed according to the polling settings. I waited a few minutes and it wasn't refreshed at all - I haven't found any logs about failures, and the counters were equal to 0. |
Beta Was this translation helpful? Give feedback.
-
Did OPA try to pull the bundle from the remote service? Try using
What's different about the cluster with the failure? Is OPA pulling the bundle from your remote service or is it loading the persisted bundle? |
Beta Was this translation helpful? Give feedback.
-
I will try with debug logging tomorrow - I am afraid that debug logging will output a request, which in our case contains token :/
There actually is nothing different, but we are having networking issues in this cluster. We use Istio service mesh, and the OPA also utilizes Istio proxy sidecar. However, we had to omit Istio proxy when fetching bundle, because of some weird issue. I am starting to feel that there is something that breaks the connection - I don't know the details about bundle download, but I assume that the bundle is fetched in portions, and extracted on fly. I feel that this might mean that part of the bundle was downloaded and extracted, but then, all of the sudden, the network connection breaks, and it cannot read more data. Any thoughts? :) |
Beta Was this translation helpful? Give feedback.
-
Hi! We use OPA in production, in a few different k8s clusters.
Here is the config that we use:
Basically, the bundle is fetched from policy-service, and persisted under /var/opa.
This is a mechanism that allows us to have a fallback in case there is a problem to fetch bundle from policy-service (network issue, or a service is down).
The
/var/opadir is a mounted volume, and is shared between multiple pods:The problem we are encountering is that OPA deployment in one of our k8s clusters fails to load bundle in most cases with such an error:
From time to time bundle is loaded successfully, but in most cases it fails (based on the
bundle_failed_load_counterandbundle_loaded_counter).I wonder if you can help to find what might be the reason of this failure? The bundle is pretty big, it contains a lot of data. However, in four different kubernetes clusters, which use the same bundle, we don't have this issue.
Beta Was this translation helpful? Give feedback.
All reactions