No data exported. Fails to load data. No dashboards in the list. #649
Description
Its a standalone deployed not included with viya deployment. I confirmed there is sufficient resource in the cluster,
Opensearch content is not exported. There is no data connection. No dashboards available in the dashboard list.
INFO Helm client version: 3.14.4
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
INFO Kubernetes client version: v1.27.12
INFO Kubernetes server version: v1.27.9
Deploying logging components to the [logging] namespace [Fri Jun 7 16:48:51 CEST 2024]
secret/internal-user-kibanaserver labeled
secret/v4m-root-ca-tls-secret created
secret/v4m-root-ca-tls-secret annotate
secret/v4m-root-ca-tls-secret labeled
secret/kibana-tls-secret created
secret/kibana-tls-secret annotate
secret/kibana-tls-secret labeled
"opensearch" has been added to your repositories
secret/v4m-osd-tls-enabled created
INFO Deploying OpenSearch Dashboards
Release "v4m-osd" does not exist. Installing it now.
NAME: v4m-osd
LAST DEPLOYED: Fri Jun 7 16:51:00 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
- Get the application URL by running these commands:
https://myexample.com/dashboards
INFO OpenSearch Dashboards has been deployed
serviceaccount/v4m-osd-dashboards patched
secret/internal-user-admin labeled
secret/internal-user-logcollector labeled
secret/internal-user-metricgetter labeled
secret/es-transport-tls-secret created
secret/es-transport-tls-secret annotate
secret/es-transport-tls-secret labeled
secret/es-rest-tls-secret created
secret/es-rest-tls-secret annotate
secret/es-rest-tls-secret labeled
secret/es-admin-tls-secret created
secret/es-admin-tls-secret annotate
secret/es-admin-tls-secret labeled
secret/opensearch-cert-subjects created
secret/opensearch-cert-subjects labeled
configmap/run-securityadmin.sh created
configmap/run-securityadmin.sh labeled
"opensearch" has been added to your repositories
INFO Copying OpenSearch Security Configuration files from [/home/ashokm/viya4-monitoring-kubernetes/logging/opensearch/securityconfig]
secret/opensearch-securityconfig created
secret/opensearch-securityconfig labeled
INFO Deploying OpenSearch
Release "opensearch" does not exist. Installing it now.
NAME: opensearch
LAST DEPLOYED: Fri Jun 7 16:56:13 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Watch all cluster members come up.
$ kubectl get pods --namespace=logging -l app.kubernetes.io/component=v4m-search -w
INFO Waiting on OpenSearch pods to be Ready
pod/v4m-search-0 condition met
Looping: 0 Fri Jun 7 14:59:01 UTC 2024
RC: 0
| run_securityadmin.sh script starting [Fri Jun 7 14:59:01 UTC 2024]
| **************************************************************************
| ** This tool will be deprecated in the next major release of OpenSearch **
| ** opensearch-project/security#1755 **
| **************************************************************************
| Security Admin v7
| Will connect to localhost:9200 ... done
| Connected as "CN=es-admin,O=v4m"
| OpenSearch Version: 2.12.0
| Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
| Clustername: opensearch-cluster
| Clusterstate: GREEN
| Number of nodes: 3
| Number of data nodes: 3
| .opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
| Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
| Will update '/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
| SUCC: Configuration for 'config' created or updated
| Will update '/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
| SUCC: Configuration for 'roles' created or updated
| Will update '/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
| SUCC: Configuration for 'rolesmapping' created or updated
| Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
| SUCC: Configuration for 'internalusers' created or updated
| Will update '/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
| SUCC: Configuration for 'actiongroups' created or updated
| Will update '/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
| SUCC: Configuration for 'tenants' created or updated
| Will update '/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml
| SUCC: Configuration for 'nodesdn' created or updated
| Will update '/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml
| SUCC: Configuration for 'whitelist' created or updated
| Will update '/allowlist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/allowlist.yml
| SUCC: Configuration for 'allowlist' created or updated
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| Done with success
INFO OpenSearch has been deployed
INFO Deploying Elasticsearch metric exporter ...
"prometheus-community" has been added to your repositories
Release "es-exporter" does not exist. Installing it now.
NAME: es-exporter
LAST DEPLOYED: Fri Jun 7 17:00:34 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
- Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace logging -l "app=v4m-es-exporter" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:9108/metrics to use your application"
kubectl port-forward $POD_NAME 9108:9108 --namespace logging
INFO Elasticsearch metric exporter has been deployed
INFO Loading Content into OpenSearch
ERROR Unable to identify the temporary port used for port-forwarding [v4m-search]; exiting script.
ERROR Exiting script [deploy_logging.sh] due to an error executing the command [logging/bin/deploy_opensearch_content.sh].
Resources deployed in logging namespace:
NAME READY STATUS RESTARTS AGE
pod/v4m-es-exporter-6854fd79f7-gl72g 1/1 Running 0 31m
pod/v4m-fb-459n2 0/1 Pending 0 11m
pod/v4m-fb-4n57z 1/1 Running 0 11m
pod/v4m-fb-9ztnh 1/1 Running 0 12m
pod/v4m-fb-kvrnl 1/1 Running 0 12m
pod/v4m-fb-lm79d 1/1 Running 0 12m
pod/v4m-fb-ncb6h 1/1 Running 0 12m
pod/v4m-fb-pkcl4 1/1 Running 0 12m
pod/v4m-fb-wqw25 1/1 Running 0 12m
pod/v4m-fb-zrg9p 1/1 Running 0 12m
pod/v4m-osd-78f8ddfc57-j98k5 1/1 Running 0 40m
pod/v4m-search-0 1/1 Running 0 35m
pod/v4m-search-1 1/1 Running 0 35m
pod/v4m-search-2 1/1 Running 0 35m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/v4m-es-exporter ClusterIP 10.0.3.144 9108/TCP 31m
service/v4m-fb ClusterIP 10.0.180.50 2020/TCP 12m
service/v4m-osd ClusterIP 10.0.253.147 5601/TCP 40m
service/v4m-search ClusterIP 10.0.121.64 9200/TCP,9300/TCP 35m
service/v4m-search-headless ClusterIP None 9200/TCP,9300/TCP,9600/TCP 35m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/v4m-fb 9 9 8 2 8 12m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/v4m-es-exporter 1/1 1 1 31m
deployment.apps/v4m-osd 1/1 1 1 40m
NAME DESIRED CURRENT READY AGE
replicaset.apps/v4m-es-exporter-6854fd79f7 1 1 1 31m
replicaset.apps/v4m-osd-78f8ddfc57 1 1 1 40m
NAME READY AGE
statefulset.apps/v4m-search 3/3 35m
I also tried executing deploy_osd_conten.sh that fails with same response:
INFO Helm client version: 3.14.4
INFO Kubernetes client version: v1.27.12
INFO Kubernetes server version: v1.27.9
INFO Loading user environment file: /home/ashokom/viya4-monitoring-kubernetes/logging/user.env
INFO Configuring OpenSearch Dashboards...this may take a few minutes
INFO Waiting for OpenSearch Dashboards pods to be ready (Fri Jun 7 17:29:02 CEST 2024 - timeout 10m)
pod/v4m-osd-78f8ddfc57-j98k5 condition met
INFO Waiting (up to more 8 minutes) for OpenSearch Dashboards API endpoint to be ready
ERROR Unable to identify the temporary port used for port-forwarding [v4m-osd]; exiting script.
ERROR Unable to obtain the URL for the OpenSearch Dashboards API Endpoint