HOSTEDCP-1184: Document IPv6/IPv4/DualStack deployments for Hypershif…

…t in Baremetal Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com>
openshift · Oct 2, 2023 · 9f2a5b5 · 9f2a5b5
1 parent 046001c
commit 9f2a5b5
Show file tree

Hide file tree

Showing 111 changed files with 3,517 additions and 0 deletions.
diff --git a/.gitignore b/.gitignore
@@ -25,6 +25,7 @@ tools/bin
 *~
 .vscode
 .envrc
+.DS_Store
 
 .kube
 /kubeconfig

diff --git a/docs/content/images/diagram-hypershift-on-baremetal.png b/docs/content/images/diagram-hypershift-on-baremetal.png
diff --git a/docs/content/images/watch-cp.png b/docs/content/images/watch-cp.png
diff --git a/docs/content/images/watch-dp.png b/docs/content/images/watch-dp.png
diff --git a/docs/content/labs/Dual/dns.md b/docs/content/labs/Dual/dns.md
@@ -0,0 +1,120 @@
+The DNS configuration is a critical aspect of our setup. To enable name resolution in our virtualized environment, follow these steps:
+
+1. Create the primary DNS configuration file for the dnsmasq server:
+
+- `/opt/dnsmasq/dnsmasq.conf`
+```conf
+strict-order
+bind-dynamic
+#log-queries
+bogus-priv
+dhcp-authoritative
+
+# BM Network IPv4
+dhcp-range=dual,192.168.126.120,192.168.126.250,255.255.255.0,24h
+dhcp-option=dual,option:dns-server,192.168.126.1
+dhcp-option=dual,option:router,192.168.126.1
+
+# BM Network dual
+dhcp-range=dual,2620:52:0:1306::11,2620:52:0:1306::20,64
+dhcp-option=dual,option6:dns-server,2620:52:0:1306::1
+
+resolv-file=/opt/dnsmasq/upstream-resolv.conf
+except-interface=lo
+dhcp-lease-max=81
+log-dhcp
+no-hosts
+
+# DHCP Reservations
+dhcp-leasefile=/opt/dnsmasq/hosts.leases
+
+# Include all files in a directory depending on the suffix
+conf-dir=/opt/dnsmasq/include.d/*.dual
+```
+
+Create the upstream resolver to delegate the non-local environments queries
+
+- `/opt/dnsmasq/upstream-resolv.conf`
+```
+nameserver 8.8.8.8
+nameserver 8.8.4.4
+```
+
+Create the different component DNS configurations
+
+- `/opt/dnsmasq/include.d/hosted-nodeport.dual`
+```
+host-record=api-int.hosted-dual.hypershiftbm.lab,192.168.126.20
+host-record=api-int.hosted-dual.hypershiftbm.lab,192.168.126.21
+host-record=api-int.hosted-dual.hypershiftbm.lab,192.168.126.22
+host-record=api.hosted-dual.hypershiftbm.lab,192.168.126.20
+host-record=api.hosted-dual.hypershiftbm.lab,192.168.126.21
+host-record=api.hosted-dual.hypershiftbm.lab,192.168.126.22
+
+## IMPORTANT!: You should point to the node which is exposing the router.
+## You can also use MetalLB to expose the Apps wildcard.
+address=/apps.hosted-dual.hypershiftbm.lab/192.168.126.30
+
+dhcp-host=aa:aa:aa:aa:04:11,hosted-worker0,192.168.126.30
+dhcp-host=aa:aa:aa:aa:04:12,hosted-worker1,192.168.126.31
+dhcp-host=aa:aa:aa:aa:04:13,hosted-worker2,192.168.126.32
+dhcp-host=aa:aa:aa:aa:11:01,hosted-worker0,[2620:52:0:1306::30]
+dhcp-host=aa:aa:aa:aa:11:02,hosted-worker1,[2620:52:0:1306::31]
+dhcp-host=aa:aa:aa:aa:11:03,hosted-worker2,[2620:52:0:1306::32]
+```
+
+- `/opt/dnsmasq/include.d/hub.dual`
+```
+host-record=api-int.hub-dual.hypershiftbm.lab,192.168.126.10
+host-record=api.hub-dual.hypershiftbm.lab,192.168.126.10
+address=/apps.hub-dual.hypershiftbm.lab/192.168.126.11
+dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20
+dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21
+dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22
+dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25
+dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26
+
+host-record=api-int.hub-dual.hypershiftbm.lab,2620:52:0:1306::2
+host-record=api.hub-dual.hypershiftbm.lab,2620:52:0:1306::2
+address=/apps.hub-dual.hypershiftbm.lab/2620:52:0:1306::3
+dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5]
+dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6]
+dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7]
+dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8]
+dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9]
+```
+
+- `/opt/dnsmasq/include.d/infra.dual`
+```
+host-record=registry.hypershiftbm.lab,2620:52:0:1306::1
+host-record=registry.hypershiftbm.lab,192.168.126.1
+```
+
+To proceed, we must create a systemd service for the management of the dnsmasq service and disable the system's default dnsmasq service:
+
+- `/etc/systemd/system/dnsmasq-virt.service`
+```
+[Unit]
+Description=DNS server for Openshift 4 Clusters.
+After=network.target
+
+[Service]
+User=root
+Group=root
+ExecStart=/usr/sbin/dnsmasq -k --conf-file=/opt/dnsmasq/dnsmasq.conf
+
+[Install]
+WantedBy=multi-user.target
+```
+
+The commands to do so:
+
+```
+systemctl daemon-reload
+systemctl disable --now dnsmasq
+systemctl enable --now dnsmasq-virt
+```
+
+!!! note
+
+    This step is mandatory for both Disconnected and Connected environments. Additionally, it holds significance for both Virtualized and Bare Metal environments. The key distinction lies in the location where the resources will be configured. In a non-virtualized environment, a more robust solution like Bind is recommended instead of a lightweight dnsmasq.
diff --git a/docs/content/labs/Dual/hostedcluster/baremetalhost.md b/docs/content/labs/Dual/hostedcluster/baremetalhost.md
@@ -0,0 +1,128 @@
+## Bare Metal Hosts
+
+A **BareMetalHost** is an openshift-machine-api object that encompasses both physical and logical details, allowing it to be identified by the Metal3 operator. Subsequently, these details are associated with other Assisted Service objects known as Agents. The structure of this object is as follows:
+
+```yaml
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: hosted-dual-worker0-bmc-secret
+  namespace: clusters-hosted-dual
+data:
+  password: YWRtaW4=
+  username: YWRtaW4=
+type: Opaque
+---
+apiVersion: metal3.io/v1alpha1
+kind: BareMetalHost
+metadata:
+  name: hosted-dual-worker0
+  namespace: clusters-hosted-dual
+  labels:
+    infraenvs.agent-install.openshift.io: hosted-dual
+  annotations:
+    inspect.metal3.io: disabled
+    bmac.agent-install.openshift.io/hostname: hosted-dual-worker0
+spec:
+  automatedCleaningMode: disabled
+  bmc:
+    disableCertificateVerification: true
+    address: redfish-virtualmedia://[192.168.126.1]:9000/redfish/v1/Systems/local/hosted-dual-worker0
+    credentialsName: hosted-dual-worker0-bmc-secret
+  bootMACAddress: aa:aa:aa:aa:02:11
+  online: true
+```
+
+**Details**:
+
+- We will have at least 1 secret that holds the BMH credentials, so we will need to create at least 2 objects per worker node.
+- `spec.metadata.labels["infraenvs.agent-install.openshift.io"]` serves as the link between the Assisted Installer and the BareMetalHost objects.
+- `spec.metadata.annotations["bmac.agent-install.openshift.io/hostname"]` represents the node name it will adopt during deployment.
+- `spec.automatedCleaningMode` prevents the node from being erased by the Metal3 operator.
+- `spec.bmc.disableCertificateVerification` is set to `true` to bypass certificate validation from the client.
+- `spec.bmc.address` denotes the BMC address of the worker node.
+- `spec.bmc.credentialsName` points to the Secret where User/Password credentials are stored.
+- `spec.bootMACAddress` indicates the interface MACAddress from which the node will boot.
+- `spec.online` defines the desired state of the node once the BMH object is created.
+
+To deploy this object, simply follow the same procedure as before:
+
+!!! important
+
+    Please create the virtual machines before you create the BareMetalHost and the destination Nodes.
+
+To deploy the BareMetalHost object, execute the following command:
+
+```bash
+oc apply -f 04-bmh.yaml
+```
+
+This will be the process:
+
+- Preparing (Trying to reach the nodes):
+```
+NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
+clusters-hosted   hosted-worker0   registering              true             2s
+clusters-hosted   hosted-worker1   registering              true             2s
+clusters-hosted   hosted-worker2   registering              true             2s
+```
+
+- Provisioning (Nodes Booting up)
+```
+NAMESPACE         NAME             STATE          CONSUMER   ONLINE   ERROR   AGE
+clusters-hosted   hosted-worker0   provisioning              true             16s
+clusters-hosted   hosted-worker1   provisioning              true             16s
+clusters-hosted   hosted-worker2   provisioning              true             16s
+```
+
+- Provisioned (Nodes Booted up successfully)
+```
+NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
+clusters-hosted   hosted-worker0   provisioned              true             67s
+clusters-hosted   hosted-worker1   provisioned              true             67s
+clusters-hosted   hosted-worker2   provisioned              true             67s
+```
+
+## Agents registration
+
+After the nodes have booted up, you will observe the appearance of agents within the namespace.
+
+```
+NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411             true       auto-assign
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412             true       auto-assign
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413             true       auto-assign
+```
+
+These agents represent the nodes available for installation. To assign them to a HostedCluster, scale up the NodePool.
+
+## Scaling Up the Nodepool
+
+Once we have the BareMetalHosts created, the statuses of these BareMetalHosts will transition from `Registering` (Attempting to reach the Node's BMC) to `Provisioning` (Node Booting Up), and finally to `Provisioned` (Successful node boot-up).
+
+The nodes will boot with the Agent's RHCOS LiveISO and a default pod named "agent." This agent is responsible for receiving instructions from the Assisted Service Operator to install the Openshift payload.
+
+To accomplish this, execute the following command:
+
+```bash
+oc -n clusters scale nodepool hosted-dual --replicas 3
+```
+
+After the NodePool scaling, you will notice that the agents are assigned to a Hosted Cluster.
+
+```
+NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411   hosted    true       auto-assign
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412   hosted    true       auto-assign
+clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413   hosted    true       auto-assign
+```
+
+And the NodePool replicas set
+
+```
+NAMESPACE   NAME     CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION                              UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
+clusters    hosted   hosted    3                               False         False        4.14.0-0.nightly-2023-08-29-102237                                      Minimum availability requires 3 replicas, current 0 available
+```
+
+So now, we need to wait until the nodes join the cluster. The Agents will provide updates on their current stage and status. Initially, they may not post any status, but eventually, they will.
-Original file line number
+Diff line change
@@ Expand Up / @@ -25,6 +25,7 @@ tools/bin @@
     *~
     .vscode
     .envrc
+    .DS_Store
     .kube
     /kubeconfig
@@ Expand Down @@