Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(agw): Automated Deployment of Magma in Kubernetes Cluster #15480

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

brunohcfaria
Copy link
Collaborator

Summary

This PR allows the deployment of Magma on the nodes without the requirement to prepare the nodes specially before the deployment.

This PR is also the implementation of the proposal discussed in PR #15357.

Following is a list of fatures provided by this PR:

  • Added automation for the vport-gtp custom kernel module deployment
  • A DeamonSet is used to deploy a vport-gtp custom kernel module, allowing OVS to handle GTP packets.
  • The openvswitch daemons run as pod (or containers)
  • Ansible roles prepare the master and worker nodes before deployment

Test Plan

  • Test the building of new ovs image.
  • Run the LTE dockerized integration tests
  • Tested 5GNR with UERANSIM

Additional Information

Added a new docker image for openvswitch-switch service.

  • This change is backwards-breaking

Security Considerations

@pull-request-size pull-request-size bot added the size/XXL Denotes a Pull Request that changes 1000+ lines. label Aug 2, 2024
Copy link
Contributor

github-actions bot commented Aug 2, 2024

Thanks for opening a PR! 💯

A couple initial guidelines

Howto

  • Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
  • Checks. All required CI checks must pass before merge.
  • Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

If this is your first Magma PR, also consider reading

@github-actions github-actions bot added component: agw Access gateway-related issue component: ci All updates on CI (Jenkins/CircleCi/Github Action) component: cwf component: feg FEG-gateway related issues labels Aug 2, 2024
Copy link
Contributor

github-actions bot commented Aug 2, 2024

✔️ The Semantic PR check ended with status success. See instructions on formatting your commit and pull request titles.

ENV MAGMA_DEV_MODE 0
ENV XDG_CACHE_HOME ${MAGMA_ROOT}/.cache

RUN apt-get update && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

cwf/gateway/docker/c/Dockerfile Outdated Show resolved Hide resolved
cwf/gateway/docker/c/Dockerfile Outdated Show resolved Hide resolved
ln -s /usr/sbin/bazelisk-linux-"${DEB_PORT}" /usr/sbin/bazel

# Install dependencies required for building
RUN apt-get update && apt-get install -y \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

cwf/gateway/docker/c/Dockerfile Outdated Show resolved Hide resolved
# Add the magma apt repo
COPY keys/linux_foundation_registry_key.asc /etc/apt/trusted.gpg.d/magma.asc
RUN echo "deb https://linuxfoundation.jfrog.io/artifactory/magma-packages-test focal-ci main" > /etc/apt/sources.list.d/magma.list
RUN apt-get update && apt-get install -y \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

ENV LINUX_HEADERS_VER 5.4.0-186-generic

## Install packages
RUN apt-get -q update && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

ENV LINUX_HEADERS_VER 5.4.0-186-generic

## Install packages
RUN apt-get -q update && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [hadolint] <DL3009> reported by reviewdog 🐶
Delete the apt-get lists after installing something

lte/gateway/docker/services/openvswitch/Dockerfile Outdated Show resolved Hide resolved
lte/gateway/docker/services/openvswitch/Dockerfile Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

markdownlint

[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

- You shall see the joined nodes with the command `kubectl get nodes`.


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

#### Deploying in single node


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

If you are running a single node cluster, make sure that the master node is available to the scheduler by untainting it.


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

Follow the AGW Helm Deployment guide in `$MAGMA_ROOT/lte/gateway/deploy/agwc-helm-charts` directory.


[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

If you are deploying AGW in multiple workers, ensure the Node Selection is configured properly in the Helm charts deployment files.

Copy link
Contributor

github-actions bot commented Aug 2, 2024

FeG Lint & Test

    2 files  203 suites   39s ⏱️
374 tests 374 ✔️ 0 💤 0
388 runs  388 ✔️ 0 💤 0

Results for commit b788780.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Aug 2, 2024

DP Lint & Test

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0

Results for commit b788780.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Aug 2, 2024

Orc8r Lint & Test

1 148 tests   1 148 ✔️  2m 13s ⏱️
   365 suites         0 💤
       7 files           0

Results for commit b788780.

♻️ This comment has been updated with latest results.

@brunohcfaria brunohcfaria changed the title Automated Deployment of Magma in Kubernetes Cluster feat(agw): Automated Deployment of Magma in Kubernetes Cluster Aug 6, 2024
ln -s /usr/sbin/bazelisk-linux-"${DEB_PORT}" /usr/sbin/bazel

# Install dependencies required for building
RUN apt-get update && apt-get install -y --no-install-recommends \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

# Add the magma apt repo
COPY keys/linux_foundation_registry_key.asc /etc/apt/trusted.gpg.d/magma.asc
RUN echo "deb https://linuxfoundation.jfrog.io/artifactory/magma-packages-test focal-ci main" > /etc/apt/sources.list.d/magma.list
RUN apt-get update && apt-get install -y --no-install-recommends \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [hadolint] <DL3008> reported by reviewdog 🐶
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

title: Deploy AGW in Kubernetes
hide_title: true
---
# Deploy AGW in Kubernetes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "# Deploy AGW in Kubernetes"]

hide_title: true
---
# Deploy AGW in Kubernetes
## Prerequisites
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Above] [Context: "## Prerequisites"]

hide_title: true
---
# Deploy AGW in Kubernetes
## Prerequisites
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Prerequisites"]

Make sure you setup the master and the worker nodes. They must run Ubuntu Server 20.04 LTS, and the master node must have ssh key access to the worker nodes with the user `magma`.
Optionally you can setup an ansible controller where you will run the deploy script or run it directly from the master node.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]


The worker nodes must satisfy the following requirements:


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]

- aarch64 or 64bit-X86 machine.
- Two ethernet ports.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]


The interfaces are renamed to `eth0` and `eth1` during the deploy process. Interface `eth0` will carry any traffic that is not S1. That is,data plane traffic (SGi), control plane traffic (Orc8r HTTP2) and management (ssh). The interface `eth1` carries S1 traffic.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]



> NOTE:
> - Interface names might have different names on your hardware, so just
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "> - Interface names might have..."]


- Configure ssh key access from the worker to the nodes for the `magma` user.

- Add your master and worker nodes to the ansible inventory file at `$MAGMA_ROOT/lte/gateway/deploy/agw_k8s_hosts.yml`. Make sure that the `ansible_user` is set to `magma` and the `ansible_sudo_pass` password is right. Set local ansible connection for the master node if you are deploying from it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

hide_title: true
---
# Deploy AGW in Kubernetes
## Prerequisites
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
## Prerequisites
## Prerequisites

---
# Deploy AGW in Kubernetes
## Prerequisites
Make sure you setup the master and the worker nodes. They must run Ubuntu Server 20.04 LTS, and the master node must have ssh key access to the worker nodes with the user `magma`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
Make sure you setup the master and the worker nodes. They must run Ubuntu Server 20.04 LTS, and the master node must have ssh key access to the worker nodes with the user `magma`.
Make sure you setup the master and the worker nodes. They must run Ubuntu Server 20.04 LTS, and the master node must have ssh key access to the worker nodes with the user `magma`.

Make sure you setup the master and the worker nodes. They must run Ubuntu Server 20.04 LTS, and the master node must have ssh key access to the worker nodes with the user `magma`.
Optionally you can setup an ansible controller where you will run the deploy script or run it directly from the master node.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change


The worker nodes must satisfy the following requirements:


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change

- aarch64 or 64bit-X86 machine.
- Two ethernet ports.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change


The interfaces are renamed to `eth0` and `eth1` during the deploy process. Interface `eth0` will carry any traffic that is not S1. That is,data plane traffic (SGi), control plane traffic (Orc8r HTTP2) and management (ssh). The interface `eth1` carries S1 traffic.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change



> NOTE:
> - Interface names might have different names on your hardware, so just
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
> - Interface names might have different names on your hardware, so just
>
> - Interface names might have different names on your hardware, so just


- Configure ssh key access from the worker to the nodes for the `magma` user.

- Add your master and worker nodes to the ansible inventory file at `$MAGMA_ROOT/lte/gateway/deploy/agw_k8s_hosts.yml`. Make sure that the `ansible_user` is set to `magma` and the `ansible_sudo_pass` password is right. Set local ansible connection for the master node if you are deploying from it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
- Add your master and worker nodes to the ansible inventory file at `$MAGMA_ROOT/lte/gateway/deploy/agw_k8s_hosts.yml`. Make sure that the `ansible_user` is set to `magma` and the `ansible_sudo_pass` password is right. Set local ansible connection for the master node if you are deploying from it.
- Add your master and worker nodes to the ansible inventory file at `$MAGMA_ROOT/lte/gateway/deploy/agw_k8s_hosts.yml`. Make sure that the `ansible_user` is set to `magma` and the `ansible_sudo_pass` password is right. Set local ansible connection for the master node if you are deploying from it.

brunohcfaria and others added 10 commits August 7, 2024 23:02
* Fixed magmad starting command to create MME/AMF
config files upon start.

* Added StorageClass and PersistentVolume
objects needed for deployment

* Set the Magma image repository URL to
base jfrog address.

Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
module

The Init Container is responsible for loading the
vport-gtp module into the host node.

Signed-off-by: Bruno Faria <bruno.faria@ektrum.com>
* Added openvswitch Dockerfile and compose for running
openvswitch in container

Signed-off-by: Bruno Faria <bruno.faria@ektrum.com>
…yaml

* The ovs image is build if the option "--profile ovs" is passed to docker compose
* Added the ovs container healthcheck script
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
…interface

during pod startup

Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
…mentation

Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
cwf/gateway/docker/go/Dockerfile Outdated Show resolved Hide resolved
feg/gateway/docker/go/Dockerfile Outdated Show resolved Hide resolved
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
@github-actions github-actions bot added component: orc8r Orchestrator-related issue and removed component: cwf component: feg FEG-gateway related issues labels Aug 7, 2024
Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
@github-actions github-actions bot removed the component: orc8r Orchestrator-related issue label Aug 8, 2024
@brunohcfaria
Copy link
Collaborator Author

@panyogesh, I'm struggling to fix some of the failing checks in the PR. They are related to go builds, but the PR didn't change anything at that level. Do you have any thoughts on that?

Signed-off-by: Bruno Faria <brunohcfaria@gmail.com>
@lucasgonze
Copy link
Contributor

Blocked on #15573

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: agw Access gateway-related issue component: ci All updates on CI (Jenkins/CircleCi/Github Action) size/XXL Denotes a Pull Request that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants