MAGMA Testing Proposal

[arora-sagar]

Proposal Owner : OpenAirInterface Software Alliance (OSA)

Summary

Testing and integration is a continuous effort that is to be managed by an organization that has experience in community software management and experience in telecom R&D. The community members should all share a cozy feeling that such an organization is working for no other purpose than “the common good”. This team also needs to have a certain degree of depth and stability in its governance structure, as well as neutrality, to be able to manage a community-driven initiative. We believe OpenAirInterface Software Alliance (OSA) meets these criteria.

We propose to take the charge of Magma Continuous Integration (CI) pipelines/workflows and release cycle. We will take the charge of maintaining the existing build and testing infrastructure and extending it when new features are added or if testing of some features is missing. Our goal would be to deliver a robust, user-friendly (documented) and fully tested MAGMA core. On top of that, we also propose to extend/improve MAGMA’s deployment compatibility on different platforms like Docker, Vanilla Kubernetes, Enterprise Grade Kubernetes and Public clouds (AWS, Azure and GCP). Once implemented we will maintain these types of deployments and their proper documentation.

All produced code, binaries and documentation will use the Magma-compatible license "BSD 3-Clause License"

Use the google document link for detailed information related to task list and milestones.

[m-govind]

Few comments:

1. All the feature development, Integration testing, unit testing have to be platform neutral and work across different CPU architectures. Currently, magma CI jobs are running only on x86. Please add a high priority work item in the task list to port all the x86 CI jobs to Arm CI for Magma AGW component as a first step. Rest of the magma components can be ported later. Arm is working with Meta team to setup a basic Arm CI with docker build jobs and sanity traffic tests. OAI can leverage the Arm CI and bring it on par with x86 CI. Without prioritizing the complete porting of all x86 CI jobs to Arm CI, it is not guaranteed that all the LTE/5G features will work in Arm. Please add this work item in Milestone 1.
2. The proposal covers only functional test for 2 years. Please add a work item to add a performance framework in magma. This will help different vendors to measure KPIs like throughput, latency and GTPu call setup/tear down rate and publish the results in magma dashboard.
3. Performance test with OAI eNodeb/gNodeb: One of the work item is to do functional test with OAI e/gNodeb's. Whether this can be reused/extended for performance test also ? Whether OAI e/gNodeb Data plane/Transport stack is on par with Magma AGW Transport (OVS-kernel) in terms of features and perf ? Whether OAI Transport stack support all the features supported by OVS ? Whether OAI Transport stack is multi core friendly to scale up/down the data plane performance ?
4. We can add another work item to add Magma AGW support in cloud by generating AWS AMI for every Magma release. This is basically packaging containerized magma under AMI envelope or equivalent.
5. We need to make the S1AP test in magma repo to work in Arm. Currently, it doesn't work due to Vagrant/Virtual box dependency which is not supported in Arm. Meta team is working to port this test from Vagrant VM to container environment. This test case have to be added as part of Magma CI after Meta team complete the work.
6. Add/Maintain a new CI job to generate all the 3rd party debian packages on all the CPU architectures (Both Arm/X86). Meta team is currently working on this to generate those packages on Arm. OAI have to maintain this CI job going forward.

[arora-sagar]

Thank you for the comments @m-govind , most of the tasks you mentioned are already in the document. We have made a small editing to highlight certain things. The porting of MAGMA components depends on MAGMA dev community and containerized testing environment. Depending on how these two tasks are carry forward we will work on creating pipelines for ARM architecture. We updated the milestone based on what we think could work the best.

Here you can access the updated document

[ssanadhya]

@arora-sagar , in keeping with the TSC Grant Review criteria can you comment on the following?

1. security considerations of the testing framework
2. key performance indicators for the functional tests and scale tests

[thinhnt1983]

Hi @ssanadhya, thank you for your comments. We updated the proposal by adding two new paragraphs (Security and KPIs) to address your comments. Please feel free to take a look at the updated document and come back to us if necessary. Thanks