Testing a 5G network, end to end

Foundations

2. What a test framework is

Six pieces, mapped to this system:

Designing one: the order that works

1. Driver abstraction first. Tests express intent ("attach 50 UEs", "push an A1 policy", "cut the fronthaul fibre"), never raw protocol. Each driver has a simulated and a real backend behind the same interface.
2. Topology as code. One declarative spec for the network under test drives docker-compose, Kubernetes, and lab reservation.
3. Deterministic identity. Every run gets an ID; every SUPI range, tenant, and PDU session created carries it. Reruns are idempotent; teardown is one predicate.
4. Async oracles. eventually, always, observed. Telecom assertions are never synchronous; forbid sleep().
5. Fault injection as an API. Fibre pull, PTP loss, pod kill, partition, latency. If chaos is manual it never runs.
6. Evidence on failure by default. Capture at the seam that failed, plus one seam either side.

Foundations

3. Levels on a RAN and core

On coverage

80% line coverage is the usual target for stack code, 60% a floor, and neither is a quality score. An ASN.1 decoder can be 100% covered by tests that never assert on a single IE value. Branch coverage is more honest; mutation testing (module 9) is the real answer.

Foundations

4. Types by purpose

A type answers what property is being checked. Types run at every level; a type is not a level. Functional tests exist at unit (does the encoder encode), integration (does F1 Setup carry the cell config), and system (does the UE get an IP). Keeping the two axes separate is the most common vocabulary fix a test team needs.

Foundations

5. How tests get executed

Manual

A human executes. Scripted manual (follow the test case) is what you automate. Exploratory is what you cannot automate and is where a good system tester earns their pay: every new radio, every new vendor pairing, every new release gets an exploratory session before the automation is trusted.

Automation

Tests executed by code. Not only "automated manual tests": TRex load runs, 7-day soaks, chaos injections, fuzzing, and per-commit simulator suites were never manual. Automation pays only when the suite is reliable enough that red means something. A flaky E2E suite gets ignored within a month.

Black, white, grey box

Black box: through the external interface only (system, field). White box: with code knowledge (unit). Grey box: external stimulus, internal observation. A system tester who reads the AMF's Prometheus counters and the DU's log while driving a UE through registration is testing grey box, and that is the right default for distributed systems.

Static

Examine code without running it. Non-negotiable for C/C++ stack code.

Tools: clang-tidy, cppcheck, CodeQL, Semgrep, golangci-lint, mypy, MISRA checkers on embedded targets; ASan/UBSan builds in CI count as nearly static.

Practices: TDD, BDD, shift-left

TDD for protocol encoders and state machines works well: the spec gives you the test before the code. BDD (Given/When/Then) is common in operator test plans because acceptance criteria are already written that way. Shift-left in telecom means: simulator in CI before the lab, static analysis before the simulator, and conformance subsets before the OTIC visit.

Practice

6. Test design from the spec

Telecom is spec-driven, which is an advantage: the test cases are latent in 3GPP and O-RAN documents. The techniques below are how you extract them without producing ten thousand redundant cases.

Procedures become scenarios

Every 3GPP procedure (TS 23.502 for the core, TS 38.331 for RRC, TS 38.413 for NGAP, TS 38.473 for F1AP) is a sequence diagram with a success path and enumerated failure paths. Each success path is one E2E scenario. Each failure path, with its cause code, is a negative scenario. Registration alone yields the accept path plus rejects for illegal UE, PLMN not allowed, roaming not allowed, no suitable cells, congestion, and the abnormal cases (T3510 expiry, lower-layer failure, collision with deregistration).

State machines become transition tables

5GMM (deregistered, registered-initiated, registered, deregistered-initiated, service-request-initiated), 5GSM, RRC (idle, inactive, connected), SCTP associations. Test every valid transition, then every event arriving in a state that must ignore or reject it. The invalid transitions are where implementations diverge.

IE tables become boundary cases

Every message has an IE table with presence (mandatory, optional, conditional), criticality (reject, ignore, notify), range, and multiplicity. Each row is a set of cases: absent when mandatory, present when conditional condition is false, at minimum, at maximum, one over maximum, unknown IE with criticality reject versus ignore. This is equivalence partitioning and boundary value analysis applied to ASN.1, and it finds most interop bugs before the plugfest.

Timers become negative cases

T3510, T3580, T3346, RRC inactivity, PFCP heartbeat, SCTP heartbeat, NGAP guard timers. For each: expiry once, expiry at the retry limit, response arriving one tick after expiry, and clock skew between peers.

Configuration space becomes a pairwise set

Band, bandwidth, subcarrier spacing, TDD pattern, MIMO layers, slice count, security algorithms, PDU session type: exhaustive is thousands of combinations; pairwise is dozens. Generate the set, then add the specific combinations the operator will deploy.

Mobility and multi-UE become scenario grids

Intra-DU, inter-DU intra-CU (F1), inter-CU (Xn), inter-AMF (N2) handover; RRC re-establishment; RRC inactive to connected; two UEs with the same SUPI; a UE that never answers the security-mode command. Each cell in the grid is a case; each failing cell is usually a retainability KPI in the field.

Risk-based depth

Not every case gets every rung. Rank by likelihood (new code, new vendor pairing, historic defect density) times impact (service-affecting, security, regulatory) and give the top of the list all three system rungs and a soak; give the bottom the simulator and move on.

Practice

7. Strategy by rung

A telecom test strategy is organised around the three rungs of system test, because they are the resource constraint. Everything cheap runs on every commit; everything that needs hardware is scheduled.

Entry and exit criteria that mean something

Entry to the RF rung: sim suite green, PTP locked, spectrum clear in the shielded box, lab reserved. Exit from the scale rung: KPI thresholds met with the method named, no NF crash in the soak, no open severity-1. Exit criteria written as "all tests pass" get waived on release day; criteria written as specific KPIs with thresholds do not.

PoC exit criteria

For a proof of concept with an operator or enterprise, write the exit criteria before the hardware ships: which procedures, on which rung, at which KPI threshold, with which test method, witnessed by whom. A PoC without written exit criteria ends in a disagreement about whether it passed.

Estimation

Simulator cases cost minutes each to run and hours to write. RF cases cost a lab slot plus setup; budget a day per new radio for bring-up before any case runs. Scale cases cost the rig for a day and analysis for another. Soak is calendar time: budget two weeks of continuous rig for a release. Write the assumptions next to the numbers.

Practice

8. Lab and environment

Most telecom test programmes fail in the lab, not in the test design. A lab nobody can reproduce, a radio three teams fight over, a grandmaster with drifting config: these produce more false failures than any number of bad assertions.

Environment tiers

Rules that keep the lab trustworthy

• Provision from code, including the grandmaster config, switch VLANs, and NIC settings. If it cannot be rebuilt from a repo, it will drift.
• Reserve every piece of hardware through a queue. A radio taken by a chat message is a radio with unknown state.
• Pin versions of simulators and test tools as strictly as product versions. A UERANSIM upgrade that silently changes NAS behaviour looks like a product regression.
• Timing first: verify PTP lock and offset before any RF test run, automatically, as an entry check.
• Shielded box for real UEs: legal, repeatable, and it keeps the neighbours' networks out of your pcaps.
• Reset after every destructive test, and snapshot config before every exploratory session.

Test data

Subscriber provisioning (SUPI ranges, K/OPc, slices, DNNs) generated per run and loaded into UDM/UDR by the harness. Every UE a test creates carries the run ID in its SUPI range so cleanup is one query and parallel runs never collide. Real SIM cards for the RF rung are inventory: tracked, provisioned to match, never hand-edited.

Practice

9. Telecom techniques

Practice

10. Defects and RCA

Lifecycle

Verification happens on the rung where the defect was found, at minimum. A fronthaul bug found on the bench is not verified on the simulator.

Severity in telecom terms

Priority is set separately by the customer impact and timeline. A minor-severity defect blocking an OTIC certification is high priority.

What a telecom defect report contains

• Rung and environment: sim, bench RF (which radio, which firmware), scale, field. Build IDs for every element in the path.
• Capture-point discipline: which seam the pcap is from, which plane, which timing domain. A fronthaul eCPRI capture and an SBI HTTP/2 capture are different worlds.
• The procedure and the step it failed at, with the cause code if one was sent.
• Minimal reproduction: one UE, one PDU session, exact config.
• Evidence: pcaps from the failing seam and one either side, NF logs, PTP offset, core dump if any, config snapshot.
• Frequency across runs, and whether it reproduces on a lower rung.
• KPI impact: which KPI, by how much, for which users.

Root cause analysis

For every escaped defect and every critical, ask which level should have caught it. A fronthaul IE mismatch that reached the field should have been a one-seam integration test; a session leak found by a customer should have been a soak. The answer is a new test at that level, and the escape count by level is the metric that tells you where the strategy is thin.

Practice

11. KPIs and reporting

Telecom has a settled KPI vocabulary. Use it in test plans so the operator's acceptance team and your system test team are measuring the same things.

Test-programme metrics

Requirement coverage by procedure and by rung; pass rate trend per rung; open defects by severity and by seam; escapes by level (the RCA metric); flaky rate on the sim suite; lab utilisation and queue wait; time from commit to first sim red.

The release report

One page per rung. What ran, on which hardware and firmware, KPI results against thresholds with the method named, open defects with severity and seam, known limitations, and a recommendation: ready, ready with conditions, not ready. Conformance evidence is attached as its own artifact set because it will be asked for by a certification body or an operator's procurement, not by engineering.

Tooling

12. Tools and stack

Browser automation (Selenium, Playwright) belongs to the SMO and portal UI only. Nothing in the RAN or core is browser-driven; a telecom test role that leads with Selenium is a portal QA role.

Tooling

13. Run a pipeline

A CI gate for this stack, cheap first, expensive last. Run it clean, then inject a bug at each level and watch where it gets caught and what the failure message can tell you.

The cost curve: the NAS bug named the function in 90 seconds. The F1 mismatch needed a pcap. The PTP drift passed every software stage and only failed on real hardware. The failover bug survived to the chaos step and produced a message that names no component at all, which is where the next module begins.

Tooling

14. The debug ladder

When a system test on this stack goes red, the failure rarely names the culprit. Work the ladder top to bottom; do not skip a rung because the symptom "looks like" a later one.

Product, harness, or environment

Before opening a defect: did the simulator version change, did the lab config drift, did someone else have the radio. In a young suite a third of system reds are not product bugs. The harness must make this cheap to answer: pinned versions, provisioning from code, reservation logs.

Where gdb and valgrind enter

A crash anywhere in the ladder ends with a core dump. gdb on the core: bt, thread apply all bt, frame, info locals, why the symbols are missing (stripped binary, no debuginfo, cross-compiled target needing set sysroot). Know how cores are produced in a container: ulimit -c, core_pattern, systemd-coredump, and why a pod has none.

valgrind appears in one telecom scenario: a soak run shows RSS growing on the AMF or DU. Memcheck localises it, but at 10 to 50x slowdown it cannot run under real traffic or real-time threads, so reproduce at reduced scale on a debug build, or use ASan/LSan compiled in. A leak and unbounded growth are not the same thing: glibc arena behaviour, caches, and fragmentation all look like leaks.

The output of a debug session is not a fix. It is a defect report with the pcap from the right seam, the core, the logs, the config, the rung it was seen on, and a reproduction that takes the developer straight to the frame.

Wrap up

15. Check yourself

Wrap up

16. Interface glossary

Uu

Air interface between UE and gNB. RRC and NAS ride on it. 3GPP TS 38.300 series.

Open Fronthaul (CUS-plane)

O-RAN 7.2x split between O-DU and O-RU: control, user, synchronisation planes over eCPRI. Microsecond timing domain.

M-Plane

Management plane for the O-RU: NETCONF/YANG from the O-DU or SMO. Configuration, software, notifications, sync state.

F1-C / F1-U

O-DU to O-CU: F1AP control (TS 38.473) and GTP-U user plane.

E1

CU-CP to CU-UP: E1AP bearer context management (TS 38.463).

N2

gNB to AMF: NGAP over SCTP (TS 38.413). Registration, context setup, PDU session resources.

N3

gNB to UPF: GTP-U user plane.

N4

SMF to UPF: PFCP session control (TS 29.244).

N6

UPF to data network.

SBI

Service-based interface between core NFs: HTTP/2 with 3GPP-published OpenAPI (TS 29.5xx series).

Xn

gNB to gNB: handover and dual connectivity (TS 38.423).

E2

Near-RT RIC to E2 nodes (DU and CU): E2AP with service models such as KPM and RC.

A1

Non-RT RIC to Near-RT RIC: policy management.

O1

SMO to O-RAN managed elements: NETCONF/YANG configuration, VES telemetry, PM files.

O2

SMO to O-Cloud: infrastructure and deployment management.

PTP / SyncE

IEEE 1588 phase and SyncE frequency distribution from grandmaster through O-DU to O-RU. ±1.5 µs end-to-end phase budget is the usual TDD target.

OTIC

O-RAN Open Test and Integration Centre: accredited labs that run conformance and interoperability specs and issue certificates.

Rung

One of the three system-test environments: simulator, real RF, scale. Results are always labelled by rung.

Written by telecogeco, a ugeco brand. The levels, types and techniques are general test practice for a RAN and core; the architecture map, the pipeline and the quiz are illustrations with synthetic results, not a record of any test run.