Learned Automatic Modulation Recognition#

Goal#

Train an I/Q-only classifier to recognize BPSK, QPSK, and 16-QAM when each received frame has an unknown carrier phase, residual frequency offset, and AWGN. The learned model replaces only the Classifier block; the waveform source, impairment distribution, class vocabulary, SNR grid, and evaluation protocol remain fixed.

The comparison contains three methods:

  • Blind differential cumulant: a deployable I/Q-only classical baseline;

  • Learned blind-carrier classifier: the returned ONNX model;

  • Oracle-synchronized likelihood: a diagnostic reference that receives simulator-only carrier phase, frequency offset, and noise variance.

The oracle is not a deployable competitor. It shows the performance available when the nuisance parameters hidden from the two blind methods are known exactly.

CLI training summary#

Run this complete block from the repository. It exports a fresh training bundle, captures disjoint datasets, trains and evaluates the example model, and runs the paired benchmark.

(
  set -euo pipefail
  ROOT="$(git rev-parse --show-toplevel)"
  BUNDLE="$ROOT/.noema/training_exports/amc_blind_carrier"
  cd "$ROOT"

  uv sync --extra onnx
  uv run --project "$ROOT" --extra onnx noema differentiable export \
    "$ROOT/recipes/modulation_recognition_awgn.yaml" \
    --training-plan \
    "$ROOT/demo_trainings/modulation_recognition_supervised_cnn/training_plan.yaml" \
    --out "$BUNDLE"
  uv run --project "$ROOT" --extra onnx python \
    "$ROOT/demo_trainings/prepare_example.py" modulation-recognition "$BUNDLE" \
    --project-root "$ROOT"

  uv run --project "$ROOT" --extra onnx noema dataset-capture run \
    "$BUNDLE/capture_train_recipe.yaml" --out "$BUNDLE/data/train"
  uv run --project "$ROOT" --extra onnx noema dataset-capture run \
    "$BUNDLE/capture_validation_recipe.yaml" --out "$BUNDLE/data/validation"
  uv run --project "$ROOT" --extra onnx noema dataset-capture run \
    "$BUNDLE/capture_test_recipe.yaml" --out "$BUNDLE/data/test"

  cd "$BUNDLE"
  uv run --project "$ROOT" --extra onnx python validate_contract.py
  uv run --project "$ROOT" --extra onnx python train_demo.py
  uv run --project "$ROOT" --extra onnx python evaluate_demo.py

  cd "$BUNDLE/reference_training"
  uv run --project "$ROOT" --extra onnx python build_benchmark.py
  cd "$ROOT"
  uv run --project "$ROOT" --extra onnx noema benchmark validate \
    "$BUNDLE/reference_training/benchmark_pack.yaml"
  uv run --project "$ROOT" --extra onnx noema benchmark run \
    "$BUNDLE/reference_training/benchmark_pack.yaml"
)

Each capture command reports progress for its split. The final command prints the benchmark result ID and the paths to result.json, metrics.csv, recipes.csv, and summary.md.

1. Open the reusable scenario#

Start Noema from the repository root:

cd "$(git rev-parse --show-toplevel)"
uv run noema ui serve --port 8766

Open http://127.0.0.1:8766, select Browse template recipes, and open Physical layer & resource optimization > Automatic modulation recognition > Automatic modulation recognition with blind carrier impairments.

The scenario uses:

  • 128 complex symbols per frame;

  • balanced bpsk, qpsk, and qam16 classes;

  • SNR coordinates [-2, 2, 6, 10, 14, 18] dB;

  • carrier phase uniformly distributed over ([-\pi,\pi]);

  • residual frequency offset uniformly distributed over ([-0.006,0.006]) cycles per symbol.

The template opens with the blind differential-cumulant classifier. These system parameters belong to the recipe; no training architecture or loss is encoded in it.

2. Export and capture from Workbench#

Open Workbench and configure:

  1. Under Operation Training Capabilities, select Train/replace for Classifier.

  2. Under Dataset definition > Captured signals, keep:

    • iq_frames from observation.observation;

    • modulation_labels from data.labels.

    The labels are offline training targets, not runtime classifier inputs.

  3. Under Capture coordinates, leave Parameter sweep empty. The recipe matrix already supplies the six SNR coordinates.

  4. Under Dataset size and splits, set:

    • Total modulation frames: 10368;

    • Train: 66.6667%;

    • Validation: 16.6667%;

    • Held-out test: 16.6666%.

  5. Under Training bundle, set:

    • Bundle directory: .noema/training_exports/amc_blind_carrier;

    • Support framework: PyTorch.

  6. Select Export training bundle.

Attach the checked-in example:

cd "$(git rev-parse --show-toplevel)"
uv run --extra onnx python demo_trainings/prepare_example.py modulation-recognition \
  .noema/training_exports/amc_blind_carrier

Return to Workbench, select Capture all datasets, and wait until train, validation, and held-out test contain 6,912, 1,728, and 1,728 frames. These counts divide evenly across the six SNR coordinates.

3. Train and return the classifier#

From the bundle directory:

cd "$(git rev-parse --show-toplevel)/.noema/training_exports/amc_blind_carrier"
uv run --project ../../.. --extra onnx python validate_contract.py
uv run --project ../../.. --extra onnx python train_demo.py
uv run --project ../../.. --extra onnx python evaluate_demo.py

The model receives only [batch, 128, 2] real-valued I/Q frames. It does not receive SNR, carrier phase, frequency offset, or oracle synchronization. Its example architecture uses communication knowledge without changing the generic runtime interface:

  • per-frame power normalization removes an irrelevant amplitude scale;

  • adjacent-symbol differential features cancel constant carrier phase and expose modulation-order structure under residual frequency offset;

  • second- and fourth-order local products describe BPSK/QPSK rotational symmetries;

  • the differential-cumulant rule supplies an explicit classical prior;

  • a smaller, dropout-regularized temporal CNN learns residual corrections;

  • random carrier rotations augment the training frames, and validation-loss early stopping can retain the untrained prior if a learned correction does not generalize.

This compact design combines the low-SNR strength of raw-I/Q temporal convolutional classifiers described by O’Shea et al. with the phase-invariant signal structure used by the deployable cumulant baseline. Rotation augmentation is supported by the AMC study of Huang et al.; the residual path follows the broader finding that structured complex/residual models improve high-SNR classification without requiring a much deeper network (Krzyston et al.).

The returned files are:

  • artifacts/modulation_classifier.onnx;

  • trained_artifact.yaml;

  • reference_training/training_history.json;

  • reference_training/evaluation_metrics.json.

Workbench detects the returned artifact automatically. Select Validate returned model and continue when the status is model interface valid.

4. Run the paired comparison#

The checked-in benchmark uses six SNR coordinates, three fresh paired seeds, three methods, and 1,536 balanced frames per method and coordinate. At each coordinate, all methods receive the same transmitted symbols, carrier impairments, and AWGN realization.

cd "$(git rev-parse --show-toplevel)"
cd .noema/training_exports/amc_blind_carrier/reference_training
uv run --project ../../../.. --extra onnx python build_benchmark.py
cd ../../../..
uv run --extra onnx noema benchmark validate \
  .noema/training_exports/amc_blind_carrier/reference_training/benchmark_pack.yaml
uv run --extra onnx noema benchmark run \
  .noema/training_exports/amc_blind_carrier/reference_training/benchmark_pack.yaml

The primary metric is balanced accuracy. Accuracy and macro F1 are companion metrics; the per-method confusion matrices show which modulation pairs remain ambiguous.

Validity Checks#

A comparable demonstration requires:

  1. balanced class support and disjoint train, validation, test, and benchmark records;

  2. identical paired payload, noise, and carrier conditions across methods;

  3. an oracle-synchronized likelihood reference labeled as non-deployable;

  4. complete classwise denominators and confusion matrices;

  5. one frozen artifact hash across every learned benchmark run.

The demonstration is CPU-sized and controlled. It establishes the full Noema replacement and benchmarking workflow; it is not a RadioML or over-the-air performance claim.

Completed benchmark result#

The internally validated experimental result 20260727T150344Z_neural_receiver_ai_phy.learned_modulation_recognition_blind_carrier_post_training_v1 contains 54 completed runs: three methods, six SNRs, and three paired held-out seeds per SNR. Each run classifies 1,536 balanced frames. The bands are two-sided Student-t 95% confidence intervals over the three paired seeds and show tutorial-scale uncertainty.

Completed paired benchmark means#

SNR (dB)

Blind accuracy

Learned accuracy

Oracle accuracy

Blind macro F1

Learned macro F1

Oracle macro F1

-2

0.333333333333

0.421006944444

0.724826388889

0.166666666667

0.377604738152

0.724711837349

2

0.333333333333

0.678819444444

0.890407986111

0.166666666667

0.675407567309

0.890393212521

6

0.333333333333

0.822265625

0.999565972222

0.166666666667

0.82197136978

0.999565970567

10

0.668619791667

0.976345486111

1

0.561457752294

0.976314677174

1

14

1

0.994791666667

1

1

0.994791275475

1

18

1

0.996961805556

1

1

0.99696171449

1

The learned classifier improves substantially on the deployable blind cumulant baseline throughout the difficult −2 to 10 dB range. Its mean accuracy rises from 42.1% at −2 dB to 82.2% at 6 dB and 97.6% at 10 dB; the cumulant baseline obtains 33.3%, 33.3%, and 66.9% at those coordinates. At 14 and 18 dB the cumulant and oracle methods reach 100%, while the learned model reaches 99.5% and 99.7%. The remaining errors are a small number of 16-QAM frames classified as QPSK, so this result does not claim that learning dominates the classical rule at every SNR. The oracle-synchronized likelihood remains the diagnostic upper reference because it receives the simulator-only carrier phase and frequency offset.

The compact benchmark projection, chart data, and snapshot manifest preserve the plotted values, run identities, source hashes, paired-seed protocol, and training-evidence binding without copying runtime tensors into the documentation.

Verify a Local Result#

After the benchmark prints RESULT_ID:

cd "$(git rev-parse --show-toplevel)"
uv run --extra onnx noema benchmark verify RESULT_ID