Benchmark Submissions#

This document defines the local submission rules for Noema benchmark results.

Submission route#

  1. Validate the complete bundle locally as described below.

  2. Open a proposal at M0574F4/noema-lab#new with the benchmark ID, comparison division, bundle size, and a digest. Do not attach private data or security-sensitive details.

  3. Submit small metadata/fixture changes through a pull request. For large result bundles, wait for the acting maintainer to approve an immutable artifact location before upload.

The current repository does not operate a private leaderboard or automatic intake service.

Allowed Modifications#

Closed division submissions may change only the method contribution:

  • codec, semantic encoder, receiver, channel model, channel code, modulation block, metric, or dataset adapter when the benchmark allows that contribution type;

  • runtime/backend choices declared in the recipe and manifest;

  • method-specific hyperparameters declared in the recipe.

Closed division submissions must not change:

  • canonical benchmark dataset/split/sample IDs;

  • metric definitions or reductions;

  • rate/channel fixed points;

  • reference data or ground truth;

  • benchmark pack ID/version;

  • seed policy, except where the protocol explicitly defines a seed grid.

Open division submissions may change more of the system, but must declare every difference in the submission metadata and should not be compared directly with closed-division results.

Required Metadata#

Submissions must include:

  • method name and authors;

  • comparison division: closed or open;

  • adapter manifest path/name/version when an adapter is used;

  • benchmark ID and version;

  • result bundle path;

  • recipe SHA-256;

  • hardware, runtime, software dependencies, and optional accelerator details;

  • seed policy and concrete seeds;

  • any external data, checkpoint, or pretrained artifact identifiers.

Validate Locally#

uv run noema submission validate path/to/submission.yaml

The validator requires a normalized relative result_bundle path confined to the submission directory and rejects traversal, absolute paths, and symlinks. It invokes the full benchmark verifier, including result-local run evidence, protocol identity, artifacts, reports, plots, resource admission, and training-lineage checks. It also cross-checks every recipe digest, concrete master seed, recorded seed-policy digest, and the comparison division frozen in benchmark.json.

The default successful verdict is verified_internal_consistency_untrusted_producer. That verdict is appropriate for an honest producer and accidental-drift detection; it does not authenticate hostile input. API callers can pass an independently obtained trusted_bundle_root_sha256 to receive verified_pinned_bundle_integrity after the exact bundle inventory matches. A pin proves byte identity, not author identity or scientific truth; signatures and producer identity remain release-policy responsibilities outside this local validator.

For multi-recipe results, reproducibility.recipe_sha256s must map every benchmark recipe ID to the recorded run recipe digest. reproducibility.seed_policy_sha256s must list the distinct SHA-256 identities of the full recorded seed-policy objects. A single-recipe submission may put the one seed-policy digest directly in seed_policy; the human-readable field remains available when the digest list is supplied. The frozen benchmark must declare either metadata.comparison_division or metadata.allowed_comparison_divisions.

Minimal Submission YAML#

schema_version: 1
comparison_division: closed
result_bundle: .noema/benchmarks/20260708T000000Z_benchmark_v1.image_reconstruction.kodak
method:
  name: My Semantic Codec
  authors:
    - A. Researcher
adapter:
  manifest: adapters/my_codec/noema_adapter.yaml
  name: my_codec
  version: 0.1.0
benchmark:
  id: benchmark_v1.image_reconstruction.kodak
  version: "1.0.0"
runtime:
  hardware: CPU
  software: noema 0.2.0.dev0
  dependencies:
    - numpy
reproducibility:
  recipe_sha256: "<sha256 from manifest>"
  seed_policy: noema deterministic step-derived seeds
  seed_policy_sha256s:
    - "<sha256 of the complete recorded seed_policy object>"
  seeds:
    - 23