// Benchmarks

What we measure ourselves against,
and where we lose.

Every result is measured against an established tool or a published result, each answer checked, not only timed. Below, every comparison in the same format; where PQCF is slower or incomplete is part of the format.

// Every comparison, one format

Fifteen comparisons,
in one format.

1
faster
PQCF faster than the reference
0
comparable
within run-to-run variation of the reference
3
slower
the reference, or the classical method, is faster
3
mixed
faster at some tasks, slower at others
8
not timed
not yet benchmarked for speed: the answers compared
0
Independently reviewed
none yet: every comparison here is ours

The hardware, unless a comparison says otherwise: one NVIDIA DGX Spark: CPU 20 Arm cores (10 Cortex-X925 and 10 Cortex-A725), 128 GB of memory shared with its GB10 GPU, Ubuntu 24.04 (Linux 6.17, aarch64), every program in Docker. The machine is shared with other work, so times vary from run to run by up to about a third. Both programs of a comparison run on the same machine and the same cores, one after the other. Open a comparison for its details.

Large-scale symbolic computation

PQCF's symbolic engine against FORM 5, every week

mixed
Problem
Expanding large polynomials (dense and sparse products, powers, symbolic permanents, an interferometer's symbolic matrix); compiling them into programs; gcd and factorisation.
Version
PQCF commit 92e2aeb (after release 1.4.0)
Result
Dense products 3 to 5 times faster; permanents from a third of FORM's time to the same; sparse products and powers 1.5 to 2.8 times slower; compiled programs need fewer operations than FORM's O4 but take longer to find; gcd level with FORM, factorisation 4 to 8 times slower.
Reference
FORM 5 (Vermaseren et al.), at the version we measured.
Agreement
Identical term for term on every task.
Hardware
One core each, the same core.
Reproduction
comparison/form/REPRODUCE.md; you should see: every task ends in ok; about 80 seconds once FORM is built. Details →
Limitations
FORM is still ahead on sparse products, powers, factorisation and the time to compile; its parallel version and disk-based expressions reach far beyond these sizes.

PQCF against the photonic simulators

The Walrus, Piquasso, Perceval, DeepQuantum, Strawberry Fields, every week

mixed
Problem
Permanents, torontonians, exact boson sampling, Fock-space simulation of a 10-mode network.
Version
PQCF commit 92e2aeb
Result
The boson sampler is the fastest at every size (4 to 360 times Piquasso, 13 to 66 times Perceval). The permanent is the slowest of the four compiled programs (Perceval 1.7 times faster); the torontonian second (Piquasso 2.7 times faster); the Fock backend 3 times slower than Piquasso's.
Reference
The five programs at fixed versions.
Agreement
Every check agrees (50 of 50 in the run of 15 November): permanents within 1e-8, torontonians within 1e-7, samples by chi-square tests against the exact distribution.
Hardware
One core, one thread per program.
Reproduction
comparison/photonic/REPRODUCE.md; you should see: 50 of 50 checks agree (the count depends on the sizes reached); about 13 minutes. Details →
Limitations
Three kernels behind the best; DeepQuantum's GPU paths not compared.

Reducing Feynman integrals

Against FIRE 6 on its own double box

faster
Problem
Reducing the two-loop massless double box to master integrals.
Version
PQCF 1.4.0; rerun at commit 92e2aeb
Result
3 to 7 times faster: 1.21 s against 7.99 for FIRE's target, 3.73 s against 11.99 for a harder one.
Reference
FIRE 6.5.2 with the FLINT simplifier, built from source.
Agreement
Equal master by master at a random point modulo a large prime.
Hardware
One core each, the same core.
Reproduction
comparison/fire/REPRODUCE.md; you should see: every case equal; about 90 seconds. Details →
Limitations
The double box is small by research standards; Kira not compared (it needs Fermat, x86 only, with a restrictive licence).

Four-loop integrals from QCD

Against FORM's Forcer on FORM's own checks

slower
Problem
The four-loop integrals of FORM's Forcer checks, reduced to master integrals.
Version
PQCF commit 92e2aeb
Result
Every answer exact. Forcer, built for these integrals, is 17 to 25 times faster on four loops; PQCF is 13 times faster on the three-loop example.
Reference
Forcer's results and the numbers FORM's checks assert.
Agreement
All 10 integrals equal to Forcer's as expansions in ε; all 5 MS-bar results equal to FORM's.
Hardware
FORM with Forcer on 1 core; PQCF on 16.
Reproduction
comparison/forcer/REPRODUCE.md; you should see: every check equal; PQCF's results unchanged; about 7 minutes on 8 cores. Details →
Limitations
PQCF builds and solves a system for every problem; Forcer solves its rules in advance.

Quantum optics

Against QuTiP 5.3.1, QuantumCumulants.jl and QuTiP's PIQS

mixed
Problem
Time evolution, steady states and spectra of open quantum systems with closed forms; the driven, damped Jaynes–Cummings model.
Version
PQCF commit aef6143 (not in a release)
Result
On four problems with closed-form solutions, PQCF agrees to 36 digits, QuTiP to 5 to 16 at its default and tightest tolerances (atol 1e-14, rtol 1e-12); PQCF at 128 bits. QuTiP is 12 to 280 times faster at time evolution of the driven Jaynes–Cummings model; PQCF is faster on the steady state of resonance fluorescence.
Reference
The closed forms; QuTiP at default and tight tolerances; exact published solutions (Drummond and Walls; Bartolo et al.).
Agreement
Closed forms to 36 digits; QuantumCumulants.jl to 1e-13; PIQS within its single precision.
Hardware
One core each, the same core.
Reproduction
comparison/quantum-optics/REPRODUCE.md; you should see: every digit count and difference as published. Details →
Limitations
Arbitrary precision costs time: QuTiP's double-precision sparse algebra is faster on larger master equations. Not in a release.

Holographic QCD

Seven computations against their papers, SymPy, mpmath and SciPy

not timed
Problem
Meson and glueball spectra, free energy, the Weyl anomaly and the speed of sound in seven holographic models, from the action.
Version
PQCF commit d77ead6; the hard wall corrected at 2c971f2 (not in a release)
Result
Six agree with their papers to the papers' rounding or exactly; improved holographic QCD within 0.6% after a correction of our own error.
Reference
The papers, exact formulas, and independent SciPy solutions.
Agreement
Soft and hard wall to 30 digits; Sakai–Sugimoto to 3e-11 with SciPy; Gubser–Nellore to the table's 6 decimals.
Hardware
One core; improved holographic QCD on 16.
Reproduction
comparison/holographic/README.md; you should see: each result file unchanged but for its commit line, date and times. Details →
Limitations
Improved holographic QCD runs at 12 digits, and its last 0.6% is open.

One-loop integrals and Dirac traces

Against mpmath, LoopTools, QCDLoop and FORM

not timed
Problem
One-loop integrals of two, three and four points at arbitrary precision; Dirac traces in d dimensions.
Version
PQCF commit 7570e62 (not in a release)
Result
Two-point functions to mpmath's 35 digits; C0 in 40 configurations; D0 to mpmath's 16 digits and LoopTools' 1e-15; 60 traces equal to FORM.
Reference
mpmath's integrals; LoopTools 2.16 and QCDLoop 2.1.0 (double precision); FORM.
Agreement
As above. In our ARM builds LoopTools and QCDLoop disagree with mpmath in several C0 configurations, and QCDLoop returns no number for two of four D0 configurations above threshold.
Hardware
One core; D0 above threshold on 20.
Reproduction
comparison/oneloop/REPRODUCE.md; you should see: the reference files rewritten byte for byte; the tests pass. Details →
Limitations
D0 is slow above threshold (45 s to 5 minutes at 18 digits on 20 cores). No x86 builds of the other programs tested.

Feynman diagrams and rules

Against FORM's diagram generator, zero-dimensional field theory and Peskin and Schroeder

not timed
Problem
Every Feynman diagram with its weight; Feynman rules from a Lagrangian.
Version
PQCF commit 7570e62 (not in a release)
Result
Equal to FORM's generator in all 24 cases compared; the three-loop gluon self-energy's 324 diagrams in 0.1 s.
Reference
FORM 5's diagrams_; zero-dimensional field theory; Peskin and Schroeder.
Agreement
Same counts and weights; exact; vertices term by term.
Hardware
One core.
Reproduction
comparison/diagrams/REPRODUCE.md; you should see: the 24 cases; the reference file byte for byte; 23 tests pass; seconds. Details →
Limitations
24 tested cases, not every diagram; FORM's generator not timed against ours; vacuum graphs with many identical vertices are slow.

The running of QCD

Against mpmath and CRunDec

not timed
Problem
β and γm to five loops; αs and quark masses run and decoupled.
Version
PQCF commit 7570e62 (not in a release)
Result
Exact coefficients; running to mpmath's 28 digits.
Reference
mpmath at 45 digits; CRunDec 3.1.2.
Agreement
28 digits with mpmath; within CRunDec's own accuracy (1e-8; 1e-14 for decoupling).
Hardware
One core.
Reproduction
comparison/qcd/REPRODUCE.md; you should see: mpmath and CRunDec agree; 5 tests pass; seconds. Details →
Limitations
Four-loop decoupling and mass decoupling not yet done.

The three-loop β-function of QCD

Against the published coefficients

not timed
Problem
A complete calculation: diagrams, Feynman rules, colour, integrals, renormalisation.
Version
PQCF commit eef0e6f
Result
Every coefficient exact and equal to the known result.
Reference
Tarasov, Vladimirov and Zharkov (1980); Larin and Vermaseren.
Agreement
Exact.
Hardware
8 cores.
Reproduction
comparison/beta/REPRODUCE.md; you should see: beta functions: as known; about 15 minutes. Details →
Limitations
At three loops the integrals were computed by FORM with Forcer; PQCF cannot yet do them.

GenHub2: virus classification

Against held-out NCBI Virus genomes and Nextclade

not timed
Problem
Classifying virus genomes and reads; mutations; trees.
Version
GenHub2 d6a1733 on PQCF d77ead6; phase B GenHub2 3a68ed5
Result
Species right for 95.1% of 593 held-out genomes (96.5% with proteins); mutations identical to Nextclade's for 885 of 1,000 genomes, every other difference at a genome end or next to an indel; the specification's targets met.
Reference
NCBI's taxonomy of the held-out genomes; Nextclade; each added method against the one it adds to.
Agreement
As above; on two nodes, results identical to one node's.
Hardware
One DGX Spark, 10 cores.
Reproduction
acceptance/REPRODUCE.md; you should see: all 20 checks hold; about 20 minutes on 10 cores. Details →
Limitations
18 of the 29 species misses are naming; maximum-likelihood trees beyond the 5-minute target with FastTree. Research software, not a validated diagnostic product.

Can PQCF help genome research?

Photonic routes against the best classical methods

slower
Problem
Assembly, haplotype phasing, gene-network modules, genome classification and encoding.
Version
PQCF branch research/genome, commit bb8a5dd
Result
Classical exact methods solve these tasks in under a second; the photonic routes need hardware that does not exist, or lose.
Reference
Proven optima; the best classical solvers.
Agreement
Every answer checked against a proven optimum.
Hardware
1 to 8 cores.
Reproduction
research/genome/REPRODUCE.md; you should see: 6 of 6 checks hold. Details →
Limitations
A negative result, published as one.

Measured against Qiskit Aer and Stim

QxSim and QxStab, our simulators on the BEAM

slower
Problem
State-vector simulation and stabilizer sampling.
Version
quantum_computing_simulations commit 3c31766
Result
QxSim is about 100 times slower than Qiskit Aer at 19 qubits; QxStab is 8 to 30 times slower than Stim on as many cores.
Reference
Qiskit Aer 0.18.2; Stim 1.16.0.
Agreement
The same circuits; QxSim's states against Qiskit's to a worst fidelity of 1 − 7 × 10⁻¹⁵.
Hardware
20 cores; the BEAM in a 1 GB container.
Reproduction
reproduce/benchmarks.md; you should see: the circuits as published; your own times. Details →
Limitations
The BEAM loses at raw numerical speed; that is why PQCF's kernels are in Rust.

The surface code under circuit noise

QxStab's samples against Stim's

not timed
Problem
Rotated surface codes of distance 3 to 7 under gate-level noise.
Version
quantum_computing_simulations commit 3c31766
Result
The curves cross between 0.6% and 0.8%, in line with published values.
Reference
Stim 1.16.0 sampling the same circuits; PyMatching decoding both.
Agreement
Exact binomial tests on every detector, detector pairs and the logical rates.
Hardware
20 cores.
Reproduction
reproduce/noise.md; you should see: every count as published; about 7 minutes. Details →
Limitations
A reproduction of known results; the decoder is PyMatching's.

Photonic schemes from the literature

Against the papers and Perceval

not timed
Problem
HOM, the KLM gates, fusion, detectors and loss, GKP and cat states, erasure decoding.
Version
quantum_computing_simulations commit 3c31766
Result
The schemes reproduced; slips in the review we started from found and listed.
Reference
The primary papers; Perceval to 2e-16; PyMatching for erasure decoding.
Agreement
As stated per scheme.
Hardware
One machine.
Reproduction
reproduce/photonic.md; you should see: the output files identical to the published ones. Details →
Limitations
Simulations of schemes, not photonic hardware.
// Benchmarks · every comparison on this page

What we measure ourselves against,
and where we lose.

Every result on this page is measured against an established tool, with the method, the data and the code public, and each answer checked, not only timed. The standing comparisons run every week.

comparisonagainsthow often
Large-scale symbolic computationFORM 5every week
PQCF against the photonic simulatorsThe Walrus, Piquasso, Perceval, DeepQuantum, Strawberry Fieldsevery week
Reducing Feynman integralsFIREevery week
Four-loop integrals from QCDFORM's Forcera result, 9 November
Holographic QCD, from the action to the spectrumthe published results of seven holographic computations; SymPy, mpmath, SciPyresults, 14–15 November
Quantum opticsQuTiP, QuantumCumulants.jl, QuTiP's PIQS, exact published solutionsresults, 14–16 November
One-loop integrals and Dirac tracesmpmath, LoopTools, QCDLoop, FORMresults, 16 November
Feynman diagrams and rulesFORM's diagram generator, zero-dimensional field theory, Peskin and Schroederresults, 18 November
The running of QCDCRunDec, mpmathresults, 17 November
The three-loop β-function of QCDthe published coefficientsa result, 12 November
GenHub2: virus classificationheld-out NCBI Virus genomes, Nextclade, each method against the one it adds towith every change
Can PQCF help genome research?the best classical methods, proven optimaa result, 12–13 November
Measured against Qiskit Aer and StimQiskit Aer, Stim29 October
The surface code under circuit noiseStima reproduction
Photonic schemes from the literaturethe published resultsa reproduction

How to reproduce every result: one list, with the step-by-step page of each result, the commit it was tested at, how long it takes and what it needs.