What we measure ourselves against,
and where we lose.
Every result is measured against an established tool or a published result, each answer checked, not only timed. Below, every comparison in the same format; where PQCF is slower or incomplete is part of the format.
Fifteen comparisons,
in one format.
PQCF faster than the reference
within run-to-run variation of the reference
the reference, or the classical method, is faster
faster at some tasks, slower at others
not yet benchmarked for speed: the answers compared
none yet: every comparison here is ours
The hardware, unless a comparison says otherwise: one NVIDIA DGX Spark: CPU 20 Arm cores (10 Cortex-X925 and 10 Cortex-A725), 128 GB of memory shared with its GB10 GPU, Ubuntu 24.04 (Linux 6.17, aarch64), every program in Docker. The machine is shared with other work, so times vary from run to run by up to about a third. Both programs of a comparison run on the same machine and the same cores, one after the other. Open a comparison for its details.
Large-scale symbolic computation
PQCF's symbolic engine against FORM 5, every week
mixed
Large-scale symbolic computation
PQCF's symbolic engine against FORM 5, every week
- Problem
- Expanding large polynomials (dense and sparse products, powers, symbolic permanents, an interferometer's symbolic matrix); compiling them into programs; gcd and factorisation.
- Version
- PQCF commit
92e2aeb(after release 1.4.0) - Result
- Dense products 3 to 5 times faster; permanents from a third of FORM's time to the same; sparse products and powers 1.5 to 2.8 times slower; compiled programs need fewer operations than FORM's O4 but take longer to find; gcd level with FORM, factorisation 4 to 8 times slower.
- Reference
- FORM 5 (Vermaseren et al.), at the version we measured.
- Agreement
- Identical term for term on every task.
- Hardware
- One core each, the same core.
- Reproduction
- comparison/form/REPRODUCE.md; you should see: every task ends in
ok; about 80 seconds once FORM is built. Details → - Limitations
- FORM is still ahead on sparse products, powers, factorisation and the time to compile; its parallel version and disk-based expressions reach far beyond these sizes.
PQCF against the photonic simulators
The Walrus, Piquasso, Perceval, DeepQuantum, Strawberry Fields, every week
mixed
PQCF against the photonic simulators
The Walrus, Piquasso, Perceval, DeepQuantum, Strawberry Fields, every week
- Problem
- Permanents, torontonians, exact boson sampling, Fock-space simulation of a 10-mode network.
- Version
- PQCF commit
92e2aeb - Result
- The boson sampler is the fastest at every size (4 to 360 times Piquasso, 13 to 66 times Perceval). The permanent is the slowest of the four compiled programs (Perceval 1.7 times faster); the torontonian second (Piquasso 2.7 times faster); the Fock backend 3 times slower than Piquasso's.
- Reference
- The five programs at fixed versions.
- Agreement
- Every check agrees (50 of 50 in the run of 15 November): permanents within 1e-8, torontonians within 1e-7, samples by chi-square tests against the exact distribution.
- Hardware
- One core, one thread per program.
- Reproduction
- comparison/photonic/REPRODUCE.md; you should see:
50 of 50 checks agree(the count depends on the sizes reached); about 13 minutes. Details → - Limitations
- Three kernels behind the best; DeepQuantum's GPU paths not compared.
Reducing Feynman integrals
Against FIRE 6 on its own double box
faster
Reducing Feynman integrals
Against FIRE 6 on its own double box
- Problem
- Reducing the two-loop massless double box to master integrals.
- Version
- PQCF 1.4.0; rerun at commit
92e2aeb - Result
- 3 to 7 times faster: 1.21 s against 7.99 for FIRE's target, 3.73 s against 11.99 for a harder one.
- Reference
- FIRE 6.5.2 with the FLINT simplifier, built from source.
- Agreement
- Equal master by master at a random point modulo a large prime.
- Hardware
- One core each, the same core.
- Reproduction
- comparison/fire/REPRODUCE.md; you should see: every case
equal; about 90 seconds. Details → - Limitations
- The double box is small by research standards; Kira not compared (it needs Fermat, x86 only, with a restrictive licence).
Four-loop integrals from QCD
Against FORM's Forcer on FORM's own checks
slower
Four-loop integrals from QCD
Against FORM's Forcer on FORM's own checks
- Problem
- The four-loop integrals of FORM's Forcer checks, reduced to master integrals.
- Version
- PQCF commit
92e2aeb - Result
- Every answer exact. Forcer, built for these integrals, is 17 to 25 times faster on four loops; PQCF is 13 times faster on the three-loop example.
- Reference
- Forcer's results and the numbers FORM's checks assert.
- Agreement
- All 10 integrals equal to Forcer's as expansions in ε; all 5 MS-bar results equal to FORM's.
- Hardware
- FORM with Forcer on 1 core; PQCF on 16.
- Reproduction
- comparison/forcer/REPRODUCE.md; you should see: every check
equal; PQCF's results unchanged; about 7 minutes on 8 cores. Details → - Limitations
- PQCF builds and solves a system for every problem; Forcer solves its rules in advance.
Quantum optics
Against QuTiP 5.3.1, QuantumCumulants.jl and QuTiP's PIQS
mixed
Quantum optics
Against QuTiP 5.3.1, QuantumCumulants.jl and QuTiP's PIQS
- Problem
- Time evolution, steady states and spectra of open quantum systems with closed forms; the driven, damped Jaynes–Cummings model.
- Version
- PQCF commit
aef6143(not in a release) - Result
- On four problems with closed-form solutions, PQCF agrees to 36 digits, QuTiP to 5 to 16 at its default and tightest tolerances (atol 1e-14, rtol 1e-12); PQCF at 128 bits. QuTiP is 12 to 280 times faster at time evolution of the driven Jaynes–Cummings model; PQCF is faster on the steady state of resonance fluorescence.
- Reference
- The closed forms; QuTiP at default and tight tolerances; exact published solutions (Drummond and Walls; Bartolo et al.).
- Agreement
- Closed forms to 36 digits; QuantumCumulants.jl to 1e-13; PIQS within its single precision.
- Hardware
- One core each, the same core.
- Reproduction
- comparison/quantum-optics/REPRODUCE.md; you should see: every digit count and difference as published. Details →
- Limitations
- Arbitrary precision costs time: QuTiP's double-precision sparse algebra is faster on larger master equations. Not in a release.
Holographic QCD
Seven computations against their papers, SymPy, mpmath and SciPy
not timed
Holographic QCD
Seven computations against their papers, SymPy, mpmath and SciPy
- Problem
- Meson and glueball spectra, free energy, the Weyl anomaly and the speed of sound in seven holographic models, from the action.
- Version
- PQCF commit
d77ead6; the hard wall corrected at2c971f2(not in a release) - Result
- Six agree with their papers to the papers' rounding or exactly; improved holographic QCD within 0.6% after a correction of our own error.
- Reference
- The papers, exact formulas, and independent SciPy solutions.
- Agreement
- Soft and hard wall to 30 digits; Sakai–Sugimoto to 3e-11 with SciPy; Gubser–Nellore to the table's 6 decimals.
- Hardware
- One core; improved holographic QCD on 16.
- Reproduction
- comparison/holographic/README.md; you should see: each result file unchanged but for its commit line, date and times. Details →
- Limitations
- Improved holographic QCD runs at 12 digits, and its last 0.6% is open.
One-loop integrals and Dirac traces
Against mpmath, LoopTools, QCDLoop and FORM
not timed
One-loop integrals and Dirac traces
Against mpmath, LoopTools, QCDLoop and FORM
- Problem
- One-loop integrals of two, three and four points at arbitrary precision; Dirac traces in d dimensions.
- Version
- PQCF commit
7570e62(not in a release) - Result
- Two-point functions to mpmath's 35 digits; C0 in 40 configurations; D0 to mpmath's 16 digits and LoopTools' 1e-15; 60 traces equal to FORM.
- Reference
- mpmath's integrals; LoopTools 2.16 and QCDLoop 2.1.0 (double precision); FORM.
- Agreement
- As above. In our ARM builds LoopTools and QCDLoop disagree with mpmath in several C0 configurations, and QCDLoop returns no number for two of four D0 configurations above threshold.
- Hardware
- One core; D0 above threshold on 20.
- Reproduction
- comparison/oneloop/REPRODUCE.md; you should see: the reference files rewritten byte for byte; the tests pass. Details →
- Limitations
- D0 is slow above threshold (45 s to 5 minutes at 18 digits on 20 cores). No x86 builds of the other programs tested.
Feynman diagrams and rules
Against FORM's diagram generator, zero-dimensional field theory and Peskin and Schroeder
not timed
Feynman diagrams and rules
Against FORM's diagram generator, zero-dimensional field theory and Peskin and Schroeder
- Problem
- Every Feynman diagram with its weight; Feynman rules from a Lagrangian.
- Version
- PQCF commit
7570e62(not in a release) - Result
- Equal to FORM's generator in all 24 cases compared; the three-loop gluon self-energy's 324 diagrams in 0.1 s.
- Reference
- FORM 5's
diagrams_; zero-dimensional field theory; Peskin and Schroeder. - Agreement
- Same counts and weights; exact; vertices term by term.
- Hardware
- One core.
- Reproduction
- comparison/diagrams/REPRODUCE.md; you should see: the 24 cases; the reference file byte for byte; 23 tests pass; seconds. Details →
- Limitations
- 24 tested cases, not every diagram; FORM's generator not timed against ours; vacuum graphs with many identical vertices are slow.
The running of QCD
Against mpmath and CRunDec
not timed
The running of QCD
Against mpmath and CRunDec
- Problem
- β and γm to five loops; αs and quark masses run and decoupled.
- Version
- PQCF commit
7570e62(not in a release) - Result
- Exact coefficients; running to mpmath's 28 digits.
- Reference
- mpmath at 45 digits; CRunDec 3.1.2.
- Agreement
- 28 digits with mpmath; within CRunDec's own accuracy (1e-8; 1e-14 for decoupling).
- Hardware
- One core.
- Reproduction
- comparison/qcd/REPRODUCE.md; you should see:
mpmath and CRunDec agree; 5 tests pass; seconds. Details → - Limitations
- Four-loop decoupling and mass decoupling not yet done.
The three-loop β-function of QCD
Against the published coefficients
not timed
The three-loop β-function of QCD
Against the published coefficients
- Problem
- A complete calculation: diagrams, Feynman rules, colour, integrals, renormalisation.
- Version
- PQCF commit
eef0e6f - Result
- Every coefficient exact and equal to the known result.
- Reference
- Tarasov, Vladimirov and Zharkov (1980); Larin and Vermaseren.
- Agreement
- Exact.
- Hardware
- 8 cores.
- Reproduction
- comparison/beta/REPRODUCE.md; you should see:
beta functions: as known; about 15 minutes. Details → - Limitations
- At three loops the integrals were computed by FORM with Forcer; PQCF cannot yet do them.
GenHub2: virus classification
Against held-out NCBI Virus genomes and Nextclade
not timed
GenHub2: virus classification
Against held-out NCBI Virus genomes and Nextclade
- Problem
- Classifying virus genomes and reads; mutations; trees.
- Version
- GenHub2
d6a1733on PQCFd77ead6; phase B GenHub23a68ed5 - Result
- Species right for 95.1% of 593 held-out genomes (96.5% with proteins); mutations identical to Nextclade's for 885 of 1,000 genomes, every other difference at a genome end or next to an indel; the specification's targets met.
- Reference
- NCBI's taxonomy of the held-out genomes; Nextclade; each added method against the one it adds to.
- Agreement
- As above; on two nodes, results identical to one node's.
- Hardware
- One DGX Spark, 10 cores.
- Reproduction
- acceptance/REPRODUCE.md; you should see: all 20 checks hold; about 20 minutes on 10 cores. Details →
- Limitations
- 18 of the 29 species misses are naming; maximum-likelihood trees beyond the 5-minute target with FastTree. Research software, not a validated diagnostic product.
Can PQCF help genome research?
Photonic routes against the best classical methods
slower
Can PQCF help genome research?
Photonic routes against the best classical methods
- Problem
- Assembly, haplotype phasing, gene-network modules, genome classification and encoding.
- Version
- PQCF branch research/genome, commit
bb8a5dd - Result
- Classical exact methods solve these tasks in under a second; the photonic routes need hardware that does not exist, or lose.
- Reference
- Proven optima; the best classical solvers.
- Agreement
- Every answer checked against a proven optimum.
- Hardware
- 1 to 8 cores.
- Reproduction
- research/genome/REPRODUCE.md; you should see:
6 of 6 checks hold. Details → - Limitations
- A negative result, published as one.
Measured against Qiskit Aer and Stim
QxSim and QxStab, our simulators on the BEAM
slower
Measured against Qiskit Aer and Stim
QxSim and QxStab, our simulators on the BEAM
- Problem
- State-vector simulation and stabilizer sampling.
- Version
- quantum_computing_simulations commit
3c31766 - Result
- QxSim is about 100 times slower than Qiskit Aer at 19 qubits; QxStab is 8 to 30 times slower than Stim on as many cores.
- Reference
- Qiskit Aer 0.18.2; Stim 1.16.0.
- Agreement
- The same circuits; QxSim's states against Qiskit's to a worst fidelity of 1 − 7 × 10⁻¹⁵.
- Hardware
- 20 cores; the BEAM in a 1 GB container.
- Reproduction
- reproduce/benchmarks.md; you should see: the circuits as published; your own times. Details →
- Limitations
- The BEAM loses at raw numerical speed; that is why PQCF's kernels are in Rust.
The surface code under circuit noise
QxStab's samples against Stim's
not timed
The surface code under circuit noise
QxStab's samples against Stim's
- Problem
- Rotated surface codes of distance 3 to 7 under gate-level noise.
- Version
- quantum_computing_simulations commit
3c31766 - Result
- The curves cross between 0.6% and 0.8%, in line with published values.
- Reference
- Stim 1.16.0 sampling the same circuits; PyMatching decoding both.
- Agreement
- Exact binomial tests on every detector, detector pairs and the logical rates.
- Hardware
- 20 cores.
- Reproduction
- reproduce/noise.md; you should see: every count as published; about 7 minutes. Details →
- Limitations
- A reproduction of known results; the decoder is PyMatching's.
Photonic schemes from the literature
Against the papers and Perceval
not timed
Photonic schemes from the literature
Against the papers and Perceval
- Problem
- HOM, the KLM gates, fusion, detectors and loss, GKP and cat states, erasure decoding.
- Version
- quantum_computing_simulations commit
3c31766 - Result
- The schemes reproduced; slips in the review we started from found and listed.
- Reference
- The primary papers; Perceval to 2e-16; PyMatching for erasure decoding.
- Agreement
- As stated per scheme.
- Hardware
- One machine.
- Reproduction
- reproduce/photonic.md; you should see: the output files identical to the published ones. Details →
- Limitations
- Simulations of schemes, not photonic hardware.
What we measure ourselves against,
and where we lose.
Every result on this page is measured against an established tool, with the method, the data and the code public, and each answer checked, not only timed. The standing comparisons run every week.
| comparison | against | how often |
|---|---|---|
| Large-scale symbolic computation | FORM 5 | every week |
| PQCF against the photonic simulators | The Walrus, Piquasso, Perceval, DeepQuantum, Strawberry Fields | every week |
| Reducing Feynman integrals | FIRE | every week |
| Four-loop integrals from QCD | FORM's Forcer | a result, 9 November |
| Holographic QCD, from the action to the spectrum | the published results of seven holographic computations; SymPy, mpmath, SciPy | results, 14–15 November |
| Quantum optics | QuTiP, QuantumCumulants.jl, QuTiP's PIQS, exact published solutions | results, 14–16 November |
| One-loop integrals and Dirac traces | mpmath, LoopTools, QCDLoop, FORM | results, 16 November |
| Feynman diagrams and rules | FORM's diagram generator, zero-dimensional field theory, Peskin and Schroeder | results, 18 November |
| The running of QCD | CRunDec, mpmath | results, 17 November |
| The three-loop β-function of QCD | the published coefficients | a result, 12 November |
| GenHub2: virus classification | held-out NCBI Virus genomes, Nextclade, each method against the one it adds to | with every change |
| Can PQCF help genome research? | the best classical methods, proven optima | a result, 12–13 November |
| Measured against Qiskit Aer and Stim | Qiskit Aer, Stim | 29 October |
| The surface code under circuit noise | Stim | a reproduction |
| Photonic schemes from the literature | the published results | a reproduction |
How to reproduce every result: one list, with the step-by-step page of each result, the commit it was tested at, how long it takes and what it needs.