ci: benchmark variance reduction#1107
Merged
Merged
Conversation
Member
Author
|
Result: |
Contributor
|
Sounds good, especially the regression threshold (though I'm mostly looking at these as statistics, not paying attention to the +/- that much). Interleaving might be especially nice on CI, where we don't have dedicated hardware. |
henryiii
approved these changes
Mar 7, 2026
ngoldbaum
pushed a commit
to ngoldbaum/packaging
that referenced
this pull request
Apr 1, 2026
This was referenced Apr 13, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does 4 things to reduce benchmark variance:
1. PYTHONHASHSEED=0
Set
PYTHONHASHSEED=0viaenv_nobuildinasv.conf.jsonto eliminatefrozensetiteration order variance across process spawns.This happens because
self._specsin aSpecifierSetis a frozenset and iterating over it can have very different performance characteristics per run, we may be able to tackle this and improve performance in a separate PR, but either way we don't want hash seed differences causing benchmark differences anywhere.2. Interleave rounds
Added
--interleave-roundstoasv continuousin CI to avoid commit-ordering bias.3. More rounds and repeats
Set
rounds=4on all benchmark classes to sample across longer time periods, I found that the default 2 rounds is simply not enough for untuned machines.4. Regression threshold
Reset
regressions_thresholdsfrom0.01(1%) to the default0.05(5%). Noise on micro-benchmarks easily exceeds 1%, we should only consider performance wins or gains on micro-benchmarks of >5%, we could set tighter or looser results for specific benchmarks if we wanted.