[CI] Tier verification by depth and compute profile #85
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
Redesign BookWave CI around three explicit optimization metrics:
The current PR gate is broad and valuable, but monolithic. A cheap formatting failure can consume almost the same compute as a meaningful deep verification failure. The target should be smarter CI, not simply lighter CI: preserve or improve confidence while avoiding work that cannot add useful evidence.
Measured current state
Recent Forgejo PR verification runs are very consistent:
A normal successful
verifyDebuginvocation is roughly 8m10s–8m20s; Gradle dominates the PR runtime.Run #247 is the strongest waste example. It ultimately had one KtLint
standard:function-signaturedefect, yetverifyDebug --continueproceeded through compilation, tests, Kover, Android Lint and APK assembly before reporting the formatting problem.At the same time, that run demonstrated the value of the deep gate:
:playback:ktlintTestSourceSetCheckfailed.The right optimization is therefore ordering and scoping, not removing the full regression suite.
Current PR gate observations
1.
verifyDebugincludes packagingThe build convention defines Android module
verifyDebugas:ktlintCheck;lintDebug;testDebugUnitTest;assembleDebug.PRODUCT_SPEC.md§16.5 currently documents debug assembly as part ofverifyDebug.This means PR CI builds installable/debug APK outputs even though PR verification only needs quality evidence. Packaging performs D8/dex/resource/native merging and
packageDebugwork and increases CPU/disk pressure.The APK workflow already exists for explicit APK production.
2. Cheap failures do not fail cheaply
The deep Gradle graph is invoked in one
verifyDebug --continuecall. This is useful once deep verification has started because it exposes multiple independent failures in one run, but it is wasteful as the first quality gate.KtLint should fail before expensive test/coverage/packaging work.
3. Full JVM/Robolectric regression coverage is worth retaining
The repository has more than 2,500 JVM/Robolectric tests across app, data, domain, persistence and playback boundaries. Cross-module behavior means a changed-module-only merge gate could miss regressions.
Changed-file classification should optimize early checks/cache decisions, not reduce the final Standard acceptance suite.
4. Coverage enforcement is narrower than the test suite
Current Kover thresholds enforce:
domain+core:model+core:common: 80% line coverage;core:common: 90% line coverage.The repository comments record about 89.7% and 92.7% respectively when wired.
However:
:playbackhas tests and applies Kover but has no coverage threshold;:data:downloadshas tests and applies Kover but has no coverage threshold;core:networkand otherdata:*modules are tested but not currently coverage-gated;5. Kover appears to generate unnecessary variant artifacts
Recent PR logs include work such as:
app:koverGenerateArtifactDebug;app:koverGenerateArtifactRelease;app:koverGenerateArtifactBenchmark.The benchmark variant explicitly has host/instrumented tests disabled, yet Kover still generates an artifact for it. Release/benchmark coverage artifact generation during a debug PR gate should be investigated and constrained if it does not contribute to the actual gate.
This may reduce compute/fan load without sacrificing any test or threshold.
6. PR cache is read-only from main
The PR workflow restores the trusted Gradle cache produced by
main, which is correct from a supply-chain perspective, but repeated attempts on one PR cannot publish reusable branch-local state.A safe optimization is a PR-private cache namespace:
This should significantly reduce repeated-run compute without weakening the trust boundary.
7.
--rerun-tasksclassification is too broadThe current classifier treats
.forgejo/workflows/*changes as classpath/build changes and therefore may force--rerun-tasks.Workflow YAML does not alter the Kotlin/Android classpath. CI-infrastructure changes deserve validation, but not necessarily a full Gradle cache bypass.
Split:
Only the first category should automatically require
--rerun-tasks.8. Dependency resolution is duplicated
After the deep Gradle gate, PR CI separately runs:
:app:dependencies --configuration debugRuntimeClasspath.The preceding build has already resolved the graph under strict Gradle dependency verification. A separate dependency report is most useful when:
For ordinary source PRs this is duplicated work.
9. Gitleaks can be incremental on PRs
Main should already be clean. PR verification chiefly needs to prove the branch introduced no new secret.
Recommended split:
main...HEAD/ changed commits;10. CI diagnostics currently miscount KtLint findings
scripts/ci/pr-diagnostics.pycounts every non-empty line in KtLint text reports. Run #247 displayed "3 finding(s)" while KtLint's rule summary showed one actualstandard:function-signaturesource violation.Use KtLint XML or parse only concrete
file:line:columnentries.This does not affect speed but directly affects confidence in CI diagnostics.
Proposed architecture
Separate verification depth from compute profile.
Verification depth
Quick
Purpose: rapid development feedback; not sufficient for merge acceptance.
Suggested work:
Expected result: formatting/policy failures should fail in seconds or low minutes rather than after an 8-minute deep graph.
Standard
Purpose: normal PR merge gate.
Suggested work:
No APK packaging.
Retain
--continueinside the deep stage to collect independent failures once Quick has passed.Intensive Assurance
Purpose: rare high-confidence run, manually dispatched and/or scheduled.
Candidate work:
--rerun-tasksdeep verification independent of incremental outputs;Compute profile
Compute profile must not change the evidence or pass/fail criteria.
Quiet
Balanced
Fast
Do not reduce heap solely for quiet mode without measurement; excessive GC may increase CPU work. Control concurrency first.
Coverage recommendations
Add branch coverage to existing Kover gates
Kover already has execution data, so branch-coverage thresholds should cost little relative to collecting coverage.
For policy-heavy BookWave code, branch coverage provides stronger evidence than simply raising line coverage.
First measure the current baseline, then establish a non-regression floor rather than choosing an arbitrary percentage.
Extend coverage gates to high-risk policy modules
Measure and then introduce focused non-regression thresholds for:
Avoid gating generated/framework glue merely to inflate a number.
Do not enforce global 100% line coverage
Global 100% Android line coverage would incentivize low-value tests over framework wiring, generated/Hilt-adjacent paths, Compose glue and defensive branches.
Instead, Intensive Assurance may require 100% branch coverage for selected pure critical policy classes where every branch has semantic meaning.
Add targeted mutation testing
Mutation testing asks whether changing/removing behavior actually makes a test fail. This is particularly valuable for BookWave because repository history includes defects where tests executed code without actually protecting production behavior.
Use mutation testing selectively on critical deterministic policy rather than across Android/Compose/framework glue.
Workflow organization
Do not create separate Quiet/Balanced/Fast YAML copies.
Recommended top-level workflows:
Within PR verification, expose two independent inputs:
This supports combinations such as Intensive + Quiet for maximum confidence at lower acoustic/thermal load without duplicating workflow logic.
Main versus branch model
Treat green
mainas the trusted baseline.PR diff classification should answer:
Use that for:
Do not use diff classification to shrink the final Standard JVM/Robolectric regression suite.
CI telemetry to add before tuning worker counts
Record per run:
Provide a manual profiling mode for Gradle task-duration reporting rather than running profiling on every PR.
Then compare at least:
and:
Optimize based on measured wall time and compute/thermal behavior rather than assumption.
Suggested implementation sequence
--rerun-tasksclassification.Acceptance criteria
IMPLEMENTATION READINESS
Frozen investigation state
88ae231ddcThis record supersedes the historical assumptions in the issue where current main has already changed.
1. Reconciliation of the proposals already in #85
2. Current CI execution graph
PR verification — .forgejo/workflows/pull-request.yml
Trigger:
Graph:
identity -> preflight -> verify -> pr-status
identity:
preflight:
verify:
verifyDebug --continue -x lintDebug -x testDebugUnitTest -x assembleDebug -x koverVerify
verifyDebug --continue -x assembleDebug, optionally --rerun-tasks.
pr-status:
Merge-gate reality:
Recent real runtime:
Main assurance — .forgejo/workflows/main.yml
Triggers:
One linux-amd64 pinned-image job:
Recent main-push runtime:
Important: the file comment assumes accepted PR commits reach main as the exact checked commit, but the frozen main is a merge commit distinct from PR #89’s checked head. Combined with disabled branch protection, keep main push assurance in v1.
APK — .forgejo/workflows/apk.yml
Trigger:
Purpose:
Recent success:
CI image — build-ci-image.yml
Contract capture — contract-capture.yml
APK branch dropdown refresh — sync-apk-pr-options.yml
3. Actual verifyDebug decomposition
QualityConventionPlugin registers verifyDebug in each project.
Per module:
Additional coupling:
Therefore:
4. Exact Quick specification
Purpose:
Trigger/default:
Execution:
gitleaks git . --redact --verbose --no-banner --log-opts="origin/main..HEAD"
Ancestry is checked first, so this range is exactly branch-introduced commits.
Forgejo itself remains the YAML parser; do not add a second YAML parser that cannot validate Forgejo semantics.
Run:
./gradlew quickKtlint --max-workers= -Pshelfplayer.warningsAsErrors=true
Quick deliberately does NOT prove:
Acceptance target:
5. Exact Standard specification
Purpose:
Default manual dispatch:
Status:
Execution:
./gradlew verifyDebug --continue -x assembleDebug --max-workers= -Pshelfplayer.warningsAsErrors=true
Rerun rule:
6. Intensive Assurance v1
Intensive v1 should be useful immediately, not a wishlist.
Triggers:
Trusted scheduled/manual main Intensive:
gitleaks git . --redact --verbose --no-banner
./gradlew verifyDebug --continue --rerun-tasks --no-build-cache --max-workers= -Pshelfplayer.warningsAsErrors=true
Do not add clean: --rerun-tasks + --no-build-cache already proves task execution and avoids extra delete/rebuild I/O ceremony.
PR Intensive:
Not in Intensive v1:
Current scheduled run #198 at 12m48s is the historical runtime reference. Initial v1 Balanced acceptance budget is <=20 minutes; if exceeded, diagnose before adding more checks.
7. Compute profile contract
Profiles alter compute pressure only. They MUST NOT alter selected tests, Kover filters, thresholds, security checks or diagnostics.
Implementation:
Fast promotion rule:
8. Kover / coverage architecture
Current facts:
Observed in real PR logs even after APK removal:
This is stale work for a debug PR gate.
Exact fix:
Required proof of the Kover fix:
Branch coverage rollout:
Candidate high-value playback policy classes inspected:
Candidate download policy classes inspected:
The download classes are more IO/Room-coupled than the small playback policies. Do not demand 100% class branch coverage from them merely to obtain a pleasing number.
Selected 100% branch gates:
9. Mutation strategy
Decision: later #85 ratchet, not Intensive v1 mandatory.
Why:
Compatibility/measurement slice:
Acceptance for adopting the tool:
If that spike fails, reject automated PIT for this Android layout and retain targeted manual mutation proof rather than weakening the signal.
10. Cache/trust boundary
Keep the current trusted namespace model in v1.
Current safe direction:
gradle-cache-key.sh is already correctly focused on Gradle/build inputs:
It does not need to include workflow YAML.
PR-private cache decision:
11. Exact change classification
Move classification out of inline YAML into scripts/ci/classify-changes.sh and unit-test it.
force_rerun remains an explicit override.
12. Dependency and secret scanning
Dependency:
Secrets:
13. Diagnostics contract
Already fixed:
Still change:
14. Telemetry design
New script: scripts/ci/ci-telemetry.py.
Output:
Cheap every-run fields:
Do not require host-only telemetry.
Profiling mode:
Profile campaign acceptance:
15. Workflow/API ergonomics and trigger matrix
New pull-request.yml inputs:
Automatic pull_request ignores manual defaults and resolves Quick + Balanced.
16. File-by-file implementation contract
Slice 1 — control plane, true Quick, profiles, classifier, diagnostics/telemetry foundation
Modify:
Do NOT change:
Validation:
Rollback:
Expected effect:
Slice 2 — Kover CI variant + coverage measurement
Modify:
Validation:
Rollback:
Expected effect:
Slice 3 — Intensive v1 and trusted security/supply-chain split
Modify:
Validation:
Rollback:
Slice 4 — measured ratchets, not guessed thresholds
Precondition:
Then:
PR-private cache is NOT part of any v1 slice.
17. Acceptance and measurement strategy
Functional acceptance:
Performance baselines to compare:
Measure after each slice:
Never compare worker profiles from different SHAs or different cache generations.
18. Known risks
19. Rollback strategy
Decision
The issue is ready for staged implementation. The first implementation agent should start with Slice 1, not with the historical numbered list in the issue body.
#85 IMPLEMENTATION READY
Implementation correction for the staged #85 contract:
verifyDebug --continue -x assembleDebug; packaging is not acceptance evidence.Latest successful pre-#85 PR verification run #274 completed
verifyDebugin 5m13s with 2,621 JVM/Robolectric tests, 0 failures, KtLint/Detekt/Android Lint clean, Room clean, dependency resolution clean, and the existing Kover gate. The shorter Gradle time came from normal cache/up-to-date reuse (506 executed / 157 from cache / 458 up-to-date), not from reducing the Standard regression graph.The remaining #85 coverage work is still staged separately: debug-only Kover variant scoping, branch coverage measurement, and measured playback/download non-regression floors.