Repository navigation
Build Speed
A build spent much of its time on work other than compiling. Scheduling steps, analyzing dependencies, checking files before the first step starts, and rerunning steps that had nothing to rebuild all added up. The changes below cut that overhead or ran it in parallel.
Dependency tracking outside the source tree. siso did not record dependencies for outputs stored outside it, so every build rescheduled all 166,000 steps. After the fix, a repeat build finishes with 0 steps.
Commits: build/siso a1fb7a7, build/soong 1e8564b, build/soong 3fced42, external/xos fd1c512, external/xos f428e88, build/siso 607980c, build/siso e53b2fc, build/siso 0275dfb, build/soong e268a42, build/blueprint b753706, build/blueprint 6aec609.
Scheduling latency. siso stat'ed every source file in series before the first step. Running stats in parallel dropped scheduling of 5.4 million nodes from 32 minutes to under two. A separate check of all stored output files also ran before any step could start. Workers now size by throughput and each file defers its check to first use, cutting time to the first step from 68 seconds to 32.
Commits: build/siso aa17e0f, build/siso f3573a1, build/siso e1f5bf6, build/siso 908251d.
Deps log loading. The deps log (645 MB) loaded in 35 seconds because each append copied the whole record list. Factor growth, direct decoding and parallel reads brought it under nine seconds.
Commits: build/siso 709ab3a, build/siso 6ef85ea.
Memory-aware step execution. siso ran one step per CPU. Builds were killed when large steps overlapped in memory. Steps now adjust from free memory and CPU, using per-step cgroups and persisted memory peaks.
Commits: build/siso f3573a1, build/siso 2f88ed2, build/siso 9ef8d29, build/siso bc1f989, build/siso 0d6987d, build/siso 2777eb8, build/siso ca3201c, build/siso bc83110, build/siso 3382840, build/siso aaba837, build/soong ea38e5e, build/soong f0d9121, external/xos 3cbf1d3.
Input prefetching. A prefetch reader now loads inputs for upcoming steps into the page cache, so they are in memory when the step runs.
Commits: build/siso f3573a1.
Soong glob overhead. Soong globs sent 75,000 stats for three tree-wide patterns, one directory at a time. Globs now read each directory once, cache results, and walk in parallel. A single glob dropped from 12 to 1.6 seconds. The check that follows reuses the finder's stats instead of repeating 156,000 of them, dropping from 7.8 to 1.9 seconds.
Commits: build/blueprint 3037722, build/blueprint e5b9a15, build/blueprint 14713a3, build/soong b87b913, build/soong fbdc285.
soong_build memory. soong_build reached 54 GB without a memory limit. soong_ui passed half of available memory to
soong_build as --memory_limit. That value was part of the analysis command, so its hash changed on every run and siso
reran the full Soong analysis even without source changes. soong_build now reads available memory at startup and sets
half of it as its soft limit. The analysis command is unchanged across runs, so a build without source
changes skips the analysis. Measured on userdebug full build without source changes: Soong analysis from
8 min 54 s to 1.9 s, regular build phase from 11 min 36 s to 2 min 42 s.
Commits: build/soong 6557b4a, build/soong f0d9121, build/blueprint d2a8ef5, build/soong 91bdf0d, build/soong 28a06b3.
Release-config flag reading. release-config read 1,250 flag files two at a time. 32 readers brought it from 21 seconds to 720 ms. A wildcard BoardConfig.mk lookup dropped lunch dumpvars from 49 to 14 seconds.
Commits: build/soong c959b27, build/make 5bb24fe.
File server serialization. File lookups in one parent directory were serialized by the file server. Strided dispatch cut the finder probe from 10 to 8 seconds and siso source stats from 3.4 to under one second.
Commits: build/soong afa57f0, build/siso 25c0b3d, build/siso 4b868e5.
Source lookups by open directory. Before the first build step, siso looked up every source file by full path. On a
FUSE or network file system the kernel looks up every directory component of every path again on each call. The FUSE
client runs lookups in one directory in series, regardless of how many threads siso starts. siso now groups the source
files by directory and opens each directory once relative to the one above it. It calls fstatat for each name on the
open directory. The file system then resolves one lookup per name and one per directory, in place of one per directory
component of every path. In a build without source changes on a FUSE source share, the scheduling phase took 28.5 to
36.7 seconds over three builds before the change and 25.3 seconds in one build after it. The walk of the first target,
which contains the source check, took 14.8 to 19.5 seconds before and 11.5 seconds after.
Commits: build/siso 9037bd8.
Kati and Ninja generation. Kati used one CPU and parsed 676 MB of C/C++ export lists that nothing compiled. Parallel writes and numbered export lists shrank the Ninja file from 1,010 MB to 337 MB.
Commits: build/kati 9419e88, build/kati 824367c, build/soong 6ca7aa0, build/make 5527163, build/soong 8780007, build/kati 2c86ec2, external/xos c36e5b6.
Kati stamp checking. The Kati stamp check expanded 11,806 globs in series. Checking on every CPU dropped it from 24 to 6.5 seconds.
Commits: build/kati ff722c6, build/kati 6127cb9, build/make b90873a.
Kati symbol table. Kati's symbol table spent 13% of CPU interning and 7.6% destroying diagnostic messages. An open-addressing table cut about 12%.
Commits: build/kati 1265c82, build/kati c726778.
Kati allocator overhead. A CPU profile of the main Kati run showed about 12 % of its time in the glibc allocator. ckati now links mimalloc. Measured on two guacamole builds with Kati evaluation forced: the main Kati run dropped from 213.3 s to 198.5 s wall time (7 % less) and evaluation from 152.8 s to 138.2 s. Peak memory rose from 19.48 GB to 19.85 GB (1.9 %). Kati runs before the build steps start, so the increase does not add to the memory peak of the build.
Commits: external/xos 5d97bf8.
Kati evaluation overhead. CPU profiles of the main Kati run showed about 25 % of its time in allocation and string growth, up to 5.5 % in a std::set lookup of variables under evaluation, 3 to 4 % in the parser, and 2 to 2.5 % building evaluation frames even when tracing was off. A bitmap replaces the set lookup, function arguments evaluate into reused strings, a table finds expression terminators, and evaluation frames are built only when tracing is on. Measured on a full build with Kati evaluating against two earlier builds: wall time dropped from 207 and 213 s to 198 s, with 23 % fewer allocator samples and 14 % fewer libstdc++ samples. On a synthetic makefile with 300,000 modules user CPU dropped 16 to 19 %.
Commits: build/kati d08b3c1, build/kati 09eb9f0, build/kati 3614127, build/kati 6d267e5.
Kati find-output ordering. Kati regenerated the Ninja file when rewriting a file changed the order in which find listed names, even though the set of names was unchanged. The stamp check now compares find output as a set of words. Adding or removing a file still triggers regeneration.
Module Ninja file partitioning. Soong sorted all modules and wrote build actions into ten Ninja files. A rerun rewrote the ten files, 9.4 GiB in total. Each module now goes into the Ninja file for the first two directory levels of the Android.bp that defines it. Soong writes a file only when the content has changed, so a rerun on unchanged inputs writes 0 files.
Commits: build/blueprint 097cc9d, build/blueprint 6dfddba.
Sandboxed step reruns. A change to sbox reran 110,000 steps because each overwrote its outputs with a new mtime.
Sandboxed rules now pass --write-if-changed and are marked restat.
Commits: build/soong af86ccb.
Jarjar class renaming. Soong renamed classes inside jars with jarjar, a Java tool built on ASM that started a JVM for each step. In a guacamole build, 990 of 1,007 jarjar steps had rule files with only exact class renames, such as aconfig flag classes moved to a hidden package. A Rust tool, rename_classes, now handles those 990 steps; the 17 steps with wildcard rules still run jarjar. Measured on 59 Settings jar shards run sequentially with files in memory: jarjar took 67.7 CPU-seconds and 56.9 s wall time, rename_classes took 0.07 CPU-seconds and 1.2 s wall time. The framework-minus-apex jar also stopped renaming its header jar a second time.
Commits: build/soong 287c5a5, build/soong d4acfa8, build/soong 9cf4211.
Copy and link overhead. Copy steps ran a shell at 3.4 seconds each for a few kilobytes. siso now copies in-process. SDK links depended on the full NDK copy set; they now reference stubs where Soong built them.
Commits: build/soong 598e78d, build/soong be77da2, build/make 2738756, build/siso 33dd0cf.
Linker debug info. Linking soong_build included 11 MB of DWARF in a 62 MB binary. Go programs now link with -w.
Commits: build/blueprint 01670cd.
Kernel rebuild checks. The kernel step took over an hour with nothing to rebuild because git rehashed 78,000 files
on the network mount. core.checkStat=minimal brought git status from two minutes to three seconds. Parallel reads
cut the input check from nine to under four minutes. The git state step listed all 78,700 kernel-tree files as inputs
and ran find over the tree after each run, about 38 seconds each time. It now depends only on the four git files a
commit, checkout or tag writes, and kbuild's own depfile catches edited sources.
Commits: build/custom f9b80b1, build/custom 838a9f9, build/custom cb42a29, build/custom 94c29b5.
Signing and packaging. A no-change signing run still checked all 675,000 stored files. It now checks only the delta directory and derives the date from file mtimes. The target-files intermediate used full deflate. It now uses level 0, and the final package stays compressed. Even when every step was up to date, siso recorded the size and mtime of every main-build file before and after signing to detect a step that changed one, about four minutes each way. It now records them only when a signing step runs a command, dropping an unchanged signing run from about ten minutes to two and a half.
Commits: build/siso b13e27c, build/siso 916a0f1, build/soong f99b51c, build/soong f0d9121, build/make a17e64d, external/xos 26a624e, build/siso a429673.
API stubs after a method body edit. Metalava builds the API stubs and signature files of every droidstubs module
from its sources. An edit inside a method body of a framework source reran every metalava run that reads the source,
although stubs do not contain method bodies and the outputs did not change. Metalava now reads a copy of the sources
written by strip-method-bodies, a Kotlin tool built on javaparser, which empties the method and constructor bodies.
The copy contains the bodies read by metalava: those of a method with a synchronized block, reported by API lint, and
of a method returning a typedef, where metalava checks the returned constants against the typedef. The tool rewrites a
copy only when its content changes. So an edit inside a body not read by metalava runs the copy rule once per stub
module and does not rerun metalava. An API edit still reruns metalava.
Compliance database import. compliance_metadata_import writes compliance-metadata.db, read by the NOTICE files
and the SBOM. It wrote each table unsorted into the database file, copied it sorted into a second table and let
auto_vacuum move the pages after dropping the first. For 277 MB of input that wrote 3.4 GB for a 330 MB database and
took 446 seconds in a full build. The importer now sorts the lines in memory, keeping only each line and its sort
values, and writes every row once: 330 MB written, with byte-identical rows.
Gensrcs shard inputs. Each shard of a gensrcs module had every source file of the module as an input, and sbox
copied all of them into the sandbox. In core_oj_api_files, each shard copied 1,417 files and used 50. The shard steps
ran 32 to 490 s each, 11,181 s of step time in total. A gensrcs module can now set reads_other_srcs: false, so sbox
copies only the files assigned to the shard and those expanded from $(location) or $(locations) in the command. The
default is true because commands such as protoc read imports among the sources. With the flag set on
core_oj_api_files, each of the 29 shards copied 50 files, the steps ran 70 to 145 s each, and total step time dropped
to 3,997 s.
Installer payload verification. web-installer dist verified the installer page against the signed package by
compressing every 2 MiB chunk of about 2.9 GB of partition images with xz and bzip2, using the shorter result, in
one loop on one thread. The step now compresses chunks on as many worker threads as the machine has CPUs, and writes
results in chunk order so the output bytes are unchanged. Measured on 32 CPUs with the same inputs: wall time dropped
from 16 min 45 s to 4 min 1 s. Peak memory rose from 41 MB to 1.2 GB because each worker runs encoders on one chunk.