Back in 2017 we ran into out of space issues in tmpfs with the metapackages package
(since the package pulls in tons of packages), see git rev 71c396b802
.
But as noted with
https://jenkins.mgm.sipwise.com/job/metapackages-piuparts/1237/console a
run might take 1h48min, which is way longer than expected and useful.
dpkg as well as SW-RAID (with /tmp being located on rootfs which is
powered by SW-RAID) are the ones limiting our I/O:
| mprokop@jenkins-slave13 ~ % df -h /
| Filesystem Size Used Avail Use% Mounted on
| /dev/md3 3.6T 623G 2.8T 18% /
| mprokop@jenkins-slave13 ~ % sudo iotop
| Total DISK READ: 0.00 B/s | Total DISK WRITE: 160.25 K/s
| Current DISK READ: 0.00 B/s | Current DISK WRITE: 1545.23 K/s
| TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
| 272 be/3 root 0.00 B/s 38.15 K/s 0.00 % 89.93 % [jbd2/md3-8]
| 1505823 be/4 root 0.00 B/s 64.86 K/s 0.00 % 0.44 % dpkg --purge acpi acpi-support-base acpid approx apt-utils aptitude aptitude-common [...]
| 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize 87
| 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
| 3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
| 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
| [...]
Let's try running those jobs in tmpfs to get more reasonable runtimes
again. If we still run into out of space situations we also might
consider further increasing tmpfs size on our jenkins-slaves.
FTR: we currently use 32GB of RAM with 50% for tmpfs of
/var/cache/pbuilder on all our production jenkins-slaves:
| mprokop@jenkins-slave13 ~ % grep /var/cache/pbuilder /etc/fstab
| tmpfs /var/cache/pbuilder tmpfs size=50% 0 0
Our typical jenkins-slave system looks like this:
| mprokop@jenkins-slave15 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.1G 596K 3.1G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 4.0G 12G 26% /var/cache/pbuilder
| tmpfs 3.1G 0 3.1G 0% /run/user/10012
And even during package builds we seem to have >8GB of space left in
/var/cache/pbuilder:
| mprokop@jenkins-slave10 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.2G 944K 3.2G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 7.0G 8.6G 46% /var/cache/pbuilder
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.119195/dev/shm
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.2177009/dev/shm
| tmpfs 3.2G 0 3.2G 0% /run/user/10012
NOTE: our "smallest" jenkins-slave is jenkins-test-slave1 which pretty
sure will run into problems (though then let's handle this different):
| mprokop@jenkins-test-slave1 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 1.6G 632K 1.6G 1% /run
| tmpfs 7.9G 0 7.9G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 7.9G 4.1G 3.8G 52% /var/cache/pbuilder
| tmpfs 1.6G 0 1.6G 0% /run/user/10012
Change-Id: Ibdeff1c9bde7b53cabdfe6cd2417d1c018da3d58
master
parent
00d15f6ffe
commit
0994f60e35
Loading…
Reference in new issue