MT#56879 Run piuparts jobs for metapackages + rainbow-misc also in tmpfs

Back in 2017 we ran into out of space issues in tmpfs with the metapackages package
(since the package pulls in tons of packages), see git rev 71c396b802.

But as noted with
https://jenkins.mgm.sipwise.com/job/metapackages-piuparts/1237/console a
run might take 1h48min, which is way longer than expected and useful.

dpkg as well as SW-RAID (with /tmp being located on rootfs which is
powered by SW-RAID) are the ones limiting our I/O:

| mprokop@jenkins-slave13 ~ % df -h /
| Filesystem      Size  Used Avail Use% Mounted on
| /dev/md3        3.6T  623G  2.8T  18% /
| mprokop@jenkins-slave13 ~ % sudo iotop
| Total DISK READ:         0.00 B/s | Total DISK WRITE:       160.25 K/s
| Current DISK READ:       0.00 B/s | Current DISK WRITE:    1545.23 K/s
|     TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
|     272 be/3 root        0.00 B/s   38.15 K/s  0.00 % 89.93 % [jbd2/md3-8]
| 1505823 be/4 root        0.00 B/s   64.86 K/s  0.00 %  0.44 % dpkg --purge acpi acpi-support-base acpid approx apt-utils aptitude aptitude-common [...]
|       1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd --system --deserialize 87
|       2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
|       3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
|       4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_par_gp]
| [...]

Let's try running those jobs in tmpfs to get more reasonable runtimes
again. If we still run into out of space situations we also might
consider further increasing tmpfs size on our jenkins-slaves.

FTR: we currently use 32GB of RAM with 50% for tmpfs of
/var/cache/pbuilder on all our production jenkins-slaves:

| mprokop@jenkins-slave13 ~ % grep /var/cache/pbuilder /etc/fstab
| tmpfs   /var/cache/pbuilder     tmpfs   size=50%        0       0

Our typical jenkins-slave system looks like this:

| mprokop@jenkins-slave15 ~ % df -h -t tmpfs
| Filesystem      Size  Used Avail Use% Mounted on
| tmpfs           3.1G  596K  3.1G   1% /run
| tmpfs            16G     0   16G   0% /dev/shm
| tmpfs           5.0M     0  5.0M   0% /run/lock
| tmpfs            16G  4.0G   12G  26% /var/cache/pbuilder
| tmpfs           3.1G     0  3.1G   0% /run/user/10012

And even during package builds we seem to have >8GB of space left in
/var/cache/pbuilder:

| mprokop@jenkins-slave10 ~ % df -h -t tmpfs
| Filesystem      Size  Used Avail Use% Mounted on
| tmpfs           3.2G  944K  3.2G   1% /run
| tmpfs            16G     0   16G   0% /dev/shm
| tmpfs           5.0M     0  5.0M   0% /run/lock
| tmpfs            16G  7.0G  8.6G  46% /var/cache/pbuilder
| tmpfs            16G     0   16G   0% /var/cache/pbuilder/build/cow.119195/dev/shm
| tmpfs            16G     0   16G   0% /var/cache/pbuilder/build/cow.2177009/dev/shm
| tmpfs           3.2G     0  3.2G   0% /run/user/10012

NOTE: our "smallest" jenkins-slave is jenkins-test-slave1 which pretty
sure will run into problems (though then let's handle this different):

| mprokop@jenkins-test-slave1 ~ % df -h -t tmpfs
| Filesystem      Size  Used Avail Use% Mounted on
| tmpfs           1.6G  632K  1.6G   1% /run
| tmpfs           7.9G     0  7.9G   0% /dev/shm
| tmpfs           5.0M     0  5.0M   0% /run/lock
| tmpfs           7.9G  4.1G  3.8G  52% /var/cache/pbuilder
| tmpfs           1.6G     0  1.6G   0% /run/user/10012

Change-Id: Ibdeff1c9bde7b53cabdfe6cd2417d1c018da3d58
master
Michael Prokop 2 years ago
parent 00d15f6ffe
commit 0994f60e35

@ -18,16 +18,6 @@ rm -f piuparts.tap
[ -n "${architecture:-}" ] || architecture="$(dpkg --print-architecture)"
[ -n "${mirror:-}" ] || mirror='http://debian.sipwise.com/debian'
if [ -n "${PROJECTNAME:-}" ] ; then
case "${PROJECTNAME}" in
metapackages|rainbow-misc)
echo "PROJECTNAME is ${PROJECTNAME}, using /tmp for the piuparts run instead of the tmpfs"
PIUPARTS_BASEDIR="$(sudo mktemp --dry-run -d --tmpdir=/tmp/ --suffix=_piuparts_basetmp)"
PIUPARTS_TMPDIR="$(sudo mktemp -d --tmpdir=/tmp/ --suffix=_piuparts_tmp)"
;;
esac
fi
echo "*** Executing $0 ***"
echo "*** Using $PIUPARTS_BASEDIR as \$PIUPARTS_BASEDIR ***"
echo "*** Using $PIUPARTS_TMPDIR as \$PIUPARTS_TMPDIR ***"

Loading…
Cancel
Save