Our cowbuilder based chroots that are also used for piuparts, provide
/etc/localtime by default (seems to get created by mmdebstrap):
% ls -la /var/cache/pbuilder.persistent/base-trixie-amd64.cow/etc/localtime
lrwxrwxrwx 1 root root 23 Feb 2 18:24 /var/cache/pbuilder.persistent/base-trixie-amd64.cow/etc/localtime -> /usr/share/zoneinfo/UTC
Now as of tzdata v2017a-1, tzdata removes file /etc/localtime on
purge, see https://bugs.debian.org/854141
Until piuparts also ignores this file change (known as
https://bugs.debian.org/1080295), let's make sure we don't report this
as a problem, by ignoring it within our piuparts wrapper.
Fixes:
| ERROR: FAIL: After purging files have disappeared:
| /etc/localtime -> /usr/share/zoneinfo/UTC not owned
FTR:
% piuparts --help 2>&1 | grep -A3 -- ' -i '
-i FILENAME, --ignore=FILENAME
Add FILENAME to list of filenames to be ignored when
comparing changes to chroot.FILENAMES prefixed with
':' will be reported verbosely.
Change-Id: Ie1c1e0246b7540a7806aa94fc9338fd3a1e56457
hylafaxplus is gone since mr4.3 (see commit
96a31de4afadf4c376dcc72337e5d0b6a91a82cd in repos-scripts.git), so
there's no need to further ship this code.
Stumbled upon this while reviewing the /etc/localtime issue.
Change-Id: I9f39a9ba2026fa96d9fb4b8075cc3ba1f8f970fa
This reverts commit 00d15f6ffe.
Now that deb.rainbow.sipwise.com is no longer relevant and will disappear,
let's remove the rainbow specific code.
Change-Id: I047f01de47dc794f09b88dd42835a6d7e9b66b5a
Back in 2017 we ran into out of space issues in tmpfs with the metapackages package
(since the package pulls in tons of packages), see git rev 71c396b802.
But as noted with
https://jenkins.mgm.sipwise.com/job/metapackages-piuparts/1237/console a
run might take 1h48min, which is way longer than expected and useful.
dpkg as well as SW-RAID (with /tmp being located on rootfs which is
powered by SW-RAID) are the ones limiting our I/O:
| mprokop@jenkins-slave13 ~ % df -h /
| Filesystem Size Used Avail Use% Mounted on
| /dev/md3 3.6T 623G 2.8T 18% /
| mprokop@jenkins-slave13 ~ % sudo iotop
| Total DISK READ: 0.00 B/s | Total DISK WRITE: 160.25 K/s
| Current DISK READ: 0.00 B/s | Current DISK WRITE: 1545.23 K/s
| TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
| 272 be/3 root 0.00 B/s 38.15 K/s 0.00 % 89.93 % [jbd2/md3-8]
| 1505823 be/4 root 0.00 B/s 64.86 K/s 0.00 % 0.44 % dpkg --purge acpi acpi-support-base acpid approx apt-utils aptitude aptitude-common [...]
| 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize 87
| 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
| 3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
| 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
| [...]
Let's try running those jobs in tmpfs to get more reasonable runtimes
again. If we still run into out of space situations we also might
consider further increasing tmpfs size on our jenkins-slaves.
FTR: we currently use 32GB of RAM with 50% for tmpfs of
/var/cache/pbuilder on all our production jenkins-slaves:
| mprokop@jenkins-slave13 ~ % grep /var/cache/pbuilder /etc/fstab
| tmpfs /var/cache/pbuilder tmpfs size=50% 0 0
Our typical jenkins-slave system looks like this:
| mprokop@jenkins-slave15 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.1G 596K 3.1G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 4.0G 12G 26% /var/cache/pbuilder
| tmpfs 3.1G 0 3.1G 0% /run/user/10012
And even during package builds we seem to have >8GB of space left in
/var/cache/pbuilder:
| mprokop@jenkins-slave10 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.2G 944K 3.2G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 7.0G 8.6G 46% /var/cache/pbuilder
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.119195/dev/shm
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.2177009/dev/shm
| tmpfs 3.2G 0 3.2G 0% /run/user/10012
NOTE: our "smallest" jenkins-slave is jenkins-test-slave1 which pretty
sure will run into problems (though then let's handle this different):
| mprokop@jenkins-test-slave1 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 1.6G 632K 1.6G 1% /run
| tmpfs 7.9G 0 7.9G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 7.9G 4.1G 3.8G 52% /var/cache/pbuilder
| tmpfs 1.6G 0 1.6G 0% /run/user/10012
Change-Id: Ibdeff1c9bde7b53cabdfe6cd2417d1c018da3d58
For the rainbow specific packages we have specific needs regarding
repository access, but potentially also regarding package outside
of our control, like:
| Setting up cloud-wrg (1.166.0~202206270731.84f00df-972+deb11) ...
| /var/lib/dpkg/info/cloud-wrg.postinst: 5: systemctl: not found
| dpkg: error processing package cloud-wrg (--configure):
| installed cloud-wrg package post-installation script subprocess returned error exit status 127
| [...]
| Setting up mongodb-org-server (3.6.23) ...
| [...]
| Done.
| /var/lib/dpkg/info/mongodb-org-server.postinst: 43: systemctl: not found
Handle this by detecting the project "rainbow-misc" and then set up a custom
cowbuilder based environment for piuparts usage:
1) Retrieve latest ngcp-rainbow-keyrings-pro package from
release-trunk-$distribution for repository keys
2) Set up /etc/apt/sources.list.d/rainbow.list and
/etc/apt/preferences as needed
3) Deploy further workarounds like the one for systemctl mentioned above
Change-Id: I8253aaa645c3b69d784f29a3b050267c3a8a03dc
It went kind of unnoticed, that the piuparts version installed
on our Debian/stretch based system included our custom apt retry
patch, while our Debian/buster based systems were using the
plain piuparts version from buster.
Change-Id: I0c02b2cdb350e7279317a618be3aaa40566d60d6
Rerun apt-get in case of failure to avoid hashsum mismatch errors
up to three times.
Depends on a specific piuparts version, available internally
for now only.
Change-Id: Id74e09b0f3b916a98a620681ac0b716b430341d6
When the same source package generates multiple binary packages,
some of which conflict with each other, the piuparts check will
fail due to the way we invoke piuparts. So provide workaround to
solve the license-client dummy package situation (see TT#26751
and https://gerrit.mgm.sipwise.com/#/c/17447/).
Change-Id: I6cdb8ff7cb99c85e10059e8ff6198bc934668bb4
metapackages pulls in tons of packages and we run out of space
in tmpfs, so let's try executing it on disk instead.
Change-Id: Id55cd7f20c3535c044712f2d74df1fad01b1b79c
Nowadays it's just ngcp-upgrade-ce + ngcp-upgrade-pro, so the
workaround to just run the checks for the current latest +
latest LTS release isn't needed anymore.
FTR: if this should fail for older releases we have to investigate
how to support (or ignore) that, but for now let's make sure that
master builds work as needed.
Change-Id: I86a991e6dbcf60c614111e740dfef1af1f3abd4c
The ngcp-support-access + ngcp-support-noaccess packages conflict
with each other, therefore we can't run them under piuparts
at the same time.
Change-Id: Iebb3142c93aa42be8c81e554dc71fe356a3cfae2
Nowadays we have two major improvements in our setup:
* *-get-code Jenkins jobs provide files for sources.list and apt_preferences
setup, so we no longer need to duplicate logic for generating those files
inside our piuparts wrapper
* the cowbuilder environments ("/var/cache/pbuilder/base-$distri-amd64.cow")
provide all the required apt keys, so we don't have to deal with
ngcp-keyring install nor wget runs to retrieve the keys any longer
As a result get rid of the piuparts tar file generation (like
/var/cache/pbuilder/base-wheezy-amd64.tgz), instead copy the
existing cowbuilder environment to a temporary place and use that
as input for piuparts' --existing-chroot option.
Change-Id: I8ba458e6088361f3f5a793512fa61d87ee711306
We've our own Debian/jessie mirror nowadays, so there's no
need to use an external mirror for that anymore.
Using "DEBOOTSTRAP_OPTIONS=${!debootstrap_options}" doesn't work
(anymore, for unknown reasons). Since we need wget for further
checks we want to include it in our default environment. Also
we need to pass our trusted.gpg keyring so that bootstrapping
the piuparts environment works in a trusted way.
I'm not really happy about this behaviour change, but there
are broken symlinks in upstream Debian packages and there's]
no mechanism available in piuparts to ignore such problems.
Until we've a decent way to handle that let's ignore
broken symlinks as we used to do in the past.
The tarball that is used for unpacking doesn't include Sipwise'
apt-key so (dist-)upgrading it fails and the wrong checksums
are used. E.g. our tarball ships an older version of perl-base
than you get with the 7.7 point release nowadays and therefore
we get:
| After purging files have been modified: /usr/share/doc/perl/changelog.Debian.gz owned by: perl-base
as a result. By ensuring that our chroot is properly updated
we can avoid this issue.
To use this feature it requires the patch I provided in
Debian's #767485, I'll provide an according backport on our
infrastructure.
piuparts uses our internal mirror by default:
| Err http://debian.sipwise.com jessie/main amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
| Err http://debian.sipwise.com jessie/contrib amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
| Err http://debian.sipwise.com jessie/non-free amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
But we don't provide jessie on it (yet), so explicitely use
piuparts --mirror option to point it to the according place.
As noted in https://jenkins.mgm.sipwise.com/view/Failing/job/metapackages-piuparts/25/console
| 12:29:46 10m36.7s ERROR: FAIL: debsums reports modifications inside the chroot:
| 12:29:46 /etc/init.d/ngcp-hylafaxplus
| 12:29:46 /etc/default/ngcp-iaxmodem
| 12:29:46
So let's try to ignore those files which are generated
via ngcpcfg build ... by intention and therefore the
checksum of the files gets modified.
We need release, SCRIPTSDIR, MIRROR, distribution and DEBOOTSTRAP_OPTIONS
in all the piuparts runs, so avoid diverging settings.
This should partly address the recent piupart failurs.
Michael Prokop