This reverts commit 00d15f6ffe.
Now that deb.rainbow.sipwise.com is no longer relevant and will disappear,
let's remove the rainbow specific code.
Change-Id: I047f01de47dc794f09b88dd42835a6d7e9b66b5a
Back in 2017 we ran into out of space issues in tmpfs with the metapackages package
(since the package pulls in tons of packages), see git rev 71c396b802.
But as noted with
https://jenkins.mgm.sipwise.com/job/metapackages-piuparts/1237/console a
run might take 1h48min, which is way longer than expected and useful.
dpkg as well as SW-RAID (with /tmp being located on rootfs which is
powered by SW-RAID) are the ones limiting our I/O:
| mprokop@jenkins-slave13 ~ % df -h /
| Filesystem Size Used Avail Use% Mounted on
| /dev/md3 3.6T 623G 2.8T 18% /
| mprokop@jenkins-slave13 ~ % sudo iotop
| Total DISK READ: 0.00 B/s | Total DISK WRITE: 160.25 K/s
| Current DISK READ: 0.00 B/s | Current DISK WRITE: 1545.23 K/s
| TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
| 272 be/3 root 0.00 B/s 38.15 K/s 0.00 % 89.93 % [jbd2/md3-8]
| 1505823 be/4 root 0.00 B/s 64.86 K/s 0.00 % 0.44 % dpkg --purge acpi acpi-support-base acpid approx apt-utils aptitude aptitude-common [...]
| 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize 87
| 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
| 3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
| 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
| [...]
Let's try running those jobs in tmpfs to get more reasonable runtimes
again. If we still run into out of space situations we also might
consider further increasing tmpfs size on our jenkins-slaves.
FTR: we currently use 32GB of RAM with 50% for tmpfs of
/var/cache/pbuilder on all our production jenkins-slaves:
| mprokop@jenkins-slave13 ~ % grep /var/cache/pbuilder /etc/fstab
| tmpfs /var/cache/pbuilder tmpfs size=50% 0 0
Our typical jenkins-slave system looks like this:
| mprokop@jenkins-slave15 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.1G 596K 3.1G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 4.0G 12G 26% /var/cache/pbuilder
| tmpfs 3.1G 0 3.1G 0% /run/user/10012
And even during package builds we seem to have >8GB of space left in
/var/cache/pbuilder:
| mprokop@jenkins-slave10 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 3.2G 944K 3.2G 1% /run
| tmpfs 16G 0 16G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 16G 7.0G 8.6G 46% /var/cache/pbuilder
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.119195/dev/shm
| tmpfs 16G 0 16G 0% /var/cache/pbuilder/build/cow.2177009/dev/shm
| tmpfs 3.2G 0 3.2G 0% /run/user/10012
NOTE: our "smallest" jenkins-slave is jenkins-test-slave1 which pretty
sure will run into problems (though then let's handle this different):
| mprokop@jenkins-test-slave1 ~ % df -h -t tmpfs
| Filesystem Size Used Avail Use% Mounted on
| tmpfs 1.6G 632K 1.6G 1% /run
| tmpfs 7.9G 0 7.9G 0% /dev/shm
| tmpfs 5.0M 0 5.0M 0% /run/lock
| tmpfs 7.9G 4.1G 3.8G 52% /var/cache/pbuilder
| tmpfs 1.6G 0 1.6G 0% /run/user/10012
Change-Id: Ibdeff1c9bde7b53cabdfe6cd2417d1c018da3d58
For the rainbow specific packages we have specific needs regarding
repository access, but potentially also regarding package outside
of our control, like:
| Setting up cloud-wrg (1.166.0~202206270731.84f00df-972+deb11) ...
| /var/lib/dpkg/info/cloud-wrg.postinst: 5: systemctl: not found
| dpkg: error processing package cloud-wrg (--configure):
| installed cloud-wrg package post-installation script subprocess returned error exit status 127
| [...]
| Setting up mongodb-org-server (3.6.23) ...
| [...]
| Done.
| /var/lib/dpkg/info/mongodb-org-server.postinst: 43: systemctl: not found
Handle this by detecting the project "rainbow-misc" and then set up a custom
cowbuilder based environment for piuparts usage:
1) Retrieve latest ngcp-rainbow-keyrings-pro package from
release-trunk-$distribution for repository keys
2) Set up /etc/apt/sources.list.d/rainbow.list and
/etc/apt/preferences as needed
3) Deploy further workarounds like the one for systemctl mentioned above
Change-Id: I8253aaa645c3b69d784f29a3b050267c3a8a03dc
It went kind of unnoticed, that the piuparts version installed
on our Debian/stretch based system included our custom apt retry
patch, while our Debian/buster based systems were using the
plain piuparts version from buster.
Change-Id: I0c02b2cdb350e7279317a618be3aaa40566d60d6
Rerun apt-get in case of failure to avoid hashsum mismatch errors
up to three times.
Depends on a specific piuparts version, available internally
for now only.
Change-Id: Id74e09b0f3b916a98a620681ac0b716b430341d6
When the same source package generates multiple binary packages,
some of which conflict with each other, the piuparts check will
fail due to the way we invoke piuparts. So provide workaround to
solve the license-client dummy package situation (see TT#26751
and https://gerrit.mgm.sipwise.com/#/c/17447/).
Change-Id: I6cdb8ff7cb99c85e10059e8ff6198bc934668bb4
metapackages pulls in tons of packages and we run out of space
in tmpfs, so let's try executing it on disk instead.
Change-Id: Id55cd7f20c3535c044712f2d74df1fad01b1b79c
We need to set the password for user root, otherwise the
sudo package refuses to get removed. See e.g.
See https://jenkins.mgm.sipwise.com/view/piuparts/job/metapackages-piuparts/290/consoleFull
| Removing sudo (1.8.10p3-1+deb8u3) ...
| You have asked that the sudo package be removed,
| but no root password has been set.
| Without sudo, you may not be able to gain administrative privileges.
|
| If you would prefer to access the root account with su(1)
| or by logging in directly,
| you must set a root password with "sudo passwd".
|
| If you have arranged other means to access the root account,
| and you are sure this is what you want,
| you may bypass this check by setting an environment variable
| (export SUDO_FORCE_REMOVE=yes).
|
| Refusing to remove sudo.
| dpkg: error processing package sudo (--purge):
| subprocess installed pre-removal script returned error exit status 1
Change-Id: Id49bfa2cdf1f4018d26c13d240682967a8507caa
Nowadays it's just ngcp-upgrade-ce + ngcp-upgrade-pro, so the
workaround to just run the checks for the current latest +
latest LTS release isn't needed anymore.
FTR: if this should fail for older releases we have to investigate
how to support (or ignore) that, but for now let's make sure that
master builds work as needed.
Change-Id: I86a991e6dbcf60c614111e740dfef1af1f3abd4c
The ngcp-support-access + ngcp-support-noaccess packages conflict
with each other, therefore we can't run them under piuparts
at the same time.
Change-Id: Iebb3142c93aa42be8c81e554dc71fe356a3cfae2
Before piuparts is recording its metadata of the chroot it's
supposed to be brought up2date, so that installed packages match
what's provided by the repositories being refered to in
sources.list.
piuparts until and including v0.66 executes the apt-get
dist-upgrade only when either the basetgz or schroot option is
present. We've switched from the basetgz approach to
--existing-chroot= though, because that speeds up build times.
With the recent Debian point release the libsqlite3-0 package
(which is a dependency of aptitude and therefore pre-installed in
the chroot) received an update which caused our piuparts runs to
fail because of wrongly recorded metadata, as mentioned above.
Reported against upstream including a patch in http://bugs.debian.org/798266
Change-Id: I4427d339f163bd92756e6ea71f0c0c99b766be7f
Nowadays we have two major improvements in our setup:
* *-get-code Jenkins jobs provide files for sources.list and apt_preferences
setup, so we no longer need to duplicate logic for generating those files
inside our piuparts wrapper
* the cowbuilder environments ("/var/cache/pbuilder/base-$distri-amd64.cow")
provide all the required apt keys, so we don't have to deal with
ngcp-keyring install nor wget runs to retrieve the keys any longer
As a result get rid of the piuparts tar file generation (like
/var/cache/pbuilder/base-wheezy-amd64.tgz), instead copy the
existing cowbuilder environment to a temporary place and use that
as input for piuparts' --existing-chroot option.
Change-Id: I8ba458e6088361f3f5a793512fa61d87ee711306
Otherwise fails with:
| 0m8.8s ERROR: Command failed (status=1): ['chroot', '/var/cache/pbuilder/tmp.upxyAfmDXA_piuparts_tmp/tmpN0qCK0', 'tmp/scripts/pre_install_add_sipwise_repos.sh']
| ** Setting up http://deb.sipwise.com/autobuild/680FBA8A.asc for apt-get usage ***
| mktemp: failed to create file via template `/var/cache/pbuilder/tmp.upxyAfmDXA_piuparts_tmp/tmp.XXXXXXXXXX': No such file or directory
We've our own Debian/jessie mirror nowadays, so there's no
need to use an external mirror for that anymore.
Using "DEBOOTSTRAP_OPTIONS=${!debootstrap_options}" doesn't work
(anymore, for unknown reasons). Since we need wget for further
checks we want to include it in our default environment. Also
we need to pass our trusted.gpg keyring so that bootstrapping
the piuparts environment works in a trusted way.
I'm not really happy about this behaviour change, but there
are broken symlinks in upstream Debian packages and there's]
no mechanism available in piuparts to ignore such problems.
Until we've a decent way to handle that let's ignore
broken symlinks as we used to do in the past.
The tarball that is used for unpacking doesn't include Sipwise'
apt-key so (dist-)upgrading it fails and the wrong checksums
are used. E.g. our tarball ships an older version of perl-base
than you get with the 7.7 point release nowadays and therefore
we get:
| After purging files have been modified: /usr/share/doc/perl/changelog.Debian.gz owned by: perl-base
as a result. By ensuring that our chroot is properly updated
we can avoid this issue.
To use this feature it requires the patch I provided in
Debian's #767485, I'll provide an according backport on our
infrastructure.
piuparts uses our internal mirror by default:
| Err http://debian.sipwise.com jessie/main amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
| Err http://debian.sipwise.com jessie/contrib amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
| Err http://debian.sipwise.com jessie/non-free amd64 Packages
| 404 Not Found [IP: 77.244.249.93 80]
But we don't provide jessie on it (yet), so explicitely use
piuparts --mirror option to point it to the according place.
Michael Prokop