1. Add new screenshot with memtest v7 which is in trixie.
2. Use new screenshot for trixie builds.
3. Check the RC. According to the help page of screenshot-compare RC
higher then 100 means 'high difference'.
4. Bump REFRESHED_AT ro regenerate docker image.
Change-Id: I99fe27c146aefba3778d03b7d19c1c0bce52a635
(cherry picked from commit b8e842398a)
It's necessary to build docker image with 'MT#62936 Wait for memtest
starts' commit.
Change-Id: If9f50e8a241542d29a91d2e62d5fd02e2aa568c8
(cherry picked from commit 3eff75344d)
Wait 5 secs before taking screenshot so the memtest is started.
Change-Id: If2d4c316498630827e1726f04e33abbd7665a8bf
(cherry picked from commit 2713ded181)
Noted in daily-build-carrier-runner installation (see
https://jenkins.mgm.sipwise.com/job/daily-build-install-vm/148320/console):
| Installing for i386-pc platform.
| grub-install: error: unable to identify a filesystem in mduuid/ad9e241cd5c1093cd46de7932bc3fce1; safety check can't be performed.
| Error: failed to execute 'grub-install --no-floppy --target=i386-pc /dev/md0'.
One can run 'grub-install /dev/md0' without problems on EFI systems, as
this doesn't really touch the disks then. But in non-EFI systems, this
fails hard, because GRUB doesn't support direct installation to the
SW-RAID.
Instead we need to iterate over all the disks separately, being already
handled within stage swraidinstallgrub:
| +11:37:12 (deployment.sh:2457): set_deploy_status swraidinstallgrub
| +11:37:12 (deployment.sh:102): set_deploy_status(): '[' -n swraidinstallgrub ']'
| +11:37:12 (deployment.sh:103): set_deploy_status(): echo swraidinstallgrub
| +11:37:12 (deployment.sh:2458): for disk in "${SWRAID_DISK1}" "${SWRAID_DISK2}"
| +11:37:12 (deployment.sh:2459): grml-chroot /mnt grub-install /dev/sda
| Writing /etc/debian_chroot ...
| Installing for i386-pc platform.
| Installation finished. No error reported.
| +11:37:12 (deployment.sh:2458): for disk in "${SWRAID_DISK1}" "${SWRAID_DISK2}"
| +11:37:12 (deployment.sh:2459): grml-chroot /mnt grub-install /dev/sdb
| Writing /etc/debian_chroot ...
| Installing for i386-pc platform.
| Installation finished. No error reported.
| +11:37:13 (deployment.sh:2462): grml-chroot /mnt update-grub
| Writing /etc/debian_chroot ...
| Generating grub configuration file ...
| Found linux image: /boot/vmlinuz-6.12.33+deb13-amd64
| Found initrd image: /boot/initrd.img-6.12.33+deb13-amd64
| Adding boot menu entry for UEFI Firmware Settings ...
| done
So install GRUB only to the *first* SW-RAID disk.
Change-Id: I436e3deca37d68ef505d44fd24620028f2ded0f4
(cherry picked from commit 75b0f9274e)
With kernel versions >=6.3, efivarfs is loaded even when
the system wasn't booted in EFI mode. But loading efivarfs
fails:
| root@web01a ~ # lsmod | grep efi
| efivarfs 28672 0
|
| root@web01a ~ # ls /sys/firmware/efi/
| ls: cannot access '/sys/firmware/efi/': No such file or directory
|
| root@web01a ~ # mount -t efivarfs efivarfs /sys/firmware/efi/efivars
| mount: /sys/firmware/efi/efivars: fsopen() failed: Operation not supported.
| dmesg(1) may have more information after failed mount system call.
Adjust our EFI support detection code to what we're also doing in
grml-debootstrap.git (see commit
b8a6a2c90f5bffceeb7d85cdc30ac7a7e58f4c0b).
Furthermore by now we have grml-debootstrap v0.121 available in our
current deployment ISO, so drop the now deprecated efivars_workaround.
Change-Id: I2210a7b87d0df2ae41872380e458e29e755f6886
(cherry picked from commit fd55984db1)
mdadm upstream commit
e97c4e18c8
changed its behavior, now (pretty much) always promting the user for the
expected bitmap configuration. :(
Now with mdadm v4.4-11 as present in current Debian/trixie,
automatically creating a mdadm array fails for us:
| (netscript.grml:704): set_up_partition_table_swraid(): echo y
| (netscript.grml:704): set_up_partition_table_swraid(): mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Mon Jun 16 13:39:42 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Mon Jun 16 13:39:42 2025
| mdadm: size set to 468218880K
Behavior with mdadm versions older than 4.4 (e.g. with mdadm 4.2-5 as
present with our previous deployment system):
| (netscript.grml:703): set_up_partition_table_swraid(): echo y
| (netscript.grml:703): set_up_partition_table_swraid(): mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Mon Jun 16 13:39:42 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Mon Jun 16 13:39:42 2025
| mdadm: size set to 468218880K
| mdadm: automatically enabling write-intent bitmap on large array
| Continue creating array? mdadm: Defaulting to version 1.2 metadata
| mdadm: array /dev/md0 started.
Now with this mdadm behavior change, we seem to have the following
options to handle this:
1) Explicitly enable the `--bitmap=internal` command line option:
| root@grml ~ # echo y | mdadm --create --verbose --bitmap=internal /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:26:41 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:26:41 2025
| mdadm: size set to 468218880K
| Continue creating array [y/N]? mdadm: Defaulting to version 1.2 metadata
| mdadm: array /dev/md0 started.
This option enables an explicit choice, causing the bitmap to be
stored with the metadata on the array (and so is replicated on all
devices). While this might be the preferred option for now, mdadm
internals might change over time, so I would prefer not to(tm) rely
on this option.
2) Piping `yes` into mdadm
| root@grml ~ # yes | mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:25:35 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:25:35 2025
| mdadm: size set to 468218880K
| Continue creating array [y/N]? mdadm: Defaulting to version 1.2 metadata
| mdadm: array /dev/md0 started.
This would be my preferred choice, but it still causes an error within
mdadm, returning with exit 141 while not reporting an actual error:
| root@grml ~ # set -o pipefail
| root@grml ~ # yes | mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 10:02:32 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 10:02:32 2025
| mdadm: size set to 468218880K
| Continue creating array [y/N]? mdadm: Defaulting to version 1.2 metadata
| mdadm: array /dev/md0 started.
| 141 root@grml ~ #
We could disable the pipefail option around the mdadm execution in our
script, though this might have unindented side-effects for us with
further failures. So I would prefer not to(tm) rely on this option,
neither.
3) Enable mdadm's `--run` command line option:
| root@grml ~ # echo y | mdadm --create --run --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| mdadm: /dev/sda3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:23:48 2025
| mdadm: Note: this array has metadata at the start and
| may not be suitable as a boot device. If you plan to
| store '/boot' on this device please ensure that
| your boot-loader understands md/v1.x metadata, or use
| --metadata=0.90
| mdadm: /dev/sdb3 appears to be part of a raid array:
| level=raid1 devices=2 ctime=Thu Jun 19 09:23:48 2025
| mdadm: size set to 468218880K
| mdadm: creation continuing despite oddities due to --run
| mdadm: Defaulting to version 1.2 metadata
| mdadm: array /dev/md0 started.
The `--run` option is documented as follows in mdadm(8):
| Insist that mdadm run the array, even if some of the
| components appear to be active in another array or filesystem. Normally
| mdadm will ask for confirmation before including such components in an
| array. This option causes that question to be suppressed.
and under "CREATE MODE" it also says:
| --run insist on running the array even if some devices look like they
| might be in use.
It seems to be the only way to disable user prompting, so it seems to
be our preferred choice.
BTW: not even the `--force` option prevents this question:
| # mdadm --create --force --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
| To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? ^C
OH: "--run is the new --force"
Change-Id: I26bcef8f8e5da057ba3029e0bdbbb7cf527f0374
In jenkins-configs commit 765cbf270df321c8c08d567a9803064696c78028
AKA "TT#62000 Add puppet key to Grml-sipwise image" we
included the puppetlabs upstream key into our deployment ISO.
Updating the file is error prone and requires quite some effort
(including changes to jenkins-configs!), whenever the file needs to be
adjusted. In commit 8647b3d7b5 we went
for usage of http://apt.puppetlabs.com/DEB-GPG-KEY-puppetlabs, which is
the *expired* key though. :-/
| % gpg DEB-GPG-KEY-puppetlabs
| gpg: WARNING: no command supplied. Trying to guess what you mean ...
| pub rsa4096/1054B7A24BD6EC30 2010-07-10 [SC] [expired: 2017-01-05]
| 47B320EB4C7C375AA9DAE1A01054B7A24BD6EC30
| uid Puppet Labs Release Key (Puppet Labs Release Key) <info@puppetlabs.com>
|
| % md5sum DEB-GPG-KEY-puppetlabs
| 7b4ed31e1028f921b5c965df0a42e508 DEB-GPG-KEY-puppetlabs
So let's check for the expired key, and if present then retrieve the
*current* and proper repository key instead. But instead of using
upstream infra http://apt.puppetlabs.com/pubkey.gpg (which we don't have
under control on our own and might be unavailable during our
deployments), we placed a copy of the key on our own infrastructure:
| mprokop@jenkins1 ~www/files % wget --quiet -O puppetlabs-pubkey-2025.gpg http://apt.puppetlabs.com/pubkey.gpg
| mprokop@jenkins1 ~www/files % ls -lah puppetlabs-pubkey-2025.gpg
| -rw-r--r-- 1 mprokop sipwise 3.2K Apr 9 19:56 puppetlabs-pubkey-2025.gpg
| mprokop@jenkins1 ~www/files % gpg puppetlabs-pubkey-2025.gpg
| gpg: WARNING: no command supplied. Trying to guess what you mean ...
| pub rsa4096 2019-04-08 [SC]
| D6811ED3ADEEB8441AF5AA8F4528B6CD9E61EF26
| uid Puppet, Inc. Release Key (Puppet, Inc. Release Key) <release@puppet.com>
| sub rsa4096 2019-04-08 [E]
| mprokop@jenkins1 ~www/files % md5sum puppetlabs-pubkey-2025.gpg
| d6368c2df370ff2093831daad16d9eeb puppetlabs-pubkey-2025.gpg
| mprokop@jenkins1 ~www/files %
Given that the file puppet.gpg is also in the wrong format for direct
usage with apt-key with its gpg file extension, let's copy it as
puppet.asc then, as expected by apt.
FTR: this used to work in the past only, as in jenkins-config's
jobs/internal/grml/build_grml_image.sh we converted the key for usage as
puppet.gpg within apt, via:
| puppet_key='puppet.gpg'
| gpg --export --no-default-keyring --keyring /etc/apt/trusted.gpg \
| --output "${puppet_key}" release@puppet.com
Change-Id: I7fe7dd20d89ed638112638930f578b6bb3783a5c
grml-live had >410 commits since v0.47.7 (which we used so far),
including getting rid of FAI (through our own so called minifai
implementation), supporting mmdebstrap for bootstrapping (and using it
by default nowadays), and usage inside non-privileged containers.
Misc changes:
* No longer run any docker containers with --privileged, now that this
is handled all internally from within grml-live's build driver
* Upgrade everything to Debian/trixie (given that we also switched our
trunk/master branch towards trixie also)
* Provide wrapper.sh script to avoid maintaining it also inside
jenkins-configs' jobs/internal/grml/build_grml_image.sh (which meant
going through yet another layer and every single change to it required
a full jenkins-config change). Update README.txt accordingly to
provide usage examples.
* Set up keyring for custom bootstrapping using our own mirror, using
grml_build/config/bootstrap-keyring/SIPWISE and
deploying /usr/share/keyrings/sipwise-archive-keyring.gpg via
updatebase hook script
* Ship .gitignore files to ignore directories that are relevant for
the build process, though we also don't need to explicitly create them
on every single build any longer.
* Update configuration layout for grml-live change as of upstream
version v0.51.0 (see grml-live.git commit f1dc42a), moving from
/etc/grml/fai/config into /usr/share/grml-live/config.
* Update configuration layout for grml-live change as of upstream
version 0.53.0 (see grml-live.git commit 3a4bd41), changing schema
files/<path>/<CLASS> into files/<CLASS>/<path>,
class/<CLASS>.var into env/<CLASS> and
hooks/<hookname>.<CLASS> into hooks/<CLASS>/<hookname>.
* Add memtest86+ + ipxe to SIPWISE class, so they are available for boot
menu integration (grml-live uses addons from grml_chroot and no longer
from host system).
grml_build/Dockerfile related changes:
* Adjust ENV usage (fixes `LegacyKeyValueFormat: "ENV key=value" should
be used instead of legacy "ENV key value" format (line 9)`)
* Add snippet for usage with debian:trixie-slim as base docker image
(useful for debugging e.g. without access to Sipwise systems)
* Install ca-certificates (needed for access to https://github.com/ with
debian:trixie-slim)
* Switch from grml-live v0.47.7 to v0.53.2
* Drop deprecated dependencies that are no longer relevant with recent
grml-live versions (fai-client + fai-server get replaced by our own
minifai implementation, mksh is no longer relevant for grml-live, and
isolinux, ipxe, memtest86+ + syslinux are used from the grml_chroot
instead of from the host system, also see above)
* Instead install mmdebstrap (which is much faster for bootstrapping
than debootstrap), now being supported and the default, thanks to our
minifai implementation
* Drop /root/.bash_history, as we switched towards our custom wrapper.sh
script
t/Dockerfile related changes:
* Mark /code as safe directory for git, to not fail with
"fatal: detected dubious ownership in repository at '/code'"
* Adjust ENV usage (fixes `LegacyKeyValueFormat: "ENV key=value" should
be used instead of legacy "ENV key value" format (line 8)`)
Addendum: it's yet unclear why we need the `ulimit -n 1048576`
workaround to fix an apt/apt-mark performance issue. It feels similar to
what has been observed for fakeroot at https://bugs.debian.org/920913.
Credits to Chris Hofstaedtler for review and working on grml-live's
minifai, to support all our needs.
Change-Id: I85111806a550f1aeffcb5263af2455ec8b90cc32
grml2usb had 47 commits since v0.19.2 (which we used so far), time to
bump it to its most recent stable version (being v0.20.6), which is also
part of current Debian/trixie.
While at it, drop deprecated grml64-small_2018.04.11-efi workaround.
Change-Id: Iac59b6ba9cf1f3782e10dd612153cd8e451c2ea3
Nowadays puppetlabs upstream provides the puppet-agent package
for Debian/bookworm, so we no longer need to work around this
by installing the puppet-agent package from bullseye on bookworm
systems.
Change-Id: Iad081fa02f4fb619330177c38673c86db60f2034
We no longer install any system with bullseye by default (we're using
bookworm internally and starting with trixie support f.e. for NGCP).
The buster workaround LVM support on Debian/buster and the dpkg
workaround for stretch/buster are no longer relevant.
Change-Id: Id96c0c1ce2b49d6332a4eb092a9f559dc221e607
Quoting from grml-live upstream:
| commit 3edddaef70f137f4f2874cef669c05948a41438c
| Author: Michael Prokop <mika@grml.org>
| Date: Mon Apr 11 18:47:19 2022 +0200
|
| SW: move from ntp/ntpdate to ntpsec/ntpsec-ntpdate in GRML_SMALL + GRML_FULL
|
| Starting with bookworm (current Debian/testing), ntpsec replaces ntp,
| and ntpsec-ntpdate replaces ntpdate. Now both the ntp + ntpdate packages
| are transitional packages. Given that ntpsec/ntpsec-ntpdate are
| available even for oldstable, let's follow this transition now (also see
| https://sources.debian.org/src/ntpsec/1.2.1%2Bdfsg1-6/debian/NEWS/).
| [...]
Now that ntpdate no longer exists in Debian/trixie, we need to switch
from ntpdate to ntpsec-ntpdate in our deployment ISO (the later package
also provides the /usr/sbin/ntpdate binary, so nothing further needs to
be adjusted).
Given that ntpsec-ntpdate exists even since oldoldstable AKA buster, we
can further adjust it in our deployment script for our puppet
deployments (to not break when switching to trixie there).
Change-Id: Ie332023aa2e6b4afedddf5a5077cec1c41ecb75b
Quoting from grml-live upstream:
| commit 61b77ff04fb60e09123a79b5f0ededbb6455add5
| Author: Michael Prokop <mika@grml.org>
| Date: Thu Dec 13 12:08:01 2018 +0100
|
| SW: drop deprecated cpufrequtils (+ drop scripts/GRMLBASE/36-cpufrequtils)
|
| The cpufreq drivers are autoloaded and the powersave/ondemand driver
| is mature enough. The linux-cpupower tools provide the binaries
| as replacement for what cpufrequtils provided so far and we ship
| them (with GRML_FULL) already.
|
| Also see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877016
|
| Thanks: Michael Biebl for the pointer
| Closes: https://github.com/grml/grml/issues/51
Now that cpufrequtils no longer exists as of Debian/trixie and newer,
drop it accordingly in our deployment ISO.
Change-Id: I4373ef907824a844cdf8adf240fa3cc2f878a1cd
We need to use the same VirtualBox Guest Additions ISO in both sp1 and
sp2, and instead of duplicating part of the installation logic, from
where to get the original files, just install the ISO on the target,
so that we can then copy it to sp2 and install it from there.
Change-Id: If1b6d0c4609be8504ff2d1bfcb0d9d7e36897539
If /sbin/fsck.ext4 (being a symlink pointing to /sbin/e2fsck) isn't
available, then update-initramfs complains with:
| update-initramfs: Generating /boot/initrd.img-6.12.27-amd64
| W: /sbin/fsck.ext4 doesn't exist, can't install to initramfs
Ensure to have e2fsprogs always available, given that its priority was
decreased from required to important with e2fsprogs v1.47.2-1, so it's
no longer necessarily part of a fresh Debian system as of trixie.
See https://bugs-devel.debian.org/cgi-bin/bugreport.cgi?bug=897277
Change-Id: Ia3fbfed9d499cce09d46cdfb6c6a290d16fb3f1b
We had to revert commit 2974f71127,
as we're relying on the /sys/block autodetection code in our VM
builds that don't specify the target disk explicitly.
So instead let's reintroduce the code, but also support
NVMe disks and make the code more generic WRT sda vs vda.
While at it, also report a message whenever we're running through this
autodetection code.
Change-Id: I39fe04def1694f127a99c97f91fabc2768fec1c7
This reverts commit 2974f71127.
This breaks our automated VM builds which don't set the target disk.
Change-Id: Iad3c21ae69e983de651fedbc6d50633d2dde0a3b
This was actually hiding a problem and is a f'up related to the previous
commit 001c9c6073.
When we're deploying a system with SW-RAID to sd* based disks (with boot
options like swraiddisk1=sda swraiddisk2=sdb), we're not setting
TARGET_DISK, but instead old autodetection code kicks in. This only
supports block devices like vd* + sd*, but then sets DISK to the first
such present disk.
As a result, we're ending up with a grml-debootstrap command line like
this:
| (netscript.grml:2219): grml-debootstrap --arch amd64 --grub /dev/sda --filesystem ext4 [...] -r bookworm -t /dev/mapper/ngcp-root --efi /dev/sda2 [...]
This propagates through grml-debootstrap into GRUB's debconf settings
for the /dev/sda disk, showing up like this:
| root@db01a:~# debconf-show grub-cloud-amd64
| * grub-cloud-amd64/install_devices: /dev/disk/by-id/ata-MTFDDAK480TDT-1AW1ZA_02JG544D7A44709LEN_3416E3FC
This is unexpected behavior, as GRUB would get installed on one disk only.
Also, when deploying to disks other than vda and sd* (e.g. with
NVMe ones), this shouldn't just use unrelated disks, but instead either
work with the expected ones or abort hard (as has been observed in the
previous commit with the NVMe disks).
Furthermore there's no point in further shipping this autodetection code
any longer at all, as we're using the TARGET_DISK approach only via
disk_selection.sh. Otherwise we're expecting the deployment system
(Grml-Deployment ISO, PXE boot environment,...) to fully control and
pass expected disk options.
It's 2025, and NVMe disks are a thing, so good riddance!
Thanks: Volodymyr Fedorov for reporting and assistance
Change-Id: I3fd7aa75a38dc0f07bc07d5b22139a89498f7b96
Since grml-debootstrap v0.116 there's a reworked GRUB setup, to support
installing both BIOS and UEFI bootloaders, and manage its updates via
grub-cloud-amd64 (see
2891f1730f
for details).
Now that we're shipping grml-debootstrap v0.119 in our latest
Grml-Sipwise deployment ISO, we're making use of its new behavior.
As it turned out, when deploying a SW-RAID based NGCP system, we invoke
grml-debootstrap with incomplete `--grub /dev/`. This is of course
incomplete and unexpected, but only started to fail with the more recent
grml-debootstrap version.
The underlying issue is grml-debootstrap setting GRUB's debconf
configuration to:
grub-cloud-amd64/install_devices: /dev/
And if we're installing grub-cloud-amd64 (or its not an EFI system),
then grml-debootstrap also executes:
grub-install --no-floppy --target=i386-pc "${device}"
Which of course won't work with the device being set to `/dev/`:
| Setting up grub-cloud-amd64 (0.0.5) ...
| Skipping installation without enable flag for cloud.
| Processing triggers for libc-bin (2.36-9+deb12u10) ...
| Installing grub on /dev/:
| Installing for x86_64-efi platform.
| Installation finished. No error reported.
| Installing for i386-pc platform.
| grub-install: error: cannot read `/dev': Is a directory.
| Error: failed to execute 'grub-install --no-floppy --target=i386-pc /dev/'.
To avoid any such failure, make sure grml-debootstrap's --grub option
uses a valid block device, also when installing SW-RAID (and even if it
actually shouldn't matter for EFI environments).
See https://github.com/grml/grml-debootstrap/issues/339 for further
details.
Change-Id: I6b839f708806c5107a1c86adbee1407cbeadddcc
We get failing ngcp-status on our trunk systems, due to:
| root@spce:~# systemctl --failed
| UNIT LOAD ACTIVE SUB DESCRIPTION
| ● vboxadd-service.service loaded failed failed VirtualBox Guest Additions Services Daemon
| ● vboxadd.service loaded failed failed vboxadd.service
While we're not even running inside a VirtualBox environment:
| root@spce:~# systemd-detect-virt
| kvm
We install /etc/systemd/system/vboxadd-service.service.d/override.conf
and /etc/systemd/system/vboxadd-service.service.d/override.conf with
`ConditionVirtualization=oracle` through our templates (see its git rev
a2521b609300965e5f8dfdeee63c5961f2d6a559) to address this. But this is
too late to the party, as the vboxadd* services are started *before* our
overrides get relevant.
So ensure we place the overrides already *before* even installing the
VBoxGuestAdditions.iso and executing its VBoxLinuxAdditions.run.
This then prevents the execution inside our Proxmox VMs, and not
reporting failing vboxadd-service + vboxadd services then:
| systemd[1]: vboxadd.service was skipped because of an unmet condition check (ConditionVirtualization=oracle).
Change-Id: I1a3a04bdc5d92d3fc79b336018ac5f588266c59b
virtualbox-guest-additions-iso v7.0.20-1 as present in current Debian/trixie
doesn't yet support kernel v6.12.22-1 (being the current kernel version
in Debian/trixie), while upstream supports kernel 6.12 as of VirtualBox 7.1.4.
Reported towards Debian as https://bugs.debian.org/1104024
FTR:
| mprokop@jenkins1 ~ % cd /var/www/files
| mprokop@jenkins1 ~www/files % wget https://download.virtualbox.org/virtualbox/7.1.8/VBoxGuestAdditions_7.1.8.iso
| [...]
| mprokop@jenkins1 ~www/files % curl -s https://download.virtualbox.org/virtualbox/7.1.8/SHA256SUMS | sha256sum -c --ignore-missing
| VBoxGuestAdditions_7.1.8.iso: OK
Change-Id: I32aa7806e375c4b85084a99d5a6903f632807694
Our deployment ISO might be outdated and when installing any additional
packages, we might get stuck in dpkg:
| +10:10:34 (netscript.grml:311): ensure_packages_installed(): DEBIAN_FRONTEND=noninteractive
| +10:10:34 (netscript.grml:311): ensure_packages_installed(): apt-get -o dir::cache=/tmp/ngcp-deployment-ensure-tmp.BKSocMV4KB/cachedir -o dir::state=/tmp/ngcp-deployment-ensure-tmp.BKSocMV4KB/statedir -o dir::etc=/tmp/ngcp-deployment-ensure-tmp.BKSocMV4KB/etc -o dir::e
| tc::trustedparts=/etc/apt/trusted.gpg.d/ -y --no-install-recommends install jq
| Reading package lists...
| Building dependency tree...
| The following additional packages will be installed:
| [...]
| Get:33 https://debian.sipwise.com/debian trixie/main amd64 libnss-myhostname amd64 257.5-2 [113 kB]
| Preconfiguring packages ...
| Fetched 25.3 MB in 4s (6777 kB/s)
| (Reading database ... 32224 files and directories currently installed.)
| Preparing to unpack .../base-files_13.7_amd64.deb ...
| Unpacking base-files (13.7) over (12.4+deb12u10) ...
| Setting up base-files (13.7) ...
| Installing new version of config file /etc/debian_version ...
|
| Configuration file '/etc/issue'
| ==> Modified (by you or by a script) since installation.
| ==> Package distributor has shipped an updated version.
| What would you like to do about it ? Your options are:
| Y or I : install the package maintainer's version
| N or O : keep your currently-installed version
| D : show the differences between the versions
| Z : start a shell to examine the situation
| The default action is to keep your current version.
|
| *** issue (Y/I/N/O/D/Z) [default=N] ? #
Avoid this, by setting DPKG option `--force-confnew`.
Change-Id: Ic5fed3dbe4744e07290159cec6952468c0557c29
vboxadd-service.service fails on our Debian/trixie systems:
| root@spce:~# lsb_release -c
| Codename: trixie
|
| root@spce:~# systemctl --failed
| UNIT LOAD ACTIVE SUB DESCRIPTION
| ● vboxadd-service.service loaded failed failed VirtualBox Guest Additions Services Daemon
|
| Legend: LOAD → Reflects whether the unit definition was properly loaded.
| ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
| SUB → The low-level unit activation state, values depend on unit type.
|
| 1 loaded units listed.
|
| root@spce:~# sudo systemctl status vboxadd-service.service
| × vboxadd-service.service - VirtualBox Guest Additions Services Daemon
| Loaded: loaded (/etc/systemd/system/vboxadd-service.service; disabled; preset: disabled)
| Drop-In: /etc/systemd/system/vboxadd-service.service.d
| └─override.conf
| Active: failed (Result: exit-code) since Thu 2025-04-24 09:08:15 CEST; 34min ago
| Invocation: 4e151a29f0054a90a717a928fcfb3f8d
| Mem peak: 2.2M
| CPU: 17ms
|
| Apr 24 09:08:15 spce systemd[1]: Starting vboxadd-service.service...
| Apr 24 09:08:15 spce vboxadd-service[1934]: vboxadd-service.sh: Starting VirtualBox Guest Addition service.
| Apr 24 09:08:15 spce vboxadd-service.sh[1937]: Starting VirtualBox Guest Addition service.
| Apr 24 09:08:15 spce vboxadd-service[1940]: VBoxService: error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND
| Apr 24 09:08:15 spce vboxadd-service.sh[1943]: VirtualBox Guest Addition service started.
| Apr 24 09:08:15 spce systemd[1]: vboxadd-service.service: Control process exited, code=exited, status=1/FAILURE
| Apr 24 09:08:15 spce systemd[1]: vboxadd-service.service: Failed with result 'exit-code'.
| Apr 24 09:08:15 spce systemd[1]: Failed to start vboxadd-service.service.
|
| root@spce:~# cat /etc/systemd/system/vboxadd.service.d/override.conf
| [Unit]
| ConditionVirtualization=oracle
|
| root@spce:~# cat /var/log/vboxadd-setup.log
| Building the main Guest Additions 7.0.6 module for kernel 6.12.22-amd64.
| Error building the module. Build output follows.
| make V=1 CONFIG_MODULE_SIG= CONFIG_MODULE_SIG_ALL= -C /lib/modules/6.12.22-amd64/build M=/tmp/vbox.0 SRCROOT=/tmp/vbox.0 -j2 modules
| make[1]: warning: -j2 forced in submake: resetting jobserver mode.
| [...]
| [,,,] /tmp/vbox.0/VBoxGuest-common.c
| /tmp/vbox.0/VBoxGuest-linux.c:196:21: error: ‘no_llseek’ undeclared here (not in a function); did you mean ‘noop_llseek’?
| 196 | llseek: no_llseek,
| | ^~~~~~~~~
| | noop_llseek
| /tmp/vbox.0/VBoxGuest-linux.c: In function ‘vgdrvLinuxParamLogGrpSet’:
| /tmp/vbox.0/VBoxGuest-linux.c:1364:9: error: implicit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Wimplicit-function-declaration]
| 1364 | strlcpy(&g_szLogGrp[0], pszValue, sizeof(g_szLogGrp));
| | ^~~~~~~
| | strncpy
| make[2]: *** [/usr/src/linux-headers-6.12.22-common/scripts/Makefile.build:234: /tmp/vbox.0/VBoxGuest-linux.o] Error 1
| make[2]: *** Waiting for unfinished jobs....
| [...]
We get virtualbox-guest-additions-iso v7.0.6-1 for Debian
stable/bookworm, but virtualbox-guest-additions-iso v7.0.20-1 is
available in current Debian testing AKA trixie. Ensure we use the
package from trixie for trixie based systems, even though the the
VirtualBox Guest Additions v7.0.20 don't work for kernel 6.12.22 either,
yet.
Also adjust ensure_packages_installed to fail installation, if we're
using a yet unknown/unexpected Debian release, to not fall back to
Debian/bookworm, to prevent issue like it has been observed here.
See MT#60815 for main tracking issue WRT Debian/trixie
Change-Id: I030525d37edbe1cf75065d021b51d38273ce81ef
As reported when sending new deployment-iso reviews,
triggered by newer docker image / shellcheck:
| not ok 1 source/templates/scripts/includes/deployment.sh:1543:10: warning: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. [SC2206]
| not ok 2 source/templates/scripts/includes/deployment.sh:1903:22: warning: Prefer mapfile or read -a to split command output (or quote to avoid splitting). [SC2207]
| not ok 3 source/templates/scripts/includes/deployment.sh:2275:20: warning: Prefer mapfile or read -a to split command output (or quote to avoid splitting). [SC2207]
| not ok 4 source/templates/scripts/includes/deployment.sh:2486:12: note: Not following: ./etc/profile.d/puppet-agent.sh was not specified as input (see shellcheck -x). [SC1091]
Let's take this as a chance to properly parse ip(8) output via its JSON
output, instead of awk/sed magic.
Change-Id: I723959626fb514ab9e57202b0e5f415b411f5a01
We have made these services conditional on running inside a VirtualBox
VM, so we do not need to remove them anymore.
Change-Id: I6dc563688ba5b0c5e935b0cb88767fcb05ab9a19
On Debian/trixie we get a failing efi.mount systemd unit:
| root@sp1:~# systemctl --failed
| UNIT LOAD ACTIVE SUB DESCRIPTION
| ● efi.mount loaded failed failed EFI System Partition Automount
|
| Legend: LOAD → Reflects whether the unit definition was properly loaded.
| ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
| SUB → The low-level unit activation state, values depend on unit type.
|
| 1 loaded units listed.
|
| root@sp1:~# systemctl status efi.mount
| × efi.mount - EFI System Partition Automount
| Loaded: loaded (/run/systemd/generator.late/efi.mount; generated)
| Active: failed (Result: exit-code) since Fri 2024-11-15 17:20:59 CET; 28min ago
| Invocation: 62c7b659dfd540e294f4b1f6fcda5e13
| TriggeredBy: ● efi.automount
| Where: /efi
| What: /dev/disk/by-diskseq/9-part2
| Docs: man:systemd-gpt-auto-generator(8)
| Mem peak: 1.5M
| CPU: 8ms
|
| Nov 15 17:20:59 sp1 systemd[1]: Mounting efi.mount - EFI System Partition Automount...
| Nov 15 17:20:59 sp1 mount[631]: mount: /efi: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or helper program, or other error.
| Nov 15 17:20:59 sp1 mount[631]: dmesg(1) may have more information after failed mount system call.
| Nov 15 17:20:59 sp1 systemd[1]: efi.mount: Mount process exited, code=exited, status=32/n/a
| Nov 15 17:20:59 sp1 systemd[1]: efi.mount: Failed with result 'exit-code'.
| Nov 15 17:20:59 sp1 systemd[1]: Failed to mount efi.mount - EFI System Partition Automount.
|
| root@sp1:~# ls -la /efi
| ls: cannot open directory '/efi': No such device
|
| root@sp1:~# ls -la /dev/disk/by-diskseq/9-part2
| lrwxrwxrwx 1 root root 10 Nov 15 17:20 /dev/disk/by-diskseq/9-part2 -> ../../sda2
|
| root@sp1:~# blkid /dev/sda2
| /dev/sda2: PARTLABEL="EFI System" PARTUUID="fa67b52e-c018-401d-ac71-fad324cad193"
The efi.mount systemd unit is automatically generated by
systemd-gpt-auto-generator. Quoting from systemd-gpt-auto-generator(8):
| The ESP is mounted to /boot/ if that directory exists and is not used
| for XBOOTLDR, and otherwise to /efi/
This got introduced as of systemd v254, see
6a488fa7cc
for details. Now having systemd v256.7-3 in current Debian/testing AKA
trixie, we need to make sure to not present an EFI partition if we
actually don't use it.
So when we don't run in an EFI environment do not mark the second
partition as EFI one.
Change-Id: I546c77fce862a41594500a33da1178c5c6182a1a
If we set up /etc/ssh/sshd_config early in early system deployment, we
end up with an empty /etc/ssh/sshd_config configuration file with only
our own changes:
| root@spce:~# cat /etc/ssh/sshd_config
| # added by deployment.sh
| PerSourcePenalties no
| # end of deployment.sh changes
| ### Added by ngcp-installer
| PermitRootLogin yes
The other defaults of sshd are OK for us, but for automated SSH logins
we also need:
AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/sipwise_vagrant_key
And for SCP-ing files we also need:
Subsystem sftp /usr/lib/openssh/sftp-server
Otherwise our Jenkins job fail due to failing ssh/scp actions.
So instead move our trixie specific code in deployment.sh for adjusting
/etc/ssh/sshd_config to be executed *after* installing base system. Then
the openssh-server package sets up /etc/ssh/sshd_config as expected, and
we only extend its configuration then. While at it, explicitly mark
beginning and end of our changes.
Change-Id: I68a235b55e9cf18c39e9034b7f3b2ed0ffd237f0
Our https://jenkins.mgm.sipwise.com/job/daily-build-matrix-debian-boxes/
matrix no longer provides builds for debian/trixie, because its
daily-build-images subproject Jenkins job with its proxmox-vm-clean-fs
job failed to run.
After running proxmox-vm-clean-fs under `set -x`, and also overriding
the ssh_wrapper function with `ssh -v ...`, I managed to grab this from
the Jenkins job execution:
| + ssh -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o 'ServerAliveInterval 10' -o 'ConnectTimeout 15' 192.168.210.101 'rm -vf /etc/udev/rules.d/70-persistent-net.rules'
| OpenSSH_9.2p1 Debian-2+deb12u3, OpenSSL 3.0.14 4 Jun 2024
| debug1: Reading configuration data /var/lib/jenkins/.ssh/config
| debug1: /var/lib/jenkins/.ssh/config line 7: Applying options for 192.168.*
| debug1: Reading configuration data /etc/ssh/ssh_config
| debug1: /etc/ssh/ssh_config line 50: Applying options for *
| debug1: /etc/ssh/ssh_config line 57: Deprecated option "useroaming"
| debug1: Connecting to 192.168.210.101 [192.168.210.101] port 22.
| debug1: fd 3 clearing O_NONBLOCK
| debug1: Connection established.
| debug1: identity file /var/lib/jenkins/.ssh/id_rsa_sipwise type 0
| debug1: identity file /var/lib/jenkins/.ssh/id_rsa_sipwise-cert type -1
| debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 0
| debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
| debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
| debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
| debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3
| debug1: kex_exchange_identification: banner line 0: Not allowed at this time
The `Not allowed at this time` pointed to a new OpenSSH feature, which
triggered the regression for us.
OpenSSH introduced options to penalize undesirable behavior, see
https://undeadly.org/cgi?action=article;sid=20240607042157 and
https://www.openssh.com/releasenotes.html#9.9p1 and
https://sources.debian.org/src/openssh/1:9.9p1-1/sshd.c/?hl=576#L573
This is now present as of openssh-server v1:9.9p1-1 since end of
September 2024 also in Debian/trixie.
Now, when too many SSH logins fail, a client system can't necessarily no
longer connect via SSH due this new penalty behavior. And indeed, within
our Jenkins job "daily-build-install-vm" we try to collect several log
files through our grab_log and SSH wrapper:
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /mnt/tmp/ngcp-installer-debug.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/ngcp-installer-debug.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/ngcp-installer-debug.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/ngcp-installer-debug.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /mnt/tmp/ngcp-installer.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/ngcp-installer.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/ngcp-installer.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/ngcp-installer.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /tmp/ngcp-installer-cmdline.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/ngcp-installer-cmdline.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /mnt/var/log/deployment.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/deployment.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/deployment.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/deployment.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /mnt/var/log/grml-debootstrap.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/grml-debootstrap.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/grml-debootstrap.log /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/grml-debootstrap.log
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/syslog /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/syslog
| + timeout 20 sipwise-ssh-copier 192.168.210.101 root sipwise /var/log/boot /buildtmpfs/tmp_jenkins-vm-builder/vmbuilder101/192.168.210.101/boot
We even execute this grab_log wrapper twice: once for the running Grml
live system, and once when we booted into the actually deployed system.
This works fine for the Grml live system situation, but as root logins
aren't allowed by default in OpenSSH since quite some time, all the
sipwise-ssh-copier runs with user/password against a plain Debian system
then fail.
As a consequence, we lock ourselves out of the system with all those SSH
login failures, and the Jenkins job proxmox-vm-clean-fs then runs into
the OpenSSH penalty, which causes the trixie/debian job to fail.
We use our Debian images as base for further configuration, where we
control the sshd_config file through our ngcpcfg system anyways, so the
`PerSourcePenalties no` setting is supposed to disappear then.
FTR: We could also enable `PermitRootLogin yes` in sshd_config to get
the grab_log working, though this didn't have any relevance for us so
far. Disabling only the `PerSourcePenalties` feature feels like a better
trade-off, at least security wise, for now.
Change-Id: Ibf16019b4787cc63d450501c8bccebeac77dd9f1
We install the qemu-guest-agent package in ensure_packages_installed().
Try to start the qemu-guest-agent service only afterwards therefore.
Fixup for commit 82e6638b40
Change-Id: Ic4aa2e493851b4c92ac134d68a9a76e05485658d
/dev/virtio-ports/org.qemu.guest_agent.0 usually is a symlink to the
character device /dev/vport1p1. So adjust the device check accordingly
and only verify it exists, but don't expected any special file type.
This actually matches the behavior we also have in ngcp-installer.
Fixup for commit 82e6638b40
Change-Id: I0aa93c1f0e1086847eb7ed6967692a52e183bdc3
Now that we enabled the QEMU Guest Agent option for our PVE VMs, we need
to have qemu-guest-agent present and active. Otherwise the VMs might
fail to shut down, like with our debian/sipwise/docker Debian systems
which are created via
https://jenkins.mgm.sipwise.com/job/daily-build-matrix-debian-boxes/:
| [proxmox-vm-shutdown] $ /bin/sh -e /tmp/env-proxmox-vm-shutdown7956268380939677154.sh
| [environment-script] Adding variable 'vm1reset' with value 'NO'
| [environment-script] Adding variable 'vm2' with value 'none'
| [environment-script] Adding variable 'vm1' with value 'none'
| [environment-script] Adding variable 'vm2reset' with value 'NO'
| [proxmox-vm-shutdown] $ /bin/bash /tmp/jenkins14192704603218787414.sh
| Using safe VM 'shutdown' for modern releases (mr6.5+). Executing action 'shutdown'...
| Shutting down VM 106
| Build timed out (after 10 minutes). Marking the build as aborted.
| Build was aborted
| [WS-CLEANUP] Deleting project workspace...
Let's make sure qemu-guest-agent is available in our Grml live system.
We added qemu-guest-agent to the package list of our Grml Sipwise ISO
(see git rev 65c3fea4c), but to ensure we don't strictly depend on this
brand new Grml Sipwise ISO yet, make sure to install it on-the-fly if
not yet present (like we already did for git, augeas-tools + gdisk).
Also make sure qemu-guest-agent service is enabled if socket
/dev/virtio-ports/org.qemu.guest_agent.0 is present (indicating that the
agent feature is enabled on VM level).
Furthermore ensure qemu-guest-agent is present also in the installed
Debian system. Otherwise when rebooting the VM once it's no longer
running the Grml live system but the installed Debian system, it might
also fail to shutdown. So add it to the default package list of packages
for bootstrapping.
Change-Id: Id6adac55a47cfaed542cad2f9ac9740783e6d924
With this variable we had some tricks in ngcp-initial-configuration if
the Pro sp2 node is installer via iPXE/cm image.
Now we support installation of sp2 via iPXE only so no need to pass this
variable.
But we need to keep parent ngcppxeinstall parameter as we need this
information for netcardconfig.
Change-Id: I20491289917cbb427ad6f5670f108c632838be71
We are dropping the scenario when sp2 node is installed from cd image so
remove appropriate part of the code.
Change-Id: Idced6b43a21add903dca070aa68f84b77acba28e
The code trying to fetch the OpenPGP certificate from a keyserver has
been non-functional for a while as the GPG_KEY_SERVER variable was
removed in commit 316c28bcc2. Instead of
restoring the variable with an up-to-date keyserver (not part of the
SKS pool, as that network is dead), we remove the support entirely as
it's a potential security issue due to fingerprint collisions for
example.
As a side effect this removes apt-key usage which has been deprecated
upstream and is slated for removal.
Change-Id: I63171a66201c631da9233d54579bd1601ff22e3e
Mykola Malkov
Michael Prokop
Sipwise Jenkins Builder
Guillem Jover