Current trunk installations based on bullseye using recent Grml
environments are broken, as EFI environments running with recent kernel
versions (>=5.10) aren't properly detected anymore.
This is caused by the missing efivars kernel module.
CONFIG_EFI_VARS is no longer available since
20146398c4
(tagged initially as debian/5.10.1-1_exp1 + shipped with kernel package
5.10.1-1~exp1 and newer, incl. 5.10.38-1 as present in current
Debian/unstable). Therefore the kernel module efivars is no longer
available on more recent Debian kernel systems.
Quoting from https://wiki.debian.org/UEFI:
| The older interface was efivars, showing files under
| /sys/firmware/efi/vars, and this is what was used by default in both
| Wheezy and Jessie.
|
| The new interface is efivarfs, which will expose things in a slightly
| different format under /sys/firmware/efi/efivars. This is the new
| preferred way of using UEFI configuration variables, and Debian switched
| to it by default from Stretch onwards.
CONFIG_EFI_VARS is no longer required, instead efivarfs seems to be
available starting with kernel v3.10 and newer (see linux.git):
| commit a9499fa7cd3fd4824a7202d00c766b269fa3bda6
| Author: Tom Gundersen teg@jklm.no
| Date: Fri Feb 8 15:37:06 2013 +0000
|
| efi: split efisubsystem from efivars
|
| This registers /sys/firmware/efi/{,systab,efivars/} whenever EFI is enabled
| and the system is booted with EFI.
|
| This allows
| *) userspace to check for the existence of /sys/firmware/efi as a way
| to determine whether or it is running on an EFI system.
| *) 'mount -t efivarfs none /sys/firmware/efi/efivars' without manually
| loading any modules.
|
| [ Also, move the efivar API into vars.c and unconditionally compile it.
| This allows us to move efivars.c, which now only contains the sysfs
| variable code, into the firmware/efi directory. Note that the efivars.c
| filename is kept to maintain backwards compatability with the old
| efivars.ko module. With this patch it is now possible for efivarfs
| to be built without CONFIG_EFI_VARS - Matt ]
and:
| commit d68772b7c83f4b518be15ae96f4827c8ed02f684
| Author: Matt Fleming matt.fleming@intel.com
| Date: Fri Feb 8 16:27:24 2013 +0000
|
| efivarfs: Move to fs/efivarfs
|
| Now that efivarfs uses the efivar API, move it out of efivars.c and
| into fs/efivarfs where it belongs. This move will eventually allow us
| to enable the efivarfs code without having to also enable
| CONFIG_EFI_VARS built, and vice versa.
|
| Furthermore, things like,
|
| mount -t efivarfs none /sys/firmware/efi/efivars
|
| will now work if efivarfs is built as a module without requiring the
| use of MODULE_ALIAS(), which would have been necessary when the
| efivarfs code was part of efivars.c.
But we also need to ensure /sys/firmware/efi/efivars is mounted,
otherwise efibootmgr fails to execute:
| # efibootmgr
| EFI variables are not supported on this system.
| # lsmod| grep efi
| efi_pstore 16384 0
| efivarfs 16384 1
| # mount -t efivarfs none /sys/firmware/efi/efivars
| # efibootmgr
| BootCurrent: 0002
| Timeout: 3 seconds
| BootOrder: 0001,0002,0003,0000,0004
| Boot0000* UiApp
| Boot0001* UEFI QEMU QEMU HARDDISK
| Boot0002* UEFI PXEv4 (MAC:02B31C8CA0AA)
| Boot0003* UEFI PXEv4 (MAC:92097BD02A48)
| Boot0004* EFI Internal Shell
FTR: we can't test only for existence of directory
/sys/firmware/efi/efivars, as it exists but is empty by default, so we
need to look inside the directory instead.
See https://github.com/grml/grml-debootstrap/pull/174 for the related
grml-debootstrap upstream change, which is supposed to be released as of
grml-debootstrap v0.97.
But as a) grml-debootstrap v0.97 isn't released yet, b) it's unclear
whether grml-debootstrap v0.97 will make it into bullseye (soonish, or
if at all) and c) we don't have the Grml repositories available via our
approx Debian mirror (as used in our PRO/Carrier environments) and don't
want to update our Grml squashfs system for this change neither, we need
to apply a workaround for this efivars vs efivarfs situation. Otherwise
Debian installation fails in EFI environments using Debian kernel
>=5.10. Thankfully we can work around this using according pre/post
scripts in grml-debootstrap, that's what efivars_workaround() is all
about.
Thanks: Manuel Montecelo <mmontecelo@sipwise.com> for the initial patch and Volodymyr Fedorov <vfedorov@sipwise.com> for underlying research
Change-Id: I5374322cb0a39cfed6563df6c4c30f1eafe560c1
The "Building database of manual pages ..." of mandb(8) is invoked
during Debian package installations, and takes a considerable amount of
time[1]. By disabling this, we can speed up our installation process,
similar to what we already do with all our build environments.
If someone really needs the man-db database (for apropos(1) or
whatis(1) usage), then invoking `systemctl restart man-db.service`
provides that on demand.
FTR: there are also /etc/cron.daily/man-db + /etc/cron.weekly/man-db,
though they don't do anything when running under systemd. There's also
man-db.timer, though we don't have it enabled by default on our NGCP
systems.
[1] Demo from a running PRO system:
| root@sp2:~# rm -rf /var/cache/man
| root@sp2:~# time systemctl restart man-db.service
|
| real 1m18.357s
| user 0m0.000s
| sys 0m0.009s
Change-Id: If98007860490adc5ad954e8c36000abd7281931b
The dmraid package executes udevadm in its postinst script:
| jenkins@jenkins-slave11:/tmp/grml-live/deployment-iso$ docker run --rm -it -v "$(pwd)":/deployment-iso/ -v "$(pwd)/grml_build/":/grml/ docker.mgm.sipwise.com:5000/grml-build-bullseye:latest /bin/bash
| root@fa6b983da364:/code/grml-live# apt install dmraid
| [...]
| Setting up dmraid (1.0.0.rc16-8+b1) ...
| Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/uevent': Read-only file system
| dpkg: error processing package dmraid (--configure):
| installed dmraid package post-installation script subprocess returned error exit status 1
| Processing triggers for libc-bin (2.31-4) ...
| Errors were encountered while processing:
| dmraid
| E: Sub-process /usr/bin/dpkg returned an error code (1)
Also see https://bugs.debian.org/962300.
Installing the dmraid package on bullseye requires execution of docker
containers with privileged mode, which is something we would like to
avoid. dmraid also had its latest Debian upload in 2017, the last
upstream release dates back to 2010 (see
http://people.redhat.com/~heinzm/sw/dmraid/src/) and it has plenty of
bugs. It's furthermore relevant for so called fake RAIDs only, something
we don't support and therefore shouldn't matter for us.
Change-Id: I7ed50d8d732f75c56e94b8bfd97a71613577f3bd
Add options to install bullseye in all places where buster is used, use
it as default when possible, and keep these for the moment.
Switch to bullseye in Dockerfile.
Change-Id: I2f693982ba92a671a6f2254c5a245a1d05231404
The call:
UTS_RELEASE="${KERNELVERSION}" LD_PRELOAD="${FAKE_UNAME}" \
grml-chroot "${TARGET}" /media/cdrom/VBoxLinuxAdditions.run --nox11
fails with:
Running in chroot, ignoring request: daemon-reload
Before 8a54cd1374 it was skipped so hide
it with '|| true'.
Use 'grml-chroot' instead of 'chroot' as 'grml-chroot' is a wrapper
which also cares about required mountpoints.
Use single style for "${TARGET}" variable.
Change-Id: Icc625c9a58b114f62350fc1e540ddac8a4147f28
Quoting from "man bash" about `-E` (AKA errtrace):
| If set, any trap on ERR is inherited by shell functions, command
| substitutions, and commands executed in a subshell environment.
| The ERR trap is normally not inherited in such cases.
To demonstrate the problem see this short shell script:
| % cat foo
| set -eu -o pipefail
|
| bailout() {
| echo "Bailing out because of error" >&2
| exit 1
| }
| trap bailout 1 2 3 6 9 14 15 ERR
|
| foo() {
| echo "Executing magic"
| magic
| }
|
| foo
| echo end
If "magic" can't be executed, then this fails as follows:
| % bash ./foo
| Executing magic
| ./foo: line 11: magic: command not found
But it doesn't invoke the bailout function via trap.
When using `set -eE` (AKA errexit + errtrace), instead of only
`set -e` (errexit), then it behaves as expected though:
| % bash ./foo
| Executing magic
| ./foo: line 11: magic: command not found
| Bailing out because of error
Change-Id: I26396b87d4a391a75997c061e866709daa57870e
When running in EFI mode, it's as simple as executing "fwsetup".
This is taken over from upstream change
8c0c9d9228
Change-Id: I0fffee2aa814597ade9772bd93ab6d69b1ea4f9c
This pretty much never works for me, while exiting GRUB switches to the
next device in the boot order, so let's just "exit" GRUB and name the
menu entries accordingly.
This is taken over from upstream change
89c43982fb
Change-Id: I23106b202bfc57f3173cb1ee0326c5b7eddd0474
grml-live v0.38.0 is the current release, including a change to use 1m
block size for squashfs, which reduces ISO size (see
fada6dea0f),
improvements to EFI, GRUB, documentation and several bug fixes.
Change-Id: I3382bd81a8a41d3672a1a709200740cb7284cbbd
grub-pc >=2.04-11 has a new behavior regarding /boot/grub/i386-pc/
handling, where we end up with an empty /boot/grub/i386-pc/ after
*successful* grub-install execution:
| root@grml ~ # vgchange -ay
| 3 logical volume(s) in volume group "ngcp" now active
| root@grml ~ # mount /dev/mapper/ngcp-root /mnt
| root@grml ~ # grml-chroot /mnt /bin/bash
| Writing /etc/debian_chroot ...
| (spce)root@grml:/# cd
| (spce)root@grml:~# grub-install /dev/sda
| Installing for i386-pc platform.
| Installation finished. No error reported.
| (spce)root@grml:~# ls -la /boot/grub/i386-pc/
| total 16
| drwxr-xr-x 2 root root 12288 Dec 16 12:04 .
| drwxr-xr-x 4 root root 4096 Dec 16 12:07 ..
This causes the installed system to fail to boot with:
| GRUB loading..
| Welcome to GRUB!
|
| error: file `/boot/grub/i386-pc/normal.mod' not found.
| grub rescue> _
The underlying issue is that recent grub versions unlink the files
inside /boot/grub/i386-pc, though it doesn't report anything about it
(even under `--verbose` execution).
This is triggered in our situation, as lvm2's vgs binary isn't present
yet. In earlier versions of grub this wasn't causing any problems and
grub-install happily installed the files inside /boot/grub/i386-pc, even
though we installed lvm2 only afterwards via our metapackages. To
ensure lvm2 is available during installation time within
grml-debootstrap, explicitly add to it list of packages to be installed.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=977544 for further
details regarding the grub bug.
Change-Id: I27a1cd18777526eb26b838fae88d4d87b6e93467
We install virtualbox-guest-additions in the target system for usage
with VirtualBox and shared folders via Vagrant. We invoke the
VBoxLinuxAdditions.run machinery from the running Grml live system. But
the target systems usually has a different kernel package and version
installed, so we have to apply some tricks to get it working. This is
where we rely on fake-uname.so.
Since commit a91baa2 (TT#48647 Ship fake_uname lib in package) we're
relying on fake-uname.so from ngcp-deployment-scripts, instead of
building and shipping it via deployment.sh itself.
But we have ngcp-deployment-scripts available only when installing NGCP
- as we're installing it there and only afterwards invoke
vagrant_configuration() - whereas it's missing when we install a plain
Debian system (like with our debian_bullseye_plain_vagrant.box),
therefore failing with:
| cp: cannot stat '/mnt/usr/lib/ngcp-deployment-scripts/fake-uname.so': No such file or directory
| ERROR: ld.so: object '/usr/lib/ngcp-deployment-scripts/fake-uname.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Change-Id: I639a43c3deafd2fc188350936e15f48482103209
The ensure_packages_installed function ensures that specified packages
are present during runtime. This is used e.g. for installation of
virtualbox-guest-additions-iso Debian package from within
vagrant_configuration(), which is used to execute
/media/cdrom/VBoxLinuxAdditions.run inside the target system.
We can't use random Debian repositories though, as the package
dependencies need to match the running live system. So far we only used
the buster repository, as our current grml-sipwise ISOs are based on
something close to buster.
On the other hand we can't use virtualbox-guest-additions-iso from
Debian/buster in our Debian/bullseye Vagrant boxes, as
/sbin/mount.vboxsf doesn't work then.
So use the bullseye repository if the release of the target system is
bullseye, which seems to work with our current Grml ISOs and current
state of bullseye.
Change-Id: Iaf965daa6ff7a62e2b3bd8c55b8f761abd94c241
Nowadays we only deploy stretch + buster based Debian systems, so drop
those release specific checks to also support bullseye and newer Debian
releases.
Change-Id: Ibf3d1527ccaeba60526a730e6886e6521c08d20e
grml2usb v0.18.1 included a regression that caused us to revert to
grml2usb v0.17.0 in commmit 9fbc9afeda.
The according fix for our needs is
657b880f02
and was released as v0.18.2
grml2usb v0.18.3 supports Grml's new Secure Boot approach and switches
in grml2iso from isohybrid to xorriso/isohybrid to properly support EFI
boot, besides some further code cleanups, docs improvements and
bugfixes.
Change-Id: I0ce25d05825767120144315f07ded2c66bfffc0f
This is revert of commit aadbef82b0
The version 0.18.1 fails during the building of mr8.5.1:
===============================
10:59:57 Executing grml2usb version v0.18.1 (git)
10:59:57 Checking for boot flag
10:59:57 Fatal: /tmp/grml2iso.tmp/cddir: unrecognised disk label
===============================
So restore working 0.17.0 version.
Change-Id: Ib5b3369152378dc3aeb7282d098383c5df82904d
The /usr/bin/python symlink/binary no longer exists in recent
Grml-Sipwise ISOs and python3 doesn't ship SimpleHTTPServer but
http.server instead.
Change-Id: I6677e8a416b142034d99d5b1d2b11ba74d87a6ec
No need to install this package to non-vagrant system.
Do not add this package to Sipwise-grml image - it's too heavy (86M)
and not needed on real systems.
Change-Id: I9ec9ff76d588f4ced30ba199f05bb167eec5288a
Previously the functions and the code were fixed so it was hard to
understand the flow of execution.
Move all functions to a single section at the beginning of the script
so it's easier to understand the code.
Fix shellcheck warnings:
SC2086: Double quote to prevent globbing and word splitting.
SC2154: efidev1 is referenced but not assigned.
Change-Id: Ie4bf28c166e4a9ff236eff807ee97adae6ecddd0
This variable is used in some ngcpcfg *.services file.
Specifically 'ngcp-provisioning-tools/mysql_values.cfg.services' uses
this variable to specify mysql db host and if it's necessary to reload
kamailio.
Change-Id: Ibf137cdd0ad6f6492a30cfa715c468e4ac22832f
Initially wanted to sync with https://github.com/grml/grml-network/ as
of commit 49409a5587d, though templates/scripts/includes/netcardconfig
existed only to provide the VLAN patch until we had this available from
upstream and our own ISO. Now that everything has been upstreamed and we
have up2date Grml ISOs, let's switch to the one included in upstream
Grml.
Change-Id: I685cd0ad95033ad2b50036e53c2ee9814a776e63
Otherwise execution of FAI might fail:
| Calling task_faiend
| /usr/lib/fai/subroutines: line 142: ps: command not found
| [...]
This is supposed to be fixed with FAI 5.9.4, while
version 5.8.4 suffers from this bug, so until >=5.9.4
is available in buster/stable let's fix this via an
explicit dependency.
Change-Id: I99490f263d1b2a1aec65f55feebe429b62628918
We no longer support linux-image-amd64-grml, so there's
no point in sitting at kernel version 4.19.0-1-grml-amd64,
instead switch to the plain Debian kernel.
Change-Id: I00efa274acf9724241762ef43a15ecec61e2a409
This grml2usb version is supposed to be used for releasing
the upcoming Grml stable release and its related release candidate.
Change-Id: I172cabb02d245794856f2101990e5a445325f7c2
grml-live v0.35.3 is the current release, planned to be the
base for the upcoming new Grml stable release and its release candidate.
Change-Id: Ia5916250975b90f8d5f75d6fd1aefa5a9bd17d4c
Instead of having to maintain this ISO in our web server off-band,
we switch to use the packaged version, which makes validation
unnecessary as apt gives us that. And it also gives us a newer
version (currently 6.0.4 with Debian buster 10.3 vs the old 5.2.26).
Change-Id: Id89280bbe7fadeb35d391b5dc46e930935017588
grml2usb <=0.14.14 works when building with ISO grml64-small_2018.04.11-efi.
Newer grml2usb versions provide SecureBoot support (which was introduced in
grml2usb v0.16.0). This is failing with our grml64-small_2018.04.11-efi ISO
though, because its provided EFI image doesn't contain a valid/mountable FAT
file system:
| % file /tmp/grml2iso.tmp/grml2usb55ey_q68/boot/efi.img
| /tmp/grml2iso.tmp/grml2usb55ey_q68/boot/efi.img: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows
Whereas the efi file from more recent Grml(-Sipwise) ISOs like
https://deb.sipwise.com/deployment-iso/grml/grml-sipwise-buster-20191022_addons.iso
looks like this:
| % file /mnt/test1/boot/efi.img
| /mnt/test1/boot/efi.img: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "mkfs.fat", sectors/cluster 4, root entries 512, sectors 8192 (volumes <=32 MB) , Media descriptor 0xf8, sectors/FAT 6, sectors/track 32, heads 64, serial number 0xef681600, label: "GRML ", FAT (12 bit)
and can be properly mounted for further adjusting by grml2usb/grml2iso.
So use grml2usb version v0.17.0 except when handling ISO
grml64-small_2018.04.11-efi ISO, then we use the version we had available on
our Debian/stretch environments back then, AKA grml2usb v0.14.14.
Change-Id: I452d7cbac138d59dc11fb1773ca3d6f6c307a6df
From the packages' deps and content POV Pro and Carrier packages are
equal to each other so get rid of this separation.
Change-Id: I587dba3147bdc9b3c0f10da820ba4cdc8a0a0f08
Newer gcc versions have become more picky on their argument order, due
to the --as-needed default, and require the libraries to be linked to,
to be passed after the code/objects that use them, otherwise they will
get dropped as unused.
Change-Id: I8ace79186b0c8709ccb9cb93f51a9890ce6e1043
* Addon fixes:
* Provide custom addon template according to our needs, instead of
relying on the default one being present in grml-live.git/templates
* Install memtest86+ as addon for BIOS usage
* Install netboot.xyz.lkrn for BIOS usage and netboot.xyz.efi for EFI usage
* Install ipxe.lkrn for BIOS usage and ipxe.efi for EFI usage
* Stick grml-live version to v0.34.3, instead of relying on some random
git master version
* Stick grml2usb version to v0.17.0, instead of relying on the grml2usb
version available on the host system (being 0.14.14 on Debian/stretch as
present on our current build hosts). For arbitrary addon file names we
need grml2iso (which uses grml2usb underneath) from grml2usb >=0.17.0.
FTR, grml2iso and grml2usb can be executed from within the git repository,
assuming all relevant tools are present
* No longer invoke isohybrid on the resulting ISO, instead rely on
grml2iso behaviour (which also checks for EFI support and enables
according switches as needed)
* Fix usage instructions in t/Dockerfile:
* it's "deployment-iso-buster" and not "lua-ngcp-kamailio-buster"
* refer to working directory instead of "deployment-iso.git",
which very probably isn't named as such on any of our systems,
while the $(pwd) approach works for c/p
* Fix docker build usage in grml_build/Dockerfile (for building we need
to provide a PATH (being current working directory for us)
* Provide testing tools in grml-build-buster docker environment
* Provide new testing script t/iso-tester to compare generated ISO
against pre-defined screenshot (only testing memtest feature using
./t/screenshots/01-memtest.jpg for now)
Change-Id: I67e3f85bbe86bd1b3ee709161504b5250ca5d7fe
Michael Prokop
Mykola Malkov
Manuel Montecelo
Sipwise Jenkins Builder
Guillem Jover
Sergii Kipot