We want to get rid of old sipwise gpg file as it contains weak key. To
do it we need either update this hardcoded value (and do it every time
when key is updated) or use the same behavior as it's used in
installer.
Change-Id: I77882d3b9f52156ce345f46217f3d438466e018f
grml2usb <=0.14.14 works when building with ISO grml64-small_2018.04.11-efi.
Newer grml2usb versions provide SecureBoot support (which was introduced in
grml2usb v0.16.0). This is failing with our grml64-small_2018.04.11-efi ISO
though, because its provided EFI image doesn't contain a valid/mountable FAT
file system:
| % file /tmp/grml2iso.tmp/grml2usb55ey_q68/boot/efi.img
| /tmp/grml2iso.tmp/grml2usb55ey_q68/boot/efi.img: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows
Whereas the efi file from more recent Grml(-Sipwise) ISOs like
https://deb.sipwise.com/deployment-iso/grml/grml-sipwise-buster-20191022_addons.iso
looks like this:
| % file /mnt/test1/boot/efi.img
| /mnt/test1/boot/efi.img: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "mkfs.fat", sectors/cluster 4, root entries 512, sectors 8192 (volumes <=32 MB) , Media descriptor 0xf8, sectors/FAT 6, sectors/track 32, heads 64, serial number 0xef681600, label: "GRML ", FAT (12 bit)
and can be properly mounted for further adjusting by grml2usb/grml2iso.
So use grml2usb version v0.17.0 except when handling ISO
grml64-small_2018.04.11-efi ISO, then we use the version we had available on
our Debian/stretch environments back then, AKA grml2usb v0.14.14.
Change-Id: I452d7cbac138d59dc11fb1773ca3d6f6c307a6df
(cherry picked from commit 608dd03674)
This is a *partial* cherry-pick from commit c3d753562, excluding all the
testing related changes and keeping only viable and safe changes.
Relevant changes included here:
* Stick grml2usb version to v0.17.0, instead of relying on the grml2usb
version available on the host system (being 0.14.14 on Debian/stretch as
present on our current build hosts). For arbitrary addon file names we
need grml2iso (which uses grml2usb underneath) from grml2usb >=0.17.0.
FTR, grml2iso and grml2usb can be executed from within the git repository,
assuming all relevant tools are present
* No longer invoke isohybrid on the resulting ISO, instead rely on
grml2iso behaviour (which also checks for EFI support and enables
according switches as needed)
* Fix usage instructions in t/Dockerfile (it's "deployment-iso-buster" and not
"lua-ngcp-kamailio-buster")
This is in preparation for backporting/cherry-pick commit 608dd0367
("TT#73210 Use grml2usb v0.14.14 when handling grml64-small_2018.04.11-efi.iso")
to use a specific grml2usb version when dealing with special
case ISO grml64-small_2018.04.11-efi.iso for mr6.5 release.
Change-Id: I67e3f85bbe86bd1b3ee709161504b5250ca5d7fe
(*partially* cherry picked from commit c3d753562d)
The Debian security repository layout was modified. Nowadays only
`http://security.debian.org/dists/testing-security/` exists and
`http://security.debian.org/ testing/updates' is gone.
Quoting from https://lists.debian.org/debian-security/2019/06/msg00015.html:
| I would like to switch to *-security instead of */updates starting with
| bullseye. There will likely be some complications, but they should be
| solvable by the time we will publish packages in bullseye-security.
Since we're invoking `apt-get update` on the Grml live system, without
checking what's configured inside apt's sources.list setup, installation
might[1] fail with:
| E: The repository 'http://security.debian.org testing/updates Release' does not have a Release file.
Instead set up a custom temporary apt configuration, similar to what
we're already doing in ensure_packages_installed() of deployment.sh.
[1] This depends on the version of the underlying Grml ISO. ISO builds
being based on grml-live commit 3e832a3a10 or newer include a modified
sources.list setup that's supposed to be working fine.
Change-Id: I420c189115cc173847a826832f7dbb76324cb418
Remove links manually as we are in chroot now so can not use systemctl calls
Change-Id: Iebfe5952d403a2cab658264eb9b921c6354c39d8
(cherry picked from commit d4273164f6)
In order to detect if it is necessary to run init or join actions
during the initial configuration JOIN_CLUSTER option is used so
there is no sense to pass RETRIEVE_MGMT_CONFIG to it.
Change-Id: I89c0bf9a96511b2de5e1bf53acd0f631a32f7429
(cherry picked from commit 2084d1d6c8)
Make it enabled by default.
This option is needed only for initial configuration tools so
we can safely change it here.
Since we use the same way of installation as Carrier for Pro we
need this option in Pro config also.
Change-Id: Id72cb92c2b808143c9380dc23160574061c6c225
(cherry picked from commit b18bf5e13b)
In some cases ICMP is blocked on a gateway.
To get connectivity status (if the server is online) ping several
servers:
1) gateway
2) deb.sipwise.com
3) dns server or 1.1.1.1
Based on the pull request
https://github.com/sipwise/deployment-iso/pull/1/
Thanks: Isaac McDonald for the PR.
Change-Id: Iebb31de8105e3b329bd0b8fc068abc9b55326ed8
(cherry picked from commit f7db085697)
For Pro installation from another node we use internal network for
installation and we do not need to configure gateway on internal interface.
Change-Id: I8e95851c1b2132728e8846cf0c1d0d5149d06d74
(cherry picked from commit 69b58f4fd1)
In interactive mode user choose an interface and its number is used
in the code. In function 'configiface' the actual interface is taken
from NETDEVICES by this number.
In non-interactive mode we have an interface name and it was used as
index in NETDEVICES which failed and the 1st interface was always taken.
It does not matter what interface was passed in NET_DEV variable script
tried to configure eth0.
Fix it with getting interface number by its name.
Change-Id: I1a31c6f4a068ae75c20978e33b77f55f3c0aed76
(cherry picked from commit 04774ae39d)
By default we do assumption that we are installing to standard block devices,
this is not working for NVMe hardware name spaces.
Here we try to detect underlying device for named partition.
Change-Id: I7d2ea339a3aee2a8458a72cfc392441721d350c7
(cherry picked from commit cc95e9be62)
We perform all packages installation on 'root' partition.
Some packages, like faxserver for now, want to create
something on 'data' partition. They do it successfully in
Debian maintainer files (like postinst), so the 'data'
partition has to be mounted properly, otherwise the data
will be gone on reboot.
Also we need to unmount it properly at the end of installation.
P.S. we need to call mount under grml-chroot as chroot fails:
> (netscript.grml:1353): main(): chroot /mnt mount /ngcp-data
> mount: /ngcp-data: mount failed: Unknown error -1
Change-Id: Ief90d380d71ea6e0a64eba45f403465989865952
(cherry picked from commit b1d78cb40c)
The code was using old, non-existing variable names from previous versions of
the code.
Change-Id: Ia5fc090fbd37cc54d6a353d8792e6e1b8cca4e90
(cherry picked from commit 9de6138cba)
Starting with the mr6.5 release we no longer set up swap partition/LV via
deployment.sh (see deployment-iso commit 6a92f155).
Instead, the system will now use swapfiles, which are easier for operations like
changing the size dynamically or removing them completely.
This change implements the configuration for a swapfile during deployment,
enabled by default and with 1/2 the size of the main memory, limited between 4
and 16GB. The file itself will be created later, during the initial NGCP
configuration phase.
Change-Id: I09a5a77ecfed3924184fcc7c84c6b6b21dcacc98
(cherry picked from commit a807de1145)
Add menu if Software RAID should be configured and which disks
should be part of it. Only 2 disks allowed.
Move all disk selection code to separate script.
Script will write variables to /tmp/disk_selection file to pass
it to main.sh. It is necessary because 'dialog' is used for
interactivity so we can not capture output of script.
Change-Id: I3dd772c9a6ac6db6809688ff38f242271d9551a8
NGCP software is fully functional without '/ngcp-fallback',
the boot process should NOT be aborted on the errors here.
'/ngcp-fallback' is necessary for better usability of the system
and during upgrade where we will check mount point in RW state anyway.
Also tune layout here to match with spaces count to '/ngcp-data'.
Change-Id: Ic63f87e38ee5e8582060e9ca515492274ba8c55b
It is necessary to test Carrier installations on real hardware,
where we have partitions/RAID configs from the previous tests.
Change-Id: Ifec5ab28ee365737775ef883d152049bc5733b2b
We need to store FS size into config.yml to be able to tune iPXE setup
to install second+ Carrier node on internal Proxmox-based test
Carrier installation (as Jenkins control web01a node there only).
Change-Id: I03db9702e540b7e90b564645e9933316c08f3b59
We need an ability to install all our products in a limited
HW environment and also be able to pack them into different boxes
(like Vagrant, VirtualBox, VmWare, Docker).
The new boot option 'fallbackfssize' gives us ability to manage
the virtual VMs 'fallback' partition size.
Also mount '/ngcp-fallback' in read-only mode by default,
it is very useful to have access to the old 'code' partition.
Rename '/ngcpdata' to '/ngcp-data' for better readability and
matching to 'ngcp-fallback' ('ngcpfallback' is hardly readable).
Change-Id: I512d65254d2f163482734d94cfc36190fb297d8e
Otherwise PRO/Carrier cannot be installed because NGCP
depends on a lot of huge packages including:
* NGCP CloudPBX firmwares ~1Gb ad for mr6.5 and growing
* GRML squashfs ~300Mb
* Sipwise prompts ~400Mb
At the moment of installing all the packages are downloaded
into apt cache (~2.9Gb in total as for mr6.5) + have to be
installed/unpacked (~3.2Gb in total as for mr6.5).
All-to-all I was not able to install Carrier web01a node
even with ROOTFS_SIZE=6Gb. I had to increase it till '7Gb'
to install Carrier successfully.
Lets use 10GB for root FS to have free space protection here.
The possible options were checked:
* move apt cache into RAM, has some disadvantages, like
we need 4GB+ RAM for cache only => hard to test on Jenkins
* move apt cache to 'data' partition is also bad idea as
it is a 'code' cache so have to be consistent with 'code'
(so it is release specific and must be stored on 'root'/'fallback')
Change-Id: Ia34dcc15230e152e894e813dfbaa8688a5145de1
Mcollective is deprecated in v6.0 and I don't see
any quick fix for that. So let's keep using v5.5
while deciding how to proceed further.
Change-Id: Ie72e5a57afc0a7235f9d7d29d52af7d796db1ea8
The users can assign disks using the full path:
'swraiddisk1=/dev/sda' instead of expected 'swraiddisk1=sda'.
As a result the installation fails with not a user friendly errors.
Let's be more user friendly here and remove initial '/dev/' is available.
Change-Id: I094814433b294ca8f084b0412cb9b67537cba8cf
Assuming we have two disks /dev/sda + /dev/sdb and boot the
system with the (new) boot options "swraiddisk1=sda
swraiddisk2=sdb", then we will get a SW-RAID (Software RAID)
setup which looks like (the /boot/efi is only present with EFI
support and only present for the active partition):
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| |-sda1 8:1 0 1M 0 part
| |-sda2 8:2 0 486M 0 part /boot/efi
| `-sda3 8:3 0 15.5G 0 part
| `-md0 9:0 0 15.5G 0 raid1
| |-ngcp-root 253:0 0 5G 0 lvm /
| |-ngcp-fallback 253:1 0 5G 0 lvm
| `-ngcp-data 253:3 0 5G 0 lvm /ngcpdata
| sdb 8:16 0 16G 0 disk
| |-sdb1 8:17 0 1M 0 part
| |-sdb2 8:18 0 486M 0 part
| `-sdb3 8:19 0 15.5G 0 part
| `-md0 9:0 0 15.5G 0 raid1
| |-ngcp-root 253:0 0 5G 0 lvm /
| |-ngcp-fallback 253:1 0 5G 0 lvm
| `-ngcp-data 253:3 0 5G 0 lvm /ngcpdata
The /dev/md0 resides in the 3rd partition, where we usually have
plain LVM. /dev/md0 is RAID level 1, on top of the two disks.
Notes:
* we hardcode /dev/md0 as raid-device for mdadm/SW-RAID
* existing data on EFI partitions will be re-used (as grml-debootstrap
won't overwrite the partition it if contains a filesystem already)
* neither the BIOS (legacy, being first partition) nor the UEFI
partition (being the second partition) can be part of the SW-RAID
setup (see https://wiki.debian.org/UEFI -> "RAID for the EFI
System Partition"). Therefore we need to make sure, that the BIOS
+ UEFI partitions are kept in sync, or at least provide data as
needed. We will have to handle this during e.g. upgrades +
re-installations.
* we need to invoke grub-install for both disks, otherwise
GRUB is available only from the first disk
* install gdisk package to have sgdisk binary available (missing
on grml-small)
Change-Id: I6ebe2c25326c2fdf9ed8fe2bd7b9bf540c7689fd
Requested features:
* switch from msdos partition table to GPT
* (optional) support for (U)EFI
* use a new LVM partition layout, with option to
rollback/install/upgrade via second rootfs partition
(which is called "fallback" in this implementation)
* drop swap partition
Note:
* we no longer support msdos partition tables but GPT only,
as (U)EFI systems can only boot from GPT (and not from
BIOS/legacy boot), while BIOS systems can also boot from GPT
* https://wiki.archlinux.org/index.php/Partitioning +
https://wiki.archlinux.org/index.php/GRUB provide a decent
overview, if you're not familiar with BIOS/GPT/(U)EFI
New partition layout:
* 1st partition: 1M BIOS boot, for BIOS/GPT (legacy) boot
- this allows fallback to grub-pc package
(needs to be partition type GUID 21686148-6449-6E6F-744E-656564454649
AKA set to bios_grub with parted, then installing grub to disk
so it properly embeds core.img, can be done from a
rescue/live system)
* 2nd partition: ~500MB EFI System, for UEFI/GPT boot
- used as /boot/efi, iff EFI support is available
* 3rd partition: LVM with:
- /dev/mapper/ngcp-root with 5GB (rootfs target) +
/dev/mapper/ngcp-fallback with 5GB (for rollback/install/upgrade)
- 10% or >=500MB unassigned of the remaining space (whatever is bigger)
- /dev/mapper/ngcp-data data partition with rest of
available disk space, used as /ngcpdata on system
Old layout with a 16GB disk for NGCP systems:
| # lsblk
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| └─sda1 8:1 0 16G 0 part
| ├─ngcp-root 254:0 0 14.3G 0 lvm /
| └─ngcp-swap 254:1 0 1004M 0 lvm [SWAP]
| # pvs
| PV VG Fmt Attr PSize PFree
| /dev/sda1 ngcp lvm2 a-- 16.00g 764.00m
| # vgs
| VG #PV #LV #SN Attr VSize VFree
| ngcp 1 2 0 wz--n- 16.00g 764.00m
| # lvs
| LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
| root ngcp -wi-ao---- 14.27g
| swap ngcp -wi-ao---- 1004.00m
New layout with a 16GB disk (+ EFI support) for NGCP systems:
| # lsblk
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| |-sda1 8:1 0 1M 0 part
| |-sda2 8:2 0 486M 0 part /boot/efi
| `-sda3 8:3 0 15.5G 0 part
| |-ngcp-root 254:0 0 5G 0 lvm /
| |-ngcp-fallback 254:1 0 5G 0 lvm
| `-ngcp-data 254:2 0 5G 0 lvm /ngcpdata
| # pvs
| PV VG Fmt Attr PSize PFree
| /dev/sda3 ngcp lvm2 a-- 15.52g 564.00m
| # vgs
| VG #PV #LV #SN Attr VSize VFree
| ngcp 1 3 0 wz--n- 15.52g 564.00m
| # lvs
| LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
| data ngcp -wi-ao---- 4.97g
| fallback ngcp -wi-a----- 5.00g
| root ngcp -wi-ao---- 5.00g
Change-Id: I39206b225edb25fd1e27de49a818ad7a54532f82
We have confusing situation for an ages here.
We were writing a lot of log files:
> /var/log/deployment-installer-debug.log
> /var/log/ngcp-installer.log
> /var/log/ngcp-installer-debug.log
The first one contains all the GRML boot, Debian debootstrap
and ngcp-installer-debug.log already. Also having both
ngcp-installer.log and ngcp-installer-debug.log confuses
new users as they do not know which logs they need to
check first to see ngcp-installer logs.
There is no need to keep both files anymore.
Nowadays we have a nice way to distinguish logs between components,
so /var/log/deployment-installer-debug.log is enough here:
> +15:07:48 (netscript.grml:1403): main(): echo 'Generating ngcp-installer run script ...'
> +15:07:48 (netscript.grml:1404): main(): cat
> +15:07:48 (netscript.grml:1417): main(): echo 'Execute ngcp-installer inside deployment chroot environment ...'
> ...
> Running ngcp-installer via grml-chroot.
> ...
> +17:07:48 (ngcp-installer:27): main(): . /usr/share/ngcp-installer//system_pro.inc
> +17:07:48 (ngcp-installer:28): main(): . /usr/share/ngcp-installer//packages.inc
> +17:07:48 (ngcp-installer:31): main(): . /usr/share/ngcp-installer//cfg.inc
> ...
> +17:07:57 (packages.inc:58): packages_ppa_sourceslist(): echo '(system installed using NGCP PPA: gerrit_gtid_imp)'
> +17:07:57 (ngcp-installer:54): main(): packages_aptconfig
> +17:07:57 (packages.inc:90): packages_aptconfig(): mkdir -p /etc/apt/preferences.d
> +17:07:57 (packages.inc:92): packages_aptconfig(): cat
The commit here should minimize amount of log files we
produce and should simplify the debug process.
Change-Id: I26a2ca8e21a3385f7d57d830feb207bbb1039b39
We do not use /tmp/ngcp-installer.log since mr6.5 and
using directly /var/log/ngcp-installer.log inside chroot.
Change-Id: I35b098fb09046a0fbc0a9bfc56bd125362c34fd0
We're executing `apt-get update` even when nothing needs to be
installed. This adds unnecessary overhead and can fill ram/disk
on live system on consecutive runs, so let's avoid that.
Change-Id: Ic1a40d82988caf454f317b4803e03f4c9de3feea
Implement suggestions from review I16e4f19f3b9270ccfef6c7c1274bb8b8d95695bd
1. In some cases there can be multiple gateways. Return only the 1st one.
2. Remove configuration of the management (IMM2/IPMI) interface of
lenovo servers.
Change-Id: Ia1bb6e9f0e44bcb0c6be87407a8c11899163c03a
All non-zero returned values are caught by '-e' option and passed
to function wait_exit by 'trap'.
So if it is an error we can set 'error' status in this function.
Change-Id: I762e5470d94a537b4f8f0e32fe6d0c1575bbb075
Sipwise Jenkins Builder
Mykola Malkov
Michael Prokop
Volodymyr Fedorov
Alexander Lutay
Manuel Montecelo
Sergii Kipot