When bootstrapping into a tmpfs (running inside a VirtualBox VM using a
Grml-Sipwise ISO), debootstrap takes 2min 17seconds, while mmdebstrap
takes only ~20 seconds. This is a notable change that's worth using it
by default.
Quoting mmdebstrap's package description:
| Downloads, unpacks and installs Debian packages to either directly create a
| directory which can be chrooted into, or a tarball of it. In contrast to
| debootstrap it uses apt, supports more than one mirror, automatically uses
| security and updates mirrors for Debian stable chroots, is 3-6 times faster,
| produces smaller output by removing unnecessary cruft, is bit-by-bit
| reproducible if $SOURCE_DATE_EPOCH is set, allows unprivileged operation using
| Linux user namespaces, fakechroot or proot and can setup foreign architecture
| chroots using qemu-user.
Further differences noted between debootstrap + mmdebstrap:
* debootstrap requires exec + dev permissions on the target,
while mmdebstrap doesn't need them (being a good thing, actually)
* mmdebstrap pulls in gnupg1/gpgv1 on the target system (stretch-only),
while debootstrap doesn't
* debootstrap leaves the Debian packages in /var/cache/apt/archives behind,
while mmdebstrap doesn't
* debootstrap leaves the Debian repository files in /var/lib/apt/lists behind,
while mmdebstrap doesn't
* mmdebstrap doesn't consider apt, debconf as Priority 'required' but as 'important',
gcc-8-base, libacl1, libattr1 + zlib1g as 'required' instead of 'optional',
libbz2-1.0 + libpcre3 as 'important' instead of 'optional';
libdb5.3 + libtasn1-6 as 'standard' instead of 'optional'
None of those issues should cause any issues for us, though.
Change-Id: I93616263c2fed45ab8063fce024b98a7c6272660
As we build and use our specific Grml images with all required packages
we don't need to install any additional ones.
Change-Id: I20df3b0e676fc49439cb9a7cfe250e71f71c6238
The version 5.2.18 building is very slow on the recent
Debian stretch kernel 4.9.0-8-amd64:
VBoxGuestAdditions version 5.2.18:
> 12:32:52 (netscript.grml:1854): vagrant_configuration(): grml-chroot /mnt /media/cdrom/VBoxLinuxAdditions.run --nox11
> Writing /etc/debian_chroot ...
> Verifying archive integrity... All good.
> Uncompressing VirtualBox 5.2.18 Guest Additions for Linux........
> VirtualBox Guest Additions installer
> Copying additional installer modules ...
> Installing additional modules ...
> VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules. This may take a while.
> VirtualBox Guest Additions: Starting.
> VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules. This may take a while.
> +12:39:15 (netscript.grml:1856): vagrant_configuration(): ...
VBoxGuestAdditions version 5.2.26:
> +13:35:50 (netscript.grml:1854): vagrant_configuration(): grml-chroot /mnt /media/cdrom/VBoxLinuxAdditions.run --nox11
> Writing /etc/debian_chroot ...
> Verifying archive integrity... All good.
> Uncompressing VirtualBox 5.2.26 Guest Additions for Linux........
> VirtualBox Guest Additions installer
> Copying additional installer modules ...
> Installing additional modules ...
> VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
> modules. This may take a while.
> VirtualBox Guest Additions: To build modules for other installed kernels, run
> VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup <version>
> VirtualBox Guest Additions: Building the modules for kernel 4.9.0-8-amd64.
> VirtualBox Guest Additions: Starting.
> Running in chroot, ignoring request: daemon-reload
> +13:39:12 (netscript.grml:1856): vagrant_configuration():
5.2.26 is the latest stable version from VirtualBox,
reported as such by upstream (see
https://download.virtualbox.org/virtualbox/LATEST-STABLE.TXT)
Change-Id: Ieb4b158344b3e4d0bf2719e8897cdfcdf133082b
Instead of runtime compiling during the installation compile this lib
in package building and deliver as part of the package.
Change-Id: Ic97adb0c958c57976ac5d23974b0efc306ccb326
This is the same solution for a similar problem and that it was implemented in
5ab1f5418a, but extending it to other parts of the
code that did not wait-and-retry.
Apparently commands like "blockdev --flushbufs" (flush buffers of block devices)
do not solve the situation in all cases, so this is a more foolproof --if
inelegant-- solution that should not slow down the deployment more than a few
seconds at most.
Change-Id: If74e134262475ab0b100981f94fa310536f0a7ab
In newer systems it can be under ngcp-data partition (/ngcp-data/home/sipwise),
in older systems without this partition it can be under /var/sipwise.
Also this way is more future-proof, if the location changes again.
Change-Id: If2d3a3b55ea81871071bf846c8ca981e703d3d88
HOSTNAME env variable is not set from 'ip=' but seems to be set
outside of deployment.sh script and exported to it. HOSTNAME
variable is set differently if newer grml20181230 is set in
dnsmasq dhcp.conf. We have 'ip=' option where we explicitly set
hostname for the host so let's use it (instead of uncontrolled
HOSTNAME variable) for puppet installation case.
Change-Id: I3fa2cc7ec982b270302d2d0940d6477b666eaf5c
GRML 2018.12 adds 'iface lo inet dhcp' line to /etc/network/interfaces
which is used in stretch system. This line breaks networking service on
boot so it isn't properly restarted in system_restart_network function
so network configuration is not complete.
Change-Id: I5e2ec763fea7db6f605e87b171514a985b0de621
This part is the installation of the packages in GRML system which is
testing/buster now so we need to use this debian name in source list.
Change-Id: I417065021bb08b704bf614181f68187705e09f8b
We seem to be hitting https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=918590
which causes installations to take "forever" as soon as LVM is involved, spitting
messages like:
| WARNING: Device /dev/loop0 not initialized in udev database even after waiting 10000000 microseconds.
grml-scripts >=2.8.4 + grml-debootstrap >=0.86 provide workarounds for this,
so when we're installing Debian/buster then make sure we have
recent versions of grml-scripts (providing grml-chroot tool)
and grml-debootstrap available.
Change-Id: I07409790989445a0a30b7373a37bee3bda47ae89
The option INTERNAL_NETMASK is set to either passed parameter
ngcpnetmask or to DEFAULT_INTERNAL_NETMASK value so there is no sense to
pass this variable to ngcp-initial-configuration.
Change-Id: I9d5f4aa72534286b7a2db9d28db42435eaf49fb3
This reverts commit 805bfcbabb.
Debian's virtualbox 5.2.18-dfsg-2 (which provides a working
virtualbox-guest-dkms) included a fix to build against kernel 4.18,
nowadays virtualbox 5.2.22-dfsg-2 is available in Debian/testing
(AKA upcoming buster) repository.
Change-Id: Id8ee9d5d646a46275447e0e5ea4e9f7d14588962
This is an ongoing discussion within the Debian project, see
https://wiki.debian.org/UsrMerge. As far as we know it won't give *us*
any benefits, while it's a one-way decision - so once we'd have a merged
/usr we can't back. We decided to force disabling it therefore.
While at it kill the unused $EXTRA_DEBOOTSTRAP_OPTS variable, which is
not present anymore. I tracked this down to a change in netscript.git
back in 2011 (see commit 6ec9cb274), where we used this for assigning it
to '--pre-scripts /etc/debootstrap/pre-scripts --keep_src_list'. This is
no longer existing as such, so drop this unused variable.
JFTR, debootstrap lists the `--merged-usr` option in its `--help` output
starting with debootstrap v1.0.89, though with v1.0.102 it was changed
to `--no-merged-usr`. We could either check for that or use `grep -q --
--no-merged-usr /usr/sbin/debootstrap`. But the option is available
since debootstrap v1.0.83 and in Debian/stretch we have debootstrap
v1.0.89 available, so enable the option by default.
Change-Id: I3e115afbb9095b0f36d45e8b69e7aeb89e6e7dbe
In some cases ICMP is blocked on a gateway.
To get connectivity status (if the server is online) ping several
servers:
1) gateway
2) deb.sipwise.com
3) dns server or 1.1.1.1
Based on the pull request
https://github.com/sipwise/deployment-iso/pull/1/
Thanks: Isaac McDonald for the PR.
Change-Id: Iebb31de8105e3b329bd0b8fc068abc9b55326ed8
In order to detect if it is necessary to run init or join actions
during the initial configuration JOIN_CLUSTER option is used so
there is no sense to pass RETRIEVE_MGMT_CONFIG to it.
Change-Id: I89c0bf9a96511b2de5e1bf53acd0f631a32f7429
Make it enabled by default.
This option is needed only for initial configuration tools so
we can safely change it here.
Since we use the same way of installation as Carrier for Pro we
need this option in Pro config also.
Change-Id: Id72cb92c2b808143c9380dc23160574061c6c225
For Pro installation from another node we use internal network for
installation and we do not need to configure gateway on internal interface.
Change-Id: I8e95851c1b2132728e8846cf0c1d0d5149d06d74
In interactive mode user choose an interface and its number is used
in the code. In function 'configiface' the actual interface is taken
from NETDEVICES by this number.
In non-interactive mode we have an interface name and it was used as
index in NETDEVICES which failed and the 1st interface was always taken.
It does not matter what interface was passed in NET_DEV variable script
tried to configure eth0.
Fix it with getting interface number by its name.
Change-Id: I1a31c6f4a068ae75c20978e33b77f55f3c0aed76
Updating to Virtualbox 5.2.18 is a requirement but not enough
yet to build against Debian/buster, as building its vboxsf is
failing:
| /tmp/vbox.0/utils.c: In function ‘sf_init_inode’:
| /tmp/vbox.0/utils.c:165:28: error: passing argument 1 of ‘sf_ftime_from_timespec’ from incompatible pointer type [-Werror=incompatible-pointer-types]
| sf_ftime_from_timespec(&inode->i_atime, &info->AccessTime);
| ^
| /tmp/vbox.0/utils.c:53:13: note: expected ‘struct timespec *’ but argument is of type ‘struct timespec64 *’
Debian's virtualbox 5.2.18-dfsg-2 (which provides a working
virtualbox-guest-dkms) includes a fix to build against kernel 4.18,
see:
https://tracker.debian.org/news/983571/accepted-virtualbox-5218-dfsg-2-source-into-unstable/
We don't use Debian's virtualbox-guest-dkms though. Instead let's
patch the sources until upstream provides an ISO which includes the
according change for compiling vboxsf against more recent kernel
versions.
I adapted the VirtualBox-kernel-4.18.patch, based on the relevant
change from Debian::
b00c7b4d53/debian/patches/kernel-4.18.patch
NOTE: We need to patch the source after it was installed by
`/media/cdrom/VBoxLinuxAdditions.run --nox11`. If we'd invoke
`/media/cdrom/VBoxLinuxAdditions.run` again it would overwrite our
modified sources. Instead patch the source and directly invoke the
relevant steps to compile and install the module.
Change-Id: Iea86c07009838dea5b42af91cbfb0dc233179533
5.2.18 is the latest stable version from VirtualBox,
reported as such by upstream (see
https://download.virtualbox.org/virtualbox/LATEST-STABLE.TXT)
as well as present in current stretch-backports.
Change-Id: Iff5cb0e8ada2419aebbd8b26a507d09d8537ae92
By default we do assumption that we are installing to standard block devices,
this is not working for NVMe hardware name spaces.
Here we try to detect underlying device for named partition.
Change-Id: I7d2ea339a3aee2a8458a72cfc392441721d350c7
We perform all packages installation on 'root' partition.
Some packages, like faxserver for now, want to create
something on 'data' partition. They do it successfully in
Debian maintainer files (like postinst), so the 'data'
partition has to be mounted properly, otherwise the data
will be gone on reboot.
Also we need to unmount it properly at the end of installation.
P.S. we need to call mount under grml-chroot as chroot fails:
> (netscript.grml:1353): main(): chroot /mnt mount /ngcp-data
> mount: /ngcp-data: mount failed: Unknown error -1
Change-Id: Ief90d380d71ea6e0a64eba45f403465989865952
Starting with the mr6.5 release we no longer set up swap partition/LV via
deployment.sh (see deployment-iso commit 6a92f155).
Instead, the system will now use swapfiles, which are easier for operations like
changing the size dynamically or removing them completely.
This change implements the configuration for a swapfile during deployment,
enabled by default and with 1/2 the size of the main memory, limited between 4
and 16GB. The file itself will be created later, during the initial NGCP
configuration phase.
Change-Id: I09a5a77ecfed3924184fcc7c84c6b6b21dcacc98
Add menu if Software RAID should be configured and which disks
should be part of it. Only 2 disks allowed.
Move all disk selection code to separate script.
Script will write variables to /tmp/disk_selection file to pass
it to main.sh. It is necessary because 'dialog' is used for
interactivity so we can not capture output of script.
Change-Id: I3dd772c9a6ac6db6809688ff38f242271d9551a8
NGCP software is fully functional without '/ngcp-fallback',
the boot process should NOT be aborted on the errors here.
'/ngcp-fallback' is necessary for better usability of the system
and during upgrade where we will check mount point in RW state anyway.
Also tune layout here to match with spaces count to '/ngcp-data'.
Change-Id: Ic63f87e38ee5e8582060e9ca515492274ba8c55b
It is necessary to test Carrier installations on real hardware,
where we have partitions/RAID configs from the previous tests.
Change-Id: Ifec5ab28ee365737775ef883d152049bc5733b2b
We need to store FS size into config.yml to be able to tune iPXE setup
to install second+ Carrier node on internal Proxmox-based test
Carrier installation (as Jenkins control web01a node there only).
Change-Id: I03db9702e540b7e90b564645e9933316c08f3b59
We need an ability to install all our products in a limited
HW environment and also be able to pack them into different boxes
(like Vagrant, VirtualBox, VmWare, Docker).
The new boot option 'fallbackfssize' gives us ability to manage
the virtual VMs 'fallback' partition size.
Also mount '/ngcp-fallback' in read-only mode by default,
it is very useful to have access to the old 'code' partition.
Rename '/ngcpdata' to '/ngcp-data' for better readability and
matching to 'ngcp-fallback' ('ngcpfallback' is hardly readable).
Change-Id: I512d65254d2f163482734d94cfc36190fb297d8e
Otherwise PRO/Carrier cannot be installed because NGCP
depends on a lot of huge packages including:
* NGCP CloudPBX firmwares ~1Gb ad for mr6.5 and growing
* GRML squashfs ~300Mb
* Sipwise prompts ~400Mb
At the moment of installing all the packages are downloaded
into apt cache (~2.9Gb in total as for mr6.5) + have to be
installed/unpacked (~3.2Gb in total as for mr6.5).
All-to-all I was not able to install Carrier web01a node
even with ROOTFS_SIZE=6Gb. I had to increase it till '7Gb'
to install Carrier successfully.
Lets use 10GB for root FS to have free space protection here.
The possible options were checked:
* move apt cache into RAM, has some disadvantages, like
we need 4GB+ RAM for cache only => hard to test on Jenkins
* move apt cache to 'data' partition is also bad idea as
it is a 'code' cache so have to be consistent with 'code'
(so it is release specific and must be stored on 'root'/'fallback')
Change-Id: Ia34dcc15230e152e894e813dfbaa8688a5145de1
Mcollective is deprecated in v6.0 and I don't see
any quick fix for that. So let's keep using v5.5
while deciding how to proceed further.
Change-Id: Ie72e5a57afc0a7235f9d7d29d52af7d796db1ea8
The users can assign disks using the full path:
'swraiddisk1=/dev/sda' instead of expected 'swraiddisk1=sda'.
As a result the installation fails with not a user friendly errors.
Let's be more user friendly here and remove initial '/dev/' is available.
Change-Id: I094814433b294ca8f084b0412cb9b67537cba8cf
Assuming we have two disks /dev/sda + /dev/sdb and boot the
system with the (new) boot options "swraiddisk1=sda
swraiddisk2=sdb", then we will get a SW-RAID (Software RAID)
setup which looks like (the /boot/efi is only present with EFI
support and only present for the active partition):
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| |-sda1 8:1 0 1M 0 part
| |-sda2 8:2 0 486M 0 part /boot/efi
| `-sda3 8:3 0 15.5G 0 part
| `-md0 9:0 0 15.5G 0 raid1
| |-ngcp-root 253:0 0 5G 0 lvm /
| |-ngcp-fallback 253:1 0 5G 0 lvm
| `-ngcp-data 253:3 0 5G 0 lvm /ngcpdata
| sdb 8:16 0 16G 0 disk
| |-sdb1 8:17 0 1M 0 part
| |-sdb2 8:18 0 486M 0 part
| `-sdb3 8:19 0 15.5G 0 part
| `-md0 9:0 0 15.5G 0 raid1
| |-ngcp-root 253:0 0 5G 0 lvm /
| |-ngcp-fallback 253:1 0 5G 0 lvm
| `-ngcp-data 253:3 0 5G 0 lvm /ngcpdata
The /dev/md0 resides in the 3rd partition, where we usually have
plain LVM. /dev/md0 is RAID level 1, on top of the two disks.
Notes:
* we hardcode /dev/md0 as raid-device for mdadm/SW-RAID
* existing data on EFI partitions will be re-used (as grml-debootstrap
won't overwrite the partition it if contains a filesystem already)
* neither the BIOS (legacy, being first partition) nor the UEFI
partition (being the second partition) can be part of the SW-RAID
setup (see https://wiki.debian.org/UEFI -> "RAID for the EFI
System Partition"). Therefore we need to make sure, that the BIOS
+ UEFI partitions are kept in sync, or at least provide data as
needed. We will have to handle this during e.g. upgrades +
re-installations.
* we need to invoke grub-install for both disks, otherwise
GRUB is available only from the first disk
* install gdisk package to have sgdisk binary available (missing
on grml-small)
Change-Id: I6ebe2c25326c2fdf9ed8fe2bd7b9bf540c7689fd
Requested features:
* switch from msdos partition table to GPT
* (optional) support for (U)EFI
* use a new LVM partition layout, with option to
rollback/install/upgrade via second rootfs partition
(which is called "fallback" in this implementation)
* drop swap partition
Note:
* we no longer support msdos partition tables but GPT only,
as (U)EFI systems can only boot from GPT (and not from
BIOS/legacy boot), while BIOS systems can also boot from GPT
* https://wiki.archlinux.org/index.php/Partitioning +
https://wiki.archlinux.org/index.php/GRUB provide a decent
overview, if you're not familiar with BIOS/GPT/(U)EFI
New partition layout:
* 1st partition: 1M BIOS boot, for BIOS/GPT (legacy) boot
- this allows fallback to grub-pc package
(needs to be partition type GUID 21686148-6449-6E6F-744E-656564454649
AKA set to bios_grub with parted, then installing grub to disk
so it properly embeds core.img, can be done from a
rescue/live system)
* 2nd partition: ~500MB EFI System, for UEFI/GPT boot
- used as /boot/efi, iff EFI support is available
* 3rd partition: LVM with:
- /dev/mapper/ngcp-root with 5GB (rootfs target) +
/dev/mapper/ngcp-fallback with 5GB (for rollback/install/upgrade)
- 10% or >=500MB unassigned of the remaining space (whatever is bigger)
- /dev/mapper/ngcp-data data partition with rest of
available disk space, used as /ngcpdata on system
Old layout with a 16GB disk for NGCP systems:
| # lsblk
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| └─sda1 8:1 0 16G 0 part
| ├─ngcp-root 254:0 0 14.3G 0 lvm /
| └─ngcp-swap 254:1 0 1004M 0 lvm [SWAP]
| # pvs
| PV VG Fmt Attr PSize PFree
| /dev/sda1 ngcp lvm2 a-- 16.00g 764.00m
| # vgs
| VG #PV #LV #SN Attr VSize VFree
| ngcp 1 2 0 wz--n- 16.00g 764.00m
| # lvs
| LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
| root ngcp -wi-ao---- 14.27g
| swap ngcp -wi-ao---- 1004.00m
New layout with a 16GB disk (+ EFI support) for NGCP systems:
| # lsblk
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
| sda 8:0 0 16G 0 disk
| |-sda1 8:1 0 1M 0 part
| |-sda2 8:2 0 486M 0 part /boot/efi
| `-sda3 8:3 0 15.5G 0 part
| |-ngcp-root 254:0 0 5G 0 lvm /
| |-ngcp-fallback 254:1 0 5G 0 lvm
| `-ngcp-data 254:2 0 5G 0 lvm /ngcpdata
| # pvs
| PV VG Fmt Attr PSize PFree
| /dev/sda3 ngcp lvm2 a-- 15.52g 564.00m
| # vgs
| VG #PV #LV #SN Attr VSize VFree
| ngcp 1 3 0 wz--n- 15.52g 564.00m
| # lvs
| LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
| data ngcp -wi-ao---- 4.97g
| fallback ngcp -wi-a----- 5.00g
| root ngcp -wi-ao---- 5.00g
Change-Id: I39206b225edb25fd1e27de49a818ad7a54532f82
We have confusing situation for an ages here.
We were writing a lot of log files:
> /var/log/deployment-installer-debug.log
> /var/log/ngcp-installer.log
> /var/log/ngcp-installer-debug.log
The first one contains all the GRML boot, Debian debootstrap
and ngcp-installer-debug.log already. Also having both
ngcp-installer.log and ngcp-installer-debug.log confuses
new users as they do not know which logs they need to
check first to see ngcp-installer logs.
There is no need to keep both files anymore.
Nowadays we have a nice way to distinguish logs between components,
so /var/log/deployment-installer-debug.log is enough here:
> +15:07:48 (netscript.grml:1403): main(): echo 'Generating ngcp-installer run script ...'
> +15:07:48 (netscript.grml:1404): main(): cat
> +15:07:48 (netscript.grml:1417): main(): echo 'Execute ngcp-installer inside deployment chroot environment ...'
> ...
> Running ngcp-installer via grml-chroot.
> ...
> +17:07:48 (ngcp-installer:27): main(): . /usr/share/ngcp-installer//system_pro.inc
> +17:07:48 (ngcp-installer:28): main(): . /usr/share/ngcp-installer//packages.inc
> +17:07:48 (ngcp-installer:31): main(): . /usr/share/ngcp-installer//cfg.inc
> ...
> +17:07:57 (packages.inc:58): packages_ppa_sourceslist(): echo '(system installed using NGCP PPA: gerrit_gtid_imp)'
> +17:07:57 (ngcp-installer:54): main(): packages_aptconfig
> +17:07:57 (packages.inc:90): packages_aptconfig(): mkdir -p /etc/apt/preferences.d
> +17:07:57 (packages.inc:92): packages_aptconfig(): cat
The commit here should minimize amount of log files we
produce and should simplify the debug process.
Change-Id: I26a2ca8e21a3385f7d57d830feb207bbb1039b39
We do not use /tmp/ngcp-installer.log since mr6.5 and
using directly /var/log/ngcp-installer.log inside chroot.
Change-Id: I35b098fb09046a0fbc0a9bfc56bd125362c34fd0
We're executing `apt-get update` even when nothing needs to be
installed. This adds unnecessary overhead and can fill ram/disk
on live system on consecutive runs, so let's avoid that.
Change-Id: Ic1a40d82988caf454f317b4803e03f4c9de3feea
Implement suggestions from review I16e4f19f3b9270ccfef6c7c1274bb8b8d95695bd
1. In some cases there can be multiple gateways. Return only the 1st one.
2. Remove configuration of the management (IMM2/IPMI) interface of
lenovo servers.
Change-Id: Ia1bb6e9f0e44bcb0c6be87407a8c11899163c03a
All non-zero returned values are caught by '-e' option and passed
to function wait_exit by 'trap'.
So if it is an error we can set 'error' status in this function.
Change-Id: I762e5470d94a537b4f8f0e32fe6d0c1575bbb075
We need it to be able to connect from mgmt node to installed one
to run 'ngcp-initial-configuration'.
Change-Id: Id907094ba2eae32c05ad8d42d8a504e498a57ad4
wget in Debian/buster:
| Package: wget
| Version: 1.19.5-1
| [...]
| Priority: standard
wget in Debian/stretch and earlier:
| Package: wget
| Version: 1.18-5+deb9u2
| [...]
| Priority: important
To ensure we still have wget available inside the
Debian system, e.g. for retrieving the ngcp-installer*.deb
file, we need to add it to the package selection.
The ngcp-nth-* packages also include wget in their dependencies,
ngcp-installer also checks and adds it, though it's too late
there for bootstrapping our systems, as we're e.g. retrieving the
installer deb + checking for PPAs using wget.
While at it move the slightly misleading comment to the
appropriate place.
Change-Id: I971b2d06528a33b1dc4c310feef83c32e10ae557
Switch from:
> +10:43:04 (/usr/sbin/ngcp-status:10): main(): export LC_ALL=C.UTF-8
to:
> +10:43:04 (ngcp-status:10): main(): export LC_ALL=C.UTF-8
Since we have unique names for all the script we use,
the full path is not so important as the space we spent of the console.
Change-Id: Ie6fe2ce1ca409033189e1c033a4aebf9e037eaba
1706:9 shellcheck:warning When used with -p, -m only applies to the deepest directory. [SC2174]
1726:9 shellcheck:warning When used with -p, -m only applies to the deepest directory. [SC2174]
Change-Id: I98efba1253c0aefd9e97ac34500476936b860ca5
257:9 shellcheck:error Arrays implicitly concatenate in [[ ]]. Use a loop (or explicit * instead of @). [SC2199]
Change-Id: I49504d5b1a8d8ae021d4d6c6465639d58a2670e3
If the installation device is virtual it is necessary to redefine
DEFAULT_INSTALL_DEV variable to the name of real interface but
save the origin name to stop it.
Change-Id: I9e59ca8411e713235a0e0e5407ef0c3acc30e299
This list is used to add all network interfaces to network.yml but
we do not need to add virtual interfaces there - they will be added
automatically from installation's configuration.
Change-Id: I3f21abc066b4f0c714b1780dcfdf4335e7506d1d
The deployment.sh script as well as installer should not configure system
or services.
Remove network's configuration code.
Remove services' configuration code.
Remove unused variables and parameters.
Copy existing /e/n/i into chroot env so the server starts with the same
network configuration.
Move gateway detection to the top - we need this information to generate
proper /e/n/i
Change-Id: I16e4f19f3b9270ccfef6c7c1274bb8b8d95695bd
1. 'localhost' has to be resolved to ipv4 only not ipv6.
2. There should be 'hostname.' with dot record.
Change-Id: I9f34f45b00453746b254f41494d7652a10de0d04
Without explicit return function returns the exit code of the last function
which in not 0 in this case so it causes amiss exit of the script.
Change-Id: Ic7f1f534048cb051fe116cb76ee386bf4ca07ff4
If 'systemctl reboot|halt' is called within script 'trap' function
is not called so deployment.sh does not wait and jenkins can miss
the status.
Move waiting to separate function and call it directly so there is
waiting in the end of deployment.sh in any case.
Change-Id: Ib93bf6fd4e6f133c9342d02d412ea1f18cdceba6
For now if there is an error in deployment.sh it is ended immediately
so jenkins node can miss deploy status.
Add wait_exit function which is called on any exit and waits
STATUS_WAIT seconds.
Change-Id: I715c2484a25222221d61a01bd220973e89547929
We are NOT installing non-free firmware packages on NGCP (since TT#42672),
as a result we should not purge them. See commit 9161d0f57 in
metapackages.git for more information.
Change-Id: I84ff3486c0b03bbc228e5332955f6654b3519137
The hardcoded list of accepted drives is never really up2date
and was meant as a safety net in the beginnings of deployment.sh.
Let's drop those checks to avoid failures in automated installations,
interactive installer/deployment mode is supposed to prompt for
before actually executing anything anyway.
Change-Id: Ib6dcaec8a221dc268638a98055219a5ae3c92a00
deployment.sh can install plain Debian systems (without NGCP),
in commit d2afaf9 such functionality has been lost. Fixing it here.
Change-Id: I883ef3c69c67cc04ae796c373f3b52d75780b4de
Currently grml-debootstrap only sets /etc/localtime, but not
/etc/timezone, so this confuses the system with libraries/tools
that read the former and others the latter. Reconfigure tzdata so
that it can get both files into a consistent state.
Change-Id: Id34c1a088809fbac19957a750c31fb8f3ceda893
Fix endless cycle in case of multiple interfaces and non interctive
mode.
Add exit if interface was configured.
Change-Id: I8ddb00f9b4dbcd997bb6fc95e100d7db6f8870b2
It is necessary to use separate apt database and source list because
non management node has no internet access so is installed from
management node so these additional packages have to be
installed/accessible from sipwise repo.
Change-Id: I4c1310320c4d66aba01af5ce00127f4757ec71ad
Make a single list of all required packages.
For each package from the list check if it is installed or not.
Install all packages at once.
Remove custom source.list for these additional packages. Previously it
was used because GRML image and its default source.list were too old.
Now a new GRML is used so we have a vise-versa situation - deb.sipwise.com
contains older packages than default source.list so there is no sense to
use any custom locations and options.
Change-Id: I33239c255f5af666ca9bbead9d3a584de9335bb6
Network device and parameters can be passed via environment variables
so script will create the network configuration without asking for
parameters.
Limitations - wlan and vlan configurations are not supported.
The list of parameters:
NET_DEV - name of network interface to configure.
METHOD - method of obtaining IP address. Can be dhcp|static|manual.
If both of these parameter defined then non-interactive mode is
used.
IPADDR - IP address for network interface.
NETMASK - netmask for network interface.
If method is 'static' or 'manual' then these parameters are mandatory.
GATEWAY - address of default route.
DNS - comma separated list of dns servers.
Add '--force' key to ifdown command before configuration of the interface.
It is required for interface reconfiguration.
Change-Id: I267d5e5ae1a6fb9c4a6234af582165c6c5fa5dc5
This code is called only once so there is no sense to make it as function.
Optimize the code - use 'sort' instead of 'for' cycle to remove duplicates.
Change-Id: I2658b2fbbb3a9478e11337327d8a76a259f22c26
Our goal is to separate installation and configuration parts so
any network related things have to be configured after the installation
and reboot. But during the deployment.sh we need to configure one
interface which will be used after the reboot. So we need consistent
interfaces' name between reboot.
Change-Id: Ib4cd19bf0455c1365a2d33fc5a90209d0a5d62d7
Several changes:
- Promote the variables to top level. Local variables are good for some things,
but if they represent external resources, it's probably better to have them
defined outside as a kind of constants, instead of deeply buried inside
functions.
- Make 'wget' a bit more robust and less verbose (to not have dozens of lines of
useless output with the progress of a big file).
Change-Id: I679de4e976497d0cf6ca3da2e13ff2fb8a76d881
The variables related to Pro edition (at least Role) needs to be present when
installing some nodes Carrier, otherwise the installation doesn't work as
expected.
Change-Id: Ibea819889f42559626164b9e344670edea5cad1b
If the dir is owned by "_apt", apt-get can use this user for network operations
instead of having to stay as "root".
Change-Id: Ib41d591d1da539c3d43fa945262d0b9766a6be23
Add choice continue or not if virtualized system was not detected and
it is interactive mode.
If it is non interactive mode - continue the deployment.
Change-Id: I8913dda6121c0e46a6de21ad1ab95eceb2c45227
The file /tmp/ngcp-installer-cmdline.log is available when GRML is running only
and will gone right after the reboot. In the same time original
"${TARGET}/etc/ngcp-installer/config_deploy.inc" is also available
during GRML is running and will stay after the reboot.
For all the time I had no reason to check /tmp/ngcp-installer-cmdline.log,
as I always used config_deploy.inc directly (which is also covered by etckeeper).
Removing it as a part of minimizing log files program.
Change-Id: Idcc2ced1bf2aba261bd519abcdb91c1f8fe0336b
* ngcp-installer*deb can be deleted right after the package installation
* ${TARGET}/config_*yml and ${TARGET}/constants_*.yml are leftovers.
They are no longer available on NGCP (long long time ago we created
them to upload CFG schema dump into git every night).
See commit ba06d2b145e3e20468de532e77fa501cd38a72e7 to netscript.git
Change-Id: I75a575233a5074143d7d35de26b33f161cf77e4f
We have a lot of different log files on GRML/deployment.sh stage,
let's minimize the count here and use main $INSTALL_LOG for all
the messages printed by deployment.sh (it will simpify further debug).
Previously logit() printed to GRML /var/log/syslog which is not
available on NGCP after reboot, the new code will use $INSTALL_LOG.
Change-Id: I675c45ebc2852b00253e7b51e7abd4a10a0fb714
First of all the information here was outdated and confusing.
Moreover it duplicates info in deployment-installer-debug.log.
The line "NGCP_INSTALLER_CMDLINE=..." didn't represent the actual
running command for several years already. Also the line SCRIPT_VERSION
is already printed to all log files several times already.
The only missing line was CMD_LINE, so printing it if debug is enabled.
Change-Id: I19255354c8314201992385f5b5bb7e60910bc26a
This is an old option we do not use anymore.
The option 'ngcpppa' is more flexible and powerful nowadays.
Change-Id: I829189a9e92ee352daa4f826dd3720723c6df3e2
Those firmwares have been moved to NGCP metapackages to ensure
they are always available.
Also Puppet server ensures they are available on
internal infrastructure.
Change-Id: I88cd1d7592ec5290c10f05b1f202114f8c86299a
If NGCP PPA is defined and does NOT contain ngcp-installer package,
we should use the default (release and product specific) ngcp-installer.
Change-Id: Iac0863698e174e333108a0d01bf687c3de85ef99
ngcp-installer installs it already also puppet depends on it.
Also metapackages have dependency on it now.
Actually it looks like Debian stretch debootstrap installs
apt-transport-https out of the box which recommends 'ca-certificates'
as a result both are installed after the debootstrap.
Change-Id: I84fa358b61aed25594a38c21efaec27ba96d6918
NGCP metapackages depends on 'ssh' already. So no need to
install it during the debootstrap process. So far
deployment.sh is being used to recover from Puppet
(ssh module cares about 'openssh-server/-client already)
and also for Debian plain installation where we still need
to install 'openssh-server' manually.
Change-Id: If9c052f0517b1610f0e65ed4a574539b23e23928
The package 'gettext-base' is actually currenlty installed as a dependency:
> root@web01a:~# aptitude why gettext-base
> i grub-pc Depends grub-common (= 2.02~beta3-5)
> i A grub-common Depends gettext-base
Change-Id: If8224f0e67eb1ccdd38ee6523e3a20424e6fe2c3
Originally the code has been added back in 2014 with commit message:
> Support blade disks we have on our internal proxmox cluster
So far at the moment the package 'firmware-qlogic' is not installed on
internal Proxmox blades (as I can see). I would consider it as
unnecessary any more and remove it here. Also it is a bit dangerous to
install some packages on internal test servers only, as production might
behave the different way then. So good to remove it here.
Change-Id: I9817ca0904eade6c51479c72f4cff428a044896f
The package 'apt-transport-https' is available on freshly debootstrapped
Debian stretch already, also NGCP depends on it via ngcp-nth-*.
Additionaly ngcp-installer check 'apt-transport-https' availability in 'check.inc'.
Moreover we will have to remove it here after migration to Debian buster,
as 'apt-transport-https' has been merged into 'apt' there.
Change-Id: I263e9010093096d7b50bb92ae39a996dc4838755
If the systemd status is 'degraded' the exit code of the command
is not 0 so script is interrupted due to '-e' option.
Change-Id: I147da71569dc527004d598c48338fc04dd2ada92
All the services which are run within chroot are stopped when they
are not needed anymore. So it makes no sense trying to kill them
by list moreover the list is outdated.
Here is the ps output before and after these kills:
https://paste.mgm.sipwise.com/show/4107/
there is no services from chroot.
Add debug output of 'systemctl status' if someone uses mountpoint
$TARGET.
Change-Id: I90f0c18157040929aaa52ec572babea1b0bf2eb1
File /etc/network/interfaces is generated by ngcpcfg so deployment.sh
should not modify it further.
Remove unused variables.
Change-Id: I287b5e5bf32834309dc09efab637bf083f479d4c
Remove LVM variable and related code as deployment.sh is version specific
now and in mr6.4+ we definitely use ony LVM.
Change-Id: I1adad68a439ee5356d243f937f37197be1055156
Remove outdated checks of jessie release.
Some shellcheck improvements.
Remove PROFILE variable as it does not exist/used.
Remove outdated comments.
Change-Id: Ie77e9477f0b38019c2a62465426509cdfe6e309a
Collect the list of running process before and after kills in the end
of deployment to catch if we do not stop them properly.
Change-Id: Ie01853723d053a1ef198e47b89566bd4b9a93bbc
Use systemctl (reboot|poweroff) instead of sysrq triggers, we do not need
this workaround in a new grml.
Change-Id: I27d03a402e96e08c902f18eef91ae2144fe128fb
When all variables (DEBIAN|SIPWISE)_REPO_HOST, (DEBIAN|SIPWISE)_REPO_TRANSPORT
are set create and use (DEBIAN|SIPWISE)_URL variables to simplify and unify
their usage.
Change-Id: I2b4578efad0d285a5e5519cad8e5f78b870073a2
Remove workarounds which were used to install old debian releases/ngcp releases.
Remove old versions of Debian from installation list.
Add trailing space to the end of boot parameters to make manual modification
easier.
Change-Id: I201dac4a1d546a83743823cb9dff3546209e6404
Function checkBootParam is fuzzy about search pattern so it
returns false positive result on similar parameters, like 'ngcpppa'
and 'ngcpppainstaller'. It leds to incorrect call of getBootParam
on nonexistent parameter so it returns 1 which causes error exit
of deployment.sh due to 'set -e'.
Add '=' symbol to the end of search pattern for parameters with argument.
Change-Id: I636a66b3e74a395de33d4bdfa9b31eb07fbe03d0
Exporting HOSTNAME inside the loading 'boot parameters' section
causes the following issue (inside GRML chroot):
> (sp1)root@sp1:/# hostname --fqdn
> hostname: Name or service not known
NOTE: it happens if ngcphostname is not defined.
We should separate loading options and using them to
have clear system behaviour, so we move the export code forward.
In this case TARGET_HOSTNAME will be defined from options ngcpsp1/ngcpsp2
and 'export HOSTNAME="$TARGET_HOSTNAME"' will do the job even if
ngcphostname is not defined.
We should consider deprecating/removing 'ngcphostname' completely
if possible, as currently the option description is a bit confusing:
> ngcphostname=... - hostname of installed system (defaults to ngcp/sp[1,2])
> NOTE: do NOT use when installing Pro Edition!
(which says we should not use it on PRO, while PRO installation
cannot happens without ngcphostname option).
AFAIK, it is necessary for Carrier only, to define Carrier host name, like web01a.
Change-Id: Idba31fd8286d41bd0c1aaf7ee762371e6b1d50d5
Stop of mysql was added in the end of ngcp-installer in
installer.git:80ce05d706ed28e906d6b171498f3d2d9f55c84e
but mysql server is required for ngcp-prepare-translations so add
start/stop for this case.
Change-Id: Id0e030533240718517721d12647766eb283607bf
Add option -e to deplayment.sh to catch the installation's errors.
Run deployment.sh as a script not via external interpreter
'bash <name>'.
Change-Id: I320b535f41f08937c4924c603f8e1a17b23e80e8
Check dhclient process as well as boot parameter to decide if
dhcp should be used for external interface.
Change-Id: I5f92cca5a513edcacb99318b7b59d33c1fd55ef5
All used services in installer have to be manually stopped when
they aren't needed so we do not need such a stop cycle.
Moreover services with existing init.d script only can be stopped
in this way. Other services just produce error message:
Stopping <service name> ...
<service name>: unrecognized service
Change-Id: I9aec0b8aa7c4878e538532b9d8997e2438a1277f
fai-setup-storage has no real advantage over plain parted
usage for us nowadays, instead we can avoid depending on
it by using the plain tools.
For ngcp systems reproduce the same setup, for plain Debian
installations use a static size of 8GB for rootfs. This
makes installations more flexible for us.
Change-Id: I5e03ad0a5dd3dac7cc547b8cdc64277db4af4c22
We don't have einfo + eend available in deployment.sh as
we don't source lsb-functions
Change-Id: Ie07d67d105ec2bcb8849c7b04d3b99c7580d6de5
Thanks: Mykola Malkov for spotting this bug
The message `Started Cleanup of Temporary Directories` shows up
during deployment, caused by systemd-tmpfiles-clean.timer which
originally looks like:
[Timer]
OnBootSec=15min
OnUnitActiveSec=1d
To avoid this message to appear during deployment stage let's
disable the systemd timer by masking it.
NOTE: we might disable this timer on default Grml, but this
needs to be discussed yet in further detail and we need
a faster workaround anyway.
Thanks: Guillem Jover <gjover@sipwise.com> for the suggestion to just mask it
Change-Id: Ic85dc58322280f4eaece61af220ae00c64c74e52
There is no any source of mount info (mtab or /proc fs) so this
call is useless. Moreover due to absence of mount info it can catch
segfault:
117e8f0763
Change-Id: I0defcc8e6cb25c3534fef1f0281291c07e02f2d5
deployment.sh is release specific and GRML in trunk
provides grml-debootstrap version 0.81 already.
Change-Id: I193cb8412afb6a8984d5e8263fdaa8417aabbc14
Git is necessary for nightly puppet recovery tests while
it is missing on 'small' GRML we are currently using.
Also fix outdated debug message as it is confusing.
Change-Id: Ie4333eda2a7ef260456c63035c8084d921f0d366
There is problem with package 'moreutils' which contains 'ifdata'
so replace it with parsing of 'ip-address' output with 'sed'
Change-Id: Ic4af3e32b5cd36aa8efd706d21805acff7a50459
In 'small' GRML utils nmap, curl, ifdata are missing so replace them
with another ones and install required packages for installcd in
one place.
Change-Id: I7269bb53b66805c792f5e91ff97a2fd2ebd14d1f
We are based on the latest GRML nowadays which is based on Debian sid 2018-11-04.
apt-transport-https has been merged into apt package since version 1.5,
so far there is no need to install it manually, moreover sid contains
dummy transitional package 'apt-transport-https' which cause:
>> apt-get install apt-transport-https
> The following packages will be REMOVED:
> apt apt-utils tasksel tasksel-data
> The following NEW packages will be installed:
> apt-transport-https
> WARNING: The following essential packages will be removed.
> This should NOT be done unless you know exactly what you are doing!
> apt
> 0 upgraded, 1 newly installed, 4 to remove and 0 not upgraded.
> Need to get 171 kB of archives.
> After this operation, 5,342 kB disk space will be freed.
> You are about to do something potentially harmful.
> To continue type in the phrase 'Yes, do as I say!'
The fool protection is always good. Deleting manual apt-transport-https
installation from GRML part of deployment.sh
P.S. there is one more 'apt-transport-https' stay inside debootstrap,
as we use Debian stretch in NGCP currently while apt-transport-https
has been merged in Debian buster+, it should stay for a while.
Change-Id: Ifcf84f3d79ecc44465f97d42bbae12be04d8407e
Michael Prokop
Mykola Malkov
Sergii Kipot
Alexander Lutay
Manuel Montecelo
Volodymyr Fedorov
Guillem Jover