We don't have any other architectures besides the amd64 one on
our own repositories and nowadays with MultiArch people might have
e.g. i386 enabled (via 'dpkg --add-architecture i386'), resulting
in errors like:
| W: Failed to fetch http://deb.sipwise.com/spce/3.0/dists/wheezy/main/binary-i386/Packages [^] 404 Not Found
| W: Failed to fetch http://deb.sipwise.com/wheezy-backports/dists/wheezy-backports/main/binary-i386/Packages [^] 404 Not Found
| E: Some index files failed to download. They have been ignored, or old ones used instead.
Avoid retrieval of i386 Packages files by limiting the deb entry
to the amd64 architecture.
Now having MT#4463 resolved let's also get rid of "Applying
Vagrant performance optimisations for VM" steps from
daily-build-vagrant.
Implemented as separate boot option (independent from "vagrant"
boot option) to apply it only for the VM 2XX builds (to e.g. not
execute it on plain Debian systems).
gcc is already present in ngcp systems and therefore the
recommended package libc6-dev won't be pulled in anymore, the
uname fake code depends on stdio.h etc though
Virtualbox Guest Addition installation fails inside chroot
with different kernel version of live system vs installed one.
Sadly there doesn't seem to be a way to reliably control kernel
version/header location forVBoxLinuxAdditions.run (e.g. via
KERN_DIR=... as advertised) because there are several calls
to `uname -r` in use. :(
I sometimes get a failing network setup in Vagrant's ce-trunk VM
where the default route is missing then:
| # ip r
| 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
| 192.168.88.0/24 dev eth1 proto kernel scope link src 192.168.88.234
|
| # cat /etc/network/interfaces
| # This file describes the network interfaces available on your system
| # and how to activate them. For more information, see interfaces(5).
| # The loopback network interface
| auto lo
| iface lo inet loopback
|
| # The primary network interface
| allow-hotplug eth0
| iface eth0 inet dhcp
| #VAGRANT-BEGIN
| # The contents below are automatically generated by Vagrant. Do not modify.
| auto eth1
| iface eth1 inet dhcp
| post-up route del default dev $IFACE
| #VAGRANT-END
Of course the default route comes back on a manual refresh of eth0:
| # ifdown eth0
| # ifup eth0
| # ip r
| default via 10.0.2.2 dev eth0
| 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
| 192.168.88.0/24 dev eth1 proto kernel scope link src 192.168.88.234
The culprit seems to be the allow-hotplug where the device is handled
different in the startup process:
http://screeni.s3.amazonaws.com/screenshot.2013-09-17T11:43:37.png
The ordering is important because of Vagrant's
"post-up route del default dev $IFACE". While there's a workaround
available within Vagrant:
73c8299ecd
I don't think that's the proper way. Since we're using "auto"
everywhere else in our setups anyway let's make this change,
hopefully also fixing the Vagrant network issue.
ngcp-services-pro depends on linux-headers-3.2.0-4-all,
which depends on linux-headers-3.2.0-4-all-amd64,
which depends on linux-headers-3.2.0-4-amd64.
linux-headers-3.2.0-4-amd64 is installed already in version
3.2.46-1+deb7u1 on our installation, being the current version
from security.debian.org. Because this package depends on all the
other packages being in version 3.2.46-1+deb7u1 as well this
fails due to our apt-pinning. Our apt-pinning considers
security.debian.org to be at it default pinning level 500 whereas
deb.sipwise.com is pinned to 990. This results in package
dependencies that can't be resolved by using packages from
security.debian.org.
We have to think further regarding the wanted behaviour before we
actually make that MIRROR switch. :(
On Jenkins host there was this symlink present:
| cd /srv/repository
| ln -s . debian
To avoid unnecessary duplicates during repository search
I removed the symlink, let's see whether this change is
enough or if apt-get then still needs it.
dkms in Debian/wheezy fails when directly invoked via chroot(8):
| /usr/sbin/dkms: line 1868: /dev/fd/62: No such file or directory
| /usr/sbin/dkms: line 1799: /dev/fd/62: No such file or directory
and returns with exit code 0. By invoking grml-chroot we get the
according mount binds for devfs etc.
Current daily VMs are known to fail because of switching from
raw image to vmdk on proxmox. :( Abort installation immediately
if actions during setup-storage, parted and mkswap fail.
Otherwise grabbing a screenshot after deployment finished
will include to much noise and not the useful data...
From: Michael Prokop <mprokop@sipwise.com>
Make it obvious which platform releases use which underlying Debian release,
once we have wheezy support the default will be changed and we don't want
to have wrong assignments for existing releases then.
From: Michael Prokop <mprokop@sipwise.com>
The '5%-10G' sadly doesn't work as expected and 5% of the
*disk* size is a bad default, so let's see whether XX% of
RAM size makes a better choice. We might change the actual
percentage value but we need some testing anyway....
From: Michael Prokop <mprokop@sipwise.com>
When using the "ngcplvm" boot option then deployment automatically
sets up LVM (volume group "ngcp" with volumes "root" and "swap"),
otherwise it works just as it used to do so far (AKA no LVM).
Needs performance checks (Kirill volunteered to do that), once
we're happy with the results we can make it the new default
deployment method as planned.
Testing: https://bugtracker.sipwise.com/view.php?id=2465
From: Michael Prokop <mprokop@sipwise.com>
This works on a freshly deployed PRO system:
| ngcpcfg init-mgmt $MGMT_IP
Now let's try to include this step also during deployment.
From: Michael Prokop <mprokop@sipwise.com>
Boot option ngcpcpip2 might differ from default, so we can't hardcode
$DEFAULT_IP2 when deploying sp1. Instead try to autoconfigure it
when we're deploying sp2, maybe we don't need it while deploying
sp1. If this doesn't work then we'd have to provision the interface
using $IP2 in the meanwhile.
While at it try to pull changes from shared storage on sp1 when
deploying sp2 so both nodes are fully up2date after fresh
installation.
From: Michael Prokop <mprokop@sipwise.com>
This should give us a full-fledged network.yml as eth0 of
the 2nd system is the only missing NIC during deployment.
Testing!
From: Michael Prokop <mprokop@sipwise.com>
this *might* fail due to ssh key setup, but we have to avoid
merge conflicts and therefore need to get ngcpcfg pull working...
From: Michael Prokop <mprokop@sipwise.com>
With the way it is the second node gets the SSH keys from the first
node's glusterfs share. Let's try to set up SSH before actually
running ngcp-installer, then in ngcp-installer skip the SSH key
setup if keys already exist (will follow in upcoming svn commit
for ngcp-installer).
From: Michael Prokop <mprokop@sipwise.com>
As supported by ngcpcfg-api as of svn r11744.
If this works as expected then the only:
ssh-keyscan $MANAGEMENT_IP >> ~/.ssh/known_hosts
should be needed for automatic SSH login between nodes in carrier
environment. A working version of "ngcpcfg init_mgmt
$MANAGEMENT_IP" should be quite close then.
Let's see what's the opinion of Jenkins + our autodeploy jobs...
From: Michael Prokop <mprokop@sipwise.com>
Let's see whether this enough to close issue #2367.
Open question: do we want to have locales-all also on systems
we did not deploy on our own (AKA CE systems installed using
ngcp-installer)? Iff so then we'd have to add it as dependency
somewhere.
Testing: https://bugtracker.sipwise.com/view.php?id=2367
From: Michael Prokop <mprokop@sipwise.com>
This reverts commit 2c859c9940caa2599b6eb71f7a1d8832064a6133.
This doesn't fix the issue (as you might guess I know the
real source of the problem now, ha!)
From: Michael Prokop <mprokop@sipwise.com>
network.yml looks good on sp1 now. Something fishy is still
causing broken config files though, a manual 'ngcpcfg build'
fixes that, why the 'ngcpcfg build' in the deployment script
doesn't solve that yet needs to be investigated...
Sadly bootstrapping sp2 is still quite tricky. We might even
have to set initial network.yml on PRO systems to what we ship
with CE systems so we don't run into merge conflicts. Hmpf.
From: Michael Prokop <mprokop@sipwise.com>
Seems to work, at least for sp1 hosts in PRO installs. Let's see
whether PRO deployments work again with new templates,
network.yml and ngcp-network being in place now.
Happy PRO deploying dear Jenkins while I'm switching office->home. :)
From: Michael Prokop <mprokop@sipwise.com>
On Dell PowerEdge R310 servers we have the PERC H700, on the next
generation model R320 we have PERC H710. Don't be too picky
about the whitelist and just check for "PERC" in the model
information and "DELL" as vendor.
From: Michael Prokop <mprokop@sipwise.com>
acpi-support-base in squeeze depends on console-tools (or kbd)[1],
therefore we can't remove console-tools if we want to get
acpi-support-base installed by default
[1] http://packages.debian.org/squeeze/acpi-support-base
From: Michael Prokop <mprokop@sipwise.com>
VMs on Proxmox 2.1-14 (pve-qemu-kvm 1.1-8, 2.6.32-14-pve) have an
annoying problem with our installations, where the deployed
system freezes as soon as some action takes place on the network
stack. For example invoking "lsof -i -n" on the rebooted system
causes such a freeze.
The only solution to work around this issue seems to be stopping
and then restarting the VM. As we need to handle this
automatically we need some kind of API to retrieve system state.
By writing the current deployment state into some file and
providing it through http://$DEPLOYMENT_SYSTEM:4242/status this
should get us there.
From: Michael Prokop <mprokop@sipwise.com>
The udev rules need the real MAC addresses and we can't mess
with them before the rules are in place
Thanks Richard for helping with debugging
From: Michael Prokop <mprokop@sipwise.com>
logit() isn't available inside the subshell in the chroot so don't
use it there, we already have it at the according place...
From: Michael Prokop <mprokop@sipwise.com>
When doing automated pro deployments we also have to stop monit
to be able to unmounted the disk in a clean way...
From: Michael Prokop <mprokop@sipwise.com>
Fix what has been broken in r10823: the "Dump completed on"
is present iff "--skip-comments" option is not used.
From: Michael Prokop <mprokop@sipwise.com>
We write data into the DB in revision scripts, like e.g.:
| $SVN/dev/ngcp/db-schema/trunk$ cat db_scripts/diff/9675.up
| INSERT INTO provisioning.voip_preferences (attribute, type, dom_pref, usr_pref, peer_pref, data_type, max_occur, description) VALUES('mobile_push_enable', 1, 1, 1, 0, 'boolean', 1, 'Send inbound call to Mobile Push server when called subscriber is not registered. This can not be used together with CFNA as call will be then simply forwarded.');
As suggested by Daniel let's also put data into the DB dumps.
From: Michael Prokop <mprokop@sipwise.com>
Might become useful for low-mem VMs once, implemented while
figuring out installing an etch system for Andi :)
From: Michael Prokop <mprokop@sipwise.com>
Starting with recent trunk versions we no longer ship
kamailio init script but just our kamailio-lb + kamailio-proxy.
From: Michael Prokop <mprokop@sipwise.com>
Inside the pool there might be versions which have been released inside a
maintenance branch but which don't cover recent changes in trunk.
This caused current trunk installations to fail with:
| mv: cannot stat `/etc/ngcp-config/config.yml': No such file or directory
So get rid of every file without "svn" in the filename, so e.g.:
ngcp-installer-ce_0.7.2+0~1339173026.svn9034.165_all.deb (trunk version)
is preferred over:
ngcp-installer-ce_0.7.3_all.deb (release into 2.5 repository)
When we're installing trunk we don't care about released versions,
so get rid of them.
Tricky.
From: Michael Prokop <mprokop@sipwise.com>
This avoids the svn commits with the only change being:
| --- Dump completed on 2012-06-06 6:09:22
| +-- Dump completed on 2012-06-06 16:35:16
From: Michael Prokop <mprokop@sipwise.com>
The:
| ERROR 2002 (HY000): Can't connect to local MySQL server through socket ...
gets displayed several times, though it's not an issue.
Seems to be some async foo or race condition I couldn't identify
yet, until it's resolved display a message to the user so nobody
thinks that's a real error.
From: Michael Prokop <mprokop@sipwise.com>
Hopefully this solves our failing daily builds where we get:
| 06:09:57 curl: (28) Operation timed out after 30000 milliseconds with 0 bytes received
like in https://jenkins.mgm.sipwise.com/job/vmbuilder-ce/192/
From: Michael Prokop <mprokop@sipwise.com>
By default we use eth1 for the crosslink / interconnect device,
but there are VMs which use eth0 - so support this in the deployment
process so we get a happy heartbeat service...
From: Michael Prokop <mprokop@sipwise.com>
Pro looks different, at least for the 2nd node,
we might have to investigate on that, but for the time
being just stay at CE to keep svn logs sane...
From: Michael Prokop <mprokop@sipwise.com>
I don't see any reason why we shouldn't have openssh-server on
each system we install, if manually testing installations of
ngcp-installer on plain Debian systems with the deployment ISO
it's annoying to not be able to login after initial deployment,
so let's change this...
From: Michael Prokop <mprokop@sipwise.com>
Until we've according DHCP/DNS/VM infrastructure that's the easiest
solution...
Thanks for the template, Richard.
From: Michael Prokop <mprokop@sipwise.com>
This is hackish™ and we should figure out a way how to get
rid of this kill-foo in the long run.
Actually the installation still works but if you care about the
error messages or want to debug the resulting system in *live*
mode then it's annoying if you're not aware of what to do...
From: Michael Prokop <mprokop@sipwise.com>
We want to have /etc/fstab being up2date on the live
system after installation, so force processing the
according uevents between kernel and userspace...
From: Michael Prokop <mprokop@sipwise.com>
Booting with Debian installation and bootoption puppet=192.168.51.25
sets up the host for puppet node deployment.
From: Michael Prokop <mprokop@sipwise.com>
For whatever reason the swap partition isn't enabled by default
anymore. Used to work in all my previous test runs (which
confused me already - haha), I've no idea why it's failing now
again. So re-add the mkswap command so we can at least release
2.4.1 now. Meh… ☠
From: Michael Prokop <mprokop@sipwise.com>
parted enables the swap partition during partition creation time
already, so we'd have to rewrite the partition stuff as well.
Turns out the difference between VM sizes (no swap vs. swap) is just
about 30MB, so let's drop the efforts regarding removal-but-enabling-swap-afterwards
and instead we get a even more homogeneous solution. :)
From: Michael Prokop <mprokop@sipwise.com>
As reported by Andrew CE systems tend to run in OOM conditions
pretty soon sometimes. So lets enable a swap partition in all
our systems by default.
To avoid a bigger VM image we create an init script on CE systems
which enables the swap partition during bootup and then destroys
itself (the script, not the swap partition :)).
From: Michael Prokop <mprokop@sipwise.com>
bootoption ngcprelease is too confusing, so drop it.
instead ngcpautobuildrelease does exactly what it says:
ngcpautobuildrelease=2.4-rc0
will skip generation of sources.list inside ngcp-installer and instead
use the defaults, except for the release it will use:
deb http://deb.sipwise.com/autobuild/ release-2.4-rc0 main
so we can test release candidates of upcoming stable releases without
manual intervention or adjusting ngcp-installer for each single test
release.
When installing a trunk release we also take the most recent version
of the installer, the installer will use the sources.list.trunk file
then anway. \o/
From: Michael Prokop <mprokop@sipwise.com>
we need a decent way to test unfinished releases during the release
cycle, using ngcprelease=2.4-rc1 allows us to test the installer
without any manual intervention against the 2.4-rc1 release on jenkins
From: Michael Prokop <mprokop@sipwise.com>
Building the offline mirror on request is slow and we don't
use it anymore since we provide the mirror on the ISO itself,
so instead test for /srv/mirror/debian only.
While at it don't hardcode "squeeze" in the mirror test but
instead use ${DEBIAN_RELEASE}.
From: Michael Prokop <mprokop@sipwise.com>
Michael Prokop
Andrew Pogrebennyk
Andreas Granig
Jon Bonilla
Kirill Solomko
