If the extend-diff-ignore option is set in a dpkg format 1.0
package we still need to set the -i option, otherwise
we again end up with .git files inside the tarball and
dpkg-source fails with
| dpkg-source: error: cannot represent change to .git/[...] binary file contents changed
| [...]
| dpkg-source: warning: the diff modifies the following upstream files:
| [...]
While at it also ensure that we skip leading spaces, which is
what Dpkg::Conf is doing:
https://anonscm.debian.org/cgit/dpkg/dpkg.git/tree/scripts/Dpkg/Conf.pm#n172
Development sponsored by Sipwise GmbH, recorded as
TT#15013 in customers' ticket system.
Fixes 8fe677820f
Thanks: Guillem Jover for feedback and code review
If a debian git tree doesn't provide debian/source/format then
the dpkg 1.0 format is assumed. The 1.0 format needs according
dpkg-buildpackage defaults (-i -I), otherwise the .git tree
gets included in the tarball.
Development sponsored by Sipwise GmbH, recorded as
TT#13353 in customers' ticket system.
Fixes 8fe677820f
Thanks: Guillem Jover for feedback and code review
If you have a debian/source/options with specific filter regexs for
diffs and/or tarballs, using a blank '-i' or '-I' will override them and
use the defaults.
When building with the branch specifier set to a commit ID,
Jenkins detaches the HEAD and sets GIT_BRANCH to detached.
The `git checkout -f ${GIT_BRANCH}` at the end of
generate-git-snapshot fails because a branch named "detached"
does not exist.
Closes#17 @ GH
Thanks: Jan Alexander Steffens <jan.steffens@gmail.com> for bugreport + patch
In cases where the source package comes already with a debian
folder which is located in some subfolder, it can be needed to
specify the path to that file explicitly for git-dch to work
correctly. That's what "$DCH_CHANGELOG_FILE" can be used for.
For example in the Kamailio repository at
https://github.com/kamailio/kamailio you see that there is no
debian folder in the project root. The debian folder is located
under pkg/kamailio/deb/$distribution. When symlinking this folder
to ./debian before starting the build, then git-dch inspects
./debian/changelog but it finds no commit for the latest
changelog entry. So we have to point git-dch to the changelog
file where it is actually committed. By using
"DCH_CHANGELOG_FILE=pkg/kamailio/deb/$distribution/changelog"
this works as intended. Otherwise git-dch will start building a
changelog entry for all commits that exist in the project, but
that will run forever.
This applies only in cases where the debian/changelog is explicitly
set to "UNRELEASED" which makes the generate-git-snapshot use plain
'dch' instead of 'gbp dch' or 'git-dch' which in turn passes the
generated version string directly as the new package version.
This is implemented as an additional option to avoid breaking
existing users which are just happy with the current behaviour.
Since git-buildpackage v0.6.24 the `git-buildpackage` and
`git-dch` commands no longer exist, instead we need to use
`gbp buildpackage` and `gbp dch`.
Closes: #799183
Thanks: Daniele E. Domenichelli <daniele.domenichelli@gmail.com> for the bug report
Introduce the env variable JENKINS_DEBIAN_GLUE_QUIET to turn off the
bash option 'set -x' in various scripts.
To mute them, simply set the variable to any value.
Wikimedia uses more specific distributions reusing the upstream name and
suffixed with '-wikimedia'. Whenever $distribution is set, it is
injected in the version string by version_information().
When generating the original tarball, git buildpackage is given a
version such as:
NoEpochVer: 2.0.0-wikimedia1+0~20150413152850+jessie-wikimedia~1.gbpe39408
From Debian Policy `5.6.12 Version`:
"package management system will break the version number apart at
the last hyphen in the string to determine the upstream_version and
debian_revision"
git buildpackage yields:
Uptream version: 2.0.0-wikimedia1+0~20150413152850+jessie
Debian revision: wikimedia~1.gbpe39408
Instead of an upstream version of '2.0.0'.
When concatenating the version and distribution, replace any dashes in
$distribution by underscores.
NoEpochVer: 2.0.0-wikimedia1+0~20150413152850+jessie_wikimedia~1.gbpe39408
Upstream version: 2.0.0
Debian revision: wikimedia1+0~20150413152850+jessie_wikimedia~1.gbpe39408
DCH_LOG_RESTRICTION can be used to pass additional (path restriction)
parameters to git-dch. For example to restrict the repository path that
is looked at.
A path is for example required if debian is not a directory but a link to a
sub directory created during source package build (e.g. with
PRE_DCH_HOOK). In such a case the changelog would always
contain all commits of the repository if the path wasn't restricted.
Note: An extra parameter was chosen because DCH_OPT couldn't be used
since it isn't the last argument and overriding DCH_EXTRA_OPTS is also
not easily possible if the adjusted VERSION_STRING should be used.
The *-gerrit job which triggers the *-source job should
use the "Current build parameters" option. Otherwise
the *-source job might have a GIT_COMMIT present in
its working space which doesn't match the according
patchset under review.
So always assume that we can rely on parameter
GERRIT_PATCHSET_REVISION to fetch the patchset under
review that should be used for building gerrit stuff.
Development sponsored by Sipwise GmbH
If the *-source job gets triggered directly via Gerrit then we
already have the according commit in $GIT_COMMIT and there's
nothing further special to do (we already supported this use case
since commit 017756fb40).
But if Gerrit does NOT directly trigger the *-source job then
we're usually not standing on the according $GIT_COMMIT. Also the
Gerrit patchset/review change possibly hasn't been even fetched
yet into the Git checkout/workspace. In $GIT_URL we have the
repository URL and rely on being able to fetch from there, using
the "changes" refs. (If someone has the need to make $GIT_URL
even more flexible this shouldn't be a problem, but until someone
actually has the needs I'm following KISS.)
Then we assume that $GERRIT_EVENT_TYPE is unset (which indicates
that it's not directly triggered via Gerrit) and
$GERRIT_PATCHSET_REVISION is set (and points to the according Git
commit that gets fetched).
To set this up all you need to do is triggering the *-source from
your *-gerrit job (or whatever the name of the Jenkins job that
receives the Gerrit events is) with parameter
GERRIT_PATCHSET_REVISION=$GERRIT_PATCHSET_REVISION and then the
*-source job does the rest.
Development sponsored by Sipwise GmbH
If the GERRIT_REFSPEC variable is set then our job was triggered
via Gerrit Code Review [http://gerrit.googlecode.com/]. If that's
the case we don't want to modify the Git repository checkout by
branch or tag because we already have what we need inside
$GIT_COMMIT. Instead let's create the temporary branch based
on $GIT_COMMIT, otherwise (if we don't create a branch on our own)
we're lacking a branch and git-dch fails with:
| + git-dch --auto -S --multimaint-merge --ignore-branch --new-version=[...]
| fatal: ref HEAD is not a symbolic ref
| gbp:error: Currently not on a branch
Using a branch like [origin/]refs/changes/$ID/$ID/$PATCHSET when triggering
the *-source job using $branch setting from Gerrit fails, because
the ref isn't directly available for checkout as branch, but
Jenkins does the job for us so we can rely on $GIT_COMMIT.
When DIST is set in a source job, generate-git-snapshot now uses that
dist in the debian/changelog. As this is only useful if one keeps the
built .changes file, this introduces a new setting KEEP_SOURCE_CHANGES
which must be set as well.
In the binary job configuration, one can get to the dist like this:
export REPOS=$(awk '/^Distribution:/ { gsub("Distribution: ", ""); print $0 }' tmpdir/*_source.changes)
Also, in the binary job one should remove the source changes file, ex:
rm -f tmpdir/*_source.changes
I'm using this successfully to build per-branch dists, in a development
environment where work happens on multiple repositories with shared
branch names.
With changes from @mika to turn the fatal error into a warning.
In commit 9d5366f9db the gbp_opts function was reused, so rename
it to gbp_conf instead. Also do not fail if 'git config --file ...'
calls fail.
Additionally support the IGNORE_GBP_CONF setting to allow skipping
execution of gbp_conf.
The "Please run the script in the jenkins workspace." itself is
a bit misleading if someone just forgot to configure "source"
directory as the checkout directory.
The recent commit to make sure quilt can't break the build (git
commit 63d6c47) made a switch from dpkg-source to
dpkg-buildpackage. dpkg-buildpackage generates a *_source.changes
file which isn't present when using dpkg-source. While this file
is nice it doesn't provide any real value for us, but what's more
important, it might break existing setups:
If all files are copied around from one Jenkins job to another
then in the final jenkins job the changes file from the binaries
job and the source.changes one end up together. If those are
passed to other tools then things might break (see github issue
67 as an example). So let's just remove the file after the
dpkg-buildpackage call to provide the used behaviour.
When using the dpkg 3.0 (quilt) format (see debian/source/format)
there might be quilt patches still applied in the source tree,
either because of a broken build or under certain - though not yet
clearly identified - situations which seem to end with a
"diff: standard output: Broken pipe" message in the build log.
This then might end up in a build failure like:
| + cd /var/lib/jenkins/workspace/glusterfs-source
| + dpkg-source -i -I -b source
| dpkg-source: info: using source format `3.0 (quilt)'
| diff: standard output: Broken pipe
| diff: standard output: Broken pipe
| dpkg-source: info: building glusterfs using existing ./glusterfs_3.2.7.orig.tar.gz
| dpkg-source: info: local changes detected, the modified files are:
| source/ChangeLog
| source/doc/gluster.8
| source/doc/glusterfs.8
| source/libglusterfs/src/common-utils.c
| source/rpc/rpc-lib/src/rpcsvc.c
| source/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c
| dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/glusterfs_3.2.7-2~bpo60+1.diff.3Wkisr
| dpkg-source: info: you can integrate the local changes with dpkg-source --commit
While everything has been cleanly integrated in debian/patches/
the build process fails because it looks like there have been
modifications to upstream sources whereas it's just wrong
information about patches applied.
So instead make sure we enter a clean git repository before doing
any changes (this can be skipped via SKIP_GIT_CLEANUP=true) and
also unapply all present quilt patches (this can be skipped via
SKIP_QUILT_CLEANUP=true). Finally replace the dpkg-source call
with dpkg-buildpackage and its --source-option=--unapply-patches
feature. While the combination of the git cleanup, quilt cleanup
and dpkg-source->dpkg-buildpackage is more than explicitely
necessary this provides more flexibility if needed.
Thanks: Raphael Hertzog <hertzog@debian.org> for his dpkg related feedback
* Do not test if the file exist. It blocks the usage of 'export PRE_SOURCE_HOOK=/home/git/foo.sh version branch'
* Use the same declaration between generate-git-snapshot & generate-svn-snapshot
* Match the similar declaration in the git & svn scripts
If the changelog contains the string "Version" then when
identifying the original version number this can go wrong
obviously, e.g.:
| + version_information
| ++ dpkg-parsechangelog --count 1
| ++ awk '/Version/ {print $2}'
| + ORIG_VERSION='1.17.2
| Version'
Adjust the dpkg-parsechangelog command line to what's being
used in all the other command lines.
Previously, it was doing "cd .." to move up one level, which doesn't
work correctly if $SOURCE_DIRECTORY is multiple levels deep, e.g.
SOURCE_DIRECTORY=source/subproject
Reported in https://github.com/mika/jenkins-debian-glue/issues/47
Compare:
| % git-buildpackage -nc --git-force-create --git-ignore-new -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true
| gbp:debug: ['git', 'rev-parse', '--show-cdup']
| gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
| gbp:debug: /bin/true [] []
| gbp:debug: ['git', 'symbolic-ref', 'HEAD']
| gbp:error: Currently not on a branch
with:
| % git-buildpackage -nc --git-force-create --git-ignore-new -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true --git-ignore-branch
| gbp:debug: ['git', 'rev-parse', '--show-cdup']
| gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
| gbp:debug: /bin/true [] []
| gbp:debug: ['git', 'symbolic-ref', 'HEAD']
| gbp:debug: ['git', 'show-ref', 'refs/heads/pristine-tar']
| gbp:debug: ['git', 'ls-tree', 'upstream/1.7.4']
| gbp:error: upstream/1.7.4 is not a valid treeish
using a recent git-buildpackage version.
The --git-ignore-branch option allows to build packages
from detached HEAD state, also see
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661598
As the option is already present (at least) in gbp 0.5.10
this should be a safe way how to prevent problems with
recent versions of gbp.
Verified against git-buildpackage 0.6.0~git20130506 and 0.5.10.
At least when using the 3.0 quilt format in a Debian package there
might be changes left around in the working directory after
dpkg-source's execution:
| + dpkg-source -i -I -b source
| dpkg-source: info: using source format `3.0 (quilt)'
| dpkg-source: warning: patches have not been applied, applying them now (use --no-preparation to override)
| dpkg-source: info: applying 0001-Added-assertTrue-assertFalse-and-assertNotEquals-fun.patch
| dpkg-source: info: applying 0002-Fix-typo.patch
| dpkg-source: info: applying 0003-Added-assertItemsEquals-function.patch
| dpkg-source: info: building lua-unit using existing ./lua-unit_1.5.orig.tar.gz
| dpkg-source: info: building lua-unit in lua-unit_1.5-1~bpo70+1+0~1368438751.21+wheezy~1.gbp63ac0c.debian.tar.gz
| dpkg-source: info: building lua-unit in lua-unit_1.5-1~bpo70+1+0~1368438751.21+wheezy~1.gbp63ac0c.dsc
| ++ readlink -f debian/changelog
| + git checkout -- /var/lib/jenkins/workspace/lua-unit-source/source/debian/changelog
| + git checkout origin/pristine-tar
| error: You have local changes to 'luaunit.lua'; cannot switch branches.
| Build step 'Execute shell' marked build as failure
So avoid failing builds because of that and force switching
branch.
In some cases the '-I' param to dpkg-source is not enough. In this case
builds fail since dpkg-source complains about modified files.
This additionally adds (previously used) '-i' to the params, which
fixes those cases.
otherwise we might be checking out a branch which doesn't exist locally yet,
failing with:
| fatal: git checkout: updating paths is incompatible with switching branches.
| Did you intend to checkout '$BRANCH' which can not be resolved as commit?
Thanks: Stefan Schlesinger <sts@ono.at> for the bug report and testing
* modify the changelog when building "$branch", otherwise the
package version wouldn't increase and look like a tag [closes#38]
* when using "none" as tag setting then ignore the request for building
from tag, this supports Jenkins jobs with "branch" *and* "tag" setting
but having "tag" set to "none" as default
* support setting $SOURCE_DIRECTORY instead of hardcoding "source" as
default source directory
* follow symlinks when restoring debian/changelog so the file gets
restored more reliable
* restore debian/changelog *before* changing back to original branch,
otherwise git might refuse to do the switch if data has been modified
in the working directory
* retrieve version information *after* checking out the according
branch/tag, otherwise we might end up with the wrong version number
* move code into functions
Development sponsored by Sipwise.com.
If we are changing back to the previous branch before running dpkg-source
then we always build the latest version instead of the requested one.
So make sure to actually generate the tarballs and *then* switch back
to the original GIT_BRANCH.
Depending on the dpkg-source version the tar ball includes the
.git directory even though 'i\.git' is specified, which isn't the
expected behaviour.
Quoting the dpkg-source manpage:
| -I by itself adds default --exclude options that will filter out
| control files and directories of the most common revision control
| systems, backup and swap files and Libtool build output
| directories.
So instead of using this broken 'i\.git' option let's just use
-I option without any arguments and everything seems to be good. :)
Verified for dpkg-source versions:
* 1.14.31 (lenny)
* 1.15.8.13 (squeeze)
* 1.16.8 (wheezy)
* 1.16.9 (sid as of 2012-10-28)
for the jenkins-debian-glue repository itself, using the 3.0
native format as well as running it through 'dpkg-source
--format=1.0'.
Thanks: yanfalies for the bugreport
Closes: #20
Michael Prokop
Will DeBerry
Sebastian Damm
Kane
Antoine Musso
Walter Kleynscheldt
Michael Prokop
nextime
Bernhard Miklautz
Christian Hofstaedtler
Mathieu Parent
Carl Fürstenberg
Sylvestre Ledru
Michael Prokop
Calvin Walton
Slawomir Bochenski
Patrick Schoenfeld
Antoine Beaupré
Marcel Beck
Marcelo Martins