A funny bug was discovered where if the buildroot's path
has the name of the build target within it, it will also be substituted
along with the stampfile's name for each program,
causing an attempt to touch a file in a directory that doesn't exist.
...
...
touch: cannot touch '/Volumes/touch/openwrt/staging_dir/host/stamp/.touch_installed': No such file or directory
touch: cannot touch '/Volumes/ln/openwrt/staging_dir/host/stamp/.ln_installed': No such file or directory
touch: cannot touch '/Volumes/chown/openwrt/staging_dir/host/stamp/.chown_installed': No such file or directory
make[2]: *** [Makefile:50: /Volumes/coreutils/openwrt/staging_dir/host/stamp/.coreutils_installed] Error 1
...
...
Split up the path with $(dir) and $(notdir) before substitution to fix
the syntax.
Reported-by: Georgi Valkov <gvalkov@gmail.com>
Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Some individual build items install a group of programs
instead of a program matching the name of the build item.
Add support for installing stampfiles for each of the
programs installed by that build item,
which will allow more control and awareness
of what is installed by the rest of the build system,
if, for example, prereq symlink checks are looking
for the same program which is built already.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
A package may run git as part of its build process, and if the package
source code is not from a git checkout, then git may traverse up the
directory tree to find buildroot's repository directory (.git).
For instance, Poetry Core, a Python build backend, will read the
contents of .gitignore for paths to exclude when creating a Python
package. If it finds buildroot's .gitignore file, then Poetry Core will
exclude all of the package's files[1].
This exports GIT_CEILING_DIRECTORIES for both package and host builds so
that git will not traverse beyond $(BUILD_DIR)/$(BUILD_DIR_HOST).
[1]: https://github.com/python-poetry/poetry/issues/5547
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Recent versions of Automake
have changed dependency tracking significantly
(reference commit below)
causing breakage in some package builds
when using newer Automake with packages that need autoreconf
that were bootstrapped with an old version of Automake.
Those changes cause a great inconsistency between packages over time
where some packages may or may not use this feature,
and may or may not update the .ac and .am files
to work with the new methods.
This problem might exist in many packages
where autoreconf is not currently required,
but would cause build failure if autoreconf is used.
Fortunately, this feature is practically useless
for the purposes of Openwrt and the average developer,
so we can disable it.
GNU Automake manual states in part:
"Because dependencies are only computed as a side-effect of compilation...
no dependency information exists the first time a package is built...
dependency tracking is completely useless for one-time builds..."
A nice side-effect is that build times are slightly faster.
Ref: 6a675ef17edf7109da189f5ae70e2dc6b7665896 (automake.git)
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Packages in general use 4 check to trigger a recompile:
- timestamp for the build_dir
- timestamp for the staging stamp dir
- depends hash for the build_dir prepared file
- presence of package archieve in dl
If host tools are prebuilt and shipped in a container or manually
installed from an archieve, it would be ideal to skip including the
package archieve and just provide the build_dir prepared files and the
staging stamp file (and the actualy prebuilt tools).
Add some logic to skip dl download for host tools if AUTOREMOVE is
selected and checks for the presence of staging dir stamp file and build
dir stamp file.
If one of these requirements are not met, the package is redownloaded
and rebuilt.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
While experimenting with the AUTOREMOVE option in search of a way to use
prebuilt host tools in different buildroot, it was discovered that the
md5 generated by find_md5 in depends.mk is not reproducible.
Currently the hash is generated by the path of the file in addition to
the file mod time. Out of confusion, probably, there was an idea that
such command was used on the package build_dir. Reality is that this
command is run on the package files. (Makefile, patches, src)
This is problematic because the package Makefile (for example) change at
each git clone and base the hash on the Makefile mtime doesn't really
reflect if the Makefile actually changes across a buildroot or not.
A better approach is to generate an hash of each file and then generate
an hash on the sort hash list. This way we remove the problem of git
clone setting a wrong mtime while keeping the integrity of checking if a
file changed for the package as any change will result in a different
hash.
Introduce a new kind of find_md5 function, find_md5_reproducible that
apply this new logic and limit it only with AUTOREMOVE option set to
prevent any kind of slowdown due to additional hash generation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Package with whitespace in their build directory are not correctly
removed when CONFIG_AUTOREMOVE is enabled. This is caused by xargs that
use whitespace as delimiters. To handle this use \0 as the delimiter and
set find to use \0 as the delimiter.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Added HOST_CXXFLAGS to specify CXXFLAGS during host-compile
(e.g. to specify c++ standard: HOST_CXXFLAGS += -std=c++11)
Signed-off-by: Sergey V. Lobanov <sergey@lobanov.in>
4e19cbc5533: [download: handle possibly invalid local tarballs] added a
FORCE rule to downloaded files, so that they will be always checked by
download.pl.
As a side-effect, check-compile will fail, forcing unnecessary package
rebuilds.
The check-compile.txt log shows (for libxml2 for example):
Considering target file '.../dl/libxml2-2.9.12.tar.gz'.
...
prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does
not exist.
Must remake target '.../dl/libxml2-2.9.12.tar.gz'.
...
Giving up on target file '...libxml2-2.9.12/.prepared_...'.
Giving up on target file '...libxml2-2.9.12/.configured_...'.
Giving up on target file '...libxml2-2.9.12/.built'.
Giving up on target file '...stamp/.libxml2_installed'.
Giving up on target file '.compile'.
Then the package is rebuilt even if it is not otherwise needed.
To fix this, instead of always forcing the download target to be remade,
check its hash first: if it matches, then the FORCE is not added.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
With commit 2ca084cc ("build: improve ccache support") these variables
are being set globally and we don't need them for specific targets.
Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
This causes various issues in other places that assume that host
binaries are staged in STAGING_DIR_HOST.
Since all the right places use HOST_BUILD_PREFIX, override that instead.
This fixes some issues with quilt on toolchain dirs
Signed-off-by: Felix Fietkau <nbd@nbd.name>
quilt.mk needs to be included first, to ensure that STAMP_PREPARED does
not include the hash if quilt is used.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Calling the clean target removes all .ipk files and un-stages the
package. Add a new target just for clearing the build dir and call that
one instead of the full clean target
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Instead of passing HOST_CONFIGURE_VARS as arguments to the configure script,
pass it as environment variables which brings the logic in line with the
behaviour of package-defaults.mk.
The change is needed since passing environment variables as configure
parameters only works with GNU autoconf which evaluates command line arguments
looking like variable assignments. Doing the same with non-autoconf configure
scripts is not guaranteed to work since such scripts might terminate due to
unknown argument errors.
One example case is the cmake configure script which bails out when called
as "./configure LDFLAGS=..." but not when called as "LDFLAGS=... ./configure".
Also change the SHELL override to CONFIG_SHELL in the default
HOST_CONFIGURE_VARS as the former is not properly propagated through the
various GNU configure invocations since it gets lost when configure re-
executes itself.
A prior attempt to change the variable placement had to be reverted due to
the missing SHELL -> CONFIG_SHELL change, leading to misgenerated libtool
executables in various packages.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This reverts commit 8395b63aac616f72fd835c59240fc2a4a6b28106.
Various host builds currently rely on the broken behaviour of
HOST_CONFIGURE_VARS so roll back to the previous state.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Ensure that HOST_CONFIGURE_VARS are set before the actual configure command
instead of passing them as configure command arguments.
This change brings host-build.mk in line with package-defaults.mk and makes
host configure environment variables work as expected.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Instead of hardcoding $(STAGING_DIR)/host, use the new $(STAGING_DIR_HOSTPKG)
variable to refer to the directory.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
The normal Prepare step for a build is unpack, apply patches.
But for certain packages, patches contain whole files, which
would be nice to have separately and copied over as a last step
in the Prepare phase.
We need it for some other packages + patches, but I think
the 'hostapd' package can be used as a test for this.
As a quick note:
the reason the condition is being evaluated as
`[ ! -d ./src/ ] || $(CP) ./src/* $(HOST_BUILD_DIR)`
and not with
`[ -d ./src/ ] && $(CP) ./src/* $(HOST_BUILD_DIR)`
is that the latter would translate in a build failure if the `src`
folder is not present (the exit code would be 1).
The first one, succeeds for both cases (if `src` present or not).
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Having a different PATH in Host/Install than in other steps like Host/Build
is confusing and easily leads to mistakes. Setting all of Host/Exports
makes host builds match target builds (Build/Install is part of
$(STAMP_BUILT), which has Build/Exports set).
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
It seems the intention was to add both $(STAGING_DIR_HOST)/... and
$(STAGING_DIR)/host/... instead of passing $(STAGING_DIR_HOST) twice. This
makes the definition match HOST_CPPFLAGS and HOST_LDFLAGS.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Using HOST_BUILD_PREFIX instead of STAGING_DIR_HOST will make the argument
work as expected from packages.
Nothing changes for tools, for which HOST_BUILD_PREFIX and STAGING_DIR_HOST
are equivalent.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Make usually passes -j and jobserver arguments as part of MAKEFLAGS. LEDE
removes MAKEFLAGS to have better control of the build, and re-adds the
jobserver arguments with -j to the Make commandline where desired.
Make 4.2+ behave differently with these arguments passed on the commandline
than in MAKEFLAGS: -j will override the jobserver argument and the job
count will be unlimited.
Moving the flags to MAKEFLAGS will need many packages to be changed and
tested; therefore, we opt for a less invasive change for now and just
remove -j for Make 4.2+, as the jobserver argument alone is enough to
enable parallel builds for these Make versions.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Michael Pratt