Run the invocation of Make with verbosity in order to
prevent the printing of Makefile level and subtarget status.
e.g. make[3] -C target/linux val.DEFAULT_PACKAGES val.ARCH_PACKAGES
Remove piping of stderr, which is only useful when using
the "communicate" method over the "run" method,
and this script would not be written to handle a captured error anyway.
For error testing, stdout and stderr can be set to a file object
with the open() function like this:
out = open('json_out', 'w')
err = open('json_err', 'w')
...
...
stdout=out,
stderr=err,
Signed-off-by: Michael Pratt <mcpratt@pm.me>
This only affects typos in comments or user-facing output.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
[only picks changes to scripts, drop "commandline" replacement,
fix case for "arbitrary", improve commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
If the image generation doesn't add any profiles to the output the
*profile merge* will fail. To avoid that set an empty profile as
fallback.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Keep other profiles.json content if the data belongs to the current
build version.
Also useful for the ImageBuilder, which builds for a single model each
time. Without this commit the profiles.json would only contain the
latest build profile information.
Signed-off-by: Moritz Warning <moritzwarning@web.de>
[improve commit message]
Signed-off-by: Paul Spooren <mail@aparcar.org>
This became a bit of a tragedy, caused by a corner cases which wasn't
put into account during testing. DEFAULT_PACKAGES are defined in
target/linux/<target>/Makefile but also in
target/linux/<target>/<subtarget>/target.mk.
The latter was no longer imported when using DUMP=1, however not using
DUMP=1 while running the Makefile in target/linux/<target>/ caused duplicate
packages in the list.
As a solution, which should have been used from day 0, `make` runs in
target/linux/ without DUMP=1, resulting in no duplicate packages and all
inclusions from include/target.mk, linux/target/<target>/{Makefile,
<subtarget>/target.mk}
While at it, sort the list of default packages.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The commit "1bf2b3fe90 build,json: fixup missing arch_packages" fixes
the missing package architecture locally but runs $(TOPDIR)/Makefile
rather than a target specific one. While this works on local builds just
fine, it causes the buildbots to add garbage to the `arch_packages`
variable:
cd \"/builder/shared-workdir/build\"; git log --format=%h -1
toolchain > /builder/shared-workdir/build/tmp/.ver_check\ncmp -s
/builder/shared-workdir/build/tmp/.ver_check
/builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check
|| { \\\n\trm -rf
/builder/shared-workdir/build/build_dir/target-x86_64_musl
/builder/shared-workdir/build/staging_dir/target-x86_64_musl
/builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl
/builder/shared-workdir/build/build_dir/toolchain-x86_64_gcc-8.4.0_musl;
\\\n\tmkdir -p
/builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp;
\\\n\tmv /builder/shared-workdir/build/tmp/.ver_check
/builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check;
\\\n}\nx86_64
Only the last line contains the desired string.
Future investigation should check why the build system prints this to
stdout rather than stderr.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Fix 7f4c2b1 "build,json: fix duplicates in default_packages" which
removed duplicate default packages but also removed the package
architecture from the profiles.json.
If DUMP=1 is set, the `ARCH_PACKAGES` is no longer exported and
therefore empty. Fix this by running make twice, once with DUMP=1 and
once without.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Calling without the DUMP=1 argument causes the target specific Makefile
to be "included" again which adds the target specific packages twice,
once on the actual run and once included from `include/target.mk`.
This led to duplicate package entries, causing confusion in downstream
projects using the generated JSON files.
While at it, apply `black` style to Python script.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The f-string feature was introduced in Python 3.6. As Buildbots may run
on Debian 9, which comes per default with Python 3.5, this would cause
an issue. Instead of f-strings use the *legacy* `.format()` function.
Signed-off-by: Paul Spooren <mail@aparcar.org>
On a system python3 is linked to python3.6, fail to perform json_overview_image_info
and got `TypeError: __init__() got an unexpected keyword argument 'capture_output'`.
This patch emulate the behaviour on python 3.7+.
Signed-off-by: Huangbin Zhan <zhanhb88@gmail.com>
Only collect arch_packages if actually generating any output.
Fixes: commit f09b9319 ("build,json: store arch_packages in profiles.json"(
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
The `arch_packages` contains the supported package architecture.
Previously it was necessary to parse the `Packages` index for the line
`Architecture:`, requiring both an additional parser and file download.
Signed-off-by: Paul Spooren <mail@aparcar.org>
With this commit the `profiles.json` contain both the target specific
`default_packages` as well as the device specific `device_packages` as a
array of strings.
This information is required for downstream projects like the various
web-based interactive firmware generators.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new
image generation methods from `image.mk` are used. However some targets
like `armvirt` do not use it yet, so the folder is never created.
The `json_overview_image_info.py` script used to raise an error if the
given `WORK_DIR` isn't a folder, however it should just notify about
missing JSON files.
This patch removes the Python assert and exists with code 0 even if no
JSON files were found, as this is not necessarily an error but simply
not yet implemented. Using `glob` on an not existing `Path` results in
an empty list, therefore the for loop won't run.
Signed-off-by: Paul Spooren <mail@aparcar.org>
CC: Petr Štetiar <ynezz@true.cz>
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Michael Pratt