linux.git/tools/testing/kunit, branch v5.11-rc2

Merge tag 'kbuild-v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

2020-12-22T22:02:39+00:00

Pull Kbuild updates from Masahiro Yamada:

 - Use /usr/bin/env for shebang lines in scripts

 - Remove useless -Wnested-externs warning flag

 - Update documents

 - Refactor log handling in modpost

 - Stop building modules without MODULE_LICENSE() tag

 - Make the insane combination of 'static' and EXPORT_SYMBOL an error

 - Improve genksyms to handle _Static_assert()

* tag 'kbuild-v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
  Documentation/kbuild: Document platform dependency practises
  Documentation/kbuild: Document COMPILE_TEST dependencies
  genksyms: Ignore module scoped _Static_assert()
  modpost: turn static exports into error
  modpost: turn section mismatches to error from fatal()
  modpost: change license incompatibility to error() from fatal()
  modpost: turn missing MODULE_LICENSE() into error
  modpost: refactor error handling and clarify error/fatal difference
  modpost: rename merror() to error()
  kbuild: don't hardcode depmod path
  kbuild: doc: document subdir-y syntax
  kbuild: doc: clarify the difference between extra-y and always-y
  kbuild: doc: split if_changed explanation to a separate section
  kbuild: doc: merge 'Special Rules' and 'Custom kbuild commands' sections
  kbuild: doc: fix 'List directories to visit when descending' section
  kbuild: doc: replace arch/$(ARCH)/ with arch/$(SRCARCH)/
  kbuild: doc: update the description about kbuild Makefiles
  Makefile.extrawarn: remove -Wnested-externs warning
  tweewide: Fix most Shebang lines

tweewide: Fix most Shebang lines

2020-12-08T14:30:04+00:00

Change every shebang which does not need an argument to use /usr/bin/env.
This is needed as not every distro has everything under /usr/bin,
sometimes not even bash.

Signed-off-by: Finn Behrens 
Signed-off-by: Masahiro Yamada

kunit: kunit_tool: Correctly parse diagnostic messages

2020-12-01T22:31:04+00:00

Currently, kunit_tool expects all diagnostic lines in test results to
contain ": " somewhere, as both the subtest header and the crash report
do. Fix this to accept any line starting with (minus indent) "# " as
being a valid diagnostic line.

This matches what the TAP spec[1] and the draft KTAP spec[2] are
expecting.

[1]: http://testanything.org/tap-specification.html
[2]: https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/T/

Signed-off-by: David Gow 
Acked-by: Marco Elver 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: Introduce get_file_path() helper

2020-11-30T20:56:30+00:00

Helper allows to derive file names depending on --build_dir argument.

Signed-off-by: Andy Shevchenko 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: fix extra trailing \n in raw + parsed test output

2020-11-10T20:44:35+00:00

For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

`isolate_kunit_output()` yielded liens with trailing \n, which results
in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46]     Expected 1 + 1 == 3, but

[16:16:46]         1 + 1 == 2

[16:16:46]         3 == 3

[16:16:46]     not ok 1 - example_simple_test

[16:16:46]

After this change:
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46]     Expected 1 + 1 == 3, but
[16:16:46]         1 + 1 == 2
[16:16:46]         3 == 3
[16:16:46]     not ok 1 - example_simple_test
[16:16:46]

We should *not* be expecting lines to end with \n in kunit_tool_test.py
for this reason.

Do the same for `raw_output()` as well which suffers from the same
issue.

This is a followup to [1], but rebased onto kunit-fixes to pick up the
other raw_output() fix and fixes for kunit_tool_test.py.

[1] https://lore.kernel.org/linux-kselftest/20201020233219.4146059-1-dlatypov@google.com/

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Tested-by: David Gow 
Signed-off-by: Shuah Khan

kunit: tool: print out stderr from make (like build warnings)

2020-11-10T20:43:22+00:00

Currently the tool redirects make stdout + stderr, and only shows them
if the make command fails.
This means build warnings aren't shown to the user.

This change prints the contents of stderr even if make succeeds, under
the assumption these are only build warnings or other messages the user
likely wants to see.

We drop stdout from the raised exception since we can no longer easily
collate stdout and stderr and just showing the stderr seems fine.

Example with a warning:

[14:56:35] Building KUnit Kernel ...
../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’:
../lib/kunit/kunit-test.c:19:6: warning: unused variable ‘unused’ [-Wunused-variable]
   19 |  int unused;
      |      ^~~~~~

[14:56:40] Starting KUnit Kernel ...

Note the stderr has a trailing \n, and since we use print, we add
another, but it helps separate make and kunit.py output.

Example with a build error:

[15:02:45] Building KUnit Kernel ...
ERROR:root:../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’:
../lib/kunit/kunit-test.c:19:2: error: unknown type name ‘invalid_type’
   19 |  invalid_type *test = data;
      |  ^~~~~~~~~~~~
...

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Signed-off-by: Shuah Khan

kunit: Do not pollute source directory with generated files (test.log)

2020-11-10T20:31:16+00:00

When --build_dir is provided use it and do not pollute source directory
which even can be mounted over network or read-only.

Signed-off-by: Andy Shevchenko 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: Do not pollute source directory with generated files (.kunitconfig)

2020-11-10T20:12:54+00:00

When --build_dir is provided use it and do not pollute source directory
which even can be mounted over network or read-only.

Signed-off-by: Andy Shevchenko 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: fix pre-existing python type annotation errors

2020-11-10T20:11:28+00:00

The code uses annotations, but they aren't accurate.
Note that type checking in python is a separate process, running
`kunit.py run` will not check and complain about invalid types at
runtime.

Fix pre-existing issues found by running a type checker
$ mypy *.py

All but one of these were returning `None` without denoting this
properly (via `Optional[Type]`).

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Signed-off-by: Shuah Khan

kunit: Fix kunit.py parse subcommand (use null build_dir)

2020-11-10T20:09:15+00:00

When JSON support was added in [1], the KunitParseRequest tuple was
updated to contain a 'build_dir' field, but kunit.py parse doesn't
accept --build_dir as an option. The code nevertheless tried to access
it, resulting in this error:

AttributeError: 'Namespace' object has no attribute 'build_dir'

Given that the parser only uses the build_dir variable to set the
'build_environment' json field, we set it to None (which gives the JSON
'null') for now. Ultimately, we probably do want to be able to set this,
but since it's new functionality which (for the parse subcommand) never
worked, this is the quickest way of getting it back up and running.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit-fixes&id=21a6d1780d5bbfca0ce9b8104ca6233502fcbf86

Fixes: 21a6d1780d5b ("kunit: tool: allow generating test results in JSON")
Signed-off-by: David Gow 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan