Bug 205223 - Add --allconfig option to kunit_tool
Summary: Add --allconfig option to kunit_tool
Status: RESOLVED CODE_FIX
Alias: None
Product: Tools
Classification: Unclassified
Component: KUnit (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Tools/KUnit virtual default assignee
URL:
Keywords:
Depends on: 205771
Blocks:
  Show dependency tree
 
Reported: 2019-10-16 21:47 UTC by Brendan Higgins
Modified: 2020-03-31 18:29 UTC (History)
2 users (show)

See Also:
Kernel Version: kselftest/test (5.4)
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Brendan Higgins 2019-10-16 21:47:23 UTC
It would be great if there was a way to automagically run all KUnit tests.

Right now the long term solution, as suggested by Luis Chamberlain, seems to be to be to try to detect which CONFIGs enable KUnit tests and then to produce a set of .configs which when taken in sum will run all tests. However, this would likely not be as easy as it sounds as it is possible to have mutually exclusive CONFIGs. Luis pointed out that some researchers are currently working on a sat solver for Kconfig that we could use to potentially address this problem. Nevertheless, a complete solution in this regard is actually somewhat difficult.

Shuah proposed what is looking like the best short term solution: use .config fragments similar to what kselftest already does.

Most recent iteration of this discussion was brought up by Ted Ts'o here: https://groups.google.com/d/msg/kunit-dev/QmA27YEgEgI/hvS1kiz2CwAJ
Comment 1 Brendan Higgins 2019-11-15 00:50:19 UTC
Kees Cook proposed a good temporary solution (if not long term solution) in just using allyesconfig to generate a config with every test available for that architecture turned on.

His comment can be found here: https://lore.kernel.org/linux-kselftest/201910301201.404F0E3BB@keescook/
Comment 2 Brendan Higgins 2019-12-04 19:00:59 UTC
Heidi Fahim discovered that allyesconfig is broken for UML.

I tracked down all the offending configs and it appears to be quite a few:

CONFIG_STATIC_LINK=y
CONFIG_UML_NET_VECTOR=y
CONFIG_UML_NET_VDE=y
CONFIG_UML_NET_PCAP=y
CONFIG_NET_PTP_CLASSIFY=y
CONFIG_IP_VS=y
CONFIG_BRIDGE_EBT_BROUTE=y
CONFIG_BRIDGE_EBT_T_FILTER=y
CONFIG_BRIDGE_EBT_T_NAT=y
CONFIG_MTD_NAND_CADENCE=y
CONFIG_MTD_NAND_NANDSIM=y
CONFIG_BLK_DEV_NULL_BLK=y
CONFIG_BLK_DEV_RAM=y
CONFIG_SCSI_DEBUG=y
CONFIG_NET_VENDOR_XILINX=y
CONFIG_NULL_TTY=y
CONFIG_PTP_1588_CLOCK=y
CONFIG_PINCTRL_EQUILIBRIUM=y
CONFIG_DMABUF_SELFTESTS=y
CONFIG_COMEDI=y
CONFIG_XIL_AXIS_FIFO=y
CONFIG_EXFAT_FS=y
CONFIG_STM_DUMMY=y
CONFIG_FSI_MASTER_ASPEED=y
CONFIG_JFS_FS=y
CONFIG_UBIFS_FS=y
CONFIG_CRAMFS=y
CONFIG_CRYPTO_DEV_SAFEXCEL=y
CONFIG_CRYPTO_DEV_AMLOGIC_GXL=y
CONFIG_KCOV=y
CONFIG_LKDTM=y
CONFIG_REED_SOLOMON_TEST=y
CONFIG_TEST_RHASHTABLE=y
CONFIG_TEST_MEMINIT=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y

Some of these prevent UML from building, others cause UML to crash before reaching the end of init.

One important note: It is not clear that CONFIG_NULL_TTY=y is actually broken on UML; however, it does cause the null TTY it creates to become console, which prevents UML from printing output after it is initialized.
Comment 3 Brendan Higgins 2019-12-04 19:49:22 UTC
I posted a config and script that can be used to generate an allyesconfig minus the above configs:

https://kunit-review.googlesource.com/c/linux/+/3109

Note You need to log in before you can comment on or make changes to this bug.