Bug 205223

Summary: Add --allconfig option to kunit_tool
Product: Tools Reporter: Brendan Higgins (brendanhiggins)
Component: KUnitAssignee: Tools/KUnit virtual default assignee (tools_kunit)
Status: RESOLVED CODE_FIX    
Severity: normal CC: brendanhiggins, pefoley
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: kselftest/test (5.4) Subsystem:
Regression: No Bisected commit-id:
Bug Depends on: 205771    
Bug Blocks:    

Description Brendan Higgins 2019-10-16 21:47:23 UTC 
It would be great if there was a way to automagically run all KUnit tests.

Right now the long term solution, as suggested by Luis Chamberlain, seems to be to be to try to detect which CONFIGs enable KUnit tests and then to produce a set of .configs which when taken in sum will run all tests. However, this would likely not be as easy as it sounds as it is possible to have mutually exclusive CONFIGs. Luis pointed out that some researchers are currently working on a sat solver for Kconfig that we could use to potentially address this problem. Nevertheless, a complete solution in this regard is actually somewhat difficult.

Shuah proposed what is looking like the best short term solution: use .config fragments similar to what kselftest already does.

Most recent iteration of this discussion was brought up by Ted Ts'o here: https://groups.google.com/d/msg/kunit-dev/QmA27YEgEgI/hvS1kiz2CwAJ
Comment 1 Brendan Higgins 2019-11-15 00:50:19 UTC 
Kees Cook proposed a good temporary solution (if not long term solution) in just using allyesconfig to generate a config with every test available for that architecture turned on.

His comment can be found here: https://lore.kernel.org/linux-kselftest/201910301201.404F0E3BB@keescook/
Comment 2 Brendan Higgins 2019-12-04 19:00:59 UTC 
Heidi Fahim discovered that allyesconfig is broken for UML.

I tracked down all the offending configs and it appears to be quite a few:

CONFIG_STATIC_LINK=y
CONFIG_UML_NET_VECTOR=y
CONFIG_UML_NET_VDE=y
CONFIG_UML_NET_PCAP=y
CONFIG_NET_PTP_CLASSIFY=y
CONFIG_IP_VS=y
CONFIG_BRIDGE_EBT_BROUTE=y
CONFIG_BRIDGE_EBT_T_FILTER=y
CONFIG_BRIDGE_EBT_T_NAT=y
CONFIG_MTD_NAND_CADENCE=y
CONFIG_MTD_NAND_NANDSIM=y
CONFIG_BLK_DEV_NULL_BLK=y
CONFIG_BLK_DEV_RAM=y
CONFIG_SCSI_DEBUG=y
CONFIG_NET_VENDOR_XILINX=y
CONFIG_NULL_TTY=y
CONFIG_PTP_1588_CLOCK=y
CONFIG_PINCTRL_EQUILIBRIUM=y
CONFIG_DMABUF_SELFTESTS=y
CONFIG_COMEDI=y
CONFIG_XIL_AXIS_FIFO=y
CONFIG_EXFAT_FS=y
CONFIG_STM_DUMMY=y
CONFIG_FSI_MASTER_ASPEED=y
CONFIG_JFS_FS=y
CONFIG_UBIFS_FS=y
CONFIG_CRAMFS=y
CONFIG_CRYPTO_DEV_SAFEXCEL=y
CONFIG_CRYPTO_DEV_AMLOGIC_GXL=y
CONFIG_KCOV=y
CONFIG_LKDTM=y
CONFIG_REED_SOLOMON_TEST=y
CONFIG_TEST_RHASHTABLE=y
CONFIG_TEST_MEMINIT=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y

Some of these prevent UML from building, others cause UML to crash before reaching the end of init.

One important note: It is not clear that CONFIG_NULL_TTY=y is actually broken on UML; however, it does cause the null TTY it creates to become console, which prevents UML from printing output after it is initialized.
Comment 3 Brendan Higgins 2019-12-04 19:49:22 UTC 
I posted a config and script that can be used to generate an allyesconfig minus the above configs:

https://kunit-review.googlesource.com/c/linux/+/3109