I've tracked this down to "udevadm trigger --type=devices --action=add"
invoked from /usr/lib/systemd/system/systemd-udev-trigger.service .

To control when the panic occurs:
- Boot the kernel with "init=/bin/sh" kernel option.
- In the init shell, type
  sh# chmod -x /usr/bin/udevadm
  to make systemd-udev service fail.
- Continue booting by typing
  sh# exec /usr/lib/systemd/systemd

The system doesn't panic when udevadm is disabled.

To cause the panic:
- sh# chmod +x /usr/bin/udevadm
- sh# udevadm trigger --type=devices --action=add

After 46 seconds, the kernel will panic.
Invoking "udevadm monitor &" beforehand does not log anything suspicious;
udevd seems quiscent.

Since systemd-udevd is related, this bug may not be a memory management
but something related to systemd-udevd<=>kernel interface (inotify?)

FYI: kernel-4.19.94 also panics, but at different location:

[  202.109493] BUG: unable to handle kernel paging request at eb800000
[  202.114991] *pde = 00000000
[  202.115635] Oops: 0002 [#1] SMP
[  202.116538] CPU: 0 PID: 184 Comm: kworker/0:4 Not tainted 4.19.94-1.el8.i586 #1
[  202.116882] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012
[  202.116882] Workqueue: events hot_add_req [hv_balloon]
[  202.116882] EIP: sparse_add_one_section+0xcb/0x12e
[  202.116882] Code: 45 ec e8 ce 5b 00 00 8b 55 e4 89 45 e8 89 d0 c1 e0 04 8d b0 c0 f7 dc c4 f6 80 c0 f7 dc c4 01 75 44 b0 ff b9 00 00 0a 00 89 df <f3> aa 89 f0 2d c0 f7 dc c4 c1 f8 04 3b 05 80 f7 dc c4 7e 05 a3 80
[  202.116882] EAX: 000000ff EBX: eb800000 ECX: 000a0000 EDX: 0000000d
[  202.116882] ESI: c4dcf890 EDI: eb800000 EBP: df8bbe48 ESP: df8bbe2c
[  202.116882] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010046
[  202.116882] CR0: 80050033 CR2: eb800000 CR3: 014fb000 CR4: 003406d0
[  202.116882] Call Trace:
[  202.116882]  __add_pages+0x89/0x100
[  202.116882]  arch_add_memory+0x3c/0x50
[  202.116882]  add_memory_resource+0x125/0x180
[  202.116882]  __add_memory+0xad/0x130
[  202.116882]  add_memory+0x2c/0x3a
[  202.116882]  hot_add_req+0x3de/0x60b [hv_balloon]
[  202.116882]  process_one_work+0x17b/0x340
[  202.116882]  worker_thread+0x39/0x3d0
[  202.116882]  kthread+0xf0/0x110
[  202.116882]  ? pwq_unbound_release_workfn+0xc0/0xc0
[  202.116882]  ? kthread_bind+0x30/0x30
[  202.116882]  ret_from_fork+0x2e/0x40
[  202.116882] Modules linked in: intel_rapl_perf pcspkr i2c_piix4 sg hv_balloon hv_utils joydev rfkill zram ext4 mbcache jbd2 loop nls_utf8 isofs sr_mod cdrom sd_mod ata_generic 8021q garp mrp stp llc hv_netvsc hv_storvsc scsi_transport_fc hyperv_keyboard hid_hyperv ata_piix libata hyperv_fb crc32_pclmul hv_vmbus serio_raw sunrpc xts lrw dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_intel raid1 raid0 iscsi_ibft squashfs cramfs be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi edd scsi_transport_iscsi
[  202.116882] CR2: 00000000eb800000
[  202.116882] ---[ end trace 8cf405307c8a67b5 ]---
[  202.116882] EIP: sparse_add_one_section+0xcb/0x12e
[  202.116882] Code: 45 ec e8 ce 5b 00 00 8b 55 e4 89 45 e8 89 d0 c1 e0 04 8d b0 c0 f7 dc c4 f6 80 c0 f7 dc c4 01 75 44 b0 ff b9 00 00 0a 00 89 df <f3> aa 89 f0 2d c0 f7 dc c4 c1 f8 04 3b 05 80 f7 dc c4 7e 05 a3 80
[  202.116882] EAX: 000000ff EBX: eb800000 ECX: 000a0000 EDX: 0000000d
[  202.116882] ESI: c4dcf890 EDI: eb800000 EBP: df8bbe48 ESP: c4c633bc
[  202.116882] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010046
[  202.116882] CR0: 80050033 CR2: eb800000 CR3: 014fb000 CR4: 003406d0
[  202.116882] Kernel panic - not syncing: Fatal exception
[  202.116882] Kernel Offset: 0x3000000 from 0xc1000000 (relocation range: 0xc0000000-0xe07effff)
[  202.116882] ---[ end Kernel panic - not syncing: Fatal exception ]---

After experimenting,
disabling hv_balloon driver by
"module_blacklist=hv_balloon" kernel option
will not cause the panic.

Seems something wrong for 32-bit drivers/hv/hv_balloon.c driver.

By passing module parameter via kernel commandline by
"hv_balloon.pressure_report_delay=60", the time between
"udevadm trigger --type=devices --action=add" and panic changes accordingly,
so something after pressure_report_delay expires
(default static uint pressure_report_delay=45)
seems to be wrong, in drivers/hv/hv_balloon.c:post_status() .

It doesn't seem to be wrong anywhere though from my eyes...

adding "hv_balloon.hot_add=0" kernel commandine seems to
make it not panic.

Something seems wrong with 32-bit memory hot-add codepath.
I've also seen several times a panic going through 
hot_add_req() [hv_balloon] .

Have succeeded in capturing a panic involving drivers/hv/hv_balloon.c:hot_add_req() .
Reproducability unknown.

[   73.268544] BUG: unable to handle kernel paging request at eb800000
[   73.270157] *pde = 00000000
[   73.271192] Oops: 0002 [#1] SMP
[   73.271897] CPU: 0 PID: 5 Comm: kworker/0:0 Not tainted 4.19.95-1.el8.i586 #1
[   73.273535] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012
[   73.275487] Workqueue: events hot_add_req [hv_balloon]
[   73.276610] EIP: sparse_add_one_section+0xcb/0x12e
[   73.277715] Code: 45 ec e8 ce 5b 00 00 8b 55 e4 89 45 e8 89 d0 c1 e0 04 8d b0 c0 f7 1c c6 f6 80 c0 f7 1c c6 01 75 44 b0 ff b9 00 00 0a 00 89 df <f3> aa 89 f0 2d c0 f7 1c c6 c1 f8 04 3b 05 80 f7 1c c6 7e 05 a3 80
[   73.277753] EAX: 000000ff EBX: eb800000 ECX: 000a0000 EDX: 0000000d
[   73.277753] ESI: c61cf890 EDI: eb800000 EBP: dbd2be48 ESP: dbd2be2c
[   73.277753] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010046
[   73.277753] CR0: 80050033 CR2: eb800000 CR3: 06228000 CR4: 003406d0
[   73.277753] Call Trace:
[   73.277753]  __add_pages+0x89/0x100
[   73.277753]  arch_add_memory+0x3c/0x50
[   73.277753]  add_memory_resource+0x125/0x180
[   73.277753]  __add_memory+0xad/0x130
[   73.277753]  add_memory+0x2c/0x3a
[   73.277753]  hot_add_req+0x3de/0x60b [hv_balloon]
[   73.277753]  process_one_work+0x17b/0x340
[   73.277753]  worker_thread+0x39/0x3d0
[   73.277753]  kthread+0xf0/0x110
[   73.277753]  ? pwq_unbound_release_workfn+0xc0/0xc0
[   73.277753]  ? kthread_bind+0x30/0x30
[   73.277753]  ret_from_fork+0x2e/0x40
[   73.277753] Modules linked in: xfs libfc rfkill zram intel_rapl_perf i2c_piix4 pcspkr sg hv_utils hv_balloon joydev ext4 mbcache jbd2 loop nls_utf8 isofs sr_mod cdrom sd_mod 8021q garp ata_generic mrp stp llc hv_netvsc hv_storvsc scsi_transport_fc hyperv_keyboard hid_hyperv ata_piix libata hyperv_fb crc32_pclmul hv_vmbus serio_raw sunrpc xts lrw dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_intel raid1 raid0 iscsi_ibft squashfs cramfs be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi edd scsi_transport_iscsi
[   73.277753] CR2: 00000000eb800000
[   73.277753] ---[ end trace 6d1ee839ab38607d ]---
[   73.277753] EIP: sparse_add_one_section+0xcb/0x12e
[   73.277753] Code: 45 ec e8 ce 5b 00 00 8b 55 e4 89 45 e8 89 d0 c1 e0 04 8d b0 c0 f7 1c c6 f6 80 c0 f7 1c c6 01 75 44 b0 ff b9 00 00 0a 00 89 df <f3> aa 89 f0 2d c0 f7 1c c6 c1 f8 04 3b 05 80 f7 1c c6 7e 05 a3 80
[   73.277753] EAX: 000000ff EBX: eb800000 ECX: 000a0000 EDX: 0000000d
[   73.277753] ESI: c61cf890 EDI: eb800000 EBP: dbd2be48 ESP: c60633bc
[   73.277753] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010046
[   73.277753] CR0: 80050033 CR2: eb800000 CR3: 06228000 CR4: 003406d0
[   73.277753] Kernel panic - not syncing: Fatal exception
[   73.277753] Kernel Offset: 0x4400000 from 0xc1000000 (relocation range: 0xc0000000-0xe07effff)
[   73.277753] ---[ end Kernel panic - not syncing: Fatal exception ]---

Created attachment 286859 [details]
add printk's to see hv_balloon memory hot-add function

Not for production

Patched hv_balloon.c with lots of printk's (pr_info()).
I first thought hv_balloon was adding memory outside MAXMEM range,
but actually it wasn't the case; it was failing after 
first add_memory() of 128MB.

Is hot-add add_memory() in X86_32 functional?


[   56.719004] hv_balloon: Max. dynamic memory size: 1048576 MB
[   57.275031] hv_balloon: Received DM_MEM_HOT_ADD_REQUEST size 24
[   57.277162] hv_balloon: DM_MEM_HOT_ADD_REQUEST received size 24, should be 16
[   57.280537] hv_balloon: Received partial hot-add request
[   57.282442] hv_balloon: ha_wrk.ha_page_range{.finfo.start_page=65536, .finfo.page_cnt=16896}
[   57.284524] hv_balloon: ha_wrk.ha_region_range{.finfo.start_page=65536, .finfo.page_cnt=229888}
[   57.287729] hv_balloon: Invoking process_hot_add(pg_start=65536, pfn_cnt=16896, tg_start=65536, rg_sz=229888
[   57.290013] hv_balloon: process_hot_add: rg_size=229888 ha_region={ .start_pfn=65536, .ha_end_pfn=65536, .covered_start_pfn=65536, .covered_end_pfn=65536, .end_pfn=295424 }
[   57.293727] hv_balloon: add_memory(nid=0 PFN_PHYS(start_pfn=65536)=0x10000000, HA_CHUNK<<PAGE_SHIFT=134217728)
[   57.557961] BUG: unable to handle kernel paging request at d3fff000
[   57.564008] *pde = 00000000
[   57.566879] Oops: 0002 [#1] SMP
[   57.567597] CPU: 0 PID: 405 Comm: systemd-udevd Tainted: G            E     4.19.95-1.el8.i586 #1
[   57.567597] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012
[   57.567597] EIP: wp_page_copy+0x9c/0x780
[   57.567597] Code: 75 e8 85 f6 0f 84 9c 05 00 00 8b 45 e8 e8 1c 5d e7 ff 89 45 d4 8b 45 e4 e8 11 5d e7 ff 8b 55 d4 8d 78 04 8b 0a 83 e7 fc 89 d6 <89> 08 8b 8a fc 0f 00 00 89 88 fc 0f 00 00 89 c1 29 f9 89 55 d4 29
[   57.567597] EAX: d3fff000 EBX: c566df04 ECX: 00000000 EDX: c3868000
[   57.567597] ESI: c3868000 EDI: d3fff004 EBP: c566dec8 ESP: c566de98
[   57.567597] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00210282
[   57.567597] CR0: 80050033 CR2: d3fff000 CR3: 051bc000 CR4: 003406d0
[   57.567597] Call Trace:
[   57.567597]  do_wp_page+0x8a/0x600
[   57.567597]  handle_mm_fault+0x8b0/0xfb0
[   57.567597]  __do_page_fault+0x1c3/0x480
[   57.567597]  do_page_fault+0x25/0xf0
[   57.567597]  ? __do_page_fault+0x480/0x480
[   57.567597]  common_exception+0x11d/0x12e
[   57.567597] EIP: 0xb7b3f104
[   57.567597] Code: 29 f9 89 4c 24 10 83 f9 0f 0f 86 92 00 00 00 8b 45 40 8d 14 3e 8b 4c 24 0c 39 48 0c 75 74 8b 4c 24 0c 81 7c 24 10 ef 03 00 00 <89> 42 08 89 4a 0c 89 55 40 89 50 0c 76 0e c7 42 10 00 00 00 00 c7
[   57.567597] EAX: b7c657d8 EBX: 00001190 ECX: b7c657d8 EDX: 0169a108
[   57.567597] ESI: 016990f8 EDI: 00001010 EBP: b7c657a0 ESP: bfefcc80
[   57.567597] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00210293
[   57.567597] Modules linked in: rfkill snd_pcm snd_timer crc32_pclmul snd soundcore intel_rapl_perf pcspkr hv_balloon(E) hv_netvsc i2c_piix4 hyperv_fb hv_utils sg joydev ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod ata_generic ata_piix hyperv_keyboard hid_hyperv hv_storvsc scsi_transport_fc libata crc32c_intel serio_raw hv_vmbus
[   57.567597] CR2: 00000000d3fff000
[   57.567597] ---[ end trace 08a505f0f046453d ]---
[   57.567597] EIP: wp_page_copy+0x9c/0x780
[   57.567597] Code: 75 e8 85 f6 0f 84 9c 05 00 00 8b 45 e8 e8 1c 5d e7 ff 89 45 d4 8b 45 e4 e8 11 5d e7 ff 8b 55 d4 8d 78 04 8b 0a 83 e7 fc 89 d6 <89> 08 8b 8a fc 0f 00 00 89 88 fc 0f 00 00 89 c1 29 f9 89 55 d4 29
[   57.567597] EAX: d3fff000 EBX: c566df04 ECX: 00000000 EDX: c3868000
[   57.567597] ESI: c3868000 EDI: d3fff004 EBP: c566dec8 ESP: c32633bc
[   57.567597] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00210282
[   57.567597] CR0: 80050033 CR2: d3fff000 CR3: 051bc000 CR4: 003406d0
[   57.567597] Kernel panic - not syncing: Fatal exception
[   57.567597] Kernel Offset: 0x1600000 from 0xc1000000 (relocation range: 0xc0000000-0xc87effff)
[   57.567597] ---[ end Kernel panic - not syncing: Fatal exception ]---

When you squelch calling register_memory() from drivers/base/memory.c:init_memory_block(), it won't panic.

drivers/base/memory.c:register_memory(struct memory_block mem) initializes
struct device mem->dev and passes it down to device_register(&memory->dev),

but my wild guess is memory->dev initialization is not enough.
I don't have idea what is missing.
Current register_memory():

/*
 * register_memory - Setup a sysfs device for a memory block
 */
static
int register_memory(struct memory_block *memory)
{
        int ret;

        memory->dev.bus = &memory_subsys;
        memory->dev.id = memory->start_section_nr / sections_per_block;
        memory->dev.release = memory_block_release;
        memory->dev.groups = memory_memblk_attr_groups;
        memory->dev.offline = memory->state == MEM_OFFLINE;

        ret = device_register(&memory->dev);
        if (ret)
                put_device(&memory->dev);

        return ret;
}

drivers/base/memory.c:hotplug_memory_register() looks like this:

/*
 * need an interface for the VM to add new memory regions,
 * but without onlining it.
 */
int hotplug_memory_register(int nid, struct mem_section *section)
{
        int ret = 0;
        struct memory_block *mem;

        mutex_lock(&mem_sysfs_mutex);

        mem = find_memory_block(section);
        if (mem) {
                mem->section_count++;
                put_device(&mem->dev);
        } else {
                ret = init_memory_block(&mem, section, MEM_OFFLINE);
                if (ret)
                        goto out;
                mem->section_count++;
        }

out:
        mutex_unlock(&mem_sysfs_mutex);
        return ret;
}

There's nothing suspicious here, but difference between nonproblematic
memory add during boot is init_memory_block(,,MEM_ONLINE);
if I changed it to init_memory_block(,,MEM_ONLINE); in 
hotplug_memory_register(), it doesn't panic.

Seems like hot-adding offline memory (and callback for onlining it?)
seems to have some problem.

Created attachment 286941 [details]
do not __online_page_free() the hot-added memory in hv_balloon

Patch which makes it not panic, and the hot-added memory properly recognized by the Hyper-V guest

The patch posted as Comment 11
https://bugzilla.kernel.org/show_bug.cgi?id=206181#c11
seems to make kernel not panic and recognize the hot-added memory
triggered by memory pressure, and added by hv_balloon module.

I guess you shouldn't __online_page_free(pg) the page you just hot-added 
and brought online.
__online_page_free() should be called when memory hot-remove was requested,
which current hv_balloon driver doesn't implement now.

Any thoughts?
I don't know why memory hot-add was working for others before the patch.