Bug 213667

why NVM checksum is bad?
(in original NVM with good checksum the driver won't try to recovery checksum)

why NVM checksum is bad?
(in original NVM with good checksum the driver won't try to recovery checksum)

I looked at the attached log - first crash reported from i915 (graphic)
I would suggest work with your vendor and check your system.

Created attachment 297863 [details]
v1-0001-e1000e-Do-not-take-care-about-NVM-checksum

Created attachment 297865 [details]
v1-0001-e1000e-Do-not-take-care-about-NVM-checksumBACK

Hi,
Intel worked with Dell and their partners to confirm that the issue is related to incorrect checksum in GbE NVM. Dell and their partners are updating their process to write correct checksum in GbE NVM after they update anything in GbE NVM. In parallel, Intel will propose to remove checksum correction from e1000e driver due to new design change on GbE since recent platforms.

I upload two patches; I will request Dell and their partners to verify.

Rex

Confirmed the patches fix the issue.

Thanks for the confirmation. I will provide the commit id for tracking.
Please, work with your vendor to properly calculate checksum

commit	7b0827770dee8c5c08f97ea65568b834e60438f6

On the Dell Precision 7560 the e1000e driver doesn't load due to an error:

e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid

I tested this with Gentoo with kernel 5.10.76 and also tried Ubuntu 20.04.3 LTS
and 21.10 (kernel 5.13)

It seems that the patch causes the NVM is not updated, however, the checksum is
still verified, and if its not correct, the driver is not loaded, network card
doesn't work.

+		if (hw->mac.type < e1000_pch_cnp) {
+			data |= valid_csum_mask;
+			ret_val = e1000_write_nvm(hw, word, 1, &data);
+			if (ret_val)
+				return ret_val;
+			ret_val = e1000e_update_nvm_checksum(hw);
+			if (ret_val)
+				return ret_val;
+		}


I added some prints to the driver and it seems that the expected checksum is 
0xbaba but the calculated value is 0xbcba. I do not know if it helps but:

The e1000_probe function selects

const struct e1000_info *ei = e1000_info_tbl[ent->driver_data];
ent->driver_data = 14 => board_pch_tgp

I checked EEPROM and the selected element is:

#define E1000_DEV_ID_PCH_TGP_I219_LM14  0x15F9


I know I can remove the checksum verification code, but I am curious if there is
any other option. Otherwise, I will have to do this every time my distribution
updates the kernel.

Many thanks
Marcin

Can anyone advise me at least if this error is important? What does it really mean?
Can this problem be solved somehow or is it better to return the hardware?

-Marcin

Created attachment 299645 [details]
attachment-32179-0.html

Out of office. Expected delayed response

I have same identical problem on my Precision 7560 with Ubuntu 20.04 OEM, kernel 5.13.0-1019-oem:

+ e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid
+ e1000e: probe of 0000:00:1f.6 failed with error -5 

Is there any plans to fix this at the kernel level or is it necessary to request hardware replacement (even if it works correctly despite the wrong checksum)?
Unfortunately it is not possible for me to compile the form at each update nor is it possible on this model to overwrite the checksum ("Unable to write default configuration to EEPROM").

(In reply to pindi from comment #13)
> I have same identical problem on my Precision 7560 with Ubuntu 20.04 OEM,
> kernel 5.13.0-1019-oem:
> 
> + e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid
> + e1000e: probe of 0000:00:1f.6 failed with error -5 
> 
> Is there any plans to fix this at the kernel level or is it necessary to
> request hardware replacement (even if it works correctly despite the wrong
> checksum)?
> Unfortunately it is not possible for me to compile the form at each update
> nor is it possible on this model to overwrite the checksum ("Unable to write
> default configuration to EEPROM").

There is no need HW replacement I thought. You should contact your PC vendor and update NVM (part of BIOS)
There is no more option to update NVM by SW tools on new HW (drivers, etc...)

I have a brand new Dell Latitude E5420 with latest BIOS (1.14.1) and got the same error message as previous comment about NVM.

Running Ubuntu 20.04.3 LTS and kernel 5.11.0-43-generic #47~20.04.2-Ubuntu SMP

I don't see any download for a NVM update

What should I do ?

(In reply to Sasha Neftin from comment #14)
> (In reply to pindi from comment #13)
> > I have same identical problem on my Precision 7560 with Ubuntu 20.04 OEM,
> > kernel 5.13.0-1019-oem:
> > 
> > + e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid
> > + e1000e: probe of 0000:00:1f.6 failed with error -5 
> > 
> > Is there any plans to fix this at the kernel level or is it necessary to
> > request hardware replacement (even if it works correctly despite the wrong
> > checksum)?
> > Unfortunately it is not possible for me to compile the form at each update
> > nor is it possible on this model to overwrite the checksum ("Unable to
> write
> > default configuration to EEPROM").
> 
> There is no need HW replacement I thought. You should contact your PC vendor
> and update NVM (part of BIOS)
> There is no more option to update NVM by SW tools on new HW (drivers, etc...)

The only solution to the NIC problem was to replace the motherboard, as DELL does not allow overwriting of the ROM.
Any other solution applied does not solve the problem in any way (except by rewrite the e1000 module and recompiling the kernel, but it is not applicable in my case).
Regards.

On my side, it seems to work with Intel's upstream version 3.8.7 from https://sourceforge.net/projects/e1000/. I installed it with https://github.com/koljah-de/e1000e-dkms-debian, so as DKMS. The question remains regarding what to do in order to make the Ubuntu driver to work natively...

I also find strange that now the bootutil64e utility is complaining, while the NIC seems to work:

---
Connection to QV driver failed - please reinstall it!

Intel(R) Ethernet Flash Firmware Utility
BootUtil version 1.37.34.3
Copyright (C) 2003-2021 Intel Corporation

ERROR: The adapter (location 0:31.6) cannot be initialized due to inaccessible device memory.
Update the device driver and reboot the system before running this utility again.
Consult the utility documentation for more information.

Type BootUtil -? for help

Port Network Address Location Series  WOL Flash Firmware                Version
==== =============== ======== ======= === ============================= =======
  1   (Cannot initialize adapter)

and then I hit https://bugzilla.kernel.org/show_bug.cgi?id=213651 with a network speed issue...

This issue has been fixed since v5.14 by below commit

commit 4051f68318ca9f3d3becef3b54e70ad2c146df97
Author: Sasha Neftin <sasha.neftin@intel.com>
Date:   Sun Jul 18 07:10:31 2021 +0300

    e1000e: Do not take care about recovery NVM checksum
    
    On new platforms, the NVM is read-only. Attempting to update the NVM
    is causing a lockup to occur. Do not attempt to write to the NVM
    on platforms where it's not supported.
    Emit an error message when the NVM checksum is invalid.
    
    Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=213667
    Fixes: fb776f5d57ee ("e1000e: Add support for Tiger Lake")
    Suggested-by: Dima Ruinskiy <dima.ruinskiy@intel.com>
    Suggested-by: Vitaly Lifshits <vitaly.lifshits@intel.com>
    Signed-off-by: Sasha Neftin <sasha.neftin@intel.com>
    Tested-by: Dvora Fuxbrumer <dvorax.fuxbrumer@linux.intel.com>
    Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>

I'm running `5.15.0-41-generic` on Ubuntu 20.04 LTS and still have the issue, the interface is not working (This is a Dell 5420):


[    0.944599] e1000e: Intel(R) PRO/1000 Network Driver
[    0.944601] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    0.945221] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.166829] e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid
[    1.231542] e1000e: probe of 0000:00:1f.6 failed with error -5

Hi @all.

short question: what part of the problem was fixed with the patch discussed here? Correction of nvm-checksum seems not to be done in current version to avoid lockup. But what about testing the checksum? Will the driver be loaded if checksum is correct or not?

Still having the problem that the e1000e driver is not loaded due to checksum-error on actual ubuntu 22.04 lts with 5.17.0-1024-oem kernel on dell precision 7760 with current bios version dated 2022-11-23.

Rex Tsai stated that he had contact with Dell and their partners on 2021-07-14 - that's quite a long time ago...

so i'm wondering on wich side we have the problem. is this dells responsibility? or do i have to file a bug in the ubuntu-bugtracker?

thank,
karsten