Could you pass 'scsi_mod.use_blk_mq=1' in kernel command line and see if
it makes a difference?

And can you build the upstream kernel and see if can boot well?

The patch of 'scsi: hpsa: fix selection of reply queue' has been in 4.16.8
already, which supposes to fix this kind of issue, but it may not be
same issue.

Thanks,

(In reply to Lei Ming from comment #1)
> Could you pass 'scsi_mod.use_blk_mq=1' in kernel command line and see if
> it makes a difference?

Hello,
I tested with parameter that you suggested to me, but I am sorry no any result,
nothing changed

GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1"

> 
> And can you build the upstream kernel and see if can boot well?
> 

Upstream kernel, what do you mean?
I tested kernel "shipped" with Ubuntu, and also from Ubuntu ppa, and at last, before open this bug the "vanilla" stable kernel, build by myself from kernel.org

> The patch of 'scsi: hpsa: fix selection of reply queue' has been in 4.16.8
> already, which supposes to fix this kind of issue, but it may not be
> same issue.
> 
> Thanks,

however this "bug" come from kernel 4.14, kernel 4.13.16 from Ubuntu ppa is the last working, if you want and it's necessary to solve it, I can download old kernel (4.13 or/and 4.14) from kernel.org and build to check it, but give me 5-6 hours for building and testing kernel

thank you very much!

(In reply to Roberto M. from comment #2)
> (In reply to Lei Ming from comment #1)
> > Could you pass 'scsi_mod.use_blk_mq=1' in kernel command line and see if
> > it makes a difference?
> 
> Hello,
> I tested with parameter that you suggested to me, but I am sorry no any
> result,
> nothing changed
> 
> GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1"
> 
> > 
> > And can you build the upstream kernel and see if can boot well?
> > 
> 
> Upstream kernel, what do you mean?
> I tested kernel "shipped" with Ubuntu, and also from Ubuntu ppa, and at
> last, before open this bug the "vanilla" stable kernel, build by myself from
> kernel.org

I mean 'mainline' kernel in page of www.kernel.org

(In reply to Lei Ming from comment #3)

> I mean 'mainline' kernel in page of www.kernel.org

OK, my kernel 4.16.8 is downloaded and build from kernel.org, I will try before next Monday last 4.13 kernel from kernel.org, and first 4.14 kernel from kernel.org, build and install.

I think the "bug" was introduced in 4.14 mainstream kernel, but I only tested Ubuntu kernel from Ubuntu PPA archive, I will check from kernel.org and report here.

Downloaded from kernel.org 4.13.16 kernel, build and tested it works

uname -a
Linux Server-Ubuntu-01 4.13.16 #1 SMP Mon May 14 19:51:55 CEST 2018 x86_64 x86_64 x86_64 GNU/Linux

Downloaded from kernel.org 4.13.1 kernel, build and tested, it doesn't boot, same result as picture

I am not a programmer, but I think bug was introduced in 4.14.1 version, I can test patch or other kernels if necessary...
I am not able to check changelog file to see what is modified

Sorry I mean kernel 4.14.1 not 4.13.1

Created attachment 276781 [details]
P400 SCSI card removed

I removed card to test if it boot with live CD

Without P400 SCSI card and any live CD it boot regular, I think there are any doubt that is 100% HPSA driver problem, it crash kernel at boot

Here the video :

https://www.youtube.com/watch?v=d05vwUg5WtI

I tested to pass at grub command line parameters to disable spectre and meltdown problem, but now I am sure

I notice this:

13:08.0 RAID bus controller [0104]: Hewlett-Packard Company Smart Array E200i (SAS Controller) [103c:3238]
        Subsystem: Hewlett-Packard Company Smart Array E200i [103c:3211]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr+ DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR+ INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 29
        Region 0: Memory at fde80000 (64-bit, non-prefetchable) [size=512K]
        Region 2: I/O ports at 5000 [size=256]
        Region 3: Memory at fde70000 (32-bit, non-prefetchable) [size=32K]
        [virtual] Expansion ROM at fde00000 [disabled] [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: cciss
        Kernel modules: cciss, hpsa

It looks like both the cciss drivere and the hpsa driver are competing for the same controller E200i. One is a block driver, the other is a SCSI driver.

So, we need to blacklist one of them.

What is the format for blacklisting a driver for Ubuntu?

rdblacklist=???

(In reply to Don from comment #9)
> I notice this:
> 
> 13:08.0 RAID bus controller [0104]: Hewlett-Packard Company Smart Array
> E200i (SAS Controller) [103c:3238]
>         Subsystem: Hewlett-Packard Company Smart Array E200i [103c:3211]
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr+ DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR+ INTx-
>         Latency: 64
>         Interrupt: pin A routed to IRQ 29
>         Region 0: Memory at fde80000 (64-bit, non-prefetchable) [size=512K]
>         Region 2: I/O ports at 5000 [size=256]
>         Region 3: Memory at fde70000 (32-bit, non-prefetchable) [size=32K]
>         [virtual] Expansion ROM at fde00000 [disabled] [size=16K]
>         Capabilities: <access denied>
>         Kernel driver in use: cciss
>         Kernel modules: cciss, hpsa
> 
> It looks like both the cciss drivere and the hpsa driver are competing for
> the same controller E200i. One is a block driver, the other is a SCSI driver.
> 
> So, we need to blacklist one of them.
> 
> What is the format for blacklisting a driver for Ubuntu?
> 
> rdblacklist=???

Actually, I see that the cciss driver has claimed the E200i first.
So, blacklist the hpsa driver.

The support for this controller was added by Hannes Reinecke some time ago and perhaps there is a bug in the init code when the device is already claimed by another driver....

commit 135ae6edeb51979d0998daf1357f149a7d6ebb08
Author: Hannes Reinecke <hare@suse.de>
Date:   Tue Aug 15 08:58:04 2017 +0200

    scsi: hpsa: add support for legacy boards
    
    Add support for legacy boards, ensuring to enable the driver for
    those boards only when 'hpsa_allow_any' is set.
    The attribute 'legacy_board' is set to '1' if the device is
    a legacy board, and '0' otherwise.
    
    Signed-off-by: Hannes Reinecke <hare@suse.com>
    Acked-by: Don Brace <don.brace@microsemi.com>
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

One more thing. the hpsa_allow_any module parameter was removed by this patch:

commit c8cd71f1f32a6227ecadbbbaaf3147a41292ecb5
Author: Hannes Reinecke <hare@suse.de>
Date:   Tue Aug 15 08:58:09 2017 +0200

    scsi: hpsa: Remove 'hpsa_allow_any' module option
    
    As the cciss driver has been removed there are no overlapping
    PCI IDs anymore, and the 'hpsa_allow_any' flag can be removed.
    
    Signed-off-by: Hannes Reinecke <hare@suse.com>
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>


And, since the cciss driver has been removed from the kernel but still show up in your lspci output shows that the cciss driver is still in your initramfs?

lsinitrd <path to your boot image> | egrep "cciss|hpsa"

If you remove cciss from here, your devices will change over to SCSI devices. Depending on your fstab and grub, your system may not boot.

Hi Don,

Thank you for your answer

1) for your comment #9 you saw my last lspci output with a working kernel, I mean the original one from kernel.org 4.13.16, it boot, there is still cciss drive inside, it was removed in 4.14 kernel and above, so no lspci output  from kernel > 4.13.16 because it crash at boot, I can only made a video

please habe a look :
https://kernelnewbies.org/Linux_4.14
https://www.systutorials.com/docs/linux/man/4-hpsa/


2) for comment #10 and #11 I tested hpsa_allow_any, if I use it, it doesn't boot also with a working kernel, for me it means that my HP P400 card work only with cciss driver.
I read P400 card use cciss (and maybe cciss use hpsa) instead e200i use hpsa, so is why I have both drivers loaded with kernel 4.13.16, but I am not sure at 100%

3) check https://access.redhat.com/solutions/874443

4) maybe is my problem I misunderstood that cciss driver was removed because it is/will been replaced by hpsa driver, it will take care also of old board working only cciss, but it not seems so, I don't get why at booting it crash  so without any possibility to do anything I have to remove power

https://github.com/torvalds/linux/blob/master/drivers/scsi/hpsa.c
https://patchwork.kernel.org/patch/9947333/

sorry for my bad English, I think hpsa didn't be tested on older card, I am available for test some patch and so on...

bye

I mean I can't blacklist cciss, I can't blacklist a module that doens't exist anymore

I can't blacklist HPSA, because from kernel 4.14 it's only driver for this boards.. (in my case E200i e P400)

I'm wondering how you updated.

If you were booted from a cciss block driver before, the disk mapping would be to a /dev/cciss/c0dX boot device.

Do you use by-label, by-uuid, ...?

If not and you simply updated the kernel and the kernel switched to hpsa, your disk mapping would not be changed over to /dev/sdX, it would still be to /dev/cciss/...

Can you boot into rescue mode? If so, can you post your grub.cfg, /etc/fstab files?


Also, do you know how to obtain ilo vsp console output? 

IE. In RBSU enabling VSP logging and setting the BIOS console and BAUD(115200), then
    updating the boot line with console=ttyS0,11500 console=tty0

Then script -c "ssh <IP of ilo" > /tmp/E200boot.log from another machine?

This will capture what is happening and you can post E200boot.log

Hi,
my answer below 

(In reply to Don from comment #14)
> I'm wondering how you updated.

It is a prolian ML350 G5 with Ubuntu 16.04 LTS, I upgraded to 18.04 LTS, after upgrade at first boot with his own official 4.15 kernel, it crashed, I tryed to choose 4.4 kernel (from grub, the previous LTS kernel from 16.04) and it boot

that's all

I tested all kernels from ubuntu kernel PPA, all <= 4.13.16 boot, from 4.14 not..Also tested kernels from kernel.org (compiled by myself), before open this bug, same results

> 
> If you were booted from a cciss block driver before, the disk mapping would
> be to a /dev/cciss/c0dX boot device.
> 
> Do you use by-label, by-uuid, ...?

by uuid

> 
> If not and you simply updated the kernel and the kernel switched to hpsa,
> your disk mapping would not be changed over to /dev/sdX, it would still be
> to /dev/cciss/...
> 
> Can you boot into rescue mode? If so, can you post your grub.cfg, /etc/fstab
> files?

I will post here and add as attachment asap (I already know that migrating from cciss to hpsa I have to change fstab, but I tryed some live cd, to check and test with fdisk new partition, but no any luck to boot, with card inside, removing it, it boot with live cd like video on youtube)
> 
> 
> Also, do you know how to obtain ilo vsp console output? 
> 
> IE. In RBSU enabling VSP logging and setting the BIOS console and
> BAUD(115200), then
>     updating the boot line with console=ttyS0,11500 console=tty0
> 
> Then script -c "ssh <IP of ilo" > /tmp/E200boot.log from another machine?
> 
> This will capture what is happening and you can post E200boot.log

No, I will surely try it, please give me time, if I can ask, before start I think that I have to do it booting with newest kernel from kernel.org? 4.17.2?? (Ubuntu official kernel are patched) no any problem for me to download and compile it

Really thank you for your answers (from RHEL faq page only solution is to buy a supported hp scsi card )

thanks again, 
bye

Created attachment 277077 [details]
boot not working log

this is the output of last linux kernel 4.17.3, every 120 second I get a message that for me is a crash

Created attachment 277079 [details]
boot woking log with last working kernel

(In reply to Don from comment #14)

Hello,
I am late, I am really sorry


> If you were booted from a cciss block driver before, the disk mapping would
> be to a /dev/cciss/c0dX boot device.
> 
> Do you use by-label, by-uuid, ...?

Here blkid  output:

/dev/cciss/c0d0p2: UUID="229d158a-4614-474d-ac53-82c403c47e49" TYPE="ext4" PARTUUID="785a40ac-5098-44bb-b7f4-be917bd359df"
/dev/cciss/c0d0p3: UUID="d5514912-b652-4ac6-8273-65c17e96996f" TYPE="swap" PARTUUID="0c7bc0a1-a917-40b2-9857-f3f36ea9901c"
/dev/cciss/c0d0p4: UUID="292a7797-5a28-4ead-ae7b-483a467e8a63" TYPE="ext4" PARTUUID="e892e834-8bf2-4e5d-a642-6f0ecd1b4559"
/dev/cciss/c0d0p5: UUID="58573ae4-9cb1-4ffb-a3a6-59b96dc25b63" TYPE="ext4" PARTUUID="db7acdaa-503e-42b6-89ab-f292242092a7"
/dev/sda1: LABEL="System Reserved" UUID="BC2EEBBB2EEB6D38" TYPE="ntfs" PARTUUID="61637073-01"
/dev/sda2: UUID="C0EEECA7EEEC96C6" TYPE="ntfs" PARTUUID="61637073-02"
/dev/cciss/c0d0: PTUUID="23b2be73-1e88-40c4-8990-7f285985743b" PTTYP


here these is fdisk -l output :

Disk /dev/cciss/c0d0: 5,5 TiB, 6001026883584 bytes, 11720755632 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 23B2BE73-1E88-40C4-8990-7F285985743B

Dispositivo             Start        Fine     Settori  Size Tipo
/dev/cciss/c0d0p1        2048        4095        2048    1M BIOS boot
/dev/cciss/c0d0p2        4096      618495      614400  300M Linux filesystem
/dev/cciss/c0d0p3 11642626048 11720755199    78129152 37,3G Linux swap
/dev/cciss/c0d0p4      618496    84504575    83886080   40G Linux filesystem
/dev/cciss/c0d0p5    84504576 11642626047 11558121472  5,4T Linux filesystem

Partition table entries are not in disk order.


Disk /dev/sda: 189,9 GiB, 203928109056 bytes, 398297088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x61637073

Dispositivo Avvio  Start      Fine   Settori   Size Id Tipo
/dev/sda1   *       2048    206847    204800   100M  7 HPFS/NTFS/exFAT
/dev/sda2         206848 398295039 398088192 189,8G  7 HPFS/NTFS/exFAT


> 
> If not and you simply updated the kernel and the kernel switched to hpsa,
> your disk mapping would not be changed over to /dev/sdX, it would still be
> to /dev/cciss/...
> 
> Can you boot into rescue mode? If so, can you post your grub.cfg, /etc/fstab
> files?


here /etc/default/grub:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 4.13.16-041316-generic"
#GRUB_HIDDEN_TIMEOUT="0"
GRUB_HIDDEN_TIMEOUT_QUIET="true"
GRUB_TIMEOUT="40"
GRUB_DISTRIBUTOR="`lsb_release -i -s 2> /dev/null || echo Debian`"
#GRUB_CMDLINE_LINUX_DEFAULT=""
#GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=ttyS1 console=ttyS0,115200"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL="console"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE="640x480"

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID="true"

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

GRUB_SAVEDEFAULT="false"
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,115200"
#UNNAMED_OPTION=""

GRUB_DISABLE_OS_PROBER="true"

here cat /boot/grub/grub.cfg :

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="Advanced options for Ubuntu>Ubuntu, with Linux 4.13.16-041316-generic"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function recordfail {
  set recordfail=1
  if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1
terminal_input console serial
terminal_output console serial
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=40
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=40
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
#set_background_image "images/tile.png";

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
if background_color 0,0,0; then
  clear
fi
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux_proxy ###

function gfxmode {
        set gfxpayload="${1}"
        if [ "${1}" = "keep" ]; then
                set vt_handoff=vt.handoff=1
        else
                set vt_handoff=
        fi
}
if [ "${recordfail}" != 1 ]; then
  if [ -e ${prefix}/gfxblacklist.txt ]; then
    if hwmatch ${prefix}/gfxblacklist.txt 3; then
      if [ ${match} = 0 ]; then
        set linux_gfx_mode=keep
      else
        set linux_gfx_mode=text
      fi
    else
      set linux_gfx_mode=text
    fi
  else
    set linux_gfx_mode=keep
  fi
else
  set linux_gfx_mode=text
fi
export linux_gfx_mode



menuentry "Ubuntu" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-292a7797-5a28-4ead-ae7b-483a467e8a63' {
        recordfail
        load_video
        gfxmode $linux_gfx_mode
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
        else
          search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
        fi
        linux   /vmlinuz-4.17.3-vanilla root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro console=ttyS1 console=ttyS0,115200 console=ttyS1,115200
        initrd  /initrd.img-4.17.3-vanilla
}
submenu "Advanced options for Ubuntu"{
menuentry "Ubuntu, with Linux 4.13.16-041316-generic" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.13.16-041316-generic-advanced-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.13.16-041316-generic ...'
                linux   /vmlinuz-4.13.16-041316-generic root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro console=ttyS1 console=ttyS0,115200 console=ttyS1,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.13.16-041316-generic
}
menuentry "Ubuntu, with Linux 4.13.16-041316-generic (recovery mode)" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.13.16-041316-generic-recovery-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.13.16-041316-generic ...'
                linux   /vmlinuz-4.13.16-041316-generic root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro recovery nomodeset console=ttyS1 console=ttyS0,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.13.16-041316-generic
}
menuentry "Ubuntu, with Linux 4.15.0-24-generic" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-24-generic-advanced-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.15.0-24-generic ...'
                linux   /vmlinuz-4.15.0-24-generic root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro console=ttyS1 console=ttyS0,115200 console=ttyS1,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.15.0-24-generic
}
menuentry "Ubuntu, with Linux 4.15.0-24-generic (recovery mode)" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-24-generic-recovery-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.15.0-24-generic ...'
                linux   /vmlinuz-4.15.0-24-generic root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro recovery nomodeset console=ttyS1 console=ttyS0,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.15.0-24-generic
}
menuentry "Ubuntu, with Linux 4.17.3-vanilla (recovery mode)" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.17.3-vanilla-recovery-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.17.3-vanilla ...'
                linux   /vmlinuz-4.17.3-vanilla root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro recovery nomodeset console=ttyS1 console=ttyS0,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.17.3-vanilla
}
menuentry "Ubuntu, with Linux 4.17.3-vanilla" --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.17.3-vanilla-advanced-292a7797-5a28-4ead-ae7b-483a467e8a63' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
                else
                  search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
                fi
                echo    'Loading Linux 4.17.3-vanilla ...'
                linux   /vmlinuz-4.17.3-vanilla root=UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 ro console=ttyS1 console=ttyS0,115200 console=ttyS1,115200
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.17.3-vanilla
}
}
### END /etc/grub.d/10_linux_proxy ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry 'Memory test (memtest86+)' {
        insmod part_gpt
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
        else
          search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
        fi
        knetbsd /memtest86+.elf
}
menuentry 'Memory test (memtest86+, serial console 115200)' {
        insmod part_gpt
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  229d158a-4614-474d-ac53-82c403c47e49
        else
          search --no-floppy --fs-uuid --set=root 229d158a-4614-474d-ac53-82c403c47e49
        fi
        linux16 /memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry "Windows 2008"{
        set root='(hd1,1)'
        search --no-floppy --fs-uuid --set BC2EEBBB2EEB6D38
        drivemap -s (hd0) ${root}
        chainloader +1
}
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###


here cat /etc/fstab :

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/cciss/c0d0p4 during installation
UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 /               ext4    errors=remount-ro 0       1
# /boot was on /dev/cciss/c0d0p2 during installation
UUID=229d158a-4614-474d-ac53-82c403c47e49 /boot           ext4    defaults        0       2
# /home was on /dev/cciss/c0d0p5 during installation
UUID=58573ae4-9cb1-4ffb-a3a6-59b96dc25b63 /home           ext4    defaults        0       2
# swap was on /dev/cciss/c0d0p3 during installation
UUID=d5514912-b652-4ac6-8273-65c17e96996f none            swap    sw              0       0
#win2008
UUID=C0EEECA7EEEC96C6                     /mnt/win2008    ntfs    defaults,auto,users,uid=1000,gid=1000,nls=utf8,umask=0222,rw  0       0


> 
> 
> Also, do you know how to obtain ilo vsp console output? 

please see the attachment, I done it


please if I can ask I see  every 120 second repeat this "error" :

[  488.412034] INFO: task systemd-udevd:211 blocked for more than 120 seconds.
[  488.445659]       Tainted: G          I       4.17.3-vanilla #1
[  488.474729] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  488.514686] systemd-udevd   D    0   211    195 0x80000004
[  488.600643] Call Trace:
[  488.612778]  ? __schedule+0x291/0x870
[  488.630664]  ? sched_clock+0x5/0x10
[  488.647681]  schedule+0x28/0x80
[  488.663709]  schedule_timeout+0x1ee/0x380
[  488.681632]  ? pwq_adjust_max_active+0x90/0xe0
[  488.700638]  io_schedule_timeout+0x19/0x40
[  488.720681]  wait_for_completion_io+0xc0/0x150
[  488.742644]  ? wake_up_q+0x70/0x70
[  488.759070]  hpsa_scsi_do_simple_cmd.isra.64+0xa9/0xf0 [hpsa]
[  488.788656]  hpsa_scsi_do_simple_cmd_with_retry+0x65/0x170 [hpsa]
[  488.819659]  hpsa_scsi_do_inquiry+0x62/0xc0 [hpsa]
[  488.842660]  hpsa_init_one+0x165e/0x1d90 [hpsa]
[  488.866860]  local_pci_probe+0x42/0xa0
[  488.880659]  ? pci_assign_irq+0x27/0x130
[  488.896676]  pci_device_probe+0x141/0x1b0
[  488.916631]  driver_probe_device+0x30b/0x480
[  488.935888]  __driver_attach+0xb8/0xe0
[  488.952676]  ? driver_probe_device+0x480/0x480
[  488.974718]  bus_for_each_dev+0x76/0xc0
[  488.994699]  bus_add_driver+0x161/0x260
[  489.013748]  ? 0xffffffffc03b5000
[  489.029719]  driver_register+0x57/0xc0
[  489.049646]  ? 0xffffffffc03b5000
[  489.065737]  hpsa_init+0x3d/0x1000 [hpsa]
[  489.076638]  do_one_initcall+0x46/0x1c8
[  489.095737]  ? _cond_resched+0x15/0x40
[  489.113687]  ? kmem_cache_alloc_trace+0x3e/0x1c0
[  489.136643]  do_init_module+0x5b/0x201
[  489.154635]  load_module+0x1989/0x1f10
[  489.173672]  ? __do_sys_finit_module+0xe9/0x110
[  489.193708]  __do_sys_finit_module+0xe9/0x110
[  489.215686]  do_syscall_64+0x55/0x110
[  489.233665]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  489.260659] RIP: 0033:0x7f17748b2839
[  489.277654] RSP: 002b:00007ffcdc23f928 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  489.317686] RAX: ffffffffffffffda RBX: 00005646293181e0 RCX: 00007f17748b2839
[  489.344659] RDX: 0000000000000000 RSI: 00007f17745910e5 RDI: 000000000000000c
[  489.381732] RBP: 00007f17745910e5 R08: 0000000000000000 R09: 00007ffcdc23fa40
[  489.415636] R10: 000000000000000c R11: 0000000000000246 R12: 0000000000000000
[  489.453682] R13: 00005646292fa310 R14: 0000000000020000 R15: 00005646293181e0


thank you very much and sorry for my terrible English and my answer so late

or better with any live cd with kernel >= 14.1 and array adapter inserted, don't boot and kernel crash, without raid adapter (removed from pcie slot) it works

100% HPSA bug that crash

The file boot_working_4.13.16.log show the cciss block driver managing your boot disk, and the file boot_not_working_kernel_4-14_and_above.log show the
change to the hpsa driver.

The 3.4.20-125 driver does have Hannes's patches to support legacy cciss controllers, I see the following:

Gave up waiting for root file system device.  Common problems:^M
 - Boot args (cat /proc/cmdline)^M
   - Check rootdelay= (did the system wait long enough?)^M
 - Missing modules (cat /proc/modules; ls /dev)^M
ALERT!  UUID=292a7797-5a28-4ead-ae7b-483a467e8a63 does not exist.  Dropping to a shell!^M

This UUID is your root '/' partition.

Later on we see:

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)^M
Enter 'help' for a list of built-in commands.^M
^M
(initramfs) ^[[6n[  242.652041] INFO: task systemd-udevd:211 blocked for more than 120 seconds.^M
[  242.684084]       Tainted: G          I       4.17.3-vanilla #1^M
[  242.714202] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M


So, the hpsa driver is attempting to send commands down to the controller, but they are not coming back. 

Do you see any messages in ilo IML logs?

Checked hpasmcli no IML messages

Thank you very much for explain me about the bug, any news about the patch? I am available to test it

perfect timing!!

i am having the exact same trying to boot 4.15.15 kernel (built from source) (yes same HP400 smart array controller)

it seems the config parameter CONFIG_BLK_CPQ_CISS_DA (and its routines?) has been removed. It appears it was dropped with kernel 4.15; but havent been able to find any details about this on kernel.org Doc, wiki, git or other websites.

and using the kernel boot parameter "hpsa_allow_any=1" does work either.

Guess the question is what replaced CONFIG_BLK_CPQ_CISS_DA if anything?

thanks

Error: above should read:
and using the kernel boot parameter "hpsa_allow_any=1" does NOT work either.

UPDATE:
* found out CONFIG_BLK_CPQ_CISS_DA was removed and Replaced with HPSA driver.
* the "hpsa_allow_any=1" boot parameter does sort-of find the raid/disk -- and creates a "/dev/disk/by-path/pci-0000:00:06.1-scsi-0:0:0:0" -- but thats it. (no "by-uuid" or "by-id" entries)
* blkid just hangs completely when tried

I was able to take the kernel.org driver and load over a P400, which I believe is older than your E200.

The E200 requires a working cache module for RAID 5 volumes. But you are using a  RAID 0...correct?


[ 2674.521073] HP HPSA Driver (v 3.4.20-125)
[ 2674.531481] hpsa 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 2674.552657] hpsa 0000:04:00.0: Logical aborts not supported
[ 2674.560739] hpsa 0000:04:00.0: HP SSD Smart Path aborts not supported
[ 2674.661018] scsi host6: hpsa
[ 2674.664782] hpsa can't handle SMP requests
[ 2674.670676] hpsa 0000:0a:00.0: legacy board ID: 0x3234103c
[ 2674.691098] hpsa 0000:0a:00.0: legacy board ID: 0x3234103c
[ 2674.696897] hpsa 0000:0a:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 2674.720169] hpsa 0000:0a:00.0: Logical aborts not supported
[ 2674.727691] hpsa 0000:0a:00.0: HP SSD Smart Path aborts not supported
[ 2674.868022] scsi host7: hpsa
[ 2674.871929] hpsa can't handle SMP requests
[ 2675.112139] hpsa 0000:0a:00.0: scsi 7:0:0:0: masked Direct-Access     HP       DF146BB6C2       PHYS DRV SSDSmartPathCap- En- Exp=0
[ 2675.113001] hpsa 0000:0a:00.0: scsi 7:1:0:0: added Direct-Access     HP       LOGICAL VOLUME   RAID-1(+0) SSDSmartPathCap- En- Exp=1
[ 2675.113001] hpsa 0000:0a:00.0: scsi 7:3:0:0: added RAID              HP       P400             controller SSDSmartPathCap- En- Exp=1


[root@localhost scsi]# lsscsi
[0:0:0:0]    disk    ASMT     2105             0     /dev/sdb 
[6:0:0:0]    storage HP       P421             6.68  -        
[6:1:0:0]    disk    HP       LOGICAL VOLUME   6.68  /dev/sda 
[7:1:0:0]    disk    HP       LOGICAL VOLUME   7.24  /dev/sdc 
[7:3:0:0]    storage HP       P400             7.24  -

What is the version of your Smart Array controller firmware?
i have a mix of 10 HP DL380 G5 machines with different versions of Smart Array P400 firmware (some have v5.06, some v5.20 and some v6.86) and only the servers with SmartArray firmware 5.06 fail to boot with new kernel.
the ones with 5.20 and 6.86 booted just fine after upgrade to kernel 4.15.0-23.
Can you try to update your firmware on the disk controllers with HP SPP and see if it helps?

(In reply to Don from comment #25)
> I was able to take the kernel.org driver and load over a P400, which I
> believe is older than your E200.
> 
> The E200 requires a working cache module for RAID 5 volumes. But you are
> using a  RAID 0...correct?
> 

No Raid 1-0, E200 isn't used, cable are connected to P400, I solved it, old FW doesn't work with HPSA driver

> Can you try to update your firmware on the disk controllers with HP SPP and
> see if it helps?

It solved my problem! thank you!, you are better than RHEL support, on their faq is written no solution, please change the card with one compatible (I have bought for less 30 euro a HP P410)

My FW was 4.12 now 7.24

bug, isn't a bug, my error, old FW not compatible with HPSA driver

Old FW card problem, 4.12 updated to 7.24, everything works

(In reply to Roberto M. from comment #27)
> (In reply to Don from comment #25)
> > I was able to take the kernel.org driver and load over a P400, which I
> > believe is older than your E200.
> > 
> > The E200 requires a working cache module for RAID 5 volumes. But you are
> > using a  RAID 0...correct?
> > 
> 
> No Raid 1-0, E200 isn't used, cable are connected to P400, I solved it, old
> FW doesn't work with HPSA driver

No Raid 0 sorry, however testing it seems works at 100%

Hi

I had the same issue. HP Smart Array P400 controller in a HP DL385 migrating from Ubuntu 16.04 to 18.04, so a kernel going from 4.10.xx to 4.15.yy and kernel couldn't find the disk controller.

I tried all the hpsa_xx, cciss_xx and rootdelay combinations in the boot parameters with no results. I ended up upgrading the P400 firmware (4.08 to 7.24) and it worked straight without any parameter change.

Server and disk access is still ok after a few days of testing.

Thanks for the tip.
Loic

(In reply to Loic S. from comment #31)
> Hi
> 
> I had the same issue. HP Smart Array P400 controller in a HP DL385 migrating
> from Ubuntu 16.04 to 18.04, so a kernel going from 4.10.xx to 4.15.yy and
> kernel couldn't find the disk controller.
> 
> I tried all the hpsa_xx, cciss_xx and rootdelay combinations in the boot
> parameters with no results. I ended up upgrading the P400 firmware (4.08 to
> 7.24) and it worked straight without any parameter change.
> 
> Server and disk access is still ok after a few days of testing.
> 
> Thanks for the tip.
> Loic

Hello,
I am happy to know I helped you but you should tahnks "Dmytro M" comment 26 he solved it

bye!