Smart Array P410 is a RAID controller that does not expose anything about the SAS fabric (expanders or physical drives) to the OS; it just presents logical drives.  You need to use HP management tools to download firmware to expanders and drives behind the controller.

If you attach the expander to an HBA, then you can use sg_write_buffer to download microcode.  Version 1.39 added a --bpw option to download in small chunks - an expander might require a small chunk size like 4 KiB (see http://sg.danny.cz/sg for the latest sg3_utils code).

Although they sound like DOS programs, .scexe files from HP are really linux executable scripts that you can view in a text editor (at least the first part).  You can do this to extract the files:
mkdir tmp
./CPxyzzy.scexe --unpack=tmp

One of those is the binary image that can be sent with sg_write_buffer.

BTW, use lsscsi -g to see the /dev/sgNN paths for all the devices presented by the LLD, including expanders.

In this case the hardware seems broken.

The P410 controller is only used for upgrading the HP SAS expander firmware. This time we started with upgrading the controller firmware:
===
./ccissflash

	This program consists of two phases: device discovery and device update.
	No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

Do you want to upgrade the device that has older ROM?
(yes/no) yes
1 devices will be updated.
Updating: P410 Slot: 2 from [5.70] to [6.60]
Updating: P410 Slot: 2 from [5.70] to [6.60]

As part of the reboot process, you must power cycle the server and any external array storage devices.
===

After power cycling the first three reboots were ok. Then a controller lockup 0x00130000 occurred 468 seconds after loading the hpsa module. Since then only one boot/module loading was ok and since then the "time outs" are occurring:
 
===
[  121.613269] HP HPSA Driver (v 3.4.0-1)
[  121.613344] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[  121.613447] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[  121.613669] hpsa 0000:01:00.0: MSIX
[  121.613770] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[  121.613843] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[  121.613909] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[  121.613981] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[  121.614046] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[  121.614117] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[  121.614182] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[  121.614251] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[  121.634560] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[  127.171906] scsi4 : hpsa
[  127.174883] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[  127.175961] scsi 4:3:0:0: RAID              HP       P410             6.60 PQ: 0 ANSI: 5
[  127.176335] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[  424.473998] HP HPSA Driver (v 3.4.0-1)
===
[   16.127857] HP HPSA Driver (v 3.4.0-1)
[   16.127927] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   16.128020] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   16.128231] hpsa 0000:01:00.0: MSIX
[   16.128327] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[   16.128394] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[   16.128460] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[   16.128524] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[   16.128590] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[   16.128660] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[   16.128726] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[   16.128791] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[   16.149110] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[   34.767566] scsi4 : hpsa
[   34.770583] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   34.771054] scsi 4:3:0:0: RAID              HP       P410             6.60 PQ: 0 ANSI: 5
===
[   48.389920] HP HPSA Driver (v 3.4.4-1)
[   48.389985] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   48.390059] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   48.390239] hpsa 0000:01:00.0: MSIX
[   48.390312] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   48.390362] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   48.390453] hpsa 0000:01:00.0: Logical aborts not supported
[   48.410617] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   53.947809] scsi host4: hpsa
[   53.953488] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   53.953986] scsi 4:3:0:0: RAID              HP       P410             6.60 PQ: 0 ANSI: 5
===
[   17.076307] HP HPSA Driver (v 3.4.4-1)
[   17.076388] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   17.076483] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   17.076695] hpsa 0000:01:00.0: MSIX
[   17.076782] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   17.076837] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   17.076942] hpsa 0000:01:00.0: Logical aborts not supported
[   17.097108] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   33.659110] scsi host4: hpsa
[   33.663958] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   33.664360] scsi 4:3:0:0: RAID              HP       P410             6.60 PQ: 0 ANSI: 5
[   33.665053] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[  485.066696] hpsa 0000:01:00.0: Controller lockup detected: 0x00130000 
===
[   49.567169] HP HPSA Driver (v 3.4.4-1)
[   49.567251] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   49.567389] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   49.567576] hpsa 0000:01:00.0: MSIX
[   49.567649] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   49.567705] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   49.567799] hpsa 0000:01:00.0: Logical aborts not supported
[   49.587966] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   55.125143] scsi host4: hpsa
[   55.129439] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   55.129832] scsi 4:3:0:0: RAID              HP       P410             6.60 PQ: 0 ANSI: 5
===
[   38.305788] HP HPSA Driver (v 3.4.4-1)
[   38.305854] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   38.306052] hpsa 0000:01:00.0: MSIX
[   38.306127] hpsa 0000:01:00.0: irq 29 for MSI/MSI-X
[   38.306179] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[  163.163631] hpsa 0000:01:00.0: board not ready, timed out.
===
[   23.324890] HP HPSA Driver (v 3.4.0-1)
[   23.324957] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
[   23.325060] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   23.325281] hpsa 0000:01:00.0: MSIX
[   23.325379] hpsa 0000:01:00.0: irq 45 for MSI/MSI-X
[   23.325453] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[   23.325518] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[   23.325591] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[   23.325655] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[   23.325728] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[   23.325793] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[   23.325865] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[  148.185212] hpsa 0000:01:00.0: board not ready, timed out.
===

Right now loading the hpsa module permanently errors out with "board not ready, timed out" and the "DS1: Diagnostics Error" LED turns Amber (=One of the server diagnostics utilities has detected a controller error.)

Seems like a hardware issue, therefore this case can be closed.

Controller lockup means the firmware crashed (e.g. an ASSERT statement triggered).  That can be due to hardware problems or firmware bugs.  You might see if it boots with nothing attached; then with one expander; then with two; then with drives attached to the expanders.

If you're not really using the P410, though, just use sg_write_buffer through the Adaptec 7805H HBA.

@Robert Elliott
Thanks for all your help.
The issue turns out not to be the HP P410 controller.
Using another mainboard the same HP P410 controller is ok.

After replacing the faulty hardware, the HP P410 boots normally again.

However the P410+hpsa still does not detect either of the HP SAS expander cards, neither does ./ccissflash detect any connected HP SAS expander.

LED status of the HP SAS expander card:
CR1 = solid green
CR2 = solid green
CR3 = solid green
CR4 = off
CR5 = off
CR6 = blinks green

Note: the Adaptec pm80xx module fails loading in sas expander discovery, I will file a separate bug report for that.

The HP Smart Array P410 still has its issues: sometimes the diagnostics LED turns amber, other times it works ok.

As a fall back a Marvell controller detects the HP SAS expander card fine:
# lsscsi -g | grep HP
[6:0:0:0]    enclosu HP       HP SAS EXP Card  2.08  -          /dev/sg1

I am not able to figure out a correct sg_write_buffer command. Run from the directory where the unpacked binary image is stored:
# sg_write_buffer --bpw=4096 -v --in=PUF21000.bin /dev/sg1
tried to read 8388608 bytes from PUF21000.bin, got 1509632 bytes
will write 1509632 bytes
sending write buffer, mode=0x0, mspec=0, id=0,  offset=0, len=4096
    Write buffer cmd: 3b 00 00 00 00 00 00 10 00 00
write buffer:  Fixed format, current;  Sense key: Illegal Request
 Additional sense: Invalid field in cdb
Write buffer failed: Illegal request sense key, apart from Invalid opcode

Any suggestions for the correct sg_write_buffer statement for writing to an HP SAS expander card?

You need to specify a download microcode mode.  Using deferred activation, the most modern approach:

1. sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=xxx.bin /dev/sgXX
2. sg_write_buffer --mode=activate_mc /dev/sgXX

Older devices might only support mode 0x7.

Confirmation of successful firmware download/update/flash to the HP SAS expander card using the commands:
# sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=PUF21000.bin /dev/sg1
# sg_write_buffer --mode=activate_mc /dev/sg1
# reboot

# lsscsi -g | grep HP
[6:0:0:0]    enclosu HP       HP SAS EXP Card  2.10  -          /dev/sg1

This is expected behavior and not a bug because the HP P410 device is a SAS RAID controller and not a serial attached scsi host bus adapter (SAS HBA).

A SAS RAID controller 
quote: "does not expose anything about the SAS fabric (expanders or physical drives) to the OS"