
ijustpostwhenimhi
1108
50
4

Still have to write up the full procedure, but I finally got a gen3 PCIe switch to function properly!
The short is that I was having usb issues with a NIC I was testing, and as a shot in the dark, I ran rpi-update to switch my firmware and kernel to the rpi dev branch. (I tried other kenerls before, just not the exact one rpi-update gives you) Well, my usb issues went away, and just out of curiosity I tried some of the dead-end switch cards that I had lying around.
The Asmedia ASM2812 based 2-port m.2 AIC works now!
It's the best I could have asked for because it's only 2 downstream ports, which is all I need, and reduces complexity & power

pci-stub grabs the 2nd port on the NIC to free up MSI-X interrupts for the SAS controller

For the 2 downstream ports, I'm using m.2 to u.2 adapters with built-in clock buffers from "Micro Sata Cables".
So far they make the best quality stuff for these types of boards.
For the RPi root port, I'm using an m.2 to u.2 adapter with built-in redriver circuit to extend the weak RPi PCIe signals. Thankfully the switch's downstream ports don't require a redriver. (The redriver boards cost much more)

Back to that sweet sweet PCIe NIC goodness
@StackMySwitchUp
llebkcir
great work.
PubMed12927120suckssobad
Wut?
zertmaster
usernametakenisthestoryofmylife
Dude, it's sweet.
Regeny
Jesus these grep and sed lines... that's why I love and hate shellscript.
StackMySwitchUp
Congrats! Nice numbers
UnitConversionBot
2 m ≈ 6 feet, 7 inches or 1.3 small boulders
QuakThat
how much storage you planning to put behind this?
ijustpostwhenimhi
To start, 80TB In HDDs, and some SSDs for cache. It'll be multi-use, primarily for media library streaming and archiving
LeftRightThere
You might want to look into grep -Po “\K{regex stuff}” instead of using sed. grep -A 5 would also return the first 5 lines per match.
ijustpostwhenimhi
Arcygenical
What sort of throughput do you get with these?
ijustpostwhenimhi
Last image shows iperf3 tests. For SMB transfer to/from a SAS ZFS pool (raid-z) I get 600MB/s
Arcygenical
That's actually really respectable for a RPI with so many bottlenecks. I opted for a small 4x Sata SSD array. I get about the same transfer speeds and my 2.5gb network is limiting transfers anyways.