My PiNAS project, part 38, it's a xmas miracle

Dec 26, 2024 11:31 AM

Still have to write up the full procedure, but I finally got a gen3 PCIe switch to function properly!
The short is that I was having usb issues with a NIC I was testing, and as a shot in the dark, I ran rpi-update to switch my firmware and kernel to the rpi dev branch. (I tried other kenerls before, just not the exact one rpi-update gives you) Well, my usb issues went away, and just out of curiosity I tried some of the dead-end switch cards that I had lying around.
The Asmedia ASM2812 based 2-port m.2 AIC works now!
It's the best I could have asked for because it's only 2 downstream ports, which is all I need, and reduces complexity & power

pci-stub grabs the 2nd port on the NIC to free up MSI-X interrupts for the SAS controller

For the 2 downstream ports, I'm using m.2 to u.2 adapters with built-in clock buffers from "Micro Sata Cables".
So far they make the best quality stuff for these types of boards.
For the RPi root port, I'm using an m.2 to u.2 adapter with built-in redriver circuit to extend the weak RPi PCIe signals. Thankfully the switch's downstream ports don't require a redriver. (The redriver boards cost much more)

Back to that sweet sweet PCIe NIC goodness
@StackMySwitchUp

raspberry_pi

engineering

technology

pinas

great work.

7 months ago | Likes 2 Dislikes 0

Wut?

6 months ago | Likes 1 Dislikes 0

7 months ago | Likes 4 Dislikes 0

Dude, it's sweet.

7 months ago | Likes 2 Dislikes 0

Jesus these grep and sed lines... that's why I love and hate shellscript.

7 months ago | Likes 6 Dislikes 0

Congrats! Nice numbers

7 months ago | Likes 2 Dislikes 0

2 m ≈ 6 feet, 7 inches or 1.3 small boulders

7 months ago | Likes 1 Dislikes 1

how much storage you planning to put behind this?

7 months ago | Likes 2 Dislikes 0

To start, 80TB In HDDs, and some SSDs for cache. It'll be multi-use, primarily for media library streaming and archiving

7 months ago | Likes 2 Dislikes 0

You might want to look into grep -Po “\K{regex stuff}” instead of using sed. grep -A 5 would also return the first 5 lines per match.

7 months ago | Likes 1 Dislikes 0

I was mostly using sed to pretty-fy the output for sharing, removing some words from the device names, and highlighting the important information. The head command is just trimming the last 5 lines which belong to the on-board pcie devices. This is the grep output before the sed stuff:

7 months ago | Likes 1 Dislikes 0

What sort of throughput do you get with these?

7 months ago | Likes 2 Dislikes 0

Last image shows iperf3 tests. For SMB transfer to/from a SAS ZFS pool (raid-z) I get 600MB/s

7 months ago | Likes 4 Dislikes 0

That's actually really respectable for a RPI with so many bottlenecks. I opted for a small 4x Sata SSD array. I get about the same transfer speeds and my 2.5gb network is limiting transfers anyways.

7 months ago | Likes 4 Dislikes 0