Hello, with Truenas you have locked kernel and linux system, there is very few room to try to test/tweak things when you have a problem.
Thats’s why i try run truenas on top of proxmox (proxmox is basic debian, with full shell and linux commands allowed).
Proxmox also has more recent kernel with more recent drivers.
With proxmox you can complile / install / run a driver, with Truenas you can’t.
With Proxmox 9 install when installed it it did not permit me to fromat my USB M.2 drive with default EXT4, so i user ZFS, the only that worked.
Unfortunatly after last kernel/proxmox update, when i boot it can’t auto select/mount zfs, so i need to type “zfs import rpool” every time i restart the system.
Proxmox 9 is too recent and has many problems. Never had update that broke the boot with Proxmox 8.
On the EMMC i have my Truenas direct install (1st test install), on the USB M.2 i have Proxmox (2nd test install) : i can boot on both.
In the Beelink Me mini i have the 10gbe M2 ethernet card + 5 Crucial T500 4tb drives.
In BIOS i desactivated power management, all my settings :
https://drive.google.com/drive/folders/1qKy8X5LHsY8KTqGCWHt1RucNrNHCf933?usp=drive_link
Unfortunatly now i am having problems with my 5 Crucial T500 M.2 4tb drives witch are going to D3cold state and break my RaidZ1 in tuenas.
2 of 5 M.2 drop from the system (they go to D3cold state and never come back to D0 state) : the only way to have them come back is power off completely the Beelink Me Mini, then very quickly they drop again.
JayTwoCents has the exact same problem on windows :
He says all M.2 drives witch have Physon E25 controller (not only the T500) have the same problem (in his tests with windows).
He says this is related to the last Windows update.
But i have the same problem (M.2 power off to D3cold and can not go back to D0 until full system power off) in Linux (debian proxmox 9).
This leads me to think that Crucial T500 (CT4000T500SSD3) has bad firmware (version : P8CR013) that does not support all fonctions needed to bring back it from D3cold to D0.
root@pxm:~# bash /var/lib/vz/snippets/nvme-firmware.sh
[] Scanning PCI NVMe controllers…
[*] Device 0000:03:00.0 driver=none desc=“03:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology T500 NVMe PCIe SSD [c0a9:5415] (rev 01)” vendor=c0a9 device=5415
[] Device 0000:03:00.0 appears to be in D3cold.
[*] Attempting to wake 0000:03:00.0 from D3cold…
[!] 0000:03:00.0 remains in D3cold — skipping
[] Device 0000:04:00.0 driver=none desc=“04:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology T500 NVMe PCIe SSD [c0a9:5415] (rev 01)” vendor=c0a9 device=5415
[*] Device 0000:04:00.0 appears to be in D3cold.
[] Attempting to wake 0000:04:00.0 from D3cold…
[!] 0000:04:00.0 remains in D3cold — skipping
[*] Device 0000:05:00.0 driver=none desc=“05:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology T500 NVMe PCIe SSD [c0a9:5415] (rev 01)” vendor=c0a9 device=5415
[] Adding PCI ID c0a9:5415 to nvme driver new_id
[*] Binding 0000:05:00.0 to nvme
==> 0000:05:00.0 (nvme0) MODEL “CT4000T500SSD3” FW “P8CR013”
[] Device 0000:07:00.0 driver=none desc=“07:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology T500 NVMe PCIe SSD [c0a9:5415] (rev 01)” vendor=c0a9 device=5415
[*] Adding PCI ID c0a9:5415 to nvme driver new_id
[] Binding 0000:07:00.0 to nvme
==> 0000:07:00.0 (nvme1) MODEL “CT4000T500SSD3” FW “P8CR013”
[*] Device 0000:08:00.0 driver=none desc=“08:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology T500 NVMe PCIe SSD [c0a9:5415] (rev 01)” vendor=c0a9 device=5415
[] Adding PCI ID c0a9:5415 to nvme driver new_id
[*] Binding 0000:08:00.0 to nvme
==> 0000:08:00.0 (nvme2) MODEL “CT4000T500SSD3” FW “P8CR013”
[] Restoring 0000:03:00.0 back to none
[*] Restoring 0000:04:00.0 back to none
[] Restoring 0000:05:00.0 back to none
[*] Restoring 0000:07:00.0 back to none
[] Restoring 0000:08:00.0 back to none
[*] Done.
nvme-firmware.sh : script generated by Google Gemini to get firmware version for my drives
#!/usr/bin/env bash
set -euo pipefail
# nvme-firmware.sh
# Finds Crucial T500 NVMe devices, binds to nvme, reads firmware, restores original driver.
# Handles D3cold and gracefully skips ghost/unpowered devices.
log() { echo "[*] $*"; }
err() { echo "[!] $*" >&2; }
require_root() { [[ $EUID -eq 0 ]] || { err "Run as root"; exit 1; }; }
require_tools() {
for b in nvme lspci; do
command -v "$b" >/dev/null 2>&1 || { err "Missing required tool: $b"; exit 1; }
done
}
load_modules() { modprobe nvme >/dev/null 2>&1 || true; modprobe vfio-pci >/dev/null 2>&1 || true; }
list_nvme_bdfs() {
for d in /sys/bus/pci/devices/*; do
[[ -f "$d/class" ]] || continue
if grep -q '^0x0108' "$d/class"; then
basename "$d"
fi
done
}
current_driver() {
local bdf="$1"
[[ -L "/sys/bus/pci/devices/$bdf/driver" ]] && basename "$(readlink -f "/sys/bus/pci/devices/$bdf/driver")" || echo "none"
}
unbind_from_driver() {
local bdf="$1" drv="$2"
local unbind="/sys/bus/pci/drivers/$drv/unbind"
[[ -w "$unbind" ]] && echo "$bdf" > "$unbind" 2>/dev/null || true
}
bind_to_driver() {
local bdf="$1" drv="$2"
local bind="/sys/bus/pci/drivers/$drv/bind"
if [[ -w "$bind" ]]; then
if ! echo "$bdf" > "$bind" 2>/dev/null; then
return 1
fi
return 0
else
return 1
fi
}
ensure_nvme_knows_id() {
local vendor="$1" device="$2"
if [[ -w /sys/bus/pci/drivers/nvme/new_id ]]; then
log "Adding PCI ID $vendor:$device to nvme driver new_id"
echo "$vendor $device" > /sys/bus/pci/drivers/nvme/new_id 2>/dev/null || true
fi
}
bdf_to_nvme_ctrl() {
local bdf="$1"
for ctrl in /sys/class/nvme/nvme*; do
[[ -e "$ctrl" ]] || continue
if [[ "$(readlink -f "$ctrl/device")" == *"/$bdf" ]]; then
basename "$ctrl"
return 0
fi
done
return 1
}
wait_for_ctrl() {
local bdf="$1"; local timeout="${2:-20}"
local i=0
while (( i < timeout )); do
if bdf_to_nvme_ctrl "$bdf" >/dev/null; then return 0; fi
sleep 0.2
(( i++ ))
done
return 1
}
read_model_fw() {
local ctrl="$1"
local mn fr
mn="$(nvme id-ctrl "/dev/$ctrl" 2>/dev/null | awk -F: '/^[[:space:]]*mn[[:space:]]*:/ {print $2}' | xargs || true)"
fr="$(nvme id-ctrl "/dev/$ctrl" 2>/dev/null | awk -F: '/^[[:space:]]*fr[[:space:]]*:/ {print $2}' | xargs || true)"
[[ -n "$mn" || -n "$fr" ]] || return 1
echo "${mn}|${fr}"
}
is_d3cold() {
local bdf="$1"
if [[ -f "/sys/bus/pci/devices/$bdf/power_state" ]] && grep -q 'D3cold' "/sys/bus/pci/devices/$bdf/power_state"; then
return 0
fi
if [[ -f "/sys/bus/pci/devices/$bdf/uevent" ]] && grep -q 'D3cold' "/sys/bus/pci/devices/$bdf/uevent"; then
return 0
fi
return 1
}
try_wake_from_d3cold() {
local bdf="$1"
log "Attempting to wake $bdf from D3cold..."
[[ -w "/sys/bus/pci/devices/$bdf/power/control" ]] && echo on > "/sys/bus/pci/devices/$bdf/power/control" 2>/dev/null || true
[[ -w "/sys/bus/pci/devices/$bdf/power/wakeup" ]] && echo 1 > "/sys/bus/pci/devices/$bdf/power/wakeup" 2>/dev/null || true
sleep 1
! is_d3cold "$bdf"
}
main() {
require_root
require_tools
load_modules
declare -A ORIGINAL_DRIVER=()
log "Scanning PCI NVMe controllers…"
mapfile -t BDFS < <(list_nvme_bdfs)
[[ ${#BDFS[@]} -gt 0 ]] || { err "No NVMe-class PCI devices found."; exit 1; }
for bdf in "${BDFS[@]}"; do
drv="$(current_driver "$bdf")"
ORIGINAL_DRIVER["$bdf"]="$drv"
vendor="$(printf "%04x" $(( $(cat /sys/bus/pci/devices/$bdf/vendor) )))"
device="$(printf "%04x" $(( $(cat /sys/bus/pci/devices/$bdf/device) )))"
desc="$(lspci -nn -s "$bdf" 2>/dev/null || true)"
log "Device $bdf driver=$drv desc=\"$desc\" vendor=$vendor device=$device"
if ! echo "$desc" | grep -qi 'T500'; then
log "Skipping non-T500 $bdf"
continue
fi
if is_d3cold "$bdf"; then
log "Device $bdf appears to be in D3cold."
if try_wake_from_d3cold "$bdf"; then
log "Woke $bdf from D3cold."
else
err "$bdf remains in D3cold — skipping"
continue
fi
fi
ensure_nvme_knows_id "$vendor" "$device"
if [[ "$drv" != "nvme" ]]; then
[[ "$drv" != "none" ]] && unbind_from_driver "$bdf" "$drv"
log "Binding $bdf to nvme"
if ! bind_to_driver "$bdf" nvme; then
err "$bdf did not respond to bind (likely absent/unpowered) — skipping"
continue
fi
if ! wait_for_ctrl "$bdf" 20; then
err "No nvme controller appeared for $bdf — skipping"
continue
fi
fi
ctrl="$(bdf_to_nvme_ctrl "$bdf" || true)"
if [[ -z "$ctrl" ]]; then
err "No /dev/nvme* controller found for $bdf — skipping"
continue
fi
info="$(read_model_fw "$ctrl" || true)"
if [[ -n "$info" ]]; then
echo "==> $bdf ($ctrl) MODEL \"${info%%|*}\" FW \"${info##*|}\""
else
err "Failed to read model/fw for $bdf ($ctrl)"
fi
done
for bdf in "${BDFS[@]}"; do
orig="${ORIGINAL_DRIVER[$bdf]}"
[[ "$orig" == "nvme" ]] && continue
log "Restoring $bdf back to $orig"
unbind_from_driver "$bdf" nvme || true
[[ "$orig" != "none" ]] && bind_to_driver "$bdf" "$orig" || true
done
log "Done."
}
main "$@"
