Skip to content

Hardware validation checklist

Use this checklist for release-candidate hardware runs. A check is complete only when its log identifies the physical device, firmware artifact, artifact hash, command result, and cold/warm boot outcome. Do not infer success from a tool's exit code when the tool has a documented false-success mode.

Current marathon ledger (2026-08-30)

Hardware Stable identity Current state Next blocking check
Seeed XIAO nRF52840 B35E71C1C3726CE7 A stalled Companion transport survived the stop token and exact USB reset, then recovered after a 36-second exact application DFU. Current Full Companion USB, bonded BLE, clock sync, terminal/Binary switching, serial-mOTA folder attach, and reversible dual-CDC logging all pass Prove current-build cold persistence; separately reproduce the stall before using a power cut to test whether power alone clears it; then transfer an actual image through BLE-controlled LoRa OTA
Seeed Tracker T1000-E 34A9141999729D5D A temporary Full Companion and one of two same-visible-version OTAFIX candidates are running; the exact candidate bytes are not yet proven. The bonded warm baseline passed at 3.34 kB/s. Two lab-only pre-START 15 ms/latency-0 DFUs with 0/0 no-preference event-length hints passed at 6.23 and 6.52 kB/s; a 10 ms event-length control timed out before START Prove installed bootloader bytes by readback, exercise a physical cold 20-to-244-byte readiness transition, restore the protected Repeater identity, cold boot, then force LR1110 reset
RAK3401 0B81C9C68D8D01B4; FICR 8D8D01B4 0B81C9C6 OTAFIX test version 0x02040403 passes bidirectional application UF2, exact SWD application/bootloader readback, unchanged UICR, and a 74-block direct V4 LoRa apply from exact 3caf9dcf to current HEAD 9fd580c8. Post-apply USB and LoRa identity/hash checks pass and the temporary 0 dBm bench setting is restored to 22 dBm Add exact RF packet counters, then repeat the supported-bandwidth and controlled/passive/mixed routed-hop matrix
Heltec MeshTower V2 with SD 9352162A72082314 Exact SD LoRa-OTA Repeater e26d48e4 is running after identity-gated BLE recovery. Card format/cooldown/forced-format/raw-erase/remount pass; live bootloader contract reports ABI 3, FULL+INPLACE, and SD apply Signed interrupted-download resume, corruption/signature rejection, then full/delta apply
Heltec V4 USB MAC 44:1B:F6:6A:E8:44; BLE 44:1B:F6:6A:E8:45 USB/BLE/Wi-Fi and all three TCP services pass on hardware. The final fresh-NTP-gated Full Companion image builds with 4.31 MB app space free Flash the exact final NTP build, prove NTP-before-TLS success/failure ordering, then OTA seeding
SenseCAP Indicator LoRa CH340 plus USB MAC D8:3B:DA:75:23:AC; BLE D8:3B:DA:75:23:AD The current dark-layout application is SHA-256 81bebdc07b6a8349c1c975cb5a0e30c5f6019d35bcac641e5f3d4df951f7406b; identity-gated flash, USB ASCII/Binary switching, runtime logging, and separate configured-Wi-Fi, BLE, and TCP checks pass. Fresh-NTP HTTPS recovery, strict Range resume, STAGEV2 install, and RP USB readback passed on earlier exact artifacts; the final coordinator source still needs its blocked-UDP/123 hardware gate Physical display-wrap and four-mode render/heap checks, exact-final blocked-NTP recovery gate, LoRa/TempRadio, and non-empty OTA seeding

The current source checkpoint passes 1,132/1,132 native cases, 68/68 host broker/endpoint cases, all 318 LoRa-OTA chain/route/bundle cases, and 41/41 MOTA format/signature/delta cases under the documented pipx detools interpreter. Indicator geometry, dark-theme, and font-recovery contracts pass 4/4, 6/6, and 25/25. A serialized real Indicator ESP32-S3 Wi-Fi/USB profile also compiles at 95,944 bytes RAM and 1,589,433 bytes flash. The direct system-Python MOTA invocation predictably lacks detools; that environment boundary is not a firmware failure and must not be worked around by installing the package globally.

The RAK failure log is /home/mikec/hwtest/runs/rak3401-syncidle-clean-tSp7PLgA/result.txt on the Mercerwood Pi. The tested UF2 SHA-256 was 650c9fda456685cdf99c1f2996c60f1f551cc9216d2b746a22c8cb78464b0ff7. cp returned success in 95 ms, but sync -f failed with EIO after 157,188 ms. Post-failure SWD readback proved that all seven ordinary sectors whose bytes differed between the old and intended builds still contained the old values, and the settings page was still erased. The only available pre-copy SWD dump was taken several hours earlier, however, so it cannot prove that zero unchanged blocks were written during this attempt or establish when the stale S140-v7 tail in the first application page appeared. The next run must capture an immediate pre-copy dump as well as the post-copy dump. Serial DFU recovery then completed in 35,698 ms with the exact Device programmed. marker. All 1,830 UF2 payload blocks matched SWD readback after recovery, and the application passed a subsequent real power-removal boot.

The corrected 2.4.4-preview.2 candidate was installed through identity-gated SWD from combined-image SHA-256 9b1654baccd2a920f68651893e31d79170400d1a7982a3f854ae640f1d3513a8. An immediate pre-copy dump closed the earlier evidence gap. A deliberately old UF2 then copied and flushed successfully in 16,568 ms, and the exact formerly failing UF2 above copied and flushed successfully in 16,601 ms. After each direction, all 1,830 payload blocks matched a full SWD readback with zero header or payload mismatches. The return update changed 123 bytes across five expected application pages, UICR remained byte-identical, and a subsequent downstream USB power removal plus Pi reboot returned the RAK as the same stable identity. It reported the intended 3caf9dcf application, board, node name, USA radio tuple, and 22 dBm setting after that cold boot. The new bootloader's CLI source string is truncated before the injected preview number, so the independently hashed SWD artifact is the authoritative candidate identity for this run.

The later TinyUSB lifecycle/generation candidate identifies itself as test version 0x02040403. Its exact 40,960-byte bootloader readback SHA-256 is f609eb18754070c54f127325b1cb48547baf5c6b1c695d36fd0c8e0874713c13. With that candidate installed, a full CURRENT.UF2 read completed before each write. The 936,960-byte bf092b24 application UF2 (SHA-256 12477a9734cf12eb920361bd21024e39bf303953fcf79d0bf720e00847e2ae92) copied and flushed in 28,203 ms and returned to application mode four seconds later. The exact 3caf9dcf application UF2 (SHA-256 650c9fda456685cdf99c1f2996c60f1f551cc9216d2b746a22c8cb78464b0ff7) then copied and flushed in 26,479 ms and also returned in four seconds. Both boots retained RAK3401-OTA-BENCH and 910.525/62.5/SF7/CR5.

Identity-gated SWD then read the entire 468,480-byte padded UF2 application range. Its SHA-256 exactly matched the converted source on both sides at b2e37a7681027ca91d1b2bd8e39b55395d9e91d4863aed17131e50c1ba1d2227. The bootloader still matched the installed 0x02040403 image byte-for-byte, and UICR still matched the pre-test image at d029ea630c2f632a1b690cb52b2a96a28b6f870c656c3e1bd74ba3a571c53b9c. The authoritative logs are under /home/mikec/hwtest/runs/rak3401-uf2-02040403-20260830-170829 and /home/mikec/hwtest/runs/rak3401-uf2-return-02040403-sfHiGEqA on Mercerwood.

The subsequent direct LoRa qualification used the V4 as controller/seeder and the RAK's exact bf092b24 body 91CF4EC2882F23D6 as the base. Its mandatory fixed three-minute rehearsal passed on 909.95/250/SF5/CR5, including natural expiry and exact normal-radio recovery. The 2,132-byte, two-block in-place delta reached 2/2 and ready in six seconds; install was accepted, and the new body appeared at the 60-second reboot probe. The automation took 514 seconds including rehearsal, clock/radio safety checks, and post-install verification. The final body was 27A223DDFC3A8F47, version 1.17.1.5 at 3caf9dcf, and the bootloader reported blrc:B8, which is OTAFIX's successful application apply result rather than an error. Exact target TX power was wrapped from 22 dBm to 0 dBm for the co-located transfer and restored to 22 dBm afterward; the source RXPS configuration and both normal radio tuples were also restored.

Identity-gated SWD then read the live application at 0x26000. All 468,384 bytes matched the intended release image exactly (SHA-256 b496e3f4419c9f771564ed3c74b3407822f45acdf60cbd6c4625a54b181ca084), and the independently recomputed EndF hash was 27A223DDFC3A8F47. The target ID was 2FA509C1, hardware ID was RAK_3401, and version word decoded to 1.17.1.5. The 40,024-byte bootloader readback also matched the exact installed ffb1580/test-version-0x02040402 artifact byte-for-byte; UICR was unchanged. After SWD reset-halt/resume, USB again reported the expected application, identity, name, radio, 22 dBm, no TempRadio lease, and blrc:00 (the retained success diagnostic is intentionally cleared by the next normal boot).

The current 0x02040403 bootloader candidate then passed a second direct V4 LoRa qualification using a materially larger application change. The unsigned in-place package was 75,614 bytes with 75,108 payload bytes and 74 transfer blocks. Its mandatory fixed three-minute rehearsal passed, including natural TempRadio expiry and exact normal-channel recovery; one lost read-only ota status reply was reconciled by its bounded exact retry. The transfer reached 74/74 and READY in 122 seconds, the explicit unsigned install was accepted, and reboot produced exact HEAD 9fd580c8 with body 744BC4443891ED10. The complete safety-wrapped run took 589.487 seconds, including rehearsal, recovery, and final verification. The first postboot ver reply was lost and its exact read-only retry passed. Independent USB checks then confirmed target 2FA509C1, hardware RAK_3401, unchanged identity/name/radio/RXPS/watchdog state, bootloader CRC 6683C066, ABI 3, apply result blrc:B8, and no active TempRadio/download/seeder/key state. After the separate host-recovery exercise, the near-field TX setting was restored from 0 dBm to its exact original 22 dBm and read back over an authenticated LoRa session. The sanitized authoritative log is /home/mikec/hwtest/runs/rak3401-02040403-head-vzWOXVxT/qualification-redacted.log with SHA-256 3e53dcaa5922c7c18a7890963b8fc092311d2db01caa5603640f43f1482802f6.

A serialized VM build of the six connected-board qualification profiles then completed 6/6 in 136.122 seconds: XIAO nRF52 BLE Companion, T1000-E BLE Companion, RAK3401 LoRa-OTA Repeater, SenseCAP Indicator LoRa USB/Wi-Fi Companion, MeshTower V2 SD LoRa-OTA Repeater, and the consolidated Heltec V4 V4.2/V4.3 Full Companion. The earlier V4 selector typo was rejected before any compile began and is not a firmware build failure.

After the fresh-NTP download gate, shared SNTP coordinator, MQTT-to-MeshCore RTC handoff, and daily refresh policy were added, the complete native matrix passed 1,102/1,102 cases in 93.953 seconds. The scheduler is host-tested at 24-hours-minus-one-millisecond, the exact 24-hour boundary, normal and wrapped millis() values, a short retry that overrides the daily cadence, the zero deadline sentinel, and a Wi-Fi event latch. A reconnect after a successful sample preserves the existing deadline, so successful samples are requested after boot and then only once per day. All current Python and shell contract suites passed, and the expanded LoRa-OTA automation suite passed 317/317 cases.

Serialized real builds also passed after that change. The final event-safe implementation uses an atomic GOT_IP latch so even a short initial connection before the 10-second MQTT status sample cannot skip the required boot NTP request. The final Heltec V4 MQTT Observer application is 1,853,304 bytes with SHA-256 1938ae88fcdbe601cf8698df44b548656079562d899801886b808af913e5f9ac. The consolidated Full V4 image used 2,250,801/6,553,600 flash bytes and left 4,302,799 bytes of app-image space. Its 2,251,272-byte application image has SHA-256 05131eac29164be46d1bcceeb4f03455f5ea8d61c8d4f529b77d8adfe25fdb6a; the 2,316,808-byte merged image has SHA-256 c4c1fc639b6d0087c795a81aca79c07eba56dc4e071eb374dc70da0bec81815e. The focused daily-NTP policy suites passed all 42 cases after the latch was added, and the Full capability manifest verified all ten expected features. The reduced RAK3401 LoRa-OTA Repeater also compiled with the verified WisBlock I2C aliases. At that source state its link used 471,344/815,104 flash bytes; the reduced external-sensor contract retains INA219/INA226/INA260/INA3221 as the voltage/current entries in the optional environmental-telemetry table. These are not the image's only I2C consumers. The image separately retains its SSD1306 display, auto-discovered I2C RTCs, RAK12500 I2C GPS, and RAK12501/L76K UART GPS paths. The configured INA3221 and RAK12500 both default to I2C address 0x42; those two devices cannot coexist at those addresses. The supported combined arrangement leaves RAK12500 at 0x42, straps INA3221 A0 to SCL for 0x43, and uses a build with -DTELEM_INA3221_ADDRESS=0x43. The nRF52840 die-temperature and ADC battery-voltage paths are also retained. The 472,227-byte DFU ZIP SHA-256 is fdd369005a61635756efa9e38b217376a952f334b0864f927a46799b65264fa9, and the matching 943,104-byte UF2 SHA-256 is 2155632fde730e0ae2e4a72ce64f89910861f6c5105a4bd7ac96aaa538a33dca. These VM results do not replace the remaining exact-final hardware test with UDP/123 blocked.

The later serialized RAK3401 contract build, after the non-blocking nRF52 USB logging change, used 471,576 bytes before its 56-byte EndF trailer. Its EndF body hash is 026749684bf6ee8f, target ID is 2FA509C1, hardware ID is RAK_3401, and version word is 0x01110105. The 472,435-byte DFU ZIP has SHA-256 cf0880b26f2d6bd8157f8af76a6718fe1f998de8ea7df8cd952e734448201fb9; the 943,616-byte UF2 has SHA-256 f893a8cebf309af12a4a6f5483acb28f94158e18632f660e011658c846e8ca7f. The linked ELF contains GPS, SSD1306, RTC, and the four INA families, while an exact symbol audit finds none of AHT10/20, BME280, BMP280, SHTC3, SHT4X, LPS22HB, MLX90614, VL53L0X, BME680/BSEC, BMP085, or RAK12035. The complete native matrix passed 1,112/1,112 cases at that intermediate source state. These are VM gates only; flash the exact artifacts and repeat the physical peripheral and blocked-clock checks before marking any later source state hardware-qualified.

Subsequent source hardening changed RAK peripheral discovery: the shared 3V3_S rail remains enabled, UART GPS requires a complete checksum-valid NMEA sentence, an exact INA3221 identity blocks a conflicting u-blox probe, and a successful I2C GPS claim suppresses a sensor only on the same bus and address. The hashes and sizes above describe their recorded intermediate artifacts; they are not current-HEAD evidence and must not be reused for the hardened source.

The immutable font endpoint was re-probed only after the VM reported NTPSynchronized=yes. GitHub returned HTTP 206 for bytes 0-65535, an exact 65,536-byte match to the checked-in font (range SHA-256 8b0dd308eb01469d1f9e732fba260720a80cc86f6cb5385c28e8a16c76dcfbf3), with total length 1,302,608 and strong ETag "39ff0cfbe37e36905507697d042869c05374c3202dd96d5a9f98812153f324c1". This validates the current strict Range assumptions against the live service; the exact-final Indicator still needs the negative hardware run proving that blocked UDP/123 prevents every TLS request and RP2040 staging write.

Mercerwood subsequently recovered without a target write. Both its Tailscale and LAN addresses again accepted SSH, Bluetooth was idle, and the exact T1000-E stable serial 34A9141999729D5D was present as application USB 239a:8029. The RAK3401 simultaneously uses the same VID/PID, which confirms that every continuation must resolve the stable serial/by-id identity rather than VID/PID. The T1000-E remained in the exact bonded Full Companion application; no stale DFU, scanner, meshcli, or HCI-capture process was running.

The post-reboot seven-TTY inventory was also resolved without assuming that one board owned adjacent device names. ttyUSB0 is the SenseCAP Indicator's ESP32-S3 CH340 UART and ttyACM5 is its RP2040; their sibling physical USB paths and the Indicator's live TCP board reply agree. The separately attached ttyACM4 is the MeshTower V2 native HT-n5262 endpoint and returned the qualified MeshTower build over its own console. The remaining ACM endpoints independently identify the RAK3401, XIAO nRF52840, Heltec V4, and T1000-E.

ModemManager had restarted after the Pi reboot and could probe otherwise idle radio consoles. Six serial-bearing boards now use stable-serial-only ID_MM_DEVICE_IGNORE rules. Because the Indicator CH340 has no stable serial, its separate rule requires the exact current physical port chain and exact 1a86:7523 identity; there is no broad CH340, product, or VID/PID-only rule. A controlled ModemManager restart reported zero modems, displaced no TTY owner, caused no USB disconnect/re-enumeration, and left all seven endpoints ignored. The same mctomqtt process retained sole ownership of the RAK, while a subsequent Indicator TCP query still returned the expected board and build. Reversible root-only backups were retained on the Pi before each rule update.

Run-wide gates

  • [ ] Record commit, version string, role/profile, artifact filename, size, and SHA-256 before flashing.
  • [ ] Archive the exact ELF and symbol/objdump map beside each qualification firmware. A rebuilt ELF is acceptable for diagnosis only after its firmware binary matches byte-for-byte; compiler __DATE__/__TIME__ strings can otherwise change both bytes and the EndF body hash.
  • [ ] Bind every operation to a stable USB serial number or physical USB path; never rely on a changing ttyACM/ttyUSB number.
  • [ ] After erasing/reflashing a BLE board that reuses its Bluetooth address, remove any stale host bond and pair again with the board's current PIN before diagnosing service-discovery disconnects. On Linux, verify the BlueZ agent actually accepted the passkey; a scanner seeing the device is not proof that Companion GATT is usable.
  • [ ] Do not use BlueZ's advertisement service_uuids as a same-address warm handoff identity gate. After application/bootloader transitions BlueZ can merge the bootloader's cached GATT UUIDs into the live application's advertisement. Gate the application advertisement by exact address and name, then require the live connected BLEDfu service, control/revision characteristics, retained bond/encryption, notification subscription, and observed target disconnect before accepting the handoff.
  • [ ] Do not treat meshcli -P as an infallible pairing probe. Immediately after one successful post-DFU session, a second invocation with the exact paired XIAO address hit the meshcore-cli NoneType ... pair/disconnect path before it created a Bleak client. BlueZ still reported the exact bond and the same command without -P connected and synchronized the clock successfully. Capture this as a host-tool failure, then verify the device independently before blaming firmware.
  • [ ] After a successful BleakClient.connect(), start a fresh disconnect generation before beginning DFU. BlueZ can fail one internal LE attempt, retry successfully, and still deliver the failed attempt's callback. A stale event caused the host to stop notifications and issue a local HCI disconnect after a valid image CRC but before ACTIVATE; the target and bootloader had not dropped the link. Preserve an HCI trace through activation so local-host and remote-target disconnects remain distinguishable.
  • [ ] Before a BLE pairing/throughput test on a Wi-Fi-managed Pi, record both management addresses and a same-LAN recovery vantage point. The XIAO pairing run completed, but Mercerwood then stopped answering on both its Wi-Fi and Tailscale addresses; BellevueBBS confirmed the old LAN address was absent at ARP and no replacement LAN host exposed SSH. This is an infrastructure outage until Pi logs prove a cause, not a firmware or BLE failure. The T1000-E pairing run reproduced the management outage: its exact application paired successfully and exposed the complete DFU service, then Mercerwood stopped accepting SSH and ultimately stopped answering Tailscale pings. No buttonless DFU write had been sent, and the T1000 remained in its valid application. Capture Pi Wi-Fi/firmware logs after recovery and avoid leaving discovery enabled between operations.
  • [ ] Confirm the selected board and role before every erase or flash.
  • [ ] Before every SWD write, read the target FICR device ID and match it to the intended board's stable USB serial (and physical wiring record). A USB product name or remembered cable position is not sufficient.
  • [ ] Confirm no other PlatformIO process is active; run only one PlatformIO build, test, clean, or upload at a time.
  • [ ] Record the installed bootloader, SoftDevice, partition table, or flash layout independently of the application banner where applicable.
  • [ ] Test both a software reboot and a real power removal/cold boot.
  • [ ] After an update, read back identity, version, radio tuple, node name, and persistent settings instead of assuming they survived.
  • [ ] Save command output and elapsed milliseconds. Mark a test FAIL, not PASS, if the transport reports an I/O error even when firmware later boots.
  • [ ] Before calling a node hung, validate that the probe uses that profile's command terminator, check a second transport, and capture PC/LR/watchdog state over identity-gated SWD before reset. Resume once and retest before using reset as recovery so the diagnostic state is not destroyed first.
  • [ ] Treat a remote login acknowledgement and the following command reply as separate LoRa packets. A lost login acknowledgement is not proof of failed authentication when a command-matched private reply from the selected contact key is present; an explicit negative login result still fails closed.
  • [ ] Restore the intended USA radio preset and normal-radio mode after tests.
  • [ ] Compare both semantic version and exact EndF/body hash. A transition between two commits carrying the same version string must stop at the normal reinstall guard unless the run plan explicitly records and passes --allow-non-upgrade; a different commit name alone is not authorization.
  • [ ] Do not allow a non-interactive transmission-failure prompt to continue retry cycles forever. Optional capability probes must fail/fall back after a bounded interval, while mutations and cleanup retain their stronger bounded-lease recovery rules.
  • [ ] Before the first remote command after any controller/source reboot, read its RTC and compare it with a trusted host clock. Advance a clock which is behind and prove readback; never move a radio clock backward. Fail closed when it is more than 10 minutes ahead. During the RAK rehearsal the V4 rebooted to 2026-03-01 while the RAK remained at 2026-08-30; the RAK received the direct packets but correctly rejected their old timestamps as replays. Setting the V4 to host epoch immediately restored login and the exact public-key reply at SNR -1.5 dB.
  • [ ] Before every external HTTPS download, require a fresh SNTP response for that bounded operation, a plausible signed wall clock, connected Wi-Fi, and a non-expired proof immediately before opening TLS. Re-check the proof before every resumed/Range TLS connection. A retained plausible RTC or mesh time is not a substitute for observing NTP, and failure must occur before any HTTP request body or flash writer begins.
  • [ ] Treat an unacknowledged immediate tempradio mutation as potentially active, not merely lost. On the tested repeater, scheduled-radio expiry deliberately waits for hasOutbound() to clear; a CLI reply stranded on the temporary tuple can therefore hold the node there beyond its nominal lease. Prefer one fixed absolute tempradioat window per node, so duplicate delivery cannot extend it, and prove the return on both tuples before a transfer. If recovering an older immediate handoff, join the exact temporary tuple to drain the reply before issuing normalradio.
  • [ ] Do not diagnose a co-located high-power LoRa bench failure as weak RF without an overdrive check. On the RAK3401/V4 bench, the RAK at 22 dBm produced only 3/6 status replies around -1.5 dB SNR, while a reversible 0 dBm setting produced 6/6 replies in 5--8 seconds at +12 to +12.5 dB SNR. Read the original power, lower it only for the near-field test, and require exact restoration afterward.

Compute-placement gate

  • [ ] Run firmware builds, PlatformIO, mOTA/delta generation, compression, full-image hashing sweeps, symbol generation, and other sustained CPU or memory work on the VM. Do not offload these jobs to a Pi merely because the hardware and source artifacts are attached there.
  • [ ] Use the Mercerwood Pi as the identity-gated hardware gateway, SWD/USB/BLE test host, and LoRa/Wi-Fi seeder. Copy only the completed artifact plus its manifest/hash to the Pi, then verify the copied SHA-256 before use.
  • [ ] Run CPU work on a Pi only when Pi performance or deployment behavior is itself the test subject, and label that exception in the run log.
  • [ ] Do not assume a non-interactive SSH session has the same PATH as an interactive shell. Resolve pipx-installed tools such as esptool and detools to their explicit venv path, record the version, and fail before erase if the pinned executable is unavailable.
  • [ ] Treat short serial/BLE/SWD control scripts on the Pi as hardware I/O, not as permission to generate artifacts there. Use the explicit Python from the relevant pipx environment when a probe needs one of its dependencies; the Pi system Python did not contain pyserial during the RAK run.

LoRa-controlled host recovery gate

  • [x] The privileged recovery boundary is a root-owned, group-restricted AF_UNIX broker with a fixed command grammar and fixed systemctl argv. The unprivileged MQTT endpoint cannot gain privileges under NoNewPrivileges, cannot select an executable or unit, and commits an action only after its signed challenge/reply has been published and confirmed. The installed broker passed its cold activation, peer credential, socket owner/mode, injection, duplicate, timeout, durable reservation, and ambiguous-outcome checks.
  • [x] A signed host network restart was accepted exactly once in 4.088 seconds. The broker retained one network-restart record at sequence 1, and the fixed unit journal had exactly one start/finish pair. Independent management probing measured 7.184 seconds from last successful probe to first recovered success and 6.314 seconds from first failure to recovery. Pi-local loopback MQTT delivered 107/107 samples; mosquitto, mctomqtt, and the endpoint retained their exact PIDs, and mctomqtt remained the sole owner of the RAK USB port. One read-only LoRa status reply was lost at the temporary 0 dBm bench setting, then authenticated status returned scheduled; no mutation was replayed. One monitor parsed two systemctl --value columns in reverse order, so its boolean labels are discarded; its PID/MQTT trace and the independent postchecks remain valid.
  • [x] A signed host reboot was accepted exactly once in 3.312 seconds and armed the fixed 10-second timer. A changed boot ID and one observed outage prove one reboot; management recovery took 35.763 seconds from the last pre-outage success and 31.276 seconds from the first failed probe. Postboot signed LoRa uptime replied in 3.651 seconds. All seven exact USB identities returned, only the RAK port was owned, the deployed configuration hashes were unchanged, and NetworkManager, Tailscale, mosquitto, mctomqtt, the endpoint, and both privileged sockets were active. Broker action state is intentionally under /run and persistent journaling was unavailable, so exactly-once evidence is the accepted reply, one boot-ID transition, one outage, an inactive timer, and no second outage; do not claim a durable postboot operation record.
  • [x] Reboot auto-started the still-enabled ModemManager service, but all seven exact devices retained ID_MM_DEVICE_IGNORE=1, zero modems appeared, and no TTY was probed or claimed. The service was returned to its pretest inactive state without disabling or masking it. The RAK then passed exact identity/body/radio gates; an explicit radio-busy rejection left TX at 0 dBm, one bounded idempotent retry after idle returned OK, and authenticated readback proved exact restoration to 22 dBm. V4 identity, power-saving, RXPS, TempRadio, and seeder state also matched their pretest values.

The sanitized host-recovery evidence bundle is under out/host-actions-live-20260831/. Its SHA256SUMS manifest has SHA-256 5acff71ed98f483bb769e8c99c577bf358d0491355ac72855ee7093cd658bf21, and sha256sum -c verifies every deployed-hash, pre/post-state, radio, OTA, network, and reboot record in that manifest.

Mercerwood Pi USB power-cycle gate

  • [ ] Confirm the Pi is reachable through Wi-Fi/Tailscale and that eth0 is not the active management path before cycling its downstream USB tree.
  • [ ] Stop all flashes/transfers, run sync, and release SWD GPIO 8 and GPIO 11 to inputs with no pulls before removing USB power. This prevents an SWD signal from partially back-powering an otherwise unpowered nRF52.
  • [ ] Never issue a standalone uhubctl ... -a off to Pi root hub location 1. Once its only port is off, the hub disappears from uhubctl discovery and a separate -a on command fails with No compatible devices detected at location 1!. This has interrupted two hardware runs.
  • [ ] A downstream hub may be cycled only after resolving the board's stable USB serial to an exact sysfs path and proving that hub advertises per-port power switching. Record the hub location and port, cycle only that port, and gate the re-enumerated VID/PID/serial before continuing. The T1000-E qualification safely used child hub 1-1.2, port 3; this is not permission to cycle the ganged root hub at location 1.
  • [ ] Do not use uhubctl off, on, or atomic cycle on Mercerwood root hub location 1. Physical testing showed that even one atomic cycle powers the ganged root hub down and then cannot rediscover it to restore power. Use a planned Wi-Fi-issued sudo reboot, which is the verified recovery, or arrange a true upstream VBUS removal when that distinction is part of the test.
  • [ ] Treat root hub 1 as a ganged operation: every attached test board and USB Ethernet adapter loses power. After recovery, enumerate every stable /dev/serial/by-id identity again before resuming any board-specific test.
  • [ ] On this heavily populated root hub, inspect kernel logs for real transfer faults before and after a flash, and stop ModemManager while it could open a test TTY. A CH340/ESP32-S3 write which lost the chip at 460800 baud completed and verified at 115200 after a planned Pi reboot; start at the board-qualified conservative rate rather than treating a fast partial transfer as firmware failure.
  • [ ] Under non-interactive SSH, do not let udisksctl fall through to an interactive polkit password prompt. After resolving the exact UF2 block device by board serial, either use an already-mounted volume or an explicit sudo -n mount at a board-specific mount point; re-check INFO_UF2.TXT model/version before copying.

Seeed XIAO nRF52840

  • [ ] Stable USB identity distinguishes application (2886:8044) from UF2 bootloader (2886:0044).
  • [ ] Do not repeat the known-failing full-application UF2 copy under the installed initial 2.4.4 candidate. The identity-gated 897,024-byte copy returned exit 0, then sync -f failed with EIO in 850 ms; the kernel logged an offline device, lost queued writes, and FAT errors before USB disappeared. Recover through a separately hash-gated BLE/serial path and qualify a corrected bootloader candidate before retrying UF2.
  • [ ] SWD readback exactly matches the intended bootloader-region SHA-256.
  • [ ] Actual SoftDevice FWID and runtime application base match the artifact (S140 7.3.0: FWID 0x0123, application base 0x27000).
  • [ ] Exercise and independently verify the exact-board combined bootloader+SoftDevice recovery path before relying on it in the field. A wrong-layout nRF52840 application can overwrite part of S140 while the UF2 bootloader still accepts files and the application still reads flash; internal writes or InternalFS.format() may then hang or reboot. An application-only UF2 and a filesystem erase cannot repair that state. The known XIAO incident in upstream issue #3284 recovered only after the matching Sense/non-Sense OTAFIX bootloader+S140 ZIP was installed. Keep board identity, SoftDevice FWID, application base, and package contents as four separate pre-write gates.
  • [ ] Invalid settings enter persistent UF2 recovery after a cold boot.
  • [ ] UF2 application copy returns success; sync -f returns success; the host has no offline-sector, lost-write, or FAT I/O errors.
  • [ ] Application enumerates after update and reports the expected board, version, role, and radio tuple.
  • [ ] Legacy serial DFU crosses all application pages, emits the exact Device programmed. success marker, and rejects false-success output. Two application serial-DFU attempts on this board timed out waiting for the start ACK while adafruit-nrfutil still returned exit 0, so exit status alone is explicitly insufficient.
  • [ ] Legacy BLE DFU verifies each cumulative PRN receipt exactly and fails on a missing, short, regressive, or overreported offset; it may adapt only among the packet-size-specific safe receipt windows after timely exact receipts and must step back down on slow exact receipts. Final application/SoftDevice/bootloader CRC and a peer disconnect are mandatory before declaring success.
  • [ ] Every upward PRN move is a bounded adjacent-level probe. A failed probe or a later real demotion blocks further increases for that transfer, while additional slow exact receipts may still negotiate down one level at a time. This prevents 4 -> 8 -> 4 -> 8 churn without disabling a safe initial probe or a 32 -> 16 -> 8 fallback.
  • [x] The exact Full Companion ZIP Xiao_nrf52_companion_radio_full-v1.17.1.5-halo-keymind-cascade-3caf9dcf.zip (SHA-256 9a916c1a3ea83d8f6dd704465fe9c044a363478f852a4369cf6b13424a7a9430) installed over BLE in 237,297 ms at about 2,110 B/s using ATT MTU 247, 244-byte writes, PRN 4, and exact cumulative receipts. Validate and the target-initiated disconnect passed. USB then reported the exact expected version/hash, USA radio tuple, key, RXPS policy, and reversible Terminal/Binary mode; the new application advertised NUS as MeshCore-4610BBD2.
  • [x] The post-update application was paired through BlueZ with the expected PIN. Companion GATT then returned the same public key, USA radio tuple, 22 dBm limit, non-repeater role, board, and build over BLE. Its RTC was 18,038 seconds behind the trusted host; clock sync advanced it and the immediate readback was within one second of the host interval. No backward clock write was performed.
  • [x] A second exact same-image update exercised application buttonless DFU and retained-peer bootloader entry. The bootloader deliberately reused the bonded application address E5:C3:A8:B0:60:66 rather than the cold-entry address ending in :67; exact name, Legacy DFU service, and DIS model gates still passed. The first adaptive threshold remained at PRN 4 and completed all 448,416 bytes at about 877 B/s in 527,832 ms; validate, target disconnect, USB identity, application key, role, and USA radio readback all passed. The application clock was then about 10 minutes behind because its RTC did not advance during the bootloader transfer; it was advanced to the host clock and proved by immediate readback.
  • [x] A third same-image A/B run used bounded neutral-window exploration. At 244-byte writes, two exact neutral PRN-4 receipts triggered one adjacent PRN-8 probe. PRN 8 measured about 845 B/s against an approximately 879 B/s PRN-4 baseline, so the helper rolled back to PRN 4 after two receipts and never re-probed. All 448,416 bytes completed at about 870 B/s in 531,804 ms; CRC validation, target disconnect, USB return, exact Companion key/role/radio readback, and post-DFU forward clock sync passed. This proves up/down sender-window adaptation and no churn, while also showing that PRN receipt overhead is not the bonded-path bottleneck on this run.
  • [x] A bootloader candidate built from exact ZIP SHA-256 9c70200fc15a315a50ec68d2a905b5bcf2fc07a796a21fbd439535ec19a608be disabled inherited slave latency locally during DFU DATA. The controller trace still reported the peer-negotiated 30 ms interval and latency 4, with no connection-parameter update after START; the change is therefore a local event-attendance override, not a GAP speed renegotiation. A cold address-generation control completed 448,416 bytes at 2,231 B/s and passed validation/activation in 226,103 ms.
  • [x] Two subsequent bonded application-to-bootloader runs completed the exact same 448,416-byte image at 3,524 and 3,534 B/s. Their complete elapsed times, including exact application advertisement proof, buttonless handoff, bootloader scan, erase, DATA, CRC validation, activation, and reboot, were 169,414 and 169,221 ms. Both targets initiated the final disconnect; USB returned as stable serial B35E71C1C3726CE7; the public key, 22 dBm setting, USA 910.525/62.5/SF7/CR5 tuple, board, and build read back exactly. Compared with the prior roughly 877 B/s bonded run, the DATA rate improved by about 4.0x without increasing the PRN window beyond 8.
  • [x] An exact bonded B-A-B link-profile sequence used the same Full Companion ZIP SHA-256 9a916c1a3ea83d8f6dd704465fe9c044a363478f852a4369cf6b13424a7a9430, lab helper SHA-256 4909c617de37b5666cc622b7190568f002280f55bf26a05843ea283f2be8f566, handoff helper SHA-256 1bad570173079b623bbe2bbb9e5ba7c4f8ac379bf966bfd4382c5266fb9fd214, stable USB serial, bonded address, and 448,416-byte application throughout. First B, xiao-ble-event-length-prestart-20260830-072641-W7COxm, used one exact pre-START 15 ms/latency-0 update with 0/0 no-preference event-length hints and completed DATA in 72.030 seconds at 6,225 B/s; DFU through target disconnect was 102.792 seconds and the full workflow was 117,064 ms. Control A, xiao-ble-control-prestart-20260830-073149-yKf0lO, requested no update, retained the bootloader connection's 30 ms/latency-4 link, and completed DATA in 134.131 seconds at 3,343 B/s; DFU was 165.137 seconds and the full workflow was 182,624 ms. Final B, xiao-ble-exact15-prestart-20260830-073554-FuGPfH, repeated exact 15 ms and completed DATA in 71.970 seconds at 6,231 B/s; DFU was 102.799 seconds and the full workflow was 117,179 ms. Both B rates were about 1.86x control and differed by only 0.096%.
  • [x] The B-A-B protocol and link gates passed independently. Both B runs probed PRN 4 -> 8 after an exact 0.135-second four-packet receipt, rolled back once to PRN 4 after the exact 0.345-second eight-packet receipt, and never re-probed. Control A likewise probed after 0.269 seconds and rolled back after 0.600 seconds. Each B bootloader connection had exactly one matching HCI Update Complete before START; control A had no bootloader-connection update. All three captures contained zero connection updates from START through DATA, CRC validation, activation, and the target-initiated disconnect. Each run ended RESULT=PASS, RC=0, STAGE=complete; final B returned the exact Full Companion application as USB 2886:8044, stable serial B35E71C1C3726CE7, with current application name MeshCore-4610BBD2. A fresh post-B binary Companion query independently re-gated the unchanged public identity, name, Full Companion role, 22 dBm power, and USA 910.525/62.5/SF7/CR5 radio tuple. A separate bonded BLE application query then returned the same board, build, and Full Companion role, so the USB return did not mask a broken Companion BLE path or stale bootloader alias.
  • [x] The current host-controller ceiling is independently characterized. The Pi Zero 2 W Broadcom UART controller reports HCI/LMP 4.2 but supports only LE 1M; LE Read Maximum Data Length and Read Suggested Default Data Length both return Unknown HCI Command. The final B capture contains 1,837 full 244-byte firmware writes, each split into ten 27-byte HCI ACL fragments; all 18,425 transmitted ACL fragments received completed-packet accounting. The median full-write gap was 29.947 ms at 15 ms versus 59.936 ms in control A. A BLE 5 controller with working Data Length Extension is therefore the next host-side speed experiment; this built-in controller cannot exercise DLE or LE 2M.
  • [x] The host helper now clears a disconnect callback left by an unsuccessful internal BlueZ connection attempt only after connect() has returned a live link, and labels PRN changes as adaptation rather than link negotiation. All 81 strict host tests pass. The first physical run before this fix still proved the 3,520 B/s DATA rate and returned a successful validation response, but is correctly recorded as FAIL because the stale host event prevented ACTIVATE.
  • [ ] Warm app-to-bootloader handoff is tested with the application watchdog active; cold recovery is tested separately.
  • [ ] BLE Companion/recovery behavior and radio-reset recovery are exercised.
  • [x] The post-Pi-recovery application still enumerated as exact 2886:8044 / B35E71C1C3726CE7, but one identity-gated, bounded USB Binary Companion attempt received no protocol response. meshcore-cli 1.6.0 printed both of its no-response error lines and nevertheless exited zero; the supervisor treated the attempt as a failure and independently proved that no process retained the serial port. The exact terminal-stop token and an identity/path-gated USB bus reset also failed, which excludes an ordinary startup Terminal-mode hold and host USB endpoint state. The external tool defect remains in upstream meshcore-cli 1.6.3: its serial creation failure returns normally, so callers must validate command output instead of trusting exit zero. The LoRa-OTA controller has an exact three-diagnostic regression and continues to fail closed.
  • [x] An exact hash-recorded Full Companion application DFU erased/replaced only the application and completed in 36 seconds. The transferred ZIP was 440,083 bytes with SHA-256 9946b8417c389784ac92c20e4ce5cf0b6a0123a58136627c0689815637f96992. The same stable USB serial returned as interface 00; USB and the existing bonded BLE identity both reported Seeed Xiao-nrf52, the 31-Aug-2026 build, and Companion role. Preserved settings included a roughly six-month stale clock, which Binary Companion clock sync advanced to one second of the Pi without changing the role.
  • [x] The recovered application passed explicit Terminal start/stop, exact board query, saved usb.logging off, source-only OTA status, and a real serial motatool serve attach/detach with an empty folder (COUNT -> 0). Binary Companion returned after both terminal and mOTA ownership. Enabling USB logging and rebooting produced exact interfaces 00 and 02; interface 02 emitted 123 bytes of entirely printable log data while Binary Companion stayed usable on 00. Disabling and rebooting removed 02, restored the single-interface descriptor, and read back logging off.
  • [ ] A physical power-removal cold test is still needed to prove persistence of the current application. It cannot retrospectively determine whether a power cycle alone would have cleared the original stall. Answer that only after reproducing the stall under a bounded stress loop, capturing state, and power-cycling before any DFU. The application DFU proves recovery but intentionally does not claim the underlying stall trigger is fixed.

Seeed Tracker T1000-E

  • [x] Stable application identity 34A9141999729D5D is recorded and is not confused with the RAK3401 even though both currently enumerate with USB product ID 239a:8029. Application is 239a:8029; the exercised bootloader is 2886:0057, with the same stable serial and exact T1000-E/T1KE_DFU model/name gates.
  • [x] A post-Pi-recovery, identity-gated, read-only USB Companion query returned the exact T1000-E model, non-repeater Full Companion role, 22 dBm maximum, and 910.525/62.5/SF7/CR5. Its clock readback was within the query's few-second host interval. No clock, radio, role, key, or settings write was issued.
  • [ ] Board, version, repeater role, bootloader status, and USA radio tuple are correct after erase/install and after reboot. The live baseline and a 2,184 ms software reboot passed: 3caf9dcf, Seeed Tracker T1000-E, 22 dBm, 910.525/62.5/SF7/CR5, OTA target 7B071FA0, bootloader 0.11.0-OTAFIX2.4.3, and full OTA base hash 3E92B157128B6963. A real power-removal boot is still required.
  • [ ] UF2 and serial DFU installation paths complete without host I/O errors. On OTAFIX 2.4.3, an identity-gated root copy of the exact old UF2 returned success in 112 ms, but the device disconnected before host writeback was durable: sync -f failed with EIO after 17,695 ms and the kernel logged lost asynchronous writes/FAT read errors. The old application happened to boot and retained its settings, but this transport is a FAIL. Retest with the corrected 2.4.4 candidate before qualifying T1000-E UF2.
  • [x] The installed t1000e_repeater_lora_ota_no_external_sensors profile does not compile the Companion BLE transport and did not advertise during a post-reboot BlueZ scan; that is expected for this repeater image. BLE Companion qualification belongs to the separate T1000-E Companion build.
  • [x] The exact latency-disable bootloader candidate ZIP (SHA-256 ebe9739ce40c2c7cc1f52076f9614194959fcf6df93d37666b69b7bd415fab68) installed through identity-gated serial DFU. The final output contained exactly one terminal Device programmed. success line, the bootloader returned as 2886:0057, and its embedded manifest identifies T1KE_DFU, S140 7.3/FWID 0x0123, application base 0x27000, and test version 0x02040403.
  • [ ] The later connection-policy-reapply combined ZIP (SHA-256 c2de1fb2704154b4abc371b6b4827c32ad09c4d15f3f6d97d0f4182bb51fff58) was identity-gated, transferred, validated, activated, and returned as exact 2886:0057 recovery USB. INFO_UF2.TXT reported 0.11.0-OTAFIX2.4.3-3-gffb1580-dirty-test-version-0x02040403, S140 7.3.0, and the T1000-E board. The earlier and later candidates share the same visible version string, however, so package sequencing and INFO_UF2 do not independently prove the installed bootloader bytes. Require an identity-gated live bootloader-region readback or a uniquely versioned rebuild before checking this item.
  • [x] The combined-image BLE transport itself passed: 193,688 bytes at 3,331 B/s, target CRC validation, and target-initiated activation. The package contains SoftDevice plus bootloader and APP=0, so remaining in recovery USB was the correct postcondition. Its first supervisor record is not accepted as run-level evidence: TERM inherited status zero and wrote a false PASS at STAGE=post_copy_usb. A separate audit records the transport result, and all later wrappers map INT/TERM to nonzero and refuse PASS unless STAGE=complete.
  • [x] A separate exact application recovery after that combined transfer completed 404,468 bytes at 2,095 B/s, validated, activated, and returned stable application USB in 225,899 ms. The lower rate is the expected cold/unbonded class, not the bonded warm baseline.
  • [x] A cold exact Full Companion application ZIP (SHA-256 b3be000ab2391527a325cc868e2706dec291c167b6e6cd3495908addb8af6c65) completed all 404,468 DATA bytes over BLE at 1,572 B/s. The complete workflow took 281,236 ms (about 1,438 B/s end to end, including setup, validation, activation, and reboot). ATT MTU reached 247, but BlueZ's independently cached maximum write size was still 20, so the identity-gated helper correctly used 20-byte packets, adapted PRN 8 -> 16 -> 32, validated CRC, activated, observed the target-initiated disconnect, and gated the exact USB application return.
  • [x] The Full Companion application advertised as MeshCore-F9AD7082 at the expected address generation, paired with the qualification PIN, and exposed its bonded, MITM-protected Legacy DFU service after connection. That DFU service is intentionally absent from the advertisement, so an advertisement-only check is insufficient. A 1200-baud USB touch instead selects serial-only DFU and correctly produces no BLE advertisement; use the bonded application control characteristic for a warm BLE handoff.
  • [x] Exact same-address bonded warm buttonless DFU passed. The application advertisement, Paired/Bonded state, connected BLEDfu service/control/ revision layout, encrypted notification/write gate, observed target disconnect, T1KE_DFU advertisement, DIS model T1000-E, ATT MTU 247, and 244-byte write capability were all checked. All 404,468 bytes were confirmed in 121.081 seconds at 3,340 B/s; DFU through target disconnect took 144.076 seconds and the full supervisor took 176.958 seconds. CRC, activation, and exact 239a:8029/34A9141999729D5D USB return passed.
  • [x] Two accepted in-DATA 30 ms -> 15 ms -> 30 ms controller experiments completed without corrupting either image, but were decisive negative performance results. The first finished DATA in 425.398 seconds at 951 B/s; the later candidate finished DATA in 449.063 seconds at 901 B/s. HCI capture found the cause: hcitool lecup silently encoded both connection-event-length hints as 0x0001 (0.625 ms). The Pi's Broadcom 4.2 controller consequently scheduled about one ACL fragment per event: roughly 64 fragments/s at 15 ms and 32 fragments/s after the 30 ms "rollback", versus roughly 120--144 fragments/s on the untouched link. The later pre-START controls prove that 15 ms itself is not slow; the restrictive event-length hint and in-DATA update/rollback were unsafe.
  • [x] Two exact 15 ms/latency-0, zero-event-length lab controls passed: t1000-ble-event-length-prestart-20260830-071549-QOeVuw completed all 404,468 DATA bytes in 64.905 seconds at 6,232 B/s (95.635 seconds through target disconnect; 112,639 ms full workflow), and t1000-ble-event-length-prestart-20260830-071825-VE9Re3 completed DATA in 62.070 seconds at 6,516 B/s (92.794 seconds through target disconnect; 106,807 ms full workflow). Both used exact 0/0 no-preference minimum and maximum connection-event lengths. On each bootloader connection the HCI trace contains exactly one successful update before START and none from START through DATA, validation, activation, and the target disconnect; Update Complete preceded START by 7.535 and 7.536 seconds. Completed-packet clusters followed the 15 ms connection anchors, while the median full-size DFU write gap was 29.997 ms in both traces. Exact package hash, CRC, activation, and stable application USB return passed twice.
  • [x] t1000-ble-event-length-prestart-20260830-071358-ogzmRy is the bounded negative control. HCI accepted exact 15 ms/latency 0 with both event-length hints set to 10 ms, then reported Connection Timeout (0x08) 6.810 seconds after Update Complete during the 20-second pre-START pause. The run failed closed at STAGE=pre_start_update in 44,227 ms; START and application DATA were never sent. A 10 ms event-length hint is therefore rejected for this adapter/target pair.
  • [ ] Production BLE DFU keeps the controller-negotiated link and issues no raw host connection update unless that exact adapter/target profile has been separately graduated. The opt-in lab path may issue exactly one request on the bootloader connection before START while sent/confirmed are zero: exact interval and latency, 0/0 no-preference event-length hints, bounded pause, and a matching HCI Update Complete are mandatory. Failure or mismatch disconnects before START. Never update or roll back the link during DATA; only the protocol PRN/window may adapt there. Compare complete repeated control/candidate DFUs offline before graduating a production profile.
  • [ ] Repeat the cold transfer with the host's bounded BlueZ readiness wait. When ATT MTU is greater than 23 while the cached write capability remains 20, it polls the same connected DFU characteristic for at most three seconds, aborts on disconnect, disappearance, or characteristic-handle change, and otherwise falls back safely to 20. The four readiness unit cases and all 81 strict helper tests pass; a physical 20 -> 244-byte promotion is still required.
  • [ ] Restore the protected t1000e_repeater_lora_ota_no_external_sensors application (target 0x7B071FA0, exact ZIP SHA-256 9b9be2bee9ebb6644d22806d404fe02c415d037707713d206fa43a8c689cf632) through identity-gated application DFU while retaining the candidate bootloader. Do not use the generic repeater image or the unqualified UF2 path. Verify the protected private/public identity without printing the private key, plus exact name, latitude/longitude, build/body/OTA target, USA radio, 22 dBm, GPS off, power saving on, RX gain on, RXPS on, watchdog on, telemetry access all, no TempRadio/download, and a valid admin login.
  • [ ] LoRa radio reset/recovery is forced and normal receive/transmit resumes. Normal post-reboot bidirectional LoRa is already proven: the V4 learned a zero-hop path, received the exact message-specific ACK in 558 ms, and received the T1000-E's exact 3caf9dcf version reply. A deliberately forced LR1110 recovery event is still required.
  • [x] Direct V4 LoRa OTA receive, apply, reboot, and post-update verification passed from exact bf092b24 body 9443FB5E6F23FC1B to exact 3caf9dcf body 3E92B157128B6963 on OTAFIX 2.4.3. The mandatory fixed three-minute rehearsal passed at 909.950/250/SF5/CR5, including natural expiry and return to 910.525/62.5/SF7/CR5. The 2,036-byte/two-block delta reached ready in six seconds, the new body replied at the 50-second probe, and the complete safety-wrapped runner took 519 seconds. V4 source RXPS and both normal radio tuples were restored. The co-located T1000 was temporarily reduced to 0 dBm and restored to 22 dBm using bounded retries across its RXPS busy windows.
  • [x] Independent post-LoRa-OTA USB verification matched the protected pre-run private key, public key, name, latitude, and longitude without printing private material. GPS remained off; device power saving, system watchdog, RX gain, and RXPS remained on; telemetry access remained all; no TempRadio lease or download remained. blrc:B8 is the expected OTAFIX application-apply success diagnostic. A subsequent software reboot reset uptime and preserved the exact build/body, name, radio, 22 dBm, GPS, power-saving, RXPS, watchdog, and no-TempRadio state. The CDC symlink did not disappear during this warm reset, so uptime/readback—not USB removal— is the required reboot proof on this board.
  • [x] GPS was initially off, enabled successfully under device power saving, reported active/no-fix/zero-satellites with its duty-cycle deadline, and was restored to off. Device power saving reported on, RXPS reported on,18205,20423, GPS advert policy prefs, and telemetry access all.

RAK3401

  • [x] Stable USB identity and SWD FICR identity are independently matched.
  • [x] Source-level WisBlock peripheral pins are complete and non-conflicting: I2C SDA/SCL are P0.13/P0.14 through both PIN_WIRE_* and PIN_BOARD_*; GPS-module RX crosses to MCU UART TX P0.16, GPS-module TX crosses to MCU UART RX P0.15, and PPS is P0.17. A static contract test prevents these aliases from silently disappearing or being uncrossed.
  • [x] Ordinary/full-sensor RAK3401 profiles use the source-built Adafruit BME680 provider, not Bosch's precompiled BSEC archive, so upstream issue #3292's soft-float/hard-float link mismatch is not reachable there. The reduced LoRa-OTA profile deliberately omits BME680/BSEC and the other declared optional environmental/ranging drivers. A static contract test requires the hard-float path fix and BSEC dependency if BSEC is enabled in this board or an inherited full-sensor recipe later.
  • [ ] With the RAK13302 radio installed, detect a RAK12501/L76K UART GPS from a complete checksum-valid NMEA sentence in sensor slot A. Confirm random UART bytes, an incomplete sentence, and a bad checksum do not claim GPS.
  • [ ] Detect RAK12500 I2C GPS alone at 0x42; exercise GPS power saving and confirm the shared 3V3_S rail, OLED, RTC, and other slots remain powered.
  • [ ] Detect INA3221 alone at the firmware-configured 0x42 and prove the discovery path does not send u-blox configuration traffic to it.
  • [ ] Exercise RAK12500 at 0x42 and INA3221 at 0x43 together after strapping INA3221 A0 to SCL and building with -DTELEM_INA3221_ADDRESS=0x43; verify both retain independent telemetry.
  • [ ] With a representative omitted environmental sensor attached, confirm the reduced image leaves the generic I2C bus healthy but does not advertise a driver that was intentionally removed.
  • [x] Local USB terminal probes are terminated with carriage return (\r). This CLI echoes an LF-only command without executing it; an echoed ver with no reply is therefore a bad probe, not evidence of a hung application. Confirm any suspected hang with a CR-terminated command, LoRa status, and SWD state before resetting the board.
  • [x] Assert CDC DTR when probing the RAK terminal. With DTR deasserted, writes reached the port but no CLI replies were returned; the same exact stable-identity port returned the expected version immediately after DTR was asserted. Keep 1200-baud bootloader touch separate from 115200-baud application probes, and allow a full reply window so delayed responses are not attributed to the following command.
  • [x] Installed S140 6.1.1 FWID (0x00B6) and runtime application base (0x26000) are independently verified.
  • [x] Bootloader version and apply capabilities are independently reported.
  • [x] UF2 application copy returns success; sync -f returns success; no USB reset/timeout or offline-sector errors occur; SWD readback matches every UF2 payload byte. Current 2.4.4-preview.2 result is PASS in both current→old and old→current directions.
  • [x] Board, repeater role, version, name, and USA radio tuple are verified after serial recovery and again after a real cold boot.
  • [x] Six consecutive authenticated ota self attempts leave the application healthy. In the qualification run, two login/reply pairs were lost, one login succeeded while its command reply was lost, and the next three commands replied in about 2.5 seconds each. A CR-terminated local ver probe passed after every attempt; packet loss did not become a node hang.
  • [ ] An unchanged admin login does not rewrite the ACL flash file, while an actually changed/newly learned persistent route schedules exactly one delayed save and survives reboot. Cover delayed flood-path replies, already-unknown paths, changed permissions/secrets, new/evicted clients, and the intentionally non-persistent guest case. Do not rely on an unrelated login write to capture a later PATH response.
  • [ ] Location and private-key persistence are verified without exposing the private key in logs.
  • [x] Serial recovery rejects nrfutil's false exit-0 failure, requires the exact success marker, and produces a byte-exact SWD application readback.
  • [ ] UF2/serial recovery works with both the deployed older bootloader family and the current backward-compatible application.
  • [ ] Bluetooth advertisement and connection work when enabled. The tested repeater profile does not advertise BLE during normal operation.
  • [x] Direct LoRa OTA from the V4 passes with elapsed time, bounded retry behavior, final hash/version verification, exact radio restoration, and independent SWD readback. The qualified bf092b24 base 91CF4EC2882F23D6 reached 3caf9dcf/27A223DDFC3A8F47 using a two-block, 2,132-byte delta: ready in six seconds, new body at the 60-second reboot probe, and 514 seconds for the complete safety-wrapped run. Log: /home/mikec/hwtest/runs/rak3401-0dbm-final-9KWjey61/run.log.
  • [x] The current OTAFIX test version 0x02040403 independently passes a 74-block direct LoRa update from exact 3caf9dcf body 27A223DDFC3A8F47 to HEAD 9fd580c8 body 744BC4443891ED10. READY took 122 seconds and the complete three-minute-rehearsal, transfer, install, reboot, recovery, and restore wrapper took 589.487 seconds. Lost read-only preflight and postboot replies were reconciled by bounded exact retries. USB confirmed the target/hardware/body and bootloader success result, and authenticated LoRa readback confirmed the final 22 dBm restoration. Sanitized log: /home/mikec/hwtest/runs/rak3401-02040403-head-vzWOXVxT/qualification-redacted.log.
  • [ ] Add exact RF packet sent/confirmed counters to the device/host telemetry. The qualified run records two payload blocks, all seeder reads, command retries, and final success, but those are not equivalent to LoRa packet counts and must not be presented as such.
  • [ ] Repeat at each supported bandwidth and record 0-hop, 1-hop, and 2-hop estimates from measured airtime rather than extrapolating only by file size.

RAK4631 reduced profiles

  • [x] Serialized VM contracts distinguish the three application identities: the plain and Serial2 reduced profiles retain the combined WisBlock GPS provider and all four INA voltage/current drivers; the explicit legacy Serial1 profile retains the INA drivers and omits GPS. This is build evidence only, not physical peripheral qualification.
  • [ ] On the plain reduced profile, detect and read RAK12500 I2C GPS, a RAK12501/L76K UART GPS in a separate run, the SSD1306 OLED, a supported autodiscovered RTC, and representative INA telemetry. Verify GPS sleep and repeated discovery never lower the shared WB_IO2/3V3_S rail.
  • [ ] With a UART RAK12501 installed and detected, confirm the Serial2 bridge and GPS remain usable together. Select Serial1 and try to enable the bridge with the GPS preference both on and off; verify each attempt is refused before the bridge starts and does not change GPS transport, acquisition, hold, or cached-fix state. The shared WB_IO2/3V3_S rail must remain high and GPS must remain usable after each refusal.
  • [ ] With only an I2C RAK12500 installed, and again with neither GPS installed, select Serial1 on the merged image and confirm bridge enable is refused without changing saved UART/runtime state. Silence or an I2C-only detection must not defeat the fail-closed RAK12501 reservation. Confirm Serial2 starts and I2C GPS telemetry remains usable.
  • [ ] Exercise RAK12500 at 0x42 and INA3221 at 0x43 together with the address-matched build. Separately place INA3221 alone at the configured 0x42 and prove the identity guard prevents u-blox configuration writes.
  • [ ] With a logic analyzer, hold SDA low and then SCL low before sensor discovery. Confirm the validated board pins receive at most nine open-drain recovery clocks and a STOP only after SCL rises; the firmware must skip discovery if either line remains low. Repeat on ProMicro's remapped SDA 8/SCL 7 and verify the user-button pin is never driven.
  • [ ] Record the remaining core limitation: the pinned Adafruit nRF52 Wire implementation has no transaction timeout. This preflight cannot protect display/RTC calls made earlier in boot or interrupt a peripheral that wedges after a transaction begins. Qualify those cases with a hard power cut/watchdog fixture before claiming bus-hang recovery.
  • [ ] Confirm the explicit legacy Serial1 image exposes neither UART RAK12501 nor I2C RAK12500. Record this as a combined-provider build limitation, not an assertion that RAK12500 electrically conflicts with the UART.

Heltec MeshTower V2 with SD

  • [x] Stable serial 9352162A72082314 and physical downstream-hub path 1-1.2 port 4 identify this board across application and bootloader descriptors; no other attached nRF52 endpoint was selected. The older 2.4.3 bootloader's UF2/serial failure was rejected rather than treated as a successful flash.
  • [x] Exact 1.17.1.5-halo-keymind-cascade-marathon-hwtest-e26d48e4 SD Repeater boots with hardware ID Heltec_tower_v2, target 0A9DBBF0, body hash E9282D470AAC348B, and USA radio 910.525/62.5/SF7/CR5. ota self reports SD apply ABI 3 with codec mask 0x5; ota bootloader reports board 239A0071, target 1150F50E, name TOWER_V2_OTA, and capability byte 09.
  • [x] A post-Pi-recovery identity-gated serial audit reconfirmed the exact Tower V2 model, Repeater role, build, 910.525/62.5/SF7/CR5, target 0A9DBBF0, body E9282D47, SD ABI 3/codecs 0x5, and installed bootloader identity/capability 239A0071/09. The card reported 959.6 MiB free and no active download. This profile exposes the local text console but deliberately has no runtime usb.logging setting; the unknown-setting reply is a capability result, not a transport failure.
  • [x] Card detection and destructive maintenance pass. Normal format took 3,927 ms; an immediate repeat was rejected by the five-minute cooldown; forced format took 3,933 ms; and raw erase plus format took 3,947 ms. The card remounted empty with 16.0 KiB used and 959.6 MiB free, and both completion ages were recorded. Automatic archive capture was then disabled deterministically, leaving only /mota/cache.off.
  • [ ] SD card detection, staging, hash validation, apply, cleanup, and recovery from missing/corrupt media pass.
  • [ ] Multi-page application and bootloader update paths pass across a cold boot.
  • [ ] LoRa OTA receive/apply reports the expected target ID and installed version.

Heltec V4

  • [x] Exact V4 hardware variant is reported (Heltec V4.3 OLED); OLED/TFT variants are not merged.
  • [x] Full Companion boots with USB logging off and Binary Companion selected. The text terminal independently reports the complete build string; protocol ver has a shorter fixed-width version field.
  • [ ] USB logging changes take effect immediately on this single-TTY ESP32 profile, require no reboot, and return to Binary Companion only through the explicit terminal-stop mode change.
  • [x] Bluetooth pairing and Companion protocol operations pass. Qualification required removing the Pi's stale pre-flash bond, pairing through a BlueZ KeyboardOnly agent with the current displayed/reported PIN, then reconnecting; BLE infos and ver returned the exact V4.3 OLED identity.
  • [x] Saved Wi-Fi connects to SlowFi, returns at 192.168.1.51 after the test host reboot, reports Wi-Fi power save min, and remains governed by the configured-network behavior rather than the unconfigured 30-minute shutdown policy.
  • [x] TCP Binary Companion (5000), Wi-Fi OTA (5001), and text CLI (5002) are each tested and cannot be confused with one another. Port 5000 returned the exact Companion identity; 5002 returned the full terminal, version, Wi-Fi, logging, RXPS, and memory state; and a real motatool serve --tcp 192.168.1.51:5001 -v session received COUNT -> 0 from an intentionally empty folder. Disconnect then restored folder:not connected while the 5001 listener stayed active.
  • [ ] For a shared Full-Companion controller/seeder, bind the source identity to the exact public key already read from Binary Companion port 5000. Do not require get public.key from text port 5002: the hardware-tested V4 Full CLI rejects that command. Port 5002 must still prove the local TempRadio state/tuple, while keyed on-air destination replies prove the physical LoRa handoff.
  • [ ] V4 seeds direct and routed LoRa OTA; TempRadio preflight, apply, and automatic normal-radio restoration all pass.

Seeed SenseCAP Indicator LoRa

  • [x] Flashing selects the ESP32-S3 CH340 endpoint, never the Indicator RP2040 ACM endpoint; esptool MAC probing succeeds before erase. The qualifying merged write used 115200 baud and completed with flash hash verification.
  • [x] A fresh merged install completes SPIFFS initialization and reports the intended Full Companion version/role and Seeed SenseCAP Indicator hardware identity. The exact qualifying artifact was SenseCapIndicator-LoRa_companion_radio_full-1.17.1.5-halo-keymind-cascade-marathon-e26d48e4-merged.bin, 2,070,152 bytes, SHA-256 df08b770313ca8776436456525681cf286cd9b5b40e05e7cf55f21ef097f338a; its ELF, map, manifest, and partition signature are archived beside it on the Mercerwood Pi.
  • [x] The current dark-layout Full Companion application was flashed app-only with exact CH340/MAC/partition gates, preserving identity, Wi-Fi, and settings. The 2,147,800-byte application has SHA-256 81bebdc07b6a8349c1c975cb5a0e30c5f6019d35bcac641e5f3d4df951f7406b; esptool verified its flash hash in 134.2 seconds. USB Binary/terminal, saved logging-off state, configured Wi-Fi, BLE, TCP ports 5000/5001/5002, and an empty-folder Wi-Fi mOTA COUNT -> 0 exchange all pass afterward.
  • [x] Keep one identity-gated CH340 descriptor open with DTR/RTS deasserted through startup. On the valid-current-font path, the Indicator streams a 1,302,608-byte font from its RP2040 at 1 Mbps before registering the USB terminal, so allow at least 30 seconds before the first command. A missing/corrupt-font recovery instead runs in the background after the interfaces start. Reopening short probes can reset the ESP32 repeatedly and create a false UART-hang diagnosis; the controlled run returned ver and board after the same apparent-silence condition.
  • [x] Automatic font recovery starts from a physically corrupt RP asset. The setup deliberately installed 64 all-zero bytes, CRC32 758d6336, in place of /meshcore/ui-font.vlw; the ESP32 rejected that asset and kept its built-in fallback font while recovery ran.
  • [x] Every new HTTPS recovery attempt waits for a fresh SNTP callback and a signed epoch at or after the compiled asset publication time before it opens TLS. The forced-resume run observed fresh syncs in 3,700, 400, and 699 ms across three outer attempts, including two failed TLS handshakes; the final normal run synced in 1,999 ms and only then logged the first api.github.com connection. A retained merely plausible clock is not accepted. CA validation and the compiled asset SHA-256 remain mandatory.
  • [x] The release-source normal path downloaded all 1,302,608 bytes into PSRAM in 13,205 ms, closed TLS, verified SHA-256 61bce9662db314054e7bcfaa26147a28ad7b500b51baac4cae1caacce90b7421, staged receiver-paced 512-byte MCFONT STAGEV2 chunks in 36,676 ms, committed, and verified RP CRC32 19f80d64 at 76.7 seconds from reset. An independent MCFONT GET over RP USB then returned exactly 1,302,608 bytes with the same CRC and SHA in 15.273 seconds. The exact installed application is SenseCapIndicator-LoRa_companion_radio_full-1.17.1.5-halo-keymind-cascade-marathon-font-recovery-e26d48e4.bin; its log is /home/mikec/hwtest/runs/indicator-release-recovery-lUZEmBHL/esp-serial.log on Mercerwood. The current RP service reports protocol 2 and completed STAGEV2 accounting; its installed flash was not independently read back, so no RP artifact hash is inferred from the archived candidates.
  • [x] Strict Range recovery passed a forced hardware interruption. Exact test application SHA-256 cfd5aec497cb9919f87c0f82f5deaf8614d411582a709cea1c36275529c0bb68 closed the initial response at verified offset 1,048,576. Its first same-offset resume handshake failed; bounded retry 2/2 then accepted only the remaining 254,032-byte 206 response under the original strong ETag/If-Range, retained the same PSRAM buffer and SHA stream, completed the download in 49,422 ms, staged in 36,702 ms, committed, and passed an independent exact USB readback. The full log is /home/mikec/hwtest/runs/indicator-range-v2-xbKr09wI/esp-serial.log. An earlier forced 512 KiB run proved one correct resume at 524,288 but later exhausted its second reconnect on a TLS failure; it is retained as regression-trigger evidence rather than an end-to-end pass.
  • [x] Rebooting the exact release-source application with the current valid font performed no NTP query, TLS connection, or GitHub request. The valid-font boot log is /home/mikec/hwtest/runs/indicator-release-valid-a84vd7Du/esp-serial.log.
  • [x] The post-qualification shared SNTP-coordinator source compiles in the real Arduino-ESP32 2.x LoRa USB/WiFi profile (1,587,965 bytes, 60.6% of its OTA slot), the dynamically generated ESP-NOW Full profile (2,093,465 bytes, 79.9%), and the matching RP2040 transactional service. After the VM independently reported NTPSynchronized=yes, an end-to-end request to the immutable GitHub asset returned HTTP 200, exactly 1,302,608 bytes, and SHA-256 61bce9662db314054e7bcfaa26147a28ad7b500b51baac4cae1caacce90b7421. A live 524288-1302607 continuation then returned HTTP 206, the same strong ETag, an exact 778,320-byte Content-Length, and a byte-for-byte hash match against the checked-in font tail; the pinned Sectigo E46 root independently verified the current api.github.com chain. The same coordinator also compiles in the real heltec_v4_repeater_observer_mqtt profile: its 1,852,392-byte EndF image has SHA-256 fe8352654437f67488b412e1c1a690f2874e836abb5a5d209654bd030c713d7e. MQTT's three process-global configTime() paths are lease-protected, and a busy asynchronous refresh uses a wrap-safe five-second retry deadline instead of overwriting another service's SNTP callback or recording a false successful refresh. The exact final LoRa Full artifact gate then passed 11/11 linked-image markers, including both requesting fresh NTP time before download and the subsequent TLS-open marker. Its 2,146,312-byte application has SHA-256 f557088a5ef3f49f74ab01ca95fefc76911cca4073f01b77cee4882284fc5110 and 475,128 bytes remain in its app slot; the matching 2,211,848-byte DIO merged image has SHA-256 52a4559774384b54dd7fb81da82c791e2c9564991fcf016bcad3b6ad756c1fb2. The exact ELF, map, capability manifest, and partition signature are archived together under /tmp/indicator-full-daily-ntp.WQ6ozM on the VM pending transfer to the hardware-run archive. The ESP-NOW Full sibling passed the same 11/11 gate with a 2,093,928-byte application (527,512 bytes free), SHA-256 fcd9e0e8a37551a845a234fc10eaa4a9f420d54172039b9241f4696a12d4c65f, and matching DIO merged-image SHA-256 f3637c23f58204f5b752b68a1bd494601cb68bee9c27b664832d40abd03cc186; its complete archive is /tmp/indicator-espnow-full-ntp-gate.YN9T8l. Repeat the negative UDP/123-blocked hardware gate on this exact final source before treating the coordinator refinement as hardware-qualified.
  • [ ] Full Wi-Fi SSID, setup address, and related status text wrap without clipping on the physical display. The new dark palette and exclusive lower pairing block are installed; a fresh bond removal held the random PIN request on-screen for 30 seconds and then an immediate second attempt paired, trusted, and returned the exact Indicator identity over BLE. A human physical-readability confirmation is still required; the automated geometry tests prove the PIN block is centered at four-fifths height and cannot overlap IP, inbox, or status text.
  • [ ] Boot the exact current LoRa and ESP-NOW Full artifacts in both saved secondary-transport modes and capture the Indicator render line. Require LoRa + WiFi, LoRa + BLE, and ESP-NOW + infrastructure WiFi to retain 480x480, and ESP-NOW + BLE to select 320x320 scaled to the panel. A logged emergency 320x320 fallback keeps the unit recoverable but does not pass a native-480 gate.
  • [ ] For all four boots, record free heap and largest internal block before interfaces and after the selected interface is usable. In WiFi mode prove BLE never starts; in BLE mode prove infrastructure WiFi, WebConfig, MQTT, and TCP never start. Keep the primary LoRa or ESP-NOW radio exchanging traffic throughout. For both BLE modes pair/encrypt, negotiate MTU, subscribe, exchange repeated Companion requests, disconnect/reconnect, and repeat after bond removal; exercise network/TCP traffic in both WiFi modes and record the lowest heap/largest-block values.
  • [x] With no saved SSID, setup Wi-Fi powers down after the absolute 30-minute window even if queried or used.
  • [x] A software reboot starts a fresh unconfigured setup window.
  • [x] With a saved SSID, Wi-Fi reconnects to SlowFi, reports minimum modem power save, and remains governed by normal configured-network behavior.
  • [x] Bluetooth Companion pairing and protocol operations pass after saving a deterministic qualification PIN, removing any stale host bond, pairing, disconnecting the bluetoothctl client, and reconnecting with meshcli. Binary infos/ver returned the exact Indicator identity and USA tuple. A second connection after the USB-open reboot and logging-mode exercise also returned the exact build; one initial scanner miss was resolved by an explicit BlueZ discovery pass and was not a firmware advertisement failure.
  • [x] Wi-Fi/TCP Companion, CLI, and OTA endpoints pass at 192.168.1.54: port 5000 returned Binary Companion identity, port 5002 returned the full text version/status, and motatool completed a port-5001 COUNT -> 0 protocol exchange against an empty served folder.
  • [x] Fresh Full starts with USB logging off. On this single-TTY ESP32 profile, logging on/off applies and persists without a descriptor-changing reboot; the same held-open ASCII terminal reported off -> on -> off correctly, emitted live Wi-Fi/radio diagnostics while on, then accepted the explicit terminal-stop token and returned a valid 85-byte Binary Companion device info frame for Seeed SenseCAP Indicator without rebooting. Companion remained independently usable over Bluetooth and Wi-Fi.
  • [x] Normal-radio LoRa transmit and receive pass. At the installed 22 dBm, a co-located discovery request overloaded the nearby receivers and found zero nodes; temporarily reducing only the Indicator to 0 dBm produced three repeater replies at RSSI -29 to -59 dBm, proving both directions. The saved 22 dBm value was restored and independently read back after the test; this near-field result is a lab-power gate, not a firmware failure.
  • [ ] TempRadio sanity check, OTA seeding, and normal-radio restoration pass.

Cross-radio LoRa OTA acceptance

  • [ ] Run a three-minute reachability preflight before staging or changing any repeater radio settings.
  • [ ] Test direct, fully controlled multi-hop, passive TempRadio hop, and mixed controlled/passive paths independently.
  • [ ] Record bytes, LoRa packets sent, packets confirmed, retries, useful throughput, total elapsed time, and per-phase elapsed time.
  • [ ] Confirm every controlled intermediate returns to its saved radio tuple.
  • [ ] Verify target application and bootloader hashes after apply, not merely the OTA tool's final status line.
  • [ ] Exercise interruption/retry and confirm incomplete data cannot be approved or booted.