Archived
Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down #2
Assignees
beatzaplenty
Clear assignees
beatzaplenty
No Assignees
No due date set.
Dependencies
No dependencies set.
Reference: beatzaplenty/nixos#2
Reference in New Issue
Block a user
The reachability check used
cat < /dev/tcp/${NIX_CACHE_HOST}/22, whichblocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (
exec 3<>/dev/tcp/..., no read)returns in ~60ms against a healthy nix-cache instead of always timing out.
Fixing that exposed a second, previously-dormant bug:
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"on a genuinely empty NIX_OPTS arraystill runs one format pass and yields the literal
''rather than anempty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via
eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"then rebuilta 1-element array holding an empty string instead of a 0-element array,
which broke
nix-shell "${NIX_OPTS[@]}" -p <pkg>with a bogus positionalargument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.
Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which blocks forever reading for EOF that never comes -- sshd sends its banner and then holds the connection open waiting for the client to speak next. Every single check hit the 3s timeout and reported "unreachable" unconditionally, regardless of whether the remote builder was actually up. Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read) returns in ~60ms against a healthy nix-cache instead of always timing out. Fixing that exposed a second, previously-dormant bug: `printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array still runs one format pass and yields the literal `'' ` rather than an empty string. A subprocess (e.g. sync-host-keys.sh) reusing this process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt a 1-element array holding an empty string instead of a 0-element array, which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional argument the moment NIX_OPTS was legitimately empty (nix-cache reachable) -- something the first bug had made impossible to ever hit before. Also adds a couple of retries (1s apart) to both checks as a secondary safety net against genuine multi-second blips, on top of fixing the checks themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>