BjTCUdZddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZiZded<dD]ZeeedZeeeee<dd Zdd Zdd Zdd Zddd Zdd ddZddZddZddZddZy)uShutdown forensics — capture context when the gateway receives SIGTERM/SIGINT. The gateway's ``shutdown_signal_handler`` runs synchronously inside the asyncio event loop. We can't safely block it for long, but we DO want a durable record of who/what triggered the shutdown so that "the gateway keeps dying" incidents can be diagnosed after the fact. This module exposes :func:`snapshot_shutdown_context`, a fast (<10ms), non-blocking probe that returns a structured dict the signal handler can log immediately, plus :func:`spawn_async_diagnostic`, a fire-and-forget ``ps`` walk that runs as a detached subprocess so it can't block teardown even if /proc is wedged. Anything that needs to wait (e.g. shelling out to ``ps aux``) belongs in the async helper, never in the synchronous probe. ) annotationsN)Path)AnyDictListOptionalzDict[int, str]_SIGNAL_NAME_BY_NUM)SIGTERMSIGINTSIGHUPSIGQUITSIGUSR1SIGUSR2c|y t|}tj |d|S#ttf$rt|cYSwxYw)zBReturn a human-readable signal name (or ``str(sig)`` as fallback).UNKNOWNzsignal#)int TypeError ValueErrorstrr get)sigsig_ints ?/home/ubuntu/.hermes/hermes-agent/gateway/shutdown_forensics.py _signal_namer%sS {c(  " "7ggY,? @@ z "3xs )AAc td|dd5}|D]D}|j|dzs|jdddjccdddS dddy#1swYyxYw#tt t f$rYywxYw)zIRead a single field from /proc//status. Linux only; None elsewhere./proc/z/statusutf-8encoding:N)open startswithsplitstripFileNotFoundErrorPermissionErrorOSError)pidkeyfhlines r_read_proc_fieldr-0s F3%w'' : 9b 9??39-::c1-a06688 9 9 9 9   9   8   s?A3A'$A' A3A'A3'A0,A30A33B  B c  td|dd5}|j}dddsy|j ddj dd jS#1swY=xYw#tttf$rYywxYw) zLRead /proc//cmdline as a printable string. Linux only; None elsewhere.rz/cmdlinerbN rreplace)errors)r"readr&r'r(r2decoder%)r)r+datas r_read_proc_cmdliner7<s F3%x($ / 2779D   << & - -gi - H N N PP    8s'A+AA+A($A++BBc:d|i}|dkr|St|d}|||d<t|d}|||d<t|d}| t||d <t|d }||r|jdn||d <t |}|r|dd |d <|S#t$rYJwxYw)zCompact /proc/ snapshot: pid, ppid, state, uid, cmdline. Best-effort. Missing fields are simply omitted rather than raising. r)rNameNnameStatestatePPidppidUiduid,cmdline)r-rrr$r7)r)summaryr:r<r>r@rBs r _proc_summaryrDIs %clG ax C (D  S' *E   C (D  !$iGFO 3 &C +.QC %G$Tc]  N   sB BBc tj}tj}tj}tj}||t || t |nd||t|t|d}tjjd}|r||d<tjjd}|r||d<t|xs|dk(|d< tjd |d < t|d }|N|d k7rI|jr t |n||d <|jrtt |nd|d< tjjd} | rt%| dz } | j'r. | j)d} | dd|d<d|| vxsd|| v|d<t%| dz } | j'r | j)d} | dd|d<|S|S#ttf$rYwxYw#t t"f$rYwxYw#t$rYswxYw#t$rY|SwxYw#t*$rY|SwxYw)a'Fast (<10ms) snapshot of who/what is asking us to shut down. Captures: * The signal number/name (so SIGINT vs SIGTERM is visible) * Our own PID/ppid + parent process info from /proc (Linux) * Whether systemd is our parent (``ppid==1`` or ``INVOCATION_ID`` set) * Whether takeover/planned-stop markers exist (consumed lazily by the caller) * /proc/self limits + load average (1-min) * Wall-clock and monotonic timestamps for cross-correlating later phases Pure stdlib, never raises, never blocks on subprocesses. N)ts ts_monotonicsignal signal_numr)r>parentself INVOCATION_IDsystemd_invocation_idJOURNAL_STREAMsystemd_journal_streamr! under_systemdr loadavg_1m TracerPid0 tracer_pidtracer HERMES_HOMEz.gateway-takeover.jsonrrrAtakeover_markerz"target_pid": z'target_pid': takeover_marker_for_selfz.gateway-planned-stop.jsonplanned_stop_marker)time monotonicosgetpidgetppidrrrDenvironrbool getloadavgr(AttributeErrorr-isdigitrrrexists read_text Exception) received_signalnowr[r)r>ctx invocation_idjournal_streamrUhermes_home_str takeover_pathrawplanned_stop_paths rsnapshot_shutdown_contextrphs ))+C I ))+C ::H??I I III IIg@)timeout_secondsc |jjddtjdk(ryd|dt j d} t jt|t jt jzt jzd} tjd |d d d |g|tjtjdd } t j"||j$S#t$rYywxYw#t$rYywxYw#t tf$rN t j"|n#t$rYnwxYwY t j"|y#t$rYywxYwwxYw#t$rY|j$SwxYw# t j"|w#t$rYwwxYwxYw)aFire-and-forget ``ps``-style snapshot written to ``log_path``. Runs as a detached subprocess so it can't block the asyncio event loop or compete with platform teardown. The subprocess uses its own ``timeout`` so a wedged ``ps`` still self-cleans within ``timeout_seconds``. Returns the subprocess PID on success, ``None`` on failure. Never raises. We deliberately avoid ``subprocess.run(["ps", "aux"])`` from inside the signal handler (the pre-existing pattern): on a busy host with hundreds of processes, ``ps aux`` can take >2s to walk /proc, during which the asyncio loop is frozen and adapter teardown can't begin. T)parentsexist_okNwin32z echo '=== shutdown diagnostic @ z ==='; echo '--- date ---'; date -u +%Y-%m-%dT%H:%M:%SZ; echo '--- ps auxf (top 60 by cpu) ---'; ps auxf --sort=-pcpu 2>/dev/null | head -60; echo '--- pstree of self ---'; pstree -plau a 2>/dev/null | head -40 || true; echo '--- /proc/loadavg ---'; cat /proc/loadavg 2>/dev/null || true; echo '--- recent dmesg (oom/killed) ---'; dmesg -T 2>/dev/null | tail -20 || journalctl --user -n 20 --no-pager 2>/dev/null | tail -20 || true; echo '=== end ==='itimeoutz.0fbashz-c)stdoutstderrstdinstart_new_session close_fds)rJmkdirr(sysplatformr\r]r"rO_WRONLYO_CREATO_APPEND subprocessPopenSTDOUTDEVNULLr&closer))log_path signal_namerqscriptfdprocs rspawn_async_diagnosticrs*dT: ||w +;-8  }%  WWS]BKK"**$D E? D  D  DDE</EE< EE<EE<FE,, E87E8;E<<F? FFGF10G1 F=:G<F==Gc |jdd}|jdxsi}|jdd}|jdxsd}|jdxsd}|jdrd nd }|jd }t|ttfr|d nd}g} |jd )|jd} | j d| rdnd|jd| j d|jdr| j d|d| rddj | znd} d|d|d|d|d|| d| S)z?Render a shutdown context dict as a single, scannable log line.rH?rJrBz (unknown)r:r)rPyesnorQz.2frWrXztakeover_marker_present=rKotherrYzplanned_stop_marker_present=yesrTz tracer_pid= zsignal=z under_systemd=z parent_pid=z parent_name=z loadavg_1m=z parent_cmdline=)r isinstancerfloatappendjoin) rirrJ parent_cmd parent_name parent_pidrPloadload_strextrasfor_self extras_strs rformat_context_for_logrsw ''(C C WWX  $"FI{3J**V$+KE")cJ WW_5E4M 77< D *4#u >$sCHF ww !-7756 &vw&G H  ww$%1 78 ww| C $5#678-3#((J #&( \""m$Z ,$  )cf tj|tdS#ttf$rYywxYw)zFJSON-serialise a context dict for structured ingestion. Never raises.T)default sort_keysz{})jsondumpsrrr)ris rcontext_as_jsonr:s1zz#sd;; z "s 00cjtjjd}|syd} tdd5}|D]O}d|vs|j j d}t |D]}|jds|}n|sOnddd|syd}dggfD]} tjd g|d |d d d d } | jdk7r6| jjD]b}|j!ds|j dddj } | j#r t%| }n t'| }|bn|n|y|dz } d} || z} || || | | kdS#1swYxYw#ttf$rYwxYw#ttjtf$rYwxYw)u.At startup, sanity-check that systemd's TimeoutStopSec >= drain_timeout. When the gateway is run under a stale systemd unit file (e.g. the user upgraded hermes-agent but never re-ran ``hermes setup`` to regenerate the unit), ``TimeoutStopSec`` can be smaller than the configured ``restart_drain_timeout``. Result: SIGTERM arrives, the drain starts, and systemd SIGKILLs the cgroup mid-drain — looks like a phantom kill in the journal because the journal only logs ``code=killed status=9``. Returns ``None`` when the alignment is fine OR we can't determine it (not running under systemd, ``systemctl`` unavailable, etc.). Returns a dict with ``timeout_stop_sec`` + ``drain_timeout`` + ``mismatch`` bool when we have data to report. Best-effort. Never raises. rLNz/proc/self/cgrouprrz.service/z--user systemctlshowz--property=TimeoutStopUSecTg@)capture_outputtextrvrzTimeoutStopUSec==r!g.Ag>@)unittimeout_stop_sec drain_timeout expected_minmismatch)r\r_rr"r%r$reversedendswithr(r&rrunTimeoutExpired returncoderx splitlinesr#rcr_parse_systemd_duration_to_us)rrj unit_namer+r,partsp timeout_usflagresultvaluerheadroomexpecteds rcheck_systemd_timing_alignmentrBs "JJNN?3M  $I % 8 B % JJL..s3E%e_"::j1()I!"!    !%JR  ^^UtUVUYU8TU#$F    ! MM,,. D12 3*1-335==?!$UJ!>u!EJ)   ! -0!K/Hx'H,& $x/  k   & '   ":#<#E/=E/E/E;"F/E84E;;F  F F21F2cp|sydddddddd}d}d }d }|d zD]}|js|d k(rL|rD|j|j}||sy |tt ||zz }d }d }||z }d|j r||z }z|rE|rC|j|j}|y |tt ||zz }d }d }|s|r |tt |dzz }d }|dkDr|SdS#t $rYywxYw#t $rYywxYw#t $rYywxYw) zParse 'TimeoutStopUSec=1min 30s' / '90s' style values to microseconds. systemd accepts a wide grammar; we cover the common cases (s, ms, min, h) and return None on anything unexpected. Never raises. Nr!ii@Bil$'-)usmsssecminhhrrrr.)rcrlowerrrrisalpha)rnunitstotal_ustokendigitsch multipliers rrrs    EH E FCi  ::<29"YYu{{}5 %V E&MJ$> ??H bLF ZZ\ RKE 5;;=1J! Cf :;;FE E Cf 9::FA B !|8--1"      s6D;D!D( DD D%$D%( D54D5)rrreturnr)r)rr*rr Optional[str])r)rrr)r)rrDict[str, Any])N)rgrrr)rrrrrqrr Optional[int])rirrr)rrrzOptional[Dict[str, Any]])rnrrr) __doc__ __future__rrr\rHrr~rZpathlibrtypingrrrrr __annotations___namegetattr_valrrr-r7rDrprrrrrrrrs"#  ,,')^( M/E 65$ 'D ).CI&/ A  Q>ZB! QQQ Q  QhBTn5.r