Site Reliability Engineer - Memphis

RESPONSIBILITIES

• Own monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure
• Provide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., PagerDuty), and timeline and severity hygiene
• Run blameless postmortems and drive corrective actions to closed, not filed
• Gate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process
• Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing)
• Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs
• Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects)
• Define error budgets and availability objectives at campus and service boundaries as adopted by the business
• Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus
• Demonstrate monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality
• Treats alert noise as a design failure, not an operator failure
• Improve mean time to detect (MTTD) and mean time to resolve (MTTR) for severity-class events, and ensure blameless postmortems with closed action items.
• Increase the alert actionable ratio and reduce recurrence of incident classes.
• Expand monitoring coverage of critical dependencies across compute, network, storage, power, and cooling.
• Minimize GPU fleet availability lost to infrastructure events.

BASIC QUALIFICATIONS

• Bachelor's degree in systems engineering, computer science, electrical engineering, or STEM discipline
• 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations
• Experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet)
• Experience working across at least two of: compute, network, storage, power, and cooling / facilities telemetry
• Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control room, or NOC partner
• Experience with scripting (Python, Bash) for automation and analysis

PREFERRED SKILLS AND EXPERIENCE

• General experience in at least one systems language (C, C++, Java, Go, Rust, or similar) – not required to be expert in all of them
• Excellent problem-solving skills with a data-driven approach to reliability engineering
• Ability to work collaboratively with cross-functional teams, including NOC, data center operations, hardware failure analysis, and infrastructure engineering
• Proven large-scale incident command experience and calm technical leadership on a bridge, whether from an SRE/NOC bridge or a plant control room
• Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality
• Experience in power plants, nuclear power plants, high-performance computing, data center environments, or other high-consequence industrial control environments (ICS/OT)
• Current or prior work in data centers or facilities like SpaceXSI spanning power, compute, cooling, and related plant systems, including CDU / direct liquid cooling
• Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes
• Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts
• Reliability engineering methods: MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, and vendor RCA/RMA partnership
• Experience with CMMS (e.g., Maximo) and commissioning / integrated systems testing (IST) and operational turnover
• Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries
• Experience running game days, dependency mapping, and closed-loop corrective action programs
• Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry
• Prior work in a fast-paced startup or tech company like SpaceXSI
• Experience in power plants, nuclear power plants, data centers, or facilities like SpaceXSI — current or prior — spanning power, compute, cooling, or related plant systems
• Proven large-scale incident command experience and calm technical leadership on a bridge; nuclear control-room, shift supervisor / Senior Reactor Operator (SRO), Shift Technical Advisor, or Navy nuclear experience counts as incident command
• Cross-team facilitation and systems engineering depth across software and facility boundaries