System76 Stress Test Scripts

Comprehensive hardware stress-testing framework with real-time sensor monitoring, a web UI, thermal characterization, and automated procedures. Designed for System76 hardware validation but works on any Linux system.

Architecture

s76-webui.sh          →  Web UI launcher (Flask on port 5000 by default)
webui/app.py          →  Flask application — dashboard, monitor, APIs, recording
webui/sensors.py      →  SensorManager — coordinates all hardware sensors
webui/thermal.py      →  Thermal characterization procedure
webui/filters.py      →  Journal filter profiles (include/exclude regex with context)
webui/journal.py      →  JournalEntryWatcher — polling-based journalctl reader
webui/runners.py      →  Subprocess runner with timeout, restart, and logging
webui/procedure.py    →  Procedure definition loader (multi-phase stress workflows)
webui/config.py       →  Configuration file loader/saver
webui/ambient.py      →  Ambient temperature sensor client (network thermometer)
stressmon/            →  Hardware sensor library (git submodule)
  cpuusage.py         ←  CPU utilization (per-core % via psutil)
  cpufreq.py          ←  CPU clock speeds
  cputemp.py          ←  CPU core temperatures
  cpuwatts.py         ←  CPU package power (RAPL)
  memusage.py         ←  Memory usage (percent, used, total, available)
  memtemp.py          ←  DIMM temperatures (spd5118)
  sysfan.py           ←  System fan speeds
  gpudata.py          ←  GPU sensors (temp, clock, fan, power, utilization, VRAM)
  drivetemp.py        ←  NVMe/SATA drive temperatures
  cpuinfo.py          ←  CPU model, vendor, P/E core topology
  hwsensors.py        ←  Abstract base class for all sensors
procedures/           →  JSON procedure definitions (multi-phase stress workflows)

Web UI

Start the web UI:

./s76-webui.sh              # Port 5000
./s76-webui.sh 8080         # Custom port

Dashboard (/)

Profile launcher with parameter tuning (runtime, integrated graphics toggle, etc). Shows running processes with live log streaming, stop controls, and elapsed time.

Available profiles:

• CPU + Memory Stress — stress-ng with configurable runtime
• GPU Burn — NVIDIA GPU stress test via gpu_burn
• LLVM Stress Test — CPU/memory/I/O stress via parallel compilation
• Unigine Valley GPU — OpenGL GPU benchmark
• Sensor Monitor — Real-time hardware sensor monitoring
• Full Burn-in Test — Comprehensive multi-phase system validation

Monitor (/monitor)

Real-time hardware sensor dashboard with SSE (Server-Sent Events) streaming, updated every 200ms.

Sensor groups shown:

Section	Data	Units
CPU Utilization	Per-core usage %, overall average	%
CPU Frequency	Per-core clock speed	MHz
CPU Temperatures	Per-core + package	°C
CPU Power	Package, Cores, Platform (RAPL)	W
Memory	Used %, total/used/available	GB, %
DIMM Temperatures	Per-DIMM (spd5118 sensors)	°C
Fans	Per-driver fan speeds	RPM
Drives	NVMe/SATA composite + sensor temps	°C
GPUs	Temp, clock, fan, power, VRAM usage, utilization	°C/MHz/RPM/W/MB/%

The CPU section is collapsible — by default only package-level values are shown; click "Show all cores" to expand per-core data.

Other features:

• Recording: Click "Record" to log all sensor data to CSV with timestamped samples. Downloaded as sensors.csv. When launched from a procedure, CSV is saved inside the run directory alongside reports.
• Summary bar: Quick-glance CPU model, core count, utilization, and memory.

Procedures (/procedures)

Multi-phase stress workflows defined as JSON in procedures/. Each procedure can include:

• stress-ng phases: CPU, memory, I/O, HDD, VM, cache, socket stressors with per-stressor runtime config
• GPU burn phases: NVIDIA GPU stress via gpu_burn
• Thermal characterization: Automated cooldown → stress → cooldown cycle with sensor min/max reporting
• Sequential orchestration: Phases run in order; results collated per phase

Built-in procedures:

Procedure	Phases
CPU Stress	stress-ng CPU × N cores
GPU Stress	GPU burn
CPU + GPU Mixed	stress-ng CPU + GPU burn (concurrent)
Advanced CPU	Per-core stress-ng with rotation
Compilation Stress	LLVM/clang parallel builds
Burn-in	CPU, memory, GPU, HDD, I/O, thermal in sequence
Thermal Test	Cooldown → stress-ng all cores → cooldown with sensor logging
Full System	CPU + memory + I/O + GPU (concurrent)
Full System Advanced	Full system + per-core CPU + HDD
Meta-Mix	Mixed stress-ng + GPU + compilation

Run History (/runs)

Browse past test runs with per-run detail pages showing:

• Sensor CSV data download
• Thermal reports (min/max temperatures during stress)
• Journal log with filter profiles
• Run journal (journalctl entries tagged stress-test for the run duration)

Journal Viewer (/journal)

Live journalctl viewer with filter profiles:

• Include/Exclude patterns: Plain substring or /regex/flags syntax. Multiple patterns per profile.
• Context lines: Configure context_before and context_after to capture stack traces and error context around matches.
• Built-in profiles: Errors & Warnings (3 lines before, 10 after), Hardware Errors (3 before, 5 after), Stress Test Markers, All Entries.
• Custom profiles: Create and save via the inline editor.
• Test button: Test patterns live against the current journal.
• Duplicate collapsing: Consecutive duplicate lines are grouped with a count badge.

Configuration (/config)

Edit ambient temperature sensor IP/port and file server settings.

Data Recording

Manual Recording

Click "Record" on the monitor page. All sensors are sampled every 200ms and written to logs/sensors.csv. Click "Stop" to finalize; "Download CSV" to retrieve.

Run Recording

When a procedure is launched, recording starts automatically into the run's directory (logs/<run_timestamp>/). Files created:

• <run_id>_sensors.csv — Raw sensor time series
• <run_id>_sensors.txt — Summary statistics (min/max/mean per sensor)
• run_journal.log — Journal entries tagged with stress-test
• Thermal reports — thermal_report.txt and thermal_report.json (generated during thermal characterization)

Hardware Sensors

All sensors follow the HWSensorBase abstract class and are updated concurrently via a ThreadPoolExecutor every 200ms.

Sensor	Source	Data Fields
CPU Utilization	psutil.cpu_percent(percpu=True)	Per-core + overall average %
CPU Frequency	psutil.cpu_freq()	Per-core current/min/max/mean MHz
CPU Temperature	psutil.sensors_temperatures()['coretemp'|'k10temp']	Per-core + package °C
CPU Power	RAPL sysfs (/sys/class/powercap/intel-rapl:*)	Package, Cores, Platform, Uncore W
Memory	psutil.virtual_memory()	Percent, used, total, available, free
DIMM Temp	psutil.sensors_temperatures()['spd5118']	Per-DIMM °C
Fans	psutil.sensors_fans()	Per-driver RPM
GPU	NVML (NVIDIA) or hwmon (AMD)	Temp, clock, fan %, power, utilization %, VRAM MB
Drives	pySMART (NVMe/SATA)	Composite + per-sensor °C

Journal Integration

The web UI runs a background thread that polls journalctl for entries with a custom identifier:

journalctl -t stress-test

Scripts output structured journal entries:

echo "Stressor started: cpu_all_cores" | systemd-cat -t stress-test

The journal viewer applies regex-based filter profiles with context extraction — matching lines plus N lines before/after are displayed.

Thermal Characterization

The thermal test procedure (procedures/thermal_test.json) runs a three-phase cycle:

1. Initial cooldown — Waits for system to reach idle temperatures
2. Stress phase — Runs stress-ng CPU stressors on all cores while recording sensor snapshots every second
3. Final cooldown — Monitors sensor recovery after stress stops

Outputs:

• thermal_report.txt — Human-readable min/max for all sensors during stress
• thermal_report.json — Machine-readable report for further analysis

Requirements

• OS: Linux (tested on Ubuntu 22.04+ / Pop!_OS 22.04+)
• Python: 3.10+
• Python packages: flask, psutil, pySMART, requests
• Optional: nvidia-ml-py (NVIDIA GPU monitoring), system76-power (System76 GPU switching)

CLI Scripts

Beyond the web UI, individual scripts are available for direct use:

Script	Purpose
s76-stress-tests.sh	Main CLI orchestrator with option flags
s76-stress-ng.sh	Run stress-ng with configurable load
s76-gpu-burn.sh	NVIDIA GPU burn-in test
s76-unigine-valley.sh	Unigine Valley benchmark
s76-llvm-stress.sh	CPU/memory stress via LLVM builds
s76-burn-in.sh	Terminal-based full burn-in (runs s76burnin.py)
s76-webui.sh	Web UI launcher
s76-journalctl.sh	Tail journal entries tagged for stress test
s76-disable-suspend.sh	Prevent suspend during tests
s76-htop.sh	Open htop in a new terminal
s76-setup-testpy.sh	Install dependencies for sensor monitor
s76-testpy.sh	Launch legacy curses sensor monitor
test.py	Legacy curses-based sensor monitor script
s76-check.sh	Pre-flight system check

License

MIT