mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-15 12:50:42 -07:00
XariannandGitHub
1db0fdf645
Add three environment variables to prevent miopenStatusUnknownError and system stuttering during inference on RDNA4 GPUs: - MIOPEN_USER_DB_PATH: redirect MIOpen kernel cache to writable, persistent dir - MIOPEN_CUSTOM_CACHE_DIR: same, for custom operator cache - MIOPEN_FIND_MODE=FAST: use heuristic kernel selection instead of exhaustive benchmarking, which fails on RDNA4 with ptr: 0 size: 0 workspace warnings MIOPEN_FIND_MODE=FAST does not affect output quality. All MIOpen kernel variants produce the same numerical result; fast mode selects a known-good kernel using heuristics instead of benchmarking every variant on the GPU. Tested on RX 9070 (gfx1201) with ROCm 7.2 and PyTorch 2.12.1+rocm7.2. Hardware note: tested on Ryzen 7 9800X3D + RX 9070 with Gigabyte B650M DS3H motherboard. The exhaustive benchmarking failures may be related to IOMMU behavior on this platform. This system was affected by an IOMMU bug patched upstream in kernel 6.19.10, which may be a contributing factor. May not affect all RDNA4 systems. MIOPEN_FIND_MODE=FAST is a safe default regardless. Depends on PR #862 which fixes the broken ROCm Docker build.
49 lines
1.9 KiB
YAML
49 lines
1.9 KiB
YAML
---
|
|
# ROCm (AMD GPU) overlay for Voicebox
|
|
#
|
|
# docker compose -f docker-compose.yml -f docker-compose.rocm.yml up --build
|
|
#
|
|
# Requires ROCm drivers on the host:
|
|
# https://rocm.docs.amd.com/projects/install-on-linux
|
|
# RDNA4 (RX 9000): export ROCM_VERSION=7.2 (default 6.3 covers RDNA1-3).
|
|
|
|
services:
|
|
voicebox:
|
|
build:
|
|
context: .
|
|
args:
|
|
PYTORCH_VARIANT: rocm
|
|
ROCM_VERSION: ${ROCM_VERSION:-6.3}
|
|
|
|
devices:
|
|
- /dev/kfd
|
|
- /dev/dri
|
|
|
|
environment:
|
|
# HSA_OVERRIDE_GFX_VERSION forces the ROCm runtime to treat the GPU as a
|
|
# specific GFX version when auto-detection fails or the GPU is newer than
|
|
# the ROCm release. app.py sets 10.3.0 (RDNA2) by default; override here
|
|
# for your GPU family:
|
|
# RDNA4 / RX 9000 series: 12.0.0
|
|
# (requires ROCM_VERSION=7.2)
|
|
# RDNA3 / RX 7000 series / Strix Halo: 11.0.0
|
|
# RDNA2 / RX 6000 series: 10.3.0
|
|
# RDNA1 / RX 5000 series: 10.1.0
|
|
# Vega / GCN5: 9.0.0
|
|
- HSA_OVERRIDE_GFX_VERSION=${HSA_OVERRIDE_GFX_VERSION:-}
|
|
|
|
# Tune the ROCm memory allocator
|
|
- PYTORCH_HIP_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:512
|
|
|
|
# Redirect MIOpen kernel cache to a writable, persistent directory.
|
|
# Without this, MIOpen may fail to write its cache and throw
|
|
# miopenStatusUnknownError on fresh containers.
|
|
- MIOPEN_USER_DB_PATH=/app/data/cache/miopen_db
|
|
- MIOPEN_CUSTOM_CACHE_DIR=/app/data/cache/miopen_cache
|
|
|
|
# Use fast heuristics for kernel selection instead of exhaustive
|
|
# benchmarking. On RDNA4, exhaustive mode tries kernels that fail to
|
|
# allocate workspace memory (ptr: 0 size: 0), causing system stuttering
|
|
# on every generation even when the cache is present.
|
|
- MIOPEN_FIND_MODE=FAST
|