This article guide you install Text Generation WebUI on local computer.

oobabooga text-generation-webui github repository

Windows 10

Environment:

  • Virualization: kvm/qemu
  • CPU: Intel(R) Xeon(R) E5-2696 v4 @ 2.20GHz
  • Memory: 32GiB
  • GPU: RTX 3060 12GiB
  • HDD: Virtio 500GiB
  1. Install Nvidia GPU Driver

when I wrote this blog, I choose stable release, it’s “531.68-desktop-win10-win11-64bit-international-dch-whql.exe”

  1. Install CUDA

when I wrote this blog, I choose cuda 11.7, it’s “cuda_11.7.0_516.01_windows.exe”, about 2.5 GiB size.

When the setup starts, select a location to extract the installer. Once that is done the CUDA installer will start. Over there, choose Express installation and click on Next. This will install the CUDA Toolkit on your system in the location C:\Program Files\NVIDIA GPU Computing Toolkit.

Configure Settings Variables:

CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7
CUDA_PATH_V11_7=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7
  1. Install CUDNN

when I wrote this blog, I choose cunn 8.9, it’s “cudnn-windows-x86_64-8.9.2.26_cuda11-archive.zip”, about 700 MiB size.

after download, extract it, and copy all content to cuda root folder, in my case, it’s “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7”.

and then configure System Variables:

CUDNN=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7;
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin;
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\lib\x64;
      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include

also add above CUDNN variable content to System PATH variable.

  1. Install PyTorch cuda version
pip install torch==2.0.1+cu117  torchaudio==2.0.2+cu117  torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117