pytorch 1.13 and nccl

发表于 2023-05-03

本文字数： 4.1k 阅读时长 ≈ 4 分钟

windows 11
wsl2
- ubuntu 18.04
- nvidia driver 531.68
- cuda 11.6.2

pytorch 1.13.1 docker image

docker image

1	docker pull pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime

view nccl version of pytorch

1
2
3

docker run -ti --rm pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime bash

python -c "import torch;print(torch.cuda.nccl.version())"

pytorch 1.13.1

https://github.com/pytorch/pytorch/tree/v1.13.1

https://github.com/pytorch/pytorch/tree/v1.13.1#from-source

https://github.com/pytorch/pytorch/blob/v1.13.1/CONTRIBUTING.md#tips-and-debugging

https://zrss.github.io/archives/5a3d0ab7.html

conda create -n pytorch-dev python=3.8

conda activate pytorch-dev

conda install astunparse numpy ninja pyyaml setuptools cmake cffi typing_extensions future six requests dataclasses
conda install -c pytorch magma-cuda116
conda install mkl mkl-include

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}

CUDACXX=/usr/local/cuda/bin/nvcc MAX_JOBS=8 python setup.py develop

如果 setup 过程中出现如下日志

Building wheel torch-1.13.0a0+git49444c3
-- Building version 1.13.0a0+git49444c3
Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in /root/projects/pytorch/third_party/ios-cmake
Did you run 'git submodule update --init --recursive --jobs 0'?

可以重新 update submodule，再做尝试

git submodule deinit -f .
git clean -xdf
python setup.py clean
git submodule update --init --recursive --jobs 0

如果 setup 过程中出现如下日志，可以减小 jobs 数（例如上述的 case 为 8），再做尝试

1
2
3

FAILED: third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o
/usr/bin/c++ -DFBGEMM_STATIC -I/root/projects/pytorch/third_party/cpuinfo/include -I/root/projects/pytorch/third_party/fbgemm/third_party/asmjit/src -I/root/projects/pytorch/third_party/fbgemm/include -I/root/projects/pytorch/third_party/fbgemm -I/root/projects/pytorch/cmake/../third_party/benchmark/include -isystem /root/projects/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /root/projects/pytorch/cmake/../third_party/googletest/googletest/include -isystem /root/projects/pytorch/third_party/protobuf/src -isystem /root/tools/miniconda3/envs/pytorch-dev/include -isystem /root/projects/pytorch/third_party/gemmlowp -isystem /root/projects/pytorch/third_party/neon2sse -isystem /root/projects/pytorch/third_party/XNNPACK/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -Wall -Wextra -Werror -Wno-deprecated-declarations -O3 -DNDEBUG -fPIC -fvisibility=hidden -m64 -mavx2 -mf16c -mfma -std=c++14 -Wno-uninitialized -MD -MT third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o -MF third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o.d -o third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o -c /root/projects/pytorch/third_party/fbgemm/src/FbgemmI8DepthwiseAvx2.cc
c++: internal compiler error: Killed (program cc1plus)