Failed to build tokenizers ERROR: Could not build wheels for tokenizers, which is required to instal

1.在google colab 安装transformer时遇见错误tokenizers无法编译。
python版本为3.10； transformers版本为4.15；tokenizers需要<0.11, >=0.10.0 ； 直接安装tokenizers是不可行的。
错误展示：

Building wheels for collected packages: tokenizers
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
1
2
3
4
5
6
7
8
9
10
11
12

2.安装rust，命令如下：

!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
1

遇见错误如下：

info: downloading installer
sh: 172: cannot open /dev/tty: No such device or address
1
2

免费版本是不存在终端的，所以将命令修改为：

!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
1

安装成功：

info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.72.0 (5680fa18f 2023-08-23)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"
1
2
3
4
5
6
7
8
9
10
11
12
13
14

运行命令 source “$HOME/.cargo/env”

source "$HOME/.cargo/env"
colab 里面是：source "/root/.cargo/env"
1
2

继续安装tranformer =4.15.0 仍然没有用遇见错误如上。
此时选择重启代码程序框。发现错误也一样，也无法解决。
此时只剩下最后一条路了，降低python版本为3.8.

改变colab中的python版本为3.8，参考链接为添加链接描述
1）安装 miniconda 改变 colab python 版本

%%bash
MINICONDA_INSTALLER_SCRIPT=Miniconda3-4.5.4-Linux-x86_64.sh
MINICONDA_PREFIX=/usr/local
wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT
chmod +x $MINICONDA_INSTALLER_SCRIPT
./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX

conda install --channel defaults conda python=3.8 --yes
conda update --channel defaults --all --yes
1
2
3
4
5
6
7
8
9

2）python 版本改变后的新安装包路径加入系统路径

import sys
sys.path.append("/usr/local/lib/python3.8/site-packages")
1
2

3. 查看python版本

!python -V
python 3.8.17
1
2

继续安装环境，安装成功！

Successfully built horovod sacremoses
Installing collected packages: wcwidth, tokenizers, safetensors, pytz, typing-extensions, tqdm, threadpoolctl, tabulate, six, setuptools, regex, pyyaml, psutil, Pillow, numpy, networkx, joblib, ftfy, cloudpickle, click, torch, tifffile, scipy, sacremoses, PyWavelets, python-dateutil, opencv_python, imageio, horovod, filelock, torchvision, scikit_learn, scikit_image, pandas, huggingface_hub, transformers, timm
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.65.0
    Uninstalling tqdm-4.65.0:
      Successfully uninstalled tqdm-4.65.0
  Attempting uninstall: setuptools
    Found existing installation: setuptools 68.0.0
    Uninstalling setuptools-68.0.0:
      Successfully uninstalled setuptools-68.0.0
Successfully installed Pillow-9.2.0 PyWavelets-1.4.1 click-8.1.7 cloudpickle-2.2.1 filelock-3.12.3 ftfy-6.1.1 horovod-0.28.1 huggingface_hub-0.13.4 imageio-2.31.3 joblib-1.3.2 networkx-3.1 numpy-1.21.6 opencv_python-4.6.0.66 pandas-1.3.5 psutil-5.9.5 python-dateutil-2.8.2 pytz-2023.3.post1 pyyaml-6.0.1 regex-2022.10.31 sacremoses-0.0.53 safetensors-0.3.3 scikit_image-0.19.3 scikit_learn-1.0.2 scipy-1.10.1 setuptools-63.4.1 six-1.16.0 tabulate-0.9.0 threadpoolctl-3.2.0 tifffile-2023.7.10 timm-0.8.15.dev0 tokenizers-0.10.3 torch-1.12.1 torchvision-0.13.1 tqdm-4.64.1 transformers-4.15.0 typing-extensions-4.7.1 wcwidth-0.2.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: The following packages were previously imported in this runtime:
  [PIL,_distutils_hack,dateutil,numpy,pkg_resources,psutil,setuptools,six,wcwidth]
You must restart the runtime in order to use newly installed versions.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

此外，如果你安装transformer 4.15之类的碰到安装不了，方便的情况下降低python为3.8版本，这是最好的选择，网上有人试了3.10，和3.6都会报编译错误；其次的话安装4.16的版本。如果要是遇见冲突再处理。