在模型运行时遇到报错“TVM/te/cce error.”

报错信息

• 报错截图

  

• 报错文本

  Traceback (most recent call last):
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/__init__.py", line 108, in <module>
      __import__('topi.cce')
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/topi/cce/__init__.py", line 20, in <module>
      import te.lang.cce
    File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/__init__.py", line 18, in <module>
      from .te_compute.broadcast_compute import broadcast
    File "/root/archiconda3/envs/***/Lib/python3.7/site-packages/te/lang/cce/te_compute/__init__.py", line 23, in <module>
      from .broadcast_compute import broadcast
    File "/root/archiconda3/envs/***/python3.7/site-packages/te/Lang/cce/te_compute/broadcast_compute.py", line 20, in <module>
      from .util import dtype_echeck_decorator
    File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/te_compute/util.py", line 22, in <module>
      from te.platform import intrinsic_check_support
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/__init__.py", line 38, in <module>
      from .cce_buffer import cur cce_product_params
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_buffer.py", line 20, in <module>
      from te.platform import cce_conf
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_conf.py", line 25, in <module>
      from te.platform import cce_build
    File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_build.py", line 201, in <module>
      pipline_opt_level=2
    ……
  tvm._ffi.base.TVMError: Traceback (most recent call last):
    ……
  TVMError: [EB0000] Buildconfig does hot contain field precise_bound_pipeline_opt level
  ……
  RuntimeError: execute_npu_operate: /usrl/workspace/PyTorch_Apex_Daily/CODE/aten/src/ATen/native/npu/utils/CalcuOpUtil .cpp:680 NPU error,NPU error code is:500001

问题分析

PyTorch内调用NPU类型算子时，强依赖于te、cce、TVM组件，PyTorch、CANN/nnae和te版本需要一致。在更新CANN/nnae后，te等组件不会自动更新，当版本不匹配时，则会出现该报错。

处理方法

更新te等组件版本，具体需要更新te-*.whl和topi-*.whl安装包。

在CANN安装目录或者nnae安装目录的lib64子目录下（以root安装用户为例：默认安装路径在/usr/local/Ascend/ascend-toolkit/latest/lib64目录下，更新安装包即可），找到te-*.whl和topi-*.whl安装包，使用pip3 install命令分别执行安装即可。