在模型运行时遇到报错“TVM/te/cce error.”
2023/06/06
75
问题信息
问题来源 | 产品大类 | 关键字 |
---|---|---|
官方 | 模型训练 | -- |
问题现象描述
- 报错文本
Traceback (most recent call last): File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/__init__.py", line 108, in <module> __import__('topi.cce') File "/root/archiconda3/envs/***/lib/python3.7/site-packages/topi/cce/__init__.py", line 20, in <module> import te.lang.cce File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/__init__.py", line 18, in <module> from .te_compute.broadcast_compute import broadcast File "/root/archiconda3/envs/***/Lib/python3.7/site-packages/te/lang/cce/te_compute/__init__.py", line 23, in <module> from .broadcast_compute import broadcast File "/root/archiconda3/envs/***/python3.7/site-packages/te/Lang/cce/te_compute/broadcast_compute.py", line 20, in <module> from .util import dtype_echeck_decorator File "/root/archiconda3/envs/***/lib/python3 .7/site-packages/te/lang/cce/te_compute/util.py", line 22, in <module> from te.platform import intrinsic_check_support File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/__init__.py", line 38, in <module> from .cce_buffer import cur cce_product_params File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_buffer.py", line 20, in <module> from te.platform import cce_conf File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_conf.py", line 25, in <module> from te.platform import cce_build File "/root/archiconda3/envs/***/lib/python3.7/site-packages/te/platform/cce_build.py", line 201, in <module> pipline_opt_level=2 …… tvm._ffi.base.TVMError: Traceback (most recent call last): …… TVMError: [EB0000] Buildconfig does hot contain field precise_bound_pipeline_opt level …… RuntimeError: execute_npu_operate: /usrl/workspace/PyTorch_Apex_Daily/CODE/aten/src/ATen/native/npu/utils/CalcuOpUtil .cpp:680 NPU error,NPU error code is:500001
原因分析
PyTorch内调用NPU类型算子时,强依赖于te、cce、TVM组件,PyTorch、CANN/nnae和te版本需要一致。在更新CANN/nnae后,te等组件不会自动更新,当版本不匹配时,则会出现该报错。
解决措施
更新te等组件版本,具体需要更新te-*.whl和topi-*.whl安装包。
在CANN安装目录或者nnae安装目录的lib64子目录下(以root安装用户为例:默认安装路径在/usr/local/Ascend/ascend-toolkit/latest/lib64目录下,更新安装包即可),找到te-*.whl和topi-*.whl安装包,使用pip3 install命令分别执行安装即可。
本页内容