AXCL 主控环境搭建
这一节主要是描述各个操作系统以及发行版如何开启CMA内存管理,下面列举已经支持的平台。
| 平台 | arch | OS | 内核 | 验证过程 |
|---|---|---|---|---|
| intel | x64 | centos9 | 5.14.0 | |
| intel | x64 | Ubuntu24.04 | 6.8.0 | |
| intel | x64 | Kylin 10 | 5.14.0 | |
| intel | x64 | OpenEuler 24.03 | 6.6.0 | |
| AX650 | arm64 | Linux | 5.15.73 | |
| Raspberry Pi5 | arm64 | Raspbian | 6.6.0 | |
| rk3568 | arm64 | ubuntu 20.4 | 5.10.160 | |
| rk3588 | arm64 | ubuntu 20.4 | 5.10.160 |
CentOS 9
系统信息
[test@centos ~]$ uname -a
Linux centos 5.14.0-527.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 6 13:28:51 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
[test@centos ~]$ cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[test@centos ~]$
环境搭建
更新软件包:
sudo yum update安装内核源码:
sudo yum install -y kernel-devel kernel-headers修改
grub文件添加reserved cma size,转码卡建议设置为 256MB[test@centos ~]$ cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/cs_192-swap rd.lvm.lv=cs_192/root rd.lvm.lv=cs_192/swap rhgb quiet cma=256M" GRUB_DISABLE_RECOVERY="true" GRUB_ENABLE_BLSCFG=true [test@centos ~]$
更新 grub:
sudo su grub2-mkconfig -o /boot/grub2/grub.cfg grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) cma=256M" grubby --update-kernel=ALL --args="cma=256M"
关闭 SELinux:
[test@centos ~]$ cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. # See also: # https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-selinux/#getting-started-with-selinux-selinux-states-and-modes # # NOTE: In earlier Fedora kernel builds, SELINUX=disabled would also # fully disable SELinux during boot. If you need a system with SELinux # fully disabled instead of SELinux running with no policy loaded, you # need to pass selinux=0 to the kernel command line. You can use grubby # to persistently set the bootloader to boot with selinux=0: # # grubby --update-kernel ALL --args selinux=0 # # To revert back to SELinux enabled: # # grubby --update-kernel ALL --remove-args selinux # SELINUX=disabled # SELINUXTYPE= can take one of these three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [test@centos ~]$
安装相关软件包:
sudo yum install -y patch sudo yum install -y rpm-build sudo yum install -y gcc g++ make
重启,执行
reboot或sudo rebootdmesg | grep cma查看 CMA reserved 是否设置成功:[test@centos ~]$ dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-527.el9.x86_64 root=/dev/mapper/cs_192-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/cs_192-swap rd.lvm.lv=cs_192/root rd.lvm.lv=cs_192/swap rhgb quiet cma=256M [ 0.005347] cma: Reserved 256 MiB at 0x0000000100000000 on node -1 [ 0.014230] Kernel command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-527.el9.x86_64 root=/dev/mapper/cs_192-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/cs_192-swap rd.lvm.lv=cs_192/root rd.lvm.lv=cs_192/swap rhgb quiet cma=256M [ 0.050869] Memory: 1085392K/33116588K available (16384K kernel code, 5720K rwdata, 13156K rodata, 4016K init, 5528K bss, 1080032K reserved, 262144K cma-reserved) [ 0.198289] cma: Initial CMA usage detected [test@centos ~]$
Ubuntu 22.04
系统信息
test@ubuntu:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
test@ubuntu:~$
环境搭建
Ubuntu 22.04 默认没有打开
CONFIG_CMA=y和CONFIG_DMA_CMA=y,所以需要重建内核。使用
cat /boot/config-$(uname -r) | grep CMA可以查看当前内核关于CMA的配置。
安装软件包:
sudo apt install gcc g++ make libncurses-dev flex bison libelf-dev libssl-dev获取内核源代码:
sudo apt install linux-source-6.5.0解压源码包:
cd /usr/src/; tar jxvf linux-source-6.5.0.tar.bz2进入内核源码目录:
cd /usr/src/linux-source-6.5.0/为方便操作切换为root用户:
sudo su清理旧文件:
make distclean复制内核配置文件:
cp /boot/config-$(uname -r) /usr/src/linux-source-6.5.0/.config配置内核
scripts/config --enable CMA scripts/config --enable DMA_CMA scripts/config --disable SYSTEM_TRUSTED_KEYS scripts/config --disable SYSTEM_REVOCATION_KEYS scripts/config --disable DEBUG_INFO_BTF make olddefconfig # 更新配置
构建内核 (编译时间较长):
make -j$(nproc) make modules_install -j$(nproc) make headers_install -j$(nproc) make install
配置CMA大小:
vim /etc/default/grub,添加GRUB_CMDLINE_LINUX="cma=256MB"root@ubuntu:~# cat /etc/default/grub # If you change this file, run 'update-grub' afterwards to update # /boot/grub/grub.cfg. # For full documentation of the options in this file, see: # info -f grub -n 'Simple configuration' GRUB_DEFAULT=0 GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=0 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" GRUB_CMDLINE_LINUX="cma=256MB" # Uncomment to enable BadRAM filtering, modify to suit your needs # This works with Linux (no patch required) and with any kernel that obtains # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" # Uncomment to disable graphical terminal (grub-pc only) #GRUB_TERMINAL=console # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo' #GRUB_GFXMODE=640x480 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux #GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries #GRUB_DISABLE_RECOVERY="true" # Uncomment to get a beep at grub start #GRUB_INIT_TUNE="480 440 1" root@ubuntu:~#
更新grub:
update-grub重启设备:
reboot查看CMA reserved的大小:
sudo dmesg | grep cmatest@ubuntu:~$ sudo dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.13 root=UUID=ed2bd69d-f89a-4952-baef-aee37d6b02e2 ro cma=256MB quiet splash vt.handoff=7 [ 0.005136] cma: Reserved 256 MiB at 0x0000000100000000 [ 0.087585] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.5.13 root=UUID=ed2bd69d-f89a-4952-baef-aee37d6b02e2 ro cma=256MB quiet splash vt.handoff=7 [ 0.198353] Memory: 31551428K/33116588K available (20480K kernel code, 4267K rwdata, 7276K rodata, 4772K init, 17416K bss, 1302756K reserved, 262144K cma-reserved) test@ubuntu:~$
OpenEuler 24.03
系统信息
[test@openeuler ~]$ cat /etc/os-release
NAME="openEuler"
VERSION="24.03 (LTS)"
ID="openEuler"
VERSION_ID="24.03"
PRETTY_NAME="openEuler 24.03 (LTS)"
ANSI_COLOR="0;31"
[test@openeuler ~]$
环境搭建
OpenEuler 24.03 默认没有打开
CONFIG_CMA=y和CONFIG_DMA_CMA=y,所以需要重建内核。使用
cat /boot/config-$(uname -r) | grep CMA可以查看当前内核关于CMA的配置。
安装软件包:
sudo dnf install -y rpm-build openssl-devel bc rsync gcc gcc-c++ flex bison m4 elfutils-libelf-devel dwarves下载内核源码:
sudo dnf install -y kernel-source进入内核源码目录:
cd /usr/src/linux-6.6.0-61.0.0.60.oe2403.x86_64/为方便操作切换为root用户:
sudo su加载内核配置:
make openeuler_defconfig配置内核:
scripts/config --enable CMA scripts/config --enable DMA_CMA scripts/config --disable SYSTEM_TRUSTED_KEYS scripts/config --disable SYSTEM_REVOCATION_KEYS scripts/config --disable DEBUG_INFO_BTF make olddefconfig # 更新配置
构建内核 (编译时间较长)
make -j$(nproc) make modules_install -j$(nproc) make headers_install -j$(nproc) make install
配置CMA大小:
vim /etc/default/grub,在GRUB_CMDLINE_LINUX添加"cma=256MB"[test@openeuler ~]$ cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="resume=UUID=8fc8e105-43fb-47b2-9815-3bebcc42ba3b rd.lvm.lv=openeuler/root rd.lvm.lv=openeuler/swap cgroup_disable=files apparmor=0 crashkernel=512M cma=256M" GRUB_DISABLE_RECOVERY="true" [test@openeuler ~]$
更新 grub
grub2-mkconfig -o /boot/grub2/grub.cfg grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) cma=256M" grubby --update-kernel=ALL --args="cma=256M"
关闭
SELinux,设置SELINUX=disabled[test@openeuler ~]$ cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [test@openeuler ~]$
重启,执行
reboot或sudo rebootdmesg | grep cma查看 CMA reserved 是否设置成功:[test@openeuler ~]$ sudo dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.6.0 root=UUID=bc2e1f2c-87c1-415d-ba78-67131985fa31 resume=UUID=8fc8e105-43fb-47b2-9815-3bebcc42ba3b rd.lvm.lv=openeuler/root rd.lvm.lv=openeuler/swap cgroup_disable=files apparmor=0 crashkernel=512M cma=256M [ 0.004731] cma: Reserved 256 MiB at 0x0000000100000000 on node -1 [ 0.014041] Kernel command line: BOOT_IMAGE=/vmlinuz-6.6.0 root=UUID=bc2e1f2c-87c1-415d-ba78-67131985fa31 resume=UUID=8fc8e105-43fb-47b2-9815-3bebcc42ba3b rd.lvm.lv=openeuler/root rd.lvm.lv=openeuler/swap cgroup_disable=files apparmor=0 crashkernel=512M cma=256M [ 0.050907] Memory: 848232K/33116588K available (16384K kernel code, 5200K rwdata, 5340K rodata, 3440K init, 4848K bss, 1304332K reserved, 262144K cma-reserved) [test@openeuler ~]$
Kylin 10
系统信息
系统版本:V10 SP1 2403 HWE
[axera@localhost ~]$ uname -a
Linux localhost.localdomain 5.14.0-148.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 19 13:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[axera@localhost ~]$ uname -r
5.14.0-148.el9.x86_64
[axera@localhost ~]$ cat /etc/os-release
NAME="Kylin"
VERSION="银河麒麟桌面操作系统V10 (SP1)"
VERSION_US="Kylin Linux Desktop V10 (SP1)"
ID=kylin
ID_LIKE=debian
PRETTY_NAME="Kylin V10 SP1"
VERSION_ID="v10"
HOME_URL="http://www.kylinos.cn/"
SUPPORT_URL="http://www.kylinos.cn/support/technology.html"
BUG_REPORT_URL="http://www.kylinos.cn/"
PRIVACY_POLICY_URL="http://www.kylinos.cn"
VERSION_CODENAME=kylin
UBUNTU_CODENAME=kylin
PROJECT_CODENAME=V10SP1
KYLIN_RELEASE_ID="2403"
环境搭建
查看CMA和DMA是否打开? 即
CONFIG_CMA=y和CONFIG_DMA_CMA=y[axera@localhost ~]$ cat /boot/config-5.10.0-9-generic | grep CMA CONFIG_CMA=y # CONFIG_CMA_DEBUG is not set # CONFIG_CMA_DEBUGFS is not set CONFIG_CMA_AREAS=7 CONFIG_NETWORK_SECMARK=y CONFIG_NF_CONNTRACK_SECMARK=y CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m CONFIG_NETFILTER_XT_TARGET_SECMARK=m CONFIG_B43_BCMA=y CONFIG_B43_BUSES_BCMA_AND_SSB=y # CONFIG_B43_BUSES_BCMA is not set CONFIG_B43_BCMA_PIO=y CONFIG_INPUT_CMA3000=m CONFIG_INPUT_CMA3000_I2C=m CONFIG_BCMA_POSSIBLE=y CONFIG_BCMA=m CONFIG_BCMA_BLOCKIO=y CONFIG_BCMA_HOST_PCI_POSSIBLE=y CONFIG_BCMA_HOST_PCI=y CONFIG_BCMA_HOST_SOC=y CONFIG_BCMA_DRIVER_PCI=y CONFIG_BCMA_SFLASH=y CONFIG_BCMA_DRIVER_GMAC_CMN=y CONFIG_BCMA_DRIVER_GPIO=y # CONFIG_BCMA_DEBUG is not set CONFIG_DRM_GEM_CMA_HELPER=y CONFIG_DRM_KMS_CMA_HELPER=y CONFIG_USB_HCD_BCMA=m CONFIG_COMEDI_PCMAD=m # CONFIG_ION_CMA_HEAP is not set CONFIG_CRYPTO_CMAC=m CONFIG_DMA_CMA=y # CONFIG_DMA_PERNUMA_CMA is not set CONFIG_CMA_SIZE_MBYTES=0 CONFIG_CMA_SIZE_SEL_MBYTES=y # CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set # CONFIG_CMA_SIZE_SEL_MIN is not set # CONFIG_CMA_SIZE_SEL_MAX is not set CONFIG_CMA_ALIGNMENT=8
如果kernel config没有打开CMA配置,参考 stackoverflow 帖子配置。
dmesg | grep cma查看CMA reserved的大小
[axera@localhost ~]$ dmesg | grep cma
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.10.0-9-generic root=UUID=c7a8075e-85dc-4a73-84e8-9b5bc3029204 ro cma=128MB quiet splash loglevel=0 resume=UUID=d27e8de7-dc17-47ab-92c6-5e7e34bca463 security=kysec
[ 0.005838] cma: Reserved 128 MiB at 0x0000000487800000
[ 0.026542] Kernel command line: BOOT_IMAGE=/vmlinuz-5.10.0-9-generic root=UUID=c7a8075e-85dc-4a73-84e8-9b5bc3029204 ro cma=128MB quiet splash loglevel=0 resume=UUID=d27e8de7-dc17-47ab-92c6-5e7e34bca463 security=kysec
[ 0.052526] Memory: 15721272K/16458476K available (14345K kernel code, 6114K rwdata, 7480K rodata, 3828K init, 20636K bss, 605872K reserved, 131072K cma-reserved)
如果CMA大小没有配置成合适size(推荐不小于128MB),按照如下步骤配置:
sudo vim /etc/default/grub,添加GRUB_CMDLINE_LINUX="cma=128MB", size 根据自己业务和实际内存情况设置。[axera@localhost ~]$ sudo vim /etc/default/grub 输入密码 # If you change this file, run 'update-grub' afterwards to update # /boot/grub/grub.cfg. # For full documentation of the options in this file, see: # info -f grub -n 'Simple configuration' GRUB_DEFAULT=0 GRUB_TIMEOUT=1 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_DISTRIBUTOR_RELEASE=`lsb_release -d -s | awk -F" " '{print $2 " " $3}' 2> /dev/null || echo ""` GRUB_CMDLINE_LINUX_DEFAULT="quiet splash loglevel=0" GRUB_CMDLINE_LINUX="cma=128MB" GRUB_CMDLINE_LINUX_SECURITY="security=kysec" # Uncomment to enable BadRAM filtering, modify to suit your needs # This works with Linux (no patch required) and with any kernel that obtains # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" # Uncomment to disable graphical terminal (grub-pc only) #GRUB_TERMINAL=console # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo' #GRUB_GFXMODE=640x480 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux #GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries #GRUB_DISABLE_RECOVERY="true" # Uncomment to get a beep at grub start #GRUB_INIT_TUNE="480 440 1"
sudo update-grub更新grubsudo reboot
Raspberry Pi 5
在树莓派 5 上安装 AX650 加速卡时,首先需要准备一块 M.2 HAT+ 扩展板。参考官方链接,M.2 HAT+ 的官方版本只支持 2230、2242 的 M.2 M Key 卡,通常 AX650 加速卡是 2280 的,您可能需要考虑购买第三方的支持 2280 长度的 M.2 HAT+ 扩展板。
:::{Warning} 根据树莓派硬件批次不同,可能需要更新一下树莓派的 EEPROM 设置。具体步骤如下: :::
如同 PC 中的 BIOS,EEPROM 设置独立于烧录 OS 的 TF 卡,烧录最新的树莓派镜像或者切换镜像版本并不会主动更新 EEPROM 的设置。首先执行 update:
sudo apt update && sudo apt full-upgrade
然后检查一下 EEPROM 中的版本:
sudo rpi-eeprom-update
如果看到的日期早于 2023 年 12 月 6 日,运行以下命令以打开 Raspberry Pi 配置 CLI:
sudo raspi-config
在 Advanced Options > Bootloader Version (引导加载程序版本) 下,选择 Latest (最新)。然后,使用 Finish 或 ESC 键退出 raspi-config。
执行以下命令,将固件更新到最新版本。
sudo rpi-eeprom-update -a
最后使用 sudo reboot 重新启动。重启后就完成了 EEPROM 中 firmware 的更新。
:::{Warning} 取决于使用的树莓派 kernel 状态,目前的修改是以 2024年11月18日 以前的树莓派刚烧录好的系统为例进行说明的,客户需要根据树莓派系统更新情况识别这个步骤是否必须。 :::
在当前的树莓派 kernel 和 M.2 HAT+ 组合中,可能会遇到如下限制:
PCIE Device 无法识别
PCIE MSI IRQ 无法申请多个
这些问题将导致安装失败或者子卡起不来。需要检查 Raspberry Pi 5 /boot/firmware/config.txt 文件,并进行修改。
如果是第三方的兼容 M.2 HAT+ 产品,需要注意供电问题;在 config.txt 中添加如下描述:
dtparam=pciex1
该描述可以默认打开 PCIE 功能;然后继续增加 PCIE 的设备描述:
dtoverlay=pciex1-compat-pi5,no-mip
完成修改并重启后,可以使用 lspci 命令检查加速卡是否正确被识别:
axera@raspberrypi:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0000:01:00.0 Multimedia video controller: Axera Semiconductor Co., Ltd Device 0650 (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge
其中 Multimedia video controller: Axera Semiconductor Co., Ltd Device 0650 (rev 01) 就是 AX650 加速卡。
rk3588,rk3568等其余自定义BSP的主控
rk3588定制化能力比较高,存在多个发行版和内核版本,但是主控与M.2算力卡交互需要内核开启CMA配置项进行内存管理,因此需要客户有自己修改内核配置的能力,主要流程为:
准备内核源码和交叉编译工具链
配置内核选项,开启CMA选项
编译内核和模块
安装内核到开发板,确保正常启动
dmesg验证配置生效