PCIE Driver Installation Manual

This technical article provides you an overview of how to install Xilinx XDMA drivers and auto-run them on boot for your Linux system.

Table of Contents

  1. Document History
  2. Introduction
  3. Installing Drivers
  4. Customizing Drivers
  5. Modifying Scripts for Custom Drivers
  6. Verifying PCIE Connection

Document History

Date Version Author Description of Revisions
December 18, 2024 1.0 Soosung Kim PCIE Driver Installation and Auto-Run Manual

Introduction

The Xilinx XDMA driver enables high-speed data transfer between a Xilinx FPGA configured with Xilinx XDMA IP and a host PC via a PCI Express connection. However, the host PC’s PCIe interface does not immediately recognize newly connected or reprogrammed devices. After each reprogramming, the host PC must be rebooted, and the XDMA drivers must be loaded to facilitate data transmission. This guide explains how to configure the host PC to automatically load the XDMA drivers at startup.

Installing Drivers

The embeddedsw repository includes the following :

  • XDMA drivers customized to fit our Vivado Block Design

  • Scripts to install auto-boot services to your host PC

Instructions are as follows :

1) Clone the embeddedsw repository to your host PC, preferably running Ubuntu.

git clone https://github.com/nrvcorp/embeddedsw.git

2) Navigate to pcie_setup/pcie_install.

cd ./pcie_setup/pcie_install

3) Run pcie_install.sh, giving your sudo password as a parameter.

./pcie_install.sh --password <sudo_password>

4) Run the following code on your terminal to check if the service is installed properly :

sudo systemctl status reboot_run_root.service

5) Reboot the PC.

6) Open a terminal, and run the following to check if XDMA drivers are installed properly :

lsmod | grep xdma

The drivers xdma_dvs and xdma_zcu106 should show up.

Customizing Drivers

This section is optional, for those who want to install XDMA drivers corresponding to their custom block design.

Prerequisites :

  • Check the Vendor ID and Device ID of the XDMA IP on your custom Vivado Block Design.

    • Example : Vendor ID : 10EE, Device ID : 9034

    • If you want to connect multiple boards to the host PC, each board should be programmed with a different device ID or vendor ID.

  • Respective drivers should be created for each board.

    • In the below document, <XDMA_CUSTOM_DRIVER_1> and <XDMA_CUSTOM_DRIVER_2> denote the names you want to give your custom XDMA drivers.

1) Navigate to the pcie_setup directory.

cd ./pcie_setup

2) Copy the xdma_dvs subdirectory to create your custom driver.

sudo cp -r ./xdma_dvs ./<XDMA_CUSTOM_DRIVER_1>

It is recommended to let the driver name start with xdma.

3) Repeat for the number of required drivers.

4) Navigate to the driver folder.

cd ./<XDMA_CUSTOM_DRIVER_1>

5) Edit Makefile line 14 as follows :

Original Line

TARGET_MODULE:=xdma_dvs

Modified Line

TARGET_MODULE:=<XDMA_CUSTOM_DRIVER_1>

6) Edit xdma_cdev.h line 29 as follows :

Original Line

#define XDMA_NODE_NAME "xdma_dvs"

Modified Line

#define XDMA_NODE_NAME "<XDMA_CUSTOM_DRIVER_1>"

7) Edit xdma_mod.c line 33 as follows :

Original Line

#define DRV_MODULE_NAME  "xdma_dvs"

Modified Line

#define DRV_MODULE_NAME  "<XDMA_CUSTOM_DRIVER_1>"

8) Modify xdma_mod.c line 53 to match your Vendor ID and Device ID :

Original Line

 { PCI_DEVICE(0x10ee, 0xa884), },

Modified Line

 { PCI_DEVICE(<YOUR_VENDOR_ID>, <YOUR_DEVICE_ID>), },

9) Repeat for the number of required drivers.

Modifying Scripts for Custom Drivers

This section integrates your custom xdma drivers into the auto-boot service.

1) Navigate to pcie_setup/tests.

cd ./pcie_setup/tests

2) Copy load_driver_camera_npu_modified.sh to create your custom script.

sudo cp -r ./load_driver_camera_npu_modified.sh ./load_driver_custom.sh

3) Open load_driver_custom.sh, and modify it as follows :

  #!/bin/bash

  # Make sure only root can run our script
  if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root" 1>&2
    exit 1
  fi

  # Remove the existing xdma kernel module
  lsmod | grep xdma
  if [ $? -eq 0 ]; then
    rmmod xdma
    # TODO : list the xdma driver names you prefer
    rmmod <XDMA_CUSTOM_DRIVER_1>
    rmmod <XDMA_CUSTOM_DRIVER_2>
  fi

  # Use the following command to Load the driver in the default 
  # or interrupt drive mode. This will allow the driver to use 
  # interrupts to signal when DMA transfers are completed.

  # TODO : change the following to the driver names you prefer
  insmod ../<XDMA_CUSTOM_DRIVER_1>/<XDMA_CUSTOM_DRIVER_1>.ko
  insmod ../<XDMA_CUSTOM_DRIVER_2>/<XDMA_CUSTOM_DRIVER_2>.ko

  if [ ! $? == 0 ]; then
    echo "Error: Kernel module did not load properly."
    echo " FAILED"
    exit 1
  fi

  # TODO: repeat the following code block for each custom driver

  #################################################################
  echo -n "Loading <XDMA_CUSTOM_DRIVER_1> ..."
  echo ""
  cat /proc/devices | grep <XDMA_CUSTOM_DRIVER_1> > /dev/null
  returnVal=$?
  if [ $returnVal == 0 ]; then
    # Installed devices were recognized.
    echo "The Kernel module installed correctly and the xdma devices were recognized."
  else
    # No devices were installed.
    echo "Error: The Kernel module installed correctly, but no devices were recognized."
    echo " FAILED"
    exit 1
  fi
  #################################################################

  echo -n "Loading <XDMA_CUSTOM_DRIVER_2> ..."
  echo ""
  cat /proc/devices | grep <XDMA_CUSTOM_DRIVER_2> > /dev/null
  returnVal=$?
  if [ $returnVal == 0 ]; then
    # Installed devices were recognized.
    echo "The Kernel module installed correctly and the xdma devices were recognized."
  else
    # No devices were installed.
    echo "Error: The Kernel module installed correctly, but no devices were recognized."
    echo " FAILED"
    exit 1
  fi

  echo "====================================================================="
  echo " DONE"

4) Open pcie_setup/pcie_install/start_pcie.sh, and modify line 19 as follows :

Original Line

bash "$PARENT_DIR/tests/load_driver_camera_npu_modified.sh" || { echo "Failed to load driver"; exit 1; }

Modified Line

bash "$PARENT_DIR/tests/load_driver_custom.sh" || { echo "Failed to load driver"; exit 1; }

5) Lines 10 ~ 16 are required to clean up driver subdirectories before running them. For each required custom driver, insert the following code to replace them :

cd "$PARENT_DIR/<XDMA_CUSTOM_DRIVER_1>" || { echo "<XDMA_CUSTOM_DRIVER_1> directory not found"; exit 1; }
make clean
make install

6) Reboot the host PC.

Verifying PCIE Connection

After rebooting the host PC, verifying PCIE connection is essential.

Run ls /dev to check if your boards are recognized.

One driver name should correspond to one connected board each.

If not, follow the troubleshooting manual below.

Troubleshooting

Requirements :

  • Your Host PC.

  • Installation of auto-boot services, described in this manual.

  • Boards programmed beforehand, with known Vendor ID and device ID.

  • Make sure no powered off boards are installed in the PCIE slot.

1) After rebooting the host PC, run the following in the terminal.

lsmod | grep xdma

If you have customized the drivers, your custom drivers should show up.

If not, xdma_dvs and xdma_zcu106 should show up.

2) Run lspci to check if Xilinx Devices are recognized correctly.

The Device IDs of all your connected boards should show up.

3) If 2 goes wrong, turn the host PC completely off.

4) Program your boards by a JTAG connection with an external computer.

Note : this can be replaced with a SD card boot procedure.

In this case, a delay mechanism is recommended to delay the board firmware code from running until the host PC finishes booting.

firmware/DVS_SD/xmipi_example.c does this by waiting until manually pressing a button on the board chassis.

5) Turn the host PC back on.

6) Run the following :

ls /dev

See if you can find <xdma_driver_name>0_c2h_0, <xdma_driver_name>0_h2c_0,<xdma_driver_name>0_xvc, <xdma_driver_name>0_user, etc.