Paper: (https://arxiv.org/pdf/2310.07602.pdf)

[2023.1.15] We are very sorry for the damage to some files in the published data, and the partitioning issue of imagesets has been corrected. And you can replace these images(006702.png, 006708.png, 006710.png, 006713.png, 006717.png) in the original dataset.

[2023.10.29] We have released the dataset download link.

[2023.12.10] Our Code currently supports VFF and M2Fusion.

[2023.10.27] Our Code currently supports some baselines including Voxel RCNN, Second, Pointpillars, RDIOU. Other baselines will be updated soon.

[2023.10.15] Our code and data are still being maintained and will be released soon.

Dual-Radar is a new dataset based on 4D radar that can be used for studies on 3D object detection and tracking in the field of autonomous driving. The perception system of ego vehicle includes a high-resolution camera, a 80-line LiDAR and two up-to-date and different models of 4D radars operating in different modes(Arbe and ARS548). The dataset comprises of raw data collected from ego vehicle, including scenarios such as urban and tunnels, with weather conditions of rainy, cloudy, sunny and so on. Our dataset also includes data from different time periods, including dusk, nighttime, and daytime. Our collected raw data amounts to a total of 12.5 hours, encompassing a total distance of over 600 kilometers. Our dataset covers a route distance of approximately 50 kilometers. It consists of 151 continuous time sequences, with the majority being 20-second sequences, resulting in a total of 10,007 carefully time-synchronized frames.

a) Ego vehicle's work scenario

b) Data projection visualization

Figure 1. Data collection vehicle and data projection visualization

Our ego vehicle’s configuration and the coordinate relationships between multiple sensors are shown in Figure. 2. The platform of our ego vehicle system consists of a high-resolution camera, an new 80-line LiDAR, and two types of 4D radar. All sensors have been carefully calibrated. The camera and LiDAR are mounted directly above the ego vehicle, while the 4D radars are installed in front of it. Due to the range of horizontal view limitations of the camera and 4D radars, we only collect data from the front of our ego vehicle for annotation. The ARS548 RDI captures data within approximately 120° horizontal field of view and 28° vertical field of view in front of the ego vehicle, while the Arbe Phoenix, operating in middle-range mode, collects data within a 100° horizontal field of view and 30° vertical field of view. The LiDAR collects around the ego vehicle in a 360° manner but only retains the data in the approximate 120° field of view in front of it for annotation.

Figure 2. Sensor Configuration and Coordinate Systems

• The specification of the autonomous vehicle system platform. Our proposed dataset is collected from a high-resolution camera, an 80-line mechanical LiDAR, and two types of 4D radars, the Arbe Phoenix and the ARS548 RDI radar. Our dataset provides GPS information for timing implementation. The sensor configurations are shown in Table 1.

• The statistics of number of points cloud per frame. In addition,we analyze the distribution density of the point clouds and the number of point clouds per frame, as shown in Table 2.

• The visualization of raw data collected under different weather conditions and the visualization of annotated 3D bounding boxes are performed separately. The three types of data(Lidar, Arbe and Ars548) are transformed onto a unified coordinate system.

a) Sunny,Daytime	b) Sunny,Nightime
c) Rainy,Daytime	d) Cloudy,Daytime

Figure 3. Visualization of raw data sequences under different weather conditions. On the left is a color RGB image, while on the right side, the cyan represents the Arbe point cloud, the white represents the LiDAR point cloud, and the yellow represents the ARS548 point cloud.

Figure 4. Visualization of 3D bounding box projection on data. The first column represents the 3D frame markers on the image. The Column 2, 3, and 4 represent the point cloud from Lidar, Arbe Phoenix radar point cloud, and ARS548 RDI radar point cloud, respectively. Each row represents a scenario type.(a): downtown daytime normal light. (b): downtown daytime backlight. (c): downtown dusk normal light. (d): downtown dusk backlight. (e): downtown clear night. (f): downtown daytime cloudy. (g): downtown rainy day. (h): downtown cloudy dusk. (i): downtown cloudy night. (j): downtown rainy night. (k): daytime tunnel. (l): nighttime tunnel.

• Our dataset is freely available to researchers. Please download and sign our agreement and send it to the provided email address (lwang_hit@hotmail.com). You will receive the download link within one week.

• When unzipping the data, please file and organize it by following the format below:

    └─Dual Radar
    ├─ImageSets.zip
    ├─testing
    │  ├─testing_arbe.zip
    │  ├─testing_ars548.zip
    │  ├─testing_calib.zip
    │  ├─testing_image.zip
    │  ├─testing_label.zip
    │  ├─testing_robosense.zip
    ├─training
    │  ├─training_arbe.zip
    │  ├─training_ars548.zip
    │  ├─training_calib.zip
    │  ├─training_image.zip
    │  ├─training_label.zip
    │  ├─training_robosense.zip
    └─README_dual_radar.txt

• This folder contains 10007 frames of labeled pointclouds and image data. The structure of the folder is shown as blow:

    └─Dual Radar
    ├─ImageSets
    │      test.txt
    │      train.txt
    │      trainval.txt
    │      val.txt
    ├─testing
    │  ├─arbe
    │  │      000000.bin	# Raw pointclouds (removed None) of the Arbe.
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin	# Raw pointclouds (removed None) of the ARS548.
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png	# Undistort images of the camera.
    │  │      ...............
    │  ├─label
    │  │      000000.txt	# Label in txt format, explain later.
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin	# Raw pointclouds (removed None) of the LiDAR.
    │  │      ...............
    ├─training
    │  ├─arbe
    │  │      000000.bin
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png
    │  │      ...............
    │  ├─label
    │  │      000000.txt
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin
    │  │      ...............
    └─README.txt

• The calib.txt contains tree parts. The dataset consists of two parts: the data part and the alignment calibration file. The data part is image data in png format and point cloud data in bin format. The alignment calibration file includes calibration parameters for the four sensors. The camera-LiDAR, camera-4D radar joint calibration are shown here as examples for illustration.

   Dual Radar_cam.Intrinsics.RadialDistortion: Barrel distortion of Dual Radar_cam [ k1, k2, k3 ]
   Dual Radar_cam.Intrinsics.TangentialDistortion: radial distortion of Dual Radar_cam [ p1, p2 ]
   Dual Radar_cam.IntrinsicMatrix: Dual Radar_cam's intrinsic matrix [ af, 0, 0; 0, bf, 0; u, v, 1]
   Dual Radar_LiDAR-->Dual Radar_cam: Dual Radar_lidar to Dual Radar cam's single response matrix P(4×4)
   Dual Radar_radar--> Dual Radar_cam: Dual Radar_radar to Dual Radar_cam rotation matrix + translation matrix P(3×4)

• All values (numerical or strings) are separated via spaces, each row corresponds to one object. The 19 columns represent:

  Value       Name             Description
  -------------------------------------------------------------------------------------------------------
  1        type               Describes the type of object: 'Car', 'Van', 'Truck', 'Pedestrian',  'Person_sitting', 'Cyclist', 'Tram', 'Misc' or 'DonCare'
  1        truncated          Float from 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries
  1        occluded           Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly ccluded, 2 = largely occluded, 3 = unknown
  1        alpha              Observation angle of object, ranging [-pi..pi]
  4        bbox               2D bounding box of object in the image (0-based index): contains left, top, right, bottom pixel coordinates. 
  3        dimensions         3D object dimensions: height, width, length (in meters).
  3        location           3D object location x,y,z in camera coordinates (in meters).
  1        rotation_y         Rotation ry around Y-axis in camera coordinates [-pi..pi].
  1        score              Only for results: Float,indicating confidence in detection, needed for p/r curves , higher is better.
  1        track_id           Path tracking of the same object

• Since the labeling work is done in label coordinate, the bounding box out of the image FOV(1920×1080) needs to be cut.

• Location mean the xyz in label coordinate. The same coordinate origen and the relation of axis is shown below.

Figure 5. Illustration of sensor coordinate systems

• The difference between rotation_y and alpha are, that rotation_y is directly given in camera coordinates, while alpha also considers the vector from the camera center to the object center, to compute the relative orientation of the object with respect to the camera. For example, a car which is facing along the X-axis of the camera coordinate system corresponds to rotation_y=0, no matter where it is located in the X/Z plane (bird's eye view), while alpha is zero only, when this object is located along the Z-axis of the camera. When moving the car away from the Z-axis, the observation angle will change.

Figure 6. Distribution of weather conditions.

• We separately count the number of instances for each category in the Dual-Radar dataset and the distribution of different types of weather. About two-thirds of our data are collected under normal weather conditions, and about one-third are collected under rainy and cloudy conditions. We collect 577 frames in rainy weather, which is about 5.5% of the total dataset. The rainy weather data we collect can be used to test the performance of different 4D radars in adverse weather conditions.

Figure 7. Distribution of instance conditions.

• We also conduct a statistical analysis of the number of objects with each label at different distance ranges from our vehicle, as shown in Figure 7. Most objects are within 60 meters of our ego vehicle.

This is the documentation for how to use our detection frameworks with Dual-Radar dataset. We test the Dual-Radar detection frameworks on the following environment:

• Python 3.8.16 (3.10+ does not support open3d.)
• Ubuntu 18.04/20.04
• Torch 1.10.1+cu113
• CUDA 11.3
• opencv 4.2.0.32

• After all files are downloaded, please arrange the workspace directory with the following structure:

• Organize your code structure as follows

    Frameworks
      ├── checkpoints
      ├── data
      ├── docs
      ├── pcdet
      ├── output

• Organize the dataset according to the following file structure

    dual_radar
      ├── lidar
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── velodyne
        ├── testing
            ├── calib
            ├── image
            ├── velodyne
      ├── radar_arbe
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── arbe
        ├── testing
            ├── calib
            ├── image
            ├── arbe
      ├── radar_ars548
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── ars548
        ├── testing
            ├── calib
            ├── image
            ├── ars548

• Clone the repository

 git clone https://github.com/adept-thu/Dual-Radar.git
 cd Dual-Radar

• Create a conda environment

conda create -n Dual-Radardet python=3.8.16
conda activate Dual-Radardet

• Install PyTorch (We recommend pytorch 1.10.1.)

• Install the dependencies

pip install -r requirements.txt

• Install Spconv（our cuda version is 113）

pip install spconv-cu113

• Build packages for Dual-Radardet

python setup.py develop

• Generate the data infos by running the following command:

using lidar data
python -m pcdet.datasets.dual_radar.dual_radar_dataset create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset.yaml

using arbe data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_arbe create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_arbe.yaml

using ars548 data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_ars548 create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_ars548.yaml

• To train the model on single GPU, prepare the total dataset and run

python train.py --cfg_file ${CONFIG_FILE}

• To train the model on multi-GPUs, prepare the total dataset and run

sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}

• To evaluate the model on single GPU, modify the path and run

python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}

• To evaluate the model on multi-GPUs, modify the path and run

sh scripts/dist_test.sh ${NUM_GPUS} \
    --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}

Here we provide a quick demo to test a pretrained model on the custom point cloud data and visualize the predicted results

• Download the pretrained model as shown in Table 4~8.
• Make sure you have installed the Open3d and mayavi visualization tools. If not, you could install it as follow:

pip install open3d
pip install mayavi

• prepare your point cloud data

points[:, 3] = 0 
np.save(`my_data.npy`, points)

• Run the demo with a pretrained model and point cloud data as follows

python demo.py --cfg_file ${CONFIG_FILE} \
    --ckpt ${CKPT} \
    --data_path ${POINT_CLOUD_DATA}

• Thanks for the sensor support provided by Beijing Jingwei Hirain Technologies Co., Inc.

• If you find this work is useful for your research, please consider citing:

@article{zhang2023dual,
  title={Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autononous Driving},
  author={Zhang, Xinyu and Wang, Li and Chen, Jian and Fang, Cheng and Yang, Lei and Song, Ziying and Yang, Guangqi and Wang, Yichen and Zhang, Xiaofei and Yang,Qingshan and Li, Jun},
  journal={arXiv preprint arXiv:2310.07602},
  year={2023}
}