Name and Contact Information

Title

Object Detection on Point Cloud Streams

Short description of work done

I developed a comprehensive real-time streaming pipeline for point cloud object detection using PointNet++ and PointPillars ONNX models. The pipeline supports multiple 3D vision tasks including classification, part segmentation, semantic segmentation, and 3D object detection from point cloud data.

The system features a modular architecture with configurable input sources (RealSense camera, dataset files, WebSocket streaming), multiple model backends (PointNet2 variants and PointPillars), and flexible output methods (WebSocket and OSC protocols). I implemented automatic performance monitoring with FPS tracking and created an optional visualization system using Open3D for real-time 3D rendering of results.

Key technical contributions include: developing abstract base classes for inputs, backends, and outputs to ensure modularity; implementing ONNX runtime integration for efficient model inference; creating a configuration management system using YAML files; and building a visualization manager that automatically selects appropriate visualizers based on model type. The pipeline processes point clouds in real-time, supports both CPU and GPU acceleration, and can run in headless mode for server deployments.

I also created comprehensive documentation including a detailed README with usage examples, setup scripts for easy environment configuration, and dependency management files for both conda and pip installations. The system is designed to be easily extensible for new models and input/output protocols.

What code got merged

What code didn’t get merged

NO

What’s left to do

I will optimize the pipeline to improve visualization speed and achieve higher FPS. This involves implementing asynchronous processing to separate visualization from model inference, optimizing Open3D rendering, and adding GPU acceleration where possible.

I will also retrain the models to better detect indoor persons by collecting additional indoor point cloud datasets with person annotations, fine-tuning the existing PointNet2 and PointPillars models on this specialized data, and implementing data augmentation techniques for indoor environments.

The goal is to achieve real-time performance suitable for live applications while maintaining high detection accuracy for indoor person recognition.