Table of Contents

• DeepMirror
• My work 2024
• My work 2023
• My work 2022
• My work 2020 - 2021

mindmap
WORKs
  (Online SLAM)
    Lidar SLAM
      Lidar-IMU
      Lidar-IMU-Image
    Visual SLAM
      Visual-Car
      Visual-IMU
      ML Re-Localization
  (OffLine Mapping)
    Visual Based
      Point Based SFM
      Global Averaging
      Line Based SFM
      Image MVS
      Map Summarization
    Lidar Based
      TSDF Pointcloud
      Mesh : Delaunay Ray Tracing
      Multi-Map Fusion
    Deep Learning
      NERF
      Gaussian Splatting
  (Other Image Processing)
    Multiplane Images
    Monocular Depth
    Infra-Red
    Panorama
  (MR)
    World Generation
    MR 6DOF in Vehicle
    Vertigo Reduction
    RunTime Rendering
    OpenXR Native

DeepMirror

website : MirrorSpace - City scale spatial mapping and localization.

My work 2024

A. Low Cost INS

Use low quality GPS & IMU only. Achieve state-of-art INS performance.

B. Automatic World Generation

Generate random world, based on earth topology and OSM road map. Video.

• Terrain generation (random while fits earth topology).
• Shaders : shadow & light & fog & water.
• MR rendering.

C. MR in Vehicle 6DOF

Enable MR device to have 6DOF tracking in any vehicle.

• Well designed fusion algorithm to give a comfort experience.
• Output poses both in car reference frame & world reference frame.
• Realtime on board absolute localization (w.r.t. car) using deeplearning.

D. Video Streaming

Stream outside Panorama view to MR.

• Camera hardware connection, image stitch, video encoder.
• Video Streaming using FFMPEG.
• MR video decoder, panorama rendering.

My work 2023

A. Outside-in Infrared Localization Modules

IR light detection, and fusion with imu measurement by kalman filter. Run with OpenXR & Monado.

• VR hand 6dof tracking.
• AR/VR camera tracking in difficult scenes (e.g. inside moving cars).

B. Panorama Video Visual Mapping

Panorama video (insta360 & go-pro) is the only input.

• pure visual sfm mapping pipeline based on Global Averaging method (shonan rotation average & 1dsfm translation average).
• usage of IMU:
  • extrinsics & timestamp calibration for camera-imu.
  • scale and gravity recovery of the visual map.
• MVS image depth recovery (both traditional cv method and Deep Learning method).

C. Line Mapping & Localization

see Line Mapping Page.

• Line mapping for traffic lane mapping.
• Use line feature for visual based localization : my branch of LIMAP.

D. Deep Learning

• Use Dense Match with relative pose (Global averaging) for localization and visual mapping.
• Use Nerf with our data session.
• Use Multiplane Images for render live 3d videos (as shown in Apple Vision Pro), test video in PICO.
• Use ACE for mapping, and check its descriptor quality with match test.

My work 2022

working on slam/vlp system.

A. INS fusion

working on meta-verse for cars.

• car localization - ins system : Based on iterative extended error state kalman filter. imu (motion model) + gps + chassis + visual localization.

• vr in-car localization - work with guangqi for AR-HUD & VR.
  • 广汽ADiGO SPACE升级沉浸式智能座舱体验 2022.
  • patent: 一种车载XR设备定位方法、装置、设备及存储介质 CN115690194B.
  • DeepMirror在宝马汽车元宇宙比赛中荣获第一名 2023. First place in the “Vehicle Readiness” category was secured by Chinese start-up DeepMirror Inc.

B. VLIO algorithm

lidar-imu-image slam, based on image direct method with photometric refinement (following last year’s work). patent: 基于激光视觉融合的建图方法及系统 CN115797490B.

C. pointcloud & mesh generation

Designed for multi-session lifelong map, handle environement change.

• Color Point Cloud : TSDF generation (using cuda c++).
• Mesh Generation : Delaunay + ray casting + min-cut + post-processing.

D. visual map summarization

For simplification of visual localization map.

• Vision bundle adjustment problem graph analysis. (using SNAP)
• Solve ILP (integral linear programming) problem based on paper
• Keep 10% the points, with neglectable drop in localization benchmark accuracy.

My work 2020 - 2021

A. lidar-imu-gps slam algorithm

• 2 versions (all written by our team) :
  • pose graph optimization (reference LIO-SAM)
  • iterative extended error state kalman filter (reference FAST-LIO2).
• Many additional constrains, robust backend optimzation, more robust imu initialization.
• Loop closure based on global descriptor (ISC).
• Sensor set: imu, rs-lidar-16, gps, cpu only, 20 FPS on nvidia-nx.

B. camera-chassis-gps slam algorithm

Single camera semi-dense direct method (reference DSO) to mapping fastly the whole city road map.

• tightly coupled with chassis input.
• cpu + arm refinement, 50 FPS on nvidia-nx.
• designed pipeline, mapped full Nansha.GuangZhou Area.

C. multi-session fusion

Robust Distributed Pose Graph Optimization. (robust to outliers, and multi-threading process with graph cut)

• Loop finding : gicp & global registration (based on FPFH + TEASER) & ISC (intensity scene context).
• Database structre based on s2 blocks.
• Mapped full Nansha.GuangZhou Area (until 2022).

D. car routing

• based on OSM lane map.
• lane level map & way point level map.