My work in China. (Documentation with more details)
- Single image based localization algorithm
- Mobile visual imu odometry
- Image marker tracking
- Other computer vision
Company Introduction
-
UTOPA : also called ‘GaoWei Tech’ is subcompany, fully supported by GTLAND . GTLAND is a real estate company. It owns four large plazas in the very center of GuangZhou. However, with the change of real estate industry, it is in the urge to transform. It choosed the ‘AR’ (augmented reality) industry, to fully explore its plazaa.
-
Team : we have dozen of algorithm engineers, major in Visual SLAM (simultaneous localization and mapping), Visual Deep Learning, and 3D Reconstruction. Support with a few dozens of employees working on Unity3d, and a few dozens working with server back-end. I am part of the algorithm group.
-
My Work : I am fully charged of one major aspect (signle image based visual localization algorithm), and in the direction of two other parts (mobile device visual SLAM algorithm, and 6-DOF Image marker tracking algorithm). The ‘productions’ of our group are SDKs could be used in Unity Android.
Single image based large scene localization
I am fully in charge of this part. Our system is based on Colmap SFM system, but I have our own modification to make it work for our system. videos_indoor, videos_ar and video_outdoor.
Algorithm:
- Offline Mapping:
- image pretreatments : a fast blur detection algorithm; deep learning based illumination equalization.
- sensor fusion:
- imu based scale estimation.
- lidar based pose graph refinement.
- dense reconstruction:
- lidar-image mapping with solid lidar slam, Using Neuvition.
- Deep learning based MVS methods. articles research and DeepMVS test
- Online Localization:
- various feature supported : SIFT, FAST, SuperPoint + SuperGlue.
- We have further developped an local version of the algorithm (which could run entirely in mobile phone, will with a small loss of robustness). Local Map
- Application (Unity):
- Build a scan application (in Android Devices) demo, to simultaneous filtering image, and record IMU data for scale estimation.
- Build a map analysis interface to help user better build a map with our system.
- Build a complete Unity Ar application (with automatic image sending system, image filter system, and pose smoothing filter).
Other:
- Work Flow:
- Linux Algorithm development. C++ development, followed by Java implementation.
- Build serve, and define interface. Build Android server communication system, and the message encoder/decoder.
- Build the corresponding Unity project.
- Mapping cloud demo: Colmap extremely large scene
- Some development log: ServerLocalization
- WebGL demo development.
Backups
Moblie Phone Visual SLAM
We use the ORBSLAM2 base to develop a SLAM system, we our (mostly mine) modification of the code to better suit our project. videos
- Use a marker image to initialize the SLAM system, to have a faster and better initialization, and have an accurate estimation of the scale.
- A system to offer a result with higher frequence, which is realized using optical flow tracking and pose only bundle adjustment.
- A loop clousre system based on marker images (some posters that we can put in our scene), use an other system to obatin the exact pose of all the marker in our coordinate system. As a result, this loop clousre system is perfectly match with one single global map.
- Now is working with PCG (preconditioned conjugate gradient) algorithm to accelerate the SLAM system my blog . And also prepare the implementation of incremental bundle adjustment (IBA).
Backups and Documents:
- VIO documents
- PCG Matlab
- ORBSLAM2 Android Modified : baiduyun/UTOPA/Backups/2020_05_ORB-SLAM2-based-AR-on-Android-2020-05.zip
- ORBSLAM2 Andorid Corresponding Unity project : baiduyun/UTOPA/Backups/2020_05_UnityCameraOrbslam.zip
- VINS Android Modified : baiduyun/UTOPA/Backups/2019_vins_android.zip
Work Flow:
- Linux PC algorithm developments.
- Andorid Native Java environment development, and test the algorithms.
- Build Android Library, build corresponding Unity project.
- Test and find problems.
Marker image based tracking
We have another co-worker deal with it, but I think he didn’t do a great job. So I have realize all the algorithms on my own, in my private time. videos
- Single marker detection (version of my co-worker is ORB feature match based, and version of mine is a brute force finder). Mine version could realize a faster and more robust detection (using a Branch-and-Bound optimization structure).
- Multi-marker tracking, based on optical flow tracking and a NCC patch match to refine.
- Structed markers detection and tracking (‘structed’ means we have prior of the relative poses of the markers). Particularly, I build a system to track a cube object (we track its four side faces).
- Randomly placed marker tracking system.
- Place the markers randomly at the scene.
- Using a offline reconstruction algorithm to find their relative poses.
- Used the localized markers realize AR camera tracking within the scene.
- Cooperate with a third-party SLAM system. Particularly, our system (and the demo video) is a cooperation with ARCORE (from google), and we realize a basic demo which has the potentail to achieve Vuforia’s performance.
Backups and Documents:
- Single Tracking : baiduyun/UTOPA/Backups/2020_05_Marker-based-AR-on-Android-2020-05.zip
- Single, Multi, and Cube Tracking : baiduyun/UTOPA/Backups/2020_05_Multi_Marker-based-AR-on-Android.zip
- Corresponding Unity Project : baiduyun/UTOPA/Backups/2020_05_multi_marker_arcore.zip
- Random Multi Tracking : baiduyun/UTOPA/Backups/2020_07_Multi_random_marker.zip
Work Flow:
- Same as the former one.
Other Computer Vision stuff
- Hand Tracking Python Use traditional Opencv Method and Deeplearning methods repectively for the task of hand detection.
- Probability Graph model based image segmentation Use Probability Graph model based algorithm for a simple image segmentation task, a realization of GraphCut algorithm, and an example of L1 heuristic for filling missing data.
- Lidar SLAM : Try to realize an simple ICP based lidar odometry algorithm. And try an mapping algorithm using part of the result from SFM visual reconstrction. some clouds :RealSense lidar pop art and RealSense lidar office
- Point Net ++ Kitti Detection Test : Using point cloud clustering algorithm for object detection, and using a Point Net++ network for object classification.