Home | PhD research | PhD work | Mouse trap | CNC milling | CNC milling 2 | CNC artwork | 360 Panoramic images | Camera pan tilt unit | Software | Computer vision

PhD research - Vision Based Global Localisation


Table of contents
  1. Introduction
  2. How it works
  3. Images
  4. Videos

[Top]

Introduction

My research is on vision based global localisation. What? Basically, I developed a system that can be used to estimate position and orientation (localisation) for a mobile robot, or anything that moves really. This functions similar to the GPS found in modern cars. My system uses an off the shelf webcam with a special panoramic mirror that provides a full 360 degrees field of view. A rich 3D map (similar to a first person shooter game) is used to provide the system with a working knowledge of the environment. The system works by capturing images from the webcam and determines the most likely place it was taken from relative to the 3D map. Overall, the system achieves an accuracy of within 1 metre for position and 10 degrees for orientation and can operate where a GPS is ineffective eg. indoors. A moderate size 50x50m environment was used for testing. 		 

 

[Top]

How it works

An overview of the process of building the localisation system is presented in Figure 1. The first step, and the most time consuming, was to build a 3D model of the environment. This was done by using a Riegl Z420i laser range scanner, shown in Figure 2. A high resolution 3D scan was taken at various locations in the environment and stitched together to build a complete model. Using the 3D model, 2D images were then generated (from the point of view of the webcam) every 0.5 metre and 10 degrees, and a unique scene signature was extracted from each image. The signatures are stored in a database to be used later on. To localise the robot, the same signature was extracted from the webcam and fed through a particle filter. The particle filter is a probabilistic algorithm that can be used to estimate the pose (position and orientation) of a robot given sensory and odometry information. The particle filter in this case, uses the signature from the webcam and odometry information from the robot to assigns a probability score of match to all the other signatures in the database, and outputs the estimated pose of the robot (x, y, θ).

My system is a rather brute force approach to global localisation but works effectively. The novel part is using the 3D model to sample the environment as oppose to manually driving a robot around and taking images. This not only can be a more time efficient process, but can also be more practical depending on the application, and does not suffer from accumulated odometry errors from manually driving a robot. The Riegl Z420i scanner used at the time was rather slow, capturing about 16,000 data points max, so probably took longer than manually taking images. But recently there has been a large leap in the performance of laser scanners and scanners capable of over 1 million points a second are already on the market (Veloydone), making the proposed method a practical choice.

 

Figure 1: Process of building the localisation system

Figure 2: Riegl Z420i laser range scanner. Effective up to 800m, with 360x80 degrees field of view and user defined angular resolution. Mounted on top is a Nikon SLR camera to record colour information, the laser does not provide any.

[Top]

Images

Below are the three robots I have used during my research. The first robot was a commercial ER1 robot, which I had the displeasure of using. The wheels are too small for outdoors, the motors are too slow, and the battery only lasted about 30 minutes. I also went through two controller boxes that blew up. The capacitors exploded leaving a foul smell and brown powder inside the case.

The second robot was a hand-me-down from a colleague who completed his PhD and no longer needed it. I stripped the robot apart down to the bare minimum as shown in the photo. The base is from a wheelchair with a custom controller hacked onto the factory one and uses a serial connection to communicated with the PC. Two 24V motors power this robot. This robot was simple and great to use but had malfunctioning problems that made it spun out of control whenever the STOP command was sent. I had to hack on an external relay to cut the power whenever a STOP was sent. There is also a Hokuyo URG laser scanner mounted at the bottom of the robot for collision avoidance.

The last robot can barely be called a robot. It's just a walking frame used by elderly people with the localisation system bolted on. It's meant to be a proof of concept that my localisation system can be bolted onto anything that moves. For odometry information, which is required by the particle filter, a simple visual odometry system was implemented using OpenCV's KLT tracker for optical flow and assuming the user walks at around 1 metre/sec.

ER1 robotCustom made robot using a
wheelchair for the base		No frills walking frame
  
 

[Top]

Videos

Below are some videos showing various parts of the system and some applications utilising it.
3D point cloud demo
This video shows an animated re-play of a 3D scan. This scan would have taken about 35 minutes to complete, which includes the scanning and capturing of images using a Nikon SLR mounted on top of the scanner. The user can select the angular resolution of the scan, which controls the total time and number of points captured. The black circle in the middle is the scanner's blind spot.
Stitching two point clouds together
The point clouds are stitched together manually using a custom written software. The user selects at least 3 corresponding points between the two scans and the software calculates the optimal transformation using the Iterative Closet Point algorithm. Each scan is stitched to its nearest scan.
Scanner on a motorised cart
One of the earlier equipment used to make moving the scanner around easier. It is a custom motorised cart controlled by a joystick. The cart has since been dismantled and used in a different project. I did the scans in my thesis by manually moving one piece of equipment at a time to a different scan location. The equipment includes: the laser range scanner, a table, a petrol generator, a 240V inverter, and a laptop.
3D Walkthrough
A walkthrough of the completed 3D model used in this research. This is the engineering area at Monash and measures about 50x50m.
Autonomous navigation 1
In this experiment, the robot was placed at a random location. It moves around randomly until it determines its position and automatically navigate its way to a user defined goal point. A short laser range finder was mounted at the bottom to prevent it from crashing into obstacles.
Autonomous navigation 2
Another experiment similar to above, but at a different location. The robot narrowly misses a bicycle in this experiment. For some reason the laser did not see it.
Real-time localisation of a walking frame
The localisation system can operate in real-time and is demonstrated by tracking a person using a walking frame at around 10 fps. The first segment of the video shows the particle filter in action (represented by blue dots). The second part shows the 3D view of the estimated pose. In theory, it is possible to use the system for augmented reality applications also.