CloudMerger – Java based application for merging (kinect) point clouds [Final project]

For my last three weeks at the Faculty of Geo-Information Science and Earth Observation (ITC) at the University of Twente in May 2012, I worked on an individual final assignment to end my program, Post-Graduate Diploma of Geoinformatics. My supervisors were Dr. Kourosh Khoshelham and Sander Oude Elberink.

Following is a presentation of the resulting application. I have published the source code (Processing) along with the applications (Linux, Mac and Windows) on my Github. To simply try the app, download the zip-file for your operating system and the test data.

Background

In indoor mapping, the Kinect sensor provides an affordable and highly portable possibility for capturing 3D data point clouds for analysis. For this to work, several point clouds of each room are needed to completely cover the area of interest and capture objects from all angles.

The coordinate system of the range data from the Kinect is relative to the sensor itself. This means that if you capture consecutive point clouds with it having moved the sensor in between captures, they will not display correctly relative to each other.

To amend this, the clouds have to be transformed (rotated and translated in space) into the same coordinate system. For clouds with overlapping areas, there exist several algorithms to do this.

The assignment

The assignment was to develop a Java based application that loads two point clouds, allows the user to select corresponding points, perform a transformation algorithm from one cloud to the other and display the result visually.

From the assignment, defining functional requirements were easy to identify:

  • Load point clouds.
  • Display point clouds.
  • Register corresponding points using cursor.
  • Apply transformation.
  • Display and print result to file.

The input data are two point cloud files that are plain text files with 6 space-delimited numbers for each point. These represent x-, y- and z-coordinates (distance to Kinect sensor in cm) and red, green and blue color value (0-255).

-164.81 309.59 115.65 170 145 165
-164.82 309.59 115.12 174 141 162
-166.31 312.35 115.62 181 155 160

The files from the Kinect consists of about 300000 points each, at about 10 MB each.

Two output files are created. One merged point cloud and one report file stating which points were registered, the resulting rotation matrix and translation parameters and finally, the calculated residuals.

The Application

Following is a boring walkthrough of how the application works. Feel free to test it out first and then consult this text when you get stuck.

Loading the point clouds

When the application is first opened, the user is met with a menu screen with four buttons. At first, only the ”load cloud” are active. Click the ”load cloud 1” to open a file chooser dialog and select the point cloud file to load it. It should be on the format specified in the input section. Do the same for ”load cloud 2”.

Loading point clouds in CloudMerger

The application may in some cases work on data of different format, but this is the only format it has been tested for. Other formats may yield unexpected results or cause the application to crash. If you test it out, please let me know!

The ”view cloud” buttons are now active. Click one of them to view the point cloud. You can now use the mouse to rotate and zoom around the cloud, press the TAB key on your keyboard to switch between the cloud screens and the menu screen or the BACKSPACE key to go back to the menu screen.

Registering corresponding points

To start registering points, click on ”view cloud 1”. Rotate and zoom the cloud until you see the point you want to register (see picture below). Hold the SHIFT key while clicking with your mouse over the point you want to register. If successful, you should see that the window title changes from ”Cloud 1, registered 0 points”, to ”Cloud 1, registered 1 points”. You can then press the TAB key to switch to the other cloud and repeat the process for the other cloud.

Viewing cloud and registering points.

Viewing cloud and registering points.

It is also possible to register more points in the first cloud consecutively, before registering the points in the other. Just make sure to register them in the correct sequence. When you have registered at least three points in each cloud (and the same number of points in each), you are ready to perform the transformation. Press the BACKSPACE key to go to the menu screen.

If you make a mistake while registering points, you may press BACKSPACE to go to the menu screen and click the ”reset registration” button to start over.

Perform transformation

When at least three points have been registered in each cloud, more buttons appear on the menu screen (picture below). First, press the ”merge clouds” button to perform transformation on cloud 1 to match cloud 2.

When ready, more options appear on the menu screen.

When ready, more options appear on the menu screen.

View and export merged clouds

When the transformation is performed, two more buttons appear. You are able to view the merged clouds in one screen and rotate and zoom it like with the input clouds (see picture below). The clouds are displayed in different colors, so that it is easy to see if the transformation was successful.

Viewing the merged clouds.

Press BACKSPACE to go back to the menu screen, and click the ”Print merged cloud to file” button to export the merged clouds as a .txt-file. A ”save as” dialog appears (see picture below) and you are able to set the name of the output file. The file is stored at the desired location, along with a report file.

Printing merged cloud to file.

Printing merged cloud to file.

A little data handling, csv table data to SQLite database

Thought I’d post a little step-by-step on how I imported the .csv table data from the Open Data Enschede into an SQLite database and prepare it for my EnschedeLocator. This is mostly for my own sake, so that I have it as a reference, but it might also be of interest to others.

Address data set [.csv 8.3MB]

Store your data set in a working folder. I am on a Mac and have SQLite3 installed and will use the terminal. The following steps on Windows and Linux should be similar.

The first thing I do is to open the file in Excel and remove the columns I don’t need. I only need the street name, house number, latitude and longitude. Delete the other columns and save. You don’t need the first row with the column names, so just delete that too.

Then insert a new column in front of street name. Here we will keep each address’ primary key. Just use the formula =ROW() and double-click the bottom right corner of the box to apply to all cells in that column. Save the .csv file. It should now be about 3.8 MB.

Then open Terminal and move to your working folder. Create your SQLite database by executing:

sqlite3 databasename.db

To make the database Android friendly, you have to create a new table called android_metadata and fill it with some columns. Then create a table for the addresses.

CREATE TABLE "android_metadata"("locale" TEXT DEFAULT 'en_US');
CREATE TABLE "addresses"(_id integer PRIMARY KEY,
straatnaam text, huisnummer integer, latitude double, longitude double);

Now we’re gonna use the import the data.

.separator ","
.import enschede_addresses.csv addresses

And that should be enough. Note that if you have a .tsv-file or something else, you need to change the .separator command accordingly.

The first time I tried, I only got the first row imported. This is due to the line-break format of the csv-file, SQLite requires the LF-format. This can be fixed in a text editor, such as TextWrangler.

If this happened to you too, run a DELETE FROM ADDRESSES; to erase the data you imported, change the line break format and import it again.

Here are the complete steps in terminal:

Due to removing some columns in the dataset, the database now contains some redundant entries, and since we’re storing the data locally on the device, we want to minimize the size.

Here is one way of removing all but one row containing the same street name and house number:

DELETE FROM addresses WHERE _id NOT IN(SELECT MIN(_id) FROM addresses GROUP BY straatnaam, huisnummer);
vacuum;

Only deleting rows doesn’t reduce the file size of the db, to do so, run the vacuum command in addition. The size of the database now went from 3.8 to 3.1 MB.

That’s it! The database file is now ready to be read by an Android application.

[App] EnschedeLocator Beta, Offline map and simple navigation in Enschede for Android

Background

Two weeks ago, a friend and I visited the OpenData Enschede Hackaton. The municipality has a project making a platform for serving data to the public, an overview of the datasets can be found here. I sat down with the address dataset and converted it into an SQLite database, and started working. Now I feel like the app is ready for trials, so I am making the  beta available here. You need Android 2.3 or higher (due to restrictions in the database size in the earlier versions) to run it and the app itself requires 6.6 MB and it also stores the map on your sdcard, taking up 1.93 MB there.

Download EnschedeLocator Beta [.apk]

I’m not going to monitor how many people download it, but if you do and have some notes, tips, flattering or offensive remarks, please let me know in the comments, twitter, email or any way you see fit.

I have tested this app with my own Samsung Galaxy S I9000, if you use a different phone and experience strange behaviour, please let me know.

Description

The app features a map and address database stored locally on the user’s device, allowing for simple offline navigation in Enschede.

Screens

Usage

  1. Start typing in street name. Suggestions for streets pops up, select your destination street.
  2. Select the house number from the spinner wheel and touch the Go-button.
  3. An OpenStreetMap of Enschede displays with your location (the balloon) and that of your destination. A compass needle, getting data from your device’s orientation sensor and your location, is pointing in the straight-line direction of your destination.

Tips

  • Make sure your location on the map is correct before embarking on your adventure in the night.
  • If the compass needle seems wonky, move your device in a figure-eight like this. If it still seems wonky, don’t base your journey on it.

Source code

The source code will be released on Github.

The map

The map comes from OpenStreetMap and is prepared as described in an earlier post. To save space, only one zoom level (15) is selected, this is were the starting level in the app as well. To add more zoom levels, just follow the steps in the aforementioned post and put the resulting .zip file in the /sdcard/osmdroid/ folder of your phone.

Note

The app stores the map in /mnt/sdcard/osmdroid/Enschede.zip, if it doesn’t already exist. To completely uninstall app, you must first clear data and uninstall from your application settings, then remove the osmdroid folder from your sdcard.

Smoothing out sensor values from SensorEvents, specifically for Orientation, in Android

For my EnschedeLocator app, I want the user to get a compass needle that points towards the destination address using the device’s orientation sensor and the calculated bearing between the user’s location and his destination. For this, I implemented and tweaked Google’s Compass example.

However, the compass seemed very jittery and unreliable, and frustration ensued. The simple solution came via this StackOverflow thread. Simply store a number of recent values and average them. This simple fix made the needle rotate smoothly and stay on target.

/**Tweaked method for setting an average value. Pass the compassAverage to canvas.rotate()*/
public void setmValues(float[] mValues) {
        this.mValues = mValues;
        compassValues[0]= compassValues[1];
        compassValues[1]= compassValues[2];
        compassValues[2]= compassValues[3];
        compassValues[3]= compassValues[4];
        compassValues[4]=mValues[0];
        compassAverage = (compassValues[0] +compassValues[1] +compassValues[2] +compassValues[3] +compassValues[4])/5;

}

Using OSMDroid for Offline mapping in Android, step-by-step [Tutorial]

For my EnschedeLocator app, I wanted the users to be able to store an offline map on the device and see it while navigating to their destinations. In our BeerFinder project at NTNU last year, we were able to do this using Nutiteq’s android library, but poor documentation lead me to looking for other solutions. I discovered OSMDroid which is a library that aims to be an opensource alternative to Google MapView in Android.

Below, I will outline how I did it and how I solved some small obstacles i encountered. Note that there may be other and better ways of doing this, but this is what worked for me.

Tools needed

  • Eclipse with Android SDK (assuming you know the basics here).
  • Mobile Atlas Creator, for preparing map tiles.
  • OSMDroid library, available here. I used osmdroid-android-3.0.7.jar.
  • SLF4J Android library, available here. I used slf4j-android-1.5.8.jar. Without this library I got a NoClassDefFoundError when trying to run the app. Adding this library somehow fixed it [source].

Preparing the map

To download the map tiles I used Mobile Atlas Creator.

My steps were as follows, most settings are done in the options pane to the left of the map:

  1. Set atlas format to Osmdroid ZIP [Atlas – Convert Atlas Format].
  2. Set your map area coordinates, either by clicking and dragging on the map, or by setting min and max coordinates in the coordinate selection pane to the left.
  3. Set map source to OpenStreetMap Mapnik. Note: I originally set my source to something else, and the map wouldn’t appear on my phone. I’m sure there’s a simple fix for this, but haven’t looked into it yet.
  4. Select the zoom levels you want to include. This will impact the storage space required. As you select more levels, the number of tiles needed are displayed. Each pane takes about 20kilobytes. I included all levels because my area of interest is so small that I’m not to concerned about that yet, I might make small versions available when I publish the app. For my area, 178 tiles + 20kB = 3.48MB.
  5. In the Atlas Content pane, set the name of your map first, then click “add selection” (the name really isn’t important, though).
  6. Then select “Create atlas” and your map is stored in the atlas folder under Mobile Atlas Creator.
  7. Move the resulting zip-file to /mnt/sdcard/osmdroid/ on your device. (For a slight improvement in performance, you can unzip the file and move the resulting Mapnik-folder to /mnt/sdcard/osmdroid/tiles/ instead. The difference in size isn’t that big, but the zip-file might be a good way to organize different maps if you have several areas, but I digress).

That’s it for preparing the map! Again, these are steps and settings that worked for me, others might work better, but I haven’t researched them yet.

Importing the libraries to Android project

When I attach external libraries to my projects I put the .jar-files in the project folder so that I know where they are.

  • Move the osmdroid-android-3.0.7.jar and slf4j-android-1.5.8.jar (and osmdroid-android-3.0.7-javadoc.jar for documentation) into your project folder.
  • In eclipse, right click the project and click refresh, the files should show up.
  • Right-click the files and select [Build Path -> Add to Build Path].
  • Then, go into [Build Path -> Configure Build Path] and under the Libraries tab select the osmdroid.jar and edit javadoc location to the osmdroid-android-3.0.7-javadoc.jar file.

The code

This example code shows the map parts of the Activity class and should be enough to view the map. Note that this Activity doesn’t need to specify an .xml layout as we’re doing it in the code.

package com.yourpackage.name;

import org.osmdroid.util.GeoPoint;
import org.osmdroid.views.MapView;

import android.app.Activity;
import android.os.Bundle;

public class OSMDroidMapActivity extends Activity {
    public void onCreate(Bundle savedInstanceState){
        super.onCreate(savedInstanceState);

        MapView mapView = new MapView(this, 256); //constructor
        mapView.setClickable(true);
        mapView.setBuiltInZoomControls(true);
        setContentView(mapView); //displaying the MapView
        mapView.getController().setZoom(15); //set initial zoom-level, depends on your need
        mapView.getController().setCenter(new GeoPoint(52.221, 6.893)); //This point is in Enschede, Netherlands. You should select a point in your map or get it from user's location.
        mapView.setUseDataConnection(false); //keeps the mapView from loading online tiles using network connection.
    }
}

This displays the MapView only. If you would like to add some other Views, you can remove the setContentView(mapView); and add the following to your class and remember to do the right imports.

TextView myTextView = new TextView(this);
        myTextView.setTextAppearance(this, android.R.style.TextAppearance_Large_Inverse);
        myTextView.setText("Enschede, Netherlands");
        Button myUselessButton = new Button(this);
        myUselessButton.setText("Click");

        final RelativeLayout relativeLayout = new RelativeLayout(this);
        final RelativeLayout.LayoutParams mapViewLayoutParams = new RelativeLayout.LayoutParams(
                        RelativeLayout.LayoutParams.FILL_PARENT,RelativeLayout.LayoutParams.FILL_PARENT);
        final RelativeLayout.LayoutParams textViewLayoutParams = new RelativeLayout.LayoutParams(
                        RelativeLayout.LayoutParams.FILL_PARENT, RelativeLayout.LayoutParams.WRAP_CONTENT);
        final RelativeLayout.LayoutParams buttonLayoutParams = new RelativeLayout.LayoutParams(
                        RelativeLayout.LayoutParams.FILL_PARENT, RelativeLayout.LayoutParams.WRAP_CONTENT);
        buttonLayoutParams.addRule(RelativeLayout.ALIGN_PARENT_BOTTOM);

        relativeLayout.addView(mapView, mapViewLayoutParams);
        relativeLayout.addView(myTextView, textViewLayoutParams);
        relativeLayout.addView(myUselessButton,buttonLayoutParams);
        setContentView(relativeLayout);

Basically, we did three things here:

  • Create the TextView and Button.
  • Create the RelativeLayout and the rules for the different Views.
  • Add the Views to the Layout and finally set the Activity to show the layout.

Now we should get something like this:

Populating AutoCompleteTextView with SQLite data

I’m currently working on a hobby project with working title EnschedeLocator, which is an Android app where users can type in an address and using their unit’s GPS receiver, get an arrow that points them to the destination. A simple offline navigation tool. The addresses comes from Opendata Enschede, and are stored with coordinates in a database shipped with the app.

For my EnschedeLocator app I want the user to be able to start typing the name of the street he wants and then receive suggestions as he types. The idea is that he has to find a match in order to continue or else the street wont be in the database.

After spending too much time trying to populate an AutoCompleteTextView using a Cursor directly (might be a dead end), the simple solution came from GiantFlyingSaucer. Simply create a method in your SQLiteOpenHelper class that turns you query into a String-array and then use that to populate the ArrayAdapter as shown on Android Developers.