This my latest and greatest markerless/natural feature augmented reality demo. This new code is called NAR Demo, short for Nghia’s Augmented Reality Demo, for a lack of a better name 🙂 So what’s the difference between NAR Demo and MarkerlessARDemo? A bucket load, just to name a few:
- No CUDA specific code, anyone can run it now (woohoo!)
- Irrlicht Engine for cross platform GUI, the power of a full featured game engine!
- Multi-threaded design for real-time processing
- Replaced FAST feature with more robust Difference of Gaussian blob features
- Experimental optical flow code, this improves stability a lot (less jittering)
- Bug fixes from MarkerlessARDemo
- LOTS of code cleaning
NAR Demo is meant to serve as an educational tool, there are no plans to make it anything beyond that.
I’ve made a PDF document that focuses on the computer vision aspect of NAR. It’s general and not code specific and might be useful if are writing your own AR system. Download NAR_Demo.pdf, also found in the doc directory of the 7z package.
Download
Last update 05/02/2012
Changes
- Minor bug fixes in DOG.cpp
- Added computer vision documentation in doc (work in progress)
  NAR_Demo-0.2.1.7z (7.2MB) Use 7zip to decompress. This file includes both a Linux (CodeBlocks) and Visual Studio 2010 project. I’ve only tested it on a 64bit machine. There is a pre-compiled Windows 64bit binary in the root folder if you want to test it out without compiling. But first please read the usage section down below. IMPORTANT: The OpenCV Windows dlls I provided have been compiled on an Intel i7. If you have a CPU that doesn’t support SSE3 instructions it might crash. In which case, you will have to compile your own version of OpenCV. |
GitHub
The code is also available on GitHub
https://github.com/nghiaho12/NAR_Demo
Compiling
To compile you need the following libraries installed
- OpenCV 2.3.x (or above)
- Boost C++ libraries
- Irrlicht Engine
If you are using the Visual Studio project you will have to edit the project and point it to the directories of the libraries mentioned above.
32 bit systems: If you are compiling for a 32 bit system you need to edit NAR/NAR_Config.h and comment out #define USE_SSE4. The POPCNT assembly instruction seems to work only with 64 bit compilers.
Usage
The executable NAR_Demo or NAR_Demo.exe can be found in the root directory. It accepts no arguments, but first you have to provide it an image of the object you want to detect at the following path:
NAR_Demo/media/AR_object.png
There is no default AR_object.png provided, NAR_Demo WILL NOT run without this file.
I recommend finding some software that can view your webcam/camera, hold the object in front, take a screenshot, and crop as required. The image should be no bigger than your video resolution. The object MUST be planar/flat for it work correctly.
Once NAR_Demo is running, just hold up your object for detection.
You might notice it’s very slow (< 4 FPS) when the object is not detected. This is due to OpenCV’s cv::findHomography function, which lacks an option to specify the maximum number of RANSAC iterations, it’s hard coded to 2000. I’ve submitted a suggestion to OpenCV’s bug track system, here’s hoping they read it.
License
Simplified BSD. See LICENSE.txt in the root directory.
great, this is great. thank you so much!
Works great! But my CPU fan keeps running like it will break open from the motor; any suggestions for keeping the CPU fan off?
NAR Demo is designed to hog the CPU and use every last bit of it 🙂 I don’t have any settings to reduce the CPU usage but maybe you can try lowering the priority of the process?
The camera should turn off if you close the program cleanly. Hit ESCAPE to close the program, it should turn the camera off. I don’t explicitly close the camera, I assume OpenCV will.
Also when I close the program window, the camera keeps on. Do you have any suggestions for this?
Dear doctor Nghia,
I would like to report that NAR can be built on MacOS X Lion.
Below is what to do to build.
It’s easier for me (and maybe others) to contribute if you use GitHub ;).
————————-
NAR needs OpenCV, Boost, and Irrlicht.
Install OpenCV to $HOME/opt/opencv-2.3.1 and Boost to $HOME/opt/boost_1_48_0
————————-
Build Irrlicht 1.7.2 (the latest 1.7.3 has build error on MacOS X):
mkdir -p $HOME/src
cd $HOME/src
svn export https://irrlicht.svn.sourceforge.net/svnroot/irrlicht/tags/release-1.7.2/ irrlicht-1.7.2
Open irrlicht-1.7.2/source/Irrlicht/MacOSX/MacOSX.xcodeproj with Xcode.
Config:
64bit
Base SDK 10.6
LLVM GCC 4.2
Edit Scheme, select Release
Select menu Archive, save libIrrlicht.a to $HOME/src/irrlicht-1.7.2/lib/MacOSX/libIrrlicht.a
—————————
Edit NAR:
main.cpp
#ifdef _WIN32
#include
#elif __linux__
#include
#elif __APPLE__
#include
#else
#error “Platform not supported or tested”
#endif
CpuID.cpp
#ifdef _WIN32
#include
#elif (__linux__ || __APPLE__)
#define __cpuid(out, infoType)\
asm(“cpuid”: “=a” (out[0]), “=b” (out[1]), “=c” (out[2]), “=d” (out[3]): “a” (infoType));
#else
#error “Platform not supported or tested”
#endif
Build:
Change to AR_Demo/NAR directory, then:
g++ `pkg-config –cflags opencv` `pkg-config –libs opencv` -msse4.1 -msse4.2 -I$HOME/opt/boost_1_48_0/include -L$HOME/opt/boost_1_48_0/lib -lboost_system -lboost_filesystem -lboost_thread -I/$HOME/src/irrlicht-1.7.2/include $HOME/src/irrlicht-1.7.2/lib/MacOSX/libIrrlicht.a -framework Carbon -framework Cocoa -framework OpenGL -framework IOKit *.cpp ../*.cpp
a.out will be created.
Thanks for post! Maybe I’ll have to give Github a serious look this weekend 🙂
The code is on GitHub now! Link at the top of page.
Thanks,
I’ve just commented at GitHub about version number.
I would like to ask about homography.
At “Pose Estimation For Planar Target” (nghiaho.com/?page_id=576), it seems that only model_3D and iprts parameters are enough.
Why in NAR NAR::PoseEstimation method needs the homography matrix?
cv::Mat image_pts = m_inv_camera_intrinsics * H * m_AR_object_image_pts
image_pts will always be the 4 corner points of the AR object. I decided to use the 4 corner points, which requires the Homography matrix, instead of all the points for consistent speed. I’m not how that affects the accuracy because the homography calculation is quite good, OpenCV does a refining process after RASNAC.
Hi,
I had no idea it was even possible to run it in Python! Sadly, I don’t program in Python so am not able to help you 🙁
Dr Nghia. After several configuration, I have tried to run your code several time, in both Debug and Release mode for windows 7 32bit with VS2010. I saw my camera is running after the Learning pose ended, but It could show any captured image in the NAR, the window is black and FindRADObject always 0ms, for debug mode it’s sometimes give mess buffer full. I couldn’t figure out what’s wrong. I didn’t changed anything in your code. Thanks.
I find camera support in OpenCV to be rather unreliable. I’ve had some good and bad experiences with different webcam. You’re best option is to keep up to date with the latest OpenCV and hope they fix the problem, or try a different webcam. Or if you have time you can attempt to debug it. I had to do this once. If it returns a blank image it indicates some error in the webcam driver, which you can trace back into the OpenCV code. But it’s rather time consuming,
Thank Dr Nghia. After updating Opencv from 2.3.1 to 2.4.1 I could run the code. I think there some problem with Microsoft Lifecam driver in Opencv2.3.1. When I run it without threads, it still working. Anyways, your NAR is very nice AR demo with very intuitive documentation.
Cool, glad you got it working.
i tryed to compile your application on ubuntu i got the following errors
i’m on a 32bits version
i already installer boost opencv and irrlich
i have pentium dual-core CPU
thanks for you help
————– Clean: Release in NAR_Demo —————
Cleaned “NAR_Demo – Release”
————– Build: Release in NAR_Demo —————
Compiling: NAR/AlphaBetaTracker.cpp
Compiling: NAR/BaseThread.cpp
Compiling: NAR/CpuID.cpp
Compiling: NAR/DoG.cpp
Compiling: NAR/ExtractFeatureThread.cpp
In file included from /home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/ExtractFeatureThread.cpp:7:0:
/usr/lib/gcc/i686-linux-gnu/4.6/include/smmintrin.h:32:3: erreur: #error “SSE4.1 instruction set not enabled”
Process terminated with status 1 (0 minutes, 5 seconds)
1 errors, 0 warnings
You probably don’t have SSE4.1 support on your CPU. Edit NAR_Demo/NAR/NAR_Config.h, comment out the line #define USE_SSE4.
i already done that
now i have strange more errors messages
————– Clean: Debug in NAR_Demo —————
Cleaned “NAR_Demo – Debug”
————– Build: Debug in NAR_Demo —————
Compiling: NAR/AlphaBetaTracker.cpp
Compiling: NAR/BaseThread.cpp
Compiling: NAR/CpuID.cpp
In file included from /home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/CpuID.cpp:1:0:
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/CpuID.h:21:3: attention : ISO C++ interdit les structures anonymes [-pedantic]
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/CpuID.h:108:3: attention : ISO C++ interdit les structures anonymes [-pedantic]
Compiling: NAR/DoG.cpp
Compiling: NAR/ExtractFeatureThread.cpp
Compiling: NAR/KTree.cpp
In file included from /home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.cpp:1:0:
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h: In member function ‘float KTree::DistanceSq(const float*, const float*) const’:
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:77:5: erreur: ‘__m128’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:77:12: erreur: expected ‘;’ before ‘sum’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:81:16: erreur: expected ‘;’ before ‘aa’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:82:16: erreur: expected ‘;’ before ‘bb’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:83:16: erreur: expected ‘;’ before ‘cc’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:84:9: erreur: ‘cc’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:84:31: erreur: ‘_mm_mul_ps’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:85:9: erreur: ‘sum’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:85:9: note: suggested alternative:
/usr/include/opencv2/core/core.hpp:2009:32: note: ‘cv::sum’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:85:33: erreur: ‘_mm_add_ps’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:90:24: erreur: ‘sum’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:90:24: note: suggested alternative:
/usr/include/opencv2/core/core.hpp:2009:32: note: ‘cv::sum’
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:90:27: erreur: ‘_mm_storeu_ps’ was not declared in this scope
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h: At global scope:
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:75:14: attention : unused parameter ‘a’ [-Wunused-parameter]
/home/khayati/Travail/realiteAugmente/Nar/NAR_Demo/NAR/KTree.h:75:14: attention : unused parameter ‘b’ [-Wunused-parameter]
Process terminated with status 1 (0 minutes, 7 seconds)
19 errors, 0 warnings
i just forgot
i added -msse4.1 option to the compiler
i tryed to compile your application on VS2010 and i got the following errors
i’m on a 32bits version
i already installer boost opencv and irrlich
i have pentium dual-core CPU
thanks for you help
BaseThread.obj : error LNK2022: metadata operation failed (8013119F): Existing TypeRef should have a corresponding TypeDef(dummy)，but it doesn’t: (0x01000019)。
1>LINK : fatal error LNK1215: metadata operation failed (8013119F):
Seems to be a boost issue, see if this post helps
http://stackoverflow.com/questions/1589440/linker-error-lnk2022
The error only appreared in the Release version. I revised the codes as http://stackoverflow.com/questions/1589440/linker-error-lnk2022.
There is no error in the compiling. But the release version still can’t run. It shows the dialog:The application failed to initialize properly(0xc000007b)
The debug version can work, but the FPS is only 2-4
Hmm I wonder if there is a mismatch in dll linking. Use dependency walker (http://www.dependencywalker.com/) and confirm the exe is linked to 32bit released libraries only.
Thanks for you reply. I have found the problem.
In the option: Common Language Runtime support, I chose the /clr which is false.
Hello. There is another question for the project. I run the source code in a 64 bit computer. However, it has the LNK 2019 error.
error LNK2019: Unresolved external symbol __imp__createDevice.
Not sure on this one. Maybe you’re not linking to the correct 64bit version of Irrlicht?
I have no 64bit version of Irrlicht. Do you compile it by yourself?
Yep I did.
I am sorry to bother you that here is another question. The processing of compiling and linking are both succuess in 64bit system. But when the project is runing at the code “gui::IGUIFont* font = device->getGUIEnvironment()->getFont(“media/bitstream_font.xml”);” a dialog is pop up:” NAR_Demo.exe 0x… the most possible abnormal 0xc0xC0000005 …” What can I do for it? Thanks a lot
Hmmm, that file is a relative path so if you run in Visual Studio make sure the path is relative to the exe. Else hard code the full path to make sure.
opencv/ opengl /boost /lrrlicht have done.
How to run?(step&command) Thanks~!
ubuntu 14.04 64bits
What have you tried
it’s great to open such technology. if I use haarcascade to detecte front side of hand than I can use this or I have to develope code for this by using contour detection???your help is really greatful for me
Umm not sure what you want to do.
Dear doctor Nghia,
I have installed OpenCV, Boost, and Irrlicht.
And I have compiled the demo with VS2015.
But when I run the NAR_Demo.exe, there occured one problem,
console printed an error line: unsupported texture format, then this program crashed.
VS2015 printed this error line: Could not load mesh, because file could not be opened: :media/sydney.md2, and three threads exited with one same return value -1.
I just run your NAR_Demo.exe and it did work, but my compiled NAR_Demo.exe failed.
Can you help me find out the reasons?
Thx~~
A few moment ago, I found the reason is the version of Irrlicht.dll, and I used the older version in your folder when error occured, then I replaced the Irrlicht.dll with the same dll in irrlicht-1.8.4 folder whose include & lib files I uesd for the project, then I found the error I met before has gone, but every time when I run NAR_Demo.exe the camera window could only live a few seconds, then it would crash with ‘buffer full’ error, I had no idea why this sort problem happened.
Last, I installed VS2010, and re-compiled boost for it, and I also use irrlicht-1.8.4 files, then I made it! It worked perfectly! The fps could reach to 20+, but my cpu fevered and the fan rotated fastly.
Then I will go on studying this project and I hope to gain your help when I meet new problems. Thx~~