Suppose we have a camera in 3D space and want to know where a ray emanating from a selected pixel intersects with a 3D plane. To solve this you’ll need to know the camera’s intrinsic properties (eg. focal length). It’s handy if you have the commonly used 3×3 camera intrinsic matrix available(usually from calibration), which allows you to convert from image points to normalised points as follows:
where, K is the 3×3 intrinsic matrix and p=[x, y, 1] is the image point, expressed as a homogeneous vector. The normalisation does the following:
- moves the origin (0,0) to the centre of the image (principal point)
- focal length becomes 1 unit
Using normalised image points, our problem can be illustrated as shown below, where 
A ray coming from the camera can be described by a line vector. The line vector is just a vector from the origin (0,0,0) to the normalised image point in 3D space, the vector is just 
Note: 
Now a 3D plane in space (of infinite size) can be described by the plane equation:
Ax + By + Cz + D = 0
Combining the line and plane equation and solving for t gives:
Substitute t back into Ray(t) to find the 3D point of intersection.
I have an augmented square in space. How do I use the square with it being in the camera image space and all of its features distorted?
Use it for?
I calculated the intrinsic matrix using openCV’s chessboard algorithm. I assume this is K you are talking about. How do I actually multiply the points(P) – which are 4 – with K? Should I store them how?
You can multiply K by P by converting P to a vector of 3 (remove the last element).
I need to know (3) items that your great blog brings to mind. First, does your line require a 90 degree incidence with the plane? Would like to know intersection point of line from camera but with non-90 angle of incidence. Second, following up on term, I too calibrated the square with an output of 5 distortion coefficients. Leaving them out vastly – making them all 0’s – improved my project points in openCV. I chose a very low distortion lens – would this contribute? That doesn’t seem quite right. Finally, manipulating the plane in openCV requires 4 + points yet you are somehow getting away with (3)! 🙂 I know its the calculus – but – do you have any trades offs to using it with only 3 points.
Thanks for your contribution to this area!
The line can be at any angle. What’s the context of your 2nd and 3rd question? It doesn’t seem related to this article.
Sorry I have returned to the blog so late, I don’t think my email is giving me proper notification. I have self-answered item 2. With regard to question 3 does opencv provide the groundwork to working towards your same solution here? I have successfully implemented the solve PnP solution which successfully produces camera pose and world coordinate generation but requires 4 coplanar points rather than 3 and no ray targeting in doing so.
Also how did you define your plane’s corner points for 3D solution? Did you use a x,y model with z = 0 for example?
Best
In this article the plane is assumed to be infinitely large when using the Ax + By + Cz + D = 0 equation, so there are no finite corners.
Now a 3D plane in space (of infinite size) can be described by the plane equation:
Ax + By + Cy + D = 0
Did you mean Cz?
Yes! Good catch.
It didn’t post all of my math : (
try again:
My line vector is: +t
Distribute t: +
Make parametric eqs:
x = 0+txi, y = 0+tyi, z = 0+t1
x = txi, y = tyi, z =t1
My plane is a special case(Cz = 0 where A & B = 0)
How do I get to t = -D/Axi(=0)Byi(=0)Cz
t = -D/Cz
It seems I have two unknowns?
No luck. I did use the parametric forms of the line origin 0,0,0 to (xi, yi, 1) and equation 0x + 0y + -550000z = 0
I would argue this is not a valid plane, since A and B are both zero (normal vector of zero).
https://www.math24.net/plane/
A = B = 0
see Special cases of this equation of a plane
#3
Good point but maybe instead of non-valid but special case.
Finding the plane equation with 3 planar points that make 2 vectors: both in the Z-plane result in – normals from orthogonal line to plane. Can post example –
Ah no you’re right. I made a mistake, I was thinking 2D plane in my head. A = B = 0 is fine.
from normal line must be valid to justify plane validity.
Ok, so I am normalizing image points by K-1(inverse) from camera calibration. How do I modify image point (Xi , Yi)
to vector so I can multiply by inverse K to normalize? Use (Xi, Yi, 1)?
That’s right.
Can you please comment on your normalization procedure to get your results (xi, yi, 1);
My hardware is rendering several image points but when I multiply the inverse of K by a raw image point I am getting crazy answers for normalization. I think its something simple.
There’s not much to it other than inv(K) * [x; p; 1], where (x,y) are the raw image co-ordinates.
Input Raw Image Testpoints x, y, z:
[50, 60, 1]
invK:[0.002095520423466527, 0, -0.8025476036257788;
0, 0.002096599465309339, -0.4776608362190569;
0, 0, 1]
Normalized coordinates = invK *[50, 60, 1]
Pnorm = [0.1047760211733263, 0.1257959679185603, -67.78703035443236]
octave:4> invK*[50;60;1]
ans =
-0.69777
-0.35186
1.00000
Awesome, thank you! That got it after I looked closely at your response. What units are the normalized points in? Shouldn’t they be multiplied by 1000 (guessing here)?
Confirm same result:
Input Raw Image Testpoints x, y, z:
[50;
60;
1]
invK:[0.002095520423466527, 0, -0.8025476036257788;
0, 0.002096599465309339, -0.4776608362190569;
0, 0, 1]
Normalized coordinates = invK *[50;
60;
1]
Pnorm = [-0.6977715824524525;
-0.3518648683004966;
1]
The units are dimensionless. They are relative to a focal length of 1 unit.
Thank you for your help so far. I’ve gotten to the end of this. But I am not understanding correctly:
“Substitute t back into Ray(t) to find the 3D point of intersection”. Take t = -D/ Axi + Byi + C and put it where on the Ray(t) equation?
The function Ray(t) is stated in the article.
When do we use normalized coordinates? Do we use them with raw, then line normal to plane, for vectors used in vector cross and in other functions utilizing raw image point data?
Various computer vision problems make use of normalized coordinates, this is one of them. Just depends on the problems.
How early in the program should I use the normalization of the raw point(s)?
For this algorithm, from the beginning. The raw points are not used later on.
Ok, so it turns out my image plane and model plane are the same z = 1, right on top of each other. Should I move my model plane back to gain some space for the ray? Also how do you back convert the equation intersect point Pnorm(xi, yi, 1) to equal image coordinates? pNorm * K?
The math should still works out if the image/model planes are on top of each other. To convert normalize back to pixel space do K * pNorm.
Hi,
I just stumbled on the rest of your site. It is clear you like teaching. Had I known of the rest of your site I probably wouldn’t have had so many on this one blog : ) I am going to redirect some questions in the Q & A section. Thx
I’ve been returning to your articles frequently(sorry not more consistently). I have a simple question:
How do I solve for D in your equation
t = -D/(Axi+Byi+Cz)
D is known before hand, from the plane equation you supply.
Very well. So in the following plane equation:
n.p = 0
+ = 0
a(x-x1) +b(y-y1) + c(z-z1) = 0
ax + by + cz -ax1-by1-cz1 = 0
now we group for d?
d = -(ax1+by1+cz1)
also now ax + by + cz +d = 0
is that correct?
is it critical the plane point I use for x1, y1, z1?
Thanks for your time
Looks reasonable. It shouldn’t matter what points you pick for the plane.
Hello,
Thanks for this great article.
I stumbled on this why struggling to successfully implement Inverse Perspective Mapping from a 2D Image.
I can calculate inv(K) * (u,v,1). If I want to assume all my pixel points lie on the ground plane and the ground plane is flat (z=0), Is this a valid equation?
Because all a,b,d = 0 in this and c=1. This makes t=0 according to your equation. Can you kindly help how to solve this type of flat ground plane assumption? Thank you so much in advance.
The coefficient A, B, C represents the [x, y, z] normal of the plane. So if the ground is flat at z=0, the normal is [0, 0, 1]. However be careful, the ground plane has to be relative to the camera’s co-ordinate system. The camera has +y into the ground and -y to the sky. So the plane is really at [0, -1, 0], assuming the camera is looking forward (no tilt). The coefficient D is the height relative to the camera. I might make a tutorial detailing this specific scenario, thanks for the idea.
Does your method retain accuracy when the image plane has a projected plane that is distorted by perspective effects like when the plane on the image is rotated a bit.
I’ll need to see a diagram of this scenario to say for sure.
https://ibb.co/k82yjCD
If you know the plane equation for your image then it should work as described.
That is if the plane equation is based on the undistorted plane instead of image points by themselves. For example the plane derived from solvePnP with correspondences to a model. Get the plane eq from the model. Let me know if I got this wrong.
So if that is right how do I compute the intersecting line to plane from the principle axis of the camera? You have a lot of great info on creating the line but I don’t see it as a ray from camera coordinates/camera pose.
With solvePnP you can find the 3D points of your plane, relative to the camera. Using these 3D points you can then find the plane equation.
Excellent my thoughts exactly. z = 0.
How do I then create a ray emanating from the camera?
First equation on the page. Pnorm = inv(K)*P, where P = [x, y, 1].
ooops I meant z = 1 after normalization
Some of my Pnorm values are (-). What does this mean?
eg, [x,y,z]:
[0.383621976759394, -0.2925051609396598, 1;]
[ -0.5374948605305616, -0.3194088054650904,1];
They’re just normalized co-ordinates. Refer to the co-ordinate axis in the the diagram in the post.
Hopefully these may be last questions. You have been very patient with this, thank you. : ) : )
I reviewed the axis in post diagram.
What do the normalized points refer to in the image plane — pixel space?
It’s in normalized space with (0,0,1) at the center of the image plane. So x left is negative, x right is positive. The normalization when you multiply by inv(K) is really x’ = (x – center_x)/focal_x.