last day (15 days later) » 

9:20 PM
hey
 
Hello
 
How are you Nicolas?
Thanks for your time
 
Im good ;) yea no pb
 
So from the start\
I know my camera calibration is great
 
ok so this is a good start
 
9:22 PM
as I mentioned when using empty rotation / translation
I can directly compare the results with the ones from Opennis camera to world coordinates
and they are or less the same
now with as far the projector goes,
to make a little test , I decided to do a solvePnP with the points I was feeding
and use those rvecs and tvec to transform 3d points to projector 2d points
which also works better than I expected, for the taught depth, all projector surface is corresponding quite nicely
But for the love of me the stereo calibration fails quite much
first of all, it actually changes the camera matrix and coeffs of both the projector and cameras, which I can so not explain,
 
so wait about this solvePnP
 
I use projector coordinates, and 3d world coordinates which I get from the camera
 
let me understand clearly what you are doing exactly
 
after calibrating the projector insintrics
I use the projector coordinates of the 4 edge circles,
 
how do you know which projector coordinate correspond to the 3D world coordinates you get from the camera, is it because the projector is projecting a chessboard ?
 
9:27 PM
I calculate from the camera the 3d coordinates of those 4 points
its projecting a circle pattern
 
so wait, starting again
you do the camera calibration and you checked that it worked well
 
yeah
then I project a circle pattern
 
then you calibrated the projector intrinsics, but using a single view
using the circle pattern ok
 
I read with circle pattern on the camera, and get their 3d coordinates
for 0 depth, with it being at the same depth as the physical chessboard
 
ok how do you detect the circles ? you have a custom function for that ?
 
9:30 PM
opencv detect circle grid
 
ok
right ?
and you project the circles at the same depth than you projected the chessboard
 
so with single view, after calibrating the projector
 
which with your conventions you call depth 0
 
exactly
 
9:32 PM
I have assumed that the chess corner is the 0,0,0 for all sensors
 
so wait, here it is not clear
it is not "for all sensors"
 
I can have more than one sensors, but only a single I want to calibrate with the projector.
 
you defined a coordinate frame whose origin is on the top left corner and whose Z axis is perpendicular to the wall
 
yes
that z distance of course from the camera is known
 
let's call this one the "wall" coordinate frame, but you also have two other coordinates frames, one centered on the projector, one centered on the camera
 
9:34 PM
but not used at the moment
yes
 
now, if I use the wall coordinate system,
with 0 depth,
everything seems to be quite great,
and even when doing the stereo calibration
the tvec I get seems really logical
the tvec camera projector that is\
 
that you get from solvePjnP ?
 
that I get from the Stereo calibration
 
9:37 PM
the SolvePnP is just additional rvecs and tvec , lets call them camera to wall
 
ok wait i don't get this
 
projector to wall I meant ***
 
you were talking about the projector intrinsic calibration no ?
 
yes
 
so why now you mention stereo calibration ?
 
9:39 PM
after I get the projector insintrics, I procced in doing stereo calibration between camera and projector or ?
 
ok, so this is the next step
but just to finish with the projector intrinsics calibration
you project a single circle grid, detect it with the camera, reproject it in 3D and then you run calibrateCamera with one pair of corresponding 2D/3D points to get the cameraMatrix and distCoeffs for the projector
right ?
any idea if this is accurate ?
 
on the wall depth it is yeah
 
I mean if those cameraMatrix and distCoefs are accurate ?
 
if they were not, I would assume that I wouldln't be able to use the 3dcoordinates afterwards
that I get from the camera and project with the projector on this exact position
by using project points that is
 
and the rvec & tvec for the camera you get it from a solvePnP before right ?
 
9:43 PM
yes
 
I tried also for the camera, to use the camera as point of reference
translate 0,0,0 rvecs 1,0, 0 etc etc
but then I had to provide depth for calculating the 3d points, which kind fucked the resukts
 
so let me understand this
 
maybe its better if I write down the steps in sequence:
 
9:46 PM
1) I use a chess board and detect it over 100 frames, while moving the camera ( or chessboard) around
2) I use solvepnp and get a rvecs and tvec that corresponds to the top left corner of the chess board.
3) I project a circle pattern from the projector and detected it on the camera.
4) I collect set of points. The 2d points are the ones collected by the projector. The 3d points are those I get from the camera, in relation to the top left corner of the chessboard. Eg if the first circle was directly on the top left corner, the first circle would be at 0,0,0 the the right on on 10,0,0 etc
the 10 Is the actually real world distance in mms
as good as the camera can read that ofcourse
After a few set of points I use those points to get the projector intrinsics
 
wait
for 4) ho do you collect the 2D points for the projector, using detectCircleGrid ?
 
Yes, with detect grid, I get the position of the circlers in the camera
 
in the camera ok, but in the projector ?
 
For the projector I still detect those points in the camera, and then convert them to the real world coordinates
So the 3d points of the projected circlers are actually the real world coordinates if we measure them from the top left corner
from the point of origin lets call it
 
but those 3D points you measure them manually with a ruler or you get them by backprojecting the 3D points that you detect in the camera ?
 
9:54 PM
I back project the 3d points
and also confirmed it by using a ruler
 
ok, so to get the 3D points in your step 4), you first detect the circles in the camera, then undistort them using the cameraMatrix and distCoefs of the camera, then apply the inverse rvec and tvec computed in step 2)
 
exaclty
 
also before applying the inverse rvec & tvec you need to use the depth you get from the depth camera, but ok
and now for the 2D points for your step 4), how do you get them ,
?
 
the 2d are the position of the circlers on the projector window
 
yes
you get them manually ?
 
9:57 PM
I manually generate the circle pattern
so I have access to their coordinates
 
Ah ok I see now
and last question
where is the chessboard when you do this, is it still hanged on the wall ?
 
Yes, but at this point I don't really use it
 
ok
now next step
 
so I have the 2d and 3d points
the 3d points always to relation of the top left corner
always with depth 0 since the cameras rvecs and tvec and also taught on that depth
 
10:01 PM
and I have confirmed with a ruler that the 3d points are correcrt
 
yes the camera should remain fixed with respect to the wall
ok
 
yes it is
now I do first a solvePnP with the points I have
and I get the rvecs and tvec of the projector, in relation to my point of origin
 
this is strictly in relation to the point of origin, not the camera or anything else
 
so in step 4) you actually generated multiple circle grids, you accumulated the 2D/3D points correspondences and you ran calibrateCamera to get the cameraMatrix and fistCoefs for the projector right ?
 
10:04 PM
yes
 
but you never moved the projector
 
not really, not
 
ok, might be a problem, but let's see the following
 
if I were to move the projector
or use a piece of paper to change the depth
I would have to provide different depth on the 3d points?
and it won't be possible to teach it with a zero depth all of them right:
 
if you used a piece of paper, yes, but if you move the projector and still project on the same wall, no need
actually even with the piece of paper
 
10:07 PM
but even if I didn't project on another depth, just moved the projector, I could still get an accurate calibration of the projector intrintrics right?
 
since you get the depth from your depth camera
yes
 
Ok cool
 
just to finish with the idea of the piece of paper, it is the same
 
So after the solve pvp, I save those rvecs and tvec, as projector to poi matrices
afterwards, I go on and perform the stereo calibration
I am using the same set I used before, 2d of the projector, 3d points and the circles points as detected on camera image
 
you detect the circle grid in the camera, get the depth (which would be closer to what you usually get from the wall), undistort apply the inverse rvec & tvec of the camera, then you will get a 3D point which does not have depth zero but it should still work
anyway
 
10:10 PM
and of course all the intrentrics
and I get the rvecs and tvec for the projector and camera
I mean between them
 
the values I get are actually appropriate, and the translation vector seems to be correct
one question here, for the stereo calibration I need a size of the image
do I use the camera size or the projector size?
until now I am using the camera size
 
I am not quite sure about this
in the opencv docs they say it is the size of the image used to initialize the intrinsics
so you probably don't care too much
 
ok cool
so afterwards, theoretically I should have everything I ever wanted
 
also because you did the intrinsics calibration before, you could call the calibrateStereo with the CALIB_FIX_INTRINSIC flag
yes you should
 
10:14 PM
off
ohhh
I thought the calib fix intrinsics refers to the new intrinsics
not the provided ones
 
the thing I am quite sure about is that to get a good stereo calibration you need to vary the depth at which the circles are projected
actually you should use two flags:
CALIB_FIX_INTRINSIC and CALIB_USE_INTRINSIC_GUESS
 
a cool
 
anyways, I definitely think your problems come from the lack of depth variation for the stereo calibration
 
but the thing is,
hmmm
 
but yes in you case it is tricky
 
10:18 PM
so you suggest I should be using depth values for the projector calibration?
and not only on the wall depth
 
I suggest you find a way to project not always on the same wall
but on a sheet or big paper plane that you move around
 
or another way: you mount the projector and the camera on a rolling thing
and you move both at the same time
 
if I do that, the 3d coordinates I provide, will need to have a depth right?
 
yes
but you could do that as a separate step
like do all as before for the intrinsics of the camera and of the projector
and then use new points for the stereo calibration
 
10:21 PM
but use the projector intrinsics from the wall only calibration
I seee
 
actually the problem I had, was that when providing depth values on the 3d points
the projector calibration gave huge error
 
then one thing is that your rvec and tvec might not be valid anymore if you move your projector+camera thing
 
and shitiv values
 
so its probably better to use a sheet or piece of paper to change the depth
 
10:22 PM
I have separate buttons to teach the rvecs and tvecs only
so I don't need to do a complete calibration to change them
now to finish , for a give point on the wall, I can get its 3d coordinates from the camera
and by using the rvecs and tvec of the projector relative to the point of origin
I can backproject those points
again only on the depth of the wall
but with great precision
 
the problem offourse I have currently, without the stereo calibration , is what to do on different depth
what I thought, is shouldn't it be possible lets stay if the wall is at distance of 500 cm
for a point at 300 to calculate where it would hit the wall at the wall depth and so to generate the projection this way
basically ray tracing the hit point
since I have all the matrices and distcoeffs and even the translations and rvecs between camera and projector?
 
well not really
because of parallax
 
ohh
 
the projection of the point on the wall will not project at the same place than the point a bit before the wall
 
10:30 PM
Yes , but if you find the projector point 0,0,0
you could recast it with the second point right?
 
you could probably compute the effect of the parallax and simulate it to estimate a correction to be applied to the projection of the wall point
 
rey cast it meant
 
didn't get that last comment
 
so my question is, how to find the 0,0,0 of the projector relative to the camera?
for a point in space lets say 10,10,500
if I knew the one more point ( which is the projector itself relative to the camera)
 
?
so you mean the origin of the coordinate frame of the projector
 
10:33 PM
yes
 
well if you know the rvec and tvec it is pretty easy
you get the rotation matrix which correspond to rvec via the opencv function rodrigues()
 
that's the rvecs and tvec of the projector relative to the camera I get from the stereo calibration right :?
 
yes
you need to be careful which direction they map, i.e. if (rvec, tvec) map from the projector to the camera or the opposite
 
its from camera to projector I do currently
but yeah true
so if I had those rvecs and tvec,
how could I actually calculate the 0,0,0 of the projector in camera coordinates?
 
if (rvec, tvec) map from camera to projector, then tvec is actually the position of the camera in the coordinate frame of the projector, and -R^T.tvec is the position of the projector in the camera coordinate frame
where R = rodrigues(rvec)
and R^T is the transpose of R
OK, it is getting late I really have to go
 
10:39 PM
Ok
thanks a lot!
 
sure
let me know if you work this out
 
you really helped me alot
ofcourse
have a nice evening!
 
you too bye
 

last day (15 days later) »