Description
I am trying to get the 3D position of a mouse click inside of the LumaSplats mesh.
This is part of the code that I have written:
function getClicked3DPoint(evt: any) {
// evt.preventDefault();
const canvasPosition = canvasRef.current?.getBoundingClientRect();
console.log('WTF', scene.children)
mousePosition.x = ((evt.clientX - canvasPosition!.left) / canvasPosition!.width) * 2 - 1;
mousePosition.y = -((evt.clientY - canvasPosition!.top) / canvasPosition!.height) * 2 + 1;
rayCaster.setFromCamera(mousePosition, camera); //cameraRef.current!);
let intersects = rayCaster.intersectObjects(scene.children, true);
if (intersects.length > 0)
console.log(intersects[0].point)
return intersects[0].point;
}
<Canvas
ref={canvasRef}
frameloop={frameLoop}
gl={{
antialias: false,
}}
camera={{
position: [0, 0, 0],
}}
>
<group
dispose={null}>
<lumaSplats
onPointerMissed={() => console.log("you clicked, but hit nothing at all")}
onClick={(e) => {
console.log(e.point) // this is the coordinates for the event, eg. mouse click
getClicked3DPoint(e)
}}
onLoad={() => {
setIsLoading(false);
if (camera360Ref.current) {
camera360Ref.current.object.position.x = -5;
camera360Ref.current.object.position.y = 0.4;
camera360Ref.current.object.position.z = 7;
}
}}
loadingAnimationEnabled={true}
semanticsMask={LumaSplatsSemantics.ALL}
// onBeforeRender={}
enableThreeShaderIntegration={false}
particleRevealEnabled={true}
source={src}
position={[0, 0, 0]}
scale={5}
rotation={[0, 0, 0]}
/>
<CustomLine start={[0,0,0]} end={[1,1,0]} />
</group>
</Canvas>
I have created the following luma capture: https://lumalabs.ai/capture/71f64429-3872-417a-a67c-a8b6c6ab16d3
As you can see it is a capture of a normal room. What I want to achieve is to draw lines on the splat. For this, I want to click on the canvas, this will cast a ray to the first intersecting part of the splat. I will take the coordinates of this intersection and then by clicking on another part of the splat I want to draw a line between these two points.
However, most of the times the event on onPointerMissed is being activated and thus no intersecting point is found. Do you have any idea why this is happening and what I can do in order to fix this?
Activity
abrahamezzeddine commentedon Jul 5, 2024
Try intersecting the point cloud itself rather than the splat and see how that works.
Limonka11 commentedon Jul 8, 2024
Hi @abrahamezzeddine, How can I get the point cloud?
abrahamezzeddine commentedon Jul 15, 2024
Each Gaussian splat file contains XYZ coordinates for the center coordinates of the splat. Essentially, a splat file is a point cloud where each point is stretched in different directions.
You do not need to render the point cloud, it is enough that you try to “hit” the XYZ.
Perhaps you also need to take into consideration that the points are too small for your clicks. Perhaps increase size of each point.
Limonka11 commentedon Jul 16, 2024
Yes, I am indeed trying to hit the XYZ coordinates of a point using ray-casting
This is what this function is supposed to do:
` function getClicked3DPoint(evt: any) {
// evt.preventDefault();
const canvasPosition = canvasRef.current?.getBoundingClientRect();
console.log('WTF', scene.children)
}`
However, most of the time it doesn't hit anything and sometimes it hits something that appears to be in the centre of the room in the air. I thought that maybe Luma's software is suitable only for objects and tries to represent the room as an object in the centre, not as an open space.
Do you have any suggestions? Does the function look correct to you?
abrahamezzeddine commentedon Jul 16, 2024
I do not know exactly how lunas software works, I just happened to pass by here.
I tried this with Mkkellogs splat viewer. I first tried to intersect the splat itself which did not work (since splats are “invisible” in three.js when raycasting) and then I had to try with the xyz points which did not work so good.
I had to increase the size of the point, xyz only, in order to be able to raytrace the points. I do not have that code at hand at the moment however.
It can be that your points are very very small that you will not able to hit them. That is why I suggested you try to increase the point size, or try to load the point cloud concurrently, only fetching the xyz since that is what you need.