- Home /
Best way to handle this in JS (Complex point cloud management)
I have a number of things I wish to relate to each other in particular ways.
1) Point cloud - A list of positions and timestamps, presorted by timestamp and doesn't need to change order.
2) Trajectory - This is spread over several files, the pertinent details of which are the positions, timestamps and error values (Vector3 that will be Abs()'d to an int).
Each of these files can contain millions of entries, 10's of millions of elements.
What I am wanting to do is to find the nearest trajectory point (by time) to each cloud point, and color the cloud based on the error values of the trajectory point it relates to. The display and colorization are easy, but I'm struggling to come up with an efficient and elegant way of handling the error data.
The only methods I can think of to do this are very heavy. My program currently runs quite efficiently, which is it's main source of merit, but any method I can think of to integrate this point "reliability" layer throws reasonable loading times out the window.
Pre-indexing all the information is possible, but I'd much rather avoid it due to the size of the data sets being used (10's of terrabytes) because we only ever access small chunks at a time.
EDIT
I should add, what I'm currently doing:
Load all cloud data (x,y,z,time)
Load all trajectory data (x,y,z,time,error)
function FindErrors(){
var trajTimer : double = 0; var trajNum : int = 0; for (var x = 0;x < points.Count;x++){ //cycle through each point in the cloud based on time while (trajTimer < pointTime[x]){ //move through trajectory timeline until we find a point //that is 1 step past out place in the cloud timeline trajNum++; trajTimer = trajTime[trajNum]; //once we have a trajectory point just passed our cloud point //this loop will end //this will be close enough time-wise for what we are doing } //assign an error value for the point in question //based on the point in the trajectory we found pointError[x] = trajError[trajNum]; } }
DOUBLE EDIT**
One more thing.
In some cases, trajectory data may not be be read in correct order, due to it being split over several files.
If I have one file that contains entries between 1:00:35 - 1:24:16 and a second one 0:52:15 - 1:00:34, what would be the best way to sort these into the correct order and also combine and sort their error values based on how the times are moved?