Point cloud streaming is nothing new, but certain requirements must be met. In my case, I was facing the issue of having very slow internet speed (max. 0.4 Mbits/sec of sluggish upload speed, to be exact).

Problem

At first, my friend and I formed a TCP connection on a P2P private network. We both live in somewhat close vicinity, so the ping was fine and steadying around 32ms. We decided to give it a try with the most straightforward approach, which is directly streaming point cloud data (3D vertices + UV map + Color Texture), frame by frame. This would yield us around ~7 MB of point cloud data, per frame. Hence, a real-time scenario with 30 FPS streaming would end up generating 210 MB/sec, which is totally infeasible. When I applied compression, I was able to get ~2 MB per frame, which was far better but still insufficient since I was able to upload a maximum of 0.05 MB/sec (0.4 Mbits/sec), not to mention downloading other client’s stream.

Although I had a few suspicions on the low-level data processing (I could be capturing and sending redundant data), I reminded myself that streaming 3D mesh data over the network was no simple task. Regarding my slow internet speed, it probably had to be a highly lossy 3D compression algorithm, but I wanted to avoid that path. So, I decided to go with a very simple yet effective approach.

Solution

There are two clients A and B in a scene that is identical to each other and here we assume that only B demands the point cloud of A.

1. B tells its position to A.
2. A places a camera in that received position of B.
3. A rotates the camera toward its own point cloud image.
4. A takes a snapshot/screenshot with the field of view (FOV) automatically narrowed/expanded to keep the whole point cloud in the view frustum.
5. A sends the cropped & compressed image over the network to B.
6. B receives the image, decompresses & “uncrops”, renders on a canvas (positioned at A’s position) in the scene rotated toward B’s position.

TLDR; It is a projection-based approach that degrades the point-cloud streaming problem to the image/video streaming problem.

Results

(the delays between snapshots are deliberately set)

Tools

• Unity 2019.3.1f1 64-bit
• Realsense SR300 fw: 3.21.0.0, sdk: 2016 R2
• Point cloud pipeline based on: librealsense/wrappers/unity/ .. /PointCloudDepthAndColor.unity
• LogMeIn Hamachi v2.2.0.633 (yes, that is correct)