Learning Spark

So It Begins

I've spent a few hours learning Spark. An advanced 3D Gaussian Splatting Renderer for THREE.js. As I have mentioned in previous blog posts I think World Models are going to be BIG in the coming years. We are going to need serious engineering in order to make them work well on the web. Spark is the first step in learning how all of this works.

There are some increadible applications already for this. Anything can be scanned with high definition photos/videos to create exceptional 3D models. Check out this post of a wonderfully rendered Bee using PlayCanvas another rendering engine capable of rendering Gaussian Splats. Also we are very close to being able to watch 3D live streaming of sporting events. It's very likely Guassian splats with play a role in this. This is all SOTA and still to be built! Imagine watching your favourite sports team play and being able to get the perspective view of a player on the pitch, that would be pretty cool! It would open up a whole new form of media creation.

Spark

Over the next few weeks I am going to document my understanding of Spark on this blog. Step by step. I have very little understanding of what is going on right now. Thankfully I can read the Typescript code, but it's a massive project with lots of moving parts so even though I can read it I have little idea what each thing does right now. I'm just trying to arrange the jigsaw pieces before I start to put it all together.

Lets start with the entry point, the Spark Hello Word example, and then look at the what Spark is doing at the highest level.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Spark • Hello World</title>
    <style>
      body {
        margin: 0;
      }
    </style>
  </head>

  <body>
    <script type="importmap">
      {
        "imports": {
          "three": "/examples/js/vendor/three/build/three.module.js",
          "@sparkjsdev/spark": "/dist/spark.module.js"
        }
      }
    </script>
    <script type="module">
      import * as THREE from "three";
      import { SplatMesh } from "@sparkjsdev/spark";
      import { getAssetFileURL } from "/examples/js/get-asset-url.js";

      const scene = new THREE.Scene();
      const camera = new THREE.PerspectiveCamera(
        60,
        window.innerWidth / window.innerHeight,
        0.1,
        1000
      );
      const renderer = new THREE.WebGLRenderer();
      renderer.setSize(window.innerWidth, window.innerHeight);
      document.body.appendChild(renderer.domElement);

      const splatURL = await getAssetFileURL("butterfly.spz");
      const butterfly = new SplatMesh({ url: splatURL });
      butterfly.quaternion.set(1, 0, 0, 0);
      butterfly.position.set(0, 0, -3);
      scene.add(butterfly);

      let lastTime;
      renderer.setAnimationLoop(function animate(time) {
        const deltaTime = time - (lastTime || time);
        lastTime = time;
        renderer.render(scene, camera);
        butterfly.rotation.y += deltaTime / 2500;
      });
    </script>
  </body>
</html>

As we can see below Spark is built on top of THREE.js. We must first create a Scene, Camera and Renderer to render our splats.

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
  60,
  window.innerWidth / window.innerHeight,
  0.1,
  1000
);
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

Next we have a simple internal function that takes one of the example 3DGS splat files from the project assets and returns the URL.

const splatURL = await getAssetFileURL("butterfly.spz");

Next we pass this URL to a new instantiation of SplatMesh. This is where most of the magic happens. This creates an object, in this case a butterfly made of splats, that we can then render inside of the THREE environment we have created.

const butterfly = new SplatMesh({ url: splatURL });
butterfly.quaternion.set(1, 0, 0, 0);
butterfly.position.set(0, 0, -3);
scene.add(butterfly);

The butterfly.quaternion.set(1, 0, 0, 0); sets the rotation to identity (no rotation), and then we position the butterfly at coordinates (0, 0, -3) which places it 3 units back from the camera along the Z-axis, and then we add it to the scene.

What's Happening (Super High Level)

At the highest level what is actually happening:

THREE environment Setup: The scene (the 3D space), the camera (how we view the scene), the renderer (how the 3D scene is converted to 2D for us to view)
Import Splats: We import a compressed file of splats and unzip.
Convert file splats to scene splats: We need to convert the information about each splat from the splat file into a data type suitable for our system.
THREE Objects Take our splats and make a THREE compatible object out of them to put in our scene.
Placing in Scene: For each scene object, take each splat in the object and place it in the scene.
Rendering: Rendering process to figure out how to render each pixel on the screen given the splats that a ray is passing through.