Camera Tracking System For Live Composting

Abstract

A 3D camera tracking and live compositing system includes software and hardware integration and allows users to create, in conjunction with existing programs, live composite video. A video camera, a tracking sensor, encoder, a composite monitor, and a software engine and plugin receive video and data from and integrate it with existing programs to generate real time composite video. The composite feed can be viewed and manipulated by users while filming. Features include 3D masking, depth layering, teleporting, axis locking, motion scaling, and freeze tracking. A storyboarding archive can be used to quickly load scenes with the location, lighting setups, lens profiles and other settings associated with a saved a photo. The video camera's movements can be recorded with video to be later applied to other 3D digital assets in post-production. The system also allows users to load scenes based on a 3D data set created with LIDAR.

What is claimed is: 
     
         1 . A camera tracking and live compositing system, comprising:
 a video camera having a lens and generating a video camera feed;   a tracking device attached to the video camera;   an encoder attached to the lens, wherein the encoder is configured to track a plurality of lens parameters of the lens, including a focal length, an aperture, and a focal distance;   a composite monitor; and   a processor configured to receive the video camera feed, positional tracking data from the tracking device, and the plurality of lens parameters from the encoder and to output a composited video feed to the composite monitor in real time,   wherein the composite video feed includes the video camera feed and virtual elements from a virtual camera, wherein the virtual elements from the virtual camera include at least one object that is positioned within a scene of the composite video based on video and tracking offsets derived from positional tracking data from the tracking device, wherein the virtual elements are constructed using virtual lens distortion parameters derived from the plurality of lens parameters, and   wherein the virtual elements are scaled based on a geometry scale derived from the positional tracking data, the video camera feed, and the plurality of lens parameters.   
     
     
         2 . The camera tracking and live compositing system of  claim 1 , wherein the tracking device includes a housing and the housing is attached to the video camera. 
     
     
         3 . The camera tracking and live compositing system of  claim 2 , wherein the housing includes a camera mount. 
     
     
         4 . The camera tracking and live compositing system of  claim 3 , wherein the tracking device includes stereo cameras, and wherein the stereo cameras generate a stereo camera feed. 
     
     
         5 . The camera tracking and live compositing system of  claim 4 , wherein the tracking device includes a video capture device to receive video input from the video camera, port for connecting to the processor, an ethernet connector, an SDI in port, an SDI out port, a board, an SDI cable, and a cooler. 
     
     
         6 . The camera tracking and live compositing system of  claim 5 , wherein the processor generates an intrinsic calibration file based on the plurality of lens parameters and the video camera feed, wherein the processor generates an extrinsic calibration file based on the stereo camera feed and the video camera feed, and wherein the processor generates a room scale calibration file based on the plurality of lens parameters, the positional tracking data, and the video camera feed. 
     
     
         7 . The camera tracking and live composting system of  claim 6 , wherein the processor determines configuration settings of the virtual camera based on information from the intrinsic calibration file, the extrinsic calibration file, and the room scale calibration file. 
     
     
         8 . The camera tracking and live composting system of  claim 7 , wherein the configuration settings include a virtual camera aperture and a virtual camera focus distance. 
     
     
         9 . The camera tracking and live composting system of  claim 1 , wherein the processor is further configured to receive depth data from the tracking device, generate a depth matte based at least on the depth data, and generate a refined matte based at least on the depth matte. 
     
     
         10 . The camera tracking and live composting system of  claim 9 , wherein the processor is configured to load and execute a trained machine model to generate the refined matte based at least on the depth matte and at least one matte of known quality. 
     
     
         11 . The camera and live composting system of  claim 9 , wherein the video camera feed includes a physical object and wherein the processor generates, based at least on the refined matte, the composited video feed that includes images in which the physical object is at least partially occluded by a virtual object. 
     
     
         12 . The camera tracking and live compositing system of  claim 1 , wherein the composite video feed is transmitted to the composite monitor by wireless transmission. 
     
     
         13 . The camera tracking and live compositing system of  claim 1 , wherein the video camera includes a video frame rate and the tracking device includes a tracker frame rate, and wherein the video frame rate and the tracker frame rate are synchronized to a server system clock. 
     
     
         14 . The camera tracking and live compositing system of  claim 1 , wherein the encoder further includes a lens gear connected to a follow focus. 
     
     
         15 . The camera tracking and live compositing system of  claim 1 , further including a recorder for recording the composite video feed in real time. 
     
     
         16 . The camera tracking and live compositing system of  claim 1 , further including a memory connected to the processor, wherein the memory stores an image captured by the video camera that are associated with information about the image including scene state data, the plurality of lens parameters, a plurality of video camera parameters, and metadata. 
     
     
         17 . The camera tracking and live compositing system of  claim 16 , wherein the scene state data includes geometry, lighting information, and project settings, wherein the plurality of lens parameters include focal length and distortion, wherein the plurality of video camera parameters include camera model and settings, and wherein the metadata includes a date, and a location of the image capture. 
     
     
         18 . A method of camera tracking and compositing comprising:
 receiving a video feed from a video camera;   receiving positional data of the video camera from a tracking device attached to the video camera;   receiving a plurality of lens parameters from an encoder connected to a lens of the video camera;   generating an intrinsic calibration file based on the plurality of lens parameters and the video feed;   generating an extrinsic calibration file based on the positional data and the video feed;   generating a room scale calibration file based on the plurality of lens parameters, the positional data, and the video feed;   generating a composite video feed from the video feed and a virtual camera, wherein a plurality of lens distortion parameters of the virtual camera are set based on the intrinsic calibration file; and   displaying the composite video in real time.   
     
     
         19 . The method of camera tracking and compositing according to  claim 18 , further including tracking an object in the composite video feed using camera and tracking offsets derived from the extrinsic calibration file without a use of markers. 
     
     
         20 . The method of camera tracking and compositing according to  claim 19 , further including adjusting a geometry scale of the virtual camera based on the room scale calibration file.