Improved system takes volumetric video to a new level for live sports production and more
The Canon Free Viewpoint volumetric capture system takes the next step at tonight’s NBA All-Star Game. More than 100 Canon Cinema EOS cameras and lenses in Cleveland’s Rocket Mortgage FieldHouse will be used to give replays a whole new look.
The Canon Free Viewpoint System allows an operator to control a virtual camera through volumetric video recording of tonight’s NBA All-Star Game.
“It took a long time,” says Steve Hellmuth, executive vice president of media operations and technology, NBAand noted that discussions with Canon have begun at NAB in 2019. “But Canon was great, asking us for advice and keeping us updated on the progress. We kept in touch and Canon decided to provide us with two systems.” One is installed in the Barclays Center in Brooklyn, the other in Cleveland; both will remain in place until the end of the season.
From left: Brent Valenti of Bally Sports Ohio, Steve Hellmuth of the NBA and Tsuyoshi Wakazono of Canon at the NBA All-Star Game
“We decided to install it [at Rocket Mortgage Fieldhouse] because of the All-Star Game and also because the Cavaliers are a great outfit to introduce new technology,” he says. “They gave us the rack space in the building and also arranged for the installation of the cameras around the perimeter of the arena.”
Tonight’s use will highlight new types of replays: A virtual camera can fly around and between players, going pretty much anywhere in the building.
“This allows us to fly into the arena,” notes Hellmuth. “It will be like having six drones in the building without any restrictions and any liability.” He believes that all leagues should look at volumetric capture, as it offers multiple benefits that go well beyond cool replays.
“Eventually,” he adds, “you get player tracking and player positioning, and with AI you can generate stats and also create stats that don’t exist today. At the moment, for example, deciding whether a player is defending a shot is a manual process, but with this can you write in the AI [that, if the hand is here relative to the ball,] it’s a defended shot.”
The cameras in the system capture the video information, which is processed on Canon’s proprietary imaging cabinets behind each camera and converted into data. The data from all the cameras is then sent to a server system that creates a point cloud 3D model of the players and other objects on the field. Images are created using voxels, which are units of graphical information that define a point in three-dimensional space (basically the 3D equivalent of a pixel) combined with RGB values. An operator with a joystick resembling a wired camera system flies the virtual camera within the virtual environment and builds the clips in a process similar to rendering an animation.
The system made its public debut at the 2019 Rugby World Cup, where 125 cameras were used, but three years of innovation will result in a more robust production experience.
During the Rugby World Cup, says Tsuyoshi Wakazono, Manager, Image Solutions Business Operations, Canon SV Business Planning Department, making the replay last about 15 minutes. “Now we’re integrated into the live broadcast so we can quickly flip a clip in about 10 seconds and also continue streaming so that other highlights can be created.”
The uses for volumetric capture are many: in addition to broadcasting and social media applications, uses range from video games to VR to on-duty, training and coaching. “It creates a perfect VR environment with no convergence issues and allows fans to take a seat in a VR environment, so it promises to be a great opportunity,” said Hellmuth. “All of our video gaming partners visit us to create the ideal video game dataset because they can get a perfect model of the players.”
He notes that the benefits extend to officials, teams, coaches and players. For example, officials can watch plays from any angle and zoom in from any angle, solving one of the major problems of relying on camera angles framed to include multiple players. The virtual environment will also not have the stakes supporting the hoop and backboard, giving the umpires a better view of the game under the basket.
“We can now place the virtual camera directly behind the basket and look straight at the field because the upright is gone,” Hellmuth added.
The system does require a lot of storage space, about 60 TB per game. But continued improvements in storage, the use of edge computing and even 5G could make it easier to move the massive amounts of data.
“The amazing thing is that Canon has figured out how to do all of that in just three seconds,” says Hellmuth. “It’s a unique partnership because Canon understands lenses and image processing and how to translate this to digital.”
Tonight’s efforts will be part of the Turner Sports production, the world feed, the NBA’s social media channels and more. “It’s been received worldwide,” he adds, “and the fact that I’ve had a few evening calls with China this week shows the interest.”
Brent Valenti, senior producer, Bally Sports Ohio, is looking forward to putting the system to the test for the rest of the season. “We will play with it more and it will help with our analysis.”