Databricks moves signature data storage technology upgrades to open source

Delta Lake technology is a key component of a Lakehouse designed to ensure the quality and reliability of AI-ready data. Databricks initially launched Delta Lake as an open-source software project in 2019. But until now, many of the new features it has added since then have been proprietary, available only to Databricks customers.

Databricks, based in San Francisco, makes money by renting analytics, AI, and other cloud-based software designed to help businesses extract insights from business data. The services are based on open-source Apache Spark, a real-time data analytics technology that emerged in 2009 from the University of California, Berkeley. Open source developers make software available for free, allowing programmers to modify and share the underlying information. source code and create their own apps.

Databrick’s co-founder and Chief Executive Ali Ghodsi said the decision to make the technology available for free is aimed at attracting commercial customers who are wary of being locked into a single data management provider and limited to its use. of its analysis tools.

Mr Ghodsi said more Lakehouses will drive demand for Databricks’ analytics software and other services. The company will also continue to handle security, maintenance and other software issues for customers deploying its tools in their own data lakehouse, he said.

Data kept in conventional data storage systems must be copied, reformatted and moved to a separate repository, where software developers have access to create AI apps. A data lakehouse takes that step out of the process, Mr Ghodsi said. “The Lakehouse combines these two worlds in one place, where you have all your data and where you can apply AI,” he said. “One system, one copy.”

The demand for AI and data analytics is high as tougher economic conditions push more companies to adopt software that promises to make better business decisions or find ways to improve everything from supply chains to customer service.

Ganesh Jayaram, chief information officer at agricultural and construction equipment manufacturer Deere & Co., a customer of Databricks, said Databricks’ open-source move would allow the company to develop its own custom data lakehouse.

“It allows us to scale analysis at a scale that we have not been able to do in the past,” said Mr. Jayaram on data lakehouse technology.

Among other things, the company uses AI-powered data and analytics to support predictive maintenance systems for tractors, backhoes and other vehicles in the field, which are loaded with sensors to collect data. In January, it unveiled a fully autonomous tractor.

But Mr Jayaram said he also plans to increase the use of AI in optimizing supply chains, marketing, finance and other internal business functions.

“When engineers use an open source technology, they can access the source code and create their own modified version,” said Christopher Condo, principal analyst at IT research firm Forrester Inc. how the open source software works.”

Open source software business models have become increasingly popular. It’s estimated that up to 80% of code in new software projects consists of third-party components, most of which are open source, said Mark Driver, vice president and analyst at IT research and consulting firm Gartner. Inc.

The most common way software developers monetize open source tools is through a so-called freemium strategy, said Mr. driver. Under that approach, a technology vendor gives away an app’s underlying code for free and then converts some of its users into paid customers by offering more advanced features and services.

Databricks, a nine-year-old company built in part on open source software, has a private market value of $38 billion. Earlier this year, it reported $800 million in annual recurring revenue for 2021. It has not disclosed any net income.

Before the value of the tech market started to fall this year, Databricks was one of the most anticipated IPOs in the startup market – a move that Mr Ghodsi says is not a priority now. “We don’t have to worry about running out of money for the next ten years,” he said.

write to Angus Lots at angus.loten@wsj.com