The role of artificial intelligence in achieving the Sustainable Development Goals
Three Insights You Might Have Missed From TheCUBE's Coverage Of Data + AI Summit
The competition in the data industry is intensifying as Databricks Inc. Vies for dominance. The company has thrown down the gauntlet by open-sourcing its Unity Catalog.
Data giants Databricks and Snowflake Inc. hosted events within a week of each other. Last week's Data + AI Summit showcased Databricks' increasing momentum and emphasis on collaboration. More than 66% of the contributions to Spark are coming from outside Databricks, according to Rob Strechay (pictured, left), principal analyst at theCUBE Research. The company's announcement that it was open sourcing Unity, a unified governance solution for data and artificial intelligence, follows news of Snowflake's upcoming release of Polaris, an open-source Iceberg catalog. These developments suggest that the competition grounds have moved from data formats to data catalogs.
TheCUBE analysts Rob Strechay, George Gilbert and John Furrier discuss insights from the Data + AI Summit.
"Databricks is saying, 'Forget the data format now. We got that covered. You won't have to care. The new platform is the catalog,'" George Gilbert (middle), principal analyst for data and AI at theCUBE, said during an analysis segment at the Data + AI Summit. "Microsoft is kind of weak there, and the Snowflake catalog is tied to their compute engine. So they're trying to get everyone to plug in, because once you plug in all your tools, the functionality of your tools depends on what's in the catalog."
Strechay and Gilbert were joined by John Furrier (right), theCUBE Research executive analyst, in one of several analyst segments at the Data + AI Summit 2024. They discussed how Databricks is going on the offensive with a strategy based on data interoperability and integrating generative AI throughout its platform.
Here's the complete video of theCUBE's Data + AI Summit "AnalystANGLE":
1. Unity Catalog could give Databricks a competitive advantage against Snowflake.Unlike Snowflake, which ensures all data stays inside its platform, Databricks has never owned the data and is now pursuing an open format strategy that could put Snowflake on the back foot and move the point of control from the database to the catalog.
"Databricks went from being five years behind in DBMS technology to being many years ahead in catalog technology," said Gilbert in a keynote analysis of the summit's first day. "Where Snowflake open-sourced the Polaris Catalog for a governance tool for Iceberg tables, it's just Iceberg and it's just technical metadata — whereas Unity is all your data assets even beyond Delta. Basically, they changed the playing field [from] something that favored Snowflake to something that favored Databricks."
Databricks recently acquired Tabular Technologies Inc. To accelerate interoperability between Delta Lake and Iceberg, two leading data formats. Gilbert compares this move to the first level in Maslow's hierarchy of needs, with rationalizing the two formats as food and shelter, then moving up to Unity, which manages data quality and lineage and, eventually, semantic harmonization between the different types of data as self-actualization.
TheCUBE analysts Savannah Peterson and John Furrier talk about how Databricks is transforming data governance.
"This is the evolution of Databricks. Their track record from day one with Spark was constantly popping on that next lily pad and getting to the next innovation," said Furrier, who has been following the company since early days. "You're seeing massive tooling. You're seeing the Delta Lake uniform, Unity Catalog got governance … Ali Ghodsi really wants to push the democratization. How he's doing that is by forcing the standard of table interoperability, data formats."
As the complete technical metadata operational catalog, Unity builds on Databricks' relationship with its user base. By creating interoperability across different data formats and sources, the company is hoping to better serve its customers' data engineering teams.
"They own data science; they own those guys. That's their key persona that they've been huge with with AI in general," said Strechay in a day 2 keynote analysis. "Now they're looking at it with … Delta Lake and what they've been doing there and SQL being kind of first-class citizen, the progress they've made with it and now bringing in some of the things like LakeFlow Connect."
Along with Tabular, Databricks recently acquired Mosaic AI and might be aiming to be the go-to AI operating system, Furrier theorized. But even with Databricks' new acquisitions, there is a long way to go before AI can perform the multilayered reasoning companies are looking for.
"Gen AI may be attracting all, but to get it right is a very, very tough job. We are now finally facing it, after almost two years of trying to do it," said Sanjeev Mohan, principal at SanjMo, in an analyst segment with Tony Baer, principal at dbInsight LLC. "The idea was, in 2023, we'll do experimentation. In 2024, we've got production. But they're nowhere close to being production. NLMs are amazing for basic language task[s], like summarization, recommendations. But when it comes to doing reasoning and more complex tasks, which is what the customers want, at the end of the day, we still have a lot of work to do."
Here are theCUBE's complete keynote analyses:
2. Databricks' industry partnerships are expanding as the company homes in on AI.Databricks' acquisition of Mosaic AI reflects the company's ongoing efforts to be a generative AI platform for a range of enterprises. Its goal is to help customers bring together all of their data regardless of format, according to Joel Minnick, vice president of marketing at Databricks.
"[Mosaic AI]'s that end-to-end workflow from the time that I get the data prepped to how I build the model, to how I deploy the model, to how I then ultimately evaluate whether or not the model's successful," he said in an interview with theCUBE. "The role of Unity Catalog is to underlie that."
Small language models have been gaining popularity, with businesses wanting to prioritize efficiency and interoperability. Compound systems of AI models are also being favored with customers because of their speed, according to Minnick.
Databricks' Joel Minnick discusses the company's acquisition of Mosaic AI.
"We are finding for the customers who are starting to put things into production, particularly on newer use cases, like putting AI agents out there into the world now or multimodal systems out there, the compound systems [are] moving much faster," he said. "[There's] a dense model, which means I have one single model tackling the entire problem from the time that I got the prompt from the user to the time I give you the answer back. Compound AI says, 'Well, let's actually break it up into discrete units.'"
Databricks continues to enhance its network of partnerships in order to prepare customers for the AI era, including collaborations with Fivetran, a data movement platform, and Alation, which helps customers catalog and organize their data assets.
"You need to have your data foundation really, really ready to move into the gen AI world. That's easy to say, really, really hard to do," said Jeff Veis, chief marketing officer of Impetus Technologies, which assists enterprises in moving data off legacy platforms into Databricks. "You need to be able to have it be governed; it needs to be able to be consumed and done in a trusted way. Only 30% of Databricks customers are on catalog today."
Data observability and modernized data infrastructure are crucial for companies trying to implement AI models, which require good data to produce reliable answers. Data silos are an ongoing problem, as well as accessibility. To address these issues, Databricks has partnered with Prophecy Inc. To create an AI co-pilot that gives its users the tools they need to harness their data.
"Everybody who wants data has got to have access to it … we are providing the tooling layer that makes data engineers … and data analysts and [a] line of business users, all of them productive with data," said Raj Bains, founder and chief executive officer of Prophecy, in an interview. "As Databricks is anchored on and focused on data functionality, we are anchored and focused on the data user."
Here's the complete interview with Joel Minnick:
3. Databricks is applying AI to diverse use cases.A number of Databricks' partners were present at the Data + AI Summit, and they showcased increasingly varied use cases for the data engineering and AI tools that Databricks offers. With their data in order, companies such as Condé Nast, which hosts over 60 publications, can now build language models that detect user preferences and service them accordingly.
Condé Nast's Tim Shokair talks about the company's AI use cases.
"Our data engineering team is processing all our data and storing it within Databricks, making it accessible for us," said Tim Shokair, senior director of data science at Condé Nast, in an interview with theCUBE. "My team uses it for model development. We do a lot of model exploration. We use a lot of the ML tools that they built in, and then my machine learning engineering team uses it now to take those models, turn them into services. What I get really excited about is thinking about how well these models have to understand content to make these predictions, and that's generally what I like to exploit."
Although the platform has a steeper learning curve than Snowflake, Databricks is popular among data scientists because it has unlimited scalability for large amounts of data and provides users a workspace with its machine learning runtime, Managed MLflow.
"It started for me as a data scientist, a tool that we could easily use to collaborate and building off of Spark, being able to process massive amounts of data or build models on top of massive amounts of data was really useful," Shokair said. "Then, they've just continued to build features that make it easier for us as data scientists and machine learning engineers to put models into production."
One use case for Condé Nast was creating a tool for the Vogue Runway app that allows users to search 1.2 million Runway images.
Having a strategy for integrating AI into your business is just as crucial as having the funds or the technology at your disposal, according to Anand Pradhan, senior director of technology at the AI Center of Excellence at ICE, which works with companies including Mosaic AI and Databricks to support companies' AI journeys.
"We make sure that we implement the AI use cases properly, so that each [step] of the process is followed so that we get some value addition first of all, and it is implemented the right way … that's one thing," Pradhan said, who also runs the New York Stock Exchange Launchpad Lab. "The other thing is having a strategic vision in the company to lay out the foundation so that multiple AI applications or projects can run with the same vision."
Here's theCUBE's complete interview with Tim Shokair:
To watch more of SiliconANGLE and theCUBE Research's coverage of Data + AI Summit, here's our complete video playlist:
Photo: SiliconANGLE Your vote of support is important to us and it helps us keep the content FREE. One click below supports our mission to provide free, deep, and relevant content. Join our community on YouTube Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.Com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts."TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well" – Andy Jassy
THANK YOU
Databricks Data+AI Summit 2024: The Standout Vendors
Databricks Data+AI Summit 2024: The Standout VendorsGoogle Cloud, Microsoft, Cube and Posit were among the most innovative vendors at Databricks Data+AI Summit 2024.
Tech giants, startups and one company a Databricks co-founder dubbed "the coolest open source company" people haven't heard of gathered this week for the open analytics platform provider's annual Data+AI Summit.
During the event, the San Francisco-based vendor made news by revealing upgrades to its Mosaic AI unified tooling product for artificial intelligence and machine learning and the open-sourcing of its Unity Catalog, among other announcements.
Databricks CEO and cofounder Ali Ghodsi used the event to highlight the work of the company's services-led partners, including global system integrators and regional ones, telling CRN during a question-and-answer session that "there's no way we can actually do the work they do ourselves."
"We are perfect complements," Ghodsi said. "We both need to succeed together for Databricks to succeed. … Databricks would not be what it is without the GSIs and RSIs. And it's going to be key to our success going forward. … The AI revolution is not going to happen without the SIs."
[RELATED: Databricks Data+AI Summit 2024: The Biggest News]
Databricks Data+AI Summit 2024Databricks has more than 3,800 partners worldwide, according to the vendor.
CRN scoured the Data+AI Summit 2024 floor for vendors doing some of the standout work around with Databricks. Those vendors include:
Here's more on the most compelling vendor partners working with Databricks on analysis and AI.
Amazon Web Services joined Google Cloud, Microsoft, Nvidia and Salesforce as part of Databricks' news that it has open sourced its Unity Catalog offering, which brings together data and artificial intelligence (AI) governance across clouds, data formats and data platforms.
The Seattle-based tech giant's cloud is interoperable with Unity Catalog OSS, according to Databricks. AWS representatives were on site to talk about AWS AI services for intelligent document processing, delivering portability to open data lakes with Delta Lake UniForm, managing the cost for data and AI workloads with Databricks on AWS and more.
Beyond Unity Catalog, the two vendors made news earlier this year when Databricks received a Federal Risk and Authorization Management Program (FedRamp) High Agency Authority to Operate (ATO) for its cloud services on AWS GovCloud, which should allow government agencies and contractors to leverage Databricks offerings with highly sensitive private sector and public sector data, meeting the U.S. Federal government's compliance standards.
Databricks on AWS promises better price and performance and has thousands of customer implementations, according to Databricks. Databricks clusters support AWS Graviton instances, works with AWS single sign on (SSO), integrates with Redshift and works with AWS Glue, among other capabilities.
AWS has more than 130,000 partners worldwide, according to CRN's 2024 Channel Chiefs.
Google representatives attended Databricks Data+AI Summit 2024 to discuss the vendor's Gemini large language model (LLM).
Other subjects discussed by the Mountain View, Calif.-based tech giant at Summit include how Databricks deployed on Google Kubernetes Engine (GKE) can improve data preparation, training, tuning, inference and other AI and ML workloads, among other topics.
In April, Google revealed BigQuery native support for Delta Lake for delivery of data for downstream applications such as business intelligence, reporting and other downstream applications. Now, Databricks and BigQuery users can maintain a single authoritative copy of data that they can query with both tools without exporting, copying or using manifest files, among other benefits, according to Databricks.
Google Cloud has more than 100,000 partners worldwide, according to of CRN's 2024 Channel Chiefs
IBM and Databricks maintain a level of coopetition – in May, IBM said that its Watsonx.Data with Presto C++ v0.286 and query optimizer on IBM Storage Fusion HCI provided better price performance compared to Databrick's Photon engine with equal query runtime at less than 60 percent of the cost.
But the Armonk, N.Y.-based tech giant has a number of integrations with Databricks, including allowing its Databand observability offering to work with Databricks for continuous Spark monitoring. IBM's DataStage data integration tool also has a connector with Databricks to help with data movement and transformation.
The two have even joined on a recent funding round for data ingesting and pre-processing upstart Unstructured.
IBM has thousands of partners and is a member of CRN's 2024 Channel Chiefs.
Intel is not just a partner of Databricks – the chipmaker has leveraged Databricks as part of the organization's move to the cloud, migrating data and analytics workloads to make a robust, connected data foundation and reduce infrastructure costs.
Representatives with the Santa Clara, Calif.-based vendor were at Summit to talk about the Tiber portfolio of enterprise software, services and products for AI, edge, cloud, trust and security.
Databricks can be used for autonomously optimizing central processing units (CPUs), managing resources cost effectively, improving scalability, improving elasticity, streamlining Spark operations and more to enhance large-scale data processing and analytics for AI applications, according to Intel.
Intel is a member of CRN's 2024 Channel Chiefs.
Although Databricks' new AI/BI business intelligence tool puts it competition with similar products from Microsoft, the two vendors have a close partnership that extends to the Azure Databricks fully managed first-party service that puts an open data lakehouse in Azure.
Earlier this year, The Redmond, Wash.-based AI giant revealed expanded integration between its Fabric data platform and Azure Databricks, giving users the ability to create and configure a new Azure Databricks Unity Catalog item from the Fabric portal and a way to access lakehouses and other Fabric data items as a catalog in Azure Databricks.
During Summit, Microsoft held a session on how Azure Databricks can enable confidential computing for guarding sensitive data within hardware-based trusted execution environments, previewing upcoming support for Intel Confidential VMs on Azure Databricks and expansion plans for Confidential VM availability across various regions.
Microsoft has more than 400,000 partners worldwide and is a member of CRN's 2024 Channel Chiefs.
During Summit, Qlik revealed a new integration with Databricks AI Functions, allowing enterprises to access Mosaic AI functionalities without extensive infrastructure changes or specialized training.
The integration gives users of the King of Prussia, Pa.-based company's business analytics platform access to Databricks sentiment analysis, classification and translation within the Qlik Cloud Data Integration workflow plus support for Databricks Vector Store, among other features.
Qlik also promises users the ability to leverage its Cloud Data Integration to provision AI-ready data directly into Databricks Lakehouse, automating the entire data lake pipeline without writing code.
Qlik has about 2,200 channel partners worldwide and 500 in North America, according to CRN's 2024 Channel Chiefs.
Databricks named Informatica its Data Integration Partner of the Year during Summit 2024, giving the company kudos for integrating its Intelligent Data Management Cloud and PowerCenter Modernization Solution with the Databricks Data Intelligence Platform.
These integrations between Databricks and the Redwood, Calif.-based AI-powered cloud data management vendor promises users a way to migrate existing data in on-premises warehouses to Databricks and build new data pipelines, according to Databricks.
The two companies' partnership goes back about seven years, with the companies sharing hundreds of enterprise users, according to Informatica.
Informatica has a partner program for system integrators, resellers, distributors and other partner business models.
Dataiku is Databricks' Innovation Partner of the Year – its second consecutive POTY recognition – due in part to integrations with the latter's Data Intelligence Platform to allow business and technical users to collaborate on AI projects and accelerate AI adoption.
Representatives with the New York-based data science platform provider were on site at Summit 2024 to show how users can leverage Dataiku with Databricks to build an enterprise-grade AI application with few lines of code, plus using Dataiku to monitor toxicity, latency, cost, and bias in LLMs.
Users of both companies can integrate Databricks Foundation Model APIs as part of the Dataiku LLM Mesh, execute custom Python code recipes directly on Databricks clusters and perform other actions, according to Dataiku.
Dataiku has a partner program for resellers, service providers and other partner business models.
Just before the Summit, Cube revealed that Databricks' venture arm participated in a $25 million funding round meant to go toward improving its Cube Cloud universal semantic layer.
The San Francisco-based startup – founded in 2019 – bills this semantic layer middleware component as a way to unify business logic, centralize governance and security, improve query performance and integrate with any data endpoint.
Using the companies' tools together allows users to make cloud data accessible to all data consumers with consistently modeled metrics and performant queries.
Cube launched a partner program for consultancies and other business partner models last year.
On stage at Databricks Data+AI Summit 2024, the company's co-founder and chief architect, Reynold Xin, called Posit "the coolest open source company" people haven't heard of – in part because it rebranded from RStudio in 2022.
The Boston-based vendor positions its tools as simplifying the deployment of open source data science work across enterprises, letting users share Jupyter notebooks, Plotly dashboards and interactive applications built with Shiny, Streamlit, Dash and other R and Python frameworks, according to Posit.
Using the companies' tools together has the potential to cut costs on context switching, speed up data visualization and sharing, and more. The future of integration for the two companies includes a single sign-on to Databricks in Posit Connect, according to Posit.
Posit offers a partner program for resellers and other partner business models.
World AI Summit Kicks Off In Austin
The goal of the summit is to help close the knowledge gap for business owners and any professional looking to use artificial intelligence to increase efficiency.
AUSTIN, Texas — The first World AI Summit is happening in Austin.
The use of artificial intelligence (AI) in business is growing at a very fast pace. The goal of the World AI Summit is to help close the knowledge gap for business owners and any professional looking to use AI to increase efficiency.
Co-founder Joshua Pellicer said there are 18 expert speakers from around the world and hands-on workshops. They will discuss a wide range of topics, and Pellicer himself plans to talk about how he used AI to launch a business in just six hours.
Pellicer said AI gave him the idea for his app and everything else needed to make it happen. Within two weeks, he said he had an offer to buy the company.
"So anybody that has $20 a month and can pay for ChatGPT and knows how to ask the right questions, is going to be competing in every industry in the world coming up. And it's going to blow entire industries out of the water," Pellicer said.
He said he chose to host the first summit in Austin because it's a booming tech city.
According to Flatworld Solutions research, Texas already has over four AI jobs for every 100,000 residents and that number is growing daily.
The summit is going on at Brazos Hall until June 8, but ticket sales end June 7.
Boomtown is KVUE's series covering the explosive growth in Central Texas. For more Boomtown stories, head to KVUE.Com/Boomtown.
Daranesha Herron on social media: FacebookXInstagram
KVUE on social media: FacebookXInstagramYouTube
Comments
Post a Comment