Databricks
Auto Added by WPeMatico
Databricks makes bringing data into its ‘lakehouse’ easier
Databricks today announced the launch of its new Data Ingestion Network of partners and the launch of its Databricks Ingest service. The idea here is to make it easier for businesses to combine the best of data warehouses and data lakes into a single platform — a concept Databricks likes to call “lakehouse.”
At the core of the company’s lakehouse is Delta Lake, Databricks’ Linux Foundation-managed open-source project that brings a new storage layer to data lakes that helps users manage the lifecycle of their data and ensures data quality through schema enforcement, log records and more. Databricks users can now work with the first five partners in the Ingestion Network — Fivetran, Qlik, Infoworks, StreamSets, Syncsort — to automatically load their data into Delta Lake. To ingest data from these partners, Databricks customers don’t have to set up any triggers or schedules — instead, data automatically flows into Delta Lake.
“Until now, companies have been forced to split up their data into traditional structured data and big data, and use them separately for BI and ML use cases. This results in siloed data in data lakes and data warehouses, slow processing and partial results that are too delayed or too incomplete to be effectively utilized,” says Ali Ghodsi, co-founder and CEO of Databricks. “This is one of the many drivers behind the shift to a Lakehouse paradigm, which aspires to combine the reliability of data warehouses with the scale of data lakes to support every kind of use case. In order for this architecture to work well, it needs to be easy for every type of data to be pulled in. Databricks Ingest is an important step in making that possible.”
Databricks VP of Product Marketing Bharath Gowda also tells me that this will make it easier for businesses to perform analytics on their most recent data and hence be more responsive when new information comes in. He also noted that users will be able to better leverage their structured and unstructured data for building better machine learning models, as well as to perform more traditional analytics on all of their data instead of just a small slice that’s available in their data warehouse.
Powered by WPeMatico
Anyscale, from the creators of the Ray distributed computing project, launches with $20.6M led by a16z
Open source has become a critical building block of modern software, and today a new startup is coming out of stealth to capitalise on one of the newer frontiers in open source: using it to build and manage distributed application environments, an approach being used increasingly to handle large computing projects, such as those involving artificial intelligence or scientific or other complex calculations.
Anyscale, a startup founded by the same team that built the Project Ray open-source distributed programming framework out of UC Berkeley — Robert Nishihara, Philipp Moritz and Ion Stoica, and Berkeley professor Michael I. Jordan — has raised $20.6 million in a Series A round of funding led by Andreessen Horowitz, with participation also from NEA, Intel Capital, Ant Financial, Amplify Partners, 11.2 Capital and The House Fund.
The company plans to use the money to build out its first commercial products — details of which are still being kept under wraps but will more generally include the ability to easily scale out a computing project from one laptop to a cluster of machines; and a group of libraries and applications to manage projects. These are expected to launch next year.
“Right now we are focused on making Ray a standard for building applications,” said Stoica in an interview. “The company will build tools and a runtime platform for Ray. So, if you want to run a Ray application securely and with high performance then you will use our product.”
The funding is partly strategic: Intel is one of the big companies that has been using Ray for its own computing projects, alongside Amazon, Microsoft and Ant Financial.
“Intel IT has been leveraging Ray to scale Python workloads with minimal code modifications,” said Moty Fania, principal engineer and chief technology officer for Intel IT’s Enterprise and Platform Group, in a statement. “With the implementation into Intel’s manufacturing and testing processes, we have found that Ray helps increase the speed and scale of our hyperparameter selection techniques and auto modeling processes used for creating personalized chip tests. For us, this has resulted in reduced costs, additional capacity and improved quality.”
With an impressive user list like this for the free-to-use Ray, you might ask yourself, what is the purpose of Anyscale? As Stoica and Nishihara explained, the idea will be to create simpler and easier ways to implement Ray, to make it usable whether you’re one of the Amazons of the world, or a more modest, and possibly less tech-centric operation.
“We see that this will be valuable mostly for companies who do not have engineering experts,” Stoica said.
The problem that Anyscale is solving is a central one to the future of large-scale, involved computing projects: there are an increasing array of problems that are being tackled with computing solutions, but as the complexity of the work involved increases, there is a limit to how much work a single machine (even a big one) can handle. (Indeed, Anyscale cites IDC figures estimating that the amount of data created and copied annually will reach 175 zettabytes by 2025.)
While one day there may be quantum-computing machines that can run efficiently and at scale to address these kinds of tasks, today this isn’t a realistic option, and so distributed computing has emerged as a solution.
Ray was devised as a standard to use to implement distributed computing environments, but on its own it’s too technical for the uninitiated to use.
“Imagine you’re a biologist,” added Nishihara. “You can write a simple program and run it at a large scale, but to do that successfully you need not only to be a biology expert but a computing expert. That’s just way too high a barrier.”
The people behind Anyscale (and Ray) have a long and very credible list of other work behind them that speaks to the opportunities that are being spotted here. Stoica, for example, was also the co-founder of Databricks, Conviva and one of the original developers of Apache Spark.
“I worked on Databricks with Ion and that’s how it started,” Andreessen Horowitz co-founder Ben Horowitz said in an interview. He added that the firm has been a regular investor into projects coming out of UC Berkeley. Ray, and more specifically Anyscale, is notable for its relevance to today’s computing needs.
“With Ray it was a very attractive project because of the open-source metrics but also because of the issue it addresses,” he said.
“We’ve been grappling with Moore’s Law being over, but more interestingly, it’s inadequate for things like artificial intelligence applications,” where increasing computing power is needed that outstrips what any single machine can do. “You have to be able to deal with distributed computing, but the problem for everyone but Google is that distributed computing is hard, so we have been looking for a solution.”
Powered by WPeMatico
Microsoft’s Azure Synapse Analytics bridges the gap between data lakes and warehouses
At its annual Ignite conference in Orlando, Fla., Microsoft today announced a major new Azure service for enterprises: Azure Synapse Analytics, which Microsoft describes as “the next evolution of Azure SQL Data Warehouse.” Like SQL Data Warehouse, it aims to bridge the gap between data warehouses and data lakes, which are often completely separate. Synapse also taps into a wide variety of other Microsoft services, including Power BI and Azure Machine Learning, as well as a partner ecosystem that includes Databricks, Informatica, Accenture, Talend, Attunity, Pragmatic Works and Adatis. It’s also integrated with Apache Spark.
The idea here is that Synapse allows anybody working with data in those disparate places to manage and analyze it from within a single service. It can be used to analyze relational and unstructured data, using standard SQL.
Microsoft also highlights Synapse’s integration with Power BI, its easy to use business intelligence and reporting tool, as well as Azure Machine Learning for building models.
With the Azure Synapse studio, the service provides data professionals with a single workspace for prepping and managing their data, as well as for their big data and AI tasks. There’s also a code-free environment for managing data pipelines.
As Microsoft stresses, businesses that want to adopt Synapse can continue to use their existing workloads in production with Synapse and automatically get all of the benefits of the service. “Businesses can put their data to work much more quickly, productively, and securely, pulling together insights from all data sources, data warehouses, and big data analytics systems,” writes Microsoft CVP of Azure Data, Rohan Kumar.
In a demo at Ignite, Kumar also benchmarked Synapse against Google’s BigQuery. Synapse ran the same query over a petabyte of data in 75% less time. He also noted that Synapse can handle thousands of concurrent users — unlike some of Microsoft’s competitors.
Powered by WPeMatico
Databricks announces $400M round on $6.2B valuation as analytics platform continues to grow
Databricks is a SaaS business built on top of a bunch of open-source tools, and apparently it’s been going pretty well on the business side of things. In fact, the company claims to be one of the fastest growing enterprise cloud companies ever. Today the company announced a massive $400 million Series F funding round on a hefty $6.2 billion valuation. Today’s funding brings the total raised to almost a $900 million.
Andreessen Horowitz’s Late Stage Venture Fund led the round with new investors BlackRock, Inc., T. Rowe Price Associates, Inc. and Tiger Global Management also participating. The institutional investors are particularly interesting here because as a late-stage startup, Databricks likely has its eye on a future IPO, and having those investors on board already could give them a head start.
CEO Ali Ghodsi was coy when it came to the IPO, but it sure sounded like that’s a direction he wants to go. “We are one of the fastest growing cloud enterprise software companies on record, which means we have a lot of access to capital as this fundraise shows. The revenue is growing gangbusters, and the brand is also really well known. So an IPO is not something that we’re optimizing for, but it’s something that’s definitely going to happen down the line in the not-too-distant future,” Ghodsi told TechCrunch.
The company announced as of Q3 it’s on a $200 million run rate, and it has a platform that consists of four products, all built on foundational open source: Delta Lake, an open-source data lake product; MLflow, an open-source project that helps data teams operationalize machine learning; Koalas, which creates a single machine framework for Spark and Pandos, greatly simplifying working with the two tools; and, finally, Spark, the open-source analytics engine.
You can download the open-source version of all of these tools for free, but they are not easy to use or manage. The way that Databricks makes money is by offering each of these tools in the form of Software as a Service. They handle all of the management headaches associated with using these tools and they charge you a subscription price.
It’s a model that seems to be working, as the company is growing like crazy. It raised $250 million just last February on a $2.75 billion valuation. Apparently the investors saw room for a lot more growth in the intervening six months, as today’s $6.2 billion valuation shows.
Powered by WPeMatico
Databricks brings its Delta Lake project to the Linux Foundation
Databricks, the big data analytics service founded by the original developers of Apache Spark, today announced that it is bringing its Delta Lake open-source project for building data lakes to the Linux Foundation under an open governance model. The company announced the launch of Delta Lake earlier this year, and, even though it’s still a relatively new project, it has already been adopted by many organizations and has found backing from companies like Intel, Alibaba and Booz Allen Hamilton.
“In 2013, we had a small project where we added SQL to Spark at Databricks […] and donated it to the Apache Foundation,” Databricks CEO and co-founder Ali Ghodsi told me. “Over the years, slowly people have changed how they actually leverage Spark and only in the last year or so it really started to dawn upon us that there’s a new pattern that’s emerging and Spark is being used in a completely different way than maybe we had planned initially.”
This pattern, he said, is that companies are taking all of their data and putting it into data lakes and then doing a couple of things with this data, machine learning and data science being the obvious ones. But they are also doing things that are more traditionally associated with data warehouses, like business intelligence and reporting. The term Ghodsi uses for this kind of usage is “Lake House.” More and more, Databricks is seeing that Spark is being used for this purpose and not just to replace Hadoop and doing ETL (extract, transform, load). “This kind of Lake House patterns we’ve seen emerge more and more and we wanted to double down on it.”
Spark 3.0, which is launching today soon, enables more of these use cases and speeds them up significantly, in addition to the launch of a new feature that enables you to add a pluggable data catalog to Spark.
Delta Lake, Ghodsi said, is essentially the data layer of the Lake House pattern. It brings support for ACID transactions to data lakes, scalable metadata handling and data versioning, for example. All the data is stored in the Apache Parquet format and users can enforce schemas (and change them with relative ease if necessary).
It’s interesting to see Databricks choose the Linux Foundation for this project, given that its roots are in the Apache Foundation. “We’re super excited to partner with them,” Ghodsi said about why the company chose the Linux Foundation. “They run the biggest projects on the planet, including the Linux project but also a lot of cloud projects. The cloud-native stuff is all in the Linux Foundation.”
“Bringing Delta Lake under the neutral home of the Linux Foundation will help the open-source community dependent on the project develop the technology addressing how big data is stored and processed, both on-prem and in the cloud,” said Michael Dolan, VP of Strategic Programs at the Linux Foundation. “The Linux Foundation helps open-source communities leverage an open governance model to enable broad industry contribution and consensus building, which will improve the state of the art for data storage and reliability.”
Powered by WPeMatico
Databricks open-sources Delta Lake to make data lakes more reliable
Databricks, the company founded by the original developers of the Apache Spark big data analytics engine, today announced that it has open-sourced Delta Lake, a storage layer that makes it easier to ensure data integrity as new data flows into an enterprise’s data lake by bringing ACID transactions to these vast data repositories.
Delta Lake, which has long been a proprietary part of Databrick’s offering, is already in production use by companies like Viacom, Edmunds, Riot Games and McGraw Hill.
The tool provides the ability to enforce specific schemas (which can be changed as necessary), to create snapshots and to ingest streaming data or backfill the lake as a batch job. Delta Lake also uses the Spark engine to handle the metadata of the data lake (which by itself is often a big data problem). Over time, Databricks also plans to add an audit trail, among other things.
“Today nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability. Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’,” said Ali Ghodsi, co-founder and CEO at Databricks.
What’s important to note here is that Delta lake runs on top of existing data lakes and is compatible with the Apache spark APIs.
The company is still looking at how the project will be governed in the future. “We are still exploring different models of open source project governance, but the GitHub model is well understood and presents a good trade-off between the ability to accept contributions and governance overhead,” Ghodsi said. “One thing we know for sure is we want to foster a vibrant community, as we see this as a critical piece of technology for increasing data reliability on data lakes. This is why we chose to go with a permissive open source license model: Apache License v2, same license that Apache Spark uses.”
To invite this community, Databricks plans to take outside contributions, just like the Spark project.
“We want Delta Lake technology to be used everywhere on-prem and in the cloud by small and large enterprises,” said Ghodsi. “This approach is the fastest way to build something that can become a standard by having the community provide direction and contribute to the development efforts.” That’s also why the company decided against a Commons Clause licenses that some open-source companies now use to prevent others (and especially large clouds) from using their open source tools in their own commercial SaaS offerings. “We believe the Commons Clause license is restrictive and will discourage adoption. Our primary goal with Delta Lake is to drive adoption on-prem as well as in the cloud.”
Powered by WPeMatico
Databricks raises $250M at a $2.75B valuation for its analytics platform
Databricks, the company founded by the original team behind the Apache Spark big data analytics engine, today announced that it has raised a $250 million Series E round led by Andreessen Horowitz. Coatue Management, Green Bay Ventures, Microsoft and NEA, also participated in this round, which brings the company’s total funding to $498.5 million. Microsoft’s involvement here is probably a bit of a surprise, but it’s worth noting that it also worked with Databricks on the launch of Azure Databricks as a first-party service on the platform, something that’s still a rarity in the Azure cloud.
As Databricks also today announced, its annual recurring revenue now exceeds $100 million. The company didn’t share whether it’s cash flow-positive at this point, but Databricks CEO and co-founder Ali Ghodsi shared that the company’s valuation is now $2.75 billion.
Current customers, which the company says number around 2,000, include the likes of Nielsen, Hotels.com, Overstock, Bechtel, Shell and HP.
“What Ali and the Databricks team have built is truly phenomenal,” Green Bay Ventures co-founder Anthony Schiller told me. “Their success is a testament to product innovation at the highest level. Databricks is without question best-in-class and their impact on the industry proves it. We were thrilled to participate in this round.”
While Databricks is obviously known for its contributions to Apache Spark, the company itself monetizes that work by offering its Unified Analytics platform on top of it. This platform allows enterprises to build their data pipelines across data storage systems and prepare data sets for data scientists and engineers. To do this, Databricks offers shared notebooks and tools for building, managing and monitoring data pipelines, and then uses that data to build machine learning models, for example. Indeed, training and deploying these models is one of the company’s focus areas these days, which makes sense, given that this is one of the main use cases for big data, after all.
On top of that, Databricks also offers a fully managed service for hosting all of these tools.
“Databricks is the clear winner in the big data platform race,” said Ben Horowitz, co-founder and general partner at Andreessen Horowitz, in today’s announcement. “In addition, they have created a new category atop their world-beating Apache Spark platform called Unified Analytics that is growing even faster. As a result, we are thrilled to invest in this round.”
Ghodsi told me that Horowitz was also instrumental in getting the company to re-focus on growth. The company was already growing fast, of course, but Horowitz asked him why Databricks wasn’t growing faster. Unsurprisingly, given that it’s an enterprise company, that means aggressively hiring a larger sales force — and that’s costly. Hence the company’s need to raise at this point.
As Ghodsi told me, one of the areas the company wants to focus on is the Asia Pacific region, where overall cloud usage is growing fast. The other area the company is focusing on is support for more verticals like mass media and entertainment, federal agencies and fintech firms, which also comes with its own cost, given that the experts there don’t come cheap.
Ghodsi likes to call this “boring AI,” since it’s not as exciting as self-driving cars. In his view, though, the enterprise companies that don’t start using machine learning now will inevitably be left behind in the long run. “If you don’t get there, there’ll be no place for you in the next 20 years,” he said.
Engineering, of course, will also get a chunk of this new funding, with an emphasis on relatively new products like MLFlow and Delta, two tools Databricks recently developed and that make it easier to manage the life cycle of machine learning models and build the necessary data pipelines to feed them.
Powered by WPeMatico
Big data analytics platform Databricks raises $140M Series D round led by Andreessen Horowitz
Databricks, a big data analytics platform built by a team that grew out of the Apache Spark project, today announced that it has raised a $140 million Series D round led by Andreessen Horowitz, with participation from New Enterprise Associates and Battery Ventures. This brings Databricks’ total funding raised to date to $247 million, which includes a $60 million round the company… Read More
Powered by WPeMatico
Databricks releases serverless platform for Apache Spark along with new library supporting deep learning
Today to kick off Spark Summit, Databricks announced a Serverless Platform for Apache Spark — welcome news for developers looking to reduce time spent on cluster management. The move to simplify developer experiences is set to be a major theme of the event overall. In addition to Serverless, the company also introduced Deep Learning Pipelines, a library that makes it easy to mix… Read More
Powered by WPeMatico
Databricks Launches Free Community Edition As Companion To Free Online Spark Courses
Databricks, the commercial company created from the open source Apache Spark project, announced the release of a free Community Edition today aimed at teaching people how to use Spark — and as an adjunct to the free online courses (MOOCs) it created last year. The free version is a limited edition without all of the advanced features you would find in the enterprise-pay… Read More
Powered by WPeMatico