Qubole launches Quantum, its serverless database engine

Published June 10, 2019

Qubole, the data platform founded by Apache Hive creator and former head of Facebook’s Data Infrastructure team Ashish Thusoo, today announced the launch of Quantum, its first serverless offering.

Qubole may not necessarily be a household name, but its customers include the likes of Autodesk, Comcast, Lyft, Nextdoor and Zillow . For these users, Qubole has long offered a self-service platform that allowed their data scientists and engineers to build their AI, machine learning and analytics workflows on the public cloud of their choice. The platform sits on top of open-source technologies like Apache Spark, Presto and Kafka, for example.

Typically, enterprises have to provision a considerable amount of resources to give these platforms the resources they need. These resources often go unused and the infrastructure can quickly become complex.

Qubole already abstracts most of this away, offering what is essentially a serverless platform. With Quantum, however, it is going a step further by launching a high-performance serverless SQL engine that allows users to query petabytes of data with nothing else but ANSI-SQL, giving them the choice between using a Presto cluster or a serverless SQL engine to run their queries, for example.

The data can be stored on AWS and users won’t have to set up a second data lake or move their data to another platform to use the SQL engine. Quantum automatically scales up or down as needed, of course, and users can still work with the same metastore for their data, no matter whether they choose the clustered or serverless option. Indeed, Quantum is essentially just another SQL engine without Qubole’s overall suite of engines.

Typically, Qubole charges enterprises by compute minutes. When using Quantum, the company uses the same metric, but enterprises pay for the execution time of the query. “So instead of the Qubole compute units being associated with the number of minutes the cluster was up and running, it is associated with the Qubole compute units consumed by that particular query or that particular workload, which is even more fine-grained,” Thusoo explained. “This works really well when you have to do interactive workloads.”

Thusoo notes that Quantum is targeted at analysts who often need to perform interactive queries on data stored in object stores. Qubole integrates with services like Tableau and Looker (which Google is now in the process of acquiring). “They suddenly get access to very elastic compute capacity, but they are able to come through a very familiar user interface,” Thusoo noted.

Google Cloud gets capacity reservations, extends committed use discounts beyond CPUs

Published June 5, 2019

Google Cloud made two significant pricing announcements today. Those, you’ll surely be sad to hear, don’t involve the usual price drops for compute and storage. Instead, Google Cloud today announced that it is extending its committed-use discounts, which give you a significant discount when you commit to using a certain number of resources for one or three years, to GPUs, Cloud TPU Pods and local SSDs. In return for locking yourself into a long-term plan, you can get discounts of 55% off on-demand prices.

In addition, Google is launching a capacity reservation system for Compute Engine that allows users to reserve resources in a specific zone for later use to ensure that they have guaranteed access to these resources when needed.

At first glance, capacity reservations may seem like a weird concept in the cloud. The promise of cloud computing, after all, is that you can just spin machines up and down at will — and never really have to think about availability.

So why launch a reservation system? “This is ideal for use cases like disaster recovery or peace of mind, so a customer knows that they have some extra resources, but also for retail events like Black Friday or Cyber Monday,” Google senior product manager Manish Dalwadi told me.

These users want to have absolute certainty that when they need the resources, they will be available to them. And while many of us think of the large clouds as having a virtually infinite amount of virtual machines available at any time, some machine types may occasionally only be available in a different availability zone, for example, that is not the same zone as where the rest of your compute resources are.

Users can create or delete reservations at any time and any existing discounts — including sustained use discounts and committed use discounts — will be applied automatically.

As for committed-use discounts, it’s worth noting that Google always took a pretty flexible approach to this. Users don’t have to commit to using a specific machine type for three years, for example. Instead, they commit to using a specific number of CPU cores and memory, for example.

“What we heard from customers was that other commit models are just too inflexible and their utilization rates were very low, like 70, 60% utilization,” Google product director Paul Nash told me. “So one of our design goals with committed-use discounts was to figure out how we could provide something that gives us the capacity planning signal that we need, provides the same amount of discounts that we want to pass on to customers, but do it in a way that customers actually feel like they are getting a great deal and so that they don’t have to hyper-manage these things in order to get the most out of them.”

Both the extended committed-use discounts and the new capacity reservation system for Compute Engine resources are now live in the Google Cloud.

How Kubernetes came to rule the world

Published June 4, 2019

Open source has become the de facto standard for building the software that underpins the complex infrastructure that runs everything from your favorite mobile apps to your company’s barely usable expense tool. Over the course of the last few years, a lot of new software is being deployed on top of Kubernetes, the tool for managing large server clusters running containers that Google open-sourced five years ago.

Today, Kubernetes is the fastest growing open-source project, and earlier this month, the bi-annual KubeCon+CloudNativeCon conference attracted almost 8,000 developers to sunny Barcelona, Spain, making the event the largest open-source conference in Europe yet.

To talk about how Kubernetes came to be, I sat down with Craig McLuckie, one of the co-founders of Kubernetes at Google (who then went on to his own startup, Heptio, which he sold to VMware); Tim Hockin, another Googler who was an early member on the project and was also on Google’s Borg team; and Gabe Monroy, who co-founded Deis, one of the first successful Kubernetes startups, and then sold it to Microsoft, where he is now the lead PM for Azure Container Compute (and often the public face of Microsoft’s efforts in this area).

Google’s cloud and the rise of containers

To set the stage a bit, it’s worth remembering where Google Cloud and container management were five years ago.

Microsoft makes a push for service mesh interoperability

Published May 21, 2019

Services meshes. They are the hot new thing in the cloud native computing world. At KubeCon, the bi-annual festival of all things cloud native, Microsoft today announced that it is teaming up with a number of companies in this space to create a generic service mesh interface. This will make it easier for developers to adopt the concept without locking them into a specific technology.

In a world where the number of network endpoints continues to increase as developers launch new micro-services, containers and other systems at a rapid clip, they are making the network smarter again by handling encryption, traffic management and other functions so that the actual applications don’t have to worry about that. With a number of competing service mesh technologies, though, including the likes of Istio and Linkerd, developers currently have to choose which one of these to support.

“I’m really thrilled to see that we were able to pull together a pretty broad consortium of folks from across the industry to help us drive some interoperability in the service mesh space,” Gabe Monroy, Microsoft’s lead product manager for containers and the former CTO of Deis, told me. “This is obviously hot technology — and for good reasons. The cloud-native ecosystem is driving the need for smarter networks and smarter pipes and service mesh technology provides answers.”

The partners here include Buoyant, HashiCorp, Solo.io, Red Hat, AspenMesh, Weaveworks, Docker, Rancher, Pivotal, Kinvolk and VMware . That’s a pretty broad coalition, though it notably doesn’t include cloud heavyweights like Google, the company behind Istio, and AWS.

“In a rapidly evolving ecosystem, having a set of common standards is critical to preserving the best possible end-user experience,” said Idit Levine, founder and CEO of Solo.io. “This was the vision behind SuperGloo — to create an abstraction layer for consistency across different meshes, which led us to the release of Service Mesh Hub last week. We are excited to see service mesh adoption evolve into an industry-level initiative with the SMI specification.”

For the time being, the interoperability features focus on traffic policy, telemetry and traffic management. Monroy argues that these are the most pressing problems right now. He also stressed that this common interface still allows the different service mesh tools to innovate and that developers can always work directly with their APIs when needed. He also stressed that the Service Mesh Interface (SMI), as this new specification is called, does not provide any of its own implementations of these features. It only defines a common set of APIs.

Currently, the most well-known service mesh is probably Istio, which Google, IBM and Lyft launched about two years ago. SMI may just bring a bit more competition to this market since it will allow developers to bet on the overall idea of a service mesh instead of a specific implementation.

In addition to SMI, Microsoft also today announced a couple of other updates around its cloud-native and Kubernetes services. It announced the first alpha of the Helm 3 package manager, for example, as well as the 1.0 release of its Kubernetes extension for Visual Studio Code and the general availability of its AKS virtual nodes, using the open source Virtual Kubelet project.