big data
Auto Added by WPeMatico
Microsoft Azure sets its sights on more analytics workloads
Enterprises now amass huge amounts of data, both from their own tools and applications, as well as from the SaaS applications they use. For a long time, that data was basically exhaust. Maybe it was stored for a while to fulfill some legal requirements, but then it was discarded. Now, data is what drives machine learning models, and the more data you have, the better. It’s maybe no surprise, then, that the big cloud vendors started investing in data warehouses and lakes early on. But that’s just a first step. After that, you also need the analytics tools to make all of this data useful.
Today, it’s Microsoft turn to shine the spotlight on its data analytics services. The actual news here is pretty straightforward. Two of these are services that are moving into general availability: the second generation of Azure Data Lake Storage for big data analytics workloads and Azure Data Explorer, a managed service that makes easier ad-hoc analysis of massive data volumes. Microsoft is also previewing a new feature in Azure Data Factory, its graphical no-code service for building data transformation. Data Factory now features the ability to map data flows.
Those individual news pieces are interesting if you are a user or are considering Azure for your big data workloads, but what’s maybe more important here is that Microsoft is trying to offer a comprehensive set of tools for managing and storing this data — and then using it for building analytics and AI services.
(Photo credit:Josh Edelson/AFP/Getty Images)
“AI is a top priority for every company around the globe,” Julia White, Microsoft’s corporate VP for Azure, told me. “And as we are working with our customers on AI, it becomes clear that their analytics often aren’t good enough for building an AI platform.” These companies are generating plenty of data, which then has to be pulled into analytics systems. She stressed that she couldn’t remember a customer conversation in recent months that didn’t focus on AI. “There is urgency to get to the AI dream,” White said, but the growth and variety of data presents a major challenge for many enterprises. “They thought this was a technology that was separate from their core systems. Now it’s expected for both customer-facing and line-of-business applications.”
Data Lake Storage helps with managing this variety of data since it can handle both structured and unstructured data (and is optimized for the Spark and Hadoop analytics engines). The service can ingest any kind of data — yet Microsoft still promises that it will be very fast. “The world of analytics tended to be defined by having to decide upfront and then building rigid structures around it to get the performance you wanted,” explained White. Data Lake Storage, on the other hand, wants to offer the best of both worlds.
Likewise, White argued that while many enterprises used to keep these services on their on-premises servers, many of them are still appliance-based. But she believes the cloud has now reached the point where the price/performance calculations are in its favor. It took a while to get to this point, though, and to convince enterprises. White noted that for the longest time, enterprises that looked at their analytics projects thought $300 million projects took forever, tied up lots of people and were frankly a bit scary. “But also, what we had to offer in the cloud hasn’t been amazing until some of the recent work,” she said. “We’ve been on a journey — as well as the other cloud vendors — and the price performance is now compelling.” And it sure helps that if enterprises want to meet their AI goals, they’ll now have to tackle these workloads, too.
Powered by WPeMatico
Big companies are not becoming data-driven fast enough
I remember watching MIT professor Andrew McAfee years ago telling stories about the importance of data over gut feeling, whether it was predicting successful wines or making sound business decisions. We have been hearing about big data and data-driven decision making for so long, you would think it has become hardened into our largest organizations by now. As it turns out, new research by NewVantage Partners finds that most large companies are having problems implementing an organization-wide, data-driven strategy.
McAfee was fond of saying that before the data deluge we have today, the way most large organizations made decisions was via the HiPPO — the highest paid person’s opinion. Then he would chide the audience that this was not the proper way to run your business. Data, not gut feelings, even those based on experience, should drive important organizational decisions.
While companies haven’t failed to recognize McAfee’s advice, the NVP report suggests they are having problems implementing data-driven decision making across organizations. There are plenty of technological solutions out there today to help them, from startups all the way to the largest enterprise vendors, but the data (see, you always need to go back to the data) suggests that it’s not a technology problem, it’s a people problem.
Executives can have farsighted vision that their organizations need to be data-driven. They can acquire all of the latest solutions to bring data to the forefront, but unless they combine that with a broad cultural shift and a deep understanding of how to use that data inside business processes, they will continue to struggle.
The study’s authors, Randy Bean and Thomas H. Davenport, wrote about the people problem in their study’s executive summary. “We hear little about initiatives devoted to changing human attitudes and behaviors around data. Unless the focus shifts to these types of activities, we are likely to see the same problem areas in the future that we’ve observed year after year in this survey.”
The survey found that 72 percent of respondents have failed in this regard, reporting they haven’t been able to create a data-driven culture, whatever that means to individual respondents. Meanwhile, 69 percent reported they had failed to create a data-driven organization, although it would seem that these two metrics would be closely aligned.
Perhaps most discouraging of all is that the data is trending the wrong way. Over the last several years, the report’s authors say that those organizations calling themselves data-driven has actually dropped each year from 37.1 percent in 2017 to 32.4 percent in 2018 to 31.0 percent in the latest survey.
This matters on so many levels, but consider that as companies shift to artificial intelligence and machine learning, these technologies rely on abundant amounts of data to work effectively. What’s more, every organization, regardless of its size, is generating vast amounts of data, simply as part of being a digital business in the 21st century. They need to find a way to control this data to make better decisions and understand their customers better. It’s essential.
There is so much talk about innovation and disruption, and understanding and affecting company culture, but so much of all this is linked. You need to be more agile. You need to be more digital. You need to be transformational. You need to be all of these things — and data is at the center of all of it.
Data has been called the new oil often enough to be cliché, but these results reveal that the lesson is failing to get through. Companies need to be data-driven now, this instant. This isn’t something to be working toward at this point. This is something you need to be doing, unless your ultimate goal is to become irrelevant.
Powered by WPeMatico
Databricks raises $250M at a $2.75B valuation for its analytics platform
Databricks, the company founded by the original team behind the Apache Spark big data analytics engine, today announced that it has raised a $250 million Series E round led by Andreessen Horowitz. Coatue Management, Green Bay Ventures, Microsoft and NEA, also participated in this round, which brings the company’s total funding to $498.5 million. Microsoft’s involvement here is probably a bit of a surprise, but it’s worth noting that it also worked with Databricks on the launch of Azure Databricks as a first-party service on the platform, something that’s still a rarity in the Azure cloud.
As Databricks also today announced, its annual recurring revenue now exceeds $100 million. The company didn’t share whether it’s cash flow-positive at this point, but Databricks CEO and co-founder Ali Ghodsi shared that the company’s valuation is now $2.75 billion.
Current customers, which the company says number around 2,000, include the likes of Nielsen, Hotels.com, Overstock, Bechtel, Shell and HP.
“What Ali and the Databricks team have built is truly phenomenal,” Green Bay Ventures co-founder Anthony Schiller told me. “Their success is a testament to product innovation at the highest level. Databricks is without question best-in-class and their impact on the industry proves it. We were thrilled to participate in this round.”
While Databricks is obviously known for its contributions to Apache Spark, the company itself monetizes that work by offering its Unified Analytics platform on top of it. This platform allows enterprises to build their data pipelines across data storage systems and prepare data sets for data scientists and engineers. To do this, Databricks offers shared notebooks and tools for building, managing and monitoring data pipelines, and then uses that data to build machine learning models, for example. Indeed, training and deploying these models is one of the company’s focus areas these days, which makes sense, given that this is one of the main use cases for big data, after all.
On top of that, Databricks also offers a fully managed service for hosting all of these tools.
“Databricks is the clear winner in the big data platform race,” said Ben Horowitz, co-founder and general partner at Andreessen Horowitz, in today’s announcement. “In addition, they have created a new category atop their world-beating Apache Spark platform called Unified Analytics that is growing even faster. As a result, we are thrilled to invest in this round.”
Ghodsi told me that Horowitz was also instrumental in getting the company to re-focus on growth. The company was already growing fast, of course, but Horowitz asked him why Databricks wasn’t growing faster. Unsurprisingly, given that it’s an enterprise company, that means aggressively hiring a larger sales force — and that’s costly. Hence the company’s need to raise at this point.
As Ghodsi told me, one of the areas the company wants to focus on is the Asia Pacific region, where overall cloud usage is growing fast. The other area the company is focusing on is support for more verticals like mass media and entertainment, federal agencies and fintech firms, which also comes with its own cost, given that the experts there don’t come cheap.
Ghodsi likes to call this “boring AI,” since it’s not as exciting as self-driving cars. In his view, though, the enterprise companies that don’t start using machine learning now will inevitably be left behind in the long run. “If you don’t get there, there’ll be no place for you in the next 20 years,” he said.
Engineering, of course, will also get a chunk of this new funding, with an emphasis on relatively new products like MLFlow and Delta, two tools Databricks recently developed and that make it easier to manage the life cycle of machine learning models and build the necessary data pipelines to feed them.
Powered by WPeMatico
Has the fight over privacy changed at all in 2019?
Few issues divide the tech community quite like privacy. Much of Silicon Valley’s wealth has been built on data-driven advertising platforms, and yet, there remain constant concerns about the invasiveness of those platforms.
Such concerns have intensified in just the last few weeks as France’s privacy regulator placed a record fine on Google under Europe’s General Data Protection Regulation (GDPR) rules which the company now plans to appeal. Yet with global platform usage and service sales continuing to tick up, we asked a panel of eight privacy experts: “Has anything fundamentally changed around privacy in tech in 2019? What is the state of privacy and has the outlook changed?”
This week’s participants include:
- Albert Gidari – Consulting Director of Privacy at Stanford’s Center for Internet and Society; Professor at Stanford Law School
- Gabriel Weinberg – CEO and Founder of DuckDuckGo
- Melika Carroll -Senior VP of Global Government Affairs at The Internet Association
- Johnny Ryan – Chief Policy and Industry Relations Officer at Brave
- John Miller – VP of Global Policy and Law at the Information Technology Industry Council
- Nuala O’Connor – President and CEO of the Center for Democracy & Technology
- Chris Baker – Senior VP and General Manager of EMEA at Box
- Christopher Wolf – Founder and Chair of the Future Privacy Forum; Senior counsel at Hogan Lovells focusing on internet law, privacy and data protection policy
TechCrunch is experimenting with new content forms. Consider this a recurring venue for debate, where leading experts – with a diverse range of vantage points and opinions – provide us with thoughts on some of the biggest issues currently in tech, startups and venture. If you have any feedback, please reach out: Arman.Tabatabai@techcrunch.com.
Thoughts & Responses:
Albert Gidari
Albert Gidari is the Consulting Director of Privacy at the Stanford Center for Internet and Society. He was a partner for over 20 years at Perkins Coie LLP, achieving a top-ranking in privacy law by Chambers, before retiring to consult with CIS on its privacy program. He negotiated the first-ever “privacy by design” consent decree with the Federal Trade Commission. A recognized expert on electronic surveillance law, he brought the first public lawsuit before the Foreign Intelligence Surveillance Court, seeking the right of providers to disclose the volume of national security demands received and the number of affected user accounts, ultimately resulting in greater public disclosure of such requests.
There is no doubt that the privacy environment changed in 2018 with the passage of California’s Consumer Privacy Act (CCPA), implementation of the European Union’s General Data Protection Regulation (GDPR), and new privacy laws enacted around the globe.
“While privacy regulation seeks to make tech companies betters stewards of the data they collect and their practices more transparent, in the end, it is a deception to think that users will have more “privacy.””
For one thing, large tech companies have grown huge privacy compliance organizations to meet their new regulatory obligations. For another, the major platforms now are lobbying for passage of a federal privacy law in the U.S. This is not surprising after a year of privacy miscues, breaches and negative privacy news. But does all of this mean a fundamental change is in store for privacy? I think not.
The fundamental model sustaining the Internet is based upon the exchange of user data for free service. As long as advertising dollars drive the growth of the Internet, regulation simply will tinker around the edges, setting sideboards to dictate the terms of the exchange. The tech companies may be more accountable for how they handle data and to whom they disclose it, but the fact is that data will continue to be collected from all manner of people, places and things.
Indeed, if the past year has shown anything it is that two rules are fundamental: (1) everything that can be connected to the Internet will be connected; and (2) everything that can be collected, will be collected, analyzed, used and monetized. It is inexorable.
While privacy regulation seeks to make tech companies betters stewards of the data they collect and their practices more transparent, in the end, it is a deception to think that users will have more “privacy.” No one even knows what “more privacy” means. If it means that users will have more control over the data they share, that is laudable but not achievable in a world where people have no idea how many times or with whom they have shared their information already. Can you name all the places over your lifetime where you provided your SSN and other identifying information? And given that the largest data collector (and likely least secure) is government, what does control really mean?
All this is not to say that privacy regulation is futile. But it is to recognize that nothing proposed today will result in a fundamental shift in privacy policy or provide a panacea of consumer protection. Better privacy hygiene and more accountability on the part of tech companies is a good thing, but it doesn’t solve the privacy paradox that those same users who want more privacy broadly share their information with others who are less trustworthy on social media (ask Jeff Bezos), or that the government hoovers up data at rate that makes tech companies look like pikers (visit a smart city near you).
Many years ago, I used to practice environmental law. I watched companies strive to comply with new laws intended to control pollution by creating compliance infrastructures and teams aimed at preventing, detecting and deterring violations. Today, I see the same thing at the large tech companies – hundreds of employees have been hired to do “privacy” compliance. The language is the same too: cradle to grave privacy documentation of data flows for a product or service; audits and assessments of privacy practices; data mapping; sustainable privacy practices. In short, privacy has become corporatized and industrialized.
True, we have cleaner air and cleaner water as a result of environmental law, but we also have made it lawful and built businesses around acceptable levels of pollution. Companies still lawfully dump arsenic in the water and belch volatile organic compounds in the air. And we still get environmental catastrophes. So don’t expect today’s “Clean Privacy Law” to eliminate data breaches or profiling or abuses.
The privacy world is complicated and few people truly understand the number and variety of companies involved in data collection and processing, and none of them are in Congress. The power to fundamentally change the privacy equation is in the hands of the people who use the technology (or choose not to) and in the hands of those who design it, and maybe that’s where it should be.
Gabriel Weinberg
Gabriel Weinberg is the Founder and CEO of privacy-focused search engine DuckDuckGo.
Coming into 2019, interest in privacy solutions is truly mainstream. There are signs of this everywhere (media, politics, books, etc.) and also in DuckDuckGo’s growth, which has never been faster. With solid majorities now seeking out private alternatives and other ways to be tracked less online, we expect governments to continue to step up their regulatory scrutiny and for privacy companies like DuckDuckGo to continue to help more people take back their privacy.
“Consumers don’t necessarily feel they have anything to hide – but they just don’t want corporations to profit off their personal information, or be manipulated, or unfairly treated through misuse of that information.”
We’re also seeing companies take action beyond mere regulatory compliance, reflecting this new majority will of the people and its tangible effect on the market. Just this month we’ve seen Apple’s Tim Cook call for stronger privacy regulation and the New York Times report strong ad revenue in Europe after stopping the use of ad exchanges and behavioral targeting.
At its core, this groundswell is driven by the negative effects that stem from the surveillance business model. The percentage of people who have noticed ads following them around the Internet, or who have had their data exposed in a breach, or who have had a family member or friend experience some kind of credit card fraud or identity theft issue, reached a boiling point in 2018. On top of that, people learned of the extent to which the big platforms like Google and Facebook that collect the most data are used to propagate misinformation, discrimination, and polarization. Consumers don’t necessarily feel they have anything to hide – but they just don’t want corporations to profit off their personal information, or be manipulated, or unfairly treated through misuse of that information. Fortunately, there are alternatives to the surveillance business model and more companies are setting a new standard of trust online by showcasing alternative models.
Melika Carroll
Melika Carroll is Senior Vice President, Global Government Affairs at Internet Association, which represents over 45 of the world’s leading internet companies, including Google, Facebook, Amazon, Twitter, Uber, Airbnb and others.
We support a modern, national privacy law that provides people meaningful control over the data they provide to companies so they can make the most informed choices about how that data is used, seen, and shared.
“Any national privacy framework should provide the same protections for people’s data across industries, regardless of whether it is gathered offline or online.”
Internet companies believe all Americans should have the ability to access, correct, delete, and download the data they provide to companies.
Americans will benefit most from a federal approach to privacy – as opposed to a patchwork of state laws – that protects their privacy regardless of where they live. If someone in New York is video chatting with their grandmother in Florida, they should both benefit from the same privacy protections.
It’s also important to consider that all companies – both online and offline – use and collect data. Any national privacy framework should provide the same protections for people’s data across industries, regardless of whether it is gathered offline or online.
Two other important pieces of any federal privacy law include user expectations and the context in which data is shared with third parties. Expectations may vary based on a person’s relationship with a company, the service they expect to receive, and the sensitivity of the data they’re sharing. For example, you expect a car rental company to be able to track the location of the rented vehicle that doesn’t get returned. You don’t expect the car rental company to track your real-time location and sell that data to the highest bidder. Additionally, the same piece of data can have different sensitivities depending on the context in which it’s used or shared. For example, your name on a business card may not be as sensitive as your name on the sign in sheet at an addiction support group meeting.
This is a unique time in Washington as there is bipartisan support in both chambers of Congress as well as in the administration for a federal privacy law. Our industry is committed to working with policymakers and other stakeholders to find an American approach to privacy that protects individuals’ privacy and allows companies to innovate and develop products people love.
Johnny Ryan
Dr. Johnny Ryan FRHistS is Chief Policy & Industry Relations Officer at Brave. His previous roles include Head of Ecosystem at PageFair, and Chief Innovation Officer of The Irish Times. He has a PhD from the University of Cambridge, and is a Fellow of the Royal Historical Society.
Tech companies will probably have to adapt to two privacy trends.
“As lawmakers and regulators in Europe and in the United States start to think of “purpose specification” as a tool for anti-trust enforcement, tech giants should beware.”
First, the GDPR is emerging as a de facto international standard.
In the coming years, the application of GDPR-like laws for commercial use of consumers’ personal data in the EU, Britain (post-EU), Japan, India, Brazil, South Korea, Malaysia, Argentina, and China will bring more than half of global GDP under a similar standard.
Whether this emerging standard helps or harms United States firms will be determined by whether the United States enacts and actively enforces robust federal privacy laws. Unless there is a federal GDPR-like law in the United States, there may be a degree of friction and the potential of isolation for United States companies.
However, there is an opportunity in this trend. The United States can assume the global lead by doing two things. First, enact a federal law that borrows from the GDPR, including a comprehensive definition of “personal data”, and robust “purpose specification”. Second, invest in world-leading regulation that pursues test cases, and defines practical standards. Cutting edge enforcement of common principles-based standards is de facto leadership.
Second, privacy and antitrust law are moving closer to each other, and might squeeze big tech companies very tightly indeed.
Big tech companies “cross-use” user data from one part of their business to prop up others. The result is that a company can leverage all the personal information accumulated from its users in one line of business, and for one purpose, to dominate other lines of business too.
This is likely to have anti-competitive effects. Rather than competing on the merits, the company can enjoy the unfair advantage of massive network effects even though it may be starting from scratch in a new line of business. This stifles competition and hurts innovation and consumer choice.
Antitrust authorities in other jurisdictions have addressed this. In 2015, the Belgian National Lottery was fined for re-using personal information acquired through its monopoly for a different, and incompatible, line of business.
As lawmakers and regulators in Europe and in the United States start to think of “purpose specification” as a tool for anti-trust enforcement, tech giants should beware.
John Miller
John Miller is the VP for Global Policy and Law at the Information Technology Industry Council (ITI), a D.C. based advocate group for the high tech sector. Miller leads ITI’s work on cybersecurity, privacy, surveillance, and other technology and digital policy issues.
Data has long been the lifeblood of innovation. And protecting that data remains a priority for individuals, companies and governments alike. However, as times change and innovation progresses at a rapid rate, it’s clear the laws protecting consumers’ data and privacy must evolve as well.
“Data has long been the lifeblood of innovation. And protecting that data remains a priority for individuals, companies and governments alike.”
As the global regulatory landscape shifts, there is now widespread agreement among business, government, and consumers that we must modernize our privacy laws, and create an approach to protecting consumer privacy that works in today’s data-driven reality, while still delivering the innovations consumers and businesses demand.
More and more, lawmakers and stakeholders acknowledge that an effective privacy regime provides meaningful privacy protections for consumers regardless of where they live. Approaches, like the framework ITI released last fall, must offer an interoperable solution that can serve as a model for governments worldwide, providing an alternative to a patchwork of laws that could create confusion and uncertainty over what protections individuals have.
Companies are also increasingly aware of the critical role they play in protecting privacy. Looking ahead, the tech industry will continue to develop mechanisms to hold us accountable, including recommendations that any privacy law mandate companies identify, monitor, and document uses of known personal data, while ensuring the existence of meaningful enforcement mechanisms.
Nuala O’Connor
Nuala O’Connor is president and CEO of the Center for Democracy & Technology, a global nonprofit committed to the advancement of digital human rights and civil liberties, including privacy, freedom of expression, and human agency. O’Connor has served in a number of presidentially appointed positions, including as the first statutorily mandated chief privacy officer in U.S. federal government when she served at the U.S. Department of Homeland Security. O’Connor has held senior corporate leadership positions on privacy, data, and customer trust at Amazon, General Electric, and DoubleClick. She has practiced at several global law firms including Sidley Austin and Venable. She is an advocate for the use of data and internet-enabled technologies to improve equity and amplify marginalized voices.
For too long, Americans’ digital privacy has varied widely, depending on the technologies and services we use, the companies that provide those services, and our capacity to navigate confusing notices and settings.
“Americans deserve comprehensive protections for personal information – protections that can’t be signed, or check-boxed, away.”
We are burdened with trying to make informed choices that align with our personal privacy preferences on hundreds of devices and thousands of apps, and reading and parsing as many different policies and settings. No individual has the time nor capacity to manage their privacy in this way, nor is it a good use of time in our increasingly busy lives. These notices and choices and checkboxes have become privacy theater, but not privacy reality.
In 2019, the legal landscape for data privacy is changing, and so is the public perception of how companies handle data. As more information comes to light about the effects of companies’ data practices and myriad stewardship missteps, Americans are surprised and shocked about what they’re learning. They’re increasingly paying attention, and questioning why they are still overburdened and unprotected. And with intensifying scrutiny by the media, as well as state and local lawmakers, companies are recognizing the need for a clear and nationally consistent set of rules.
Personal privacy is the cornerstone of the digital future people want. Americans deserve comprehensive protections for personal information – protections that can’t be signed, or check-boxed, away. The Center for Democracy & Technology wants to help craft those legal principles to solidify Americans’ digital privacy rights for the first time.
Chris Baker
Chris Baker is Senior Vice President and General Manager of EMEA at Box.
Last year saw data privacy hit the headlines as businesses and consumers alike were forced to navigate the implementation of GDPR. But it’s far from over.
“…customers will have trust in a business when they are given more control over how their data is used and processed”
2019 will be the year that the rest of the world catches up to the legislative example set by Europe, as similar data regulations come to the forefront. Organizations must ensure they are compliant with regional data privacy regulations, and more GDPR-like policies will start to have an impact. This can present a headache when it comes to data management, especially if you’re operating internationally. However, customers will have trust in a business when they are given more control over how their data is used and processed, and customers can rest assured knowing that no matter where they are in the world, businesses must meet the highest bar possible when it comes to data security.
Starting with the U.S., 2019 will see larger corporations opt-in to GDPR to support global business practices. At the same time, local data regulators will lift large sections of the EU legislative framework and implement these rules in their own countries. 2018 was the year of GDPR in Europe, and 2019 be the year of GDPR globally.
Christopher Wolf
Christopher Wolf is the Founder and Chair of the Future of Privacy Forum think tank, and is senior counsel at Hogan Lovells focusing on internet law, privacy and data protection policy.
With the EU GDPR in effect since last May (setting a standard other nations are emulating),
“Regardless of the outcome of the debate over a new federal privacy law, the issue of the privacy and protection of personal data is unlikely to recede.”
with the adoption of a highly-regulatory and broadly-applicable state privacy law in California last Summer (and similar laws adopted or proposed in other states), and with intense focus on the data collection and sharing practices of large tech companies, the time may have come where Congress will adopt a comprehensive federal privacy law. Complicating the adoption of a federal law will be the issue of preemption of state laws and what to do with the highly-developed sectoral laws like HIPPA and Gramm-Leach-Bliley. Also to be determined is the expansion of FTC regulatory powers. Regardless of the outcome of the debate over a new federal privacy law, the issue of the privacy and protection of personal data is unlikely to recede.
Powered by WPeMatico
Dataiku raises $101 million for its collaborative data science platform
Dataiku wants to turn buzzwords into an actual service. The company has been focused on data tools for many years, before everybody started talking about big data, data science and machine learning.
And the company just raised $101 million in a round led by Iconiq Capital, with Alven Capital, Battery Ventures, Dawn Capital and FirstMark Capital also participating.
If you’re generating a lot of data, Dataiku helps you find a meaning behind data sets. First, you import your data by connecting Dataiku to your storage system. The platform supports dozens of database formats and sources — Hadoop, NoSQL, images, you name it.
You can then use Dataiku to visualize your data, clean your data set, run some algorithms on your data in order to build a machine learning model, deploy it and more. Dataiku has a visual coding tool, or you can use your own code.
But Dataiku isn’t just a tool for data scientists. Even if you’re a business analyst, you can visualize and extract data from Dataiku directly. And because of its software-as-a-service approach, your entire team of data scientists and data analysts can collaborate on Dataiku.
Clients use it to track churn, detect fraud, forecast demand, optimize lifetime values and more. Customers include General Electric, Sephora, Unilever, KUKA, FOX and BNP Paribas.
With today’s funding round, the company plans to double its staff. The company currently works with 200 people in New York, Paris and London. It plans to open offices in Singapore and Sydney, as well.
Powered by WPeMatico
AtScale lands $50 million investment led by Morgan Stanley
AtScale, the startup that helps companies move massive amounts of data into business intelligence and analytics tools, announced a $50 million Series D round today.
Morgan Stanley led the round, with previous investors Storm Ventures and Atlantic Bridge joining in. New investor Wells Fargo also participated. The funding comes almost exactly a year after the company announced its $25 million Series C. Today’s funding brings the total amount raised to $120 million.
Bringing on an institutional investor like Morgan Stanley is often a signal that the company has reached the stage where it is at least beginning to think about the possibility of going public at some point in the future. AtScale CEO Chris Lynch acknowledged such a connection without making any broad commitment (as you would expect). “We are not close to being IPO-ready, but that was a future consideration in selecting Morgan Stanley,” Lynch told TechCrunch.
What the company does is help take big data and move it into tools where customers can make better use of it. AtScale co-founder Dave Mariani used to be at Yahoo where he helped pioneer the use of big data in the 2009/2010 timeframe. Unfortunately, systems at the time couldn’t deal with the volume of data — and that is still a problem, one that AtScale says it is designed to solve. “We take a bunch of data silos and put a semantic layer across the data platforms and expose them in a consistent way,” Mariani told TechCrunch last year at the time of the Series C round. This allows a company to get a big picture view of their data, rather than consuming it in smaller chunks.
AtScale reported a banner year, bringing on 50 new customers across their target verticals of retail, financial services, advertising and digital sales. These include Rakuten, Dell Technologies, TD Bank and Toyota. What’s more, the company stretched out this year, taking advantage of the last funding round to expand more into international markets in Europe and Asia.
The company was founded in 2013 and is based in San Mateo, California.
Powered by WPeMatico
Fivetran announces $15M Series A to build automated data pipelines
Fivetran, a startup that builds automated data pipelines between data repositories and cloud data warehouses and analytics tools, announced a $15 million Series A investment led by Matrix Partners.
Fivetran helps move data from source repositories like Salesforce and NetSuite to data warehouses like Snowflake or analytics tools like Looker. Company CEO and co-founder George Fraser says the automation is the key differentiator here between his company and competitors like Informatica and SnapLogic.
“What makes Fivetran different is that it’s an automated data pipeline to basically connect all your sources. You can access your data warehouse, and all of the data just appears and gets kept updated automatically,” Fraser explained. While he acknowledges that there is a great deal of complexity behind the scenes to drive that automation, he stresses that his company is hiding that complexity from the customer.
The company launched out of Y Combinator in 2012, and other than $4 million in seed funding along the way, it has relied solely on revenue up until now. That’s a rather refreshing approach to running an enterprise startup, which typically requires piles of cash to build out sales and marketing organizations to compete with the big guys they are trying to unseat.
One of the key reasons they’ve been able to take this approach has been the company’s partner strategy. Having the ability to get data into another company’s solution with a minimum of fuss and expense has attracted data-hungry applications. In addition to the previously mentioned Snowflake and Looker, the company counts Google BigQuery, Microsoft Azure, Amazon Redshift, Tableau, Periscope Data, Salesforce, NetSuite and PostgreSQL as partners.
Ilya Sukhar, general partner at Matrix Partners, who will be joining the Fivetran board under the terms of deal sees a lot of potential here. “We’ve gone from companies talking about the move to the cloud to preparing to execute their plans, and the most sophisticated are making Fivetran, along with cloud data warehouses and modern analysis tools, the backbone of their analytical infrastructure,” Sukhar said in a statement.
They currently have 100 employees spread out across four offices in Oakland, Denver, Bangalore and Dublin. They boast 500 customers using their product including Square, WeWork, Vice Media and Lime Scooters, among others.
Powered by WPeMatico
The economics and tradeoffs of ad-funded smart city tech
In order to have innovative smart city applications, cities first need to build out the connected infrastructure, which can be a costly, lengthy, and politicized process. Third-parties are helping build infrastructure at no cost to cities by paying for projects entirely through advertising placements on the new equipment. I try to dig into the economics of ad-funded smart city projects to better understand what types of infrastructure can be built under an ad-funded model, the benefits the strategy provides to cities, and the non-obvious costs cities have to consider.
Consider this an ongoing discussion about Urban Tech, its intersection with regulation, issues of public service, and other complexities that people have full PHDs on. I’m just a bitter, born-and-bred New Yorker trying to figure out why I’ve been stuck in between subway stops for the last 15 minutes, so please reach out with your take on any of these thoughts: @Arman.Tabatabai@techcrunch.com.
Using ads to fund smart city infrastructure at no cost to cities
When we talk about “Smart Cities”, we tend to focus on these long-term utopian visions of perfectly clean, efficient, IoT-connected cities that adjust to our environment, our movements, and our every desire. Anyone who spent hours waiting for transit the last time the weather turned south can tell you that we’ve got a long way to go.
But before cities can have the snazzy applications that do things like adjust infrastructure based on real-time conditions, cities first need to build out the platform and technology-base that applications can be built on, as McKinsey’s Global Institute explained in an in-depth report released earlier this summer. This means building out the network of sensors, connected devices and infrastructure needed to track city data.
However, reaching the technological base needed for data gathering and smart communication means building out hard physical infrastructure, which can cost cities a ton and can take forever when dealing with politics and government processes.
Many cities are also dealing with well-documented infrastructure crises. And with limited budgets, local governments need to spend public funds on important things like roads, schools, healthcare and nonsensical sports stadiums which are pretty much never profitable for cities (I’m a huge fan of baseball but I’m not a fan of how we fund stadiums here in the states).
As city infrastructure has become increasingly tech-enabled and digitized, an interesting financing solution has opened up in which smart city infrastructure projects are built by third-parties at no cost to the city and are instead paid for entirely through digital advertising placed on the new infrastructure.
I know – the idea of a city built on ad-revenue brings back soul-sucking Orwellian images of corporate overlords and logo-paved streets straight out of Blade Runner or Wall-E. Luckily for us, based on our discussions with developers of ad-funded smart city projects, it seems clear that the economics of an ad-funded model only really work for certain types of hard infrastructure with specific attributes – meaning we may be spared from fire hydrants brought to us by Mountain Dew.
While many factors influence the viability of a project, smart infrastructure projects seem to need two attributes in particular for an ad-funded model to make sense. First, the infrastructure has to be something that citizens will engage – and engage a lot – with. You can’t throw a screen onto any object and expect that people will interact with it for more than 3 seconds or that brands will be willing to pay to throw their taglines on it. The infrastructure has to support effective advertising.
Second, the investment has to be cost-effective, meaning the infrastructure can only cost so much. A third-party that’s willing to build the infrastructure has to believe they have a realistic chance of generating enough ad-revenue to cover the costs of the projects, and likely an amount above that which could lead to a reasonable return. For example, it seems unlikely you’d find someone willing to build a new bridge, front all the costs, and try to fund it through ad-revenue.
When is ad-funding feasible? A case study on kiosks and LinkNYC
A LinkNYC kiosk enabling access to the internet in New York on Saturday, February 20, 2016. Over 7500 kiosks are to be installed replacing stand alone pay phone kiosks providing free wi-fi, internet access via a touch screen, phone charging and free phone calls. The system is to be supported by advertising running on the sides of the kiosks. ( Richard B. Levine) (Photo by Richard Levine/Corbis via Getty Images)
To get a better understanding of the types of smart city hardware that might actually make sense for an ad-funded model, we can look at the engagement levels and cost structures of smart kiosks, and in particular, the LinkNYC project. Smart kiosks – which provide free WiFi, connectivity and real-time services to citizens – have been leading examples of ad-funded smart city projects. Innovative companies like Intersection (developers of the LinkNYC project), SmartLink, IKE, Soofa, and others have been helping cities build out kiosk networks at little-to-no cost to local governments.
LinkNYC provides public access to much of its data on the New York City Open-Data website. Using some back-of-the-envelope math and a hefty number of assumptions, we can try to get to a very rough range of where cost and engagement metrics generally have to fall for an ad-funded model to make sense.
To try and retrace considerations for the developers’ investment decision, let’s first look at the terms of the deal signed with New York back in 2014. The agreement called for a 12-year franchise period, during which at least 7,500 Link kiosks would be deployed across the city in the first eight years at an expected project cost of more than $200 million. As part of its solicitation, the city also required the developers to pay the greater of either a minimum annual payment of at least $17.5 million or 50 percent of gross revenues.
Let’s start with the cost side – based on an estimated project cost of around $200 million for at least 7,500 Links, we can get to an estimated cost per unit of $25,000 – $30,000. It’s important to note that this only accounts for the install costs, as we don’t have data around the other cost buckets that the developers would also be on the hook for, such as maintenance, utility and financing costs.
Source: LinkNYC, NYC.gov, NYCOpenData
Turning to engagement and ad-revenue – let’s assume that the developers signed the deal with the expectations that they could at least breakeven – covering the install costs of the project and minimum payments to the city. And for simplicity, let’s assume that the 7,500 links were going to be deployed at a steady pace of 937-938 units per year (though in actuality the install cadence has been different). In order for the project to breakeven over the 12-year deal period, developers would have to believe each kiosk could generate around $6,400 in annual ad-revenue (undiscounted).
Source: LinkNYC, NYC.gov, NYCOpenData
The reason the kiosks can generate this revenue (and in reality a lot more) is because they have significant engagement from users. There are currently around 1,750 Links currently deployed across New York. As of November 18th, LinkNYC had over 720,000 weekly subscribers or around 410 weekly subscribers per Link. The kiosks also saw an average of 18 million sessions per week, or 20-25 weekly sessions per subscriber, or around 10,200 weekly sessions per kiosk (seasonality might even make this estimate too low).
And when citizens do use the kiosks, they use it for a long time! The average session for each Link unit was four minutes and six seconds. The level of engagement makes sense since city-dwellers use these kiosks in time or attention-intensive ways, such making phone calls, getting directions, finding information about the city, or charging their phones.
The analysis here isn’t perfect, but now we at least have a (very) rough idea of how much smart kiosks cost, how much engagement they see, and the amount of ad-revenue developers would have to believe they could realize at each unit in order to ultimately move forward with deployment. We can use these metrics to help identify what types of infrastructure have similar profiles and where an ad-funded project may make sense.
Bus stations, for example, may cost about $10,000 – $15,000, which is in a similar cost range as smart kiosks. According to the MTA, the NYC bus system sees over 11.2 million riders per week or nearly 700 riders per station per week. Rider wait times can often be five-to-ten minutes in length if not longer. Not to mention bus stations already have experience utilizing advertising to a certain degree. Projects like bike-share docking stations and EV charging stations also seem to fit similar cost profiles while having high engagement.
And interactions with these types of infrastructure are ones where users may be more receptive to ads, such as an EV charging station where someone is both physically engaging with the equipment and idly looking to kill up sometimes up to 30 minutes of time as they charge up. As a result, more companies are using advertising models to fund projects that fit this mold, like Volta, who uses advertising to offer charging stations free to citizens.
The benefits of ad-funding come with tradeoffs for cities
When it makes sense for cities and third-party developers, advertising-funded smart city infrastructure projects can unlock a tremendous amount of value for a city. The benefits are clear – cities pay nothing, citizens are offered free connectivity and real-time information on local conditions, and smart infrastructure is built and can possibly be used for other smart city applications down the road, such as using locational data tracking to improve city zoning and congestion.
Yes, ads are usually annoying – but maybe understanding that advertising models only work for specific types of smart city projects may help quell fears that future cities will be covered inch-to-inch in mascots. And ads on projects like LinkNYC promote local businesses and can tap into idiosyncratic conditions and preferences of regional communities – LinkNYC previously used real-time local transit data to display beer ads to subway riders that were facing heavy delays and were probably in need of a drink.
Like everyone’s family photos from Thanksgiving, the picture here is not all roses, however, and there are a lot of deep-rooted issues that exist under the surface. Third-party developed, advertising-funded infrastructure comes with externalities and less obvious costs that have been fairly criticized and debated at length.
When infrastructure funding is derived from advertising, concerns arise over whether services will be provided equitably across communities. Many fear that low-income or less-trafficked communities that generate less advertising demand could end up having poor infrastructure and maintenance.
Even bigger points of contention as of late have been issues around data consent and treatment. I won’t go into much detail on the issue since it’s incredibly complex and warrants its own lengthy dissertation (and many have already been written).
But some of the major uncertainties and questions cities are trying to answer include: If third-parties pay for, manage and operate smart city projects, who should own data on citizens’ living behavior? How will citizens give consent to provide data when tracking systems are built into the environment around them? How can the data be used? How granular can the data get? How can we assure citizens’ information is secure, especially given the spotty track records some of the major backers of smart city projects have when it comes to keeping our data safe?
The issue of data treatment is one that no one has really figured out yet and many developers are doing their best to work with cities and users to find a reasonable solution. For example, LinkNYC is currently limited by the city in the types of data they can collect. Outside of email addresses, LinkNYC doesn’t ask for or collect personal information and doesn’t sell or share personal data without a court order. The project owners also make much of its collected data publicly accessible online and through annually published transparency reports. As Intersection has deployed similar smart kiosks across new cities, the company has been willing to work through slower launches and pilot programs to create more comfortable policies for local governments.
But consequential decisions related to third-party owned smart infrastructure are only going to become more frequent as cities become increasingly digitized and connected. By having third-parties pay for projects through advertising revenue or otherwise, city budgets can be focused on other vital public services while still building the efficient, adaptive and innovative infrastructure that can help solve some of the largest problems facing civil society. But if that means giving up full control of city infrastructure and information, cities and citizens have to consider whether the benefits are worth the tradeoffs that could come with them. There is a clear price to pay here, even when someone else is footing the bill.
And lastly, some reading while in transit:
- How California’s Efforts to Prevent Wildfires Reflect a National Crisis on Climate Change – The New Yorker, Carolyn Kormann
- A’s Propose Crazy-Looking Howard Terminal Stadium, Financial Details Still Uncertain – Field of Schemes, Neil deMause
- Berlin’s Massive Housing Push Sparks a Debate About the City’s Future – CityLab, Feargus O’Sullivan
- Magnetic Levitation: The Return of Transport’s Great ‘What If?’ – The Guardian, Christopher Beanland
- Addis Ababa Has Overtaken Dubai as the World’s Gateway into Africa – Quartz, Abdi Latif Dahir
- Planning a Neighborhood Square – The Western Planner, Dr. Suzanne H. Crowhurst Lennard
Powered by WPeMatico
Nvidia launches Rapids to help bring GPU acceleration to data analytics
Nvidia, together with partners like IBM, HPE, Oracle, Databricks and others, is launching a new open-source platform for data science and machine learning today. Rapids, as the company is calling it, is all about making it easier for large businesses to use the power of GPUs to quickly analyze massive amounts of data and then use that to build machine learning models.
“Businesses are increasingly data-driven,” Nvidia’s VP of Accelerated Computing Ian Buck told me. “They sense the market and the environment and the behavior and operations of their business through the data they’ve collected. We’ve just come through a decade of big data and the output of that data is using analytics and AI. But most it is still using traditional machine learning to recognize complex patterns, detect changes and make predictions that directly impact their bottom line.”
The idea behind Rapids then is to work with the existing popular open-source libraries and platforms that data scientists use today and accelerate them using GPUs. Rapids integrates with these libraries to provide accelerated analytics, machine learning and — in the future — visualization.
Rapids is based on Python, Buck noted; it has interfaces that are similar to Pandas and Scikit, two very popular machine learning and data analysis libraries, and it’s based on Apache Arrow for in-memory database processing. It can scale from a single GPU to multiple notes and IBM notes that the platform can achieve improvements of up to 50x for some specific use cases when compared to running the same algorithms on CPUs (though that’s not all that surprising, given what we’ve seen from other GPU-accelerated workloads in the past).
Buck noted that Rapids is the result of a multi-year effort to develop a rich enough set of libraries and algorithms, get them running well on GPUs and build the relationships with the open-source projects involved.
“It’s designed to accelerate data science end-to-end,” Buck explained. “From the data prep to machine learning and for those who want to take the next step, deep learning. Through Arrow, Spark users can easily move data into the Rapids platform for acceleration.”
Indeed, Spark is surely going to be one of the major use cases here, so it’s no wonder that Databricks, the company founded by the team behind Spark, is one of the early partners.
“We have multiple ongoing projects to integrate Spark better with native accelerators, including Apache Arrow support and GPU scheduling with Project Hydrogen,” said Spark founder Matei Zaharia in today’s announcement. “We believe that RAPIDS is an exciting new opportunity to scale our customers’ data science and AI workloads.”
Nvidia is also working with Anaconda, BlazingDB, PyData, Quansight and scikit-learn, as well as Wes McKinney, the head of Ursa Labs and the creator of Apache Arrow and Pandas.
Another partner is IBM, which plans to bring Rapids support to many of its services and platforms, including its PowerAI tools for running data science and AI workloads on GPU-accelerated Power9 servers, IBM Watson Studio and Watson Machine Learning and the IBM Cloud with its GPU-enabled machines. “At IBM, we’re very interested in anything that enables higher performance, better business outcomes for data science and machine learning — and we think Nvidia has something very unique here,” Rob Thomas, the GM of IBM Analytics told me.
“The main benefit to the community is that through an entirely free and open-source set of libraries that are directly compatible with the existing algorithms and subroutines that their used to — they now get access to GPU-accelerated versions of them,” Buck said. He also stressed that Rapids isn’t trying to compete with existing machine learning solutions. “Part of the reason why Rapids is open source is so that you can easily incorporate those machine learning subroutines into their software and get the benefits of it.”
Powered by WPeMatico
AI could help push Neo4j graph database growth
Graph databases have always been useful to help find connections across a vast data set, and it turns out that capability is quite handy in artificial intelligence and machine learning too. Today, Neo4j, the makers of the open source and commercial graph database platform, announced the release of Neo4j 3.5, which has a number of new features aimed specifically at AI and machine learning.
Neo4j founder and CEO Emil Eifrem says he had recognized the connection between AI and machine learning and graph databases for a while, but he says that it has taken some time for the market to catch up to the idea.
“There has been a lot momentum around AI and graphs…Graphs are very fundamental to AI. At the same time we were seeing some early use cases, but not really broad adoption, and that’s what we’re seeing right now,” he explained.
AI graph uses cases. Graphic: Neo4j
To help advance AI uses cases, today’s release includes a new full text search capability, which Eifrem says has been one of the most requested features. This is important because when you are making connections between entities, you have to be able to find all of the examples regardless of how it’s worded — for example, human versus humans versus people.
Part of that was building their own indexing engine to increase indexing speed, which becomes essential with ever more data to process. “Another really important piece of functionality is that we have improved our data ingestion very significantly. We have 5x end-to-end performance improvements when it comes to importing data. And this is really important for connected feature extraction, where obviously, you need a lot of data to be able to train the machine learning,” he said. That also means faster sorting of data too.
Other features in the new release include improvements to the company’s own Cypher database query language and better visualization of the graphs to give more visibility, which is useful for visualizing how machine learning algorithms work, which is known as AI explainability. They also announced support for the Go language and increased security.
Graph databases are growing increasingly important as we look to find connections between data. The most common use case is the knowledge graph, which is what lets us see connections in a huge data sets. Common examples include who we are connected to on a social network like Facebook, or if we bought one item, we might like similar items on an ecommerce site.
Other use cases include connected feature extraction, a common machine learning training techniques that can look at a lot of data and extract the connections, the context and the relationships for a particular piece of data, such as suspects in a criminal case and the people connected to them.
Neo4j has over 300 large enterprise customers including Adobe, Microsoft, Walmart, UBS and NASA. The company launched in 2007 and has raised $80 million. The last round was $36 million in November 2016.
Powered by WPeMatico