ActiveFence comes out of the shadows with $100M in funding and tech that detects online harm, now valued at $500M+

Published July 27, 2021

Online abuse, disinformation, fraud and other malicious content is growing and getting more complex to track. Today, a startup called ActiveFence is coming out of the shadows to announce significant funding on the back of a surge of large organizations using its services. ActiveFence has quietly built a tech platform to suss out threats as they are being formed and planned to make it easier for trust and safety teams to combat them on platforms.

The startup, co-headquartered in New York and Tel Aviv, has raised $100 million, funding that it will use to continue developing its tools and to continue expanding its customer base. To date, ActiveFence says that its customers include companies in social media, audio and video streaming, file sharing, gaming, marketplaces and other technologies — it has yet to disclose any specific names but says that its tools collectively cover “billions” of users. Governments and brands are two other categories that it is targeting as it continues to expand. It has been around since 2018 and is growing at around 100% annually.

The $100 million being announced today actually covers two rounds: Its most recent Series B led by CRV and Highland Europe, as well as a Series A it never announced led by Grove Ventures and Norwest Venture Partners. Vintage Investment Partners, Resolute Ventures and other unnamed backers also participated. It’s not disclosing valuation but I understand it’s over $500 million.

“We are very honored to be ActiveFence partners from the very earliest days of the company, and to be part of this important journey to make the internet a safer place and see their unprecedented success with the world’s leading internet platforms,” said Lotan Levkowitz, general partner at Grove Ventures, in a statement.

The increased presence of social media and online chatter on other platforms has put a strong spotlight on how those forums are used by bad actors to spread malicious content. ActiveFence’s particular approach is a set of algorithms that tap into innovations in AI (natural language processing) and to map relationships between conversations. It crawls all of the obvious, and less obvious and harder-to-reach parts of the internet to pick up on chatter that is typically where a lot of the malicious content and campaigns are born — some 3 million sources in all — before they become higher-profile issues. It’s built both on the concept of big data analytics as well as understanding that the long tail of content online has a value if it can be tapped effectively.

“We take a fundamentally different approach to trust, safety and content moderation,” Noam Schwartz, the co-founder and CEO, said in an interview. “We are proactively searching the darkest corners of the web and looking for bad actors in order to understand the sources of malicious content. Our customers then know what’s coming. They don’t need to wait for the damage, or for internal research teams to identify the next scam or disinformation campaign. We work with some of the most important companies in the world, but even tiny, super niche platforms have risks.”

The insights that ActiveFence gathers are then packaged up in an API that its customers can then feed into whatever other systems they use to track or mitigate traffic on their own platforms.

ActiveFence is not the only company building technology to help platform operators, governments and brands have a better picture of what is going on in the wider online world. Factmata has built algorithms to better understand and track sentiments online; Primer (which also recently raised a big round) also uses NLP to help its customers track online information, with its customers including government organizations that used its technology to track misinformation during election campaigns; Bolster (formerly called RedMarlin) is another.

Some of the bigger platforms have also gotten more proactive in bringing tracking technology and talent in-house: Facebook acquired Bloomsbury AI several years ago for this purpose; Twitter has acquired Fabula (and is working on a bigger efforts like Birdwatch to build better tools), and earlier this year Discord picked up Sentropy, another online abuse tracker. In some cases, companies that more regularly compete against each other for eyeballs and dollars are even teaming up to collaborate on efforts.

Indeed, it may well be that ultimately there will exist multiple efforts and multiple companies doing good work in this area, not unlike other corners of the world of security, which might need more than one hammer thrown at problems to crack them. In this particular case, the growth of the startup to date, and its effectiveness in identifying early warning signs, is one reason investors have been interested in ActiveFence.

“We are pleased to support ActiveFence in this important mission,” commented Izhar Armony, a general partner at CRV, in a statement. “We believe they are ready for the next phase of growth and that they can maintain leadership in the dynamic and fast-growing trust and safety market.”

“ActiveFence has emerged as a clear leader in the developing online trust and safety category. This round will help the company to accelerate the growth momentum we witnessed in the past few years,” said Dror Nahumi, general partner at Norwest Venture Partners, in a statement.

To guard against data loss and misuse, the cybersecurity conversation must evolve

Instagram will let you appeal post takedowns

Published May 9, 2019

Instagram isn’t just pretty pictures. It now also harbors bullying, misinformation and controversial self-expression content. So today Instagram is announcing a bevvy of safety updates to protect users and give them more of a voice. Most significantly, Instagram will now let users appeal the company’s decision to take down one of their posts.

A new in-app interface (rolling out starting today) over the next few months will let users “get a second opinion on the post,” says Instagram’s head of policy, Karina Newton. A different Facebook moderator will review the post, and restore its visibility if it was wrongly removed, and they’ll inform users of their conclusion either way. Instagram always let users appeal account suspensions, but now someone can appeal a takedown if their post was mistakenly removed for nudity when they weren’t nude or hate speech that was actually friendly joshing.

Blocking vaccine misinfo hashtags

On the misinformation front, Instagram will begin blocking vaccine-related hashtag pages when content surfaced on a hashtag page features a large proportion of verifiably false content about vaccines. If there is some violating content, but under that threshold, Instagram will lock a hashtag into a “Top-only” post, where Recent posts won’t show up, to decrease visibility of problematic content. Instagram says that it will test this approach and expand it to other problematic content genres if it works. Instagram will also be surfacing educational information via a pop-up to people who search for vaccine content, similar to what it’s used in the past for self-harm and opioid content.

Instagram says now that health agencies like the Center for Disease Control and World Health Organization are confirming that VACCINES DO NOT CAUSE AUTISM, it’s comfortable declaring contradictory information as verifiably false, and it can be aggressively demoted on the platform.

The automated system scans and scores every post uploaded to Instagram, checking them against classifiers of prohibited content and what it calls “text-matching banks.” These collections of fingerprinted content it’s already banned have their text indexed and words pulled out of imagery through optical character recognition so Instagram can find posts with the same words later. It’s working on extending this technology to videos, and all the systems are being trained to spot obvious issues like threats, unwanted contact and insults, but also those causing intentional fear-of-missing-out, taunting, shaming and betrayals.

If the AI is confident a post violates policies, it’s taken down and counted as a strike against any hashtag included. If a hashtag has too high of a percentage of violating content, the hashtag will be blocked. If it had fewer strikes, it’d get locked in Top-Only mode. The change comes after stern criticism from CNN and others about how hashtag pages like #VaccinesKill still featured tons of dangerous misinformation as recently as yesterday.

Tally-based suspensions

One other new change announced this week is that Instagram will no longer determine whether to suspend an account based on the percentage of their content that violates policies, but by a tally of total violations within a certain period of time. Otherwise, Newton says, “It would disproportionately benefit those that have a large amount of posts,” because even a large number of violations would be a smaller percentage than a rare violation by someone who doesn’t post often. To prevent bad actors from gaming the system, Instagram won’t disclose the exact time frame or number of violations that trigger suspensions.

Instagram recently announced at F8 several new tests on the safety front, including a “nudge” not to post a potentially hateful comment a user has typed, “away mode” for taking a break from Instagram without deleting your account and a way to “manage interactions” so you can ban people from taking certain actions like commenting on your content or DMing you without blocking them entirely.

The announcements come as Instagram has solidified its central place in youth culture. That means it has intense responsibility to protect its user base from bullying, hate speech, graphic content, drugs, misinformation and extremism. “We work really closely with subject matter experts, raise issues that might be playing out differently on Instagram than Facebook, and we identify gaps where we need to change how our policies are operationalized or our policies are changed,” says Newton.