working from home
The Internet is ripe with toxic content. Social Media companies such as Facebook, Twitter, Instagram, etc., have been using a combination of human content moderators and technology to try to limit the amount of harmful content. With the advancement of AI algorithms, companies such as Spectrum Labs are rushing into the space to use contextual AI algorithms to improve toxic content detection. They’ve just raised a 10 million round of funding in September 2020. It seems content platforms want more sophisticated solutions.
Justin Davis, Co-founder & CEO of Spectrum Labs says, “From usernames created, to search queries entered, to posts made, to images uploaded, the Internet is full of content created by users. Unfortunately, not all of that content is appropriate. Recognizing which content is and isn’t appropriate can be challenging for platforms with outdated moderation technology.”
Multiple Languages Add Complexity to AI Algorithms
Many content platforms are worldwide. They are accessed daily by content creators who are working in multiple languages. Algorithms that use “keywords” as the basis of their search for toxicity may not work well due to nuances in language. It is also a very inefficient approach. This is where contextual AI comes in. Contextual AI is when the metadata around a piece of information is used inside the algorithms in conjunction with the meaning of the speech.
Spectrum Lab's approach bypasses translation in favor of a new technique that aligns embedding across languages. This approach's benefit, versus translation, is that platforms can apply all the work done in English to other languages, which saves significant time.
Davis says, “Imagine you’re running a dating app used across the world. What is acceptable behavior in one culture may not be in another. It becomes obvious pretty quickly that you can’t rely on a list of translated banned words to accurately moderate.”
Spectrum Labs’ contextual AI is built around custom word embeddings. The assumption is that a given user will speak differently on different verticals of a platform. This user’s baseline normal conversation on gaming apps will differ from their behavior on dating apps. This is all due to the context of the content they are creating on the specific platform.
Davis says, “We founded Spectrum Labs because we realized that online toxicity was a data problem, not a tooling problem. No one needed another moderation queue. What they needed was more accurate detection. Our approach solves the accuracy problem and enables moderators to do something they’ve never been able to do: operate at scale, in real-time, across languages.”
Toxic Content Is Nuanced
The difficulty of detecting harmful content is that the intention of the content is often nuanced. Brands are increasingly concerned with the effect of harmful content can have on well-established brands. Toxic content can often be disguised as regular speech that’s hard to detect. Sometimes, even dark humorous content can be toxic content.
Davis says, “Toxic behavior that is pretty obvious, like using profanity, isn’t too hard to detect. What is significantly harder to detect are more nefarious behaviors, like sexual harassment, cyberbullying, CSAM grooming, radicalization, and terrorist recruitment. These behaviors evolve over time and can occur without using banned words.”
Contextual AI can help to understand the behavior of the user who is generating the content. Using meta-information from the user, the historical responses to their content, and the responses to their current content, the AI algorithm can continuously establish a feedback loop that allows it to understand the user's context.
Davis says, "Let's look at a dating app user as an example. That person could be interacting with a few different people on the app. Let's say that, after interacting for a while, that person sends each of their matches an explicitly sexual message. Match one responds back with a similar message. Match two doesn't respond at all. Match three responds with fear, anger or sadness, or another form of toxicity. Our technology analyzes the whole interaction between users, not just single messages or words, and will be able to tell which of these interactions are a problem for the platform."
Amplify Cultural Issues
Going beyond looking at harmful content as simply bad behavior, if you classify blatant discrimination as a type of toxic content, then, in many countries, on content platforms, this can be a significant problem. In India, where caste discrimination is often present on social media and other content platforms, biased or toxic content can amplify society's cultural issues.
Spectrum Labs' platform developed the technology to detect caste discrimination in both Hindi and English.
In western countries, especially on gaming platforms, cyberbullying can also become an important issue. With children spending more and more time on these platforms, western countries are more concerned with detecting cyberbullying behaviors.
Davis says, "If you're just looking at, a single word on a child based platform and you don't do the word embeddings to understand how kids communicate within that social dynamic. Then you're probably not going to be very accurate at detecting when actual cyberbullying is taking place, because there's things like sarcasm, they have their own coded speak."
Automation vs. Human Moderators
With AI algorithms becoming more accurate in detecting toxic content online, content moderation's labor-intensive process can become more automated.
Davis says, “Moderators are passionate people invested in the communities they serve. With us, they have a partner they can trust to give them an accurate idea of what’s happening. They rely on our technology to fuel their efforts.”
In the short term, as more people scrutinize the content that they see online, content moderators will continue to be the key in moderating content on many platforms. But, with the success of contextual AI technology such as Spectrum Labs’, more nuanced toxic content can be detected and flagged for the moderators.
Davis says, “Our mission is to make the Internet a more valuable place for all. Across our customers, which include Riot Games, Mercari, and The Meet Group, we’re protecting 1 billion (and counting) people from toxic behaviors.”
