Text Moderation (Google)

This evaluator leverages Google's advanced text analysis capabilities to assess content across various safety attributes such as toxicity, profanity, and threats. It's designed to identify potentially harmful or sensitive content in text documents.

Configuration

Option	Description	Type	Default	Required	Constraints
Attributes	Specific attributes to check within the text. An empty list implies all are checked.	[]string	[]	false	Enum: TOXICITY, DEROGATORY, VIOLENCE, SEX, INSULT, PROFANITY, THREAT, WEAPONS, PUBLIC_SAFETY, HEALTH, RELIGION, ILLICIT_DRUGS, WAR, FINANCIAL, POLITICAL, LEGAL
Threshold	The confidence threshold to flag an attribute as concerning.	float64	0.7	true	Min: 0 Max: 1

Metrics

This evaluator reports the following metrics to provide detailed insights into the moderation process:

Name	Key	Description	Min	Max
Toxicity	toxicity	The probability that the text is toxic.	0	1
Derogatory	derogatory	The probability that the text is derogatory.	0	1
Violence	violence	The probability that the text contains violent content.	0	1
Sex	sex	The probability that the text contains sexual content.	0	1
Insult	insult	The probability that the text contains insulting content.	0	1
Profanity	profanity	The probability that the text contains profanity.	0	1
Threat	threat	The probability that the text contains threats of violence.	0	1
Weapons	weapons	The probability that the text mentions firearms and weapons.	0	1
Public Safety	public_safety	The probability that the text mentions public safety issues.	0	1
Health	health	The probability that the text mentions health issues.	0	1
Religion	religion	The probability that the text mentions religion and belief.	0	1
Illicit Drugs	illicit_drugs	The probability that the text mentions illicit drugs.	0	1
War	war	The probability that the text mentions war and conflict.	0	1
Financial	financial	The probability that the text mentions finance-related topics.	0	1
Political	political	The probability that the text mentions political content.	0	1
Legal	legal	The probability that the text discusses legal issues.	0	1

Additional Information

You can learn more about Google Cloud's text moderation at Google Cloud's text moderation documentation.