# Documentation

Modelmetry provides advanced guardrails and observability for Large Language Models (LLMs) applications.

### Jump right in

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Getting Started</strong></td><td>Understand Modelmetry's concepts</td><td><a href="https://content.gitbook.com/content/Lnh3RzaEwzTzGfK3wvEc/blobs/LbbRa5viMdrldsdknHYq/3.png">3.png</a></td><td></td><td><a href="broken://pages/JjjojIyKxaiBPzzLwvtg">Broken link</a></td></tr><tr><td><strong>Evaluators</strong></td><td>Discover what you can evaluate</td><td><a href="https://content.gitbook.com/content/Lnh3RzaEwzTzGfK3wvEc/blobs/54u9P5QRv7wLO3HwDJq7/1.png">1.png</a></td><td></td><td><a href="/evaluators/all">Evaluators</a></td></tr></tbody></table>


# Concepts

Definitions of terms found in Modelmetry from its core concepts to its Observability and Safeguard concepts.

**Modelmetry** is a sophisticated platform designed to enhance the safety, quality, and appropriateness of data and models in applications that utilize Large Language Models (LLMs) such as chatbots. It integrates numerous tools to evaluate and monitor your LLM pipelines, ensuring they meet high standards of operation.

Core concepts include:

* **payloads** (data to be evaluated),&#x20;
* **evaluators** (analyzing aspects like safety, sentiment, or custom metrics),&#x20;
* **instances** (customized evaluator setups),&#x20;
* **entries** (records of evaluations), and
* **findings** (observable facts like numeric, boolean, or labeled).

Guardrail terms include:

* **guardrails** (groups of evaluators forming custom evaluation frameworks),&#x20;
* **check** (records of guardrail verifications),
* **outcome** (whether a check passed, failed, or errored out).

Observability terms include:

* **traces** (representing requests),&#x20;
* **spans** (logging specific tasks),&#x20;
* **events** (logging actions), and&#x20;
* **metrics** (quantifying performance aspects).

### Payload

In Modelmetry, a payload is the data that needs to be evaluated or checked. It can consist of `Input`, `Output`, and `Options`.

The Input and Output can be plain text (for simpler cases) or a set of messages (for multi-turn LLM use cases). The options are the usual generative AI request options such as model name, temperature, tools, etc.

### Evaluators

Evaluators are at the heart of Modelmetry. They analyze payloads and focus on specific aspects of them, such as safety and security, sentiment and emotions, helpfulness and quality, or even custom analyses via your own HTTP endpoint or LLM-as-a-Judge evaluators. An evaluator can output built-in metrics as well as custom ones, as well as give a verdict and a score.

### Instances

Nearly all evaluators offer settings to allow you to personalize and fine-tune the evaluations. An **Instance** refers to a specific setup of an evaluator, customized through a configuration schema to suit particular assessment needs. Each instance operates under a unique identifier and a defined set of options that dictate how evaluations are conducted, including thresholds and conditions that trigger different outcomes.

### Entry

An evaluation, or entry, is a record that contains the details about a specific evaluation:

* the evaluator
* the instance's configuration
* the payload
* the output
  * outcome (pass, fail, error)
  * score
  * findings (e.g., metrics, labels)

### Findings

In Modelmetry, findings are observable facts which can be classified as a:

* numeric finding – a quantifiable metric
* boolean findings – a true/false or yes/no observation
* labeled findings – a categorical or enumerable observation

Most evaluators also emit findings, which can then be tracked over time to notice trends or used within guardrail checks to ensure a specific metric doesn't exceed a particular score (and, if so, fails the guardrail check).

### Guardrail

A guardrail is a set of safety and quality controls. In Modelmetry, it's a group of multiple evaluator instances that form a comprehensive evaluation framework for your payloads. You can send payloads to your guardrail from your codebase so it can check whether they pass or fail your set of evaluations.

### Check

A check is a record of some data sent by your codebase that is being checked by a guardrail. It, therefore, contains an outcome (pass or fail) and all the findings generated by the guardrail's evaluators. It also embeds all the Entries for each evaluation from each instance.

#### Outcome

The outcome of a guardrail check can be `pass`, `fail`, or `error`. It would be best if you then handled all possible outcomes in your codebase when checking a payload. We generally recommend continuing the pipeline as normal if it's a `pass` or `error`, and focusing on handling `fail` outcomes (e.g., sending an email alert to a team lead, interrupting the chatbot thread, or even requesting a human to take over).

### Trace

A trace represents a single request or operation in the context of LLM observability. It captures the function's overall input and output, along with metadata such as user information, session details, and tags.&#x20;

A trace serves as a high-level representation of a request, containing multiple observations that log the individual steps or events occurring during that request. Traces help in capturing the full context of execution, such as API calls, prompt details, and other relevant data points.

### Span

A span is a specific type of observation within a trace representing a work duration or a particular task. Spans are used to log discrete units of work, such as function calls or interactions within the broader trace. They help understand the timing, sequencing, and nested structure of operations, making it easier to pinpoint where specific tasks start and end within a trace. Spans can represent tasks like API calls, data processing, or model generation events.

### Events

Events log specific actions or incidents during an LLM application's execution flow, helping to provide detailed insight into its behavior. They can be combined with other observations, like spans and generations, to comprehensively view an application's performance.

### Metrics

Metrics are "numeric findings" used to quantify various aspects of the application's performance, such as model usage, quality metrics, token counts, processing times, and costs. These measurements help developers optimize their LLM applications by identifying performance bottlenecks, tracking costs, and monitoring overall efficiency.


# Evaluators

In short, evaluators are small software we run that you can configure which evaluate specific aspects of a payload. A payload is your user's input or your LLM's output. Some evaluators focus on readability, others on JSON structure if you expect JSON from your model, or even safety by checking for prompt injection and jailbreaks.

To get started, you first pick the evaluator of your choice and create an [instance](/platform/evaluators/instances). An instance is essentially a configured evaluator. Nearly all evaluators are configurable so you can run multiple instances of the same evaluator with slightly different settings.

{% hint style="success" %}
[**View a list of all Modelmetry evaluators in the documentation.**](/evaluators/all)
{% endhint %}

In your team, you can list all of Modelmetry evaluators by going to `Evaluators` in the sidebar. The list is searchable and filterable.

### View details of an evaluator

1. Go to `Evaluators`
2. Locate the evaluator you want to see the details of and click the `Details` button

The details page of an evaluator shows you some general information as well as the findings generated by the evaluator, as well as its configuration settings.

In the `Instances` section, you can see a list of your current instances for that evaluator.

<figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FF7R1f2ZzmNTELkKMBd9Z%2FScreenshot%202024-11-26%20at%2018.12.06.jpg?alt=media&amp;token=729100f0-a042-491f-aedd-4cdd99c73148" alt=""><figcaption><p>Example of the details screen for <code>azure.prompt-shields.v1</code></p></figcaption></figure>

### Test an evaluator

1. Go to `Evaluators`
2. Locate the evaluator you want to see the details of and click the `Test` button
3. Fill in the configuration, request, and grading data
   1. If the evaluator uses an LLM, attach the needed secret to it

<figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FKlygE9gEO3fCsOiG60fI%2FScreenshot%202024-11-26%20at%2018.22.52.jpg?alt=media&amp;token=78c844e7-2979-422a-931f-5656ec3a2602" alt=""><figcaption><p>Example of test data</p></figcaption></figure>

<figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2F2kSN3SWifiORs1yjyKWw%2FScreenshot%202024-11-26%20at%2018.23.07.jpg?alt=media&amp;token=79f65e8c-75ae-4f2a-8580-ff398185ccb9" alt=""><figcaption><p>Example of a response when the test is executed</p></figcaption></figure>


# Instances

### List all instances

### Edit an instance

#### Configuration

#### Secrets

#### Grading


# Grading

### Expression Language

<https://github.com/expr-lang/expr>

### Assessments

#### Action

#### Expression

#### Message


# Automations

An automation is a specific flow that, given a specific rule will, if valid, execute a set of pre-defined actions. At this moment, automations get assessed for each new ingested trace and check.

You can view all your existing automations by clicking on `Automations` in the sidebar.

| Automation Attribute | Description                                                                                                                                                                                                                                         |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name**             | A name for the automation to easily identify it later.                                                                                                                                                                                              |
| **Enabled**          | Whether the automation is currently enabled. If disabled, the automation will not be evaluated and will never trigger. This is useful for drafts.                                                                                                   |
| **Sampling Rate**    | How often the automation should be evaluated, between 0 and 1. A value of 1 means every time, 0.5 half the time, and 0 never.                                                                                                                       |
| **Rule**             | Each automation has a set of rules that determine when it should trigger. The rules are evaluated against the incoming data.                                                                                                                        |
| **Actions**          | <p>The actions to perform when the automation triggers. You must ensure your actions work with the resource type detected from the rule (<code>trace</code>).<br><br>The currently available actions are:<br>- Send Webhook<br>- Run Evaluation</p> |

{% hint style="warning" %}
**An automation can only target one resource type (i.e., trace, check).** Therefore, an automation rule such as `trace.name == "some-name" && check.outcome == "pass"` will always be skipped. If you need to run the same set of actions for both resources, create two distinct automations.
{% endhint %}

### Create/Edit an automation

To create an automation:

1. Go to `Automations`
2. Click on `+ New`

To edit an automation:

1. Go to `Automations`
2. Click `Edit` for the automation you want to change

### Delete an automation

1. Go to `Automations`
2. Click `Edit` for the automation you want to change
3. Click the `Delete automation` button


# Analytics

odelmetry generates a lot of data to help you understand how your LLM-powered features behave, perform, and evolve over time. It is therefore important for our team to help you get insights and helpful data visualisations in a simple and effective manner.

### Dashboards

Each team has a shared dashboard; so the entire team sees the same data at any time. A dashboard is essentially a grid-layout with the ability for allowed members to add data visualisation widgets of various types.

A dashboard shows data that spans across a pre-defined period: 24h, 3 days, 7 days, 14 days, 30 days, 60 days, or 90 days. This setting can be adjusted via the Period dropdown in the header.

<figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FwGMZYYu2nBn5VJIeiSAr%2Fmodelmetry-dashboard-example.jpg?alt=media&amp;token=03369766-ab4f-4d26-85b8-dc8d9aa2ab63" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
We heard your feedback and are working on having team dashboards as well as member-specific private dashboards with easy switch between all. We are looking for more feedback on this so we can make sure we work on and deliver the best experience for our users. Please contact us at [modelmetry.com/contact](https://modelmetry.com/contact)!
{% endhint %}

### Widgets

A Widget is a self-contained data visualisation card that is placed on a dashboard view. There are different types of widgets which offer different options.&#x20;

A widget can be added to a dashboard via the `Add widget` button in the header.

#### Metric Timeseries

A time series chart for a specific numerical finding (i.e., a metric).&#x20;

| Attribute       | Description                                                                                                                                                                                                                                            |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Finding**     | The metric you want to chart.                                                                                                                                                                                                                          |
| **Aggregator**  | The aggregation method to run over this metric. It is one of `AVG`, `COUNT`, `SUM`.                                                                                                                                                                    |
| **Granularity** | The level of detail for the time interval in the time series. It is one of `month`, `day`, `hour`, `minute`. Please note, the available options also depend on the period used to query the data *(e.g., a 90d period only allows `month` and `day`)*. |

<div align="left"><figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FboZ1IaqJ0ZrSoaiuLl4C%2FScreenshot%202024-11-26%20at%2015.52.43.jpg?alt=media&amp;token=b367cfe4-2d58-4af8-b055-ce2ec2aa3206" alt="" width="366"><figcaption><p>A chart of the average of <code>modelmetry.text-readability.v1/fkgl</code> across a given period.</p></figcaption></figure> <figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2F5jTRIO2u3yxSBUckTmLi%2FScreenshot%202024-11-26%20at%2015.55.50.jpg?alt=media&amp;token=3b47c1cd-f931-41b7-920a-966f9e5fd62f" alt="" width="375"><figcaption><p>A chart of the sum of modelmetry.text-readability.v1/wordcount across a given period.</p></figcaption></figure></div>

#### Metric Indicator

A single figure for a specific numerical finding (i.e., a metric). This is particularly useful when you want to have a higher-level view of a particular metric that you really care about.

| Attribute      | Description                                                                         |
| -------------- | ----------------------------------------------------------------------------------- |
| **Finding**    | The metric you want to chart.                                                       |
| **Aggregator** | The aggregation method to run over this metric. It is one of `AVG`, `COUNT`, `SUM`. |

<div align="left"><figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2Frb1ndxmToHpAr0ao3JG1%2FScreenshot%202024-11-26%20at%2015.58.37.jpg?alt=media&amp;token=78dbc518-302a-4b3e-b400-b1d0991b8794" alt="" width="375"><figcaption><p>Multiple metric indicators laid out on a dashboard view.</p></figcaption></figure></div>

#### Guardrail Outcome Breakdown

A pie chart breaking down the outcomes of all guardrail checks across the given time period.

<div align="left"><figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2F9pWZ7EERyaUGxnOEEVxO%2FScreenshot%202024-11-26%20at%2015.59.26.jpg?alt=media&amp;token=9b638d20-641a-4f55-8655-63a40a1b29c5" alt="" width="354"><figcaption></figcaption></figure></div>


# API Keys

API keys are used to perform operations programmatically on behalf of a user. For example, you will need an API key to use any Modelmetry SDK.

### View all API keys

1. Go to `Settings`
2. Click on `API Keys`
3. View all api keys. You can filter api keys based on their status (Enabled, Disabled, Revoked).

{% hint style="info" %}
If you have lost your API key; you must revoke it and create a new one again. The API key will not be shown again after it's been created.
{% endhint %}

### Create an API key

1. Go to `Settings`
2. Click on `API Keys`
3. Click on `New`
4. Select the member you want to create the API key for
5. Click `Create`
6. The API key will be displayed to you and it will never be shown to you again. Make sure you save and use it securely.

### Disable an API key

1. Go to `Settings`
2. Click on `API Keys`
3. Open the action menu for the api key you want to disable
4. Click on `Disable` and the API key will be immediately disabled
   1. If you were in a filtered view, the API key will disappear for that filtered view.&#x20;
   2. You can see it by selecting the appropriate filter.
5. You can re-enable the API key by following the same steps and clicking on `Enable`

### Revoke an API key

1. Go to `Settings`
2. Click on `API Keys`
3. Open the action menu for the api key you want to revoke
4. Click on `Revoke` and the API key will be immediately revoked

{% hint style="warning" %}
A revoked API key cannot be re-enabled so you will need to create a new API key.
{% endhint %}


# Secrets

Secrets are authentication codes such as API keys, JSON service accounts, auth tokens, given to you by third-party providers that you want Modelmetry to use and store securely.

{% hint style="warning" %}
**We do not store your secrets in plain text.** They are encrypted at rest and only decrypted when used to authenticate with external services. For this reason, we cannot show you the full value of the secret once it has been set (only a small preview).

For the sake of safety, we do recommend you enable usage and billing limits with providers that allow to ensure no bug within or hack into our servers compromises your third-party access and billing usage.
{% endhint %}

### List all secrets

<figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2F8VVak22dJkG9nEJFLUwv%2FScreenshot%202024-11-26%20at%2014.26.03.jpg?alt=media&amp;token=2ec84adb-f94a-4195-8a30-4f60233823df" alt=""><figcaption></figcaption></figure>

1. Go to `Settings`
2. Click on `Secrets`
3. View all secrets

### Create a new secret

1. Go to `Settings`
2. Click on `Secrets`
3. Click on `+ New Secret`
4. Enter the appropriate data:
   1. **Provider**: which third-party is this secret used for (this will be used when you select a specific model in some of our evaluators). **You cannot change the provider once the secret is created.**
      1. OpenAI
      2. Azure
      3. Google
      4. Amazon
      5. Groq
      6. Replicate
      7. Other
   2. **Name**: give it a human-friendly name so your team knows what it refers to, and what to use it for (e.g., `openai-embeddings-api-key`, `google-gemini-json-service-account`). **You cannot change the name once the secret is created.**
   3. **Value**: the actual contents of the secret.
5. Click `Save`
6. The secret will now appear in the table, with a preview

{% hint style="info" %}
Once your secret is saved, we also save a very short preview so you can see, at a glance, what this token is. For example, if you have multiple OpenAI secrets, you will be able to see the last few characters of the token to help you distinguish them.
{% endhint %}

### Update a secret

1. Go to `Settings`
2. Click on `Secrets`
3. Click on the edit icon button
4. Change the contents of the secret's value.
5. Click `Save`

### Delete a secret

{% hint style="danger" %}
You cannot delete a secret which is still being used in one or more evaluator instances. Make sure you detach the secret you want to delete from instances in order to delete it.
{% endhint %}

* Go to `Settings`
* Click on `Secrets`
* Click on the edit icon button
* Click `Delete`


# Team

In Modelmetry, a team is the top-level organisational unit that contains all the monitoring and observability data, as well as users (members) and team-specific settings.

For team management, read the following:

* [Memberships](/platform/team/memberships)
* [Invitations](/platform/team/invitations)
* [Roles & Permissions](/platform/team/roles-and-permissions)

{% hint style="info" %}
Throughout the platform, if you dig a little deeper (e.g., API), you will see "Tenant" being used. It can be interchangeably used instead of "Team" – except in our API/backend where "Tenant .
{% endhint %}


# Memberships

A membership is the formalisation of a user belonging to a team, assuming a specific role.&#x20;

{% hint style="info" %}
In order to make a user a member of your team, [**you must invite them**](/platform/team/invitations).
{% endhint %}

### View all team members

1. Go to `Settings`
2. Click on `Team`
3. You can then see all members for the current team.

### Change a member's role

1. Go to `Settings`
2. Click on `Team`
3. Find the member you want to edit the role of
4. Select the user's role from the dropdown

   <figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FQghhwj6t9ZcG1IT5OBLn%2FScreenshot%202024-11-26%20at%2013.32.50.jpg?alt=media&amp;token=e214da37-928e-41cd-9776-e4b93a35632b" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
The "Owner" role cannot be assigned manually to another user. It is automatically assigned to the creator of the team and cannot be changed.
{% endhint %}

### Stop a membership

1. Go to `Settings`
2. Click on `Team`
3. Find the membership you want to stop
4. Click `Revoke` in the action menu

   <figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2Fa2BYwExMyHzaIaMn4Vmi%2FScreenshot%202024-11-26%20at%2013.35.41.jpg?alt=media&amp;token=651e5df0-9ddf-4329-b879-fc6fa18356a6" alt=""><figcaption></figcaption></figure>

Once a membership is stopped, the user can be re-invited if they must rejoin the team.


# Invitations

An invitation is required to invite any user to your team. An invitation can be in one of the following statuses:

* **Open**: the user is able to reply to the invitation
* **Expired**: the invitation is no longer open as the user took too long to reply
* **Accepted**: the user has accepted the invitation and became a member
* **Declined**: the user has rejected the invitation
* **Withdrawn**: the team has canceled their invitation

Upon acceptance, a user will automatically receive the assigned role from the invitation for their membership.

### View all invitations

1. Go to `Settings`
2. Click on `Team`
3. Click on `Invitations`
4. You can then see all invitations for the current team.

### Invite a user

{% hint style="info" %}
Once an invitation is sent, **it will remain open for one week** from the moment it was sent. After then, it will have an expired status and you will need to re-invite the user again.
{% endhint %}

1. Go to `Settings`
2. Click on `Team`
3. Click on `Invitations`
4. Click the `Invite User` button
5. Enter the required information:
   1. Email
   2. Role
6. Click the `Send invitation`

Once an invitation is sent, a message will land in the invitee's email inbox. They will need to click the link to accept or decline the invitation.&#x20;

If the user does not receive the email for whatever reason, you can copy and paste the invitation link directly from the list of invitees.

<div align="left"><figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FmAILF2oOVAaNhQZbW4ib%2FScreenshot%202024-11-26%20at%2013.24.04.jpg?alt=media&amp;token=115ac637-b147-4c29-8d43-1fdac3beeed5" alt="" width="375"><figcaption></figcaption></figure></div>

### Withdraw an invitation

Any team member with the `members.manage` permission can withdraw the invitation whilst it's open.

1. Go to `Settings`
2. Click on `Team`
3. Click on `Invitations`&#x20;
4. Click the invitation's right-hand menu
5. Click on `Withdraw` <br>

   <figure><img src="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2Fq1IjZqz77ixkocjgwnnE%2Fwithdraw-invitation.jpg?alt=media&amp;token=e5f81be2-e1ea-4c57-a0fa-75a19c01217f" alt=""><figcaption></figcaption></figure>
6. A confirmation message will appear, and once your confirm the withdrawal, the invitation will be closed and the invitee won't be able to use it.


# Roles & Permissions

### Default Roles

Default built-in roles (`owner`, `administrator`, `member`) are managed by Modelmetry and their permissions cannot be changed by users. Only Modelmetry will make changes when required – always informing team administrators and owners by email with enough notice.

#### Owner

The owner role is automatically assigned to the creator of a team. It essentially has all permissions and privileges within that team and their role cannot be changed by other team members.

#### Administrator

An administrator has all roles except billing within the team.

#### Member

A member has the roles needed to use the platform without being allowed critical permissions.

### Custom Roles

If you are on a suitable billing plan, you will be able to create, update, and assign custom roles which you can completely customise to your business' needs.

{% embed url="<https://youtu.be/0XUtNBwbXHw>" %}

### Permissions

<table><thead><tr><th width="250">Permission</th><th>Description</th></tr></thead><tbody><tr><td>analytics.read</td><td>Read and update the team's dashboard.</td></tr><tr><td>apikeys.manage</td><td>Manage API keys.</td></tr><tr><td>apikeys.read</td><td>View API keys.</td></tr><tr><td>automations.create</td><td>Create automations.</td></tr><tr><td>automations.delete</td><td>Delete automations.</td></tr><tr><td>automations.read</td><td>Read automations.</td></tr><tr><td>automations.update</td><td>Update automations.</td></tr><tr><td>billing.manage</td><td>Manage the team's billing.</td></tr><tr><td>billing.usage.read</td><td>View the team's usage.</td></tr><tr><td>checks.read</td><td>Read checks.</td></tr><tr><td>guardrails.create</td><td>Create guardrails.</td></tr><tr><td>guardrails.delete</td><td>Delete guardrails.</td></tr><tr><td>guardrails.read</td><td>Search and view guardrails.</td></tr><tr><td>guardrails.update</td><td>Update guardrails.</td></tr><tr><td>instances.create</td><td>Create instances of evaluators.</td></tr><tr><td>instances.delete</td><td>Delete instances of evaluators.</td></tr><tr><td>instances.read</td><td>View instances of evaluators.</td></tr><tr><td>instances.update</td><td>Update instances of evaluators.</td></tr><tr><td>members.manage</td><td>Invite and withdraw invitations. Revoke memberships.</td></tr><tr><td>members.read</td><td>View memberships.</td></tr><tr><td>roles.manage</td><td>Create custom roles, and update assigned permissions.</td></tr><tr><td>roles.read</td><td>View roles.</td></tr><tr><td>secrets.manage</td><td>Create, update, and delete secrets.</td></tr><tr><td>secrets.read</td><td>View secrets.</td></tr></tbody></table>


# Billing

### Main Subscription

### Add-ons

### Usage

### Billing Portal


# List of Evaluators

### Modelmetry evaluators

<table data-card-size="large" data-view="cards"><thead><tr><th data-card-target data-type="content-ref"></th><th data-hidden></th><th data-hidden></th><th data-hidden></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><a href="/evaluators/modelmetry.boolean-llm-as-judge.v1">Boolean LLM-as-Judge (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.score-llm-as-judge.v1">Score LLM-as-Judge (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.http-request.v1">HTTP Request (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.word-counter.v1">Word Counter (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.emotion-analysis.v1">Emotion Analysis (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.language-detector.v1">Language Detector (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/modelmetry.competitor-blocklist.v1">Competitor Blocklist (Modelmetry)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr></tbody></table>

### Google evaluators

<table data-card-size="large" data-view="cards"><thead><tr><th data-card-target data-type="content-ref"></th><th data-hidden></th><th data-hidden></th><th data-hidden></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><a href="/evaluators/google.text-moderation.v1">Text Moderation (Google)</a></td><td>Google Cloud Text Moderation</td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr><tr><td><a href="/evaluators/google.dlp-pii-detector.v1">DLP PII Detector (Google)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr></tbody></table>

### Azure evaluators

<table data-card-size="large" data-view="cards"><thead><tr><th data-card-target data-type="content-ref"></th><th data-hidden></th><th data-hidden></th><th data-hidden></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><a href="/evaluators/azure.prompt-shields.v1">Prompt Shields (Azure)</a></td><td></td><td></td><td></td><td><a href="https://1301655980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLnh3RzaEwzTzGfK3wvEc%2Fuploads%2FaZ3oIPn9JiTi3gEdltpD%2Fflat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg?alt=media&amp;token=bb004ff9-0ccf-4d34-be1b-970c357c65ae">flat-design-illustration-peope-with-color-year-2022_23-2149217364.jpg</a></td></tr></tbody></table>


# Prompt Shields (Azure)

This evaluator leverages Azure's Prompt Shields API to detect and prevent adversarial inputs targeting user prompts and documents. It's specially designed to enhance the safety of interactions with language models by analyzing and flagging potential attacks.

### Configuration <a href="#configuration" id="configuration"></a>

No specific configuration parameters are required for this evaluator as it operates with default settings designed to optimize performance and security.

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metrics to provide insights into detected adversarial activities:

| Name                   | Key                  | Description                                                    | Min | Max |
| ---------------------- | -------------------- | -------------------------------------------------------------- | --- | --- |
| **User Prompt Attack** | user\_prompt\_attack | Indicates whether the input text contains a user prompt attack | 0   | 1   |
| **Document Attack**    | document\_attack     | Indicates whether any external documents contain attacks       | 0   | 1   |

### Additional Information <a href="#additional-information" id="additional-information"></a>

Please refer to the [official Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection) for a more comprehensive understanding of how Azure Prompt Shields works and how to implement it within your systems.

<br>


# DLP PII Detector (Google)

This evaluator utilizes Google's Cloud Data Loss Prevention (DLP) API to identify and classify potentially sensitive information within text data, ensuring compliance and safety across user interactions.

### Configuration <a href="#configuration" id="configuration"></a>

| Option                | Description                                                                      | Type       | Default                                                   | Required | Constraints                                                  |
| --------------------- | -------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------- | -------- | ------------------------------------------------------------ |
| **InfoTypes**         | Types of information to detect, based on Google Cloud DLP documentation          | `[]string` | `["PHONE_NUMBER", "EMAIL_ADDRESS", "CREDIT_CARD_NUMBER"]` | `true`   | MinItems1                                                    |
| **MinimumLikelihood** | The minimum likelihood required for an information type to be considered a match | `string`   | `LIKELY`                                                  | `true`   | EnumVERY\_LIKELY, LIKELY, POSSIBLE, UNLIKELY, VERY\_UNLIKELY |

### Additional Information <a href="#additional-information" id="additional-information"></a>

The Google Cloud DLP PII Detector is a powerful tool for scanning text for personally identifiable information (PII), helping to maintain privacy and meet regulatory compliance requirements. For more detailed information on configuring and using this evaluator, refer to [Google's DLP documentation](https://cloud.google.com/sensitive-data-protection/docs/libraries).

<br>


# Text Moderation (Google)

This evaluator leverages Google's advanced text analysis capabilities to assess content across various safety attributes such as toxicity, profanity, and threats. It's designed to identify potentially harmful or sensitive content in text documents.

### Configuration <a href="#configuration" id="configuration"></a>

| Option         | Description                                                                          | Type       | Default | Required | Constraints                                                                                                                                                       |
| -------------- | ------------------------------------------------------------------------------------ | ---------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Attributes** | Specific attributes to check within the text. An empty list implies all are checked. | `[]string` | `[]`    | `false`  | `Enum: TOXICITY, DEROGATORY, VIOLENCE, SEX, INSULT, PROFANITY, THREAT, WEAPONS, PUBLIC_SAFETY, HEALTH, RELIGION, ILLICIT_DRUGS, WAR, FINANCIAL, POLITICAL, LEGAL` |
| **Threshold**  | The confidence threshold to flag an attribute as concerning.                         | `float64`  | `0.7`   | `true`   | `Min: 0` `Max: 1`                                                                                                                                                 |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metrics to provide detailed insights into the moderation process:

| Name              | Key            | Description                                                    | Min | Max |
| ----------------- | -------------- | -------------------------------------------------------------- | --- | --- |
| **Toxicity**      | toxicity       | The probability that the text is toxic.                        | 0   | 1   |
| **Derogatory**    | derogatory     | The probability that the text is derogatory.                   | 0   | 1   |
| **Violence**      | violence       | The probability that the text contains violent content.        | 0   | 1   |
| **Sex**           | sex            | The probability that the text contains sexual content.         | 0   | 1   |
| **Insult**        | insult         | The probability that the text contains insulting content.      | 0   | 1   |
| **Profanity**     | profanity      | The probability that the text contains profanity.              | 0   | 1   |
| **Threat**        | threat         | The probability that the text contains threats of violence.    | 0   | 1   |
| **Weapons**       | weapons        | The probability that the text mentions firearms and weapons.   | 0   | 1   |
| **Public Safety** | public\_safety | The probability that the text mentions public safety issues.   | 0   | 1   |
| **Health**        | health         | The probability that the text mentions health issues.          | 0   | 1   |
| **Religion**      | religion       | The probability that the text mentions religion and belief.    | 0   | 1   |
| **Illicit Drugs** | illicit\_drugs | The probability that the text mentions illicit drugs.          | 0   | 1   |
| **War**           | war            | The probability that the text mentions war and conflict.       | 0   | 1   |
| **Financial**     | financial      | The probability that the text mentions finance-related topics. | 0   | 1   |
| **Political**     | political      | The probability that the text mentions political content.      | 0   | 1   |
| **Legal**         | legal          | The probability that the text discusses legal issues.          | 0   | 1   |

### Additional Information <a href="#additional-information" id="additional-information"></a>

You can learn more about Google Cloud's text moderation at [Google Cloud's text moderation documentation](https://cloud.google.com/natural-language/docs/moderating-text).


# Boolean LLM-as-Judge (Modelmetry)

This evaluator uses a large language model (LLM) to perform a boolean (true/false) evaluation of the input based on specified instructions and a maximum token count.

### Configuration <a href="#configuration" id="configuration"></a>

| Option           | Description                                                                                       | Type     | Default                                                                                                                                             | Required | Constraints  |
| ---------------- | ------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ |
| **Model**        | The namespace model used for evaluation                                                           | `string` | `openai/gpt-4o-mini`                                                                                                                                | `true`   |              |
| **Instructions** | Directions given to the LLM to ensure responses adhere to the task's requirements                 | `string` | "You are an LLM evaluator. We need the guarantee that the output answers what is being asked on the input, please evaluate as False if it doesn't." | `true`   | MinLength: 1 |
| **MaxTokens**    | The limit on the number of tokens the entire prompt can be, to prevent excessive input processing | `int`    | `8192`                                                                                                                                              | `true`   | `Min: 1`     |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator does not report specific metrics but evaluates whether the processed content meets the boolean conditions specified.

<br>


# Competitor Blocklist (Modelmetry)

This evaluator is designed to scan the input and output text for mentions of specified competitors and trigger a failure if the mentions exceed a set threshold. It supports adjusting sensitivity to case and defining where to look for mentions.

### Configuration <a href="#configuration" id="configuration"></a>

| Option              | Description                                                             | Type       | Default          | Required | Constraints                              |
| ------------------- | ----------------------------------------------------------------------- | ---------- | ---------------- | -------- | ---------------------------------------- |
| **LookIn**          | Where to search for competitor mentions                                 | `string`   | `both`           | `false`  | `Enum: input, output, both`              |
| **Competitors**     | List of competitors to search for                                       | `[]string` |                  | `true`   | `MinItems: 1`                            |
| **CaseSensitivity** | Whether to consider the word's case when matching strings               | `string`   | `case_sensitive` | `false`  | `Enum: case_sensitive, case_insensitive` |
| **Threshold**       | The number of times a competitor must be mentioned to trigger a failure | `int`      | `1`              | `false`  | -                                        |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metrics based on the analysis results:

| Name                    | Key                  | Description                               | Min | Max |
| ----------------------- | -------------------- | ----------------------------------------- | --- | --- |
| **Competitor Mentions** | competitor\_mentions | How many times a competitor was mentioned | 0   | ∞   |


# Emotion Analysis (Modelmetry)

This evaluator analyzes messages' emotional content, specifically targeting the presence and intensity of specified emotions such as joy, sadness, anger, or frustration.

### Configuration <a href="#configuration" id="configuration"></a>

| Option            | Description                                      | Type      | Default       | Required | Constraints                              |
| ----------------- | ------------------------------------------------ | --------- | ------------- | -------- | ---------------------------------------- |
| **EmotionType**   | The type of emotion to evaluate.                 | `string`  | `frustration` | `true`   | `Enum: joy, sadness, anger, frustration` |
| **FailThreshold** | The threshold to trigger the evaluation to fail. | `float64` | `0.7`         | `true`   | `Min: 0` `Max: 1`                        |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metrics based on the analysis results:

| Name            | Key         | Description                             | Min | Max |
| --------------- | ----------- | --------------------------------------- | --- | --- |
| **Joy**         | joy         | Represents how joyful the text was.     | 0   | 1   |
| **Sadness**     | sadness     | Represents how sad the text was.        | 0   | 1   |
| **Anger**       | anger       | Represents how angry the text was.      | 0   | 1   |
| **Frustration** | frustration | Represents how frustrated the text was. | 0   | 1   |

<br>


# HTTP Request (Modelmetry)

This evaluator enables sending HTTP requests to a specified endpoint and interpreting the response to determine the outcome. It can pass, fail, or error based on the received data.

### Configuration <a href="#configuration" id="configuration"></a>

| Option      | Description                                 | Type                | Default | Required | Constraints              |
| ----------- | ------------------------------------------- | ------------------- | ------- | -------- | ------------------------ |
| **URL**     | The URL to send the HTTP request to.        | `string`            | -       | `true`   | `format: uri`            |
| **Method**  | The HTTP method to use for the request.     | `string`            | `POST`  | `true`   | `enum: POST, PATCH, PUT` |
| **Headers** | A map of headers to include in the request. | `map[string]string` | -       | `false`  | -                        |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator does not define any specific metrics but captures various response attributes that influence the pass/fail/error outcomes based on the API response.

<br>


# Language Detector (Modelmetry)

This evaluator detects the language of the input and compares it to the expected language or to the language of the output, depending on the configuration.

### Configuration <a href="#configuration" id="configuration"></a>

| Option                  | Description                                                                                                                    | Type      | Default                | Required | Constraints                                                                      |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------- | ---------------------- | -------- | -------------------------------------------------------------------------------- |
| **CheckType**           | What should be checked by this evaluator                                                                                       | `string`  | `input_matches_output` | `false`  | `Enum: input_matches_output, input_matches_language, output_matches_fulfillment` |
| **ExpectedLanguage**    | The expected language of the input and/or output                                                                               | `string`  | None                   | `false`  | `Length: 2` `Pattern: ^[a-z]{2}$`                                                |
| **WordCountThreshold**  | Minimum number of words to check, as the language detection can be unreliable for very short texts                             | `int`     | `2`                    | `false`  | `Min: 0`                                                                         |
| **ConfidenceThreshold** | Minimum confidence threshold for the language detection. If the confidence is lower than this, the evaluation will be skipped. | `float64` | `0.4`                  | `false`  | `Min: 0` `Max: 1`                                                                |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metrics based on the analysis results:

| Name                              | Key                              | Description                                               | Min | Max |
| --------------------------------- | -------------------------------- | --------------------------------------------------------- | --- | --- |
| **Input Matches Output Language** | input\_matches\_output\_language | Whether the input language matches the output language    | 0   | 1   |
| **Input Language As Expected**    | input\_language\_as\_expected    | Whether the input language matches the expected language  | 0   | 1   |
| **Output Language As Expected**   | output\_language\_as\_expected   | Whether the output language matches the expected language | 0   | 1   |

<br>


# Score LLM-as-Judge (Modelmetry)

This evaluator uses a language model to numerically score a payload based on how likely they are to be satisfied with it, ranging from 0.0 (not satisfied at all) to 1.0 (completely satisfied).

### Configuration <a href="#configuration" id="configuration"></a>

| Option               | Description                                                  | Type                 | Default                                                                                                                                                                                  | Required | Constraints    |
| -------------------- | ------------------------------------------------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------- |
| **Model**            | The specific language model used for evaluation              | `ai.NamespacedModel` | `openai/gpt-3.5-turbo`                                                                                                                                                                   | `true`   |                |
| **Instructions**     | Evaluation criteria as provided to the LLM                   | `string`             | "You are an LLM evaluator. Please score from 0.0 to 1.0 how likely the user is to be satisfied with this answer, from 0.0 being not satisfied at all to 1.0 being completely satisfied." | `true`   | `MinLength: 1` |
| **MaxTokens**        | The maximum number of tokens allowed in the prompt           | `int`                | `8192`                                                                                                                                                                                   | `true`   | `Min: 1`       |
| **PassingThreshold** | The minimum score required to consider the evaluation passed | `float64`            | `0.5`                                                                                                                                                                                    | `true`   |                |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator does not report specific metrics beyond the binary outcome of the evaluation (passed or failed).


# Word Counter (Modelmetry)

This evaluator counts the number of words in the input text and evaluates it based on a configured threshold.

### Configuration <a href="#configuration" id="configuration"></a>

| Option       | Description                   | Type       | Default | Required | Constraints            |
| ------------ | ----------------------------- | ---------- | ------- | -------- | ---------------------- |
| **Cutset**   | Characters to split on        | `[]string` | `[" "]` | `true`   | `MinItems: 1`          |
| **Operator** | Comparison operator for count | `string`   | `"<"`   | `true`   | `Enum: <,<=,>,>=,=,!=` |
| **Value**    | Value to compare against      | `int`      | `3`     | `true`   | `Min: 0`               |

### Metrics <a href="#metrics" id="metrics"></a>

This evaluator reports the following metric based on the text analysis:

| Name           | Key       | Description                      | Min | Max |
| -------------- | --------- | -------------------------------- | --- | --- |
| **Word Count** | wordcount | The number of words in the input | 0   | ∞   |