Secrets

Secrets are authentication codes such as API keys, JSON service accounts, auth tokens, given to you by third-party providers that you want Modelmetry to use and store securely.

We do not store your secrets in plain text. They are encrypted at rest and only decrypted when used to authenticate with external services. For this reason, we cannot show you the full value of the secret once it has been set (only a small preview).

For the sake of safety, we do recommend you enable usage and billing limits with providers that allow to ensure no bug within or hack into our servers compromises your third-party access and billing usage.

List all secrets

1. Go to Settings

2. Click on Secrets

3. View all secrets

Create a new secret

1. Go to Settings

2. Click on Secrets

3. Click on + New Secret

4. Enter the appropriate data:

   1. Provider: which third-party is this secret used for (this will be used when you select a specific model in some of our evaluators). You cannot change the provider once the secret is created.

      1. OpenAI

      2. Azure

      3. Google

      4. Amazon

      5. Groq

      6. Replicate

      7. Other

   2. Name: give it a human-friendly name so your team knows what it refers to, and what to use it for (e.g., openai-embeddings-api-key, google-gemini-json-service-account). You cannot change the name once the secret is created.

   3. Value: the actual contents of the secret.

5. Click Save

6. The secret will now appear in the table, with a preview

Once your secret is saved, we also save a very short preview so you can see, at a glance, what this token is. For example, if you have multiple OpenAI secrets, you will be able to see the last few characters of the token to help you distinguish them.

Update a secret

1. Go to Settings

2. Click on Secrets

3. Click on the edit icon button

4. Change the contents of the secret's value.

5. Click Save

Delete a secret

You cannot delete a secret which is still being used in one or more evaluator instances. Make sure you detach the secret you want to delete from instances in order to delete it.

• Go to Settings

• Click on Secrets

• Click on the edit icon button

• Click Delete