diff --git a/support/rotate-credentials.mdx b/support/rotate-credentials.mdx index bf3f515b..fe39903a 100644 --- a/support/rotate-credentials.mdx +++ b/support/rotate-credentials.mdx @@ -6,37 +6,93 @@ description: "Step-by-step guide to rotating every credential stored in or issue This guide lists every credential that Portkey stores or issues for an organisation, and how to rotate each one. Work through the sections in order. -Portkey does not support grace periods when you replace credentials stored in integrations, providers, MCP servers, or other settings. Saving a new value does not take effect on every gateway at once. Hybrid and self-hosted gateways clear cached keys, integrations, and configs on a sync that runs about once a minute, and values read from a vault are cached for up to 5 minutes. For each credential: create the new value at the source, update Portkey, wait for the change to propagate, confirm requests succeed with the new value on every gateway, then revoke the old value at the source. +Portkey API keys and gateway client auth keys support a grace period (minimum 30 minutes for API keys). Portkey does not support grace periods when you replace credentials stored in integrations, providers, MCP servers, or other settings. Saving a new value does not take effect on every gateway at once. Hybrid and self-hosted gateways clear cached keys, integrations, and configs on a sync that runs about once a minute, and values read from a vault are cached for up to 5 minutes. For each stored credential: create the new value at the source, update Portkey, wait for the change to propagate, confirm requests succeed with the new value on every gateway, then revoke the old value at the source. ## Credentials at a glance -| # | Credential | Where it lives | Rotate at | +| Step | Credential | Where it lives | Rotate at | |---|---|---|---| -| 1 | [Portkey API keys](#1-portkey-api-keys) (admin, workspace service, workspace user) | Portkey | Portkey | -| 2 | [AI provider credentials](#2-ai-provider-credentials) (integrations, providers, virtual keys) | Portkey | Your AI provider, then Portkey | -| 3 | [MCP credentials](#3-mcp-credentials) | Portkey | The MCP server or IdP, then Portkey | -| 4 | [Secret references](#4-secret-references-and-vault-credentials) | Your vault | Your vault | -| 5 | [JWT, sessions, and users](#5-jwt-sessions-and-users) | Portkey | Your IdP, then Portkey | -| 6 | [Plugin integration credentials](#6-plugin-integration-credentials) | Portkey | The plugin provider, then Portkey | -| 7 | [Credentials embedded in configs, guardrails, and prompts](#7-credentials-embedded-in-configs-guardrails-and-prompts) | Portkey | The upstream service, then Portkey | -| 8 | [Observability and log export credentials](#8-observability-and-log-export-credentials) | Portkey or your gateway | Your destination, then Portkey | -| 9 | [Self-hosted gateway secrets](#9-self-hosted-gateway-secrets) | Your deployment | Your deployment | +| 1 | [Admin API keys](#step-1-remove-all-admin-api-keys) | Portkey | Portkey | +| 2 | [Gateway client auth (license key)](#step-2-rotate-the-gateway-client-auth-key) for hybrid deployments | Portkey, your Helm configuration | Portkey, then your Helm configuration | +| 3a | [User and service API keys](#step-3a-rotate-user-and-service-api-keys) | Portkey | Portkey | +| 3b | [JWKS signing keys](#step-3b-update-jwks) | Your IdP, Portkey, gateway Redis | Your IdP, then Portkey and Redis | +| 4 | [MCP credentials](#step-4-rotate-mcp-credentials) | Portkey | The MCP server or IdP, then Portkey | +| 5 | [Provider LLM keys](#step-5-rotate-provider-llm-keys) (integrations, providers, virtual keys) | Portkey | Your AI provider, then Portkey | +| 6 | [A2A agent server keys](#step-6-rotate-a2a-agent-server-keys) | Portkey | The agent server, then Portkey | +| 7 | [External guardrail integration keys](#step-7-rotate-external-guardrail-integration-keys) | Portkey | The guardrail provider, then Portkey | +| 8 | [Secrets stored in secret references](#step-8-rotate-secrets-stored-in-secret-references) | Your vault | Your vault | +| 9 | [Other places credentials hide](#step-9-review-other-places-credentials-hide) (configs, prompts, log exports, gateway secrets) | Portkey or your deployment | The upstream service or your deployment | + + +**For hybrid deployments, steps 1 to 3 remove all data entry points.** Once they are complete, no old Admin API key, gateway client auth (license key), Portkey API key, or JWT signed by an old key can reach your data or change your configuration. Steps 4 to 9 then rotate the credentials Portkey uses to reach your own systems and providers. + ## Before you start -- **Use a rotation account you trust.** Sign in to the Portkey dashboard with SSO or your own credentials. Don't depend on an existing API key to run the rotation. +- **Work in the order shown.** Steps 1 to 3 close every way in; steps 4 to 9 rotate what Portkey uses to reach your systems. Each step that has a grace period must finish before you move on. +- **Use a rotation account you trust.** Sign in to the Portkey dashboard with SSO or your own credentials. Apart from the temporary admin key in step 3a, don't depend on an existing API key to run the rotation. - **Plan for downtime.** Replacing a credential breaks any application still using the old value. Roll out new values to your applications before revoking the old ones. - **Keep a list.** Record each credential you rotate, so you can confirm none was missed. Use the [checklist](#rotation-checklist) at the end of this page. - **Know your permissions.** Rotating organisation-level resources requires an organisation owner or admin. Workspace-level resources require a workspace admin or manager. -## 1. Portkey API keys +## Step 1: Remove all admin API keys -Portkey issues three kinds of API keys. See [API Keys (AuthN and AuthZ)](/product/enterprise-offering/org-management/api-keys-authn-and-authz) for how they differ. +Admin API keys are organisation-level keys that can change configuration in Portkey, for example setting up log exports, adding integrations, and creating other API keys. If one leaks, an attacker can use it to send your data elsewhere or to mint new credentials, so remove them first. + +### Remove with the dashboard + +Go to **Admin Settings > Admin Keys**. Delete every key you don't recognise or no longer use. For a key that automation still depends on, use **Rotate** and set the transition period to 30 minutes, the minimum. + +### Remove with the API + +Admin API keys are keys of type `organisation-service`. The calling key needs the `organisation_service_api_keys.list`, `.delete`, and `.rotate` scopes. + +1. List every admin key with [`GET /api-keys?type=organisation-service`](/api-reference/admin-api/control-plane/api-keys/list-api-keys). +2. [Delete each admin API key](/api-reference/admin-api/control-plane/api-keys/delete-an-api-key) with `DELETE /api-keys/{id}`. Delete any key you don't recognise or no longer use. Delete the key you are authenticating with last, otherwise later calls fail. +3. If an admin key can't be deleted because automation still depends on it, [rotate it](/api-reference/admin-api/control-plane/api-keys/rotate-api-key) with `POST /api-keys/{id}/rotate` and set `key_transition_period_ms` to `1800000` (30 minutes, the minimum). A key can't be rotated again while its previous secret is still valid. + +Whichever method you use, **wait for the grace period to end** before moving to step 2. The old secrets stop working only when it ends. + + +Keep one admin key for [step 3a](#step-3a-rotate-user-and-service-api-keys) if you rotate user and service keys through the API. Use the freshly rotated key, or create a single new one, and delete it when step 3a is done. Store it only in a place that you trust. + + +## Step 2: Rotate the gateway client auth key + +Applies to [hybrid deployments](/self-hosting/hybrid-deployments/architecture). The client auth key (`PORTKEY_CLIENT_AUTH`, also called the license key) is the credential each gateway uses to sync with the Portkey control plane. Rotate it for **every gateway deployment** in your organisation. + + + +Go to **Admin Settings > Gateway Registration** and open the deployment. + + Gateway Registration deployment page with the Rotate Key button under Gateway Authentication + + + +Under **Gateway Authentication**, click **Rotate Key**. Set **Grace period (hours)** to `0.5` for a 30-minute grace period. Leave it empty to use the default. Click **Confirm Rotation**. + + Rotate Key dialog with the Grace period (hours) field + + + +Copy the new key and click **Done**. The key is shown only once. + + New gateway auth key shown once with a copy button + + + +Replace `PORTKEY_CLIENT_AUTH` with the new key in your Helm configuration for every gateway and optional Data Service instance that uses this deployment. If your deployment references a secret store, update the underlying credential while preserving secret references and any other fields in the secret. Run `helm upgrade` and restart all consuming services before the grace period ends. Services still using the old key lose control-plane authentication once it expires. Then confirm each gateway registers and syncs with the new key. + + + +## Step 3a: Rotate user and service API keys + +Use the admin API key from [step 1](#step-1-remove-all-admin-api-keys) to rotate every remaining key, in the dashboard or programmatically. See [API Keys (AuthN and AuthZ)](/product/enterprise-offering/org-management/api-keys-authn-and-authz) for how the key types differ. -- **Admin API keys:** organisation-level keys with access across workspaces - **Workspace service API keys:** workspace-level keys for backends and automation - **Workspace user API keys:** workspace-level keys owned by individual users +- **Organisation service API keys** (type `organisation-service`, the admin keys from step 1), only the temporary one should remain ### Choose a rotation method @@ -51,6 +107,8 @@ The rotate endpoint always keeps the previous secret valid for the transition pe ### Rotate with the dashboard +This is the simplest route if you have few keys. The temporary admin key is under **Admin Settings > Admin Keys**; service and user keys are on the **API Keys** page. + Go to [**API Keys**](https://app.portkey.ai/api-keys). @@ -68,21 +126,54 @@ After traffic moves to the new key, delete the old key from the list. ### Rotate with the API -1. [Create an API key](/api-reference/admin-api/control-plane/api-keys/create-api-key) with `POST /v2/api-keys/{sub-type}`. Set the same default config and config-override setting as the old key. -2. Deploy the new key. -3. [Delete the old key](/api-reference/admin-api/control-plane/api-keys/delete-an-api-key) with `DELETE /v2/api-keys/{id}`. +Filter each list call by key type. The key types are `workspace-service`, `workspace-user`, and `organisation-service`. Step 1 removed the `organisation-service` keys, so only the temporary admin key should remain under that type. The calling key needs the `.list` and `.rotate` scopes for each type (plus `.create` and `.delete` if you replace keys): `workspace_service_api_keys`, `workspace_user_api_keys`, and `organisation_service_api_keys`. + +1. List the keys for each type with [`GET /api-keys?type=workspace-service`](/api-reference/admin-api/control-plane/api-keys/list-api-keys) and `GET /api-keys?type=workspace-user`. Page through the results so none is missed. +2. For each service and user key, either: + - call [`POST /api-keys/{id}/rotate`](/api-reference/admin-api/control-plane/api-keys/rotate-api-key) to keep the same key ID. The response contains the new secret and the time the previous secret expires, or + - [create a replacement](/api-reference/admin-api/control-plane/api-keys/create-api-key) with `POST /api-keys/workspace/service` or `POST /api-keys/workspace/user`, using the same default config and config-override setting as the old key, deploy it, then [delete the old key](/api-reference/admin-api/control-plane/api-keys/delete-an-api-key) with `DELETE /api-keys/{id}`. +3. Distribute each new secret to the owning application, CI pipeline, or secret store, and replace the `x-portkey-api-key` header value. +4. Check with `GET /api-keys?type=organisation-service` that the only admin key left is the temporary one, then delete it. + +Rotate **workspace service API keys** before **workspace user API keys**, because service keys can manage other workspace keys. Delete any key you don't recognise or no longer use. + +## Step 3b: Update JWKS + +If you authenticate with [JWTs](/product/enterprise-offering/org-management/jwt), the gateway trusts any token signed by a key in your configured JWKS, and caches the result of validated tokens. Rotate the signing key and clear those caches. + +1. **Rotate the JWKS signing key.** Generate a new key pair at your IdP, publish the new public key in the JWKS with a new `kid`, and start signing tokens with the new private key. Then remove the old key from the JWKS in Portkey (**Admin Settings > Organisation > Authentication**). +2. **Flush Redis `AKEY_*` (hybrid deployments).** Hybrid gateways cache authentication results in your own Redis under keys that start with `AKEY_`. Delete them so cached tokens signed with the old key stop being accepted. Run this against the Redis instance your gateway uses, for example: + +```sh +redis-cli --scan --pattern 'AKEY_*' | xargs -r redis-cli del +``` + + +Old tokens remain valid until their `exp` unless you flush the cache and remove the old key from the JWKS. Run the flush after the old key is removed, and use `UNLINK` instead of `DEL` on large instances. + + +Also ask users to sign out and back in, and review [organisation users and pending invites](/product/enterprise-offering/org-management/organizations). Remove any user or invite you don't recognise. -To keep the same key ID instead, call [`POST /v2/api-keys/{id}/rotate`](/api-reference/admin-api/control-plane/api-keys/rotate-api-key). The response contains the new secret and the time the previous secret expires. + +**For hybrid deployments, steps 1 to 3 remove all data entry points.** After they are done, only keys and tokens issued in these steps can authenticate to Portkey or change your configuration. + -### Order of rotation +## Step 4: Rotate MCP credentials + +[MCP integrations and servers](/product/mcp-gateway/mcp-registry) store credentials that Portkey uses to reach upstream MCP servers. Rotate each one at the source, then update Portkey. + +| Authentication type | What to rotate | Steps | +|---|---|---| +| **Static headers** ([custom auth](/product/mcp-gateway/authentication/custom-auth)) | Bearer tokens, API keys, or custom header values | Generate new credentials on the MCP server. Update the headers on the MCP integration or server in Portkey. Revoke the old credentials. | +| **Client credentials** ([client credentials](/product/mcp-gateway/authentication/client-credentials)) | OAuth client secret | Create a new secret at your IdP. Update **Client Secret** in the integration, or in the vault entry it references. Wait up to 5 minutes for vault-cached values to refresh. Revoke the old secret. | +| **OAuth Auto** ([OAuth](/product/mcp-gateway/authentication/oauth)) | OAuth client secret, if you supplied your own OAuth app. Per-user access tokens. | Create a new client secret in the upstream OAuth app and update the integration. To force users to re-authorise, remove their stored connections with [`DELETE /mcp-servers/{mcpServerId}/connections`](/api-reference/admin-api/control-plane/mcp-servers/connections/delete-mcp-server-connection). | +| **External OAuth or JWT validation** | Introspection client secret, if configured | Rotate the introspection credentials at your IdP and update the integration. JWKS URIs and public keys are not secret. | -1. Rotate **admin API keys** first. They can create or modify every other resource. -2. Rotate **workspace service API keys** next. They can manage other workspace keys. -3. Rotate **workspace user API keys** last. Ask each user to replace theirs, or rotate them as an admin. +Also rotate any tokens the MCP server itself holds for downstream systems. Those live outside Portkey. -List existing keys with [`GET /v2/api-keys`](/api-reference/admin-api/control-plane/api-keys/list-api-keys) to find every key to replace. Delete any key you don't recognise or no longer use. +Use [`GET /mcp-servers`](/api-reference/admin-api/control-plane/mcp-servers/list-mcp-servers) and [`GET /mcp-integrations`](/api-reference/admin-api/control-plane/mcp-integrations/list-mcp-integrations) to list every server and integration that needs review. -## 2. AI provider credentials +## Step 5: Rotate provider LLM keys Integrations, providers, and virtual keys hold credentials for your AI providers: API keys, cloud access keys, service account JSON, and similar. Rotating the key only in Portkey does not invalidate the old one. **Rotate the credential at the provider.** @@ -135,45 +226,33 @@ Existing virtual keys hold provider credentials too. Rotate them the same way us If your applications send provider credentials per request, such as an `Authorization` header or `x-portkey-*` provider headers, rotate those in your application secret stores as well. -## 3. MCP credentials - -[MCP integrations and servers](/product/mcp-gateway/mcp-registry) store credentials that Portkey uses to reach upstream MCP servers. Rotate each one at the source, then update Portkey. +## Step 6: Rotate A2A agent server keys -| Authentication type | What to rotate | Steps | -|---|---|---| -| **Static headers** ([custom auth](/product/mcp-gateway/authentication/custom-auth)) | Bearer tokens, API keys, or custom header values | Generate new credentials on the MCP server. Update the headers on the MCP integration or server in Portkey. Revoke the old credentials. | -| **Client credentials** ([client credentials](/product/mcp-gateway/authentication/client-credentials)) | OAuth client secret | Create a new secret at your IdP. Update **Client Secret** in the integration, or in the vault entry it references. Wait up to 5 minutes for vault-cached values to refresh. Revoke the old secret. | -| **OAuth Auto** ([OAuth](/product/mcp-gateway/authentication/oauth)) | OAuth client secret, if you supplied your own OAuth app. Per-user access tokens. | Create a new client secret in the upstream OAuth app and update the integration. To force users to re-authorise, remove their stored connections with [`DELETE /mcp-servers/{mcpServerId}/connections`](/api-reference/admin-api/control-plane/mcp-servers/connections/delete-mcp-server-connection). | -| **External OAuth or JWT validation** | Introspection client secret, if configured | Rotate the introspection credentials at your IdP and update the integration. JWKS URIs and public keys are not secret. | +Agents registered in the [Agent Gateway](/product/agent-gateway) authenticate to their upstream A2A agent server with credentials that Portkey stores, usually an `Authorization` bearer token or API key in the agent's `headers`, plus any `passthrough_headers`. See [Agent Servers](/product/agent-gateway/servers) and [Forwarding Headers](/product/agent-gateway/forwarding-headers). Rotate each one at the source, then update Portkey. -Also rotate any tokens the MCP server itself holds for downstream systems. Those live outside Portkey. - -Use [`GET /mcp-servers`](/api-reference/admin-api/control-plane/mcp-servers/list-mcp-servers) and [`GET /mcp-integrations`](/api-reference/admin-api/control-plane/mcp-integrations/list-mcp-integrations) to list every server and integration that needs review. - -## 4. Secret references and vault credentials - -[Secret references](/product/enterprise-offering/secret-references) point to secrets in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. Portkey stores the reference configuration, including the auth config it uses to reach your vault. It does not store the secret value itself. - -Rotate in this order: - -1. **Vault access credentials.** Rotate the credentials in each secret reference's `auth_config` (for example an IAM access key, an Azure client secret, or a HashiCorp Vault token or AppRole secret). Update the reference with [`PUT /v1/secret-references/{id}`](/product/enterprise-offering/secret-references#updating-a-secret-reference). Updates to `auth_config` are merged, so send only the changed fields. -2. **The secrets themselves.** Rotate the secret values in your vault. Gateways cache fetched values for 5 minutes, so the new value takes effect within that window with no change in Portkey. -3. **Revoke the old vault credentials.** - -## 5. JWT, sessions, and users + + +Open the [**Agents**](https://app.portkey.ai/agents) tab and note each agent and the headers configured on it. + + +Generate a new API key or bearer token on the agent server, or at the IdP that issues its tokens. + + +Replace the credential in the agent's configured headers, including `Authorization` and any custom header that carries a secret. + + +Fetch the agent card and invoke the agent through the Agent Gateway to confirm the new credential works. + + +Wait for the change to propagate, then delete or disable the old credential on the agent server. + + - - -Portkey validates JWTs using your IdP's public keys, which are not secret. Rotate signing keys at your IdP if you suspect they were exposed, and confirm the JWKS configuration in Portkey is current. See [JWT](/product/enterprise-offering/org-management/jwt). - - -Ask users to sign out and back in. Review [organisation users and pending invites](/product/enterprise-offering/org-management/organizations), and remove any user or invite you don't recognise. - - +Also rotate any tokens the agent server itself holds for downstream systems. Those live outside Portkey. -## 6. Plugin integration credentials +## Step 7: Rotate external guardrail integration keys -Plugin integrations store credentials for third-party guardrail and tool providers. Rotate the credential at the provider, then update it in Portkey. +Plugin integrations store credentials for external guardrail and tool providers (for example Lakera, Aporia, Pillar, or a cloud guardrail service). Rotate the credential at the provider, then update it in Portkey. @@ -202,9 +281,23 @@ Wait for the change to propagate, then delete or disable the old credential. Managed identity and similar setups store no secret. -Raw guardrails defined in JSON can carry a `credentials` object inline. Review those under [section 7](#7-credentials-embedded-in-configs-guardrails-and-prompts) as well. +Raw guardrails defined in JSON can carry a `credentials` object inline. Review those under [step 9](#credentials-embedded-in-configs-guardrails-and-prompts) as well. -## 7. Credentials embedded in configs, guardrails, and prompts +## Step 8: Rotate secrets stored in secret references + +[Secret references](/product/enterprise-offering/secret-references) point to secrets in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. Portkey stores the reference configuration, including the auth config it uses to reach your vault. It does not store the secret value itself. + +Rotate in this order: + +1. **Update the connection configuration if it uses long-term credentials.** Rotate the credentials in each secret reference's `auth_config` (for example an IAM access key, an Azure client secret, or a HashiCorp Vault token or AppRole secret). Update the reference with [`PUT /secret-references/{secretReferenceId}`](/api-reference/admin-api/control-plane/secret-references/update-secret-reference). Updates to `auth_config` are merged, so send only the changed fields. Prefer short-lived or role-based access (for example an assumed IAM role or Azure managed identity) where your vault supports it, so there is no long-term credential to rotate. +2. **Rotate all the stored secrets.** Rotate every secret value in your vault that a reference points to. Gateways cache fetched values for 5 minutes, so the new value takes effect within that window with no change in Portkey. +3. **Revoke the old vault credentials.** + +List every reference with [`GET /secret-references`](/api-reference/admin-api/control-plane/secret-references/list-secret-references) so none is missed. + +## Step 9: Review other places credentials hide + +### Credentials embedded in configs, guardrails, and prompts Credentials sometimes end up inside saved resources instead of integrations. Review each of the following for static secrets: @@ -215,7 +308,7 @@ Credentials sometimes end up inside saved resources instead of integrations. Rev For each secret found, rotate it at the upstream service, update the saved resource, and revoke the old value. Prefer [secret references](/product/enterprise-offering/secret-references) over inline values so future rotation happens in your vault. -## 8. Observability and log export credentials +### Observability and log export credentials Rotate any credential Portkey uses to write to your systems: @@ -225,11 +318,11 @@ Rotate any credential Portkey uses to write to your systems: Create the new credential at the destination, update the Portkey configuration, confirm data is arriving, then revoke the old credential. -## 9. Self-hosted gateway secrets +### Self-hosted gateway secrets For [hybrid deployments](/self-hosting/hybrid-deployments/architecture), rotate every secret in your deployment configuration: -- **`PORTKEY_CLIENT_AUTH`**: the credential your gateway uses to sync with the Portkey control plane. See [Rotate the gateway client auth key](#rotate-the-gateway-client-auth-key). +- **`PORTKEY_CLIENT_AUTH`**: the credential your gateway uses to sync with the Portkey control plane. Rotated in [step 2](#step-2-rotate-the-gateway-client-auth-key). - **Log store credentials:** for example `LOG_STORE_ACCESS_KEY` and `LOG_STORE_SECRET_KEY` (S3, GCS HMAC, or Azure storage). - **Cache and database credentials:** Redis, and any database your deployment uses. - **Provider keys and other values in your environment files, Helm values, or Kubernetes secrets.** @@ -237,35 +330,9 @@ For [hybrid deployments](/self-hosting/hybrid-deployments/architecture), rotate Redeploy the gateway after updating secrets and confirm it registers and serves requests. -### Rotate the gateway client auth key - - - -Go to **Admin Settings > Gateway Registration** and open the deployment. - - Gateway Registration deployment page with the Rotate Key button under Gateway Authentication - - - -Under **Gateway Authentication**, click **Rotate Key**. Optionally, set a **Grace period (hours)** from 0 to 24. This is how long the previous key keeps working. Leave it empty to use the default. Click **Confirm Rotation**. - - Rotate Key dialog with the Grace period (hours) field - - - -Copy the new key and click **Done**. The key is shown only once. - - New gateway auth key shown once with a copy button - - - -Update `PORTKEY_CLIENT_AUTH` to the new key for every gateway and optional Data Service instance that uses this deployment. If your deployment references a secret store, update the underlying credential while preserving secret references and any other fields in the secret. Restart all consuming services so they load the new key before the grace period ends. Services still using the old key lose control-plane authentication once it expires. - - - ## After rotating -1. **Confirm the old credentials fail.** Wait at least a few minutes for gateway sync and caches to refresh, then send a request with each old Portkey API key through every gateway, including self-hosted ones. Expect a `401`. +1. **Confirm the old credentials fail.** Wait at least a few minutes for gateway sync and caches to refresh, then send a request with each old Portkey API key, admin key, and JWT signed by the old key through every gateway, including self-hosted ones. Expect a `401`. Confirm a gateway still using an old client auth key can no longer sync. 2. **Test end-to-end.** Send a request through every integration, MCP server, and key path your applications use. 3. **Review audit logs.** Check [audit logs](/product/enterprise-offering/audit-logs) under **Admin Settings > Audit Logs**, or call [`GET /audit-logs`](/api-reference/admin-api/control-plane/audit-logs/list-audit-logs), for unexpected creation of API keys, integrations, users, or configuration changes. 4. **Review request logs and provider usage.** Check [logs](https://app.portkey.ai/logs) and your AI providers' usage dashboards for requests you didn't make. @@ -275,18 +342,17 @@ Update `PORTKEY_CLIENT_AUTH` to the new key for every gateway and optional Data Copy this list to track progress. -- [ ] Admin API keys replaced; old keys deleted -- [ ] Workspace service API keys replaced; old keys deleted -- [ ] Workspace user API keys replaced; old keys deleted +- [ ] **Step 1:** Admin API keys deleted, or rotated with the grace period (minimum 30 minutes) allowed to end +- [ ] **Step 2:** Client auth (license key) rotated for every gateway deployment; Helm configuration updated and gateways restarted +- [ ] **Step 3a:** Workspace service and user API keys rotated or replaced programmatically; old keys deleted; temporary admin key deleted +- [ ] **Step 3b:** JWKS signing key rotated; old key removed from the JWKS; Redis `AKEY_*` flushed - [ ] Application secret stores updated with new Portkey keys -- [ ] Every integration and provider credential rotated at the provider, updated in Portkey, and old credentials revoked -- [ ] Virtual key credentials rotated, or virtual keys migrated and deleted -- [ ] MCP integration and server credentials rotated; user connections reset where needed -- [ ] Secret reference vault credentials rotated; secret values rotated -- [ ] Plugin integration credentials rotated, including workspace override credentials -- [ ] Configs, guardrails, prompts, and custom hosts reviewed for embedded secrets -- [ ] OTel and log export credentials rotated -- [ ] Self-hosted gateway secrets rotated; gateway redeployed +- [ ] **Step 4:** MCP integration and server credentials rotated; user connections reset where needed +- [ ] **Step 5:** Every integration and provider credential rotated at the provider, updated in Portkey, and old credentials revoked; virtual keys rotated or migrated +- [ ] **Step 6:** A2A agent server credentials rotated +- [ ] **Step 7:** External guardrail integration credentials rotated, including workspace override credentials +- [ ] **Step 8:** Secret reference connection credentials rotated; stored secrets rotated +- [ ] **Step 9:** Configs, guardrails, prompts, and custom hosts reviewed for embedded secrets; OTel and log export credentials rotated; self-hosted gateway secrets rotated - [ ] Old Portkey API keys confirmed to return `401` - [ ] Audit logs and provider usage reviewed