Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions support/rotate-credentials.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,35 @@ For [hybrid deployments](/self-hosting/hybrid-deployments/architecture), rotate

Redeploy the gateway after updating secrets and confirm it registers and serves requests.

## Protect your data in hybrid deployments

In a [hybrid deployment](/self-hosting/hybrid-deployments/architecture), your prompt and log data stays in your own infrastructure (the data plane). Rotating credentials stops new access, and the measures below limit what an attacker can reach even if they hold a leaked API key or know a log ID. Apply them in addition to steps 1 to 9, and keep them in place until you are confident the incident is contained.
Comment thread
whysumedh marked this conversation as resolved.
Resolved

### Disable public ingress to your data plane

Fetching a log by ID is a data plane API, not a control plane API. The control plane never stores your prompt logs. Unless the data plane is reachable from the public internet, a threat actor cannot read log details, even if they have an API key and a log ID.
Comment thread
whysumedh marked this conversation as resolved.
Resolved

Remove public ingress to your data plane and keep it reachable only from your private network. If you expose the gateway to applications, put it behind your own private load balancer or VPN rather than a public endpoint.

### Temporarily disable the Data Service

The Data Service handles log exports and downloads. If you shut it down in your infrastructure, data export and download stop completely, **including for historical exports**.
Comment thread
whysumedh marked this conversation as resolved.
Resolved

If you suspect a compromise, scale the Data Service down until you have finished rotating credentials. Announce a temporary downtime for log exports and other log-related operations to your users before you do.

### Block inbound connections from the control plane

You can also block the Portkey control plane from connecting to your data plane. This stops log fetching and prompt playground requests. The two connections between the planes behave differently:

| | Control plane to data plane (gateway)<br/>**Inbound** from Portkey infra to your infra | Data plane (gateway) to control plane<br/>**Outbound** from your infra to Portkey infra |
|---|---|---|
| **What it is used for** | 1. Individual logs are fetched in real time from the data plane and rendered only in the UI. Your prompt logs are not stored persistently in the control plane.<br/>2. Prompt playground requests are made from the control plane backend to your on-prem data plane. | 1. The data plane starts an outbound connection to the control plane and syncs the delta of all configurations, keys, and similar data from the Portkey-managed database. |
| **When this connection is severed** | 1. The logs list page on `/logs` still loads, but opening any log to see the request fails.<br/>2. Prompt playground requests do not work. | 1. Your data plane continues to function as it is for the next 7 days.<br/>2. All existing data is stored in your local cache for 7 days.<br/>3. Any new changes made on the control plane are not reflected in your gateway calls. |

<Warning>
Block only the **inbound** connection. Do not sever the outbound connection from your data plane to the control plane: it carries the credentials and configuration you rotated in steps 1 to 3, and without it new changes do not reach your gateways. Restore the inbound connection once the incident is contained.
</Warning>

## After rotating

1. **Confirm the old credentials fail.** Wait at least a few minutes for gateway sync and caches to refresh, then send a request with each old Portkey API key, admin key, and JWT signed by the old key through every gateway, including self-hosted ones. Expect a `401`. Confirm a gateway still using an old client auth key can no longer sync.
Expand All @@ -354,6 +383,7 @@ Copy this list to track progress.
- [ ] **Step 8:** Secret reference connection credentials rotated; stored secrets rotated
- [ ] **Step 9:** Configs, guardrails, prompts, and custom hosts reviewed for embedded secrets; OTel and log export credentials rotated; self-hosted gateway secrets rotated
- [ ] Old Portkey API keys confirmed to return `401`
- [ ] Hybrid only: public ingress to the data plane disabled; Data Service temporarily disabled and downtime announced; inbound control plane connection blocked if needed
- [ ] Audit logs and provider usage reviewed

## Related
Expand Down
Loading