Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ go mod download
go run main.go # reads ./config/config.yaml, then ./config.yaml, then /etc/e-sp-line2
```

The API listens on `:8080`. Before you expose it beyond localhost, change `jwt.secret` in the config — see [SECURITY.md](SECURITY.md).
The API listens on `:8080`. Before you expose it beyond localhost, change `jwt.secret` in the config — see [SECURITY.md](SECURITY.md). Startup warns while the placeholder is still in place.

The default config already carries `server` timeouts and a body cap; do not delete them. Serving the UI from another origin needs that origin in `security.allowed_origins` (it gates CORS *and* the WebSocket upgrade) — `ESP_SECURITY_ALLOWED_ORIGINS` overrides the list without editing the file.

```bash
make build # build to bin/e-sp-line2
Expand Down
8 changes: 5 additions & 3 deletions .github/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,18 +52,20 @@ Out of scope:
### Backend

- Passwords are stored as bcrypt hashes, never in plaintext.
- API tokens are JWTs signed with HS256 and carry an expiry (`jwt.expiration`, 24 hours by default). Rotate `jwt.secret` in `backend/config/config.yaml` per deployment.
- API tokens are JWTs signed with HS256 and carry an expiry (`jwt.expiration`, 24 hours by default). Rotate `jwt.secret` in `config/config.yaml` per deployment; startup logs a WARN while the shipped placeholder is still in place.
- The `/api/v1` surface except `/auth/register` and `/auth/login` sits behind `AuthRequired`. `AdminRequired` exists but is not wired into any route yet.
- An adapter's `security` policy declares `sensitive_fields` (for Taobao and Xianyu, `cookie`) so a manifest can mark what must not be echoed back.
- CORS echoes a `security.allowed_origins` member instead of a wildcard, and the WebSocket upgraders (bridge, app and adapter gateway) check the same list. A cross-origin socket is refused with 403 before the upgrade, and both sockets require a valid JWT during the handshake (401 otherwise).
- The HTTP server runs with a header timeout, read/write/idle timeouts and a request-body cap, so a stalled or oversized request cannot pin a connection.

### What is not there yet

Be aware of these when you deploy, and please send a report if one of them bites you:

- The adapter instance `config` column stores its Cookie as plain JSON in the database, despite the `encrypted_fields` declaration. Treat the SQLite/PostgreSQL file as a secret.
- `AuthRequired` is still a placeholder: it checks the presence and `Bearer` shape of the header but does not yet verify the JWT signature, and it sets a fixed `user-123` identity. Do not expose the API to the internet and assume per-user isolation until that lands.
- CORS sends `Access-Control-Allow-Origin: *` with `Access-Control-Allow-Credentials: true`, and the WebSocket upgrader accepts any origin. Put the front end behind the same origin as the API in production.
- The default `jwt.secret` is a placeholder; change it before you deploy.
- The default `jwt.secret` is a placeholder; startup only warns, it does not refuse to boot. Change it before you deploy.
- `security.allowed_origins` defaults to `localhost` only. A real deployment must list its front-end origin (or set `ESP_SECURITY_ALLOWED_ORIGINS`); a `*` is not accepted, the list is matched exactly.

### Front end

Expand Down
11 changes: 9 additions & 2 deletions adapters/taobao/esp_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import asyncio
import json
import time
import urllib.parse
import uuid
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, List, Optional
Expand Down Expand Up @@ -188,9 +189,15 @@ def __init__(
self.running = False

def _ws_url(self) -> str:
"""构造后端适配器 WebSocket 地址。"""
"""构造后端适配器 WebSocket 地址。

token 走 query 参数:后端在升级握手前校验它,缺失直接回 401。
"""
base = self.backend_url.replace("http://", "ws://").replace("https://", "wss://")
return f"{base}/ws/adapter?instance_id={self.instance_id}"
url = f"{base}/ws/adapter?instance_id={self.instance_id}"
if self.token:
url += f"&token={urllib.parse.quote(self.token, safe='')}"
return url

def _tag(self) -> str:
"""短标签:instance_id 前 8 位,便于在日志中区分多个桥。"""
Expand Down
10 changes: 9 additions & 1 deletion adapters/xianyu/esp_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import asyncio
import json
import time
import urllib.parse
import uuid
from dataclasses import dataclass, field
from typing import Any, Dict, Optional, Callable
Expand Down Expand Up @@ -170,8 +171,15 @@ def __init__(
self.running = False

def _ws_url(self) -> str:
"""构造后端适配器 WebSocket 地址。

token 走 query 参数:后端在升级握手前校验它,缺失直接回 401。
"""
base = self.backend_url.replace("http://", "ws://").replace("https://", "wss://")
return f"{base}/ws/adapter?instance_id={self.instance_id}"
url = f"{base}/ws/adapter?instance_id={self.instance_id}"
if self.token:
url += f"&token={urllib.parse.quote(self.token, safe='')}"
return url

def _tag(self) -> str:
"""短标签:instance_id 前 8 位,便于在日志中区分多个桥。"""
Expand Down
22 changes: 22 additions & 0 deletions config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,31 @@ log_level: "info" # debug, info, warn, error

# JWT settings
jwt:
# The shipped value is a placeholder: while it is in place startup logs a
# WARN and anyone can forge a token. Generate one per deployment, e.g.
# openssl rand -base64 32
secret: "change-this-secret-in-production"
expiration: 24 # hours

# HTTP server settings — all seconds, 0 disables a timeout.
server:
read_header_timeout: 10
read_timeout: 60
write_timeout: 120
idle_timeout: 120
max_body_bytes: 4194304 # 4 MiB

# Browser security. One allowlist, used by CORS *and* the WebSocket Origin
# check. Add the real front-end Origin when the UI is served from elsewhere;
# ESP_SECURITY_ALLOWED_ORIGINS (comma-separated) overrides this list. An empty
# list means "allow any Origin" (development fallback).
security:
allowed_origins:
- "http://localhost:3000"
- "http://127.0.0.1:3000"
- "http://localhost:8080"
- "http://127.0.0.1:8080"

# Adapter settings
adapter:
heartbeat_interval: 30 # seconds
Expand Down
41 changes: 39 additions & 2 deletions docs/developer-guide/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -752,9 +752,26 @@ Public. Returns the message field contract that [message-fields.md](./message-fi

## WebSocket interface

Both sockets authenticate during the HTTP handshake, before the upgrade. A
missing or invalid token is answered with **401** as a plain HTTP response, and
an `Origin` outside the allowlist with **403**, so a failure is readable rather
than an immediately-closed socket.

The token is accepted in any of three forms, because the callers differ:

| Transport | Caller |
|---|---|
| `?token=<jwt>` query parameter | the Python bridges (they cannot set headers on connect) |
| `Authorization: Bearer <jwt>` header | server-to-server clients, `websocat`, tests |
| `Sec-WebSocket-Protocol: espl, <jwt>` | browsers — the only header a page can set on `WebSocket` |

`Origin` is checked against `security.allowed_origins` (the same list the CORS
middleware uses; `ESP_SECURITY_ALLOWED_ORIGINS` overrides it). A request without
an `Origin` header is a non-browser client and is allowed through.

### Adapter WebSocket

**URL**: `ws://localhost:8080/ws/adapter?instance_id=<instance_id>`
**URL**: `ws://localhost:8080/ws/adapter?instance_id=<instance_id>&token=<jwt>`

**Message format**:
```json
Expand All @@ -776,7 +793,7 @@ Public. Returns the message field contract that [message-fields.md](./message-fi

### App WebSocket

**URL**: `ws://localhost:8080/ws/app?app_id=<app_id>`
**URL**: `ws://localhost:8080/ws/app?app_id=<app_id>&token=<jwt>`

**Message format**:
```json
Expand All @@ -796,6 +813,25 @@ Public. Returns the message field contract that [message-fields.md](./message-fi
}
```

## CORS

Browsers get the request `Origin` echoed back in
`Access-Control-Allow-Origin` when it appears in `security.allowed_origins`
(defaults cover `localhost:3000` and `localhost:8080`); `Vary: Origin` is set so
caches key on it. Because the value is a concrete origin rather than `*`,
`Access-Control-Allow-Credentials: true` is meaningful and a future Cookie-based
session keeps working. Requests from an unlisted origin still reach the handler
but carry no CORS headers, so the browser blocks the response. Requests without
an `Origin` (curl, the Python bridges) are unaffected.

## Limits

- Request bodies are capped by `server.max_body_bytes` (default 4 MiB); an
oversized body is refused with **413**.
- The HTTP server enforces `server.read_header_timeout` (default 10s),
`read_timeout`, `write_timeout` and `idle_timeout`. All are seconds; `0`
disables the corresponding timeout.

## Error codes

| Error code | Description |
Expand All @@ -805,6 +841,7 @@ Public. Returns the message field contract that [message-fields.md](./message-fi
| 403 | Forbidden |
| 404 | Resource not found |
| 409 | Resource conflict |
| 413 | Request body too large |
| 500 | Internal server error |

## Rate limiting
Expand Down
16 changes: 12 additions & 4 deletions docs/developer-guide/functional-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ second insert with empty idempotency_key:

**建议**:要么回显 `Origin` 并配白名单,要么明确只走 Bearer 不打算用 Cookie,把 `Allow-Credentials` 删掉以免误导。

**现已修复**(P1/ESPL-20):走了"回显 + 白名单"这条路,因为前端将来未必一直用 Bearer。`internal/middleware/middleware.go` 的 `CORS` 现在只对 `security.allowed_origins` 里的 Origin 回 `Access-Control-Allow-Origin: <origin>` 并保留 `Allow-Credentials: true`,同时加 `Vary: Origin`;不带 `Origin` 的客户端(curl、Python 桥)完全不受影响。实测:允许的 Origin 预检拿到 204 + 回显 + `Vary`,外来 Origin 只剩 `Vary`、无 `Access-Control-Allow-Origin`。

### M3. 默认 JWT secret 两处硬编码,且没有任何"是否已改"的检查

**定位**:`internal/config/config.go:126`(viper 默认值)、`config/config.yaml:32`(随仓库提交的值),内容都是 `change-this-secret-in-production`。
Expand All @@ -176,6 +178,8 @@ second insert with empty idempotency_key:

**建议**:`config.Load` 后若 secret 仍是占位串则打 WARN(或直接拒绝启动);README/SECURITY 已有的提醒保留。

**现已修复**(P1/ESPL-20):选了 WARN 而不是拒启,避免把现有部署一下卡死。判定收敛到 `config.UsingPlaceholderJWTSecret()`(占位串或空串都算),`server.New` 启动时打一条 WARN 并带修法提示。只做前缀比较不够——`change-this-secret-in-production-prod` 已是真轮换,不该报警,测试里钉住了这条。实测启动日志:`WARN ... JWT secret is still the shipped placeholder`。

### M4. WebSocket `CheckOrigin` 恒真,且握手不做鉴权

**定位**:`internal/handler/websocket.go:17` — `return true // Allow all origins for development`。
Expand All @@ -184,6 +188,8 @@ second insert with empty idempotency_key:

**建议**:接 B1 后校验 `Sec-WebSocket-Protocol` 或 `?token=`,`CheckOrigin` 收白名单。

**现已修复**(P1/ESPL-20):两条 WS 在**升级前**校验 token,缺失或非法都回 401(不再只是"验个形状");`CheckOrigin` 改为查与 CORS 同一份 `security.allowed_origins`,跨域来源在升级前被 403 拒掉;适配器网关的 upgrader 也接了同一份列表,`GatewayConfig.AllowedOrigin` 注入。token 三种承载:`?token=`(Python 桥,已同步改 `adapters/*/esp_bridge.py`)、`Authorization: Bearer`、`Sec-WebSocket-Protocol: espl, <jwt>`(浏览器唯一可设的头)。无 `Origin` 的非浏览器客户端放行。注意这是**鉴权到位**,不是**授权**:握手只验 token 有效,"这个 token 能不能碰这个 instance_id"仍属 M1。

### M5. `POST /platforms` 等三个 handler 有实现、无路由

**现象**:`CreatePlatform`、`UpdatePlatform`、`DeletePlatform` 三个 handler 完整存在,但 `server.go` 只注册了 `GET ""` 和 `GET "/:id"`。实测 `POST /api/v1/platforms` → **404**。
Expand Down Expand Up @@ -322,6 +328,8 @@ TestReviewDoubleClose → PANIC: close of closed channel

**建议**:补 `ReadHeaderTimeout` / `ReadTimeout` / `WriteTimeout` / `IdleTimeout`,加 `MaxBytesReader`。

**现已修复**(P1/ESPL-20):四类超时全部落到 `http.Server`,值来自 `server.*` 配置(默认 10s/60s/120s/120s,0 = 不设)。请求体上限 `server.max_body_bytes`(默认 4 MiB)由全局中间件 `MaxBodyBytes` 执行:声明超限直接 413,不声明(chunked/撒谎的 Content-Length)由 `http.MaxBytesReader` 兜住。实测 8KB body → 413、正常 body → 200;只发一半头后挂住的连接 **2.0s**(配置值)被 `ReadHeaderTimeout` 断开。

### M16. `make migrate` 与 `make dev` 传的参数 `main.go` 根本不读

**现象**:`main.go` 没有任何 `flag.` / `os.Args` 解析(`grep` 无命中)。实测带 `migrate` 参数启动,进程**照常起 HTTP 服务**,不迁移、不退出。
Expand Down Expand Up @@ -456,9 +464,9 @@ Error: EROFS: read-only file system, open '.../vite.config.ts.timestamp-17940267

7. **M7 API 基址**:改 `import.meta.env.VITE_API_BASE_URL ?? '/api/v1'`,健康检查走相对路径。
8. **M8/M9 登录态**:401 统一 `clearToken` + 跳 `/login`;`handleLogout` 改用 client;加 `RequireAuth` 路由守卫。
9. **M3 JWT secret**:仍是占位串就 WARN 或拒启。
10. **M4 WS 鉴权 + CheckOrigin 白名单**(依赖 P0-1)。
11. **M2 CORS**:回显 Origin 配白名单,或删掉 `Allow-Credentials`。
9. **M3 JWT secret**:已完成(P1/ESPL-20,启动 WARN)。
10. **M4 WS 鉴权 + CheckOrigin 白名单**:已完成(P1/ESPL-20,升级前 401 / 跨域 403,白名单与 CORS 同一份)。
11. **M2 CORS**:已完成(P1/ESPL-20,回显白名单 Origin + `Vary: Origin`)。

### P2 — 正确性与可观测性

Expand All @@ -467,7 +475,7 @@ Error: EROFS: read-only file system, open '.../vite.config.ts.timestamp-17940267
14. **M5**:补齐 `POST/PUT/DELETE /platforms`,挂 `AuthRequired` + `AdminRequired`。
15. **M13 测试**:先鉴权、再 WS 端到端;`CGO_ENABLED=0` 下用探测 + `t.Skip` 兜底。
16. **M14**:已完成(`sync.Once` + 五个用例)。剩余只需把 `logger` 未初始化就 nil 的问题在 `pkg/logger` 兜掉,免得每个包写一遍 `TestMain`。
17. **M15**:补四类超时 + `MaxBytesReader`。
17. **M15**:已完成(P1/ESPL-20,四类超时 + body 上限,超限 413)。
18. **m1/m2**:`Hub` 建好后加载已启用规则并在增删改时同步内存;`Priority`/`Enabled` 改指针类型。

### P3 — 文档与卫生
Expand Down
13 changes: 12 additions & 1 deletion internal/adaptergateway/gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ type GatewayConfig struct {
ReadTimeout int
// MaxMessageSize is the maximum inbound frame size in bytes.
MaxMessageSize int64
// AllowedOrigin reports whether a browser Origin may connect. A nil value
// allows every Origin, which is what an embedder that has not configured an
// allowlist gets. The HTTP CORS middleware and the WebSocket upgrader share
// this one rule so the two can never disagree.
AllowedOrigin func(string) bool
}

// DefaultGatewayConfig returns the default gateway configuration.
Expand Down Expand Up @@ -298,7 +303,13 @@ func NewGateway(cfg GatewayConfig, svc *service.AdapterGatewayService) *Gateway
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(r *http.Request) bool {
return true // Allow all origins for development
origin := r.Header.Get("Origin")
// A missing Origin is a non-browser client; those are not
// subject to CORS at all.
if origin == "" || cfg.AllowedOrigin == nil {
return true
}
return cfg.AllowedOrigin(origin)
},
},
}
Expand Down
92 changes: 91 additions & 1 deletion internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package config
import (
"fmt"
"os"
"strings"

"github.com/spf13/viper"
)
Expand All @@ -26,10 +27,80 @@ type Config struct {
// JWT settings
JWT JWTConfig `mapstructure:"jwt"`

// HTTP server settings
Server ServerConfig `mapstructure:"server"`

// Security settings (shared by CORS and the WebSocket Origin check)
Security SecurityConfig `mapstructure:"security"`

// Adapter settings
Adapter AdapterConfig `mapstructure:"adapter"`
}

// PlaceholderJWTSecret is the value shipped in config/config.yaml and used as
// the viper default. A deployment that never changed it signs tokens with a
// public key, so startup warns when it is still in place.
const PlaceholderJWTSecret = "change-this-secret-in-production"

// DefaultAllowedOrigins is the development Origin allowlist. It covers the Vite
// dev server and the backend's own default port; a deployment adds its real
// front-end Origin via config.yaml or the ESP_SECURITY_ALLOWED_ORIGINS env var.
func DefaultAllowedOrigins() []string {
return []string{
"http://localhost:3000",
"http://127.0.0.1:3000",
"http://localhost:8080",
"http://127.0.0.1:8080",
}
}

// ServerConfig holds HTTP server timeouts and request-size limits.
type ServerConfig struct {
// ReadHeaderTimeout bounds how long a client may take to send headers.
ReadHeaderTimeout int `mapstructure:"read_header_timeout"` // seconds
// ReadTimeout bounds the whole request read.
ReadTimeout int `mapstructure:"read_timeout"` // seconds
// WriteTimeout bounds the response write.
WriteTimeout int `mapstructure:"write_timeout"` // seconds
// IdleTimeout bounds an idle keep-alive connection.
IdleTimeout int `mapstructure:"idle_timeout"` // seconds
// MaxBodyBytes caps the request body size.
MaxBodyBytes int64 `mapstructure:"max_body_bytes"` // bytes
}

// SecurityConfig holds cross-cutting browser-security settings.
type SecurityConfig struct {
// AllowedOrigins is the Origin allowlist used both by the CORS middleware
// and by the WebSocket CheckOrigin hook. One list, so the two can never
// drift apart.
AllowedOrigins []string `mapstructure:"allowed_origins"`
}

// IsAllowedOrigin reports whether origin is in the CORS/WebSocket allowlist.
// The empty Origin (non-browser clients such as the Python bridges) is allowed,
// as is an empty allowlist meaning "development default", i.e. allow all.
func (c *Config) IsAllowedOrigin(origin string) bool {
if origin == "" {
return true
}
allowed := c.Security.AllowedOrigins
if len(allowed) == 0 {
return true
}
for _, a := range allowed {
if a == origin {
return true
}
}
return false
}

// UsingPlaceholderJWTSecret reports whether the JWT secret is still the
// shipped placeholder.
func (c *Config) UsingPlaceholderJWTSecret() bool {
return c.JWT.Secret == "" || c.JWT.Secret == PlaceholderJWTSecret
}

// DatabaseConfig represents database configuration
type DatabaseConfig struct {
Driver string `mapstructure:"driver"`
Expand Down Expand Up @@ -147,9 +218,28 @@ func setDefaults(v *viper.Viper) {
v.SetDefault("log_level", "info")

// JWT defaults
v.SetDefault("jwt.secret", "change-this-secret-in-production")
v.SetDefault("jwt.secret", PlaceholderJWTSecret)
v.SetDefault("jwt.expiration", 24)

// HTTP server defaults. Without these a single slow client can hold a
// connection forever (Slowloris) and a large body is read entirely into
// memory.
v.SetDefault("server.read_header_timeout", 10)
v.SetDefault("server.read_timeout", 60)
v.SetDefault("server.write_timeout", 120)
v.SetDefault("server.idle_timeout", 120)
v.SetDefault("server.max_body_bytes", 4<<20) // 4 MiB

// Origin allowlist shared by CORS and the WebSocket Origin check.
v.SetDefault("security.allowed_origins", DefaultAllowedOrigins())

// Env is read with ESP_* names; AutomaticEnv alone cannot see a slice, so
// the comma-separated override is bound explicitly. This lets a deployment
// set its real front-end Origin without editing config.yaml.
v.SetEnvPrefix("esp")
v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
_ = v.BindEnv("security.allowed_origins")

// Adapter defaults
v.SetDefault("adapter.heartbeat_interval", 30)
v.SetDefault("adapter.reconnect_delay", 5)
Expand Down
Loading
Loading