# Troubleshooting

Use the items below to identify common Keystone Gateway issues, understand likely causes, and apply the recommended resolution.

## Handle logs and diagnostics safely

Gateway-owned structured logs redact recognized sensitive keys and, for string/error values, high-confidence credential representations, including nested structured attributes. This protection is defense in depth, not a complete secret detector: arbitrary opaque secrets, request/configuration objects, and raw error or message strings may still contain sensitive data. Keep credentials out of anything passed to logging, use safe contextual errors, and inspect logs, exporter output, and startup stderr before sharing them. Startup errors are a separate diagnostic path and are not covered by the structured-log redactor. See [customer-safe diagnostics](https://portal.cjcit.io/knowledge-base/keystone_gateway/0.2/troubleshooting/diagnostics/) for the collection procedure.

## Licensing

## Invalid license

### Cause

**Symptoms.** Service startup fails with `gateway license validation failed`, or `/readyz`/lease operations fail after expiry.

**Likely causes.** Expired, malformed, tampered, untrusted-signature, wrong Gateway ID, or unlisted replica ID.

### Resolution

**Diagnosis.** Check `journalctl -u keystone-gateway -n 100`, `node-id --config ...`, and the safe `/v1/license` projection when running. Do not send the raw signature or secrets.

**Resolution.** Install a valid license naming the configured IDs and restart. See [licensing](https://portal.cjcit.io/knowledge-base/keystone_gateway/0.2/getting-started/licensing/).

**Escalation information to collect.** Version, node-ID output, timestamp/time source status, redacted logs, and license metadata.

## License issued to the wrong node

### Cause

**Symptoms.** Startup validation fails despite an otherwise valid file.

**Likely causes.** `gateway_id` or `ha.replica_id` differs from the license claims.

### Resolution

**Diagnosis.** Compare `node-id` JSON with the license’s `gateway_id` and `nodes` values.

**Resolution.** Correct a configuration error or request a replacement license. Do not edit a signed document.

**Escalation information to collect.** Both identity values and redacted license metadata.

## Leases and capacity

## Capacity exhaustion

### Cause

**Symptoms.** Acquire returns HTTP 409 `capacity_exhausted`; product startup cannot obtain capacity.

**Likely causes.** All declared units are held by live leases, or HA reconciliation is over capacity.

### Resolution

**Diagnosis.** Query authenticated `/v1/entitlements`, inspect lease rejection metrics and `/v1/ha`.

**Resolution.** Release unused leases, wait for expiry, reduce requested counts, or obtain sufficient capacity. Do not delete state to free capacity.

**Escalation information to collect.** Entitlement projection, component, correlation ID, lease rejection metrics, and HA status.

## Failed lease renewal

### Cause

**Symptoms.** Client enters degraded state or becomes unlicensed at its recorded expiry.

**Likely causes.** Network failure, TLS/auth failure, standby target, store failure, expired license, or clock issue.

### Resolution

**Diagnosis.** Check client logs, `/readyz`, Gateway logs using correlation ID, connectivity to both endpoints, and error code. `not_active`, `store_unavailable`, and `rate_limited` are retryable; preserve the same idempotency key.

**Resolution.** Restore connectivity/authentication or route to a ready peer. A lease cannot be revived after expiry; acquire a new one.

**Escalation information to collect.** Client/Gateway version, error code, correlation ID, endpoint test, readiness, and time status.

## Telemetry

## Telemetry failure

### Cause

**Symptoms.** Collector has no recent data or exporter errors appear in logs.

**Likely causes.** Incorrect endpoint/protocol, CA/header, DNS/firewall, or collector outage.

### Resolution

**Diagnosis.** Validate telemetry configuration and network path; distinguish it from `/readyz` and lease operations.

**Resolution.** Repair Collector connectivity/configuration. Telemetry failure does not stop Gateway licensing.

**Escalation information to collect.** Redacted telemetry configuration, exporter logs, protocol/endpoint, and readiness result.

## High availability

## HA peer failure

### Cause

**Symptoms.** `peer_connected` is false, split-brain warning is true, or a node becomes `degraded_active`.

**Likely causes.** Peer process/network loss, peer TLS identity failure, clock skew, or partition.

### Resolution

**Diagnosis.** Query `/v1/ha` on both nodes; check both peer ports and certificate target/CN, then logs and time synchronization.

**Resolution.** Restore peer transport. Expect convergence; if both sides issued leases, new acquisitions remain blocked until capacity is no longer exceeded. See [Active/Standby](https://portal.cjcit.io/knowledge-base/keystone_gateway/0.2/deployment/active-standby/).

**Escalation information to collect.** Both HA projections, peer connectivity tests, clock status, correlation IDs, and redacted logs.