Skip to content

operations.md

Latest commit

 

History

History
328 lines (244 loc) · 6.58 KB

operations.md

File metadata and controls

328 lines (244 loc) · 6.58 KB

Operations Manual

This document provides guidance for deployment and Kubernetes operations.

Table of Contents

Deployment

Prerequisites

  • Kubernetes cluster with KubeBlocks installed
  • Sealos platform (recommended) or any K8s cluster
  • Container registry access

Environment Variables

Required environment variables for production:

# Database
DATABASE_URL="postgresql://..."

# NextAuth
NEXTAUTH_URL="https://your-domain.com"
NEXTAUTH_SECRET="your-secret"
AUTH_TRUST_HOST="true"

# GitHub OAuth
GITHUB_CLIENT_ID="..."
GITHUB_CLIENT_SECRET="..."

# GitHub App (optional)
GITHUB_APP_ID="..."
GITHUB_APP_PRIVATE_KEY="..."

# Sealos OAuth
SEALOS_JWT_SECRET="..."

# Kubernetes
RUNTIME_IMAGE="docker.io/limbo2342/fullstack-web-runtime:sha-ca2470e"

# Aiproxy (optional)
AIPROXY_ENDPOINT="..."
ANTHROPIC_BASE_URL="..."

# Logging
LOG_LEVEL="info"

Deploy to Sealos

  1. Build and push Docker image:
docker build -t your-registry/fulling:latest .
docker push your-registry/fulling:latest
  1. Create Sealos application with the image
  2. Configure environment variables
  3. Set up domain and SSL

Resource Requirements

Main Application:

  • CPU: 500m request, 2000m limit
  • Memory: 512Mi request, 2Gi limit

Per Sandbox:

  • CPU: 20m request, 2000m limit
  • Memory: 25Mi request, 4096Mi limit
  • Storage: 10Gi PVC

Per Database:

  • CPU: 100m request, 1000m limit
  • Memory: 102Mi request, 1024Mi limit
  • Storage: 3Gi PVC

Kubernetes Resources

Per Project Resources

Each project creates the following K8s resources:

1. StatefulSet (Sandbox):

  • Name: {project-name}-{random-6chars}
  • Image: limbo2342/fullstack-web-runtime:sha-ca2470e
  • Ports: 3000 (app), 7681 (ttyd), 8080 (filebrowser)
  • PVC: 10Gi (persistent storage)

2. Service:

  • Name: {sandbox-name}-service
  • Ports: 3000, 7681, 8080

3. Ingresses:

  • App: {sandbox-name}-app.{domain} → 3000
  • Terminal: {sandbox-name}-ttyd.{domain} → 7681
  • FileBrowser: {sandbox-name}-filebrowser.{domain} → 8080

4. PostgreSQL Cluster (KubeBlocks):

  • Name: {project-name}-{random-6chars}
  • Version: postgresql-14.8.0
  • Storage: 3Gi

Labels

All resources have these labels:

labels:
  cloud.sealos.io/app-deploy-manager: {resource-name}
  project.fullstackagent.io/name: {k8s-project-name}
  app: {resource-name}

Resource Management

Start/Stop Projects

Stop (scales replicas to 0):

# Via API
POST /api/projects/{id}/stop

# Via kubectl
kubectl scale statefulset {sandbox-name} --replicas=0 -n {namespace}

Start (scales replicas to 1):

# Via API
POST /api/projects/{id}/start

# Via kubectl
kubectl scale statefulset {sandbox-name} --replicas=1 -n {namespace}

Delete Projects

# Via API (soft delete)
POST /api/projects/{id}/delete

# Via kubectl (hard delete)
kubectl delete statefulset {sandbox-name} -n {namespace}
kubectl delete service {sandbox-name}-service -n {namespace}
kubectl delete ingress {sandbox-name}-app-ingress -n {namespace}
kubectl delete ingress {sandbox-name}-ttyd-ingress -n {namespace}
kubectl delete ingress {sandbox-name}-filebrowser-ingress -n {namespace}
kubectl delete pvc data-{sandbox-name}-0 -n {namespace}

Update Runtime Image

  1. Build and push new image:
cd runtime
./build.sh
./push-to-dockerhub.sh
  1. Update lib/k8s/versions.ts:
export const VERSIONS = {
  RUNTIME_IMAGE: 'docker.io/limbo2342/fullstack-web-runtime:sha-new',
  // ...
}

  1. Restart main app to pick up new version

  2. Existing sandboxes will use new image on next restart

Monitoring

Health Checks

Main Application:

curl https://your-domain.com/api/health

Sandbox Containers:

kubectl get pods -n {namespace} -l app={sandbox-name}

Logs

Main Application:

# Sealos
sealos logs {app-name}

# kubectl
kubectl logs -n {namespace} {pod-name}

Sandbox Containers:

kubectl logs -n {namespace} {sandbox-pod-name}

# ttyd logs
kubectl logs -n {namespace} {sandbox-pod-name} | grep ttyd

Metrics

Key metrics to monitor:

  • API response time (< 100ms)
  • Reconciliation job duration (< 5s)
  • Database connections
  • K8s API calls
  • Pod restart count

Backup and Recovery

Database Backup

Main Application Database:

pg_dump -h localhost -U user -d fullstackagent > backup.sql

Project Databases (KubeBlocks):

# KubeBlocks automatic backup
kubectl get backup -n {namespace}

# Manual backup
kubectl apply -f - <<EOF
apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: Backup
metadata:
  name: manual-backup
  namespace: {namespace}
spec:
  backupPolicyName: {cluster-name}-backup-policy
  backupType: full
EOF

Recovery

Main Application:

psql -h localhost -U user -d fullstackagent < backup.sql

Project Database:

# Restore from backup
kubectl apply -f - <<EOF
apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: Restore
metadata:
  name: restore-from-backup
  namespace: {namespace}
spec:
  backupName: manual-backup
  restoreTime: "2024-01-15T10:00:00Z"
EOF

PVC Recovery

If sandbox PVC is corrupted:

# 1. Stop sandbox
kubectl scale statefulset {sandbox-name} --replicas=0 -n {namespace}

# 2. Delete PVC
kubectl delete pvc data-{sandbox-name}-0 -n {namespace}

# 3. Start sandbox (creates new PVC)
kubectl scale statefulset {sandbox-name} --replicas=1 -n {namespace}

Common Operations

Check Resource Status

# List all sandboxes
kubectl get statefulsets -A -l project.fullstackagent.io/name

# List all databases
kubectl get clusters -A

# Check specific sandbox
kubectl describe statefulset {sandbox-name} -n {namespace}

Access Sandbox Shell

# Via kubectl
kubectl exec -it -n {namespace} {pod-name} -- /bin/bash

# Via ttyd (web terminal)
# Open https://{sandbox-name}-ttyd.{domain}?authorization={token}

Debug Network Issues

# Check ingress
kubectl get ingress -n {namespace}
kubectl describe ingress {sandbox-name}-app-ingress -n {namespace}

# Check service
kubectl get svc -n {namespace}
kubectl describe svc {sandbox-name}-service -n {namespace}

# Check endpoints
kubectl get endpoints -n {namespace}

Related Documentation