Chapter 18: DevOps and CI/CD Integration
Overview
This chapter focuses on integrating Vektagraf applications into modern DevOps workflows and CI/CD pipelines. We'll explore automated testing strategies, infrastructure as code patterns, deployment automation, security scanning, and compliance validation to ensure reliable and secure delivery of Vektagraf applications.
Learning Objectives
- Implement comprehensive CI/CD pipelines for Vektagraf applications
- Master infrastructure as code patterns and configuration management
- Set up automated deployment and rollback procedures
- Integrate security scanning and compliance validation
Prerequisites
- Understanding of CI/CD concepts and tools
- Familiarity with containerization and Kubernetes
- Knowledge of infrastructure as code principles
- Basic understanding of security scanning and compliance
Core Concepts
CI/CD Pipeline Architecture
A comprehensive CI/CD pipeline for Vektagraf applications includes multiple stages:
graph LR
A[Source Code] --> B[Build & Test]
B --> C[Security Scan]
C --> D[Package & Push]
D --> E[Deploy to Staging]
E --> F[Integration Tests]
F --> G[Security Tests]
G --> H[Deploy to Production]
H --> I[Smoke Tests]
I --> J[Monitor & Alert]
Pipeline Configuration
// lib/cicd/pipeline_config.dart
class PipelineConfig {
final BuildConfig build;
final TestConfig test;
final SecurityConfig security;
final DeploymentConfig deployment;
const PipelineConfig({
required this.build,
required this.test,
required this.security,
required this.deployment,
});
Map<String, dynamic> toJson() => {
'build': build.toJson(),
'test': test.toJson(),
'security': security.toJson(),
'deployment': deployment.toJson(),
};
}
class BuildConfig {
final String dartVersion;
final List<String> buildTargets;
final bool enableOptimization;
final Map<String, String> buildArgs;
const BuildConfig({
this.dartVersion = '3.0.0',
this.buildTargets = const ['linux', 'docker'],
this.enableOptimization = true,
this.buildArgs = const {},
});
Map<String, dynamic> toJson() => {
'dartVersion': dartVersion,
'buildTargets': buildTargets,
'enableOptimization': enableOptimization,
'buildArgs': buildArgs,
};
}
GitHub Actions CI/CD Pipeline
Complete Pipeline Configuration
# .github/workflows/ci-cd.yml
name: CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
release:
types: [published]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
# Build and Test Stage
build-and-test:
runs-on: ubuntu-latest
strategy:
matrix:
dart-version: ['3.0.0', '3.1.0']
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Dart
uses: dart-lang/setup-dart@v1
with:
dart-version: ${{ matrix.dart-version }}
- name: Cache dependencies
uses: actions/cache@v3
with:
path: |
~/.pub-cache
.dart_tool
key: ${{ runner.os }}-dart-${{ matrix.dart-version }}-${{ hashFiles('**/pubspec.lock') }}
restore-keys: |
${{ runner.os }}-dart-${{ matrix.dart-version }}-
- name: Install dependencies
run: dart pub get
- name: Verify formatting
run: dart format --output=none --set-exit-if-changed .
- name: Analyze code
run: dart analyze --fatal-infos
- name: Run unit tests
run: dart test --coverage=coverage
- name: Generate coverage report
run: |
dart pub global activate coverage
dart pub global run coverage:format_coverage \
--lcov --in=coverage --out=coverage/lcov.info \
--packages=.dart_tool/package_config.json \
--report-on=lib
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: coverage/lcov.info
flags: unittests
name: codecov-umbrella
- name: Build application
run: |
dart compile exe bin/server.dart -o build/vektagraf-server
dart compile exe bin/client.dart -o build/vektagraf-client
- name: Upload build artifacts
uses: actions/upload-artifact@v3
with:
name: vektagraf-binaries-${{ matrix.dart-version }}
path: build/
# Security Scanning Stage
security-scan:
runs-on: ubuntu-latest
needs: build-and-test
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
format: 'sarif'
output: 'trivy-results.sarif'
- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: 'trivy-results.sarif'
- name: Run Semgrep security scan
uses: returntocorp/semgrep-action@v1
with:
config: >-
p/security-audit
p/secrets
p/dart
- name: Dependency vulnerability scan
run: |
dart pub deps --json > deps.json
# Custom script to check for known vulnerabilities
python scripts/check_vulnerabilities.py deps.json
# Container Build and Push Stage
build-and-push:
runs-on: ubuntu-latest
needs: [build-and-test, security-scan]
if: github.event_name != 'pull_request'
permissions:
contents: read
packages: write
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,prefix={{branch}}-
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
VERSION=${{ github.sha }}
BUILD_DATE=${{ github.event.head_commit.timestamp }}
- name: Sign container image
run: |
cosign sign --yes ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
env:
COSIGN_EXPERIMENTAL: 1
# Deploy to Staging
deploy-staging:
runs-on: ubuntu-latest
needs: build-and-push
if: github.ref == 'refs/heads/develop'
environment: staging
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Configure kubectl
uses: azure/k8s-set-context@v3
with:
method: kubeconfig
kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}
- name: Deploy to staging
run: |
envsubst < k8s/staging/deployment.yaml | kubectl apply -f -
kubectl rollout status deployment/vektagraf-app -n vektagraf-staging
env:
IMAGE_TAG: ${{ github.sha }}
- name: Run smoke tests
run: |
kubectl wait --for=condition=ready pod -l app=vektagraf -n vektagraf-staging --timeout=300s
./scripts/smoke-tests.sh staging
# Integration Tests
integration-tests:
runs-on: ubuntu-latest
needs: deploy-staging
if: github.ref == 'refs/heads/develop'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Dart
uses: dart-lang/setup-dart@v1
- name: Install dependencies
run: dart pub get
- name: Run integration tests
run: dart test test/integration/
env:
VEKTAGRAF_ENDPOINT: ${{ secrets.STAGING_ENDPOINT }}
VEKTAGRAF_API_KEY: ${{ secrets.STAGING_API_KEY }}
- name: Run performance tests
run: |
dart run test/performance/load_test.dart
dart run test/performance/stress_test.dart
- name: Generate test report
run: |
dart run test/generate_report.dart
- name: Upload test results
uses: actions/upload-artifact@v3
with:
name: integration-test-results
path: test-results/
# Security Testing
security-tests:
runs-on: ubuntu-latest
needs: deploy-staging
if: github.ref == 'refs/heads/develop'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run OWASP ZAP security scan
uses: zaproxy/action-full-scan@v0.7.0
with:
target: ${{ secrets.STAGING_ENDPOINT }}
rules_file_name: '.zap/rules.tsv'
cmd_options: '-a'
- name: Run custom security tests
run: |
dart test test/security/
env:
TARGET_ENDPOINT: ${{ secrets.STAGING_ENDPOINT }}
- name: Upload security scan results
uses: actions/upload-artifact@v3
with:
name: security-scan-results
path: report_html.html
# Deploy to Production
deploy-production:
runs-on: ubuntu-latest
needs: [integration-tests, security-tests]
if: github.event_name == 'release'
environment: production
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Configure kubectl
uses: azure/k8s-set-context@v3
with:
method: kubeconfig
kubeconfig: ${{ secrets.KUBE_CONFIG_PRODUCTION }}
- name: Deploy with blue-green strategy
run: |
./scripts/blue-green-deploy.sh ${{ github.event.release.tag_name }}
- name: Run production smoke tests
run: |
./scripts/smoke-tests.sh production
- name: Update monitoring dashboards
run: |
./scripts/update-dashboards.sh ${{ github.event.release.tag_name }}
# Post-deployment monitoring
post-deployment:
runs-on: ubuntu-latest
needs: deploy-production
if: github.event_name == 'release'
steps:
- name: Monitor deployment health
run: |
./scripts/monitor-deployment.sh 300 # Monitor for 5 minutes
- name: Send deployment notification
uses: 8398a7/action-slack@v3
with:
status: ${{ job.status }}
channel: '#deployments'
webhook_url: ${{ secrets.SLACK_WEBHOOK }}
fields: repo,message,commit,author,action,eventName,ref,workflow
Advanced Pipeline Features
Parallel Testing Strategy
# .github/workflows/parallel-tests.yml
name: Parallel Test Execution
on:
pull_request:
branches: [main]
jobs:
test-matrix:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
test-suite:
- unit
- integration
- performance
- security
dart-version: ['3.0.0', '3.1.0']
database-mode: ['embedded', 'hosted']
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Dart
uses: dart-lang/setup-dart@v1
with:
dart-version: ${{ matrix.dart-version }}
- name: Setup test environment
run: |
case "${{ matrix.test-suite }}" in
"unit")
echo "Running unit tests"
;;
"integration")
docker-compose -f docker-compose.test.yml up -d
;;
"performance")
./scripts/setup-performance-env.sh
;;
"security")
./scripts/setup-security-env.sh
;;
esac
- name: Run tests
run: |
dart test test/${{ matrix.test-suite }}/ \
--define=database-mode=${{ matrix.database-mode }}
- name: Upload test results
uses: actions/upload-artifact@v3
with:
name: test-results-${{ matrix.test-suite }}-${{ matrix.dart-version }}-${{ matrix.database-mode }}
path: test-results/
Infrastructure as Code
Terraform Configuration
Main Infrastructure
# terraform/main.tf
terraform {
required_version = ">= 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.20"
}
}
backend "s3" {
bucket = "vektagraf-terraform-state"
key = "infrastructure/terraform.tfstate"
region = "us-east-1"
}
}
provider "aws" {
region = var.aws_region
}
provider "kubernetes" {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
# VPC Configuration
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
name = "${var.project_name}-vpc"
cidr = var.vpc_cidr
azs = var.availability_zones
private_subnets = var.private_subnets
public_subnets = var.public_subnets
enable_nat_gateway = true
enable_vpn_gateway = false
enable_dns_hostnames = true
enable_dns_support = true
tags = var.common_tags
}
# EKS Cluster
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "${var.project_name}-cluster"
cluster_version = var.kubernetes_version
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
cluster_endpoint_config = {
private_access = true
public_access = true
public_access_cidrs = var.allowed_cidrs
}
# Node groups
eks_managed_node_groups = {
main = {
min_size = var.node_group_min_size
max_size = var.node_group_max_size
desired_size = var.node_group_desired_size
instance_types = var.node_instance_types
capacity_type = "ON_DEMAND"
k8s_labels = {
Environment = var.environment
Application = "vektagraf"
}
update_config = {
max_unavailable_percentage = 25
}
}
spot = {
min_size = 0
max_size = 10
desired_size = 2
instance_types = var.spot_instance_types
capacity_type = "SPOT"
k8s_labels = {
Environment = var.environment
Application = "vektagraf"
NodeType = "spot"
}
taints = {
spot = {
key = "spot-instance"
value = "true"
effect = "NO_SCHEDULE"
}
}
}
}
tags = var.common_tags
}
# RDS for metadata storage
resource "aws_db_instance" "metadata" {
identifier = "${var.project_name}-metadata"
engine = "postgres"
engine_version = "15.3"
instance_class = var.rds_instance_class
allocated_storage = var.rds_allocated_storage
max_allocated_storage = var.rds_max_allocated_storage
storage_type = "gp3"
storage_encrypted = true
db_name = "vektagraf_metadata"
username = var.rds_username
password = var.rds_password
vpc_security_group_ids = [aws_security_group.rds.id]
db_subnet_group_name = aws_db_subnet_group.main.name
backup_retention_period = var.rds_backup_retention
backup_window = "03:00-04:00"
maintenance_window = "sun:04:00-sun:05:00"
skip_final_snapshot = var.environment != "production"
deletion_protection = var.environment == "production"
tags = var.common_tags
}
# ElastiCache for caching
resource "aws_elasticache_replication_group" "cache" {
replication_group_id = "${var.project_name}-cache"
description = "Vektagraf Redis cache"
node_type = var.redis_node_type
port = 6379
parameter_group_name = "default.redis7"
num_cache_clusters = var.redis_num_nodes
automatic_failover_enabled = var.redis_num_nodes > 1
multi_az_enabled = var.redis_num_nodes > 1
subnet_group_name = aws_elasticache_subnet_group.main.name
security_group_ids = [aws_security_group.redis.id]
at_rest_encryption_enabled = true
transit_encryption_enabled = true
tags = var.common_tags
}
Variables Configuration
# terraform/variables.tf
variable "project_name" {
description = "Name of the project"
type = string
default = "vektagraf"
}
variable "environment" {
description = "Environment name"
type = string
validation {
condition = contains(["development", "staging", "production"], var.environment)
error_message = "Environment must be development, staging, or production."
}
}
variable "aws_region" {
description = "AWS region"
type = string
default = "us-east-1"
}
variable "vpc_cidr" {
description = "CIDR block for VPC"
type = string
default = "10.0.0.0/16"
}
variable "availability_zones" {
description = "Availability zones"
type = list(string)
default = ["us-east-1a", "us-east-1b", "us-east-1c"]
}
variable "private_subnets" {
description = "Private subnet CIDR blocks"
type = list(string)
default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}
variable "public_subnets" {
description = "Public subnet CIDR blocks"
type = list(string)
default = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
}
variable "kubernetes_version" {
description = "Kubernetes version"
type = string
default = "1.27"
}
variable "node_group_min_size" {
description = "Minimum number of nodes in the node group"
type = number
default = 1
}
variable "node_group_max_size" {
description = "Maximum number of nodes in the node group"
type = number
default = 10
}
variable "node_group_desired_size" {
description = "Desired number of nodes in the node group"
type = number
default = 3
}
variable "node_instance_types" {
description = "Instance types for the node group"
type = list(string)
default = ["t3.medium", "t3.large"]
}
variable "common_tags" {
description = "Common tags to apply to all resources"
type = map(string)
default = {
Project = "vektagraf"
ManagedBy = "terraform"
}
}
Environment-Specific Configurations
# terraform/environments/production.tfvars
environment = "production"
aws_region = "us-east-1"
# EKS Configuration
kubernetes_version = "1.27"
node_group_min_size = 3
node_group_max_size = 20
node_group_desired_size = 6
node_instance_types = ["m5.large", "m5.xlarge"]
# RDS Configuration
rds_instance_class = "db.r6g.large"
rds_allocated_storage = 100
rds_max_allocated_storage = 1000
rds_backup_retention = 30
# Redis Configuration
redis_node_type = "cache.r6g.large"
redis_num_nodes = 3
# Security
allowed_cidrs = ["10.0.0.0/8"]
common_tags = {
Project = "vektagraf"
Environment = "production"
ManagedBy = "terraform"
CostCenter = "engineering"
}
Helm Charts
Vektagraf Application Chart
# helm/vektagraf/Chart.yaml
apiVersion: v2
name: vektagraf
description: A Helm chart for Vektagraf application
type: application
version: 0.1.0
appVersion: "1.0.0"
dependencies:
- name: postgresql
version: "12.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
- name: redis
version: "17.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: redis.enabled
# helm/vektagraf/values.yaml
# Default values for vektagraf
replicaCount: 3
image:
repository: ghcr.io/yourorg/vektagraf
pullPolicy: IfNotPresent
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
podSecurityContext:
fsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
service:
type: ClusterIP
port: 80
targetPort: 8080
ingress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rate-limit: "100"
hosts:
- host: api.yourdomain.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: vektagraf-tls
hosts:
- api.yourdomain.com
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 500m
memory: 1Gi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 20
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
nodeSelector: {}
tolerations: []
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- vektagraf
topologyKey: kubernetes.io/hostname
# Vektagraf specific configuration
vektagraf:
config:
database:
mode: hosted
maxConnections: 100
connectionTimeout: "30s"
vector:
algorithm: hnsw
dimensions: 768
efConstruction: 200
security:
encryptionEnabled: true
auditLogging: true
rateLimiting:
enabled: true
requestsPerMinute: 1000
# External dependencies
postgresql:
enabled: true
auth:
postgresPassword: "changeme"
database: "vektagraf"
primary:
persistence:
enabled: true
size: 100Gi
redis:
enabled: true
auth:
enabled: true
password: "changeme"
master:
persistence:
enabled: true
size: 10Gi
# helm/vektagraf/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "vektagraf.fullname" . }}
labels:
{{- include "vektagraf.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "vektagraf.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "vektagraf.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "vektagraf.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: grpc
containerPort: 9090
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 30
env:
- name: VEKTAGRAF_CONFIG_PATH
value: "/config/app.json"
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "vektagraf.fullname" . }}-secrets
key: database-password
- name: ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: {{ include "vektagraf.fullname" . }}-secrets
key: encryption-key
volumeMounts:
- name: config
mountPath: /config
- name: data
mountPath: /data
- name: tmp
mountPath: /tmp
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumes:
- name: config
configMap:
name: {{ include "vektagraf.fullname" . }}-config
- name: data
persistentVolumeClaim:
claimName: {{ include "vektagraf.fullname" . }}-data
- name: tmp
emptyDir: {}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
Configuration Management
GitOps with ArgoCD
Application Configuration
# argocd/applications/vektagraf-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: vektagraf-production
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/yourorg/vektagraf-config
targetRevision: main
path: environments/production
helm:
valueFiles:
- values.yaml
- secrets.yaml
destination:
server: https://kubernetes.default.svc
namespace: vektagraf-production
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
revisionHistoryLimit: 10
Multi-Environment Configuration
# argocd/app-of-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: vektagraf-environments
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/yourorg/vektagraf-config
targetRevision: main
path: argocd/applications
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
Secrets Management with External Secrets Operator
# external-secrets/secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: vektagraf-production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vektagraf-secrets
namespace: vektagraf-production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: vektagraf-secrets
creationPolicy: Owner
data:
- secretKey: database-password
remoteRef:
key: vektagraf/production/database
property: password
- secretKey: encryption-key
remoteRef:
key: vektagraf/production/encryption
property: key
- secretKey: jwt-secret
remoteRef:
key: vektagraf/production/jwt
property: secret
Automated Deployment and Rollback
Blue-Green Deployment Script
#!/bin/bash
# scripts/blue-green-deploy.sh
set -euo pipefail
NAMESPACE="${NAMESPACE:-vektagraf-production}"
NEW_VERSION="$1"
TIMEOUT="${TIMEOUT:-600}"
if [ -z "$NEW_VERSION" ]; then
echo "Usage: $0 <version>"
exit 1
fi
echo "Starting blue-green deployment for version: $NEW_VERSION"
# Get current active color
CURRENT_COLOR=$(kubectl get service vektagraf-active -n "$NAMESPACE" \
-o jsonpath='{.spec.selector.version}' 2>/dev/null || echo "blue")
if [ "$CURRENT_COLOR" = "blue" ]; then
NEW_COLOR="green"
else
NEW_COLOR="blue"
fi
echo "Current active: $CURRENT_COLOR, deploying to: $NEW_COLOR"
# Update deployment with new version
kubectl patch deployment "vektagraf-$NEW_COLOR" -n "$NAMESPACE" \
-p '{"spec":{"template":{"spec":{"containers":[{"name":"vektagraf","image":"ghcr.io/yourorg/vektagraf:'$NEW_VERSION'"}]}}}}'
# Wait for rollout to complete
echo "Waiting for rollout to complete..."
kubectl rollout status deployment/"vektagraf-$NEW_COLOR" -n "$NAMESPACE" --timeout="${TIMEOUT}s"
# Run health checks
echo "Running health checks..."
for i in {1..30}; do
if kubectl run health-check-$NEW_COLOR-$i \
--image=curlimages/curl \
--rm -i --restart=Never \
--timeout=10s \
--command -- curl -f "http://vektagraf-$NEW_COLOR/health" >/dev/null 2>&1; then
echo "Health check passed"
break
fi
if [ $i -eq 30 ]; then
echo "Health checks failed"
exit 1
fi
echo "Health check $i failed, retrying..."
sleep 10
done
# Run smoke tests
echo "Running smoke tests..."
if ! ./scripts/smoke-tests.sh "$NEW_COLOR" "$NAMESPACE"; then
echo "Smoke tests failed"
exit 1
fi
# Switch traffic
echo "Switching traffic to $NEW_COLOR..."
kubectl patch service vektagraf-active -n "$NAMESPACE" \
-p '{"spec":{"selector":{"version":"'$NEW_COLOR'"}}}'
# Verify traffic switch
sleep 10
echo "Verifying traffic switch..."
for i in {1..10}; do
if kubectl run verify-traffic-$i \
--image=curlimages/curl \
--rm -i --restart=Never \
--timeout=10s \
--command -- curl -f "http://vektagraf-active/health" >/dev/null 2>&1; then
echo "Traffic switch verified"
break
fi
if [ $i -eq 10 ]; then
echo "Traffic switch verification failed, rolling back..."
kubectl patch service vektagraf-active -n "$NAMESPACE" \
-p '{"spec":{"selector":{"version":"'$CURRENT_COLOR'"}}}'
exit 1
fi
sleep 5
done
echo "Blue-green deployment completed successfully"
echo "Active color is now: $NEW_COLOR"
# Optional: Scale down old version after verification period
read -p "Scale down $CURRENT_COLOR deployment? (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
kubectl scale deployment "vektagraf-$CURRENT_COLOR" --replicas=0 -n "$NAMESPACE"
echo "Scaled down $CURRENT_COLOR deployment"
fi
Automated Rollback
#!/bin/bash
# scripts/rollback.sh
set -euo pipefail
NAMESPACE="${NAMESPACE:-vektagraf-production}"
REASON="${1:-Manual rollback}"
echo "Initiating rollback: $REASON"
# Get current and previous colors
CURRENT_COLOR=$(kubectl get service vektagraf-active -n "$NAMESPACE" \
-o jsonpath='{.spec.selector.version}')
if [ "$CURRENT_COLOR" = "blue" ]; then
PREVIOUS_COLOR="green"
else
PREVIOUS_COLOR="blue"
fi
echo "Rolling back from $CURRENT_COLOR to $PREVIOUS_COLOR"
# Check if previous deployment is available
if ! kubectl get deployment "vektagraf-$PREVIOUS_COLOR" -n "$NAMESPACE" >/dev/null 2>&1; then
echo "Previous deployment not found"
exit 1
fi
# Scale up previous deployment if needed
PREVIOUS_REPLICAS=$(kubectl get deployment "vektagraf-$PREVIOUS_COLOR" -n "$NAMESPACE" \
-o jsonpath='{.spec.replicas}')
if [ "$PREVIOUS_REPLICAS" -eq 0 ]; then
echo "Scaling up previous deployment..."
kubectl scale deployment "vektagraf-$PREVIOUS_COLOR" --replicas=3 -n "$NAMESPACE"
kubectl rollout status deployment/"vektagraf-$PREVIOUS_COLOR" -n "$NAMESPACE"
fi
# Run health checks on previous deployment
echo "Running health checks on previous deployment..."
for i in {1..10}; do
if kubectl run rollback-health-check-$i \
--image=curlimages/curl \
--rm -i --restart=Never \
--timeout=10s \
--command -- curl -f "http://vektagraf-$PREVIOUS_COLOR/health" >/dev/null 2>&1; then
echo "Health check passed"
break
fi
if [ $i -eq 10 ]; then
echo "Health checks failed on previous deployment"
exit 1
fi
sleep 5
done
# Switch traffic back
echo "Switching traffic back to $PREVIOUS_COLOR..."
kubectl patch service vektagraf-active -n "$NAMESPACE" \
-p '{"spec":{"selector":{"version":"'$PREVIOUS_COLOR'"}}}'
# Scale down current deployment
echo "Scaling down current deployment..."
kubectl scale deployment "vektagraf-$CURRENT_COLOR" --replicas=0 -n "$NAMESPACE"
echo "Rollback completed successfully"
# Send notification
curl -X POST "$SLACK_WEBHOOK_URL" \
-H 'Content-type: application/json' \
--data "{\"text\":\"🔄 Rollback completed: $REASON\"}"
Security Scanning and Compliance
Container Security Scanning
# .github/workflows/security-scan.yml
name: Security Scan
on:
push:
branches: [main]
schedule:
- cron: '0 2 * * *' # Daily at 2 AM
jobs:
container-scan:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build image
run: docker build -t vektagraf:scan .
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: 'vektagraf:scan'
format: 'json'
output: 'trivy-results.json'
- name: Run Grype vulnerability scanner
uses: anchore/scan-action@v3
with:
image: 'vektagraf:scan'
format: 'json'
output-file: 'grype-results.json'
- name: Run Snyk container scan
uses: snyk/actions/docker@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: 'vektagraf:scan'
args: --severity-threshold=high --json-file-output=snyk-results.json
- name: Process scan results
run: |
python scripts/process-security-results.py \
--trivy trivy-results.json \
--grype grype-results.json \
--snyk snyk-results.json \
--output security-report.json
- name: Upload security report
uses: actions/upload-artifact@v3
with:
name: security-report
path: security-report.json
- name: Check security thresholds
run: |
python scripts/check-security-thresholds.py security-report.json
Compliance Validation
// lib/compliance/compliance_validator.dart
class ComplianceValidator {
final List<ComplianceRule> rules;
ComplianceValidator({required this.rules});
Future<ComplianceReport> validate(VektagrafConfig config) async {
final results = <ComplianceResult>[];
for (final rule in rules) {
try {
final result = await rule.validate(config);
results.add(result);
} catch (e) {
results.add(ComplianceResult(
rule: rule,
status: ComplianceStatus.error,
message: 'Validation error: $e',
));
}
}
return ComplianceReport(
timestamp: DateTime.now(),
results: results,
overallStatus: _calculateOverallStatus(results),
);
}
ComplianceStatus _calculateOverallStatus(List<ComplianceResult> results) {
if (results.any((r) => r.status == ComplianceStatus.failed)) {
return ComplianceStatus.failed;
}
if (results.any((r) => r.status == ComplianceStatus.warning)) {
return ComplianceStatus.warning;
}
return ComplianceStatus.passed;
}
}
abstract class ComplianceRule {
String get name;
String get description;
ComplianceFramework get framework;
Future<ComplianceResult> validate(VektagrafConfig config);
}
class EncryptionAtRestRule implements ComplianceRule {
@override
String get name => 'Encryption at Rest';
@override
String get description => 'Data must be encrypted at rest';
@override
ComplianceFramework get framework => ComplianceFramework.gdpr;
@override
Future<ComplianceResult> validate(VektagrafConfig config) async {
if (!config.security.encryptionEnabled) {
return ComplianceResult(
rule: this,
status: ComplianceStatus.failed,
message: 'Encryption at rest is not enabled',
);
}
return ComplianceResult(
rule: this,
status: ComplianceStatus.passed,
message: 'Encryption at rest is properly configured',
);
}
}
class AuditLoggingRule implements ComplianceRule {
@override
String get name => 'Audit Logging';
@override
String get description => 'All data access must be logged';
@override
ComplianceFramework get framework => ComplianceFramework.sox;
@override
Future<ComplianceResult> validate(VektagrafConfig config) async {
if (!config.security.auditLogging) {
return ComplianceResult(
rule: this,
status: ComplianceStatus.failed,
message: 'Audit logging is not enabled',
);
}
return ComplianceResult(
rule: this,
status: ComplianceStatus.passed,
message: 'Audit logging is properly configured',
);
}
}
enum ComplianceFramework { gdpr, hipaa, sox, pci }
enum ComplianceStatus { passed, warning, failed, error }
class ComplianceResult {
final ComplianceRule rule;
final ComplianceStatus status;
final String message;
const ComplianceResult({
required this.rule,
required this.status,
required this.message,
});
}
class ComplianceReport {
final DateTime timestamp;
final List<ComplianceResult> results;
final ComplianceStatus overallStatus;
const ComplianceReport({
required this.timestamp,
required this.results,
required this.overallStatus,
});
Map<String, dynamic> toJson() => {
'timestamp': timestamp.toIso8601String(),
'overallStatus': overallStatus.name,
'results': results.map((r) => {
'rule': r.rule.name,
'framework': r.rule.framework.name,
'status': r.status.name,
'message': r.message,
}).toList(),
};
}
Automated Compliance Testing
# .github/workflows/compliance.yml
name: Compliance Validation
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
compliance-check:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Dart
uses: dart-lang/setup-dart@v1
- name: Install dependencies
run: dart pub get
- name: Run compliance validation
run: dart run bin/compliance_validator.dart
- name: Generate compliance report
run: |
dart run bin/generate_compliance_report.dart \
--output compliance-report.json \
--format json
- name: Upload compliance report
uses: actions/upload-artifact@v3
with:
name: compliance-report
path: compliance-report.json
- name: Check compliance status
run: |
if ! dart run bin/check_compliance_status.dart compliance-report.json; then
echo "Compliance validation failed"
exit 1
fi
Best Practices
CI/CD Best Practices
-
Pipeline Security
- Use least privilege access
- Secure secrets management
- Regular security scanning
- Audit pipeline activities
-
Testing Strategy
- Comprehensive test coverage
- Parallel test execution
- Environment-specific testing
- Performance and security testing
-
Deployment Safety
- Blue-green deployments
- Automated rollback procedures
- Health checks and monitoring
- Gradual traffic shifting
Infrastructure Best Practices
-
Infrastructure as Code
- Version control all infrastructure
- Use modules and reusable components
- Implement proper state management
- Regular infrastructure testing
-
Configuration Management
- GitOps workflows
- Environment-specific configurations
- Secure secrets handling
- Configuration validation
-
Monitoring and Observability
- Comprehensive monitoring setup
- Automated alerting
- Performance tracking
- Audit logging
Advanced Topics
Multi-Cloud Deployment
# terraform/multi-cloud/main.tf
# AWS Provider
provider "aws" {
alias = "primary"
region = "us-east-1"
}
# GCP Provider
provider "google" {
alias = "secondary"
project = var.gcp_project_id
region = "us-central1"
}
# Azure Provider
provider "azurerm" {
alias = "tertiary"
features {}
}
# Multi-cloud load balancer
resource "cloudflare_load_balancer" "global" {
zone_id = var.cloudflare_zone_id
name = "vektagraf-global"
fallback_pool_id = cloudflare_load_balancer_pool.aws.id
default_pool_ids = [
cloudflare_load_balancer_pool.aws.id,
cloudflare_load_balancer_pool.gcp.id,
]
region_pools = {
"WNAM" = [cloudflare_load_balancer_pool.aws.id]
"ENAM" = [cloudflare_load_balancer_pool.gcp.id]
"EU" = [cloudflare_load_balancer_pool.azure.id]
}
}
Progressive Delivery with Flagger
# flagger/canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: vektagraf-canary
namespace: vektagraf-production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: vektagraf-app
progressDeadlineSeconds: 60
service:
port: 80
targetPort: 8080
analysis:
interval: 1m
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
- name: request-duration
thresholdRange:
max: 500
interval: 1m
- name: vektagraf-error-rate
templateRef:
name: vektagraf-error-rate
thresholdRange:
max: 1
interval: 1m
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://vektagraf-canary/api/test | grep OK"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://vektagraf-canary/"
Summary
This chapter covered comprehensive DevOps and CI/CD integration for Vektagraf applications, including:
- CI/CD Pipelines: Complete GitHub Actions workflows with testing and deployment
- Infrastructure as Code: Terraform configurations for scalable infrastructure
- Configuration Management: GitOps with ArgoCD and Helm charts
- Automated Deployment: Blue-green and canary deployment strategies
- Security Integration: Container scanning and compliance validation
- Best Practices: Security, testing, and operational excellence
Key Takeaways
- Implement comprehensive CI/CD pipelines with proper testing
- Use infrastructure as code for reproducible deployments
- Adopt GitOps for configuration management
- Implement progressive deployment strategies
- Integrate security scanning throughout the pipeline
- Maintain compliance through automated validation
Next Steps
- Chapter 19: Learn about troubleshooting and maintenance procedures