Skip to main content

Cloud Storage with Amazon S3

Cloud Storage with Amazon S3

Modern applications often need to handle file uploads, document storage, and media management at scale. While local file storage works for development and small applications, production systems require the reliability, scalability, and global distribution that cloud storage provides. This chapter explores how to integrate Amazon S3 with your Hypermodern applications for robust file management.

Understanding Cloud Storage Integration

The Hypermodern platform includes comprehensive S3 integration that allows you to seamlessly switch between local and cloud storage without changing your application logic. This flexibility means you can develop locally and deploy to production with confidence, knowing your file operations will work consistently across environments.

Why Choose S3?

  • Scalability: Handle files from bytes to terabytes without infrastructure concerns
  • Durability: 99.999999999% (11 9's) durability for your critical files
  • Global Distribution: Serve files from edge locations worldwide
  • Security: Fine-grained access controls and encryption options
  • Cost Effectiveness: Pay only for what you use with multiple storage classes

Setting Up S3 Integration

AWS Account Configuration

Before integrating S3, you'll need an AWS account and proper credentials. The Hypermodern S3 integration supports multiple authentication methods:

# Environment variables (recommended for development)
export AWS_ACCESS_KEY_ID=your_access_key_here
export AWS_SECRET_ACCESS_KEY=your_secret_key_here
export AWS_DEFAULT_REGION=us-east-1
export S3_BUCKET_NAME=your-app-bucket

Using the Credentials Service

For production applications, use Hypermodern's credentials service for secure credential management:

// Configure credentials through the service
await CredentialsService.set('aws_access_key_id', 'your_access_key');
await CredentialsService.set('aws_secret_access_key', 'your_secret_key');
await CredentialsService.set('aws_region', 'us-east-1');
await CredentialsService.set('s3_bucket_name', 'your-app-bucket');

IAM Permissions

Create an IAM user with the minimum required permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-app-bucket",
                "arn:aws:s3:::your-app-bucket/*"
            ]
        }
    ]
}

Basic S3 Operations

Direct S3 Client Usage

The S3Client provides low-level access to S3 operations:

import 'package:hypermodern_server/hypermodern_server.dart';

class FileController {
  static final s3 = S3Client();

  static Future<dynamic> uploadFile(RequestData data) async {
    try {
      // Get file data from request
      final fileData = data.files['upload'];
      if (fileData == null) {
        return {'error': 'No file provided'};
      }

      // Upload to S3
      final result = await s3.uploadFile(
        'uploads/${DateTime.now().millisecondsSinceEpoch}_${fileData.filename}',
        fileData.bytes,
        contentType: fileData.contentType,
        metadata: {
          'original_name': fileData.filename,
          'uploaded_by': data.user?.id ?? 'anonymous',
          'upload_time': DateTime.now().toIso8601String(),
        },
      );

      return {
        'success': true,
        'key': result.key,
        'url': result.url,
        'size': fileData.bytes.length,
      };
    } catch (e) {
      return {'error': 'Upload failed: $e'};
    }
  }

  static Future<dynamic> downloadFile(RequestData data) async {
    final key = data.pathParams['key'];
    
    try {
      final fileData = await s3.downloadFile(key);
      
      // Return file with appropriate headers
      return FileResponse(
        data: fileData,
        contentType: 'application/octet-stream',
        filename: key.split('/').last,
      );
    } catch (e) {
      return {'error': 'File not found'};
    }
  }

  static Future<dynamic> deleteFile(RequestData data) async {
    final key = data.pathParams['key'];
    
    try {
      await s3.deleteFile(key);
      return {'success': true, 'message': 'File deleted'};
    } catch (e) {
      return {'error': 'Delete failed: $e'};
    }
  }
}

File Existence and Metadata

Check if files exist and retrieve metadata without downloading:

class FileInfoController {
  static final s3 = S3Client();

  static Future<dynamic> checkFile(RequestData data) async {
    final key = data.pathParams['key'];
    
    final exists = await s3.fileExists(key);
    if (!exists) {
      return {'exists': false};
    }

    final metadata = await s3.getFileMetadata(key);
    return {
      'exists': true,
      'size': metadata['content-length'],
      'lastModified': metadata['last-modified'],
      'contentType': metadata['content-type'],
      'customMetadata': metadata['metadata'] ?? {},
    };
  }
}

File Storage Service Integration

The FileStorageService provides a higher-level abstraction that works with both local and S3 storage:

Configuration

class StorageConfig {
  static FileStorageConfig get production => FileStorageConfig(
    backend: StorageBackend.s3,
    s3KeyPrefix: 'app-files',
    organizeByDate: true,
    maxFileSize: 50 * 1024 * 1024, // 50MB
    allowedExtensions: ['.jpg', '.jpeg', '.png', '.pdf', '.doc', '.docx'],
  );

  static FileStorageConfig get development => FileStorageConfig(
    backend: StorageBackend.local,
    baseDirectory: 'storage/uploads',
    organizeByDate: true,
    maxFileSize: 10 * 1024 * 1024, // 10MB for dev
    allowedExtensions: ['.jpg', '.jpeg', '.png', '.pdf'],
  );
}

Service Implementation

class DocumentService {
  static final storage = FileStorageService(
    config: Environment.isProduction 
      ? StorageConfig.production 
      : StorageConfig.development
  );

  static Future<StoredFile> uploadDocument(
    UploadedFile file, 
    String userId,
  ) async {
    // Validate file
    if (!_isValidDocument(file)) {
      throw ValidationException('Invalid document type');
    }

    // Store with metadata
    final storedFile = await storage.storeFile(
      file,
      metadata: {
        'user_id': userId,
        'document_type': _getDocumentType(file.filename),
        'upload_timestamp': DateTime.now().toIso8601String(),
      },
    );

    // Log the upload
    await AuditService.logFileUpload(userId, storedFile.id);

    return storedFile;
  }

  static Future<Uint8List> getDocument(String fileId, String userId) async {
    // Check permissions
    final file = await FileMetadataService.getFile(fileId);
    if (file.metadata['user_id'] != userId) {
      throw UnauthorizedException('Access denied');
    }

    return await storage.getFileBytes(fileId);
  }

  static Future<String> generateShareLink(
    String fileId, 
    Duration expiration,
  ) async {
    return await storage.generatePresignedUrl(fileId, expiration: expiration);
  }

  static bool _isValidDocument(UploadedFile file) {
    final allowedTypes = ['application/pdf', 'image/jpeg', 'image/png'];
    return allowedTypes.contains(file.contentType);
  }

  static String _getDocumentType(String filename) {
    final extension = filename.toLowerCase().split('.').last;
    switch (extension) {
      case 'pdf': return 'document';
      case 'jpg':
      case 'jpeg':
      case 'png': return 'image';
      default: return 'unknown';
    }
  }
}

Advanced S3 Features

Presigned URLs for Direct Upload

Allow clients to upload directly to S3 without going through your server:

class DirectUploadController {
  static final s3 = S3Client();

  static Future<dynamic> generateUploadUrl(RequestData data) async {
    final filename = data.body['filename'];
    final contentType = data.body['contentType'];
    final userId = data.user!.id;

    // Generate unique key
    final key = 'direct-uploads/$userId/${DateTime.now().millisecondsSinceEpoch}_$filename';

    // Generate presigned URL for upload
    final uploadUrl = await s3.generatePresignedUploadUrl(
      key,
      contentType: contentType,
      expiration: Duration(minutes: 15),
      metadata: {
        'user_id': userId,
        'original_filename': filename,
      },
    );

    return {
      'uploadUrl': uploadUrl,
      'key': key,
      'expiresIn': 900, // 15 minutes
    };
  }

  static Future<dynamic> confirmUpload(RequestData data) async {
    final key = data.body['key'];
    
    // Verify the file was uploaded
    final exists = await s3.fileExists(key);
    if (!exists) {
      return {'error': 'Upload not found'};
    }

    // Move from temporary to permanent location
    final permanentKey = key.replaceFirst('direct-uploads/', 'files/');
    await s3.copyFile(key, permanentKey);
    await s3.deleteFile(key);

    // Store metadata in database
    await FileMetadataService.createRecord(permanentKey, data.user!.id);

    return {
      'success': true,
      'fileId': permanentKey,
    };
  }
}

Multipart Uploads for Large Files

Handle large file uploads efficiently:

class LargeFileUploadService {
  static final s3 = S3Client();
  static const int chunkSize = 5 * 1024 * 1024; // 5MB chunks

  static Future<String> initiateMultipartUpload(
    String key,
    String contentType,
  ) async {
    return await s3.initiateMultipartUpload(key, contentType);
  }

  static Future<String> uploadPart(
    String uploadId,
    String key,
    int partNumber,
    Uint8List data,
  ) async {
    return await s3.uploadPart(uploadId, key, partNumber, data);
  }

  static Future<void> completeMultipartUpload(
    String uploadId,
    String key,
    List<String> etags,
  ) async {
    await s3.completeMultipartUpload(uploadId, key, etags);
  }

  static Future<void> abortMultipartUpload(
    String uploadId,
    String key,
  ) async {
    await s3.abortMultipartUpload(uploadId, key);
  }
}

File Organization Strategies

Date-Based Organization

Organize files by date for better management:

class FileOrganizer {
  static String generateDateBasedKey(String filename, String prefix) {
    final now = DateTime.now();
    final year = now.year.toString();
    final month = now.month.toString().padLeft(2, '0');
    final day = now.day.toString().padLeft(2, '0');
    
    final fileId = generateUniqueId();
    final extension = filename.split('.').last;
    
    return '$prefix/$year/$month/$day/${fileId}.$extension';
  }

  static String generateUserBasedKey(String userId, String filename) {
    final fileId = generateUniqueId();
    final extension = filename.split('.').last;
    
    return 'users/$userId/files/${fileId}.$extension';
  }

  static String generateCategoryKey(String category, String filename) {
    final now = DateTime.now();
    final timestamp = now.millisecondsSinceEpoch;
    final extension = filename.split('.').last;
    
    return '$category/${timestamp}_$filename';
  }
}

Content-Type Based Processing

Handle different file types appropriately:

class FileProcessor {
  static Future<ProcessedFile> processUpload(UploadedFile file) async {
    switch (file.contentType) {
      case 'image/jpeg':
      case 'image/png':
        return await _processImage(file);
      
      case 'application/pdf':
        return await _processPDF(file);
      
      case 'video/mp4':
        return await _processVideo(file);
      
      default:
        return await _processGenericFile(file);
    }
  }

  static Future<ProcessedFile> _processImage(UploadedFile file) async {
    // Generate thumbnails
    final thumbnail = await ImageProcessor.generateThumbnail(file.bytes);
    
    // Store original and thumbnail
    final originalKey = FileOrganizer.generateCategoryKey('images/original', file.filename);
    final thumbnailKey = FileOrganizer.generateCategoryKey('images/thumbnails', file.filename);
    
    await S3Client().uploadFile(originalKey, file.bytes, contentType: file.contentType);
    await S3Client().uploadFile(thumbnailKey, thumbnail, contentType: 'image/jpeg');
    
    return ProcessedFile(
      originalKey: originalKey,
      thumbnailKey: thumbnailKey,
      metadata: {
        'type': 'image',
        'has_thumbnail': true,
      },
    );
  }

  static Future<ProcessedFile> _processPDF(UploadedFile file) async {
    // Extract text for search indexing
    final text = await PDFProcessor.extractText(file.bytes);
    
    final key = FileOrganizer.generateCategoryKey('documents', file.filename);
    await S3Client().uploadFile(
      key, 
      file.bytes, 
      contentType: file.contentType,
      metadata: {'extracted_text': text},
    );
    
    return ProcessedFile(
      originalKey: key,
      metadata: {
        'type': 'document',
        'searchable': true,
        'text_length': text.length,
      },
    );
  }
}

Security and Access Control

Secure File Access

Implement proper access controls for file operations:

class SecureFileController {
  static Future<dynamic> downloadFile(RequestData data) async {
    final fileId = data.pathParams['fileId'];
    final user = data.user;

    // Check permissions
    final hasAccess = await FilePermissionService.checkAccess(fileId, user?.id);
    if (!hasAccess) {
      return {'error': 'Access denied', 'code': 403};
    }

    // Generate temporary access URL
    final presignedUrl = await S3Client().generatePresignedUrl(
      fileId,
      expiration: Duration(minutes: 5),
    );

    // Log access
    await AuditService.logFileAccess(fileId, user?.id);

    return {'downloadUrl': presignedUrl};
  }

  static Future<dynamic> shareFile(RequestData data) async {
    final fileId = data.pathParams['fileId'];
    final expirationHours = data.body['expirationHours'] ?? 24;
    final user = data.user!;

    // Verify ownership
    final isOwner = await FileOwnershipService.isOwner(fileId, user.id);
    if (!isOwner) {
      return {'error': 'Only file owners can create share links'};
    }

    // Generate share token
    final shareToken = await ShareTokenService.create(
      fileId: fileId,
      createdBy: user.id,
      expiresAt: DateTime.now().add(Duration(hours: expirationHours)),
    );

    return {
      'shareUrl': '/shared/${shareToken.id}',
      'expiresAt': shareToken.expiresAt.toIso8601String(),
    };
  }
}

File Encryption

Encrypt sensitive files before storing in S3:

class EncryptedFileService {
  static final _encryptionKey = CredentialsService.get('file_encryption_key');

  static Future<String> storeEncryptedFile(
    UploadedFile file,
    String userId,
  ) async {
    // Encrypt file data
    final encryptedData = await CryptoService.encrypt(file.bytes, _encryptionKey);
    
    // Store encrypted file
    final key = FileOrganizer.generateUserBasedKey(userId, '${file.filename}.encrypted');
    await S3Client().uploadFile(
      key,
      encryptedData,
      contentType: 'application/octet-stream',
      metadata: {
        'original_content_type': file.contentType,
        'original_filename': file.filename,
        'encrypted': 'true',
      },
    );

    return key;
  }

  static Future<Uint8List> retrieveEncryptedFile(String key) async {
    // Download encrypted file
    final encryptedData = await S3Client().downloadFile(key);
    
    // Decrypt and return
    return await CryptoService.decrypt(encryptedData, _encryptionKey);
  }
}

Performance Optimization

Caching Strategies

Implement intelligent caching for frequently accessed files:

class CachedFileService {
  static final _cache = <String, CachedFile>{};
  static const maxCacheSize = 100 * 1024 * 1024; // 100MB
  static int _currentCacheSize = 0;

  static Future<Uint8List> getFile(String key) async {
    // Check cache first
    if (_cache.containsKey(key)) {
      final cached = _cache[key]!;
      if (cached.isValid) {
        return cached.data;
      } else {
        _removeFromCache(key);
      }
    }

    // Download from S3
    final data = await S3Client().downloadFile(key);
    
    // Cache if small enough
    if (data.length < 10 * 1024 * 1024) { // Cache files under 10MB
      _addToCache(key, data);
    }

    return data;
  }

  static void _addToCache(String key, Uint8List data) {
    // Evict old entries if needed
    while (_currentCacheSize + data.length > maxCacheSize && _cache.isNotEmpty) {
      final oldestKey = _cache.keys.first;
      _removeFromCache(oldestKey);
    }

    _cache[key] = CachedFile(
      data: data,
      cachedAt: DateTime.now(),
      ttl: Duration(minutes: 30),
    );
    _currentCacheSize += data.length;
  }

  static void _removeFromCache(String key) {
    final cached = _cache.remove(key);
    if (cached != null) {
      _currentCacheSize -= cached.data.length;
    }
  }
}

CDN Integration

Serve files through CloudFront for better performance:

class CDNFileService {
  static const cdnDomain = 'https://d1234567890.cloudfront.net';

  static String getCDNUrl(String s3Key) {
    return '$cdnDomain/$s3Key';
  }

  static Future<String> uploadWithCDN(
    UploadedFile file,
    String key,
  ) async {
    // Upload to S3
    await S3Client().uploadFile(
      key,
      file.bytes,
      contentType: file.contentType,
      cacheControl: 'public, max-age=31536000', // 1 year
    );

    // Return CDN URL
    return getCDNUrl(key);
  }

  static Future<void> invalidateCDNCache(List<String> keys) async {
    // Invalidate CloudFront cache
    final paths = keys.map((key) => '/$key').toList();
    await CloudFrontService.createInvalidation(paths);
  }
}

Monitoring and Analytics

File Usage Tracking

Track file operations for analytics and billing:

class FileAnalyticsService {
  static Future<void> trackUpload(String fileId, String userId, int fileSize) async {
    await AnalyticsService.track('file_upload', {
      'file_id': fileId,
      'user_id': userId,
      'file_size': fileSize,
      'timestamp': DateTime.now().toIso8601String(),
    });
  }

  static Future<void> trackDownload(String fileId, String? userId) async {
    await AnalyticsService.track('file_download', {
      'file_id': fileId,
      'user_id': userId,
      'timestamp': DateTime.now().toIso8601String(),
    });
  }

  static Future<Map<String, dynamic>> getUsageStats(String userId) async {
    final uploads = await AnalyticsService.count('file_upload', {'user_id': userId});
    final downloads = await AnalyticsService.count('file_download', {'user_id': userId});
    final totalSize = await FileMetadataService.getTotalSize(userId);

    return {
      'uploads': uploads,
      'downloads': downloads,
      'total_size': totalSize,
      'storage_cost': calculateStorageCost(totalSize),
    };
  }
}

Migration and Backup Strategies

Migrating from Local to S3

Migrate existing local files to S3:

class StorageMigrationService {
  static Future<void> migrateToS3() async {
    final localFiles = await _getLocalFiles();
    final s3 = S3Client();

    for (final localFile in localFiles) {
      try {
        // Read local file
        final data = await File(localFile.path).readAsBytes();
        
        // Generate S3 key
        final s3Key = _generateS3Key(localFile.path);
        
        // Upload to S3
        await s3.uploadFile(s3Key, data, contentType: localFile.contentType);
        
        // Update database references
        await FileMetadataService.updateStorageLocation(
          localFile.id,
          StorageBackend.s3,
          s3Key,
        );
        
        // Delete local file after successful upload
        await File(localFile.path).delete();
        
        print('Migrated: ${localFile.path} -> $s3Key');
      } catch (e) {
        print('Failed to migrate ${localFile.path}: $e');
      }
    }
  }

  static String _generateS3Key(String localPath) {
    // Convert local path to S3 key
    return localPath.replaceFirst('storage/', 'migrated/');
  }
}

Backup and Disaster Recovery

Implement backup strategies for critical files:

class BackupService {
  static Future<void> backupToSecondaryRegion() async {
    final s3Primary = S3Client();
    final s3Backup = S3Client(region: 'us-west-2');

    final files = await FileMetadataService.getCriticalFiles();
    
    for (final file in files) {
      try {
        // Download from primary
        final data = await s3Primary.downloadFile(file.s3Key);
        
        // Upload to backup region
        final backupKey = 'backup/${file.s3Key}';
        await s3Backup.uploadFile(backupKey, data);
        
        // Record backup
        await BackupMetadataService.recordBackup(file.id, backupKey);
      } catch (e) {
        print('Backup failed for ${file.s3Key}: $e');
      }
    }
  }

  static Future<void> restoreFromBackup(String fileId) async {
    final backup = await BackupMetadataService.getBackup(fileId);
    if (backup == null) {
      throw Exception('No backup found for file $fileId');
    }

    final s3Backup = S3Client(region: 'us-west-2');
    final s3Primary = S3Client();

    // Download from backup
    final data = await s3Backup.downloadFile(backup.backupKey);
    
    // Restore to primary
    final originalFile = await FileMetadataService.getFile(fileId);
    await s3Primary.uploadFile(originalFile.s3Key, data);
  }
}

Best Practices and Recommendations

1. File Naming Conventions

Use consistent, predictable file naming:

class FileNamingService {
  static String generateFileName(String originalName, String userId) {
    final timestamp = DateTime.now().millisecondsSinceEpoch;
    final extension = originalName.split('.').last.toLowerCase();
    final sanitizedName = _sanitizeFilename(originalName);
    
    return '${userId}_${timestamp}_$sanitizedName.$extension';
  }

  static String _sanitizeFilename(String filename) {
    return filename
        .replaceAll(RegExp(r'[^a-zA-Z0-9._-]'), '_')
        .replaceAll(RegExp(r'_+'), '_')
        .toLowerCase();
  }
}

2. Error Handling

Implement comprehensive error handling:

class RobustFileService {
  static Future<StoredFile> uploadWithRetry(
    UploadedFile file,
    {int maxRetries = 3}
  ) async {
    for (int attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        return await _performUpload(file);
      } on S3Exception catch (e) {
        if (attempt == maxRetries || !_isRetryableError(e)) {
          rethrow;
        }
        
        // Exponential backoff
        await Future.delayed(Duration(seconds: attempt * 2));
      }
    }
    
    throw Exception('Upload failed after $maxRetries attempts');
  }

  static bool _isRetryableError(S3Exception e) {
    return e.statusCode >= 500 || e.statusCode == 429;
  }
}

3. Cost Optimization

Monitor and optimize S3 costs:

class CostOptimizationService {
  static Future<void> implementLifecyclePolicy() async {
    // Move files to cheaper storage classes over time
    final policy = S3LifecyclePolicy([
      LifecycleRule(
        id: 'optimize-storage-costs',
        status: 'Enabled',
        transitions: [
          Transition(days: 30, storageClass: 'STANDARD_IA'),
          Transition(days: 90, storageClass: 'GLACIER'),
          Transition(days: 365, storageClass: 'DEEP_ARCHIVE'),
        ],
      ),
    ]);

    await S3Client().putBucketLifecycleConfiguration(policy);
  }

  static Future<void> cleanupOldFiles() async {
    final cutoffDate = DateTime.now().subtract(Duration(days: 365));
    final oldFiles = await FileMetadataService.getFilesOlderThan(cutoffDate);

    for (final file in oldFiles) {
      if (!file.isImportant) {
        await S3Client().deleteFile(file.s3Key);
        await FileMetadataService.markAsDeleted(file.id);
      }
    }
  }
}

The S3 integration in Hypermodern provides a production-ready solution for file storage that scales with your application. By leveraging cloud storage, you can build applications that handle files efficiently, securely, and cost-effectively while maintaining the developer experience that makes Hypermodern special.

Back to top