This Lambda-based script optimizes AWS VPC Flow Logs by eliminating redundant cloud metadata fields, reducing log size by 60%. The script responds to S3 Event Notifications (configured either directly or through Amazon SNS), pulls raw VPC Flow Logs from S3, removes repetitive columns, and writes the optimized logs to a destination S3 bucket using compression. It can optionally retain the original flow log headers, allowing compatibility with Splunk's standardized ingestion format. Despite removing fields, the reduced logs preserve full analytical capabilities through optional enrichment pipelines that add cloud context on-demand. This reduction decreases S3 storage costs, lowers Splunk licensing costs, improves query performance, and maintains searchability while using existing VPC Flow Log collection configurations.
