A few years ago, Ethan Rosenthal (now a Senior AI Engineer at Square) started building a platform to monitor the performance of ML systems by ingesting data on predictions and model outcomes and egesting model metrics. Although he ultimately decided not to pursue the idea as a startup, he is now open-sourcing some components of this tool, including a serverless event collector. The serverless event collector is designed to receive and store events (pageviews and button clicks) and provide fast access to information such as the total number of events. It includes an API Gateway with 2 lambdas, which authenticate and validate requests before transmitting them to Kinesis Firehose and updating a DynamoDB table with a count of each event type. When batches of events from Kinesis are added to S3, another Lambda function copies events from the primary bucket into user-specific buckets partitioned by event-type and time.