DEVELOPER BLOG

Here are some of the most notable improvements since our last update.

Better details for HTTP messages

Log details for HTTP messages now show the response status, headers and parsed body (if JSON).

Improved Status Surfacing

When something goes wrong in the backend, the source of the issue can be many different things. This week we improved how this status is surfaced to the user, so that, instead of just saying "Program Aborted", the explorer will show more details about what's going on.

New Internal Logging System

Server logs are extremely important for us. Membrane runs on Kubernetes and our provider has a max retention of 10MB per pod which can fill up pretty quickly.

Storage is extremely cheap (i.e. S3) so there's no reason why log retention is expensive (except maybe for indexing). So we set up a deployment of Fluentbit daemon (from the creators of Fluentd) that continously compresses and ships log chunks to S3.

Cool, now we have a bunch of .gz files in S3. Unlimited retention! But how do we read them?

That's were mlogs comes in. We built a simple CLI tool that retrieves chunks from S3 based on date, caches them locally (because logs are usually read over and over) and stitches them together along with regular kubernetes logs output (including --follow). Infinite logs ftw!.

This approach doesn't have all the bells and whistles of a hosted logging system (particularly indexing of the logs for quick search, stats and alerts) but it's extremely cheap and gives us full control of retention.