Hi, here you’ll find some general thoughts about the ETL, and some special exercises solved with a pretty attractive option we can have with modern cloud providers - Azure in my case. My example is a stream of values coming from some source, to keep it simple, let’s say we are receiving timestamp and value. The stream is very dense, and we want to compress it in time and provide some other structure of payload (basically, just transform it from one JSON format to another one).

That particular topic could be quite trivial for some of the engineers who deliver java applications as docker containers, but… If you’ve never cared about that particular thing, you’d better do it. Overwise, you will be losing a lot of memory in the container as unused. As usual the pet project for the investigation could be found here. Application for testing First, I will create a simple application exposing controller for triggering the needed load inside the application to boost the heap rise.

This is a second part of the previous writing about the option for streaming data from Amazon DynamoDb to other services. I wasn’t aware of all options of Amazon Kinesis family services when was writing the first part. But during preparation, I noticed that this is a good alternative to the old school AWS Lambda processing, which I described in the first part. Well, what’s so interesting in this alternative is that you don’t need to bother about the connectivity to your Amazon ES from Lambda code, which was quite a tricky place, to be honest.

It’s well known that DynamoDB works well for lookup queries. But if you need to run heavy, analytical queries against the data in the DynamoDB table, you would need to use other tools for indexing data. One of the solutions you can come across in the AWS documentation and AWS blog posts is to use DynamoDB streams to load data to the AWS ElasticSearch service for indexing and providing a reacher search possibilities to users.

I am going to quickly show some cool things, which could be done almost out of the box by means of Spring Actuator module. I selected just 4 cool features, but there are of course more. Before getting started with cool things, I must warn you that big power brings also high responsibility. Don’t forget to check the security part of actuator documentation before delivering a solution to production.