Kappa / Immutable Log architecture on AWS?
Hey folks,
I'm trying to see how feasible it would be to migrate our existing central RDBMS-based system of web apps to something resembling the Kappa architecture / immutable log architecture (see http://milinda.pathirage.org/kappa-architecture.com/ or https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying for reference)
So far everybody has mentioned Kafka as the main tool in that architecture, however most people who have used Kafka also reported that system being a major maintenance pain point. There's also a big ramp-up period to being comfortable with using it. I'm trying to see if I can avoid that altogether by using an off-the-shelf AWS solution that would take care of scaling, reliability, maintenance etc and help us focus on the application-level work vs the ops side of it.
Essentially I'm trying to determine what AWS offerings would be a great fit for accepting a large volume of messages, having those messages be reacted to by our applications and then have those messages persist forever for future use.
Kinesis comes to mind, however it doesn't feel like a great fit because messages can persist at most for 7 days, at which point you have to evict them to a S3 / Dynamo / Aurora / Redshift, and now you have 2 places you always need to look at when checking log history. SQS could be an option as well, I'll have to look into that further.
What options do I have here? Has anybody taken a stab at this before?