So Kafka is a message broker basically. I haven’t utilized it personally but here is the detailed documentation including the APIs that are available to utilize: https://kafka.apache.org/documentation/
The biggest question when looking at a message brokering system is how it fits into an application. Usually a system like this is specifically designed to handle the messages that then are fed into an application. The specific calls to the messages are programed into some sort of user interface. Unless you work for Apache on the Kafka product I think your main focus should be on how your project/projects are specifically targeting utilizing Kafka. If your trying to “test Kafka” then I’d assume you are working for apache as it’s a product.
I would consider targeting the following questions and getting those answers:
- What is the implementation strategy of my org for utilizing Kafka?
- Within that implementation strategy what are the interfaces that will be developed for end users to access message content managed via Kafka?
- Once you know those two answers you can then ask what technology will best automate testing of these developed interfaces which will render the managed message content?
- After you have that then you should know where the originating source data is coming from in which case then it would be how best to check the source with the final product?
- Then you can get into tracking failures and logs to ensure you find where gaps may exist…
Unless your unit testing, you need to start from the final product and go backwards. Integration testing start at the APIs specifically, not the product under the API. So your final picture will look more like a GUI/RSS feed/something else that gives me x from interacting with y functionality. Integration of y with Kafka and Kafka with the original source(s). Check the original source(s) to see if x = original source(s) x.
It’s all data being sent and managed, it’s just how it’s being constructed is the key. Think of it this way, if you had 15 different databases and pulled info from all into a mix you would want to ensure the constructed mix of data was the same as the original parts it was constructed from. Same principal here with a message broker like Kafka.