K8s Kafka Development 02/21/23

For quite some time now, I have been unable to get kcat to connect to a Kafka Pod running in my K8s Namespace.

I kept running into the following error:

$ kubectl -n prod port-forward svc/kafka 9092
$ kcat -C -b localhost -t foobar -o -1 -c 1
%3|1670010894.150|FAIL|rdkafka#consumer-1| [thrd:kafka-0.kafka-headless.prod.svc.cluster.local:9092/0]: kafka-0.kafka-headless.prod.svc.cluster.local:9092/0: Failed to resolve 'kafka-0.kafka-headless.prod.svc.cluster.local:9092': nodename nor servname provided, or not known (after 5003ms in state CONNECT)
%4|1670010894.150|OFFSET|rdkafka#consumer-1| [thrd:main]: foobar [0]: offset reset (at offset BEGINNING, broker 0) to END: failed to query logical offset: Local: Host resolution failure
%4|1670010894.150|OFFSET|rdkafka#consumer-1| [thrd:main]: foobar [1]: offset reset (at offset BEGINNING, broker 0) to END: failed to query logical offset: Local: Host resolution failure
% ERROR: Topic foobar [0] error: Failed to query logical offset END: Local: Host resolution failure

After some research, I stumbled upon this post which shows how to use iptables to DNAT the broker's internal cluster IP to localhost to force it through a tunnel.

Although the writer of said post experienced an error different than mine, the post got me thinking on a solution that looked promising:

Since K8s creates DNS records for Services and Pods, what if all I need to do is to translate the long auto-generated K8s hostname to my localhost to force the resolution of the kafka-0.kafka-headless.prod.svc.cluster.local hostname mentioned in the error to localhost? Will that suffice?

My research finally led me to this brilliant introduction to using Dnsmasq for development on OSX.

Equipped with this new knowledge, I figured that in order to solve the error that I got, all I need is to add a new resolver as mentioned in passingcuriosity's post, which uses localhost as the nameserver for resolving any request made to *.svc.cluster.local. Running on localhost will be a dnsmasq daemon which will resolve the DNS query to 127.0.0.1.

I suggest you give the links I attached a read if you want to go down one nice rabbit hole.

Once you run the following commands, you should notice that kcat now reads the topic properly:

# Install dnsmasq
brew install dnsmasq

# Add address translation of routes with svc.cluster.local suffix to localhost
echo "\naddress=/svc.cluster.local/127.0.0.1" >> /usr/local/etc/dnsmasq.conf

# Setup resolver directory
sudo mkdir -p /etc/resolver

# Add a new resolver for svc.cluster.local with the dnsmasq running locally as the nameserver
sudo tee /etc/resolver/svc.cluster.local >/dev/null <<EOF
nameserver 127.0.0.1
EOF

# Restart dnsmasq
sudo brew services stop dnsmasq
sudo brew services start dnsmasq