Setting Up kafka on multiple servers

0 votes

I'm following this guide to install kafka and zookeeper. Everything works fine, but I'm setting up kafka and zookeeper on 2 servers.

The kafka/config/server.properties are:

  • Server 1: broker.id = 0
  • Server 1: zookeeper.connect = localhost:2181,99.99.99.91:2181
  • Server 2: broker.id = 1
  • Server 2: zookeeper.connect = localhost:2181,99.99.99.92:2181

I'm confused about the following:

  • When I publish a topic, does it go to both Instances, or just the server it's loaded on?
  • In order to use multiple servers like this, would I be required to use something like HAProxy with say 3 servers?
  • Is there anything important I am missing with using 3 servers?
Aug 3, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
2,953 views

1 answer to this question.

0 votes

I'm assuming that instances you mentioned are kafka servers.

Now, to answer your questions:

#1: New created topics info is stored on zookeeper and some of the key information will be loaded to all brokers metadata cache buffer.

#2: You don't have to configure any kind of proxy servers for kafka cluster. It can manage itself to implement the fail-over and load-balancing.

#3: Just assigning an unused ID and data folder is enough like the way you did for those two brokers.

Also remember, because ZAB requires majority, it is advised to use an odd number of machines as the zookeeper quorum.

Hope this will work

Join our Kafka course online today to become certified expert.

Thank You

answered Aug 3, 2018 by DareDev
• 6,890 points

Related Questions In DevOps & Agile

+6 votes
5 answers

can't remove SSIS variables using remove method on PowerShell

you're missing the additional $environment.Alter() after your ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by ajs3033
• 7,300 points
2,536 views
0 votes
1 answer

How to set up a VM with KVM/qemu without VitualBox using Vagrant

Start the vagrant box using: vagrant up --provider=kvm But ...READ MORE

answered Apr 11, 2018 in DevOps & Agile by ajs3033
• 7,300 points
2,291 views
0 votes
1 answer

Can I run a program on release failure in TFS?

You can try and do the following ...READ MORE

answered Apr 26, 2018 in DevOps & Agile by ajs3033
• 7,300 points
699 views
+15 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
3,935 views
+2 votes
1 answer
0 votes
1 answer

Web Service with WebSocket Implementation on AWS Blue-green deployment

Try connection draining. It will send all ...READ MORE

answered Apr 2, 2018 in DevOps & Agile by DareDev
• 6,890 points
991 views
0 votes
1 answer

What OS is Jenkins installed on Azure Virtual Machine?

Under Managed Jenkins there's a section called ...READ MORE

answered Apr 4, 2018 in DevOps & Agile by DareDev
• 6,890 points
751 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP