Setting Up kafka on multiple servers

0 votes

I'm following this guide to install kafka and zookeeper. Everything works fine, but I'm setting up kafka and zookeeper on 2 servers.

The kafka/config/server.properties are:

  • Server 1: broker.id = 0
  • Server 1: zookeeper.connect = localhost:2181,99.99.99.91:2181
  • Server 2: broker.id = 1
  • Server 2: zookeeper.connect = localhost:2181,99.99.99.92:2181

I'm confused about the following:

  • When I publish a topic, does it go to both Instances, or just the server it's loaded on?
  • In order to use multiple servers like this, would I be required to use something like HAProxy with say 3 servers?
  • Is there anything important I am missing with using 3 servers?
Aug 3, 2018 in DevOps & Agile by DragonLord999
• 8,380 points
282 views

1 answer to this question.

0 votes

I'm assuming that instances you mentioned are kafka servers.

Now, to answer your questions:

#1: New created topics info is stored on zookeeper and some of the key information will be loaded to all brokers metadata cache buffer.

#2: You don't have to configure any kind of proxy servers for kafka cluster. It can manage itself to implement the fail-over and load-balancing.

#3: Just assigning an unused ID and data folder is enough like the way you did for those two brokers.

Also remember, because ZAB requires majority, it is advised to use an odd number of machines as the zookeeper quorum.

answered Aug 3, 2018 by DareDev
• 6,810 points

Related Questions In DevOps & Agile

+6 votes
5 answers

can't remove SSIS variables using remove method on PowerShell

you're missing the additional $environment.Alter() after your ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by ajs3033
• 7,280 points
96 views
0 votes
1 answer

How to set up a VM with KVM/qemu without VitualBox using Vagrant

Start the vagrant box using: vagrant up --provider=kvm But ...READ MORE

answered Apr 11, 2018 in DevOps & Agile by ajs3033
• 7,280 points
446 views
0 votes
1 answer

Can I run a program on release failure in TFS?

You can try and do the following ...READ MORE

answered Apr 26, 2018 in DevOps & Agile by ajs3033
• 7,280 points
46 views
+13 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by DragonLord999
• 8,380 points
206 views
0 votes
1 answer
0 votes
1 answer

Web Service with WebSocket Implementation on AWS Blue-green deployment

Try connection draining. It will send all ...READ MORE

answered Apr 2, 2018 in DevOps & Agile by DareDev
• 6,810 points
135 views
0 votes
1 answer

What OS is Jenkins installed on Azure Virtual Machine?

Under Managed Jenkins there's a section called ...READ MORE

answered Apr 4, 2018 in DevOps & Agile by DareDev
• 6,810 points
45 views