last day (24 days later) » 

9:40 PM
0
A: Kafka cluster authorization failure

mazaneichaCan you check (using $ kafka-acl --list --topic ...) that your consumer user has access to the consumer-group. I think your command might have not worked properly (despite not throwing any erros) because of a slightly different syntax from whats known to work as shown here: $ ../bin/kafka-acls....

 
ill try that. But I also noticed each time I try to consume from a topic, I get a different consumer group such as "consumer-group-####", where "####" are numbers that change each time. Will that fix it? I was following this website on how to use ACLs: kafka.apache.org/documentation/#security. Also, how do I switch "users" to test out the ACLs? I also updated the question in regards to the problem I encountered when starting my Kafka server. Im justing using a simple SSL with no Kerberos.
are groups necessary for this? cause even without specifying group keyword and the group name, it doesn't work.
 
You can set a consumer group for your console consumer from command line --consumer-property group.id=<some-group>, or create a consumer.properties file and pass it as --consumer.config=<consumer.properties>.
If you use TLS for client auth, your user is the CN in client certificate, and you should use it as a principal in --allow-principal option when setting Kafka ACLs.
I suggest you check out cwiki.apache.org/confluence/display/KAFKA/… document and associated KIP referenced there.
SO encourages users to ask separate questions (with informative titles) if they have multiple issues rather than keep adding questions to their original posts. Its free and helps others to find answers if they have similar problems,
 
got it. also I tried dong this: ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:swe-analyticsdb-prod2 --consumer --topic my-topic --group=*, but it saud kafka-acls.sh couldn't be found. I then tried this: ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:swe-analyticsdb-prod2 --consumer --topic my-topic --group = * and it worked but still can't run consumer
 
Maybe the CN in your client cert is something like CN=swe-analyticsdb-prod2,OU=xxxxx,O=xxxxx,... ? Then you should use this whole thing in kafka-acls command. Or, for testing only, allow all users: --allow-principal User:*. Another option -- enable authorizer debugging and look through the log to see what exactly the problem is. stackoverflow.com/questions/37536259/…
 
This is what the server gives me when I start it: org.apache.kafka.common.errors.ClusterAuthorizationException‌​: Request Request(processor=0, connectionId=10.15.164.233:9093-10.15.164.233:48606-0, session=Session(User:1.2.840.113549.1.9.1=#161164696e6840737‌​96e6f707379732e636f6‌​d,CN=swe-analyticsdb‌​-prod2,OU=Software Engineering,O=Synopsys,L=Mountain View,ST=CA,C=US,/10.15.164.233), listenerName=ListenerName(SSL), securityProtocol=SSL, buffer=null) is not authorized. How can I change user to "swe-analyticsdb-prod2" @mazaneicha
 
9:40 PM
Run ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:* --allow-host swe-analyticsdb-prod2 --add --cluster, and restart your Kafka server. Then run ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list to see all ACLs that you have.
 
please check the results above in the main post @mazaneicha. I cannot do "User=*" because ill get the error: java.lang.IllegalArgumentException: expected a string in format principalType:principalName but got --allow-host. I think that us causing my problem
 
It should be User:* not User=*. Then try ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --add --consumer --topic 'my-topic' --group='*'
 
Sorry I did use ":", but I tried the new command and got this navdeploy@swe-analyticsdb-prod2:/remote/sde108/kafka/kafka/c‌​onfig % ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --add --consumer --topic 'my-topic' --group='*' ../bin/kafka-acls.sh: No match. and when I added a space between the "*" and "User:" I got Option "[consumer]" can't be used with option "[operation]" @mazaneicha
 
Yeah, cut-n-paste problem... please remove --operation Read from it.
 
the revised command gives me this: navdeploy@swe-analyticsdb-prod2:/remote/sde108/kafka/kafka/c‌​onfig % ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --allow-principal User:* --add --consumer --topic 'my-topic' --group='*' ../bin/kafka-acls.sh: No match.. Do I need to add a space between User and *, like this: ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --allow-principal User: * --add --consumer --topic 'my-topic' --group='*' ?
 
9:40 PM
What Kafka version are you running? It works without spaces for me (exactly as shown here docs.confluent.io/4.0.0/kafka/authorization.html#adding-acls‌​) but you can try adding a space, as long as after that the --list shows that all users have Read permission on topic and all groups.
 
im running kafka 2.2.0. also can we move this discussion to chat?
 
Well, the kafka.apache.org/22/documentation.html#security_authz_exampl‌​es in second example also shows that --allow-principal User:* should be supported. So feel free to report a bug if you're confident in your setup. And sorry, I didn't work with 2.2 myself so won't be able to help.
 
I have a few more questions. can I ask you them here?
following your previous commands: ../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:'*' --allow-host swe-analyticsdb-prod2 --add --cluster
../bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list
I got this:
urrent ACLs for resource `Cluster:LITERAL:kafka-cluster`:
User:* has Allow permission for operations: All from hosts: swe-analyticsdb-prod2

Current ACLs for resource `Group:LITERAL:test`:
User: has Allow permission for operations: Read from hosts: connect-console-sink.properties

Current ACLs for resource `Group:LITERAL:controller.log`:
User:navdeploy has Allow permission for operations: Read from hosts: controller.log

Current ACLs for resource `Group:LITERAL:connect-console-sink.properties`:
 

last day (24 days later) »