Replies: 25 comments
-
|
Welcome everyone to contribute |
Beta Was this translation helpful? Give feedback.
-
|
Basically realized the function of pulsar operator，include pulsar dashboard, grafana, prometheus and so on Pulsar Operator，weclome to use and contribute. |
Beta Was this translation helpful? Give feedback.
-
|
Next step concern pulsar stateful component(include Bookeeper and Zookeeper), use operator control complex stateful component. |
Beta Was this translation helpful? Give feedback.
-
|
Great initiative! I know you have to start somewhere and cannot do everything at once but have you thought about CRD's for Sources and Sinks? What about configuration like Channels, Tenants and Namespaces? I would really like to get out of an imperative approach based on pulsar-admin for managing the cluster. Still trying to figure out how to get the whole configuration for a geo-replicated cluster in Git... |
Beta Was this translation helpful? Give feedback.
-
@chrsoo Maybe the pulsar-manager is a good idea for you. |
Beta Was this translation helpful? Give feedback.
-
|
@wolfstudy thanks for the tip (I am aware) but how is pulsar-manager not imperative? A web UI where you do point-and-click is virtually the same thing as issuing commands on the command line. At each instance in time you tell the system what to do instead of telling it the desired target state and have it figure out the changes by itself. In the end I would like to declare the state of my cluster in an external git repository and then have a CI/CD pipeline provision it for me (gitops). The CRD proposed by @sky-big is a really good start and I could work with this towards my end goal, but I need to declare other parts of the configuration as well. Point-and-click will not do the job :-) |
Beta Was this translation helpful? Give feedback.
-
|
@chrsoo Cloud you please give a sepcific example based on your sence? then we can add this function in the pulsar operator. |
Beta Was this translation helpful? Give feedback.
-
|
The difficulty here is where to draw the line between more or less static configuration and what should remain dynamically defined at runtime by the clients/users of the system. This might vary in different contexts. I think it could be useful to define CRDs for the following Pulsar configuration items:
Starting with Tenants the following CRDs might be useful
It would be good if Tenants and Namespaces are defined at the K8s cluster level as a Tenants and Namespaces then could be reused across multiple PulsarClusters defined in the different K8s namespaces. The PulsarNamespaceBinding should be defined at the namespace level and in the same namespace as the PulsarCluster to which it is bound. @sky-big - does all this make sense to you? |
Beta Was this translation helpful? Give feedback.
-
|
@wolfstudy @sijie @jiazhai Can you disscuss it together? |
Beta Was this translation helpful? Give feedback.
-
|
@chrsoo I generally understand what you mean, Starting with Tenants CRDs, Can you make a specific design and let everyone discuss？ |
Beta Was this translation helpful? Give feedback.
-
|
@chrsoo @sky-big @wolfstudy I think what @chrsoo request is an operator for the full lifecycle management for all the resources within a Pulsar instance. This includes both ClusterOperator and ResourceOperator. A ClusterOperator is what @sky-big is doing and most of the operators are doing. It is used for setting up a cluster and managing the upgrade, downgrade and scaling up-and-down a cluster. A ResourceOperator is more about using CRD for provisioning resources within a Pulsar cluster. A resource such as a role (key and tokens), a tenant, a namespace, a topic, a function and etc. Once everything is modeled as CRD, you can use kubernetes to provision both cluster and resources. Thus you can apply a GitOps patten in a CI/CD pipeline to manage everything. @chrsoo : this is kind of aligned with what we (StreamNative) are working on. We shall be able to share what we have done soon. |
Beta Was this translation helpful? Give feedback.
-
|
@sijie yes, exactly. Should we take your comment regarding StreamNative to mean that you are planning to open source an Operator for Pulsar on Kubernetes that deals with the full scope of CRD's for Pulsar management? How soon is "soon"? @sky-big regarding a specific design I would design it as explained above but I will see if I can produce a more detailed design specification as a PR. |
Beta Was this translation helpful? Give feedback.
-
|
@chrsoo OK, Looking forward to you PR. |
Beta Was this translation helpful? Give feedback.
-
|
Operator add pulsar manager component success. |
Beta Was this translation helpful? Give feedback.
-
|
This is cool to see. A team I am on was going to start making a Pulsar Operator using Ansible OperatorSDK. https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/user-guide.md Here is an Ansible Operator example for reference. Would the Pulsar team be interested in this work? Any opinions on Ansible vs Go? |
Beta Was this translation helpful? Give feedback.
-
|
I think both ansible and go operators can co-exist. I would also recommend submitting the operators to the operator hub as well so that it can be exposed to wider community. |
Beta Was this translation helpful? Give feedback.
-
|
The next step is to publish to OperatorHub |
Beta Was this translation helpful? Give feedback.
-
@sijie any feedback on this?
@sky-big sorry for not getting back to you, the pulsar project took a back-seat for while where I work. It is getting reprioritised again so I will start looking into that PR again. Note that I am not talking about code at this point but at least I should be able to put how I think it should work. |
Beta Was this translation helpful? Give feedback.
-
|
@chrsoo OK |
Beta Was this translation helpful? Give feedback.
-
|
I created slack https://pulsaroperator.slack.com/ |
Beta Was this translation helpful? Give feedback.
-
|
Why not just an #operator channel in the Pulsar Slack? We hang out in there so not sure if this is big enough effort to need a whole new Slack. Having a design spec laid out is a great idea because we are starting to build something similar. If we have an agreed upon approach, members from my team could contribute functionality to flesh out the implementation. Also we can perhaps keep separate Operators as Strimzi did for their Kafka Operators. This would allow the current big-sky Operator to exist for managing the cluster and my team could create another operator for managing the User,Topic, Tenant, Namespace type objects. |
Beta Was this translation helpful? Give feedback.
-
|
@nicknezis Hi, There's another team looking at the same thing, we can discuss it together here. We've been talking about priorities for all function points, and we've been talking about rules. |
Beta Was this translation helpful? Give feedback.
-
|
@nicknezis I‘ve invited you to join the channel |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, |
Beta Was this translation helpful? Give feedback.
-
|
Open-ended discussion. Moved to the Discussions forum. |
Beta Was this translation helpful? Give feedback.
-
Add pulsar operator for kubernetes(https://github.com/sky-big/pulsar-operator) based on deployment/kubernetes/*.yaml
Beta Was this translation helpful? Give feedback.
All reactions