Refactor Apache Kibble project

[turbaszek]

Refactor Apache Kibble project

This is an umbrella issue for collecting ideas and managing the work that has to be done.
Dev list thread: https://lists.apache.org/thread.html/r3f373be93397fc55ad6aff11030d2f3eb93f0f34782ff58fba9325b1%40%3Cdev.kibble.apache.org%3E

1. General refactor

• Move scanners to main kibble repo
• Make kibble a package
• Add command-line interface to configure kibble and start all components (including scanners)
• Rethink configuration of kibble Refactor the configuration methods of Apache Kibble #52

2. Rewrite API server

• Add tests for API endpoints to help us preserve backward compatibility and to limit the scope of work to backend stuff. The frontend can be done in future. Having the open API spec we should be able to generate some part of required code.
• Decide which framework we would like to use. The main requirement is that it should work with the existing OpenAPI spec and should do automagically as much as possible.
• Rewrite all endpoint to use new framework

Additional considerations

• Consider implementing GraphQL server
• Schedule kibble scanners using celery beat

[auvipy]

what about considering JSON-API 1.1rc+++ instead of GraphQL server?

[turbaszek]

what about considering JSON-API 1.1rc+++ instead of GraphQL server?

@auvipy thanks for the suggestion! I will take a look at that 👍 But you reminded me that we may also consider running kibble scanners using celery / airflow.

[auvipy]

ahh 🗡️

[skekre98]

Hi, I would be interested in helping build out the cli for kibble. I have experience building out command line interfaces in Golang as well as Python. Willing to help out wherever needed.

[turbaszek]

Hi, I would be interested in helping build out the cli for kibble. I have experience building out command line interfaces in Golang as well as Python. Willing to help out wherever needed.

Awesome @skekre98 ! What do you think about using https://click.palletsprojects.com/en/7.x/ ?
The cli should be launched here:
https://github.com/apache/kibble/blob/edb9a91165ff3e3eebbee99db6c86ddf87c362f4/kibble/__main__.py#L18-L23
so users can do kibble command --flags --and-more

[skekre98]

Sure @turbaszek, will finish up KibbleConfigParser and then take a stab at this.

[michalslowikowski00]

This step: Make kibble a package is done. 🚀