Add scanners to main repository

[turbaszek]

This PR moves Apache Kibble scanners from
https://github.com/apache/kibble-scanners to https://github.com/apache/kibble
as discussed here:
https://lists.apache.org/thread.html/r3f373be93397fc55ad6aff11030d2f3eb93f0f34782ff58fba9325b1%40%3Cdev.kibble.apache.org%3E

I reviewed all files and fixed licenses headers, removed unused imports and adjusted imports. It seems to look good:

➜ python kibble/scanners/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
'Kibble Scanner v/0.1.0 starting'
kibble/scanners/kibble-scanner.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(open(CONFIG_FILE))
('Loaded YAML config from '
 '/Users/turbaszek/code/kibble/kibble/scanners/config.yaml')
'Using direct ElasticSearch broker model'
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
'All done scanning for now, found 0 organisations and 0 sources to process.'

However, doing this review I found few errors in implementation (undefined variables, missing imports etc) and I'm wondering, wouldn't it be better to add the scanners one by one to improve them in this process?

Probably we should do this once we have #77

[turbaszek]

However, doing this review I found few errors in implementation (undefined variables, missing imports etc) and I'm wondering, wouldn't it be better to add the scanners one by one to improve them in this process?

What do you think @Humbedooh @michalslowikowski00 @sharanf ? Other option is that we accept what we have and then we fix it + add tests.

[michalslowikowski00]

If we move the scanners to the main repo and we leave them broken -- what are the consequences? what is the impact on the user?

[turbaszek]

If we move the scanners to the main repo and we leave them broken -- what are the consequences? what is the impact on the user?

The proper answer is: users will be surprised. But they will also be surprised now 😄 I think we don't have thousands of users currently and we can risk merging everything and then fixing. This has the advantage of doing more work in parallel (scanners fixes, configuration refactor, etc)

[sharanf]

If we move the scanners to the main repo and we leave them broken -- what are the consequences? what is the impact on the user?

The proper answer is: users will be surprised. But they will also be surprised now smile I think we don't have thousands of users currently and we can risk merging everything and then fixing. This has the advantage of doing more work in parallel (scanners fixes, configuration refactor, etc)

I agree, I don't think we have a lot of users downloading and setting up Kibble. My thought is that the majority of users are coming from within Apache to check on their own project statistics which they can do easily with the demo. That doesn't mean that we won't have some users out there that will be surprised that things are changing! Ideally if people are really upset then we will find out here or on the mailing list, and then we can look at doing something to ease them through it.

For the scanners we did have a couple of requests to add some new ones, Gitlab, Slack and bitbucket so will need to bring those across into our current issue list.

[Humbedooh]

Ignore this, I am just testing a webhook - will delete again :)

[turbaszek]

@sharanf @Humbedooh @michalslowikowski00 if there's no strong objection I will merge this PR