Skip to content
This repository was archived by the owner on Dec 16, 2022. It is now read-only.

Commit c544b15

Browse files
author
Seth Art
committed
Upload to github
0 parents  commit c544b15

8 files changed

+984
-0
lines changed

‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.idea
2+
*.pid
3+
*.sqlite3
4+
*.pyc

‎LICENSE

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Copyright 2018 Seth Art
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
4+
5+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
6+
7+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎README.md

+107
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# celerystalk
2+
3+
celerystalk automates your network scanning/enumeration process with asynchronous jobs (aka *tasks*). This allows you to scan each service the same way so you don't have to keep track of what you ran where. celerystalk works well when used against one host, but it was designed for scanning multiple hosts.
4+
5+
### Credit
6+
This project was inspired by many great tools:
7+
1. https://github.com/codingo/Reconnoitre by @codingo_
8+
1. https://github.com/frizb/Vanquish by @frizb
9+
1. https://github.com/leebaird/discover by @discoverscripts
10+
1. https://github.com/1N3/Sn1per
11+
1. https://github.com/SrFlipFlop/Network-Security-Analysis by @SrFlipFlop
12+
13+
Thanks to @offensivesecurity and @hackthebox_eu for their lab networks :)
14+
15+
Also, thanks to:
16+
1. @decidedlygray for pointing me towards celery, helping me solve python problems that were over my head, and for the extensive beta testing
17+
1. @kerpanic for inspiring me to dust off an old project and turn it into celerystalk
18+
1. My TUV OpenSky team and my IthacaSec hackers for testing this out and submitting bugs and features
19+
20+
21+
### Install/Setup
22+
23+
Directly into Kali:
24+
25+
```
26+
# git clone <repo>
27+
# ./install.sh
28+
```
29+
30+
Install with virtualenv:
31+
32+
(Below Not working yet)
33+
34+
```
35+
# apt install redis-server python-libnmap seclists -y
36+
# pip install --user pipenv
37+
# git clone --recursive <repo>
38+
# cd celerystalk && pipenv install
39+
# pipenv shell
40+
# python setup.py install
41+
42+
```
43+
44+
The basic workflow:
45+
46+
47+
1. **Run Nmap or Nessus:**
48+
49+
Run nmap against your target(s) with version detection on and output to XML (nmap target -Pn -p- -sV -oX target.xml)
50+
51+
or
52+
53+
Run nessus against your target(s) and export a .nessus file
54+
55+
1. **Configure celerystalk:** For each service type (i.e., http, https, ssh), configure what commands you want to run (config.json)
56+
1. **Launch Scan:** Run celerystalk against the nmap or nessus XML and it will submit tasks to celery workers which asynchronously execute them and log output to your output directory
57+
1. **Query Status:** Asynchronously check the status of the tasks queue, as frequently as you like
58+
1. **Cancel/Pause/Resume Tasks:** Cancel/Pause/Resume any task(s) that are currently running or in the queue.
59+
1. **Run Report:** Run a report which combines all of the tool output into an html file and a txt file (Run this as often as you like)
60+
61+
62+
### Usage
63+
```
64+
Usage:
65+
celerystalk scan -f <nmap_file> -o <output_dir> [-w <workspace>] [-t <targets>] [-d]
66+
celerystalk query [-w <workspace>] [repeat]
67+
celerystalk report <report_dir> [-w <workspace>]
68+
celerystalk cancel ([all]|[<task_ids>]|[ip]) [-w <workspace>]
69+
celerystalk pause ([all]|[<task_ids>]|[ip]) [-w <workspace>]
70+
celerystalk resume ([all]|[<task_ids>]|[ip]) [-w <workspace>]
71+
celerystalk shutdown
72+
celerystalk (help | -h | --help)
73+
74+
Options:
75+
-h --help Show this screen
76+
-v --version Show version
77+
-f <nmap_file> Nmap xml import file
78+
-o <output_dir> Output directory
79+
-t <targets> Target(s): IP, IP Range, CIDR
80+
-w <workspace> Workspace [default: Default]
81+
-d --dry_run Dry run
82+
83+
84+
Prerequisites:
85+
Scan target(s) with: Nessus and export nessus db (scan.nessus ) or
86+
nmap target(s) -Pn -p- -sV -oX /pentest/nmap.xml
87+
88+
Examples:
89+
Start from Nmap XML file: celerystalk scan -f /pentest/nmap.xml -o /pentest
90+
Start from Nessus file: celerystalk scan -f /pentest/scan.nessus -o /pentest
91+
Use non-default workspace: celerystalk scan -f <file> -o /pentest -w test
92+
Scan subset of Nmap XML: celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.1,10.0.0.3
93+
celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.100-200
94+
celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.0/24
95+
Dry Run: celerystalk scan -f <file> -o /pentest -d
96+
Query Tasks: celerystalk query [-w workspace]
97+
Create Report: celerystalk report /pentest #Create a report for all scanneed hosts in /pentest
98+
celerystalk report /pentest/10.0.0.1 #Create a report for a single host
99+
Cancel/Pause/Resume Tasks: celerystalk <verb> 5,6,10-20 #Cancel/Pause/Resume tasks 5, 6, and 10-20
100+
celerystalk <verb> all #Cancel/Pause/Resume all tasks from all workspaces
101+
celerystalk <verb> all -w test #Cancel/Pause/Resume all tasks in the test workspace
102+
celerystalk <verb> 10.0.0.1 #Cancel/Pause/Resume all tasks related to 10.0.0.1
103+
celerystalk <verb> 10.0.0.0/24 #Cancel/Pause/Resume all tasks related to 10.0.0.0/24
104+
Shutdown Celery processes: celerystalk shutdown
105+
106+
```
107+

‎celerystalk

+274
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
#!/usr/bin/env python
2+
3+
"""
4+
Usage:
5+
celerystalk scan -f <nmap_file> -o <output_dir> [-w <workspace>] [-t <targets>] [-d <domains>] [-s]
6+
celerystalk query [-w <workspace>] ([full] | [summary] | [brief]) [watch]
7+
celerystalk query [-w <workspace>] [watch] ([full] | [summary] | [brief])
8+
celerystalk report <report_dir> [-w <workspace>]
9+
celerystalk cancel ([all]|[<task_ids>]) [-w <workspace>]
10+
celerystalk pause ([all]|[<task_ids>]) [-w <workspace>]
11+
celerystalk resume ([all]|[<task_ids>]) [-w <workspace>]
12+
celerystalk shutdown
13+
celerystalk (help | -h | --help)
14+
15+
Options:
16+
-h --help Show this screen
17+
-v --version Show version
18+
-f <nmap_file> Nmap xml import file
19+
-o <output_dir> Output directory
20+
-t <targets> Target(s): IP, IP Range, CIDR
21+
-w <workspace> Workspace [default: Default]
22+
-d --domains Domains to scan for vhosts
23+
-s --simulation Simulation mode. Submit tasks comment out all commands
24+
25+
Examples:
26+
27+
Start from Nmap XML file: celerystalk scan -f /pentest/nmap.xml -o /pentest
28+
Start from Nessus file: celerystalk scan -f /pentest/scan.nessus -o /pentest
29+
Specify workspace: celerystalk scan -f <file> -o /pentest -w test
30+
Find in scope vhosts: celerystalk scan -f <file> -o /pentest -d domain1.com,domain2.com
31+
Scan subset hosts in XML: celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.1,10.0.0.3
32+
celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.100-200
33+
celerystalk scan -f <file> -o /pentest -w test -t 10.0.0.0/24
34+
Simulation mode: celerystalk scan -f <file> -o /pentest -d
35+
Query Tasks: celerystalk query [-w workspace]
36+
Create Report: celerystalk report /pentest #Create a report for all scanneed hosts in /pentest
37+
celerystalk report /pentest/10.0.0.1 #Create a report for a single host
38+
Cancel/Pause/Resume Tasks: celerystalk <verb> 5,6,10-20 #Cancel/Pause/Resume tasks 5, 6, and 10-20
39+
celerystalk <verb> all #Cancel/Pause/Resume all tasks from all workspaces
40+
celerystalk <verb> all -w test #Cancel/Pause/Resume all tasks in the test workspace
41+
celerystalk <verb> 10.0.0.1 #Cancel/Pause/Resume all tasks related to 10.0.0.1
42+
celerystalk <verb> 10.0.0.0/24 #Cancel/Pause/Resume all tasks related to 10.0.0.0/24
43+
Shutdown Celery processes: celerystalk shutdown
44+
"""
45+
46+
from docopt import docopt
47+
import os
48+
from time import sleep
49+
import sys
50+
import subprocess
51+
import lib.cancel
52+
import lib.scan
53+
import lib.resume
54+
import lib.pause
55+
import lib.utils
56+
import lib.report
57+
import lib.query
58+
#
59+
# from lib.resume import resume_paused_tasks
60+
# from lib.pause import pause_running_tasks
61+
# from lib.utils import nmap_parser, nmap_scan, start_services, \
62+
# shutdown_background_jobs, target_splitter, nessus_parser
63+
# from lib.report import report
64+
# from lib.query import query_sqlite
65+
from lib import db
66+
67+
68+
def print_banner():
69+
# print("""
70+
# _ _ _ _
71+
# __ ___| |___ _ _ _ _ __| |_ __ _| | |__
72+
# / _/ -_| / -_| '_| || (_-| _/ _` | | / /
73+
# \__\___|_\___|_| \_, /__/\__\__,_|_|_\_\\
74+
# |__/
75+
# """)
76+
print("\ncelerystalk -\tAn asynchronous network enumeration/vulnerability scanner that supports distributed workers")
77+
print("\t\tby @sethsec")
78+
print("\t\tv1.0, 20180718\n")
79+
80+
81+
def main(arguments):
82+
if arguments["-w"]:
83+
workspace = arguments["-w"]
84+
else:
85+
workspace = 'Default'
86+
87+
db.create_task_table()
88+
db.create_path_table()
89+
db.create_services_table()
90+
db.create_vhosts_table()
91+
92+
93+
94+
# Query - Inform user about job status
95+
if arguments["query"]:
96+
try:
97+
if arguments["watch"]:
98+
if arguments["summary"]:
99+
subprocess.call(["watch", "./celerystalk", "query", "summary","-w",workspace])
100+
#lib.query.query_sqlite(workspace, repeat=True,summary=arguments["summary"])
101+
else:
102+
subprocess.call(["watch", "./celerystalk", "query", "brief","-w",workspace])
103+
#lib.query.query_sqlite(workspace,repeat=True)
104+
else:
105+
if arguments["summary"]:
106+
lib.query.query_sqlite(workspace,summary=arguments["summary"])
107+
elif arguments["brief"]:
108+
lib.query.query_sqlite(workspace,repeat=True)
109+
else:
110+
lib.query.query_sqlite(workspace)
111+
except KeyboardInterrupt:
112+
sys.exit(0)
113+
except EnvironmentError as e:
114+
print(e)
115+
116+
if arguments["cancel"]:
117+
paused_tasks = db.get_paused_tasks(workspace)
118+
running_tasks = db.get_running_tasks(workspace)
119+
pending_tasks = db.get_pending_tasks(workspace)
120+
if (len(pending_tasks) == 0) and (len(running_tasks) == 0) and (len(paused_tasks) == 0):
121+
print("[+] There were no tasks to cancel. Are you sure you have the right workspace?\n")
122+
exit()
123+
124+
if arguments["all"]:
125+
lib.cancel.cancel_tasks("all", workspace)
126+
# if len(paused_tasks) > 0:
127+
# print("\n[+] Resuming all paused tasks.\n")
128+
# resume_paused_tasks("all", workspace)
129+
# if len(running_tasks) > 0:
130+
# print("\n[+] Canceling all tasks.\n")
131+
# cancel_tasks("all", workspace)
132+
elif arguments["<task_ids>"]:
133+
lib.cancel.cancel_tasks(arguments["<task_ids>"], workspace)
134+
#
135+
# if len(paused_tasks) > 0:
136+
# print("\n[+] Resuming any selected tasks that are paused.\n")
137+
# resume_paused_tasks(arguments["<task_ids>"], workspace)
138+
# if len(running_tasks) > 0:
139+
# print("\n[+] Canceling selected tasks.\n")
140+
# cancel_tasks(arguments["<task_ids>"], workspace)
141+
exit()
142+
143+
if arguments["pause"]:
144+
if arguments["all"]:
145+
num_paused = lib.pause.pause_running_tasks("all",workspace)
146+
sleep(3)
147+
num_paused = lib.pause.pause_running_tasks("all",workspace,repeat="True")
148+
elif arguments["<task_ids>"]:
149+
num_paused = lib.pause.pause_running_tasks(arguments["<task_ids>"], workspace)
150+
else:
151+
print("[!] You need to specify the tasks to pause, or specify \"all\".\n")
152+
exit()
153+
if num_paused > 0:
154+
print("\n")
155+
else:
156+
print("[+] There were no tasks to pause.\n")
157+
exit()
158+
159+
if arguments["resume"]:
160+
if arguments["all"]:
161+
num_resumed = lib.resume.resume_paused_tasks("all",workspace)
162+
elif arguments["<task_ids>"]:
163+
num_resumed = lib.resume.resume_paused_tasks(arguments["<task_ids>"], workspace)
164+
else:
165+
exit()
166+
if num_resumed > 0:
167+
print("\n")
168+
else:
169+
print("[+] There were no tasks to resume.\n")
170+
exit()
171+
172+
173+
if arguments["shutdown"]:
174+
lib.utils.shutdown_background_jobs()
175+
exit()
176+
177+
# Run Report
178+
if arguments["report"]:
179+
if arguments["<report_dir>"]:
180+
output_dir = os.path.join(arguments["<report_dir>"], '')
181+
else:
182+
print('Specify directory to recursively scan for celerystalk output')
183+
lib.report.report(output_dir)
184+
exit()
185+
186+
# Parse nmap then run celerystalk (for just some hosts in target file)
187+
if arguments["-f"] and arguments["-t"]:
188+
if arguments["-o"] is None:
189+
print('Specify report directory')
190+
else:
191+
output_dir = os.path.join(arguments["-o"], '')
192+
# Before doing anything host specific, make sure the user specified output directory exists (i.e., /pentest)
193+
try:
194+
os.stat(output_dir)
195+
except:
196+
print("[+] Output directory does not exist. Creating " + output_dir)
197+
os.mkdir(output_dir)
198+
#TODO: make this read the file and be better :)
199+
print(arguments["-f"])
200+
201+
lib.utils.start_services()
202+
#(json_config, supported_services) = read_config_ini()
203+
target_list = lib.utils.target_splitter(arguments["-t"])
204+
205+
for target in target_list:
206+
if "nessus" in arguments["-f"]:
207+
nessus_report = lib.utils.nessus_parser(arguments["-f"])
208+
# process_nessus_data(nessus_report, output_dir, arguments["--simulation"],workspace)
209+
lib.scan.process_nessus_data2(nessus_report, workspace,target=target)
210+
if arguments["<domains>"]:
211+
lib.scan.find_subdomains(arguments["<domains>"], arguments["--simulation"], workspace, output_dir)
212+
lib.scan.process_db_services(output_dir, arguments["--simulation"], workspace,target=target)
213+
214+
else:
215+
nmap_report = lib.utils.nmap_parser(arguments["-f"])
216+
# process_nmap_data(nmap_report, output_dir, arguments["--simulation"],workspace)
217+
lib.scan.process_nmap_data2(nmap_report, workspace,target=target)
218+
if arguments["<domains>"]:
219+
lib.scan.find_subdomains(arguments["<domains>"], arguments["--simulation"], workspace, output_dir)
220+
lib.scan.process_db_services(output_dir, arguments["--simulation"], workspace,target=target)
221+
exit()
222+
223+
# Parse nmap then run celerystalk (for ALL hosts in target file)
224+
if arguments["-f"]:
225+
if arguments["-o"] is None:
226+
print('Specify report directory')
227+
else:
228+
output_dir = os.path.join(arguments["-o"], '')
229+
# Before doing anything host specific, make sure the user specified output directory exists (i.e., /pentest)
230+
try:
231+
os.stat(output_dir)
232+
except:
233+
print("[+] Output directory does not exist. Creating " + output_dir)
234+
os.mkdir(output_dir)
235+
236+
lib.utils.start_services()
237+
#(json_config, supported_services) = read_config_ini()
238+
if "nessus" in arguments["-f"]:
239+
nessus_report = lib.utils.nessus_parser(arguments["-f"])
240+
#process_nessus_data(nessus_report, output_dir, arguments["--simulation"],workspace)
241+
lib.scan.process_nessus_data2(nessus_report, workspace)
242+
if arguments["<domains>"]:
243+
lib.scan.find_subdomains(arguments["<domains>"], arguments["--simulation"], workspace, output_dir)
244+
lib.scan.process_db_services(output_dir, arguments["--simulation"], workspace)
245+
246+
else:
247+
nmap_report = lib.utils.nmap_parser(arguments["-f"])
248+
#process_nmap_data(nmap_report, output_dir, arguments["--simulation"],workspace)
249+
lib.scan.process_nmap_data2(nmap_report, workspace)
250+
if arguments["<domains>"]:
251+
lib.scan.find_subdomains(arguments["<domains>"], arguments["--simulation"], workspace, output_dir)
252+
lib.scan.process_db_services(output_dir, arguments["--simulation"], workspace)
253+
254+
#commenting this out for now.
255+
# # Scan targets with nmap and then run celerystalk
256+
#
257+
# if arguments["-t"]:
258+
# if arguments["-o"] is None:
259+
# print('Specify report directory')
260+
# else:
261+
# output_dir = os.path.join(arguments["-o"], '')
262+
# lib.utils.start_services()
263+
# #(json_config, supported_services) = read_config_ini()
264+
# nmap_report = lib.scan.nmap_scan(arguments["-t"], output_dir)
265+
# lib.scan.process_nmap_data(nmap_report, output_dir, arguments["--simulation"],workspace)
266+
267+
268+
if __name__ == "__main__":
269+
arguments = docopt(__doc__, version='celerystalk 1.0')
270+
#print(arguments)
271+
print_banner()
272+
if arguments["help"]:
273+
print(__doc__)
274+
main(arguments)

0 commit comments

Comments
 (0)