|
1 | 1 | # OONI backend |
2 | 2 |
|
3 | | -Welcome. This document describes the architecture of the main components of the |
4 | | -OONI infrastructure. |
| 3 | +Welcome to the OONI backend. |
5 | 4 |
|
6 | | -The documentation is meant for core contributors, external contributors and researcher |
7 | | -that want to extract data or reuse software components in their own projects. |
8 | | - |
9 | | -This file is [rendered here](https://ooni.github.io/pipeline/README.html) |
10 | | - |
11 | | -You can also explore the [documentation tree](https://ooni.github.io/pipeline/) |
12 | | - |
13 | | -## Table of contents |
14 | | - |
15 | | -[TOC] |
16 | | - |
17 | | -## Architecture |
18 | | - |
19 | | -The backend infrastructure provides multiple functions: |
20 | | - |
21 | | -* Provide APIs for data consumers |
22 | | -* Instruct probes on what measurements to perform |
23 | | -* Receive measurements from probes, process them and store them in the database and on S3 |
24 | | - |
25 | | -## Data flow |
26 | | - |
27 | | -This diagram represent the main flow of measurement data |
28 | | - |
29 | | - |
30 | | -blockdiag { |
31 | | - Probes [color = "#ffeeee"]; |
32 | | - Explorer [color = "#eeeeff"]; |
33 | | - "S3 jsonl" [shape = ellipse]; |
34 | | - "S3 postcan" [shape = ellipse]; |
35 | | - "DB jsonl tbl" [shape = ellipse]; |
36 | | - "DB fastpath tbl" [shape = ellipse]; |
37 | | - "disk queue" [shape = ellipse]; |
38 | | - |
39 | | - Probes -> "API: Probe services" -> "Fastpath" -> "DB fastpath tbl" -> "API: Measurements" -> "Explorer"; |
40 | | - "API: Probe services" -> "disk queue" -> "API: uploader" -> "S3 jsonl" -> "API: Measurements"; |
41 | | - "API: uploader" -> "S3 postcan"; |
42 | | - "API: uploader" -> "DB jsonl tbl"; |
43 | | - "DB jsonl tbl" -> "API: Measurements" |
44 | | -} |
45 | | - |
46 | | - |
47 | | -Each measurement is processed individually in real time. |
48 | | - |
49 | | - |
50 | | -## Components: API |
51 | | - |
52 | | -The API entry points are documented at [apidocs](https://api.ooni.io/apidocs/) |
53 | | - |
54 | | -### Measurements |
55 | | - |
56 | | -Provide access to measurements to end users directly and through Explorer. |
57 | | - |
58 | | -Mounted under /api/v1/measurement/ |
59 | | - |
60 | | -The API is versioned. Access is rate limited based on source IP address and access tokens |
61 | | -due to the computational cost of running heavy queries on the database. |
62 | | - |
63 | | -[Sources](https://github.com/ooni/api/blob/master/newapi/ooniapi/probe_services.py) |
64 | | - |
65 | | -### Probe services |
66 | | - |
67 | | -Serves lists of collectors and test helpers to the probes and receive measurements from them. |
68 | | - |
69 | | -Mounted under /api/v1/ |
70 | | - |
71 | | -[Sources](https://github.com/ooni/api/blob/master/newapi/ooniapi/probe_services.py) |
72 | | - |
73 | | -### Private entry points |
74 | | - |
75 | | -Not for public consumption. Mounted under `/api/_` and used exclusively by Explorer |
76 | | - |
77 | | -[Sources](https://github.com/ooni/api/blob/master/newapi/ooniapi/private.py) |
78 | | - |
79 | | -## Fastpath |
80 | | - |
81 | | -[Documentation](af/fastpath/fastpath/core.html) |
82 | | - |
83 | | -## Database |
84 | | - |
85 | | -## Operations |
86 | | - |
87 | | -### Build, deploy, rollback |
88 | | - |
89 | | -Host deployments are done with the [sysadmin repo](https://github.com/ooni/sysadmin) |
90 | | - |
91 | | -For component updates a deployment pipeline is used: |
92 | | - |
93 | | -Look at the [Status dashboard](https://github.com/ooni/backend/wiki/Backend) - be aware of badge image caching |
94 | | - |
95 | | -Use the deploy tool: |
96 | | - |
97 | | -```bash |
98 | | -# Update all badges: |
99 | | -dep refresh_badges |
100 | | - |
101 | | -# Show status |
102 | | -dep |
103 | | - |
104 | | -# Deploy/rollback a given version on the "test" stage |
105 | | -deploy ooni-api test 0.6~pr194-147 |
106 | | - |
107 | | -# Deploy latest build on the first stage |
108 | | -deploy ooni-api |
109 | | - |
110 | | -# Deploy latest build on a given stage |
111 | | -deploy ooni-api prod |
112 | | - |
113 | | -``` |
114 | | - |
115 | | -### Adding new tests |
116 | | - |
117 | | -Update [database_upgrade_schema](https://github.com/ooni/pipeline/blob/master/af/fastpath/database_upgrade_schema.py) |
118 | | - |
119 | | -``` |
120 | | -ALTER TYPE ootest ADD VALUE '<test_name>'; |
121 | | -``` |
122 | | - |
123 | | -Update [fastpath](https://github.com/ooni/pipeline/blob/master/af/fastpath/fastpath/core.py) |
124 | | -by adding a new test to the `score_measurement` function and adding relevant |
125 | | -integration tests. |
126 | | - |
127 | | -Create a [Pull Request](https://github.com/ooni/pipeline/compare) |
128 | | - |
129 | | -Run fastpath manually from S3 on the testing stage see: [rerun fastpath manually](#rerun-fastpath-manually) |
130 | | - |
131 | | -Update the [api](https://github.com/ooni/api/blob/master/newapi/ooniapi/measurements.py#L491) |
132 | | - |
133 | | -### Adding new fingerprints |
134 | | - |
135 | | -TODO |
136 | | - |
137 | | -### API runbook |
138 | | - |
139 | | -Monitor the [API](https://mon.ooni.nu/grafana/d/CkdDBscGz/ams-pg-api?orgId=1) and |
140 | | -[fastpath](https://mon.ooni.nu/grafana/d/75nnWVpMz/fastpath-ams-pg?orgId=1) dashboards. |
141 | | - |
142 | | -Follow Nginx or API logs with: |
143 | | -```bash |
144 | | -sudo journalctl -f -u nginx --no-hostname |
145 | | -# The API logs contain SQL queries, exceptions etc |
146 | | -sudo journalctl -f --identifier gunicorn3 --no-hostname |
147 | | -``` |
148 | | - |
149 | | -### Fastpath runbook |
150 | | - |
151 | | -#### Manual deployment |
152 | | - |
153 | | -```bash |
154 | | -ssh <host> |
155 | | -sudo apt-get update |
156 | | -apt-cache show fastpath | grep Ver | head -n5 |
157 | | -sudo apt-get install fastpath |
158 | | -``` |
159 | | - |
160 | | -#### Restart |
161 | | -`sudo systemctl restart fastpath` |
162 | | - |
163 | | -#### Rerun fastpath manually |
164 | | - |
165 | | -Run as fastpath user: |
166 | | - |
167 | | -```bash |
168 | | -ssh <host> |
169 | | -sudo sudo -u fastpath /bin/bash |
170 | | -cd |
171 | | -``` |
172 | | - |
173 | | -```bash |
174 | | -fastpath --help |
175 | | -# rerun without overwriting files on disk nor writing to database: |
176 | | -fastpath --start-day 2016-05-13 --end-day 2016-05-14 --stdout --no-write-msmt --no-write-to-db |
177 | | -# rerun without overwriting files on disk: |
178 | | -fastpath --start-day 2016-05-13 --end-day 2016-05-14 --stdout --no-write-msmt |
179 | | -# rerun and overwrite: |
180 | | -fastpath --start-day 2016-05-13 --end-day 2016-05-14 --stdout --update |
181 | | -``` |
182 | | - |
183 | | -The fastpath will pull cans from S3. |
184 | | -The daemon (doing real-time processing) can keep running in the meantime. |
185 | | - |
186 | | -[Progress chart](https://mon.ooni.nu/prometheus/new/graph?g0.expr=netdata_statsd_gauge_fastpath_s3feeder_s3_download_percentage_value_average%7Bdimension%3D%22gauge%22%7D&g0.tab=0&g0.stacked=1&g0.range_input=2h&g1.expr=netdata_statsd_gauge_fastpath_load_s3_reports_remaining_files_value_average%7Bdimension%3D%22gauge%22%7D&g1.tab=0&g1.stacked=1&g1.range_input=1h) |
187 | | -#### Log monitoring |
188 | | - |
189 | | -```bash |
190 | | -sudo journalctl -f -u fastpath |
191 | | -``` |
192 | | - |
193 | | -#### Monitoring dashboard |
194 | | - |
195 | | -[https://mon.ooni.nu/grafana/d/75nnWVpMz/fastpath-ams-pg?orgId=1&refresh=5m&from=now-7d&to=now](https://mon.ooni.nu/grafana/d/75nnWVpMz/fastpath-ams-pg?orgId=1&refresh=5m&from=now-7d&to=now) |
196 | | - |
197 | | -### Analysis runbook |
198 | | - |
199 | | -The Analysis tool runs a number of systemd timers to monitor the slow query summary and more. |
200 | | -See https://github.com/ooni/pipeline/blob/master/af/analysis/analysis/analysis.py |
201 | | - |
202 | | -#### Manual deployment |
203 | | - |
204 | | -``` |
205 | | -ssh <host> |
206 | | -sudo apt-get update |
207 | | -apt-cache show analysis | grep Ver | head -n5 |
208 | | -sudo apt-get install analysis=<version> |
209 | | -``` |
210 | | - |
211 | | -#### Run manually |
212 | | -``` |
213 | | -sudo systemctl restart ooni-update-counters.service |
214 | | -``` |
215 | | - |
216 | | -#### Log monitoring |
217 | | - |
218 | | -``` |
219 | | -sudo journalctl -f --identifier analysis |
220 | | -``` |
221 | | - |
222 | | -#### Monitoring dashboard |
223 | | - |
224 | | -[https://mon.ooni.nu/grafana/d/75nnWVpMz/fastpath-ams-pg?orgId=1&refresh=5m&from=now-7d&to=now](https://mon.ooni.nu/grafana/d/75nnWVpMz/fastpath-ams-pg?orgId=1&refresh=5m&from=now-7d&to=now) |
225 | | - |
226 | | -### Deploy new host |
227 | | - |
228 | | -Deploy host from https://cloud.digitalocean.com/projects/ |
229 | | - |
230 | | -Create DNS "A" record `<name>.ooni.org` at https://ap.www.namecheap.com/ |
231 | | - |
232 | | -On the sysadmin repo, ansible directory, add the host to the inventory |
233 | | - |
234 | | -Run the deploy with the root SSH user |
235 | | -``` |
236 | | -./play deploy-<foo>.yml -l <name>.ooni.org --diff -u root |
237 | | -``` |
238 | | - |
239 | | -Update prometheus |
240 | | -``` |
241 | | -./play deploy-prometheus.yml -t prometheus-conf --diff |
242 | | -``` |
| 5 | +For up to date documentation about the backend, see: |
| 6 | +https://docs.ooni.org/backend |
0 commit comments