-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathREADME.Rmd
318 lines (244 loc) · 14.5 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/",
out.width = "100%",
fig.align = "center"
)
```
# captcha <img src="man/figures/hex_small.png" align="right" />
<!-- badges: start -->
[](https://lifecycle.r-lib.org/articles/stages.html#stable)
[](https://app.codecov.io/gh/decryptr/captcha?branch=master)
[](https://github.com/decryptr/captcha/actions/workflows/R-CMD-check.yaml)
[](https://CRAN.R-project.org/package=captcha)
[](https://joss.theoj.org/papers/027aba13cdc52465145959e84637bd56)
<!-- badges: end -->
This package is an extensible API to build models and solve Captchas (Completely Automated Public Turing test to tell Computers and Humans Apart). It provides tools to read Captchas, visualize Captchas, annotate Captchas, fit models, and share fitted models.
## Installation
One can install the development version of `{captcha}` package running
``` r
remotes::install_github("decryptr/captcha")
```
## Basic usage
The basic usage of `{captcha}` involves the functions `read_captcha()`, `plot()`, `captcha_annotate()`, `captcha_load_model()` and `decrypt()`. The diagram below summarizes the relationships between these functions. The arrows indicate the dependency of functions on objects generated by other functions.
```{r}
#| label: fig-captcha-diagram-simple
#| fig-align: center
#| echo: false
knitr::include_graphics("man/figures/diagram.png")
```
### Read and visualize
The `read_captcha()` function reads a character vector of image files and stores them in memory. Behind the scenes, the function uses the [`{magick}`](https://docs.ropensci.org/magick/) package to deal with the types of files that may appear (JPEG, PNG, among others).
```{r}
#| label: fig-example-plot-inicial
#| echo: true
#| out-width: 30%
library(captcha)
example <- "man/figures/dados_tjmg.jpeg"
captcha <- read_captcha(example)
captcha
```
The function returns an object of class `captcha`, which can be used by other methods.
```{r}
#| echo: true
class(captcha)
```
The captcha object is a list with three elements: `$img`, which contains the image read from the `{magick}` package; `$lab`, which contains the image label (by default, `NULL`); and `$path`, which contains the path of the image.
```{r}
#| echo: true
str(captcha)
```
The `read_captcha()` function has an `lab_in_path=` parameter, which indicates whether the image path contains the label. If `lab_in_path=TRUE`, the function will try to extract the label from the file (getting the text that comes after the last `_` of the path) and store it in the `$lab` element.
```{r}
#| echo: true
example <- "man/figures/mnist128c49c36e13_6297.png"
captcha <- read_captcha(example, lab_in_path = TRUE)
str(captcha)
```
The `plot()` function is a method of [class S3](https://adv-r.hadley.nz/s3.html) from base R. The function facilitates the visualization of Captchas. The function receives a list of images (obtained with the `read_captcha()` function) and displays the Captcha visually.
```{r}
#| label: fig-example-plot
#| echo: true
#| out-width: 30%
example <- "man/figures/dados_tjmg.jpeg"
captcha <- read_captcha(example)
plot(captcha)
```
An interesting aspect of the `plot()` function is that it deals with a list of Captchas. It is useful when the goal is to view several Captchas in the image simultaneously. The next image shows an example.
```{r}
#| label: fig-example-plot-multi
#| echo: true
#| out-width: 70%
#| out-height: 100px
examples <- paste0("man/figures/", c(
"dados_tjmg.jpeg",
"dados_esaj.png",
"dados_rfb.png",
"dados_sei.png"
))
captchas <- read_captcha(examples)
plot(captchas)
```
By default, the `plot()` function arranges the images into four columns. To change the default, one can modify the options using `options(captcha.print.cols = N)`, where `N` is the desired number of columns. The next image shows an example with two columns.
```{r}
#| echo: false
# backup option (hidden)
op <- getOption("captcha.print.cols")
```
```{r}
#| echo: true
#| label: fig-example-plot-multi-2col
#| out-width: 60%
options(captcha.print.cols = 2)
plot(captchas)
```
```{r}
#| echo: false
# back to original option (hidden)
options(captcha.print.cols = op)
```
When the list of Captchas is too long, the `plot()` function displays a maximum number of images accompanied by a message. By default, this number is 100, with 25 rows and four columns. One can override the option by combining the `captcha.print.cols=` and `captcha.print.rows=` options. The next image shows an example of the function’s behavior when the number of images exceeds 100.
```{r}
#| label: fig-example-plot-multi-many
#| message: true
#| echo: true
# more than 100 imagens:
examples <- rep("man/figures/dados_tjmg.jpeg", 110)
captchas <- read_captcha(examples)
plot(captchas)
```
It is possible to create subsets of `captcha` objects using the `[` operator. One can also use the `length()` function to measure the number of images. The next image shows an example of these operations.
```{r}
#| label: fig-example-plot-multi-many-subset
#| echo: true
captchas_subset <- captchas[1:20]
length(captchas_subset) # 20
plot(captchas_subset)
```
Finally, when the image has a label, the `plot()` function shows the label on the corner of the image. The following image shows an example.
```{r}
#| label: fig-example-plot-rotulado
#| echo: true
#| out-width: 30%
example <- "man/figures/mnist128c49c36e13_6297.png"
captcha <- read_captcha(example, lab_in_path = TRUE)
plot(captcha)
```
### Annotate
The `captcha_annotate()` function annotates a Captcha image, either manually or automatically. It modifies the image path and adds the text `_label` to the end of the file path. The function has the following parameters:
- `files=`: object of class captcha read with function `read_captcha()` (recommended) or character vector of file paths.
- `labels=`: (optional) character vector with image labels. It must have the same `length()` as `files=`. The value is `NULL` by default, prompting the user to enter the label manually.
- `path=`: (optional) path of the folder to save the annotated files. By default, it saves files with modified names in the same folder as the original files.
- `rm_old=`: (optional) whether or not to delete the original files. Defaults to `FALSE`.
The `captcha_annotate()` function returns a vector with the paths of the modified files. The `labels=` parameter can handle situations where one knows the Captcha label. For example, a workflow that uses an oracle might provide the label automatically.
When the label doesn't exist, the `captcha_annotate()` function opens the prompt for classification and shows the image using `plot()`. The following image shows an application example of the `captcha_annotate()` function in [RStudio](https://posit.co/download/rstudio-desktop/).
```{r}
#| label: fig-example-classify
#| echo: false
#| out-width: 70%
knitr::include_graphics("man/figures/exemplo_classify.png")
```
### Predict
The `decrypt()` function returns a label for an image using a fitted model. The function takes two arguments: `file=`, which can be either the file path or a captcha class object, and `model=`, which contains an object of class luz_module_fitted, fitted using the [{luz}](https://mlverse.github.io/luz/) package.
```{r}
#| label: fig-example-decrypt
#| out-width: 30%
#| message: false
model <- captcha_load_model("cadesp")
img <- "man/figures/dados_cadesp.jpg"
captcha <- read_captcha(img)
plot(captcha)
decrypt(captcha, model)
```
There are fitted models for several different Captchas available in the captcha package. It is possible to load a trained model using the `captcha_load_model()` function. The `path=` parameter contains either the path for a fitted model or a string with the name of a released model, like `"rfb"`. Fitted models are stored in the `{captcha}` package repository releases, which can be downloaded using the [`{piggyback}` package](https://docs.ropensci.org/piggyback/) package.
Currently, the Captchas with available fitted models are `trf5`, `tjmg`, `trt`, `esaj`, `jucesp`, `tjpe`, `tjrs`, `cadesp`, `sei` and ` rfb`. The table below describes the models and their accuracy.
```{r}
#| label: tbl-available-models
#| message: false
#| echo: false
av_models <- captcha_available_models()
description <- c(
"Centro de Apoio ao Desenvolvimento da Saúde Pública",
"Tribunal de Justiça da Bahia",
"Junta Comercial de São Paulo",
"Receita Federal",
"Sistema Eletrônico de Informações - ME",
"Tribunal de Justiça de Minas Gerais",
"Tribunal de Justiça de Pernambuco",
"Tribunal de Justiça do Rio Grande do Sul",
"Tribunal Regional Federal 5",
"Tribunal Regional do Trabalho 3"
)
acc <- purrr::map_dbl(
av_models,
\(x) {
model <- captcha_load_model(x)
dplyr::last(model$records$metrics$valid)[["captcha acc"]]
}
) |>
purrr::set_names(av_models) |>
tibble::enframe()
fs::dir_ls("man/figures", regexp = "dados") |>
tibble::enframe("name", "path") |>
dplyr::mutate(
name = basename(tools::file_path_sans_ext(name)),
name = stringr::str_remove(name, "dados_"),
img = stringr::str_glue("")
) |>
dplyr::inner_join(acc, "name") |>
dplyr::transmute(
name,
img,
desc = description,
acc = formattable::percent(value, 2)
) |>
knitr::kable(
col.names = c("Name", "Example", "Description", "Accuracy")
)
```
### Fit custom model
The `{captcha}` package provides a basic interface for fitting custom models from a fully labeled data. Annotation can be done manually using the `captcha_annotate()` function presented earlier or with another method developed by the user. The model uses a convolutional neural network architecture, similar to the [LeNet-5 model](https://en.wikipedia.org/wiki/LeNet).
The modeling step has some assumptions about the file names. Images must be in a folder and have the pattern `path/to/file/<id>_<lab>.<ext>`, where:
- `<id>`: can be any name, preferably without accents or other special characters, to avoid encoding issues. It usually contains a name for the type and a hash to identify the image uniquely. __Note__: When annotating a file, the id must be unique, as two Captchas can have the same label.
- `<lab>`: is the Captcha label. It is a string of characters between `[a-zA-Z0-9]`, which can be case-sensitive if necessary. All labels must have the same length.
- `<ext>`: file extension. It can be `.png`, `.jpeg` or `.jpg`. The operations also work for the `.svg` format, but it may have problems due to the image's transparency.
The `captcha_fit_model()` function fits a model from a folder with annotated images. The function considers the parameters:
- `dir=`: the path of the annotated files;
- `dir_valid=`: (optional) the path of annotated files for validation;
- `prop_valid`=:, the proportion of the training set to be considered as validation. `prop_valid=` is ignored when `dir_valid=` is given (by default, 20% of the dataset is considered as validation).
The `captcha_fit_model()` function also has some parameters related to the neural network. They are:
- `dropout=`: the dropout percentage applied to the hidden layers of the network (by default, 0.25);
- `dense_units=`: the number of units in the hidden layer that comes after the convolutional layers (by default, 200);
- `decay=`: the learning rate decay percentage (by default, 0.99);
- `epochs=`: the number of epochs to fit the model (by default, 100).
An important note is that the model stops fitting after 20 iterations without significant increment of accuracy (chosen as 1%; for more details, see the [advanced guide](https://decryptr.github.io/captcha/articles/advanced.html)).
The function returns a fitted model of class `luz_module_fitted`, which can be saved to disk using `luz_save()`. One can also serialize the model to use it with other packages such as PyTorch. A tutorial on serialization can be found in the [torch package documentation](https://torch.mlverse.org/docs/articles/serialization.html).
Fitting a new Captcha model can be challenging. To help with that, the `{captcha}` package documentation provides an [application example](). The example uses Captchas from `captcha_generate()` function, which generates Captchas using the `{magick}` package. The `captcha_generate()` function has the following parameters:
- `write_disk=`: save files to disk? By default, `FALSE`.
- `path=`: Path to save files on disk, if the previous parameter is `TRUE`.
- `chars=`: Which characters to use in the image.
- `n_chars=`: The length of the Captcha.
- `n_rows=`: Height of the image, in pixels.
- `n_cols=`: Width of the image, in pixels.
- `p_rotate=`: Probability of image rotation.
- `p_line=`: Probability of adding a line between the letters.
- `p_stroke=`: Probability of adding a border to the letters.
- `p_box=`: Probability of adding a box (rectangle) around the letters.
- `p_implode=`: Probability of adding implode effects.
- `p_oilpaint=`: Probability to add oil paint effects.
- `p_noise=`: Probability of adding white noise to the background of the image.
- `p_lat=`: Probability of applying the *local adaptive thresholding* algorithm to the image.
## Advanced usage
Once the premises of the classified base are met, it is possible to fit a neural network model using the `{captcha}` package. However, the training step involves many small adaptations, it was decided to export functions in two depth levels. To address that, the `{captcha}` package also provides a **procedural** approach to fit the model, using a step-by-step described in the [advanced guide](https://decryptr.github.io/captcha/articles/advanced.html).
## Contributing
One can contribute to the package through [pull requests](https://github.com/decryptr/captcha/pulls) and report problems or ask questions at the [issues section](https://github.com/decryptr/captcha/issues). See the CONTRIBUTING.md file for detailed instructions.
## Code of Conduct
Please note that the captcha project is released with a [Contributor Code of Conduct](https://decryptr.github.io/captcha/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.
## License
MIT