Skip to content

Generators that allow for multiple requests and websockets #1291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

iamnotcj
Copy link
Contributor

This change adds support for sending multiple request/response pairs as well as websockets. This was influenced by a couple of web app implementations our practice has seen over the past couple of months.

MultiRestGenerator:

There are times where the initial request with the user input doesn't immediately return a generated output. Instead another request has to be made to a specified endpoint to get the results of the input. For example, one implementation we have come across had the user send an input and a query id was returned. The succeeding request used this query id as a value for a GET parameter. The response to this GET request then contained the generated output.

It also relies on Burpsuite XML files, as Burpsuite is one of the most popular web application testing tool. This might be a stepping stone into adding garak as a dedicated burp extension some time in the future?

WebSocketGenerator:

There have been times where chatbots would communicate with websockets. I've only encountered pretty basic implementations of this, which is why this one is pretty rudimentary

iamnotcj added 4 commits June 10, 2025 09:37
Added WebSocket generator and a Rest generator that can execute multiple requests.

None should run properly
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are both patterns the project is interested in getting support for. Please see #878 and the conversations about needing some publicly available application stack to test against to maintain official generators.

Previous comments such as this one offered patterns for possibly lifting that class into a generic class configurable via --generator_options_file. The implementation here based on burp exported XML looks reasonable. I would be interested in opinions on if it can support the use cases described in the previous PR's comments.

@iamnotcj
Copy link
Contributor Author

These are both patterns the project is interested in getting support for. Please see #878 and the conversations about needing some publicly available application stack to test against to maintain official generators.

I took a look at that previous pull request, I'm not entirely sure if there is anything publicly available, but let me check back on this we might have something. If not, an idea I had was testing it with multiple endpoints with high availability (like test if you can reach google first and then subsequently make a request to wikipedia).

Previous comments such as this one offered patterns for possibly lifting that class into a generic class configurable via --generator_options_file. The implementation here based on burp exported XML looks reasonable. I would be interested in opinions on if it can support the use cases described in the previous PR's comments.

I think it can with a few edits. The generator creates its own data structure for requests and responses from an external file. I just used burpsuite because its popular to proxy requests and save them from there. I think it would be a function similar to get_reqrep_fromburp where it just gets request and responses from the config rather than a file.

I should be able to send another commit with those changes requested sometime before the end of the week. 👍

@iamnotcj
Copy link
Contributor Author

Okay, I just went ahead and re wrote it. It should support manually putting the requests and stuff in a config file, I followed the discourse in that other pull request and tried to make it as close as the last suggestion you gave on that thread. The format of the config can be changed, I'm open to all suggestions.

I also moved that example file to the directory you specified as well.

@iamnotcj iamnotcj requested a review from jmartin-tech July 19, 2025 15:58
Copy link
Collaborator

@erickgalinkin erickgalinkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to do some testing on an arbitrary websocket/streaming generator in real life to validate but this looks ok to me.

Comment on lines 51 to 53



Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should apply black on this file just for cleanup. I'd consider this a nit but others may consider the code style more important.

requests_mock.get(url2, headers={"Content-Type": "application/json"}, status_code=400, json={"message": "error details"})

output = mr_gen_burpfile_config.run("Hello")
assert output == "error details"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Newline at end of file

@iamnotcj
Copy link
Contributor Author

For adding it to the rest package, would something like this work?

garak/generators
├── __init__.py
├── __pycache__
...truncated...
├── rest
│   ├── __init__.py
│   ├── __pycache__
│   ├── multirest.py
│   ├── rest.py
│   └── wsocket.py
...truncated...

And then for rest/__init__.py we import the classes from the respective files:

from garak.generators.rest.rest import RestGenerator
from garak.generators.rest.multirest import MultiRestGenerator
from garak.generators.rest.wsocket import WebSocketGenerator

DEFAULT_CLASS = "RestGenerator"

@jmartin-tech
Copy link
Collaborator

I don't think using rest/__init__.py in that way would currently work with the plugin enumeration structure. We can think about how to make that better in a future revision.

@iamnotcj
Copy link
Contributor Author

Alright, I just went ahead and moved them into rest.py.

iamnotcj and others added 3 commits July 25, 2025 09:17
Co-authored-by: Jeffrey Martin <[email protected]>
Signed-off-by: CJ Anih <[email protected]>
Co-authored-by: Jeffrey Martin <[email protected]>
Signed-off-by: CJ Anih <[email protected]>
Co-authored-by: Jeffrey Martin <[email protected]>
Signed-off-by: CJ Anih <[email protected]>
@iamnotcj iamnotcj requested a review from jmartin-tech August 1, 2025 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants