Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -1957,6 +1957,7 @@
"tools/toolkits/web-scrape/jina-reader",
"tools/toolkits/web-scrape/newspaper",
"tools/toolkits/web-scrape/newspaper4k",
"tools/toolkits/web-scrape/olostep",
"tools/toolkits/web-scrape/spider",
"tools/toolkits/web-scrape/trafilatura",
"tools/toolkits/web-scrape/website",
Expand Down
63 changes: 63 additions & 0 deletions tools/toolkits/web-scrape/olostep.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: Olostep
description: Use Olostep with Agno to scrape, crawl, search and get AI-powered answers from the web.
---

## Overview

OlostepTools enable an Agent to scrape websites, crawl entire sites, discover URLs, run web searches, and get AI-powered answers grounded in live data using the [Olostep](https://www.olostep.com) web data API.

## Prerequisites

The following example requires the `olostep` library and an API key which can be obtained from [Olostep](https://www.olostep.com/dashboard/api-keys).
```shell
pip install -U olostep
```

## Example

The following agent will scrape the content from `https://docs.olostep.com/get-started/welcome` and return a summary:
```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.olostep import OlostepTools

agent = Agent(
model=OpenAIChat(id="gpt-4o-mini"),
tools=[OlostepTools(scrape_url=True)],
markdown=True,
)
agent.print_response(
"Summarize the key features at https://docs.olostep.com/get-started/welcome"
)
```

## Toolkit Params

| Parameter | Type | Default | Description |
| ---------------- | ---- | ------- | ------------------------------------------------------------------ |
| api_key | str | None | Olostep API key. Falls back to OLOSTEP_API_KEY env var. |
| scrape_url | bool | True | Enable single URL scraping. |
| crawl_website | bool | False | Enable website crawling. |
| map_website | bool | False | Enable URL discovery / site mapping. |
| search_web | bool | False | Enable web search returning structured links. |
| answer_question | bool | False | Enable AI-powered answers grounded in live web data. |
| batch_scrape | bool | False | Enable concurrent batch scraping of multiple URLs. |
| all_tools | bool | False | Enable all tools at once. |

## Toolkit Functions

| Function | Description |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------- |
| scrape_url | Scrape a single URL and return its content as markdown, html, text, or structured JSON. Supports parsers and LLM extraction. |
| crawl_website | Recursively crawl a website starting from a URL. Supports URL glob filtering, depth limits, and relevance search. |
| map_website | Discover all URLs on a website from sitemaps and discovered links. |
| search_web | Search the web with a natural language query and return ranked links with titles and descriptions. |
| answer_question | Search the web and return an AI-synthesized answer grounded in live data with source citations. Supports structured JSON output. |
| batch_scrape | Scrape multiple URLs concurrently in a single batch job. Up to 10,000 URLs, completes in ~5–8 minutes. |

## Developer Resources

- [Olostep Website](https://www.olostep.com)
- [Olostep Documentation](https://docs.olostep.com)
- [Python SDK](https://docs.olostep.com/sdks/python)