Open Deep Research blog #2644
Merged
Open Deep Research blog #2644
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Albert Villanova del Moral <[email protected]>
craffel
reviewed
Feb 4, 2025
open-deep-research.md
Outdated
|
|
||
| ## TLDR | ||
|
|
||
| Yesterday, OpenAI released [Deep Research](https://openai.com/index/introducing-deep-research/), a system that browses the web to summarize content and answer questions based on the summary. The system is impressive and blew our mind when we tried it for the first time. |
open-deep-research.md
Outdated
| One of the main results in the blog post is a strong improvement of performances on the [General AI Assistants benchmark (GAIA)](https://huggingface.co/gaia-benchmark), a benchmark we’ve been playing with recently as well, where they successfully reached near 67% correct answers on 1-shot on average, and 47.6% on especially challenging “level 3” questions that involve multiple steps of reasoning and tool usage (see below for a presentation of GAIA). | ||
|
|
||
|
|
||
| DeepResearch is composed of an LLM (which can be selected from the current list of LLM provided by OpenAI, 4o, o1, o3, etc) and an internal “agentic framework” which guide the LLM to use tools like web search and organize its actions in steps. |
open-deep-research.md
Outdated
|
|
||
| DeepResearch is composed of an LLM (which can be selected from the current list of LLM provided by OpenAI, 4o, o1, o3, etc) and an internal “agentic framework” which guide the LLM to use tools like web search and organize its actions in steps. | ||
|
|
||
| While powerful LLM are now freely available in open-source (see e.g. [the recent DeepSeek R1 model](https://huggingface.co/deepseek-ai/DeepSeek-R1)), OpenAI didn’t disclose much about the agentic framework underlying Deep Research… |
|
|
||
| So we decided to embark on a mission to reproduce their results and open-source the needed framework along the way! | ||
|
|
||
| The clock is ticking, let’s go! ⏱️ |
There was a problem hiding this comment.
You didn't mention the 24-hour thing so this clock reference is kind of a non-sequitur
There was a problem hiding this comment.
Oh i did but someone removed it
open-deep-research.md
Outdated
|
|
||
| Now we need to provide the agent with the right set of tools. | ||
|
|
||
| **1.** A web browser. While a fully fledged web browser interaction like [Operator](https://openai.com/index/introducing-operator/) will be needed to reach full performance, we started with an extremely simple text-based web browser for now for our first PoC. You can find the code [here](https://github.com/huggingface/smolagents/blob/gaia-submission-r1/examples/open_deep_research/scripts/text_web_browser.py) |
open-deep-research.md
Outdated
|
|
||
| Here is a short roadmap of improvements which we feel would really improve these tools’ performance (feel free to open a PR and contribute!): | ||
|
|
||
| - extending the number of file-formats which can be read. |
open-deep-research.md
Outdated
|
|
||
| ## Results 🏅 | ||
|
|
||
| In our 24h+ reproduction sprint, we’ve already seen steady improvements in the performances of our agent on GAIA! |
open-deep-research.md
Outdated
|
|
||
| In our 24h+ reproduction sprint, we’ve already seen steady improvements in the performances of our agent on GAIA! | ||
|
|
||
| We’ve quickly gone up from the previous SOTA with an open framework, around 46% for Magentic-One, to our current performances of 54% on the validation set. |
There was a problem hiding this comment.
SOTA -> SoTA if you want to be pedantic
open-deep-research.md
Outdated
|
|
||
| We’ve quickly gone up from the previous SOTA with an open framework, around 46% for Magentic-One, to our current performances of 54% on the validation set. | ||
|
|
||
| This bump in performance is due mostly to letting our agents write their actions in code! Indeed, when switching to a standard agent that writes actions in Json instead of code, performance of the same setup is instantly degraded to 33% average on the validation set. |
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Albert Villanova del Moral <[email protected]>
albertvillanova
approved these changes
Feb 4, 2025
Co-authored-by: Albert Villanova del Moral <[email protected]>
Co-authored-by: Aymeric Roucher <[email protected]>
|
should we merge @aymeric-roucher |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cc @aymeric-roucher @thomwolf @clefourrier @albertvillanova @NathanHB you can check from hf.co/new-blog