feat: add vendor-neutral web corpus loader

[sachasimov]

Vendor-neutral rework of #10, which was closed because it wired one commercial provider into the public API, dependency set, and marketing copy. This addresses each point you raised.

What changed since #10

• No brand in the public API. The public class is now WebLoader (was LinkupWebLoader), built against a small provider-agnostic WebFetchClient interface (fetch(url, ...) -> response). Nothing vendor-named is exported from the package.
• No vendor baked in / no default backend. WebLoader takes any client you supply. The config format is renamed linkup_fetch -> web_fetch and requires either an injected client or an explicitly named backend — there is no default provider.
• Specific backends are clearly-optional, pluggable adapters. They live in knowledge_base/web_adapters.py. The Linkup adapter is kept as one example, installable via the optional [linkup] extra, and is never used unless explicitly selected.
• No marketing language. Removed all promotional/endorsement copy from README.md and DOCUMENTATION.md; the example config is renamed web_fetch_config.toml with neutral wording.

Net effect for users

You can build a retrieval corpus from known public web pages through whatever fetch backend you choose, with the package's default path staying fully offline and dependency-light. No third-party SaaS is required by core, and none is endorsed.

Test plan

• Full suite passes: pytest (59 passed).
• ruff check clean.
• Public API no longer exposes any vendor name (WebLoader only).

Made with Cursor

[opensource-SantanderAI]

Thanks @sachasimov — this is a big step in the right direction and we really appreciate the rework. We're happy to accept the architecture: the vendor-agnostic WebFetchClient interface, the WebLoader (no brand in the public API), the no-default/inject-your-own-client design, and the removal of marketing copy. That decoupling is exactly what we wanted.

However, the maintainer requirement for merge is that the repository be 100% vendor-neutral, with no branded backend shipped in code or dependencies. Right now a vendor still remains as the only concrete backend:

• knowledge_base/web_adapters.py registers exactly one backend — _BACKENDS = {"linkup": linkup_fetch_client} — so the single example we ship is brand-named, which reads as a de-facto endorsement.
• pyproject.toml adds a real vendor dependency via the [linkup] extra (linkup-sdk>=0.18), i.e. supply-chain surface for one commercial provider.

A branded backend gives no technical advantage here: any generic HTTP client (stdlib urllib/httpx/requests) — or simply "bring your own client / use your agent's fetch tool" — demonstrates the interface equally well, vendor-free.

Please do one of the following:

1. (Preferred) Replace the Linkup adapter with a neutral example backend (e.g. a small stdlib-/httpx-based WebFetchClient), and drop the [linkup] extra and linkup-sdk dependency. Linkup (and any other provider) can be mentioned in the docs as one option a user could plug in, but not shipped as code or a dependency.
2. Or ship no concrete backend at all — keep only the WebFetchClient interface plus "bring your own client" documentation, and remove the [linkup] extra/dependency entirely.

A couple of notes while you're in there:

• pyproject.toml has a duplicate line: linkup-sdk>=0.18 appears twice in the [linkup] extra — please remove the duplicate (this would go away entirely with the changes above).
• Once a neutral backend is in place, please make sure the example config (examples/web_fetch_config.toml) and tests/test_web_knowledge_base.py reference the neutral backend rather than a brand name.

With a brand-free backend (or none) the repo is fully vendor-neutral and we'll be glad to merge. Thanks again for working with us on this!

[opensource-SantanderAI]

Thanks for the vendor-neutral rework — this addresses the #10 concerns well.

We verified the diff and confirmed there is no commercial provider left in the code, dependencies, or docs: the public API exposes only WebLoader against the WebFetchClient protocol, and the sole registered backend is the dependency-free stdlib http one. A few things before we can merge:

1. CI (black --check) fails on linear_adapter_trainer/config.py (escaped \"http\" where black wants single quotes). Please run black . and push.
2. PR description vs. code: the description says the Linkup adapter is kept as an optional [linkup] extra, but there is no Linkup adapter or [linkup] extra in the diff (and pyproject.toml is untouched). Please update the description to match — the code is actually cleaner than described. Consider renaming the branch (feat/linkup-web-corpus-loader) too, for clarity.
3. Optional hardening: HttpWebFetchClient uses urllib.urlopen without restricting the URL scheme, so a file:// URL would read a local file. Since URLs are operator-supplied this is low risk, but restricting to http/https would be a nice safeguard.
4. Please rebase on main (the branch is behind).

With the black fix the rest is good to go. Thanks again!