Skip to content

Fix JsoupDocumentReader dropping relative link URLs - #6803

Open
wantaekchoi wants to merge 1 commit into
spring-projects:mainfrom
wantaekchoi:fix-jsoup-relative-link-base-uri
Open

Fix JsoupDocumentReader dropping relative link URLs#6803
wantaekchoi wants to merge 1 commit into
spring-projects:mainfrom
wantaekchoi:fix-jsoup-relative-link-base-uri

Conversation

@wantaekchoi

Copy link
Copy Markdown

Problem

JsoupDocumentReader parses its resource with an empty base URI:

org.jsoup.nodes.Document doc = Jsoup.parse(inputStream, this.config.charset, "");

Link extraction then asks jsoup for the absolute form of each href:

List<String> linkUrls = links.stream().map(link -> link.attr("abs:href")).toList();

abs: resolves against the document base URI. With an empty base there is nothing to resolve against, so a relative link yields an empty string rather than a URL. Reading a page that contains <a href="guide.html"> with includeLinkUrls enabled produces this metadata:

linkUrls = ["", "https://spring.io/"]

The relative link is not shortened or left relative. It is gone, and nothing reports it.

Two places state the opposite behavior. The reference documentation, under the HTML (JSoup) reader in etl-pipeline.adoc:

The base URI, for resolving relative links, will be extracted from URL resources.

And the module README, which lists as a feature:

Extract a list of all absolute URLs of links (<a href="...">) within the document.

Commit 82b46d2 introduced the reader, that README line, and that documentation line together. The base URI has been empty since.

Changes

Take the base URI from the resource:

private String resolveBaseUri() {
    try {
        return this.htmlResource.getURL().toString();
    }
    catch (IOException ex) {
        return "";
    }
}

Resources that do not resolve to a URL, ByteArrayResource among them, keep the empty base and behave exactly as before. TextReader already treats a resource this way, catching IOException from getURL() when the resource has no URL to give.

Absolute links are unaffected, since jsoup returns those unchanged whatever the base is.

This is the only Jsoup.parse call in the repository, so no sibling reader carries the same omission.

Testing

testWithRelativeLinkUrls reads a classpath resource holding one relative and one absolute link, then asserts that no entry in linkUrls is empty and that the relative link resolved against the resource. Without the change it fails on the empty entry:

Expecting no elements of:
  ["", "https://spring.io/"]
to match given predicate but this element did:
  ""

The existing ByteArrayResource tests cover the no-URL path and still pass.

./mvnw -pl document-readers/spring-ai-jsoup-document-reader test

passes with 12 tests, 1 skipped, the skip being the pre-existing test that needs an internet connection.

`JsoupDocumentReader` parses with an empty base URI, so jsoup cannot
resolve a relative `href` and `abs:href` yields an empty string. With
`includeLinkUrls` enabled, every relative link lands in the `linkUrls`
metadata as "".

The reference documentation states that the base URI for resolving
relative links is taken from URL resources, and the module README
promises absolute URLs for every link in the document. Take the base
URI from the resource, keeping the empty one for resources that do not
resolve to a URL.

Signed-off-by: wantaek <wantaekchoi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants