URI / IRI checking should complain about ASCII control chars and maybe properly check URIs

[joernhees]

currently the _is_valid_URI method only checks if a URI includes any of the not allowed printable ASCII chars (see #620)

ASCII control chars like backspace, tab, ... won't lead to an early error / warning.

maybe we should really try to parse URIs with a python builtin lib or maybe https://pypi.python.org/pypi/rfc3987 ?

[jpmccu]

There's an additional problem that IRIs aren't being handled properly either, at least in the SPARQL store:

XMLSyntaxError: xmlns:ns30: 'http://ja.dbpedia.org/resource/セントラル・' is not a valid URI, line 45, column 52 (line 45)

[reeshabhranjan]

@joernhees me, @guvi007 and @prince17080 would like to work on this issue, can we go forward with this?

[prince17080]

@joernhees Hey, we (me, @reeshabhkumarranjan, @guvi007) want to solve this issue. We came up with the solution that a valid URI can only contain characters among 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;='. If you think this is correct, I (@prince17080) can submit a pull request for the same.

[jpmccu]

This approach will exclude IRIs, which I think we are trying to support.

On Mon, May 25, 2020 at 2:30 PM prince17080 ***@***.***> wrote: @joernhees <https://github.com/joernhees> Hey, we (me, @reeshabhkumarranjan <https://github.com/reeshabhkumarranjan>, @guvi007 <https://github.com/guvi007>) want to solve this issue. We came up with the solution that a valid URI can only contain characters among 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;='. If you think this is correct, I ***@***.*** <https://github.com/prince17080>) can submit a pull request for the same. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#625 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAETCEIVQBR36HREDD2IBP3RTK2MJANCNFSM4CDMMQCA> .

-- Jim McCusker Director, Data Operations Tetherless World Constellation Rensselaer Polytechnic Institute mccusj2@rpi.edu <mccusj@cs.rpi.edu> http://tw.rpi.edu

[tgbugs]

Last I checked is_valid_uri is one of the most time consuming checks that is run by rdflib. I second @joernhees's original suggestion to use urlparse for this, but whatever implementation is ultimately used we will need performance numbers.

[CrypticGuy]

Hi @joernhees @jpmccu @tgbugs,
I, @UditArora2000 and @Ritvik1712 would like to address this issue. We are planning to work on some of the issues of the library as part of our course on Semantic Web. We think that instead of urlparse function present in inbuilt python urllib, the method mentioned by @joernhees in the initial issue would work better.
We propose to use the match function as part of https://pypi.python.org/pypi/rfc3987 and then compare the possible matches based on the format type URI_reference or IRI_referecne https://github.com/dgerber/rfc3987/blob/master/rfc3987.py would be able to do the work.
As for the last comment on performance number we would be glad to run the analysis and perform the needful.

Hm, not quite as straightforward as it might first appear. fwiw, I decanted rfc3987 into its own file in rdflib/extras, added a small switch to rdflib/term.py:

diff --git a/rdflib/term.py b/rdflib/term.py
index c0ea4a28..44a919ff 100644
--- a/rdflib/term.py
+++ b/rdflib/term.py
@@ -70,6 +70,7 @@ from isodate import (
 
 import rdflib
 from rdflib.compat import long_type
+from rdflib.extras import rfc3987
 
 if TYPE_CHECKING:
     from .namespace import NamespaceManager
@@ -81,15 +82,26 @@ rdflib_skolem_genid = "/.well-known/genid/rdflib/"
 skolems: Dict[str, "BNode"] = {}
 
 
+def _is_valid_uri(uri: str) -> bool:
+    full_iri_check = True
+    return __is_valid_iri(uri) if full_iri_check else __is_valid_uri(uri)
+
+
 _invalid_uri_chars = '<>" {}|\\^`'
 
 
-def _is_valid_uri(uri: str) -> bool:
+def __is_valid_uri(uri: str) -> bool:
     for c in _invalid_uri_chars:
         if c in uri:
             return False
     return True
 
+def __is_valid_iri(iri: str) -> bool:
+    if not isinstance(iri, str):
+        iri = str(iri)
+    res = rfc3987.parse(iri)
+    return True if res is not None else False
+
 
 _lang_tag_regex = compile("^[a-zA-Z]+(?:-[a-zA-Z0-9]+)*$")
 

with the following rather interesting results:

Standard, full_iri_check=False

5406 passed, 24 skipped, 117 xfailed, 1 xpassed, 46 warnings in 54.56s

Experimental, full_iri_check=True

8 failed, 5398 passed, 24 skipped, 118 xfailed, 46 warnings in 58.01s

FAILED test/test_misc/test_parse_file_guess_format.py::TestFileParserGuessFormat::test_warning - ValueError: 'file:///tmp/tmppei8l7iv/?xml version="1.0"?' is not a valid 'IRI_reference'.
FAILED test/test_parsers/test_trig_w3c.py::test_manifest[file:///home/gjh/PyBench/rdflib-github-rfc3987/test/data/suites/w3c/trig/manifest.ttl#localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries-http://www.w3.org/ns/rdftest#TestTrigEval-rdf_test31]
FAILED test/test_parsers/test_turtle_w3c.py::test_manifest[file:///home/gjh/PyBench/rdflib-github-rfc3987/test/data/suites/w3c/turtle/manifest.ttl#localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries-http://www.w3.org/ns/rdftest#TestTurtleEval-rdf_test20]
FAILED test/test_serializers/test_serializer_turtle.py::testUnicodeEscaping - ValueError: 'http://example.com/zzz\U00100000zzz' is not a valid 'IRI_reference'.
FAILED test/test_sparql/test_dawg.py::test_dawg_data_sparql11[http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_05-http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#PositiveSyntaxTest11-rdf_test417]
FAILED test/test_sparql/test_dawg.py::test_dawg_data_sparql11[http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_06-http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#PositiveSyntaxTest11-rdf_test418]
FAILED test/test_sparql/test_dawg.py::test_dawg_data_sparql11[http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_07-http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#PositiveSyntaxTest11-rdf_test419]
FAILED test/test_sparql/test_service.py::test_service_with_implicit_select_and_base - ValueError: 'BASE <http://example.org/' is not a valid 'IRI_reference'.

[CrypticGuy]

@gjhiggins we tried to fix something in a similar manner and faced the same issues. Specifically the tests
test/test_parsers/test_trig_w3c.py::test_manifest[file:///home/gjh/PyBench/rdflib-github-rfc3987/test/data/suites/w3c/trig/manifest.ttl#localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries-http://www.w3.org/ns/rdftest#TestTrigEval-rdf_test31 are failing badly.
We tried to analyze it and were stuck there.
Is there any other approach you might recommend that could be worth trying?

[CrypticGuy]

@gjhiggins we tried to fix something in a similar manner and faced the same issues. Specifically the tests test/test_parsers/test_trig_w3c.py::test_manifest[file:///home/gjh/PyBench/rdflib-github-rfc3987/test/data/suites/w3c/trig/manifest.ttl#localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries-http://www.w3.org/ns/rdftest#TestTrigEval-rdf_test31 are failing badly. We tried to analyze it and were stuck there. Is there any other approach you might recommend that could be worth trying?

I'm a bit puzzled, tbh. I don't see anything in the URIs of that particular test other then ordinary ASCII. The value of the object may well be rich in unicode codepoint examples but as it's a Literal, it would be inappropriate to treat and test it as a potentially valid URI. If you had included the test failure output, maybe I'd be less puzzled.

Exception: "http://a.example/AZazÀÖØöø˿Ͱͽ΄῾‌‍⁰↉Ⰰ⿕、ퟻ﨎ﷇﷰ￯��" does not look like a valid  URI, I cannot serialize this as N3/Turtle. Perhaps you wanted to urlencode it?

This is the error that we are getting. So, I'm not sure if it's being encoded properly.

@prefix p: <http://a.example/> .
<http://a.example/s> <http://a.example/p> p:AZazÀÖØöø˿ͰͽͿ῿‌‍⁰↏Ⰰ⿯、퟿﨎﷏ﷰ￯𐀀󯿽 .

The test case if I am not wrong is trying to check the above input for a valid URI.

I remembered I'd created a branch for this so was able to return to it. Yeah, I misread the trig, it is supposed to be a URI as indicated by the prefixing p: