You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+53-46Lines changed: 53 additions & 46 deletions
Original file line number
Diff line number
Diff line change
@@ -29,80 +29,81 @@ The training set is available at `train.json`. Each document contains a number o
29
29
{
30
30
"fold": 2,
31
31
"documentId": "8313",
32
-
"documentText": "Gennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries. He settled at Brünn, in Moravia, and lived about 1756. His best picture is the altar-piece in the chapel of the chateau at Seeberg, in Salzburg. Most of his works remained in Moravia.",
32
+
"source": "DBpedia Abstract",
33
+
"documentText": "Gennaro Basile\n\nGennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries. He settled at Brünn, in Moravia, and lived about 1756. His best picture is the altar-piece in the chapel of the chateau at Seeberg, in Salzburg. Most of his works remained in Moravia.",
33
34
"passages": [
34
35
{
35
-
"passageId": "8313:0:98",
36
+
"passageId": "8313:16:114",
37
+
"passageStart": 16,
38
+
"passageEnd": 114,
39
+
"passageText": "Gennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries.",
36
40
"exhaustivelyAnnotatedProperties": [
37
-
{
41
+
{
38
42
"propertyId": "12",
39
43
"propertyName": "PLACE_OF_BIRTH",
40
44
"propertyDescription": "Describes the relationship between a person and the location where she/he was born."
41
45
}
42
46
],
43
-
"passageStart": 0,
44
-
"passageEnd": 98,
45
-
"passageText": "Gennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries.",
@@ -117,40 +118,46 @@ The training set is available at `train.json`. Each document contains a number o
117
118
The official evaluation script is also available for download and can be used to evaluate a system using the training set (via cross-validation). The script takes a gold standard file (e.g., `train.json`) and a prediction file (which needs to be produced by the system). The prediction file should look exactly like the gold standard file (same documents and fields), except for the contents of `facts` (which should contain the facts predicted by the system).
goldFile Path of the KnowledgeNet file with the gold data
125
-
predictionFile Path of the KnowledgeNet file with the predicted data
126
-
-e {span_e,span_o,uri} Choose the evaluation method: span-exact vs span-overlap vs uri
127
-
128
-
optional arguments:
129
-
-h, --help show this help message and exit
130
-
-c print raw counts of tp/fn/fp for prec/rec/F1 metrics
131
-
-a ANALYSISPATH Folder to store error analysis files (default=no analysis).
132
-
-f {1,2,3,4,5} folds to evaluate (useful during cross-validation). Default is 4.
125
+
positional arguments:
126
+
goldFile path of the KnowledgeNet file with the gold data
127
+
predictionFile path of the KnowledgeNet file with the predicted data
133
128
129
+
optional arguments:
130
+
-h, --help show this help message and exit
131
+
-e {span_exact,span_overlap,uri} choose the evaluation method: span-exact vs span-overlap vs uri
132
+
-c print raw counts of tp/fn/fp for prec/rec/F1 metrics
133
+
-a ANALYSISPATH folder to store error analysis and results files
134
+
(default=no analysis).
135
+
-f {1,2,3,4,5,-1} folds to evaluate. Default is 4. Choose -1 to evaluate on all the folds.
134
136
```
135
137
136
138
The prediction file has to keep the same unique identifiers and attributes for the given documents and passages.
137
-
Each new fact has to be described by a `factId` (obtained as explained above) and should contain the following attributes that are needed to run the evaluation script:
139
+
Each new fact must contain the following attributes that are needed to run the evaluation script:
138
140
*`subjectStart`
139
141
*`subjectEnd`
140
142
*`objectStart`
141
143
*`objectEnd`
142
-
*`subjectUri`
143
-
*`objectUri`
144
+
*`subjectUri` (can be empty)
145
+
*`objectUri` (can be empty)
144
146
*`propertyId`
145
147
148
+
A `factId` will be automatically generated from these attributes.
149
+
146
150
#### Evaluation Methods
147
151
Two facts are considered the same when they have the same property, and there is a match between the values for subject and object.
148
152
149
153
We consider three different methods to establish if there is a match:
150
-
***Span Overlap** (`span_o`): there is an overlap between the character offsets
151
-
***Span Exact** (`span_e`): the character offsets are exactly the same
154
+
***Span Overlap** (`span_overlap`): there is an overlap between the character offsets (set as default in the evaluation script)
155
+
***Span Exact** (`span_exact`): the character offsets are exactly the same
152
156
***URI** (`uri`): Wikidata URIs are the same (only applies to facts that have URIs for both the subject and the object)
153
157
158
+
#### Error Analysis
159
+
In order to facilitate error analysis the script creates a simple html file for browser visualization. It can be enabled using the option `-a`.
160
+
154
161
## Adding a system to the leaderboard
155
162
156
163
To preserve the integrity of the results, we have released the test set (fifth fold) without annotations (`test-no-facts.json`). To evaluate the results of your system and (optionally) add your system to the leaderboard, please send an email with your prediction file to filipe[at]diffbot[dot]com.
0 commit comments