Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
85dc9c0
feat:清理元素属性,保留图片的有效src(排除base64)、alt,以及所有元素的class和id"
LollipopsAndWine Jun 9, 2025
35909d6
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 9, 2025
6cffbb6
feat: 精简控制是否获取XPATH
LollipopsAndWine Jun 9, 2025
52b544b
Merge remote-tracking branch 'origin/dev' into dev
LollipopsAndWine Jun 9, 2025
c96bbf9
feat: 精简控制是否获取XPATH
LollipopsAndWine Jun 9, 2025
ceede71
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 9, 2025
460005a
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 10, 2025
c47b8c0
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 17, 2025
61fd634
feat: 自定义标签'marked-tail', 'marked-text'配置为行内标签
LollipopsAndWine Jun 17, 2025
a122c6a
Merge remote-tracking branch 'origin/dev' into dev
LollipopsAndWine Jun 17, 2025
59aff6d
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 19, 2025
a73c2b4
fix: 重命名自定义标签名称
LollipopsAndWine Jun 19, 2025
63b2467
Merge remote-tracking branch 'origin/dev' into dev
LollipopsAndWine Jun 19, 2025
87d9f82
fix: 重命名自定义标签名称
LollipopsAndWine Jun 19, 2025
ff910e9
fix: 去掉冗余代码
LollipopsAndWine Jun 19, 2025
9e8055e
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jun 30, 2025
049788b
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 9, 2025
d7e09d1
fix: 修复多语种拼接规则
LollipopsAndWine Jul 10, 2025
466e4aa
Merge remote-tracking branch 'origin/dev' into dev
LollipopsAndWine Jul 15, 2025
cd4f5cc
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 16, 2025
9113a0e
fix: 新增language参数
LollipopsAndWine Jul 16, 2025
0e6deee
fix: 修复无法处理xml
LollipopsAndWine Jul 16, 2025
e288989
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 22, 2025
f390206
feat: noclip管线新增预处理:删除表单交互式元素
LollipopsAndWine Jul 22, 2025
39870a7
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 24, 2025
d71155e
feat: noclip管线新增预处理配置
LollipopsAndWine Jul 24, 2025
d4b6305
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 31, 2025
2b0cdd4
fix: 修复title、list、table、text管线中换行不正确以及缺失内容
LollipopsAndWine Jul 31, 2025
38c9a0a
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Jul 31, 2025
0fbac26
fix: 去掉冗余代码
LollipopsAndWine Jul 31, 2025
0bd9d28
fix: 增加测试用例
LollipopsAndWine Jul 31, 2025
aa9f984
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Aug 8, 2025
12cccf6
refactor: 重构simple
LollipopsAndWine Aug 8, 2025
72b2833
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Aug 14, 2025
1a1fb18
fix: 允许获取非标准结构的list的content_list
LollipopsAndWine Aug 14, 2025
69684b8
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Aug 21, 2025
9fc1f4d
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Aug 22, 2025
24b0de5
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Aug 24, 2025
8d30d93
feat: noclip管线main_html预处理使用selectolax修复html
LollipopsAndWine Aug 24, 2025
c36134a
fix: 修复复杂嵌套table提取逻辑
LollipopsAndWine Aug 24, 2025
5e5bb1b
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Sep 1, 2025
ca49f1a
fix: image提取caption
LollipopsAndWine Sep 1, 2025
b4c641e
feat: image提取caption的单测
LollipopsAndWine Sep 1, 2025
da0f4e4
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Sep 3, 2025
58caba9
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Sep 11, 2025
12c05ca
fix: 1.简化表格,去掉非表格标签 2.修复table、list、title、text中上下标与主体被分隔开
LollipopsAndWine Sep 11, 2025
3a6ff2b
fix: 去掉冗余代码
LollipopsAndWine Sep 11, 2025
815b0f1
Merge branch 'ccprocessor:dev' into dev
LollipopsAndWine Sep 11, 2025
ad22fb5
fix: 修复段落结尾为换行时datajson拼接异常
LollipopsAndWine Sep 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions llm_web_kit/extractor/html/recognizer/text.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,10 +267,10 @@ def __get_paragraph_text_recusive(el: HtmlElement, text: str) -> str:
return text

if final := __get_paragraph_text_recusive(root, ''):
para_text.append({'c': final.replace('$br$', PARAGRAPH_SEPARATOR), 't': ParagraphTextType.TEXT})
para_text.append({'c': final, 't': ParagraphTextType.TEXT})

for item in para_text:
item['c'] = restore_sub_sup_from_text_regex(item['c'])
item['c'] = restore_sub_sup_from_text_regex(item['c']).replace('$br$', PARAGRAPH_SEPARATOR)
return para_text

def __extract_paragraphs(self, root: HtmlElement):
Expand Down
1 change: 0 additions & 1 deletion llm_web_kit/input/datajson.py
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,6 @@ def __join_one_para(self, para: list, exclude_inline_types: list = []) -> str:
c = el['c']
if not c or not c.strip():
continue
c = c.strip()
new_c = self.__escape_md_special_chars(c) # 转义特殊字符
one_para.append(new_c)
elif el['t'] == ParagraphTextType.EQUATION_INLINE:
Expand Down
Loading