高级 RAG 02：揭秘 PDF 解析

对于 RAG 系统来说，从文档里准确捞出信息，是绕不开的一步。这一步做得好不好，直接关系到最终输出质量的上下限。

在搭建RAG的时候，千万别小看这个预处理环节。要是解析的时候信息提取得七零八落，后面再怎么折腾，对原文档的理解也始终是雾里看花。

解析这一步在整个RAG流程里处于什么位置？看图1就清楚了。

图1

在实际工作中，非结构化的数据量，可比结构化数据要多得多。这些数据如果解析不出来，背后隐藏的巨大价值也就白白浪费了。而在这些非结构化数据里，PDF格式的文件又占了绝大多数。从这个角度说，能把PDF处理好，对其他非结构化文档的处理也就成功了一大半。

这篇文章就系统地梳理一下解析PDF的那些主流方法，提供一些实用的算法和建议，目标是帮大家尽可能多地从PDF里把有用的信息给“榨”出来。

解析 PDF 的挑战

PDF文档可以说是非结构化文档的“老大哥”了，但想从它里面提取信息，难度可不小。

我们得先理解一件事：PDF本质上不是一个数据格式，它更像是一套打印指令的集合。它包含的是告诉阅读器或打印机“在屏幕哪个位置显示这个字符”的指令，而不是像HTML或者doc文件那样，用标签来标明“这是个标题”、“这是个段落”。它们的区别，看图2就一目了然。

图2

解析PDF的难点，主要在于怎么准确捕捉页面的整体布局，然后把表格、标题、段落、图片这些东西，都忠实地转化成可读的文字。这个过程里，文本抽取不准确、图像识别困难、表格里行与列的关系理不清，都是常见的“坑”。

如何解析 PDF 文档

通常来说，解析PDF有三大流派：基于规则的方法、基于深度学习模型的方法，以及基于多模态大模型的方法。

• 基于规则的方法
  ：顾名思义，就是通过预设的规则来识别文档的样式和内容。但PDF的格式和布局千变万化，光靠几条固定的规则，很难覆盖所有情况，适用性比较有限。

• 基于深度学习模型的方法
  ：这是目前比较流行的做法，典型的方案是把物体检测和OCR模型结合起来用。

• 基于多模态大模型的方法
  ：利用多模态模型直接解析PDF里那些复杂的结构，或者提取关键信息。

基于规则的方法

提到基于规则的方法，就绕不开一个代表性的工具——pypdf。它被广泛用作LangChain和LlamaIndex里解析PDF的默认工具。

我们来试试用pypdf解析一下《Attention Is All You Need》论文的第6页，原页面长这样。

图3

代码写起来很简单：

import PyPDF2  
filename = "/Users/Florian/Downloads/1706.03762.pdf"  
pdf_file = open(filename, 'rb')  

reader = PyPDF2.PdfReader(pdf_file)  

page_num = 5  
page = reader.pages[page_num]  
text = page.extract_text()  

print('--------------------------------------------------')  
print(text)  

pdf_file.close()

输出结果如下（为了简洁，省略了一部分）：

(py) Florian:~ Florian$ pip list | grep pypdf  
pypdf 3.17.4  
pypdfium2 4.26.0  

(py) Florian:~ Florian$ python /Users/Florian/Downloads/pypdf_test.py  
--------------------------------------------------  
Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations  
for different layer types. nis the sequence length, dis the representation dimension, kis the kernel  
size of convolutions and rthe size of the neighborhood in restricted self-attention.  
Layer Type Complexity per Layer Sequential Maximum Path Length  
Operations  
Self-Attention O(n2·d) O(1) O(1)  
Recurrent O(n·d2) O(n) O(n)  
Convolutional O(k·n·d2) O(1) O(logk(n))  
Self-Attention (restricted) O(r·n·d) O(1) O(n/r)  
3.5 Positional Encoding  
Since our model contains no recurrence and no convolution, in order for the model to make use of the  
order of the sequence, we must inject some information about the relative or absolute position of the  
tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the  
bottoms of the encoder and decoder stacks. The positional encodings ha ve the same dimension dmodel  
as the embeddings, so that the two can be summed. There are many choices of positional encodings,  
learned and fixed [9].  
In this work, we use sine and cosine functions of different frequencies:  
PE(pos,2i)=sin(pos/100002i/d model)  
PE(pos,2i+1)=cos(pos/100002i/d model)  
where posis the position and iis the dimension. That is, each dimension of the positional encoding  
corresponds to a sinusoid. The wa velengths form a geometric progression from 2πto10000 ·2π. We  
chose this function because we hypothesized it would allow the model to easily learn to attend by  
relative positions, since for any fixed offset k,PEpos+kcan be represented as a linear function of  
PEpos.  
...  
...  
...

可以看到，pypdf直接把PDF里的字符串成了一个长长的序列，完全没有保留任何结构信息。换句话说，它就是把文档的每一行都当成一个由换行符分隔的序列来处理。这样一来，我们想准确识别出一个段落或者一张表，基本是不可能的。这个限制，也是基于规则的方法的“先天缺陷”。

基于深度学习模型的方法

深度学习方法的优势很明显——它能比较准确地识别出整个文档的布局，包括表格和段落，甚至能理解表格内部的结构。这意味着它能把文档切分成一个个定义明确、信息完整的单元，最大程度地保留原文的意图和结构。

当然，它也有自己的短板。物体检测和OCR这两个阶段都比较耗时，所以建议用GPU或者别的加速硬件，同时配合多进程、多线程来提升处理速度。

这种方法主要涉及物体检测和OCR模型。我实测了几个有代表性的开源框架：

• Unstructured
  ：已经被集成到了LangChain里。在启用 `infer_table_structure=True` 的高分辨率策略下，表格识别效果不错。但快速策略因为没用到物体检测模型，会把很多图片和表格识别错。

• Layout-parser
  ：如果PDF结构比较复杂，建议用最大的模型来保证准确率，但速度会慢一些。另外，这个模型的更新似乎已经停滞了两年多。

• PP-StructureV2
  ：用多种模型组合进行文档分析，效果在平均水平之上。它的架构图是这样的：

图4

除了开源工具，也有一些像ChatDOC这样的付费产品，采用基于布局的识别加上OCR技术来解析PDF。

接下来，我们以开源的Unstructured框架为例，深入聊聊它如何应对三大核心挑战。

挑战1：如何从表格和图片中抽取数据

用Unstructured框架来演示，检测到的表格数据可以直接导出为HTML格式，代码实现如下：

from unstructured.partition.pdf import partition_pdf  

filename = "/Users/Florian/Downloads/Attention_Is_All_You_Need.pdf"  

# infer_table_structure=True 会自动选择 hi_res 策略  
elements = partition_pdf(filename=filename, infer_table_structure=True)  
tables = [el for el in elements if el.category == "Table"]  

print(tables[0].text)  
print('--------------------------------------------------')  
print(tables[0].metadata.text_as_html)

我仔细跟踪了 `partition_pdf` 函数内部的执行流程，它的基础流程图是这样的：

图5

代码的运行结果：

Layer Type Self-Attention Recurrent Convolutional Self-Attention (restricted) Complexity per Layer O(n2 · d) O(n · d2) O(k · n · d2) O(r · n · d) Sequential Maximum Path Length Operations O(1) O(n) O(1) O(1) O(1) O(n) O(logk(n)) O(n/r)
--------------------------------------------------
Layer Type
Complexity per Layer
Sequential Operations
Maximum Path Length
Self-Attention
O(n? - d)
O(1)
O(1)
Recurrent
O(n- d?)
O(n)
O(n)
Convolutional
O(k-n-d?)
O(1)
O(logy(n))
Self-Attention (restricted)
O(r-n-d)
ol)
O(n/r)

把这段HTML标签存成一个HTML文件，用Chrome打开，效果如下：

图6

测试结果显示，Unstructured的算法基本完整地把整个表格复原出来了。

挑战2：如何对检测到的内容块进行重新排列？特别是在处理双栏PDF文件时。

在处理双栏PDF时，比如论文《BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding》，正确的阅读顺序应该像图里红色箭头所示：

图7

Unstructured框架在确定布局后，会把每个页面划分成若干个矩形区块，具体划分方式如图8所示。

图8

每个矩形块的详细信息可以通过以下格式获取：

[
LayoutElement(bbox=Rectangle(x1=851.1539916992188, y1=181.15073777777613, x2=1467.844970703125, y2=587.8204599999975), text='These approaches ha ve been generalized to coarser granularities, such as sentence embed- dings (Kiros et al., 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014). To train sentence representations, prior work has used objectives to rank candidate next sentences (Jernite et al., 2017; Logeswaran and Lee, 2018), left-to-right generation of next sen- tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto- encoder derived objectives (Hill et al., 2016). ', source=, type='Text', prob=0.9519357085227966, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=196.5296173095703, y1=181.1507377777777, x2=815.468994140625, y2=512.548237777777), text='word based only on its context. Unlike left-to- right language model pre-training, the MLM ob- jective enables the representation to fuse the left and the right context, which allows us to pre- In addi- train a deep bidirectional Transformer. tion to the masked language model, we also use a “next sentence prediction” task that jointly pre- trains text-pair representations. The contributions of our paper are as follows: ', source=, type='Text', prob=0.9517233967781067, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=200.22352600097656, y1=539.1451822222216, x2=825.0242919921875, y2=870.542682222221), text='• We demonstrate the importance of bidirectional pre-training for language representations. Un- like Radford et al. (2018), which uses unidirec- tional language models for pre-training, BERT uses masked language models to enable pre- trained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs. ', source=, type='List-item', prob=0.9414362907409668, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=851.8727416992188, y1=599.8257377777753, x2=1468.0499267578125, y2=1420.4982377777742), text='ELMo and its predecessor (Peters et al., 2017, 2018a) generalize traditional word embedding re- search along a different dimension. They extract context-sensitive features from a left-to-right and a right-to-left language model. The contextual rep- resentation of each token is the concatenation of the left-to-right and right-to-left representations. When integrating contextual word embeddings with existing task-specific architectures, ELMo advances the state of the art for several major NLP benchmarks (Peters et al., 2018a) including ques- tion answering (Rajpurkar et al., 2016), sentiment analysis (Socher et al., 2013), and named entity recognition (Tjong Kim Sang and De Meulder, 2003). Melamud et al. (2016) proposed learning contextual representations through a task to pre- dict a single word from both left and right context using LSTMs. Similar to ELMo, their model is feature-based and not deeply bidirectional. Fedus et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod- els. ', source=, type='Text', prob=0.938507616519928, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=199.3734130859375, y1=900.5257377777765, x2=824.69873046875, y2=1156.648237777776), text='• We show that pre-trained representations reduce the need for many hea vily-engineered task- specific architectures. BERT is the first fine- tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outper- forming many task-specific architectures. ', source=, type='List-item', prob=0.9461237788200378, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=195.5695343017578, y1=1185.526123046875, x2=815.9393920898438, y2=1330.3272705078125), text='• BERT advances the state of the art for eleven NLP tasks. The code and pre-trained mod- els are a vailable at https://github.com/ google-research/bert. ', source=, type='List-item', prob=0.9213815927505493, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=195.33956909179688, y1=1360.7886962890625, x2=447.47264000000007, y2=1397.038330078125), text='2 Related Work ', source=, type='Section-header', prob=0.8663332462310791, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=197.7477264404297, y1=1419.3353271484375, x2=817.3308715820312, y2=1527.54443359375), text='There is a long history of pre-training general lan- guage representations, and we briefly review the most widely-used approaches in this section. ', source=, type='Text', prob=0.928022563457489, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=851.0028686523438, y1=1468.341394166663, x2=1420.4693603515625, y2=1498.6444497222187), text='2.2 Unsupervised Fine-tuning Approaches ', source=, type='Section-header', prob=0.8346447348594666, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=853.5444444444446, y1=1526.3701822222185, x2=1470.989990234375, y2=1669.5843488888852), text='As with the feature-based approaches, the first works in this direction only pre-trained word em- (Col- bedding parameters from unlabeled text lobert and Weston, 2008). ', source=, type='Text', prob=0.9344717860221863, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=200.00000000000009, y1=1556.2037353515625, x2=799.1743774414062, y2=1588.031982421875), text='2.1 Unsupervised Feature-based Approaches ', source=, type='Section-header', prob=0.8317819237709045, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=198.64227294921875, y1=1606.3146266666645, x2=815.2886352539062, y2=2125.895459999998), text='Learning widely applicable representations of words has been an active area of research for decades, including non-neural (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Mikolov et al., 2013; Pennington et al., 2014) methods. Pre-trained word embeddings are an integral part of modern NLP systems, of- fering significant improvements over embeddings learned from scratch (Turian et al., 2010). To pre- train word embedding vectors, left-to-right lan- guage modeling objectives ha ve been used (Mnih and Hinton, 2009), as well as objectives to dis- criminate correct from incorrect words in left and right context (Mikolov et al., 2013). ', source=, type='Text', prob=0.9450697302818298, image_path=None, parent=None), 

LayoutElement(bbox=Rectangle(x1=853.4905395507812, y1=1681.5868488888855, x2=1467.8729248046875, y2=2125.8954599999965), text='More recently, sentence or document encoders which produce contextual token representations ha ve been pre-trained from unlabeled text and fine-tuned for a supervised downstream task (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018). The advantage of these approaches is that few parameters need to be learned from scratch. At least partly due to this advantage, OpenAI GPT (Radford et al., 2018) achieved pre- viously state-of-the-art results on many sentence- level tasks from the GLUE benchmark (Wang language model- Left-to-right et al., 2018a). ', source=, type='Text', prob=0.9476840496063232, image_path=None, parent=None)
]

这里 (x1, y1) 是矩形左上角的坐标， (x2, y2) 是右下角的坐标：

     (x_1, y_1) --------
                 |             |
                 |             |
                 |             |
                 ---------- (x_2, y_2)

接下来，需要重新排列页面的阅读顺序。Unstructured自带了一个内置的排序算法，但在处理双栏情况时，排序结果并不理想。

所以，一个专门为此设计的算法就很有必要了。最直接的想法是按左上角的水平坐标排序，如果水平坐标一样，再按垂直坐标排序。伪代码如下：

layout.sort(key=lambda z: (z.bbox.x1, z.bbox.y1, z.bbox.x2, z.bbox.y2))

然而，即便是同一列里的块，它们的水平坐标也可能有细微的偏移。就像图9里那样，紫色线块的水平坐标 `bbox.x1` 实际上比绿色块更靠左。如果按上面的方法排序，紫色块会被排到绿色块前面，这就完全打乱了阅读顺序。

图9

这种情况下，一个可行的算法步骤如下：

• 先对所有左上角的x坐标 x1 排序，找到最小的 x1（即 x1_min）。
• 再对所有右下角的x坐标 x2 排序，找到最大的 x2（即 x2_max）。
• 然后，计算页面中央线的x坐标，方法如下：

x1_min = min([el.bbox.x1 for el in layout])
x2_max = max([el.bbox.x2 for el in layout])
mid_line_x_coordinate = (x2_max + x1_min) /  2

接下来，如果某个块的 bbox.x1 小于 mid_line_x_coordinate，就把它归为左列；否则，就归为右列。

分类完成后，再根据每个块的 y 坐标对它们分别进行排序。最后，把右列拼接到左列的右边就可以了。

left_column = []
right_column = []
for el in layout:
    if el.bbox.x1 < mid_line_x_coordinate:
        left_column.append(el)
    else:
        right_column.append(el)

left_column.sort(key = lambda z: z.bbox.y1)
right_column.sort(key = lambda z: z.bbox.y1)
sorted_layout = left_column + right_column

值得一提的是，这个改进后的算法在处理单栏PDF时也完全适用。

挑战3：如何提取多级标题

提取标题（尤其是多级标题）的目的，是为了提高大语言模型回答问题的准确性。

比如，用户想了解图9中第2.1节的主要内容。如果能准确地把第2.1节的标题和它下面的正文一起作为上下文喂给大模型，最终答案的准确性肯定会大幅提升。

这个算法还是依赖图9中展示的那些布局块。我们可以提取出类型为 'Section-header' 的块，然后计算每个块的高度差（bbox.y2 - bbox.y1）。高度差最大的块对应一级标题，其次对应二级标题，以此类推。

基于多模态大模型解析PDF中的复杂结构

多模态模型火了之后，用它来解析表格也成了一个新思路。有这么几个选项：

• 直接把相关的PDF页面作为图像，发给GPT4-V，让它根据查询来回答。
• 把每一页PDF都当作一张图，让GPT4-V对每一页进行图像推理，然后为这些推理结果构建向量索引，最后在这个索引里查询答案。
• 用Table Transformer这类模型，从检索到的图像里裁剪出表格区域，然后把裁剪后的图像发给GPT4-V来回应查询。
• 在裁剪出的表格图像上做OCR，把文本数据发给GPT4或GPT-3.5来回答问题。

此外，我们还可以用多模态模型直接从图像里提取或总结关键信息，就像图10展示的那样。

图10

结论

总的来说，非结构化文档的灵活性极高，解析它们需要多种技术“混搭”。目前业界还没有一个公认的“最佳方案”。

不同类型的PDF，比如学术论文、书籍、财务报表，都有各自的特性，需要“对症下药”。

如果条件允许，还是推荐用基于深度学习或多模态的技术。它们能有效对文档进行分割，保证信息单元的完整和清晰，最大限度地还原文档原本的意图和结构。