For RAG, convert PDFs to Markdown when you want chunks that follow the document's sections: each heading becomes a split point, and the headings above a chunk travel with it as metadata. Check headings, tables and OCR text before you index, then test retrieval on questions whose answers you already know.

This guide is for developers building retrieval over private documents: policies, manuals, contracts, research. It assumes you already have an embedding model and a vector store in mind.

Stacks of lined index cards and paper strips beside a laptop, one stack tied with a lilac ribbon.
Chunks that follow the document's sections.

Why headings make better chunks

A fixed-size splitter cuts text into pieces of a set length, wherever the cut falls. A chunk can start in the middle of a section and lose the heading that says what it is about. "Requests are limited to 100 per minute" means one thing under "Sandbox" and another under "Production". Without the heading, neither the retriever nor the model can tell which.

Markdown keeps the document's outline as lines that start with #, ## and ###. A heading-based splitter cuts at those lines, so each chunk covers one section, and it records the headings above the chunk: its section path. LangChain's guide to splitting Markdown describes the same idea: keep text together with the headers it belongs to.

Sections vary in length, so a long section still needs a second split by size. The section path applies to every piece of it.

Convert the PDFs and check the headings

In MD Factory, add the folder with your PDFs. Each Markdown file is saved next to its PDF under the same name, with images in a folder beside it. Scanned PDFs go through OCR on your PC. Word files and other formats in the folder are converted the same way.

Open a few results before you write any splitting code:

  • Make sure chapters, sections and subsections come out as #, ## and ###. A PDF often marks a heading only with a bigger or bolder font, so any converter has to infer where sections start and how they nest. Fix the levels where they are wrong.
  • In Word files, headings carry over when the author used Word's Heading styles. Text that is only bold or large stays an ordinary paragraph.
  • Look for running headers, footers and page numbers in the middle of the text.
  • Click Needs review next to a file to see what to check. For a PDF, the list can include pages where OCR could not read part of the text or found no text at all.

Fix problems in the Markdown file before you index it. Every chunk inherits them.

Split by headings in code

LangChain's MarkdownHeaderTextSplitter splits on the heading levels you list and puts each heading into the chunk's metadata. A second splitter limits the size inside each section:

from pathlib import Path
from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    RecursiveCharacterTextSplitter,
)

corpus = Path("corpus")
by_heading = MarkdownHeaderTextSplitter(
    [("#", "h1"), ("##", "h2"), ("###", "h3")], strip_headers=False
)
by_size = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150)

chunks = []
for path in sorted(corpus.rglob("*.md")):
    sections = by_heading.split_text(path.read_text(encoding="utf-8"))
    for chunk in by_size.split_documents(sections):
        chunk.metadata["source"] = path.relative_to(corpus).as_posix()
        chunk.metadata["section"] = " > ".join(
            chunk.metadata[h] for h in ("h1", "h2", "h3") if h in chunk.metadata
        )
        chunks.append(chunk)

The chunk size and overlap here are a starting point; adjust them for your embedding model. To keep the heading lines in the chunk text, so that the text you embed also names its section, the example sets strip_headers=False.

LlamaIndex's MarkdownNodeParser does the same heading split. When you load files with SimpleDirectoryReader, each node also carries the file name:

from llama_index.core import SimpleDirectoryReader
from llama_index.core.node_parser import MarkdownNodeParser

documents = SimpleDirectoryReader(
    "corpus", recursive=True, required_exts=[".md"]
).load_data()
nodes = MarkdownNodeParser().get_nodes_from_documents(documents)
print(nodes[0].metadata["file_name"], nodes[0].metadata["header_path"])

header_path lists the headings above a node, for example /Employee handbook/Expenses/; the node's own heading is the first line of its text.

Keep the source with every chunk

Store at least three things with each chunk:

  • The file. A Markdown file has the same name as its PDF, so a chunk from handbook.md leads back to handbook.pdf. Store the path relative to the corpus folder, because two files called terms.md in different folders are different documents.
  • The section path, so an answer can name the section it used and a reader can go straight to the passage.
  • The edition or date. When a policy changes, you delete the old file's chunks and index the new ones, rather than serving both versions.

If people with different access rights will query the same index, also store who may read each document, and filter on it at query time. A retriever returns any chunk that matches the question.

The Markdown that MD Factory produces has no page numbers. If answers must cite pages, look the passage up in the original PDF.

Handle tables and OCR text with care

A Markdown table has its header row once, at the top. When a table is longer than your chunk size, the size splitter cuts it, and the later pieces arrive without column names. In our test with LangChain (langchain-text-splitters 1.1.3), a 40-row table split into 300-character chunks gave four pieces, and only the first contained the header row.

Two ways to fix this: keep each table in one chunk even if it is longer than the rest, or repeat the header row at the start of every piece.

For Word and Excel files, MD Factory marks a document Needs review when merged or complex cells were simplified. PDF tables get no such flag, so compare a few of them with the original, especially tables with merged headers.

OCR text needs its own check. Misread numbers, names and codes go into answers unchanged, and nothing in the chunk looks wrong. Compare the figures in a few chunks with the scan, and leave flagged pages out of the index until you have fixed them. What to look for is covered in our guide to checking OCR in scanned PDFs.

Test retrieval on real questions

Write down questions people will actually ask, and for each one the file and section that holds the answer. Start with a dozen or two.

  1. Run each question through your retriever and look at the top results.
  2. Note whether the right section is among them.
  3. For each miss, compare the chunks that came back with the one that should have. Typical causes: a wrong heading level, a table cut off from its header, an OCR error in the key term, or a section so long that the answer is buried.
  4. Change one thing, such as chunk size, heading levels or how tables are split, and run the same questions again.

Keep the questions in a file next to the corpus and rerun them whenever the documents or the splitting code change.

When to use MD Factory and when to use a library

MD Factory is a desktop app for Windows. It suits a private collection you want to convert on your own PC and check by eye: add a folder, look at the statuses, open the files that need review, fix them, then point your indexing script at the folder.

MD Factory keeps the documents on your PC while it converts them. If you then embed the chunks with a hosted embedding API or send them to a hosted model, their text goes to that provider. Use a local model if the documents must not leave your network.

If documents arrive all the time, live on a server, or must be converted without a person watching, use a conversion library in your code instead. For options, see our comparison of PDF to Markdown converters.