> Documentation index for agents: https://docs.rocketride.ai/llms.txt

# preprocessor_llm

A RocketRide preprocessor node that uses a connected LLM to split document text
into chunks for vector embedding storage. Choose it over rule-based text
splitters when the LLM's document-level chunking guidance is worth the model
call.

## About LangChain

LangChain is a library for language-model applications. This node's package
declares `langchain` and `langchain-core` dependencies while obtaining model
operations through RocketRide's LLM connection.

## What it does

Accumulates text for an object and, when it closes, asks the required LLM
connection for a JSON `chunks` array. The prompt requests coherent chunks that
preserve the input text plus a final summary chunk. Choose the general-text
preprocessor when deterministic local splitting is preferable, or the code
preprocessor for source-code syntax.

## Connections

| Connection | Required | Description |
| --- | --- | --- |
| `llm` | yes | LLM used to process document text. |

## Lanes

| Lane in | Lane out | Description |
| --- | --- | --- |
| `text` | `documents` | Accumulate and LLM-chunk text when the object closes. |
| `table` | `documents` | Split tables locally into document chunks. |

## Configuration

The default configuration uses 384 tokens as the prompt's target chunk size.
Connect an LLM before processing, because the node reads that connection's
context length, output length, and token counter on its first object.

### Number of tokens per document chunk

This setting is included in the LLM prompt as the target size, with a
three-quarters-as-many-words fallback. Set it to the chunk capacity appropriate
for the embedding model that will receive the documents. It also sets the
budget used to divide tables at line boundaries, so reduce it when table chunks
are too large for the downstream embedding model. It is not the limit used to
pre-split text for the LLM request: that limit comes from the connected LLM's
context and output capacities.

## Notes

### Text and table processing

Before calling the LLM, the node reserves its prompt size and 500 tokens for
JSON formatting, then uses the smaller of the remaining context budget and the
model's output capacity. It begins splitting oversized text at a 65,536-character
threshold, working through paragraphs, lines, sentences, and finally words. A
single oversized word can remain intact rather than being cut further.

Tables bypass the LLM. They are split only between lines; a line that is longer
than the configured token budget remains a chunk by itself. Empty output
content is discarded. Each emitted document gets a `chunkId`; LLM summary
chunks are marked as summaries, and chunks made from tables are marked as
tables with a shared `tableId` for their source table.

### LLM response shape

The node reads `chunks` from the LLM's JSON response. If that key is absent or
empty, text input produces no text documents; table chunks already collected
can still be emitted.

## Upstream docs

- [LangChain documentation](https://python.langchain.com/docs/)

---

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->

## Schema

| Field | Type | Description | Default |
|---|---|---|---|
| `preprocessor_llm.numberOfTokens` | `number` | **Number of tokens per document chunk. Needs to match your embedding model.** | `384` |
| `preprocessor_llm.profile` | `string` |  | `"default"` |

## Dependencies

- `langchain`
- `langchain-core`

## Source

[<svg viewBox="0 0 16 16" width="15" height="15" fill="currentColor" aria-hidden="true" style="vertical-align:-0.15em;margin-right:0.35em"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"/></svg> View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/preprocessor_llm)
<!-- ROCKETRIDE:GENERATED:PARAMS END -->
