> Documentation index for agents: https://docs.rocketride.ai/llms.txt

# preprocessor_langchain

A RocketRide preprocessor node that splits general text into documents for
embedding or LLM processing. Choose it over the code preprocessor for prose and
over the LLM preprocessor when deterministic, local splitter behavior is enough.

## About LangChain

LangChain provides the `langchain_text_splitters` classes used by this node.
The node selects one of those classes from its profile and supplies the chunk
size and length function it uses for splitting.

## What it does

Splits incoming `text` and `table` content on the `documents` lane. It offers
general, Markdown, LaTeX, sentence-oriented, and fixed-separator splitters,
whereas the code preprocessor extracts code syntax and the LLM preprocessor
asks a connected model to make chunking decisions. Text is accumulated until an
object closes; tables are split as they arrive.

## Lanes

| Lane in | Lane out | Description |
| --- | --- | --- |
| `text` | `documents` | Split accumulated text when the object closes. |
| `table` | `documents` | Split each incoming table immediately. |

## Profiles

Default: `default`, the recursive character splitter.

| Profile | Splitter |
| --- | --- |
| `default` *(default)* | `RecursiveCharacterTextSplitter` |
| `recursive` | `RecursiveCharacterTextSplitter` |
| `character` | `CharacterTextSplitter` |
| `markdown` | `MarkdownTextSplitter` |
| `latex` | `LatexTextSplitter` |
| `nltk` | `NLTKTextSplitter` |
| `spacy` | `SpacyTextSplitter` |
| `custom` | `RecursiveCharacterTextSplitter` |

## Configuration

Start with the default profile and string-length mode for ordinary prose. Pick
a content-specific profile when its boundaries matter, then set the size in
characters or estimated tokens. The generated schema lists the profile-owned
fields; the choices below explain their operational effects.

### Text splitter

The profile fixes the splitter class; changing the class field to another
value fails configuration validation. Use `markdown` for Markdown, `latex` for
LaTeX, `character` for one fixed separator, `nltk` for the NLTK splitter, or
`spacy` for the spaCy splitter. The `custom` selector is present in the
configuration but is also fixed to `RecursiveCharacterTextSplitter`, so it
currently behaves like `default`.

### Split by and chunk size

String-length mode uses the configured string length, 512 by default. Token
mode estimates tokens as UTF-8 byte length divided by 3, rounded up; this is a
conservative estimate rather than a tokenizer's count. Choose token mode when
a downstream model or embedding service has a token budget, and lower the size
when chunks are rejected for being too long.

### Split separators

For the recursive profile, write a comma-separated sequence of quoted Python
string literals, such as `'\n\n', '\n', ' ', ''`. The node parses that
sequence with `ast.literal_eval`; every item must be a string or startup fails.
For the character profile, supply exactly one such literal. Change recursive
separators to prefer domain boundaries before falling back to shorter ones.

### Model

The spaCy profile uses the selected spaCy pipeline, defaulting to
`en_core_web_sm`. Choose one of the listed English, German, French, or Spanish
models for text in that language. If the pipeline is absent, the node downloads
it before creating the splitter; this may delay the first run.

## Notes

### Downloads and output

The NLTK profile installs its extra dependencies when selected and downloads
`punkt` when it cannot find it; it also attempts to download `punkt_tab` when
needed. All splitters use zero chunk overlap. Every emitted document receives a
sequential `chunkId` per object; table documents also receive a `tableId`.

### Token-budget safety net

When token mode also has a positive `max_model_tokens` configuration value,
the node subtracts `token_safety_margin` (32 by default), caps the requested
chunk size to the result, and force-splits any remaining oversized output. The
byte-per-token estimate defaults to 3.0 and can be changed through
`bytes_per_token`; lowering it estimates more tokens and is the safer choice.

## Upstream docs

- [LangChain text splitters](https://python.langchain.com/docs/how_to/recursive_text_splitter/)

---

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->

## Schema

| Field | Type | Description | Default |
|---|---|---|---|
| `langchain.splitter.character.separator` | `string` | **Split separator** | `"\"\\n\""` |
| `langchain.splitter.character.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to CharacterTextSplitter by the 'Character Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"CharacterTextSplitter"` |
| `langchain.splitter.custom.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to RecursiveCharacterTextSplitter by the 'Custom' text-splitter profile and cannot be changed here; the custom profile currently behaves like default. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"RecursiveCharacterTextSplitter"` |
| `langchain.splitter.default.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to RecursiveCharacterTextSplitter by the 'Default' text-splitter profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile (for example Markdown for MarkdownTextSplitter). Editing this to another class fails validation with 'must be equal to constant'. | const: `"RecursiveCharacterTextSplitter"` |
| `langchain.splitter.latex.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to LatexTextSplitter by the 'Latex Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"LatexTextSplitter"` |
| `langchain.splitter.markdown.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to MarkdownTextSplitter by the 'Markdown Text Splitter' profile and cannot be changed here. To use MarkdownTextSplitter, select 'Markdown Text Splitter' in the 'Text splitter' selector above rather than editing this field. Editing this to another class fails validation with 'must be equal to constant'. | const: `"MarkdownTextSplitter"` |
| `langchain.splitter.mode` | `string` | **Split by** | `"strlen"` |
| `langchain.splitter.nltk.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to NLTKTextSplitter by the 'NLTK Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"NLTKTextSplitter"` |
| `langchain.splitter.profile` | `string` | **Text splitter**<br/>Selects the splitter profile. Each profile locks one LangChain splitter class and shows only that splitter's options, so the splitter class is not chosen independently of the profile. Pick the profile that matches your content: Markdown for .md, Latex for LaTeX, Character/NLTK/Spacy as needed. Editing a profile's splitter field to a different class fails schema validation with a 'must be equal to constant' error; change this selector instead. | `"default"` |
| `langchain.splitter.recursive.separators` | `string` | **Split separators** | `"'\\n\\n', '\\n', ' ', ''"` |
| `langchain.splitter.recursive.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to RecursiveCharacterTextSplitter by the 'Recursive Character Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"RecursiveCharacterTextSplitter"` |
| `langchain.splitter.spacy.model` | `string` | **Model** | `"en_core_web_sm"` |
| `langchain.splitter.spacy.splitter` | `string` | **Splitter class (set by profile)**<br/>Fixed to SpacyTextSplitter by the 'Spacy Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'. | const: `"SpacyTextSplitter"` |
| `langchain.splitter.strlen` | `number` | **String length** | `512` |
| `langchain.splitter.tokens` | `number` | **Number of tokens** | `512` |

## Dependencies

- `langchain`
- `langchain-text-splitters`
- `langchain-core`
- `pyyaml`
- `filelock`
- `regex`
- `tqdm`

## Source

[<svg viewBox="0 0 16 16" width="15" height="15" fill="currentColor" aria-hidden="true" style="vertical-align:-0.15em;margin-right:0.35em"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"/></svg> View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/preprocessor_langchain)
<!-- ROCKETRIDE:GENERATED:PARAMS END -->
