> Documentation index for agents: https://docs.rocketride.ai/llms.txt

# preprocessor_code

A RocketRide preprocessor node that splits supported source code into
syntax-aware documents. Choose it over a general-text or LLM preprocessor when
you want code structures, rather than prose boundaries, to drive chunking.

## About Tree-sitter

Tree-sitter is the parsing library this node uses to build source-code syntax
trees. The node installs its language-specific grammar packages and creates a
parser for the selected language.

## What it does

Receives `text` and emits `documents` made from syntax-tree nodes. It supports
the C, C++, Python, JavaScript, and TypeScript profiles; in automatic mode it
detects one of those languages from the text itself. A matched class or
function can coexist with chunks for nested constructs, so output may overlap.

## Lanes

| Lane in | Lane out | Description |
| --- | --- | --- |
| `text` | `documents` | Split source-code text into syntax-aware documents. |

## Profiles

Default: `auto`, which detects the language from the source text.

| Profile | Language |
| --- | --- |
| `auto` *(default)* | Detect from source text |
| `c` | C |
| `cpp` | C++ |
| `python` | Python |
| `javascript` | JavaScript |
| `typescript` | TypeScript |

## Configuration

Select the language profile that matches the input when its language is known;
otherwise leave the default automatic profile. The maximum string length is
present in the configuration, but the current implementation determines output
boundaries from syntax nodes.

### Code splitter profile

The automatic profile samples up to the first 5,000,000 characters and scores regex signals
for the five supported languages. It needs a score of at least 3 and a lead of
at least 2 over the runner-up; otherwise the input produces no documents and a
warning is emitted. Use an explicit profile for short snippets, unusual code,
or content that automatic detection cannot classify reliably.

For C and C++, automatic detection favors C for header-like ties and for
`extern "C"` without C++-only markers. Pick `cpp` explicitly when that
conservative choice is wrong for your input.

### Maximum string length

This value defaults to 512, but the splitter does not use it to divide a
syntax node. A large class or function is therefore emitted as one chunk even
when it exceeds that value. Change it only if a downstream component reads the
same configuration; it does not constrain this node's output today.

## Notes

### Output details

Both text and table callbacks are parsed. Table-derived documents are marked
as tables; each object starts its `chunkId` at 0. C and C++ header declarations
and whole `extern "C"` linkage blocks are included, while preprocessor lines
such as `#include` and `#define` are skipped.

The parser cache is a class attribute on `PreProcessor`, so a grammar loaded
once is reused by every instance in the process and outlives any single global
node instance. If the optional `tree_sitter_languages` package is available it
is used to load a grammar; otherwise the node loads the installed per-language
grammar module.

## Upstream docs

- [Tree-sitter documentation](https://tree-sitter.github.io/tree-sitter/)

---

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->

## Schema

| Field | Type | Description | Default |
|---|---|---|---|
| `code.splitter.auto.language` | `string` |  | const: `"auto"` |
| `code.splitter.c.language` | `string` |  | const: `"c"` |
| `code.splitter.cpp.language` | `string` |  | const: `"cpp"` |
| `code.splitter.javascript.language` | `string` |  | const: `"javascript"` |
| `code.splitter.profile` | `string` | **Code splitter profile** | `"auto"` |
| `code.splitter.python.language` | `string` |  | const: `"python"` |
| `code.splitter.strlen` | `number` | **Maximum string length** | `512` |
| `code.splitter.typescript.language` | `string` |  | const: `"typescript"` |

## Dependencies

- `tree-sitter`
- `tree-sitter-c`
- `tree-sitter-cpp`
- `tree-sitter-javascript`
- `tree-sitter-python`
- `tree-sitter-typescript`

## Source

[<svg viewBox="0 0 16 16" width="15" height="15" fill="currentColor" aria-hidden="true" style="vertical-align:-0.15em;margin-right:0.35em"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"/></svg> View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/preprocessor_code)
<!-- ROCKETRIDE:GENERATED:PARAMS END -->
