// Do not edit directly!
// This file was generated by camel-quarkus-maven-plugin:update-extension-doc-page
[id="extensions-langchain4j-ingest"]
= LangChain4j Ingest
:linkattrs:
:cq-artifact-id: camel-quarkus-langchain4j-ingest
:cq-native-supported: true
:cq-status: Experimental
:cq-status-deprecation: Experimental
:cq-description: Declarative AI document ingestion: point a knowledge base at a folder via configuration; splitting, embedding and storing are handled under the hood
:cq-deprecated: false
:cq-jvm-since: 3.39.0
:cq-native-since: 3.39.0

ifeval::[{doc-show-badges} == true]
[.badges]
[.badge-key]##JVM since##[.badge-supported]##3.39.0## [.badge-key]##Native since##[.badge-supported]##3.39.0## [.badge-key]##🧪##[.badge-unsupported]##Experimental##
endif::[]

Declarative AI document ingestion: point a knowledge base at a folder via configuration; splitting, embedding and storing are handled under the hood

[id="extensions-langchain4j-ingest-maven-coordinates"]
== Maven coordinates

https://{link-quarkus-code-generator}/?extension-search=camel-quarkus-langchain4j-ingest[Create a new project with this extension on {link-quarkus-code-generator}, window="_blank"]

Or add the coordinates to your existing project:

[source,xml]
----
<dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-langchain4j-ingest</artifactId>
</dependency>
----
ifeval::[{doc-show-user-guide-link} == true]
Check the xref:user-guide/index.adoc[User guide] for more information about writing Camel Quarkus applications.
endif::[]

[id="extensions-langchain4j-ingest-usage"]
== Usage
[id="extensions-langchain4j-ingest-usage-ingesting-a-directory"]
=== Ingesting a directory

Point a pipeline at a folder and the files in it become a knowledge base. No route, no ingestion code:

[source,properties]
----
quarkus.camel.langchain4j.ingest.products.source.directory=/var/data/product-docs
quarkus.camel.langchain4j.ingest.products.embedding-store=products
quarkus.camel.langchain4j.ingest.products.embedding-model=my-model
----

Each file is read as UTF-8 text — there is no format parsing, so convert a PDF or DOCX before it reaches the pipeline — split into overlapping segments (`max-segment-size`, `max-overlap-size`), embedded in batches and written to the store; a document is held in memory whole while it is split. `embedding-store` and `embedding-model` name CDI beans and may be omitted when the application has exactly one of each. Every segment carries `camel_quarkus_pipeline` and `camel_quarkus_document_id` metadata, so retrieval can cite which document an answer came from. Apart from `enabled` and the `source.*` settings shown here, properties are fixed at build time. A pipeline declared through runtime properties alone — nothing but a `source.directory`, say — is invisible to build-time validation; its checks, including the clash with an equally named `@Ingest` pipeline, report at startup instead.

[NOTE]
====
This experimental extension ingests what it is given and keeps no record of it: a document ingested twice leaves two copies in the store, a restart re-reads the whole directory (the duplicate register is in-memory, sized for 100,000 files), and a polled consumer re-reads its source on every poll. Keeping the store in step with a changing source — skipping unchanged documents, replacing edited ones, removing deleted ones — needs that record and arrives with the synchronising engine in a later release.
====

[id="extensions-langchain4j-ingest-usage-other-sources-declared-in-java"]
=== Other sources, declared in Java

Any Camel consumer can feed a pipeline — the roughly 300 components, each with its own options and its own documentation. Such a pipeline is declared in Java with `@Ingest` and the Camel Endpoint DSL:

[source,java]
----
@Ingest("events")
IngestPipeline events() {
    return IngestPipeline.from(Source.endpoint(dsl -> dsl.kafka("ingest-events").groupId("ingest"))
                    .documentId("CamelKafkaKey"))
            .embeddingStore("events");
}
----

Typing `dsl.` lists a factory for every component, each completing its own typed options. The method runs once at startup; it must return `IngestPipeline`, take no parameters and use a name no configured pipeline uses. `enabled=false` in configuration switches a Java-declared pipeline off. A component missing from the classpath fails at startup with an error naming the extension artifact that provides it — the DSL compiles regardless, since its factories all ship in one artifact.

Ingestion needs a stable id per document, and where it lives is the consumer's business: `documentId` names the header — the record key `CamelKafkaKey` above, `CamelAwsS3Key` for S3 — or gives a simple-language expression. Without it, the pipeline expects the `CamelIngestDocumentId` header and fails the exchange when it is absent. Mind each component's own defaults, too: the `aws2-s3` consumer *deletes objects after reading them* unless `deleteAfterRead(false)` is set — a knowledge base reads its source, it does not consume it.

After such a pipeline ingests a document, the exchange body is replaced with the `IngestResult`, so a request-reply caller receives the outcome of its call.

The same pipeline can be declared purely in properties: `source.uri` takes the consumer URI as written, `source.document-id` the header name — or a simple-language expression written `+$simple{...}+`, the one form MicroProfile Config leaves untouched. The URI is fixed at build time by design — a runtime-overridable consumer URI would be arbitrary component invocation — while property placeholders inside it still resolve at startup, keeping credentials and endpoints runtime configuration. Treat runtime configuration as the trust boundary it is: the directory and the id expression decide what the process reads into an often external store.

[source,properties]
----
quarkus.camel.langchain4j.ingest.s3docs.source.uri=aws2-s3://product-docs?region=eu-west-1&deleteAfterRead=false
quarkus.camel.langchain4j.ingest.s3docs.source.document-id=CamelAwsS3Key
quarkus.camel.langchain4j.ingest.s3docs.embedding-store=products
----

[id="extensions-langchain4j-ingest-usage-when-ingestion-fails"]
=== When ingestion fails

A failure while splitting, embedding or storing — a rate-limited model, an unreachable store — propagates to the consumer; there is no dead-letter channel in this increment. For a directory pipeline the file stays where it is and is retried on the next poll, because the duplicate-protection key is only committed on success — which also means a permanently failing file is retried forever, loudly. For a consumer-fed pipeline the component's own error handling applies: a request-reply caller receives the exception, while a Kafka consumer with default settings logs the failure and commits the offset, so the record is *dropped* — and since this engine keeps no record either, nothing remembers it. A record whose configured `document-id` resolves to nothing (a Kafka record without a key, say) fails the same way, one exchange at a time.


[id="extensions-langchain4j-ingest-quarkus-langchain4j-bom"]
== LangChain4j usage

=== Dependency management

In order to ensure alignment across all Quarkus and LangChain4j related dependencies, it is recommended to import the LangChain4j BOM as below:
[source,xml,subs=attributes+]
----
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>dev.langchain4j</groupId>
      <artifactId>langchain4j-bom</artifactId>
      <version>{langchain4j-version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
  ...
</dependencyManagement>
----

Note that the import order is paramount when using maven `dependencyManagement`.
As such, one might need to import the `langchain4j-bom` before other related Camel and Quarkus BOMs.

=== Quarkus LangChain4j support

This extension is designed and tested to work together with the https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html[Quarkus LangChain4j] extensions. The `EmbeddingStore` and `EmbeddingModel` beans a pipeline writes through are ordinary CDI beans shared by both stacks — including stores declared through Quarkus LangChain4j configuration — and with the RAG augmentor bridge an `@RegisterAiService` interface answers from the store a pipeline filled, without glue code. The integration tests run both stacks in one application.


[id="extensions-langchain4j-ingest-additional-camel-quarkus-configuration"]
== Additional Camel Quarkus configuration

[width="100%",cols="80,5,15",options="header"]
|===
| Configuration property | Type | Default


a|icon:lock[title=Fixed at build time] [[quarkus-camel-langchain4j-ingest-pipeline-name-source-uri]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-source-uri[quarkus.camel.langchain4j.ingest."pipeline-name".source.uri]`

The Camel consumer URI feeding this pipeline: any component, with its own options.
Setting it is what makes the pipeline consume from that component; leaving it unset
makes the pipeline read the directory named by the runtime `source.directory`
property instead. Fixed at build time by design — a runtime-overridable consumer
URI would be arbitrary component invocation. Property placeholders inside it still
resolve at startup, so credentials and endpoints remain runtime configuration.
| `string`
| 

a|icon:lock[title=Fixed at build time] [[quarkus-camel-langchain4j-ingest-pipeline-name-embedding-store]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-embedding-store[quarkus.camel.langchain4j.ingest."pipeline-name".embedding-store]`

Name of the `EmbeddingStore` bean to write to. When not set, the only one present is
used.
| `string`
| 

a|icon:lock[title=Fixed at build time] [[quarkus-camel-langchain4j-ingest-pipeline-name-embedding-model]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-embedding-model[quarkus.camel.langchain4j.ingest."pipeline-name".embedding-model]`

Name of the `EmbeddingModel` bean to embed with. When not set, the only one present is
used.
| `string`
| 

a|icon:lock[title=Fixed at build time] [[quarkus-camel-langchain4j-ingest-pipeline-name-max-segment-size]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-max-segment-size[quarkus.camel.langchain4j.ingest."pipeline-name".max-segment-size]`

Maximum size of one segment, in characters.
| `int`
| `500`

a|icon:lock[title=Fixed at build time] [[quarkus-camel-langchain4j-ingest-pipeline-name-max-overlap-size]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-max-overlap-size[quarkus.camel.langchain4j.ingest."pipeline-name".max-overlap-size]`

How much of the previous segment each segment repeats, in characters. Overlap keeps a
sentence split across a boundary retrievable from either side.
| `int`
| `50`

a| [[quarkus-camel-langchain4j-ingest-pipeline-name-enabled]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-enabled[quarkus.camel.langchain4j.ingest."pipeline-name".enabled]`

Whether this pipeline starts. Useful to switch ingestion off in dev mode.
| `boolean`
| `true`

a| [[quarkus-camel-langchain4j-ingest-pipeline-name-source-directory]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-source-directory[quarkus.camel.langchain4j.ingest."pipeline-name".source.directory]`

The directory to ingest documents from, for a pipeline that has no `source.uri`. A
path is a deployment concern, so unlike the URI it stays runtime configuration.
Setting both is an error.
| `string`
| 

a| [[quarkus-camel-langchain4j-ingest-pipeline-name-source-recursive]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-source-recursive[quarkus.camel.langchain4j.ingest."pipeline-name".source.recursive]`

Whether subdirectories are ingested too, when reading a directory.
| `boolean`
| `true`

a| [[quarkus-camel-langchain4j-ingest-pipeline-name-source-document-id]]`link:#quarkus-camel-langchain4j-ingest-pipeline-name-source-document-id[quarkus.camel.langchain4j.ingest."pipeline-name".source.document-id]`

Where the document id lives in the exchange the consumer delivers: normally the
name of a header, such as `CamelAwsS3Key` for an S3 consumer or `CamelKafkaKey` for
a Kafka one. For an id that is not a plain header, write a simple-language
expression in the `+$simple{...}+` form — MicroProfile Config passes it through
untouched, while a `+${...}+` in a properties file would be consumed as a config
expansion before Camel ever saw it. When not set, a pipeline reading a directory
uses the file name, and one consuming from a component uses the
`CamelIngestDocumentId` header.
| `string`
| 
|===

[.configuration-legend]
{doc-link-icon-lock}[title=Fixed at build time] Configuration property fixed at build time. All other configuration properties are overridable at runtime.

