IngestDocument

yaml
type: "io.kestra.plugin.ai.rag.ingestdocument"

Examples

yaml
id: document_ingestion
namespace: company.ai

tasks:
  - id: ingest
    type: io.kestra.plugin.ai.rag.IngestDocument
    provider:
      type: io.kestra.plugin.ai.provider.GoogleGemini
      modelName: gemini-embedding-exp-03-07
      apiKey: "{{ kv('GEMINI_API_KEY') }}"
    embeddings:
      type: io.kestra.plugin.ai.embeddings.KestraKVStore
    drop: true
    fromExternalURLs:
      - https://raw.githubusercontent.com/kestra-io/docs/refs/heads/main/content/blogs/release-0-24.md

Properties

embeddings *Chroma Elasticsearch KestraKVStore Milvus MongoDBAtlas PGVector Pinecone Qdrant Redis Weaviate

provider *AmazonBedrock Anthropic AzureOpenAI DeepSeek GoogleGemini GoogleVertexAI MistralAI Ollama OpenAI

documentSplitter IngestDocument-DocumentSplitter

drop booleanstring

Default false

fromDocuments array

SubType

fromExternalURLs array

SubType string

fromInternalURIs array

SubType string

fromPath string

metadata object

SubType string

Outputs

embeddingStoreOutputs object

ingestedDocuments integer

inputTokenCount integer

outputTokenCount integer

totalTokenCount integer

Definitions

Azure OpenAI Model Provider

endpoint *string

modelName *string

type *object

apiKey string

clientId string

clientSecret string

serviceVersion string

tenantId string

PGVector Embedding Store

database *string

host *string

password *string

port *integerstring

table *string

type *object

user *string

useIndex booleanstring

Default false

Qdrant Embedding Store

apiKey *string

collectionName *string

host *string

port *integerstring

type *object

Google VertexAI Model Provider

endpoint *string

location *string

modelName *string

project *string

type *object

Google Gemini Model Provider

apiKey *string

modelName *string

type *object

MongoDB Atlas Embedding Store

collectionName *string

host *string

indexName *string

scheme *string

type *object

createIndex booleanstring

database string

metadataFieldNames array

SubType string

options object

password string

username string

Mistral AI Model Provider

apiKey *string

modelName *string

type *object

baseUrl string

In-memory embedding store that stores data as Kestra KV pairs

type *object

kvName string

Default {{flow.id}}-embedding-store

Chroma Embedding Store

baseUrl *string

collectionName *string

type *object

Redis Embedding Store

host *string

port *integerstring

type *object

indexName string

Default embedding-index

io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection-BasicAuth

password string

username string

Milvus Embedding Store

token *string

type *object

autoFlushOnDelete booleanstring

autoFlushOnInsert booleanstring

collectionName string

consistencyLevel string

databaseName string

host string

idFieldName string

indexType string

metadataFieldName string

metricType string

password string

port integerstring

retrieveEmbeddingsOnSearch booleanstring

textFieldName string

uri string

username string

vectorFieldName string

Deepseek Model Provider

apiKey *string

modelName *string

type *object

baseUrl string

Default https://api.deepseek.com/v1

Pinecone Embedding Store

apiKey *string

cloud *string

index *string

region *string

type *object

namespace string

Anthropic AI Model Provider

apiKey *string

modelName *string

type *object

io.kestra.plugin.ai.rag.IngestDocument-DocumentSplitter

maxOverlapSizeInChars *integer

maxSegmentSizeInChars *integer

splitter string

Default RECURSIVE

Possible Values

RECURSIVEPARAGRAPHLINESENTENCEWORD

Weaviate Embedding Store

apiKey *string

host *string

type *object

avoidDups booleanstring

consistencyLevel string

Possible Values

ONEQUORUMALL

grpcPort integerstring

metadataFieldName string

metadataKeys array

SubType string

objectClass string

port integerstring

scheme string

securedGrpc booleanstring

useGrpcForInserts booleanstring

Ollama Model Provider

endpoint *string

modelName *string

type *object

OpenAI Model Provider

apiKey *string

modelName *string

type *object

baseUrl string

io.kestra.plugin.ai.rag.IngestDocument-InlineDocument

content *string

metadata object

io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection

hosts *array

SubType string

Min items 1

basicAuth Elasticsearch-ElasticsearchConnection-BasicAuth

headers array

SubType string

pathPrefix string

strictDeprecationMode booleanstring

trustAllSsl booleanstring

Elasticsearch Embedding Store

connection *Elasticsearch-ElasticsearchConnection

indexName *string

type *object

Amazon Bedrock Model Provider

accessKeyId *string

modelName *string

secretAccessKey *string

type *object

modelType string

Default COHERE

Possible Values

COHERETITAN

​Ingest​Document

IngestDocument