Pipeline Service

Central orchestrator for Haystack pipeline management and execution.

The PipelineService provides:

  • Pipeline CRUD - Create, read, update, delete pipeline definitions

  • Validation - Validates pipeline definitions against ComponentRegistry

  • Building - Constructs Haystack Pipeline objects from YAML/dict definitions

  • Execution - Asynchronous pipeline execution

  • Caching - Version-based pipeline caching for performance

Key Dependencies

  • ComponentRegistry - Whitelist of allowed Haystack components

  • PipelineValidator - Validates pipeline structure and connections

  • DocumentContract - Enforces required metadata (celex, eId)

  • PipelineRepository - Database persistence for pipelines and executions

Execution Flow

  1. Pipeline definition validated via PipelineValidator

  2. Documents validated against DocumentContract

  3. Haystack Pipeline built from component definitions

  4. Pipeline executed with input documents

  5. Results persisted via repository layer