From bdd72b5a85a0137723a23046734445b696f8ff46 Mon Sep 17 00:00:00 2001 From: Daniel Rosel Date: Fri, 28 Nov 2025 14:06:01 +0100 Subject: [PATCH] docs: what the pipeline is like now --- experiments/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/experiments/README.md b/experiments/README.md index e69de29..0e8d104 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -0,0 +1,8 @@ + +# Products +# Agents +# Pipeline + +Our pipeline technically should follow principles in a style like this: +- Each step should be defined as an inheriting child of an scikit pipeline step, the granularity of the steps is dictated by the following: a step should be a transformation, augmentation or computation independently, no single stage should run multiple in-itself. This way we can modularize properly all the components and track properly in airflow. A stage can be defined as an sklearn step but then must be transalted to a function that takes the context in our DAG of airflow. All parametrization must be done via contexts. +