stack
https://www.reddit.com/r/mlops/comments/1ephb1p/whats_your_mlops_stack/
https://ml-ops.org/content/state-of-mlops
https://docs.databricks.com/en/machine-learning/mlops/mlops-stacks.html
https://github.com/kelvins/awesome-mlops
usingdbt +feast +teradata
This article explains how feast is used: simplifying and standardizing the retrieval of a training dataset for cross project management. https://developers.teradata.com/quickstarts/manage-data/getting-started-dbt-feast-teradata-pipeline/
interested compatible FOSS stack components
| type | tool |
|---|---|
| data source | sparkkafkapulsarapiwarehousepostgresql |
| data source transform orchestration | dbtsqlmesh |
| data source/model catalog | datahubamundsen-ioOpenMetadata |
| feature store/share/documentation | feast |
| model build/train | sklearn,spark,xgboostray |
| model code dev | mlflow |
| model registry | mlflow |
| model serve | kubeflow |
| model orchestration | airflow,argoflow |
| IaC | Terraform |
| CICD | githubgitlab workflowjenkins |
| data reporting | apache-supersetmetabaseredashplotly-dash |
| service monitor | prometheusgrafana |
lineage visibility:
datahub supports sourcing lineage from the following:
- majority of data sources listed above
- feature storefeast
- model registrymlflow
- model orchestrationairflow