Daniel
@indent4.bsky.social
🤖 ML Engineer → Developing Apps & Sharing ML/AI Insights 🚀 | Always Learning #MachineLearning #DataScience #AI
Follow me for insights on real-world ML🚀
If this thread helped, drop a like 📷 or repost 📷
If this thread helped, drop a like 📷 or repost 📷
February 21, 2025 at 5:31 PM
Follow me for insights on real-world ML🚀
If this thread helped, drop a like 📷 or repost 📷
If this thread helped, drop a like 📷 or repost 📷
Why use the CPO Pattern?
✅ Scalability – Add new models & components easily.
✅ Maintainability – Elements are decoupled making it easy to change, test and debug individual parts without breaking the entire system.
✅ Reusability – Reuse components across different ML projects.
✅ Scalability – Add new models & components easily.
✅ Maintainability – Elements are decoupled making it easy to change, test and debug individual parts without breaking the entire system.
✅ Reusability – Reuse components across different ML projects.
February 21, 2025 at 5:31 PM
Why use the CPO Pattern?
✅ Scalability – Add new models & components easily.
✅ Maintainability – Elements are decoupled making it easy to change, test and debug individual parts without breaking the entire system.
✅ Reusability – Reuse components across different ML projects.
✅ Scalability – Add new models & components easily.
✅ Maintainability – Elements are decoupled making it easy to change, test and debug individual parts without breaking the entire system.
✅ Reusability – Reuse components across different ML projects.
3️⃣ Orchestrators 🤖
The orchestrator is responsible for managing the pipeline execution.
This allows us to easily switch components (e.g., swap XGBoost for RandomForest).
The orchestrator is responsible for managing the pipeline execution.
This allows us to easily switch components (e.g., swap XGBoost for RandomForest).
February 21, 2025 at 5:31 PM
3️⃣ Orchestrators 🤖
The orchestrator is responsible for managing the pipeline execution.
This allows us to easily switch components (e.g., swap XGBoost for RandomForest).
The orchestrator is responsible for managing the pipeline execution.
This allows us to easily switch components (e.g., swap XGBoost for RandomForest).
2️⃣ Pipelines 🛤️
A TrainingPipeline ties together components into a single workflow.
Pipelines ensure a clear flow of data, from raw input to final evaluation.
A TrainingPipeline ties together components into a single workflow.
Pipelines ensure a clear flow of data, from raw input to final evaluation.
February 21, 2025 at 5:31 PM
2️⃣ Pipelines 🛤️
A TrainingPipeline ties together components into a single workflow.
Pipelines ensure a clear flow of data, from raw input to final evaluation.
A TrainingPipeline ties together components into a single workflow.
Pipelines ensure a clear flow of data, from raw input to final evaluation.
1️⃣ Components 📦
Each key function (data import, preprocessing, training, evaluation) is wrapped in its own class.
This makes it reusable, testable, and modular.
Example: A DataImporter class to load data!
Each key function (data import, preprocessing, training, evaluation) is wrapped in its own class.
This makes it reusable, testable, and modular.
Example: A DataImporter class to load data!
February 21, 2025 at 5:31 PM
1️⃣ Components 📦
Each key function (data import, preprocessing, training, evaluation) is wrapped in its own class.
This makes it reusable, testable, and modular.
Example: A DataImporter class to load data!
Each key function (data import, preprocessing, training, evaluation) is wrapped in its own class.
This makes it reusable, testable, and modular.
Example: A DataImporter class to load data!
DataFramed, Super Data Science, The Data Scientist Show, Ken's Nearest Neighbours
February 19, 2025 at 3:34 PM
DataFramed, Super Data Science, The Data Scientist Show, Ken's Nearest Neighbours
Follow me for insights on real-world ML🚀
If this thread helped, drop a like 📷 or repost 📷🔁
If this thread helped, drop a like 📷 or repost 📷🔁
February 19, 2025 at 3:32 PM
Follow me for insights on real-world ML🚀
If this thread helped, drop a like 📷 or repost 📷🔁
If this thread helped, drop a like 📷 or repost 📷🔁
EDA isn’t just a formality—it’s the difference between a strong ML model and garbage results. 🚀
✅ Understand your data
✅ Spot issues before modeling
✅ Find strong predictors
These steps serve as a starting point. Analyse further depending on the needs of your project.
✅ Understand your data
✅ Spot issues before modeling
✅ Find strong predictors
These steps serve as a starting point. Analyse further depending on the needs of your project.
February 19, 2025 at 3:32 PM
EDA isn’t just a formality—it’s the difference between a strong ML model and garbage results. 🚀
✅ Understand your data
✅ Spot issues before modeling
✅ Find strong predictors
These steps serve as a starting point. Analyse further depending on the needs of your project.
✅ Understand your data
✅ Spot issues before modeling
✅ Find strong predictors
These steps serve as a starting point. Analyse further depending on the needs of your project.
💡 Why?
👉Detect multicollinearity (too many correlated features = model confusion)
👉Uncover hidden patterns
👉Reduce dimensionality if needed (PCA can help)
📊 Example: If two features are 99% correlated, you probably don’t need both in your model.
👉Detect multicollinearity (too many correlated features = model confusion)
👉Uncover hidden patterns
👉Reduce dimensionality if needed (PCA can help)
📊 Example: If two features are 99% correlated, you probably don’t need both in your model.
February 19, 2025 at 3:32 PM
💡 Why?
👉Detect multicollinearity (too many correlated features = model confusion)
👉Uncover hidden patterns
👉Reduce dimensionality if needed (PCA can help)
📊 Example: If two features are 99% correlated, you probably don’t need both in your model.
👉Detect multicollinearity (too many correlated features = model confusion)
👉Uncover hidden patterns
👉Reduce dimensionality if needed (PCA can help)
📊 Example: If two features are 99% correlated, you probably don’t need both in your model.
4️⃣ Multivariate Analysis 🔄 (Relationships Between Features)
Time to go deeper and check how features interact with each other.
🔥 Correlation heatmaps (sns.heatmap(df.corr(), annot=True))
🔥 Pairplots (sns.pairplot(df))
🔥 PCA (if needed) (PCA().fit_transform(df))
Time to go deeper and check how features interact with each other.
🔥 Correlation heatmaps (sns.heatmap(df.corr(), annot=True))
🔥 Pairplots (sns.pairplot(df))
🔥 PCA (if needed) (PCA().fit_transform(df))
February 19, 2025 at 3:32 PM
4️⃣ Multivariate Analysis 🔄 (Relationships Between Features)
Time to go deeper and check how features interact with each other.
🔥 Correlation heatmaps (sns.heatmap(df.corr(), annot=True))
🔥 Pairplots (sns.pairplot(df))
🔥 PCA (if needed) (PCA().fit_transform(df))
Time to go deeper and check how features interact with each other.
🔥 Correlation heatmaps (sns.heatmap(df.corr(), annot=True))
🔥 Pairplots (sns.pairplot(df))
🔥 PCA (if needed) (PCA().fit_transform(df))
💡 Why?
👉 Find strong predictors 📈
👉 Identify non-linear relationships (Choose a non-linear model)
👉 Detect leakage (some features might be too correlated with the target!)
📊 Example: A high correlation might mean a strong predictor… or data leakage. Always check!
👉 Find strong predictors 📈
👉 Identify non-linear relationships (Choose a non-linear model)
👉 Detect leakage (some features might be too correlated with the target!)
📊 Example: A high correlation might mean a strong predictor… or data leakage. Always check!
February 19, 2025 at 3:32 PM
💡 Why?
👉 Find strong predictors 📈
👉 Identify non-linear relationships (Choose a non-linear model)
👉 Detect leakage (some features might be too correlated with the target!)
📊 Example: A high correlation might mean a strong predictor… or data leakage. Always check!
👉 Find strong predictors 📈
👉 Identify non-linear relationships (Choose a non-linear model)
👉 Detect leakage (some features might be too correlated with the target!)
📊 Example: A high correlation might mean a strong predictor… or data leakage. Always check!
3️⃣ Bivariate Analysis 📊
See how features relate to the target variable
📈 Numerical:
Correlation heatmaps (df.corr())
Scatter plots
📊 Categorical:
Box plots (sns.boxplot(x='category', y='target', data=df))
Grouped means (df.groupby('category')['target'].mean())
See how features relate to the target variable
📈 Numerical:
Correlation heatmaps (df.corr())
Scatter plots
📊 Categorical:
Box plots (sns.boxplot(x='category', y='target', data=df))
Grouped means (df.groupby('category')['target'].mean())
February 19, 2025 at 3:32 PM
3️⃣ Bivariate Analysis 📊
See how features relate to the target variable
📈 Numerical:
Correlation heatmaps (df.corr())
Scatter plots
📊 Categorical:
Box plots (sns.boxplot(x='category', y='target', data=df))
Grouped means (df.groupby('category')['target'].mean())
See how features relate to the target variable
📈 Numerical:
Correlation heatmaps (df.corr())
Scatter plots
📊 Categorical:
Box plots (sns.boxplot(x='category', y='target', data=df))
Grouped means (df.groupby('category')['target'].mean())
💡 Why?
👉 Detect skewness (maybe you need log transformations)
👉Spot outliers
👉 Identify imbalanced categories
📊 Example: If a feature is heavily skewed, your linear models might struggle—fix it early!
👉 Detect skewness (maybe you need log transformations)
👉Spot outliers
👉 Identify imbalanced categories
📊 Example: If a feature is heavily skewed, your linear models might struggle—fix it early!
February 19, 2025 at 3:32 PM
💡 Why?
👉 Detect skewness (maybe you need log transformations)
👉Spot outliers
👉 Identify imbalanced categories
📊 Example: If a feature is heavily skewed, your linear models might struggle—fix it early!
👉 Detect skewness (maybe you need log transformations)
👉Spot outliers
👉 Identify imbalanced categories
📊 Example: If a feature is heavily skewed, your linear models might struggle—fix it early!
2️⃣ Univariate Analysis 🔍
Let’s get to know each feature individually.
🔍 What to check?
📊 Numerical: Histograms, box plots (df['feature'].hist())
📊 Categorical: Value counts, bar plots (df['feature'].value_counts())
📊 Outliers: Box plots (sns.boxplot(x=df['feature']))
Let’s get to know each feature individually.
🔍 What to check?
📊 Numerical: Histograms, box plots (df['feature'].hist())
📊 Categorical: Value counts, bar plots (df['feature'].value_counts())
📊 Outliers: Box plots (sns.boxplot(x=df['feature']))
February 19, 2025 at 3:32 PM
2️⃣ Univariate Analysis 🔍
Let’s get to know each feature individually.
🔍 What to check?
📊 Numerical: Histograms, box plots (df['feature'].hist())
📊 Categorical: Value counts, bar plots (df['feature'].value_counts())
📊 Outliers: Box plots (sns.boxplot(x=df['feature']))
Let’s get to know each feature individually.
🔍 What to check?
📊 Numerical: Histograms, box plots (df['feature'].hist())
📊 Categorical: Value counts, bar plots (df['feature'].value_counts())
📊 Outliers: Box plots (sns.boxplot(x=df['feature']))
💡 Why? This helps spot early issues (wrong dtypes, missing data, duplicates, incorrect values) before they ruin your model.
📊 Example: Checking for missing values can reveal if you need imputation or feature removal.
📊 Example: Checking for missing values can reveal if you need imputation or feature removal.
February 19, 2025 at 3:32 PM
💡 Why? This helps spot early issues (wrong dtypes, missing data, duplicates, incorrect values) before they ruin your model.
📊 Example: Checking for missing values can reveal if you need imputation or feature removal.
📊 Example: Checking for missing values can reveal if you need imputation or feature removal.
1️⃣ Understanding Data Structure & Metadata 🏗️
Before diving in, know your data.
🔍 What to check?
✅ Dataset shape (df.shape)
✅ Data types (df.dtypes)
✅ Descriptive statistics (df.describe())
✅ Missing values (df.isnull().sum())
✅ Unique values & duplicates
Before diving in, know your data.
🔍 What to check?
✅ Dataset shape (df.shape)
✅ Data types (df.dtypes)
✅ Descriptive statistics (df.describe())
✅ Missing values (df.isnull().sum())
✅ Unique values & duplicates
February 19, 2025 at 3:32 PM
1️⃣ Understanding Data Structure & Metadata 🏗️
Before diving in, know your data.
🔍 What to check?
✅ Dataset shape (df.shape)
✅ Data types (df.dtypes)
✅ Descriptive statistics (df.describe())
✅ Missing values (df.isnull().sum())
✅ Unique values & duplicates
Before diving in, know your data.
🔍 What to check?
✅ Dataset shape (df.shape)
✅ Data types (df.dtypes)
✅ Descriptive statistics (df.describe())
✅ Missing values (df.isnull().sum())
✅ Unique values & duplicates
Should you separate your SQL servers into different resource groups? This depends on your specific use case. if you have a small number of SQL databases and they are tightly integrated with your app then keeping them in the same resource group as your app might be the way to go.
February 10, 2025 at 12:07 AM
Should you separate your SQL servers into different resource groups? This depends on your specific use case. if you have a small number of SQL databases and they are tightly integrated with your app then keeping them in the same resource group as your app might be the way to go.
It's recommended to keep Data Lakes and Blob Storage in a different resource group than your application or project resource group. This limits attack potential security issues and eases resource & cost management.
February 10, 2025 at 12:07 AM
It's recommended to keep Data Lakes and Blob Storage in a different resource group than your application or project resource group. This limits attack potential security issues and eases resource & cost management.
It's a good idea to keep data associated with different projects in dedicated Blob storage containers.
February 10, 2025 at 12:06 AM
It's a good idea to keep data associated with different projects in dedicated Blob storage containers.
Create dedicated resource groups for each large project or application
February 10, 2025 at 12:06 AM
Create dedicated resource groups for each large project or application