# ENVIRONMENTS & LIBRARIES
Per project, not global
python -m venv .venv
.venv\Scripts\activate
pip install {library}
pip freeze > requirements.txt
deactivate - Deactivate the virtual environment
Jupyter Notebook - Interactive environment for data science and machine learning. Starts as jupyter notebook in terminal, opens in browser. Create and share documents with live code, equations, visualizations, and narrative text. Great for data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.
# Libraries
pip install numpy - Numerical computing
pip install pandas - Data manipulation and analysis
pip install matplotlib seaborn - Data visualization
pip install scikit-learn - Machine learning
pip install tensorflow - Deep learning
pip install keras - Deep learning runs on top of TensorFlow
pip install flask - Web development
pip install selenium - Web scraping and browser automation
pip install - beautifulsoup4 - Web scraping
#
pandas
numpy
skickit-learn
beautifulsoup4
requests
flask
scheduler
subprocess
time
matplotlib
# MACHINE LEARNING
How it works:
Data Collection, Data Preprocessing, Model Selection, Training Model, Evaluation, Prediction, Deployment
# Algorithms:
Linear Regression: predicting a continious target
Logistic Regression: predicting a binary target
Decision Trees: tree-like model of decisions and their possible consequences
Random Forest: ensemble of decision trees
Support Vector Machines (SVM): finds the hyperplane that best separates classes in the feature space
# Supervised vs Unsupervised Learning
Supervised Learning: the model is trained on labeled data, where the input data is paired
Unsupervised Learning: the model is trained on unlabeled data, where the input data is not paired with any output labels. The goal is to find patterns or structure in the data. Clustering, Dimensionality Reduction.
K-Means Clustering: unsupervised learning algorithm that partitions data into K clusters based on feature similarity
Principal Component Analysis (PCA): unsupervised learning technique for dimensionality reduction, transforming data into a new coordinate system where the greatest variance is captured in the first few principal components
Neural Networks: computational models inspired by the human brain, used for deep learning tasks
#
# AI MODELS
Scikit-learn
Machinge Learning models (algorithms):
Regression: predicting continuous numeric values
Linear Regression: Finds relationship between independent variables and a continuous dependent variable by fitting a linear equation to observed data.
Ridge Regression: Linear regression with L2 regularization, which adds a penalty term to the loss function to prevent overfitting by shrinking the coefficients of less important features.
Classification:
Logistic Regression: Classification algorythm for binary (spam vs no spam)
Support Vector Machines (SVM): Finds the hyperplane that best separates classes in the feature space, maximizing the margin between them.
Random Forest: Ensemble learning method that constructs multiple decision trees during training and outputs the mode of the classes (classification) or mean prediction (regression) of the individual trees.
Clustering:
K-Means Clustering: Unsupervised learning algorithm that partitions data into K clusters based on feature similarity, minimizing the within-cluster sum of squares.
Hierarchical Clustering: Unsupervised learning method that builds a hierarchy of clusters by either merging or splitting them based on distance metrics.
Dimensionality Reduction
Traning a model:
Evaluating a model
True Positive (TP): The model correctly predicts the positive class.
True Negative (TN): The model correctly predicts the negative class.
False Positive (FP): The model incorrectly predicts the positive class (Type I error).
False Negative (FN): The model incorrectly predicts the negative class (Type II error).
# ML WORKFLOWS
pip instasll mlflow tensorflow
MlFlow: Manage ML Lifecycle. Tracking, projects, models, registry.
TensorFlow: Streamlining Library. Keras, Extended TFX, Serving
Data Collection
Data Preprocessing
Model Training
Model Evaluation
Model Tuning
MOdel Deployment
*
# DEEP LEARNING & NEURAL NETWORKS
Neural Network: Computational model inspired by the human brain, consisting of layers of interconnected nodes (neurons) that process and learn from data. Used for deep learning tasks such as image recognition, natural language processing, and more.
Layers: Input Layer, Hidden Layers, Output Layer
Activate functions: ReLU, Sigmoid, Tanh
Loss Function
Optimizer: Gradient Descent, Adam, RMSprop
- Libraries:
TensorFlow,
Keras,
PyTorch
How it works? Deep = Hidden
# COMPUTER VISION IMAGE PROCESSING AND AUTOMATION
OpenCV: Computer vision library for image and video processing, including tasks such as object detection, face recognition, and more.
Pillow: Python Imaging Library (PIL) fork for image processing tasks such as opening, manipulating, and saving various image file formats.
# VOICE RECOGNITION
SpeechRecognition: Library for performing speech recognition, supporting various engines and APIs for converting spoken language into text.
pip install SpeechRecognition
PyAudio: Library for audio input and output, often used in conjunction with SpeechRecognition for capturing audio from microphones.
pip install PyAudio
# CHATBOTS
Customer Support, Ecoommerce, Healthcare, Education, Entertainment, Banking, HR.
ChatterBot: Python library for building chatbots that can learn from conversations and generate responses based on input data.
Rasa: Open-source framework for building conversational AI, including chatbots and voice assistants, with natural language understanding and dialogue management capabilities.
Rasa NLU: Natural Language Understanding component for intent recognition and entity extraction.
Rasa Core: Dialogue management component for handling conversations and generating responses based on user input and context.
Expand:
Personalization
Integration with APIs and Services
Handling context
Multilingual SUpport
# WEB INTERACTION AND AUTOMATION
Selenium: Library for automating web browsers, allowing you to interact with web elements, perform actions, and scrape data from websites.
Web Scraping: The process of extracting data from websites using tools like BeautifulSoup, Scrapy, or Selenium to navigate and parse HTML content.
Testing
Task automation
Selenium: browser automation, element locators, waiting mechanisms, cross browser support
# DATA ANALAYSIS AND REPORTING
pandas: DataFrames, Series. Importing and exporting data, transforming, cleaning, grouping
Matplotlib: Data visualization library for creating static, animated, and interactive plots in Python. Line plots, bar charts, histograms, scatter plots,
Seaborn: On top of Matplotlib, provides a high-level interface for drawing attractive and informative statistical graphics. Heatmaps, violin plots, pair plots, regression plots.
# CLOUD AUTOMATION
AWS. Compute, Storage, Databases, Networking. Boto3 AWS SDK for Python. Automate AWS services and resources. Create, configure, and manage AWS services programmatically.
Google Cloud Platform: Compute, cloud functions, storage, bigdata, networking.
Azure:
# IOT: Internet of Things
GPIO: General Purpose Input/Output. Raspberry Pi, Arduino. Control and interact with sensors, actuators, and other electronic components.
Pyserial: Library for serial communication in Python, allowing you to interface with devices connected via serial ports (e.g., Arduino, microcontrollers).
MQTT: Lightweight messaging protocol for IoT devices, enabling communication between devices and servers. Libraries like paho-mqtt can be used in Python to implement MQTT clients.
Flask: Web framework for building web applications and APIs, often used in IoT projects to create dashboards or interfaces for monitoring and controlling devices.
Home Assistant: Open-source platform for home automation, allowing you to control smart devices and create automation rules. Python can be used to extend Home Assistant's functionality through custom components and scripts.
# DATA ENGINEERING
# Etl Extract Transform Load
Pandas
Apache Airflow: Task scheduling, Pipeline orchesstration, Error Handling.
SqlAlqchemy: Database Abstraction, ORM, Query Building, Connection Management.
Dask: Interactive Dashboards, Data Visualization, Web Applications. Parallel computing.
Prefect
# SECURING AUTOMATION WORKFLOWS
Best practices: Use Version Control Git, Limit Code Access, Virtual environments, software up to date
Sensitive DAta: Never hardcode secrets, use secrect management tools, environment variables, secure storage solutions
Authentication and Authorization: OAuth2.0, Api keys, JWT, avoid user/pwd, Use least privilege principle, Role-based access control (RBAC)
# MONITORING
Key Aspects: Availability, performance, error handling, logs,
Tools: Nagios, Prometheus, Grafana, Datalog, Zabbix, Cloud Native
# FUTURE TRENDS in AI and Automation
1- Machine Learning and Deep Learning Advancements: Improved algorithms, architectures, and techniques for more accurate predictions and insights.
