Syllabus Outline

This subchapter covers the foundational concepts of how data is represented and preprocessed before feeding it into machine learning models. Tabular data is structured in columns (features) and rows (samples). - Properties: Every row (record) shares the exact same set of properties (column headers). - Access: Rows can be retrieved using unique identifiers (queries through key values). - Scale: Tabular databases have a virtually infinite range for mass data storage. Using the Iris Dataset ($d = 4$ features) as an introductory example: - Sample: A single data instance (e.g., a specific flower). - Features: Measurable properties (columns). - **Feature Vector (\mathbf{x**)}: A vector representing all features of a sample, e.g., $\mathbf{x} = (6.7, 3.1, 4.7, 1.5)^T$. - Class: The target category (e.g., setosa, virginica, versicolor). - Label ($y$): The ground-truth class of the sample (e.g., $y = \text{versicolor}$). High-dimensional data (hundreds/thousands of features) cannot be visualized directly. - Goal: Project data into lower dimensions (e.g., 2D or 3D) while preserving maximum information, making it extremely useful for visualizing high-dimensional datasets. - Algorithms: Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE). Note that clustering algorithms like k-Means are for grouping, not for dimensionality reduction. - Information Loss: Reducing dimensionality always leads to some loss of information. - Prerequisite: Applying dimensionality reduction is not a strict requirement before training a model (many models can train directly on high-dimensional data). Images are represented as 3D tensors: $\text{Height} \times \text{Width} \times \text{Channels}$. - Channels: Color images typically have 3 channels (Red, Green, Blue). Summing them creates the final colored image. - Color Depth: Number of possible intensity values per pixel per channel. - 8-bit depth: $2^8 = 256$ values (integer range $[0, 255]$). - 16-bit depth: $2^{16} = 65,536$ values. [Image] Artificially generating new training samples by modifying existing ones. - Techniques: Rotation, flipping, cropping, blurring, noise, input dropout, color jitter. - Pros: Increases dataset size, reduces overfitting, and increases robustness. - Cons: Can introduce artifacts, can change the task (e.g., flipping a handwritten digit `6' makes it look like another class), and is highly dependent on the task/model. Sequences are ordered elements: $a_1, a_2, \dots, a_n$. - Types: Time series (ordered in time, e.g., weather, stocks) or non-temporal ordered data (molecules, language syntax). - Standard Feed-Forward Neural Networks (FFNNs): - Require fixed-length inputs (variable-length sequences must be preprocessed/padded). - Process elements independently; thus, the relation of elements in the sequence is lost. \subsection{Subchapter 1.2: Unsupervised Machine Learning}

This subchapter covers the key techniques used to regularize deep neural networks, improve generalization, and prevent overfitting. Artificially modifying training examples by changing an irrelevant input property. - Goal: Help the model ignore irrelevant variations (e.g. flipping a cat horizontally, rotation, scaling) and focus on relevant features (shape, edges). - Effect: Increases the effective dataset size and improves model robustness. Dropout randomly deactivates neurons during training to prevent co-adaptation. - Training Time: Randomly omit (set to 0) $p$ of the hidden units. Scale the remaining weights by $\frac{1}{1-p}$ to keep the output expectation constant: - If $p=0.20$, scale weights by $1.25$. - If $p=0.50$, scale weights by $2.00$. - At Test Time: Use the full network (all units active, no scaling). - Effect: Acts as a strong regularizer. Leads to a higher, noisier training loss, but significantly reduces validation loss (prevents overfitting). [Image] [Image] Standardizing data inside hidden layers to speed up training. - Concept: Standardizing input features is easy (precompute training set mean/std once). Standardizing hidden layers is hard because distributions shift with weight updates (Internal Covariate Shift). - During Training: Normalize each mini-batch of training examples by its own mini-batch mean and standard deviation after every convolution or fully-connected layer. - During Testing: Use running statistics (mean and standard deviation) computed over the entire training set. - Effect: Speeds up optimization, acts as a regularizer due to batch-wise noise. Penalizing large weights by adding a term to the loss function: - L1 Regularization: Adds the sum of absolute weights: [ L_{\text{new}} = L + \lambda \sum_i |w_i| $- **L2 Regularization (Weight Decay)**: Adds the sum of squared weights: \[ L_{\text{new}} = L + \lambda \sum_i w_i^2$ Commonly referred to as weight decay in deep learning. \subsection{Subchapter 2.2: Deep Networks & Transfer Learning}

This subchapter introduces sequence data, outlines the architectural limitations of feed-forward networks on sequential inputs, and explains the basic recurrent neural network (RNN) structure, equations, and properties. - Fixed Input size: The input vector $\mathbf{x}$ must have a fixed dimensionality, making variable-length sequences (e.g., sentences with different lengths) difficult to handle. - Order/Relation Loss: FFNNs process elements independently, discarding the sequence order and temporal relations. - Workaround Limits: Concatenating inputs in a sliding window increases input size and parameters, making optimization intractable. [Image] - Definition: A sample $s$ is a sequence of length $T$ with $D$ features at each timestep $t$: [ s = (\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T) \quad \text{with} \quad \mathbf{x}_t \in \mathbb{R}^D $- **Dataset Constraints**: - Feature dimension$D$is **constant** across all samples in the dataset. - Sequence length$T$**can vary** between samples. - For a specific sample, the feature dimension$D$can be greater than the sequence length$T$. - **Representation Example**: Sentence "this book is great" ($T = 4$words) represented using a vocabulary of size$D = 5$as a sequence of one-hot encoded vectors. To process sequences, RNNs feed inputs timestep-by-timestep and loop previous outputs back into inputs: - **Hidden State ($\mathbf{h**_t)}: The network's internal state (memory) at timestep t$, where$\mathbf{h}_t \in \mathbb{R}^H$. - **Recursive Step**: The hidden state is computed as: \[ \mathbf{h}_t = f(\mathbf{x}_t, \mathbf{h}_{t-1}) \quad \text{with} \quad \mathbf{h}_0 = \mathbf{0}$ - Shared Weights: The same weight matrix $W$ is reused (shared) across all timesteps. [Image] The hidden state update equation is: [ \mathbf{h}t = f\left( W \cdot \begin{bmatrix} \mathbf{h}{t-1} \mathbf{x}t \end{bmatrix} + \mathbf{b} \right) $Where$f$is typically$\tanh$and$\begin{bmatrix} \mathbf{h}{t-1} \mathbf{x}t \end{bmatrix}$is the concatenated state. - **Vector and Matrix Dimensions**: - Input vector$\mathbf{x}t$: size$D \times 1$. - Previous hidden state$\mathbf{h}{t-1}$: size$H \times 1$. - Concatenated vector$\begin{bmatrix} \mathbf{h}{t-1} \mathbf{x}_t \end{bmatrix}$: size$(H+D) \times 1$. - Weight matrix$W$: size$H \times (H+D)$. - Bias$\mathbf{b}$: size$H \times 1$. - Output hidden state$\mathbf{h}_t$: size$H \times 1$. [Image] RNN architectures are classified into 5 main types based on sequence length inputs ($T_x$) and output predictions ($T_y$): - **One-to-One ($T_x = T_y = 1$)**: Standard Feed-Forward Neural Network. Given input length$T_x = 1$, output length is$T_y = 1$. - **One-to-Many ($T_x = 1, T_y > 1$)**: One input generates a sequence (e.g., Music Generation, Image Captioning). - **Many-to-One ($T_x > 1, T_y = 1$)**: A sequence generates a single label (e.g., Sentiment Classification). - **Many-to-Many (Equal,$T_x = T_y$)**: Sequence input produces sequence output of identical length (e.g., Name Entity Recognition, Video Classification frame-by-frame). - **Many-to-Many (Different,$T_x \ne T_y$)**: Sequence input produces sequence output of a different length (e.g., Machine Translation, Sequence-to-Sequence models). For all types, the overall loss$L(\hat{\mathbf{y}}, \mathbf{y})$is defined as the sum of individual losses per output timestep: \[ L(\hat{\mathbf{y}}, \mathbf{y}) = \sum_{t=1}^{T_y} L(\hat{\mathbf{y}}_t, \mathbf{y}_t)$ - Concept: RNNs are theoretically Turing complete, meaning they can represent any program (Siegelmann & Sontag, 1991). - Practical Challenge: The primary challenge is not representation capability, but finding the representation via gradient descent (learning). - Shared Weight Matrix: For each timestep, the exact same (shared) weight matrix is used. - Length Independence: The size of the weight matrix is independent of the sequence length. - Different Lengths: For sequences of different lengths, we do not need to learn multiple weight matrices. The same learned weight matrix is applied across all sequence lengths. Problem: Assume you have a sequence of 8 elements, and each element has 10 features ($T=8, D=10$). If you process it using a standard, single-layer RNN with a hidden state size of 6 ($H=6$), what is the size of the first RNN output? - Solution: The size of the RNN output at the first timestep $\mathbf{h}_1$ (and every other timestep $\mathbf{h}_t$) is exactly equal to the hidden state size $H$. - Answer: The size is 6. (It is independent of sequence length 8 or feature size 10). \subsection{Subchapter 3.2: Unrolling RNNs & BPTT}

This subchapter covers the introduction to Natural Language Processing (NLP), linguistic concepts like Chomsky's Universal Grammar, and early AI attempts at language modeling, specifically Weizenbaum's ELIZA and the Connectionist Model. Language is a highly complex, evolved system closely connected to human cognition: - Cognitive Link: Language allows humans to express, communicate, and structure thought. - Natural Language Processing (NLP): The computational modeling of language to infer structure, represent thoughts, and enable human-machine communication. Noam Chomsky (1965) proposed a highly influential (and controversial) theory of linguistics: - Innate Rules: Newborns possess an innate, genetically determined set of structural rules common to all human languages. - Constraints: During first-language acquisition, children do not learn grammar from scratch; rather, they map the specific vocabulary and rules of their native tongue onto these pre-existing, universal grammatical constraints. Joseph Weizenbaum developed ELIZA, one of the earliest conversational agents: - Role: Mimicked a Rogerian psychotherapist. - Mechanism: Relied strictly on hardcoded, hand-crafted rules and keyword pattern matching to reflect user inputs back as questions (e.g., matching "my boyfriend" and responding with "Is it important to you that your boyfriend..."). - Limit: It possessed no actual semantic understanding or representation of language. In contrast to rule-based systems, connectionists argued that language rules emerge from statistical learning: - Core Idea: Neural networks can learn linguistic structures directly from examples, without hardcoded grammatical rules. - Past Tense Learning: A neural network was trained on English verbs and successfully learned: - Regular Verbs: E.g., walk $\to$ walked, play $\to$ played. - Irregular Verbs: E.g., sing $\to$ sang, go $\to$ went. - Similarity Patterns: E.g., group patterns like sing/sang, ring/rang, drink/drank. - Significance: It demonstrated that complex, rule-like language behavior can emerge naturally from learned statistical patterns in data. - Limitations: Used simplified training settings, artificial data representations, and did not fully replicate human developmental stages. \subsection{Subchapter 4.2: Word Representations & Embeddings}

This subchapter introduces the fundamentals of drug discovery, the pharmacology paradigm, the drug development timeline, the cost structures (including Eroom's Law), the massive molecular search space, and traditional High-Throughput Screening (HTS) approaches. The biological and chemical basis of modern drug discovery: - Drug: A chemical substance of known structure, other than a nutrient or essential dietary ingredient, which produces a biological effect when administered to a living organism. - Pharmacology Paradigm: The biological effect of a drug is strongly determined by its molecular structure. - Targets: Diseases are typically associated with pathological molecular pathways. The target molecules responsible for these pathways are frequently proteins (humans have approximately 20,000 to 25,000 proteins). - Proteins: Large biomolecules consisting of chains of 50 to 2,000 amino acids. - Task: - Identify the target biomolecule (often a protein) associated with a disease. - Find a small molecule (drug candidate) that interacts with (binds to) that target to produce a useful therapeutic effect. Bringing a drug to market is a long, high-risk process: - Funnel Model: - Drug Discovery & Preclinical: Starts with $\sim$10,000 compounds. Takes $\sim$6.5 years. Filters down to $\sim$250 compounds for preclinical testing. - Clinical Trials: Filters down to $\sim$5 candidate compounds tested on humans. Takes $\sim$7 years. - Review: FDA/EMA review of the clinical data. Takes $\sim$1.5 years. - Result: Only 1 approved drug reaches the market from the initial 10,000. [Image] - Financial Cost: Bringing a new drug to market is extremely expensive, with typical estimates ranging from $1 billion to over $2 billion. - Success Rate: Only about 10% of drug candidates that enter clinical trials are eventually approved. - Discovery Share: The initial drug discovery stage contributes to about 30% of the total cost. Improving this stage using machine learning can greatly increase cost efficiency. - Eroom's Law (Moore's Law spelled backward): The number of new drugs approved per billion dollars of R&D spending has roughly halved every 9 years. This indicates a severe decline in R&D efficiency over time, raising the need for computational solutions. Finding a drug candidate is literally like finding a "needle in a haystack": - Search Space Size: - If the maximum number of heavy atoms in a molecule is 13: $\sim 9.77 \times 10^8$ (977 million) possible compounds. - If the maximum number of heavy atoms is 17: $\sim 1.66 \times 10^{11}$ (166 billion) possible compounds. - For typical drug-like molecules, the maximum number of heavy atoms is $\sim$40: this yields an astronomical search space of $10^{60}$ possible molecules. - High-Throughput Screening (HTS): The traditional approach where millions of candidate molecules are physically tested in biological assays. HTS is an expensive and time-consuming physical process. - Need for Machine Learning: By training neural networks to predict bioactivity, we can make educated guesses and computationally screen candidates (Virtual Screening), significantly narrowing down the physical testing requirements. In Virtual Screening, a trained neural network predicts whether a molecule in a database is likely to bind/react with the target of interest. \subsection{Subchapter 5.2: QSAR & Small Molecules}

This subchapter introduces Reinforcement Learning (RL), contrasts it with Supervised and Unsupervised Learning, mathematically formulates the problem using Markov Decision Processes (MDPs), details the Q-Value Function and Q-learning update updates, analyzes the exploration-exploitation trade-off, and walks through the classic Taxi toy environment. Machine Learning paradigms differ fundamentally in their learning signals and objectives: - Supervised Learning: Learning a mapping $x \to y$ from input values and corresponding target labels (predictive modeling: classification/regression). - Unsupervised Learning: Finding underlying hidden structure from unlabeled inputs $x$ (clustering, dimensionality reduction, feature learning, density estimation). - Reinforcement Learning (RL): Learning how to act to maximize cumulative rewards. An agent interacts with an active environment by executing actions and receiving numeric reward signals. An MDP is the standard mathematical formulation of the reinforcement learning problem. It is formally defined by a 4-tuple: - States ($S$): The set of all possible states $s_i$ in the environment. - Actions ($A$): The set of all possible actions $a_j$ available to the agent. - Transition Probability ($P_a(s, s')$): The probability of transitioning from state $s$ to state $s'$ when executing action $a$ at timestep $t$: [ P_a(s, s') = \Pr(s_{t+1} = s' \mid s_t = s, a_t = a) $- **Reward Function ($R_a(s, s')$)**: The immediate numeric reward received after executing action$a$and transitioning from state$s$to state$s'$. - MDP rewards can be **positive, negative, or zero**. - The reward is **associated with the transition** from one state to another state. **Learned Policy ($\pi$)**: A mapping from the state space to the action space:$a = \pi(s)$. It serves as the "heart and brain" of the agent, determining which action to execute in each state. The goal of RL is to find the **optimal policy ($\pi^*$)** that maximizes the expected cumulative reward. [Image] - **Q-Value Function ($Q(s, a)$)**: The expected cumulative future reward (return), discounted by$\gamma$, when executing action$a$in state$s$and following the optimal policy thereafter. - **Q-Learning**: An implementation of reinforcement learning. Tabular Q-learning can theoretically be applied to **any** MDP. - **Q-Learning Update Rule**: An iterative, model-free algorithm to approximate optimal Q-values: \[ Q(s_t, a_t) \leftarrow (1 - \alpha) \cdot Q(s_t, a_t) + \alpha \cdot \left[ r_t + \gamma \max_a Q(s_{t+1}, a) \right]$ Where: - $\alpha \in [0, 1]$ (Learning Rate): Specifies the update step size. - $\alpha = 0$: No update at all is performed (the agent ignores new experiences and learns nothing). - $0 < \alpha < 1$: Both the old Q-value as well as the new information will be used in the update step. - $\alpha = 1$: The agent completely discards the old estimate, overwriting it with the new temporal difference target. - $\gamma \in [0, 1]$ (Discount Factor): Specifies the significance of future rewards. - $\gamma = 0$: "Myopic" (short-term) agent that only maximizes the immediate reward $r_t$. - $\gamma = 1$: Far-sighted (long-term) agent that places full weight on future cumulative rewards. - $r_t$: The immediate reward received when transitioning from $s_t$ to $s_{t+1}$ using action $a_t$. - $\max_a Q(s_{t+1}, a)$: The estimated optimal future value in the next state. - Definition: An optimal policy $\pi^*$ maximizes the expected cumulative return from any starting state. - Method: The optimal policy may be obtained via Q-learning. - Non-Uniqueness: There does not necessarily exist exactly one optimal policy. Multiple policies can be optimal if they achieve the same maximal expected cumulative return. - Objective: Applying the optimal policy maximizes the discounted sum of future rewards. It does not minimize rewards, nor does it simply maximize the immediate reward (unless $\gamma=0$). Scenario: Consider a 2D grid world of size 5x5. A robot (the agent) is placed randomly on this grid and must move to the target destination. - Actions: 4 movements (up, down, left, right). If the robot tries to go off the grid, it remains at the same location. - Rewards: Each action has a reward of $-2$ (negative reward). Reaching the target awards $+15$ (positive reward). - Key Properties: - Optimal Policy steps: The maximum number of steps of an optimal policy starting from any location is 8 (moving from one corner to the opposite corner takes $(5-1)+(5-1)=8$ steps). - Optimal Policy reward: An optimal policy does not always have a positive cumulative reward. Starting 8 steps away, the optimal cumulative reward is $8 \cdot (-2) + 15 = -16 + 15 = -1$ (negative). - Policy Failure: With a bad policy, it might happen that the target destination is not reached at all (e.g. infinite loops). Thus, the maximum steps of any policy is infinite (not limited to 100). - The Dilemma: The agent must balance: - Exploration: Trying actions with uncertain outcomes to discover better policies. - Exploitation: Choosing the action currently estimated to yield the highest reward. - $\epsilon$-Greedy Selection: A simple, widely-used policy to balance this trade-off: [ a_t = $\begin{cases} \text{random action} & \text{with probability } \epsilon

\arg\max_a Q(s_t, a) & \text{with probability } 1 - \epsilon
\end{cases}$
$
$\epsilon$ is often decayed over time (high exploration initially, high exploitation as learning stabilizes).
- **Episodes**: An interaction sequence starting from an initial state and running until a terminal state (win/loss) is reached. If no terminal state is reached, the episode is truncated after a maximum number of steps.

The Taxi domain is a classic reinforcement learning environment used to test tabular Q-learning: - State Space Size: Calculated as the Cartesian product of sub-states: [ 25 \text{ grid cells} \times 5 \text{ passenger locations} \times 4 \text{ drop-off locations} = 500 \text{ states} $(Passenger locations include R, G, B, Y, and inside the taxi). - **Action Space**: 6 discrete actions: `move south`, `move north`, `move east`, `move west`, `pick up passenger`, and `drop off passenger`. - **Q-Table Size**:$500 \text{ states} \times 6 \text{ actions} = 3,000$values. - **Reward Structure**: -$+20$for correct drop-off. -$-10$for incorrect pick-up or drop-off (penalties). -$-1$for all other actions (including driving into walls to encourage path efficiency). - **Update Example**: For state$s_{12}$, action$a_1$(`north`), reward$r = -1$, next state$s_{174}$, learning rate$\alpha = 0.1$, and discount factor$\gamma = 1$: \[ Q(s_{12}, a_1) \leftarrow (1 - 0.1) \cdot 0 + 0.1 \cdot [-1 + 1 \cdot \max_a Q(s_{174}, a)] = 0.1 \cdot (-1 + 0) = -0.1$ [Image] \subsection{Subchapter 6.2: Q-learning & Policies}