Learning from User-generated Data - Learning from User-generated Data
Unit 1
7 min read
This subchapter establishes the foundational definitions of User-generated Data (UGD), User-generated Content (UGC), the evolution of web paradigms, Big Data characteristics, and key analytical tasks and challenges in LUD.
UGD is data created by regular, non-professional users of web technology. It is broadly categorized into two components:
- **User-generated Content (UGC)**: Media explicitly and intentionally created by the user, such as text, audio, images, and videos.
- **Corresponding Metadata**: System, content, or user interactions, identifiers, and sensor data from smart devices. These are often implicitly created, sometimes without the active awareness of the user (e.g., click logs, GPS coordinates).
According to the Organisation for Economic Co-operation and Development (OECD), to be classified as UGC, content must satisfy three strict criteria:
- **Publication Requirement**: The work must be published in a public or semi-public context accessible to a considerable sub-group of users. This explicitly **excludes** private communications such as emails, instant messages (IMs), or direct messages.
- **Creative Effort**: A non-trivial amount of creative effort must be invested. The user must add value or adapt existing works to construct something new. Simply reposting someone else's content or uploading unedited TV clips does **not** qualify as UGC.
- **Creation Outside Professional Routines**: The content must be produced by the general public outside professional routines and practices, typically without institutional or commercial market contexts, and often without expectations of profit or remuneration.
The internet has evolved through distinct phases characterized by user interaction capabilities:
- **Web 1.0 (1990s)**: The ``read-only'' web. Clear division between professional content creators and passive consumers.
- **Web 2.0 (early 2000s)**: The ``participatory'' or ``social'' web. Consumers become creators. Users actively contribute to creating, rating, collaborating on, and distributing web content, customizing internet applications.
- **Web 3.0 (late 2000s)**: The ``semantic'' web. Focuses on interlinking entities, concepts, and things (Internet of Things - IoT) to enable machine readability.
- **Web 4.0 (2020s)**: The ``intelligent'' or ``symbiotic'' web. Humans and technology work as partners. Characterized by deep personalization (via AI), security (blockchain), user-centered design, trustworthiness, and ethical aspects.
User-generated data represents a primary driver of ``Big Data,'' defined by four dimensions:
- **Volume**: The sheer scale of data (e.g., entering the Zettabyte Era ($10^{21}$ bytes), with average IP traffic exceeding 300 Exabytes ($10^{18}$ bytes) per month).
- **Velocity**: The speed at which new data is generated and needs to be analyzed (e.g., real-time streaming data, stock exchange transactions, social media feeds).
- **Variety**: The structural diversity of data formats (e.g., structured tabular databases, semi-structured XML/JSON, unstructured text, audio, video).
- **Veracity**: The uncertainty and trustworthiness of the data. Poor data quality, noise, and biases make checking accuracy a major challenge.
Learning from user-generated data enables a wide range of analytical applications:
- **Influential User Detection**: Identifying which users have high influence over others in a social network. This is treated as a network analysis task, utilizing users' social graphs and information cascades.
- **Detecting Affective/Emotional State**: Analyzing text (e.g., tweets) to monitor mood. Research shows mood varies systematically by hour and day. It distinguishes between:
- *Positive Affect (PA)*: Enthusiasm, delight, activeness.
- *Negative Affect (NA)*: Distress, fear, anger, guilt, disgust.
- **Sentiment Analysis**: Predicting a user's sentiment (positive, negative, neutral, or continuous value in $[-1, 1]$) towards a product or entity. Crucially, **sentiment is more stable** over time than fleeting emotions or moods.
- **Predicting User Traits**: Inferring personal attributes (age, gender, political orientation, etc.) from user behavior (e.g., Facebook ``Likes'' or music listening history).
- **Trend and Popularity Prediction**: Forecasting the success of products, videos, or music (e.g., ``Hit Song Science'' to predict billboard charts).
- **Games with a Purpose (GWAP)**: Using gamification to crowdsource annotations that are hard for computers (e.g., ESP Game for image tagging, TagATune for music tagging).
- **Recommender Systems**: Personalized systems that select and present interesting items to users, filtering out irrelevant content.
In a notable study (Krismayer et al., 2018), researchers modeled listening habits to predict user demographics:
- **Feature Representation**: Each user was represented by a 20,000-dimensional feature vector containing 10,000-dimensional TF-IDF weighted artist listening events and 10,000-dimensional artist tags.
- **Algorithms**: Adopted Support Vector Machines/Regression (SVM/SVR), Naive Bayes, and Decision Trees.
- **Key Results and Baselines**:
\resizebox{\linewidth}{!}{
**Target Trait** & **Task Type** & **Best Model Performance** & **Baseline Performance**
---
**Age** & Regression (MAE) & **4.13** (SVR RBF Kernel) & 6.23 (Mean Baseline)
**Gender** & Classification (Acc.) & **81.36%** (Bayesian Log. Reg.) & 72.51% (Majority Class)
**Country** & Classification (Acc.) & **69.37%** (Simple Logistic) & 19.03% (Majority Class)
}
Working with UGC introduces several technical and statistical challenges:
- **Double-counting**: The same content is often available across multiple platforms, requiring duplicate detection.
- **Inactive/Fake Accounts**: Not all registered accounts are active (hence the focus on Monthly Active Users - MAUs). Fake accounts can be used for malicious inflation of views/likes.
- **Biases**: Community bias leads to statistical bias, and popularity bias causes unfairness by over-recommending popular items.
- **Copyright and Privacy**: Tracking the original source of UGC is complex, and publicly accessible UGC often raises severe privacy concerns regarding sensitive personal data.
\subsection{Subchapter 1.2: NLP Fundamentals in LUD}
Key concepts
Unit 2
10 min read
This subchapter covers the fundamentals of Collaborative Filtering (CF) and contrasts the two primary memory-based heuristics: User-based CF (UserKNN) and Item-based CF (ItemKNN), detailing their prediction formulas and practical trade-offs.
Herlocker et al. (2004) identified 10 primary reasons why users interact with recommender systems:
- **Annotation in Context**: Showing predictions for items directly related to the user's current context (e.g., related links while browsing).
- **Find Good Items**: The core recommendation task: suggesting a top-N ranked list of items of high utility.
- **Find All Good Items**: Suggesting every single relevant item while minimizing the false negative rate (critical for patent searches or medical literature).
- **Recommend Sequence**: Generating a pleasant, coherent sequence of items (e.g., music playlist generation or a structured learning path).
- **Just Browsing**: Purely entertainment-driven exploration without a concrete purchasing intent.
- **Find Credible Recommender**: The user tests the system's quality and checks for biases to see if it can be trusted.
- **Improve Profile**: The user explicitly rates items to help the system learn their preferences faster.
- **Express Self**: Rating and reviewing items for personal satisfaction and self-expression.
- **Help Others**: Altruistically contributing reviews and ratings to benefit the community.
- **Influence Others**: Intentionally manipulating ratings to promote or devalue specific items (e.g., shilling attacks or vandalism).
These three modes of information access differ fundamentally in the user's information need and active involvement:
- **Information Retrieval (IR)**:
- *Need*: Specific, directed information need.
- *Interaction*: User is highly active, formulating and entering an explicit query.
- *System Goal*: Retrieve and rank the most relevant documents matching the query.
- **Browsing**:
- *Need*: Undirected, exploratory information need (hard to express as a query).
- *Interaction*: User actively explores the interface to discover items.
- *System Goal*: Provide an intuitive interface for collection discovery.
- **Recommending**:
- *Need*: System-predicted preference.
- *Interaction*: User is passive (or almost passive).
- *System Goal*: Filters and ranks items based on observed past actions or implicit/explicit feedback (user profile).
Collaborative Filtering is the most widely used approach for recommender systems. It is domain-independent, requiring no content analysis or domain-specific features.
- **Core Assumption**: Users who agreed on their preferences in the past will agree on their preferences in the future.
- **Data Source**: A rating matrix $R$ of size $n \times m$, where $n$ is the number of users and $m$ is the number of items. Ratings can be explicit (Likert stars, sliders, binary thumbs) or implicit (clicks, listening counts).
- **The Memory-based Paradigm**: Stores the raw user-item rating matrix in memory. It makes predictions on the fly using nearest-neighbor heuristics (lazy learning), offering high transparency and ease of interpretation, but demanding massive memory resources.
- **Core Idea**: To predict a rating for active user $a$ on target item $i$, identify a neighborhood $K$ of $k$ users who are most similar to $a$ and have rated item $i$.
- **Prediction Formula**:
\[
r'_{a, i} = \bar{r}_a + \frac{\sum_{u \in K} \text{sim}(a, u) \cdot (r_{u, i} - \bar{r}_u)}{\sum_{u \in K} |\text{sim}(a, u)|}
$
where:
- $\bar{r}_a$ is the average rating of the active user $a$ (calculated over all items rated by $a$).
- $\bar{r}_u$ is the average rating of neighbor $u$.
- $\text{sim}(a, u)$ is the similarity weight between user $a$ and user $u$ (typically Pearson correlation).
- **The Rating Bias Correction**: Subtracting the neighbor's mean ($\bar{r}_u$) and adding the active user's mean ($\bar{r}_a$) is crucial. It normalizes rating scales, accounting for the fact that some users are naturally lenient (e.g., average rating of 4.5) while others are critical (e.g., average rating of 2.0).
- **Core Idea**: To predict a rating for active user $a$ on target item $i$, identify a neighborhood $N$ of $k$ items that are most similar to $i$ and have been rated by $a$.
- **Prediction Formula**:
\[
r'_{a, i} = \frac{\sum_{j \in N} \text{sim}(i, j) \cdot r_{a, j}}{\sum_{j \in N} |\text{sim}(i, j)|}
$
where:
- $r_{a, j}$ is the rating active user $a$ gave to item $j$.
- $\text{sim}(i, j)$ is the similarity between item $i$ and item $j$ (typically adjusted cosine similarity).
- **Heuristic Difference**: Instead of finding similar *users* (rows of the matrix), ItemKNN finds similar *items* (columns of the matrix) and aggregates the active user's own ratings of those items.
In industrial and large-scale applications, Item-based CF is almost universally preferred over User-based CF:
\resizebox{\linewidth}{!}{
**Dimension** & **User-based CF (UserKNN)** & **Item-based CF (ItemKNN)**
---
**Core Similarity** & User-User similarity (row-wise) & Item-Item similarity (column-wise)
**Scalability** & Poor (Users $n \gg$ Items $m$; matrix size $n \times n$ is huge) & Excellent (Items $m$ are fewer; matrix size $m \times m$ is smaller)
**Profile Stability** & Low (User tastes and profiles change dynamically) & High (Item characteristics and rating vectors are highly stable)
**Precomputability** & Hard to precompute offline due to dynamic user profiles & Easy to precompute the item similarity matrix offline
**Online Lookup** & Slow (Must scan all users who rated the target item) & Extremely fast (O(1) lookup of similar items and user's ratings)
**Interpretability** & ``Users like you also liked...'' (Less intuitive) & ``Because you liked item X...'' (Highly transparent)
}
\subsection{Subchapter 2.2: Similarity Measures}
Key concepts
Unit 3
7 min read
This subchapter introduces model-based Collaborative Filtering, detailing the conceptual foundations of Latent Factor Models, Singular Value Decomposition (SVD), the projection of new users and items into latent spaces, and the limitations of SVD on incomplete matrices.
Latent Factor Models aim to explain user-item ratings by characterizing both users and items in a joint, lower-dimensional space of factors () automatically derived from rating patterns.
- **Latent Factors**: Hidden dimensions that represent underlying characteristics of items and users.
- *In Music*: Instrumentation, era, tempo, vocal styles (male/female), mood.
- *In Movies*: Action levels, comedic style, target demographic, character tropes.
- **Interpretability**: While some dimensions align with recognizable concepts, mathematically derived latent factors are often **not directly interpretable**. They simply capture orthogonal directions of maximum variance in the rating data.
Matrix Factorization represents both users and items as -dimensional vectors in a shared latent space :
- **User Vector**: User $u$ is represented by $\mathbf{w}_u \in \mathbb{R}^f$, measuring the user's interest in the latent dimensions. $W = [\mathbf{w}_1, \dots, \mathbf{w}_n]^T$ is the $n \times f$ user-factor matrix.
- **Item Vector**: Item $i$ is represented by $\mathbf{h}_i \in \mathbb{R}^f$, measuring the degree to which item $i$ possesses those latent dimensions. $H = [\mathbf{h}_1, \dots, \mathbf{h}_m]$ is the $f \times m$ item-factor matrix.
- **Rating Prediction**: The predicted rating $r'_{u, i}$ is modeled as the inner product of the user and item vectors:
\[
r'_{u, i} = \mathbf{w}_u^T \mathbf{h}_i = \sum_{k=1}^f w_{u, k} h_{k, i}
$
Thus, the full prediction matrix is $R' = W H$.
A classic linear algebra technique used to factorize the rating matrix : [ R = U \Sigma V^T $ where:
- $U$ is an $n \times n$ orthogonal matrix (left singular vectors, representing users).
- $\Sigma$ is an $n \times m$ diagonal matrix containing singular values (representing the strength of each latent factor, ordered by importance).
- $V$ is an $m \times m$ orthogonal matrix (right singular vectors, representing items).
W and H are calculated from the singular matrices as: [ W = U_f \Sigma_f^{1/2} \quad \text{and} \quad H = \Sigma_f^{1/2} V_f^T fUVR$ and performs dimensionality reduction.
A major advantage of SVD is the ability to project new users or items into the established latent space without recomputing the entire decomposition:
- **New User**: Let $\mathbf{a}$ be the rating vector ($m \times 1$) of a new user. Their latent representation $\mathbf{w}_a \in \mathbb{R}^f$ is projected as:
\[
\mathbf{w}_a = \mathbf{a}^T U_f \Sigma_f^{-1}
$
- **New Item**: Let $\mathbf{b}$ be the rating vector ($n \times 1$) of a new item. Its latent representation $\mathbf{h}_b \in \mathbb{R}^f$ is projected as:
\[
\mathbf{h}_b = \mathbf{b}^T V_f \Sigma_f^{-1}
$
Once projected, recommendations can be made by calculating the cosine similarity between user and item vectors in the joint space.
Classical SVD is strictly defined only on fully known matrices. This presents two severe obstacles for Collaborative Filtering:
- **Missing Values**: In real-world recommenders, $R$ is highly sparse (e.g., $> 99%$ missing). SVD cannot handle missing entries natively.
- **Imputation Drawbacks**: Standard mitigation requires imputing missing values (e.g., filling them with the user average, item average, or zero). This significantly inflates the dataset size (destroying sparsity advantages), shifts the data distribution, and introduces massive artificial noise.
Solution: Rather than factorizing the full matrix using classical SVD, modern CF directly optimizes the latent vectors and by minimizing reconstruction error only over the set of observed ratings .
\subsection{Subchapter 3.2: Training Matrix Factorization (ALS, SGD, Regularization, Biases)}
Key concepts
Unit 4
8 min read
This subchapter details the three primary evaluation paradigms in Recommender Systems, detailing their experimental designs, trade-offs, advantages, and limitations.
Offline testing relies entirely on historical datasets containing recorded user-item interactions, with no real-world users interacting with the candidate system.
- **Common Experimental Setups**:
- *Cross-Fold Validation*: Dataset is partitioned into $K$ folds; $K-1$ folds train the model, and the remaining fold acts as the test set.
- *Percentage Splits*: Random partitioning of the dataset (e.g., $80%$ training, $10%$ validation, $10%$ testing).
- *Temporal Splits*: Splitting data based on time (e.g., train on history before date $T$, test on data after $T$). This is highly critical in recommender systems to prevent **look-ahead bias** (learning from future interactions to predict past ones).
- **Pros**: Extremely low cost, rapid execution, highly reproducible, and zero risk to live users.
- **Cons**: Cannot capture dynamic user feedback or satisfaction; plagued by the *weak labeling* assumption (treating unobserved entries as negative feedback, whereas they may simply be unsearched or highly relevant items the user has not seen).
User studies involve recruiting a small cohort of users to interact with a prototype system and gathering quantitative and qualitative feedback.
- **Data Collection**:
- *Quantitative*: Likert-style questionnaire responses assessing perceived system performance, recommendation variety, or interaction effort.
- *Qualitative*: Open surveys, structured interviews, diary studies, and direct behavioral observations.
- **UX Evaluation Frameworks**:
- *ResQue* (Pu et al., 2011): Recommender systems' Quality of user experience, focusing on system usability, interface, interaction qualities, and user loyalty.
- *Knijnenburg et al. (2012)*: A comprehensive framework that uses Structural Equation Modeling (SEM) to trace causal relations between user characteristics, system design, context, perceived qualities, interaction experience, and choice satisfaction.
- **Pros**: Captures high-fidelity subjective dimensions of UX (trust, satisfaction, ease of use, explainability).
- **Cons**: Small sample size limits statistical generalization; expensive and slow; subject to Hawthorne effect (users modifying behavior because they are observed).
Online testing evaluates recommendation systems ``in the wild'' with real users performing their normal activities in real-time.
- **Experimental Design**:
- Users are randomly partitioned into two (or more) groups:
- **Group A** (Control): Exposed to recommendations generated by the existing production system.
- **Group B** (Treatment): Exposed to recommendations generated by the new candidate algorithm.
- Business/Engagement metrics (KPIs) are tracked over a set period and statistically compared.
- **Common Online Metrics**: Click-Through Rate (CTR), purchase/conversion rate, session length, retention rate, and gross revenue.
- **Pros**: The gold standard of evaluation. Measures actual, unbiased user behavior and long-term engagement.
- **Cons**: Very high engineering and traffic cost; requires a large, active user base; risks presenting poor recommendations to live users (directly impacting company revenue).
**Dimension** & **Offline Testing** & **User Studies** & **Online Testing (A/B)**
---
**User Interaction** & None & Simulated/Laboratory & Real-world, live
**Sample Size** & Large (historic logs) & Small (10--100s users) & Very large (10,000--Millions)
**Primary Metrics** & RMSE, Precision@K, NDCG & Usability, Trust, UX scale & CTR, Conversion, Retention
**Cost & Effort** & Low (computational only) & Medium to High & High (infra & risk)
**Safety Risk** & Zero & Low & High (revenue risk)
\subsection{Subchapter 4.2: Accuracy Metrics (MAE, RMSE, NDCG, MAP, F1)}
Key concepts