System-Design-Notes

[System Design] Video Search

High-level Design

Text Processing

Text normalization: lowercaseing, punctuation removal, trim whitespace, Normalization Form KD (NFKD), strip accent, lemmatization and stemming

Tokenization: word tokenization, subword tokenization, character tokenization

Tokens to IDs: lookup table, hasing

Text Encoder

Statistical methods:

• Bag of Words (BoW)
• Term Frequency Inverse Document Frerquence (TF-IDF)

ML-based Methods:

• Embedding layer
• Word2vec
• Transformer-based architectures (BERT, GPT3, BLOOM…)

Video Processing

Loss Computation

Offline Metrics

• Precision@k and mAP
• Recall@k
• Mean Reciprocal Rank (MRR)

Online Metrics

• CTR
• Videos completion rate
• Total watch time of search results

Summary