Natural Language Processing (NLP)

Definition

NLP is an AI method of communicating with intelligent systems using a natural human language (e.g., English, Hindi, French).

NLP is a sub-field of AI where computers can understand and process human language.

NLP relies mostly on Machine Learning and Deep Learning (Transformers).

NLP Objectives

Read — understand written text
Decipher — interpret meaning beyond literal words
Understand — comprehend context and intent
Make sense — extract actionable insights

Components of NLP

NLP System
├── NLU (Natural Language Understanding)
│   → Recognizes natural speech and UNDERSTANDS meaning (reading/interpreting)
└── NLG (Natural Language Generation)
    → PRODUCES a response in natural language (writing/generating)

NLU Examples: Reading comprehension, question answering, sentiment analysis NLG Examples: ChatGPT responses, news article generation, autocomplete

5 Steps in NLP Pipeline

Step	Description	Example
1. Lexical Analysis	Identify and analyze structure of words; divide text into sentences and words	"The cat sat" → ["The", "cat", "sat"]
2. Syntactic Analysis	Check grammar and word arrangement; reject grammatically wrong	"The school goes to boy" → REJECTED
3. Semantic Analysis	Derive exact/dictionary meaning from text	"Bank" near "river" vs "bank" for money
4. Discourse Integration	Meaning of sentence depends on preceding/following sentences	"It was delicious" — what was delicious?
5. Pragmatic Analysis	Re-interpret based on real-world knowledge; what was really meant	"Can you pass the salt?" = a request, not a question about ability

Key NLP Techniques

Technique	Description
Tokenization	Split text into words/sentences/subwords
Stop Word Removal	Remove common words (the, is, a, at)
Stemming	Reduce words to root form: "running" → "run"
Lemmatization	Find dictionary base form: "better" → "good"
POS Tagging	Label each word (noun, verb, adjective, etc.)
NER (Named Entity Recognition)	Identify names, organizations, dates, locations
Sentiment Analysis	Detect positive/negative/neutral tone
Text Classification	Categorize text into predefined classes

NLP Applications

Application	Description	Example
Virtual Assistants	Voice-driven interfaces	Alexa, Siri
Social Media Filtering	Filter terrorist/hate language	Twitter content moderation
Healthcare	Disease recognition from patient speech / records	Clinical NLP
Sentiment Analysis	Analyze opinions on social media	Brand monitoring tools
Spam Filtering	Classify emails as spam/ham	Gmail, Yahoo Mail
Fake News Detection	Identify false news articles	News verification tools
Talent Recruitment	Search and select candidates from resumes	LinkedIn, Naukri
Plagiarism Checkers	Detect copied content	Turnitin, Grammarly
Machine Translation	Convert text between languages	Google Translate
Question Answering	Answer questions naturally	IBM Watson, ChatGPT

Transformer Architecture (Modern NLP)

The Transformer (2017, "Attention is All You Need", Google) revolutionized NLP:

Self-Attention:
  Every word attends to every other word in the sequence
  Captures long-range dependencies
  Parallel processing (unlike sequential RNN)

Key Transformer-based Models:

Model	Developer	Key Feature
BERT	Google (2018)	Bidirectional; best for understanding
GPT-3/4	OpenAI	Generative; best for text generation
T5	Google	Text-to-text transfer transformer
Llama	Meta	Open-source, efficient
Gemini	Google	Multimodal (text+image+audio)

Natural Language Processing (NLP)