Naive Bayes: Classify with Simple Probability Evidence
Use evidence to update a class belief
Naive Bayes estimates how likely each class is after seeing the features. It combines a prior belief about the class with the likelihood of observing the features under that class.
Bayes’ rule is P(class | features) = P(features | class) × P(class) ÷ P(features). For comparing classes for the same row, the denominator is shared, so we can compare the numerator scores.
Key Points
- P(class) is the prior class probability.
- P(features | class) is the likelihood of the observed evidence under a class.
- P(class | features) is the posterior probability.
- The algorithm chooses or ranks classes using these probability ingredients.
The naive assumption makes calculation manageable
Naive Bayes assumes features are conditionally independent given the class. In plain language, once we know the class, it treats the evidence from each feature as separable when multiplying likelihoods.
Key Points
- The assumption is often unrealistic.
- The model can still classify well when the ranking of class scores remains useful.
- Duplicate or strongly related features can make the evidence count twice.
- The version used depends on feature type: word counts, continuous values, or binary indicators.
Compare two class scores
1priors = {"spam": 0.4, "safe": 0.6}
2likelihoods = {
3 "spam": 0.8 * 0.7,
4 "safe": 0.1 * 0.2,
5}
6
7scores = {label: priors[label] * likelihood for label, likelihood in likelihoods.items()}
8prediction = max(scores, key=scores.get)
9
10print("Scores:", {label: round(score, 3) for label, score in scores.items()})
11print("Prediction:", prediction)Scores: {'spam': 0.224, 'safe': 0.012}
Prediction: spamThe scores are unnormalised because the shared denominator was omitted. That is enough to choose the larger class score. A library can normalise them when calibrated probabilities are needed.
Priors describe class frequency
If spam appears in 40% of the training messages, the prior begins near 0.4 before the message words are considered. Priors can be estimated from data or chosen from a domain process, but changing them changes the prediction behaviour.
Key Points
- Class frequency affects the starting odds.
- A rare class needs strong evidence to overcome a small prior.
- Training prevalence may not equal deployment prevalence.
- Check whether the prior should reflect a future population or the sample used for fitting.
The zero-frequency problem needs smoothing
If a word never appeared in the safe messages during training, an unsmoothed likelihood can be zero. Multiplying by zero makes the whole class score zero, even when the word was simply absent from a small sample.
Laplace smoothing adds a small count before calculating probabilities. It prevents an unseen feature value from eliminating a class too aggressively.
Key Points
- Smoothing is a modelling choice.
- It is especially useful with sparse text features.
- The smoothing strength can affect probabilities and rankings.
- Evaluate it on held-out data.
Log probabilities avoid tiny products
Multiplying many small likelihoods can underflow numerically. Since log(ab) = log(a) + log(b), implementations usually add log priors and log likelihoods instead of multiplying raw probabilities.
1import math
2
3prior = 0.4
4likelihoods = [0.8, 0.7, 0.6]
5
6raw_score = prior
7for likelihood in likelihoods:
8 raw_score *= likelihood
9
10log_score = math.log(prior) + sum(math.log(likelihood) for likelihood in likelihoods)
11print(f"Raw score: {raw_score:.3f}")
12print(f"Log score: {log_score:.3f}")Raw score: 0.134 Log score: -2.007
Different Naive Bayes variants fit different evidence
Key Points
- Multinomial Naive Bayes is common for counts such as word frequencies.
- Bernoulli Naive Bayes uses binary feature presence.
- Gaussian Naive Bayes models continuous features with a class-specific distribution.
- The “naive” independence assumption is shared, but the likelihood calculation changes.
Pick the variant based on how features are represented. Passing continuous measurements to a count model, or treating counts as independent binary flags without thought, can weaken the result.
Naive Bayes is fast and useful as a baseline
The model is often quick to train and works surprisingly well for high-dimensional sparse text. It may be a strong baseline even when its probability estimates are not perfectly calibrated.
Key Points
- It handles many features efficiently.
- It can work with small datasets.
- It makes assumptions that should be checked.
- Measure classification quality and probability quality separately when needed.
Where people get confused
Key Points
- “Naive means useless.” The assumption is simple, but the resulting ranking can be effective.
- “The features must be truly independent.” Conditional independence is an approximation used for calculation.
- “The raw scores are probabilities.” Without the shared normalising denominator, they are class scores.
- “Smoothing fixes missing data.” It fixes zero likelihoods in the probability estimate; it does not repair missing feature values generally.
- “High classification accuracy means calibrated probability.” Ranking and calibration are separate.
Quick check
What does Bayes’ rule combine?
It combines a prior class probability with the likelihood of observing the features under that class, then normalises across classes to obtain a posterior probability.
What is the naive assumption?
Features are treated as conditionally independent given the class, allowing their likelihoods to be multiplied or their log likelihoods to be added.
Why is smoothing useful?
It prevents a feature value never seen in a class during training from assigning that class a zero likelihood solely because the sample was limited.
Why use log probabilities?
Adding log probabilities avoids numerical underflow from multiplying many tiny likelihoods and gives the same class ranking.
Recap
Key Points
- Naive Bayes combines class priors and feature likelihoods.
- The independence assumption makes calculation fast but is approximate.
- Smoothing and log probabilities make implementation more stable.
- The next lesson learns human-readable splits with Decision Trees.
Theia
Can you explain this concept?
Practise Naive Bayes: Classify with Simple Probability Evidence with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.