What a Dataset Is, and Where to Find Good Ones
Someone had to collect those 21 flats
Since Chapter 2 you have been loading a CSV of 21 Pune flats and treating it like it just exists. In every real project, it does not. Before anyone writes a single line of pandas, someone had to decide what to record, go and record it, and save it somewhere you could load. That step is the dataset, and how good it is decides how good your model can ever be.
This lesson is about that step: what a dataset actually is, the shapes it comes in outside a tidy CSV, and where you go to get one when nobody has handed you a pune-flats.csv.
The idea
A dataset is a collection of examples of the thing you are trying to predict or understand. Each example is one real situation: one flat that actually got rented, one match that actually got played, one loan that actually got repaid or defaulted. A model that has never seen an example cannot learn anything about it, however clever the algorithm is.
Recall from lesson 1.6: a row is one example, and a column is one feature about that example. A dataset is just a pile of rows collected honestly, one row per real thing that happened.
Tip:"Garbage in, garbage out" is the oldest line in this field, and it is about this exact step. A brilliant model trained on a bad dataset makes confident, wrong predictions. A simple model trained on a good dataset is usually more useful.
Structured and unstructured data
pune-flats.csv is structured data: rows and named columns, the same columns for every row, and it fits neatly into a table. Most of the data you will work with in this course looks like this, and pandas exists almost entirely for it.
A lot of real data is not shaped like a table at all. A WhatsApp chat, a photo of a cricket scoreboard, an audio recording of a customer support call: this is unstructured data. There is no fixed set of columns to open in a spreadsheet. It still gets used for ML, in image recognition, speech-to-text, chatbots, but it needs different tools than the ones in this chapter. You are learning tabular data first because it is the most common starting point and the ideas here, missing values, scaling, splitting, carry over everywhere else too.
A dataset is rarely just one file
pune-flats.csv is one CSV because this course kept it simple. In a real job, "the dataset" is often several tables you have to combine yourself: a table of flats, a separate table of areas with their average commute time, maybe a third table of nearby metro stations. Combining them, called joining, is a pandas skill on its own, and you will not need it for this course, but it is worth knowing the word so it does not surprise you later.
Reload the flats data the way you have been:
1import pandas as pd
2
3df = pd.read_csv("https://cod-e.vercel.app/data/pune-flats.csv")
4df.shape(21, 6)
21 rows, 6 columns. That is a tiny dataset, small enough to keep in this course, but nowhere near what a real rent-prediction product would use. A real one might have tens of thousands of flats. Size matters, and not just for accuracy:
Key Points
- More rows usually means the model sees more of the variety in the real world, and generalises better.
- More rows also means slower training and more memory, which becomes a real engineering problem at scale.
- A small, clean dataset almost always beats a huge, messy one. Size is not the whole story.
Where real datasets come from
Roughly three sources, and you will meet all three sooner or later.
Key Points
- Your own company's logs. A food delivery app already has every order, every delivery time, every rating, sitting in a database. This is usually the best data you will ever get, because it is real and it is yours, but it is also the messiest, and getting access to it is its own process.
- Public datasets, collected for practice or research. Kaggle and the UCI Machine Learning Repository are the two you will hear most. India's own data.gov.in publishes government datasets, everything from crop yields to exam results.
- Data you generate yourself. Surveys, scraping public web pages (only where it is legal and allowed), or running a small experiment to collect exactly what you need.
Tip:When you practise on your own, start on Kaggle. It has thousands of free datasets, most with other people's notebooks attached so you can see how they approached the same data.
Labels are not free
Remember from lesson 1.3: supervised learning needs a target, the answer column, like rent in our flats data. For flats, the "answer" already existed: whatever the flat actually rented for. For a lot of problems, it does not exist yet and someone has to create it.
Say you want a model that reads a customer review and tells you if it is a complaint about "delivery time" or about "food quality". Nobody wrote that label on the review. A person has to read thousands of reviews and tag each one. That is called labelling, it costs real time or real money, and it is why "just collect more data" is often harder than it sounds. Unsupervised problems, where there is no target column at all, sidestep this cost entirely, which is one reason they show up wherever labelling is too expensive.
Where people get confused
Key Points
- "A bigger dataset is always better." Only if the extra rows are correct and relevant. A thousand mislabelled rows can be worse than a hundred clean ones.
- "A dataset has to be a CSV." CSV is common because it is simple, but data also comes as JSON, from a SQL database, or straight out of an API. pandas can read most of these with a different read_ function, read_json, read_sql, and so on.
- "Public datasets are ready to use." They almost never are. Kaggle datasets still need the cleaning this whole chapter is about, just usually less of it than raw company data.
Quick check
What is the difference between structured and unstructured data, with an example of each?
Structured data fits a table with fixed columns, like the flats CSV. Unstructured data does not, like a photo or an audio clip.
Why is a labelled dataset sometimes expensive to create, even when the raw data is free?
The raw examples might already exist, but someone still has to go through them and attach the correct answer, the target, to each one by hand, which costs time or money.
Name two places you could go looking for a public dataset to practise on.
Kaggle and the UCI Machine Learning Repository are the two most common. India-specific government data lives on data.gov.in.
Recap
Key Points
- A dataset is a collection of real examples. A model only ever learns what is actually in it.
- Most of this course is tabular, structured data, rows and named columns, but unstructured data (images, audio, text) is common elsewhere in ML.
- Real datasets come from your own company's logs, from public sources like Kaggle, or from data you collect yourself.
- Labels are not automatic. Turning raw examples into a supervised dataset often means someone tagging each one by hand.
- Bigger is not automatically better. A clean, relevant dataset beats a huge, messy one.
Now that you know where a dataset comes from, the next question is what to actually do the moment you get one, before you touch a single value. That is looking at your data properly, and it is next.
Theia
Can you explain this concept?
Practise What a Dataset Is, and Where to Find Good Ones with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.