Remove duplicates from CSV files

How to remove duplicate rows from a CSV file

Duplicate records creep into every list: the same contact signs up twice, two exports get combined, or a product appears in several feeds. Importing those duplicates into a CRM, an email tool or a shop system creates double messages and confusing reports. This tool removes them before the import, without uploading your data anywhere.

Step by step

1. Load the file. Choose or drop a CSV file, or paste the text. The delimiter (comma, semicolon, tab or pipe) is detected automatically; you can override it if the preview looks wrong.
2. Select the match columns. Tick the columns that identify one record. For a mailing list, the email column alone is usually right. Leave all columns ticked to remove only rows that are identical everywhere.
3. Decide how strict the comparison is. By default, extra spaces and upper or lower case are ignored, so "Ada@Example.com " and "ada@example.com" count as the same value. If you pick a phone column, only its digits are compared, so "(555) 010-2030" matches "555.010.2030".
4. Choose which row to keep. Keep the first occurrence, or the last one if newer records were added at the end of the file.
5. Run and review. The summary shows how many rows were read, removed and kept. Switch the preview to "Removed duplicates" to check what was taken out and which row each one matched.
6. Download. Save the cleaned CSV, and optionally the removed rows as a second file for your records.

What counts as a duplicate?

There are three levels. An exact duplicate has the same characters in every compared column. A normalized duplicate is the same once you ignore formatting such as case, spaces or punctuation; this is what most people mean when they clean a contact list. A near duplicate differs slightly, for example because of a typo in a name. The free tool handles exact and normalized duplicates. Fuzzy matching for near duplicates is available in Pro, because it needs a careful review of the results.

Empty values need special care

If you deduplicate by email and several rows have no email at all, a simple comparison would treat all of those rows as duplicates of each other and delete them. The option "Never treat rows with empty match values as duplicates" is on by default to prevent exactly that. Completely empty rows are removed separately by the cleanup option.

Spreadsheet apps versus a dedicated cleaner

Excel has a Remove Duplicates command on the Data tab, and Google Sheets offers one under Data cleanup. They work well for small sheets. A dedicated cleaner helps when you want to normalize values before comparing, keep the last instead of the first occurrence, see exactly which rows were removed, or avoid opening a file with many rows in a spreadsheet at all. It also keeps the data out of cloud storage if you would rather not upload a customer list.

Good habits

• Keep an untouched copy of the original export.
• Check the removed rows before importing the cleaned file.
• Use the same rules every time you clean the same kind of export, so results stay consistent.

Frequently asked questions