Remove Duplicate Lines

What this tool does

Paste a list and this tool will remove duplicate lines from it, leaving one copy of each and telling you exactly how many went. The original order is preserved, you choose whether the first or the last copy survives, and case, surrounding whitespace and empty lines can each be ignored or respected.

It is built for the lists that arrive without warning: email addresses pasted out of three spreadsheets, a column of SKUs, a log of user agents, a redirect map that someone appended to twice.

How to use it

1. Paste the list, one item per line.
2. Decide whether the first or the last occurrence should survive.
3. Turn on the tolerances you want — ignore case, ignore surrounding spaces, drop empty lines.
4. Copy the result, or download it as a text file.

The counters underneath separate duplicates from empty lines, so you can tell a list that was 40% repeated from one that was mostly blank rows.

When this beats sort -u

If you are already at a terminal, sort -u file.txt does most of this in one command, and for large files it will always be faster. Reach for it. But three things it does not do come up constantly:

• sort -u sorts. That is not a side effect you can turn off; it is the mechanism. If the order of your list carries meaning — a redirect chain, a build sequence, records already in date order — sorting destroys the very thing you were preserving. awk '!seen[$0]++' keeps the order, and is the command worth memorising, but almost nobody has.
• sort -u cannot keep the last copy. Which line survives is undefined from your point of view, because identical lines are identical. When "later wins" is the rule, you need something that tracks position.
• sort -u tells you nothing. It prints the result. Knowing that 12,000 lines became 11,998 is usually more interesting than the result itself: it tells you the duplication was a rounding error rather than the systematic double-import you feared.

There is also the boring case, which is most cases: the list is in a browser tab or a spreadsheet cell and never was a file.

The carriage return that eats an hour

The single most common report about any deduplication tool is that it removed nothing from a list that visibly repeats. The cause is nearly always \r\n.

Windows ends lines with a carriage return followed by a line feed; Unix uses the line feed alone. Paste two sources into the same box and one set of lines ends with an invisible extra character. To any exact comparison, alpha\r and alpha are different strings — correctly, and uselessly.

This page normalises \r\n and lone \r to a plain line break before comparing anything, so mixed sources behave. What it cannot guess at is the rest of the invisible zoo: a trailing space, a non-breaking space pasted from a web page, a zero-width space that survived a PDF. Switch on ignore surrounding spaces first; if lines still refuse to match, the difference is inside the line, and it is worth pasting one of each into a hex-aware editor to see what is really there.

Unicode adds one more trap. The same accented character can be stored as one code point or as a letter plus a combining accent. They look identical and compare as different, which is exactly the sort of thing that turns up in a list of names.

After deduplicating

Two useful next steps. Sort lines puts the survivors into an order — alphabetical, natural or by length — which makes an unexpected entry stand out. The word counter gives the totals when a list has to fit a limit rather than merely be unique.