How to remove duplicate lines
- Paste your list, one item per line.
- Set the options: case sensitive, ignore leading and trailing spaces, and remove empty lines.
- Check how many lines were removed, then copy or download the result.
Examples
Before
apple banana apple cherry banana
After
apple banana cherry
Before
Paris paris PARIS London
After
Paris London
Before
red red blue
After
red blue
When to use it
- Clean email or newsletter lists before an import.
- Remove repeated keywords from an SEO keyword list.
- Deduplicate tags, product SKUs or IDs copied from a spreadsheet column.
- Merge two lists and keep each entry once.
- Remove repeated lines from logs while keeping their order.
How duplicates are detected
Lines are compared from top to bottom. The first occurrence is kept with its original text, including any spaces or capitalization, and later matches are removed. Nothing is sorted, so the order of your list stays the same. If you also want the list sorted, use the Line Sorter afterwards.
With “ignore leading/trailing spaces” on, “red” and “red ” are treated as the same line. With “case sensitive” off, “Paris” and “PARIS” match. Text that looks identical but is stored differently, such as an “é” typed as one character and an “e” plus a combining accent (NFC and NFD), is always treated as a duplicate. Other look-alikes, such as a Latin “a” and a Cyrillic “а”, are different characters and are not merged.
Empty lines are left in place and never counted as duplicates unless you turn on “remove empty lines”. Windows (CRLF) and Unix (LF) line endings are both accepted; the output uses LF.
Frequently asked questions
Which copy of a duplicate is kept?
The first one. Later lines that match it are removed, so the original order is preserved.
Does it work for lists with thousands of lines?
Yes. Typical lists are processed instantly in the browser. Very large inputs depend on your device’s memory and speed.
Why were two lines that look the same not removed?
They probably contain different characters, such as a zero-width space, a non-breaking space in the middle, or a letter from another alphabet. Paste both lines into the Unicode Normalizer to compare their code points.
Can I count how often each line appears?
No. The tool reports how many lines were removed in total, not a count per line.