Remove Duplicate Lines
Remove repeated lines from a list while keeping the original order. Count how often each line appeared, or show only the duplicates or only the unique lines.
Your text is processed locally in your browser and is not uploaded to our servers.
Result
0 charactersSummary
What this tool does
Lists collect duplicates. Merge two email exports, paste a column of tags, or combine several logs, and you end up with the same value many times over. This tool removes the repeats in one step and, crucially, keeps the lines in the order you gave them — the order is not shuffled the way a sort would shuffle it.
It can also do the opposite jobs: show you only the lines that repeat, so you can find the collisions, or only the lines that appear exactly once.
How to use it
- Paste your list with one item per line.
- Decide whether case and surrounding spaces matter. For email addresses and tags, leaving both off is usually right.
- Choose what to keep: all unique lines, only the duplicates, or only the lines that appear once.
- Turn on Show how many times each line appeared if you want a frequency next to each result.
- Copy or download the cleaned list.
How duplicates are matched
The tool walks your list from top to bottom. For each line it builds a comparison key, keeps the first line that produces a new key, and skips any later line that produces a key it has already seen. Because it keeps the first occurrence, the surviving lines stay in their original order.
The comparison key depends on your options. With Ignore leading and trailing spaces on, the key is the line with its edges trimmed. With Case sensitive off, the key is also lowercased. So with the default settings, Apple, apple and APPLE all share one key and count as the same line — though the version that survives in the output is the first one exactly as you typed it, unless trimming is on.
Blank lines are removed by default. Turn on Keep blank lines to preserve them in place, which is useful when the blank lines are separating sections you want to keep.
Windows and Mac line endings are normalized first, so a file from Notepad and a file from a Mac editor deduplicate identically.
Examples
Deduplicating this list with the default settings:
apple
Banana
apple
cherry
BANANA
bananagives:
apple
Banana
cherryThree lines survive, in first-seen order. Turn on Show how many times each line appeared and you get apple (2), Banana (3), cherry (1).
Switch the mode to Only lines that repeat and the result is just apple and Banana — the values that showed up more than once. That is how you find, say, the email addresses that appear in two different mailing lists.
Common use cases
- Cleaning mailing lists after merging exports from two systems.
- Deduplicating keyword lists before loading them into an ad platform.
- Finding repeated log lines by switching to the duplicates-only mode.
- Tidying tag and category lists that grew organically.
- Comparing two lists — paste both, and duplicates-only mode shows what they have in common.
- Preparing data for import into a system that rejects duplicate keys.
To also reorder the result, follow up with the Sort Lines tool, which has its own built-in deduplication.
Privacy and security
The list is processed in your browser. Email addresses, customer records and internal identifiers are never uploaded, so deduplicating sensitive data here is safe.
Only your option choices are stored locally. The privacy policy has the full detail.
Frequently asked questions
Does it keep the original order?
Yes. The first occurrence of each line is kept in place, so the output is your list with the repeats removed, not a sorted version. If you want it sorted as well, use the Sort Lines tool afterward.
Which copy of a duplicate is kept?
The first one, reading top to bottom. Later copies are removed. When trimming is on, the kept line is stored trimmed; otherwise it is kept exactly as you typed it.
Can I see only the duplicates?
Yes — set the mode to Only lines that repeat. That lists each value that appeared more than once, which is the fastest way to find collisions between two merged lists.
How are blank lines handled?
By default they are removed along with other duplicates. Turn on Keep blank lines to preserve them, which is helpful when blank lines separate sections you want to keep intact.
Is whitespace inside a line considered?
Only the edges are affected by the trim option. Spaces in the middle of a line are always significant, so New York and New York (two spaces) are different lines.
Related tools
Tools that pair well with the Remove Duplicate Lines.