Duplicate line remover
Remove repeated lines from pasted text while choosing case sensitivity, whitespace trimming, and whether the first or last occurrence wins.
About the Duplicate lines
The duplicate line remover scans a block of lines and keeps only one copy of each, deleting the repeats. You decide whether matching is case-sensitive, whether to trim surrounding whitespace before comparing, and whether to keep the first or the last occurrence of each duplicate. A small panel reports how many lines came in, how many remain and how many were removed.
It is built for deduplicating lists: email addresses, URLs, keywords, log lines, CSV rows or any pasted set where the same entry crept in more than once. Order is preserved for the copies you keep, so the result reads like the original list with the redundant lines quietly taken out, not a re-sorted version of it.
How to use it
- Paste your lines into the Input box.
- Choose Case-sensitive if 'Apple' and 'apple' should count as different lines; leave it off to treat them as the same.
- Keep Trim whitespace on (the default) so lines that differ only by leading or trailing spaces are still treated as duplicates.
- Pick whether to Keep first or Keep last when the same line appears more than once.
- Read the input, output and removed counts, then click Apply to rewrite the input or Copy to copy the unique lines.
Examples
Cleaning a mailing list
Paste 500 email addresses where some were pasted twice. With Trim whitespace on and Case-insensitive matching, "Ada@x.com" and "ada@x.com " collapse to one entry. The stats panel might read input 500, output 463, removed 37, so you instantly see the duplication rate.
Keeping the most recent entry
In a log where the latest line for each user should win, switch to Keep last. If "alice: offline" appears early and "alice: online" appears later, only the later line survives, letting you reduce a noisy log to its final state per key.
Case-sensitive code deduplication
For a list of identifiers where capitalisation is meaningful, turn Case-sensitive on so "userId" and "UserID" are kept as two distinct lines. Without it, they would be merged, which would be wrong for code symbols.
Frequently asked questions
- Does removing duplicates change the order of my lines?
- No. It walks the list in order and keeps the chosen occurrence of each unique line in its original position, so the surviving lines stay in their first-seen sequence. It is a dedupe, not a sort, so nothing is reordered.
- What is the difference between Keep first and Keep last?
- When a line repeats, Keep first retains the earliest occurrence and drops later ones; Keep last retains the final occurrence and drops earlier ones. The unique lines that appear only once are unaffected either way.
- Why is Trim whitespace on by default?
- Pasted lists often have invisible trailing spaces or stray indentation, so two lines that look identical can differ by whitespace. Trimming before comparing catches those near-duplicates, and the kept lines are also trimmed in the output.
- Does it remove blank lines?
- Only the duplicate blank lines. With trimming on, all empty and whitespace-only lines reduce to a single empty line in the output. If you want every blank line gone, run the text through a formatter pass that drops blank lines first.
- How is this different from sorting and deduping?
- Sorting groups identical lines together but does not delete them. This tool removes the repeats outright while preserving order. If you want both a clean order and no duplicates, sort first, then deduplicate, or the other way around depending on the order you need.
Good to know
The interaction between the toggles is where the real control lies. Case sensitivity and trimming both change what counts as a duplicate, so the same input can yield very different removed counts depending on how strict you are. For human data like emails and names, the lenient defaults (case-insensitive, trimmed) usually match intent. For code symbols, file paths on case-sensitive systems, or anything where capitalisation carries meaning, turn case sensitivity on so you do not silently merge distinct values.
Keep first versus keep last is more than cosmetic when each line carries position-dependent meaning. In an append-only log, the last entry per key is often the current state, so Keep last gives you a snapshot; in a priority list where earlier means more important, Keep first preserves the ranking. The live counts are a useful audit trail: a removed total far higher than you expected is a sign your matching is too loose, and one of zero means the toggles are stricter than the data.