`sort -u` re-orders. `awk '!seen[$0]++'` is the classic order-preserving dedupe — and it's a single line.
Bash
# Classic — keep first occurrence of each line, preserve original order
awk '!seen[$0]++' messy.txt
# Dedupe by a specific column (e.g. first column)
awk '!seen[$1]++' transactions.csv
# Dedupe a comma-separated list in one variable
echo "apple,banana,apple,cherry,banana,date" | \
tr ',' '\n' | awk '!seen[$0]++' | paste -sd ','
# apple,banana,cherry,date
# Dedupe ignoring case
awk 'BEGIN{IGNORECASE=1} !seen[tolower($0)]++' file.txtSave your own code snippets
Create a free account and build your private vault. Share publicly whenever you want.