# Remove duplicate lines without sorting or removing empty lines

I am using the following one-liner to remove duplicate non-empty lines without sorting:

perl -ne 'if ( /^\s*$/ ) { print } else { print if !$x{$_}++}'  Empty (whitespace-only) lines are to be retained. However, this code seems bulky, specially the $x{$_}++ part. Is there any way I can make this code more readable and understandable? • I'm not sure what you mean by $x{$_}++ being bulky. It seems pretty small for what it does. Does it take too much memory? What's bulky about it? In terms of readability, is changing it from a one-liner to a script on the table? Sep 24 at 7:12 • @mdfst13 if I look at $x{$_}++, after few days, I will not understand what it is actually doing. Maybe, this is due to my lack of understanding of perl. But a more understandable syntax will be much helpful. Sep 24 at 7:16 • Break it into two parts then, where latter one is $x{$_} += 1; Sep 27 at 7:42 ## 1 Answer Why not just print if /^\s*$/ || ! $SEEN{$_}++


I don't see the point of nesting here. You want to print if the line is empty or if it has not been seen previously. So say that. If you still don't find this readable and readability is your main concern, change it from a one-liner to a script. Something like

#!/usr/bin/perl -w

use strict;

my $WHITESPACE = qr{\s*}; my$EMPTY_LINE = qr{^$WHITESPACE$};

my %seen;

while (my $line = <>) { # add comments as necessary print$line if $line =~ m{$EMPTY_LINE} || ! $seen{$line}++;
}

# one line version
# perl -ne 'print if /^\s*$/ || !$SEEN{$_}++'  Now you can see that an empty line is defined as having only whitespace between the beginning and the end. It would be better to use constants or Readonly for the patterns, but I don't remember the syntax well enough without running it. There's a seen hash. We read every line either from STDIN or the files in the arguments. We print if the line matches an empty line or if it is not seen. At the end, if we wanted, we would could print how many times each line was seen. If that doesn't read naturally to you, add comments for the parts that you won't understand later. Unwrapping $seen{$line}++. The ++ is a post increment. It increments the value of the variable and then returns the previous value of the variable. In this case, the first time that a particular $line value is seen, the variable will be undefined. So Perl will treat it as zero for the purpose of the increment and either zero or undef for the Boolean. Zero and undef are falsey in Perl, so not zero/undef will be true the first time. Subsequent times, the value will be 1, 2, 3, etc. Those are all truthy, so the negated expression will be false.

Perl uses short circuit Boolean operators. So if the line matches an empty line, the second part will never be reached.

One of the major points of Perl one-liners is that they allow a very terse syntax, which is relatively difficult to read but easy to type. If you don't want that, upgrade it to a script with its much longer syntax. If you stuff it in the bin directory under your home directory in Linux, you can set the executable bit and use it like

~/bin/script_name.pl file_name


or

~/bin/script_name.pl < file_name


or

some_other_command | ~/bin/script_name.pl


And if you want to know how it works later, you can simply view or edit the file. If you forget how something works and have to research it, add a comment for the next time.