Skip to main content
replaced http://stackoverflow.com/ with https://stackoverflow.com/
Source Link

This is elegant but inefficient: O(n^2) when you could probably do this in O(n). See Prime sieve in HaskellPrime sieve in Haskell for a similar issue where the elegant code is slow.

This is elegant but inefficient: O(n^2) when you could probably do this in O(n). See Prime sieve in Haskell for a similar issue where the elegant code is slow.

This is elegant but inefficient: O(n^2) when you could probably do this in O(n). See Prime sieve in Haskell for a similar issue where the elegant code is slow.

fix link
Source Link
Quentin Pradet
  • 6.9k
  • 1
  • 24
  • 44

One general comment: this is not Python, but Haskell. :) Unfortunately Python really does not want us to write functional code (two examples: no tail call optimizationstail call optimizations, Python 3 hiding reduce() in functools). This is reflected in the culture and the community as well: nobody writes such code and you can't expect it to be liked by other Python programmers. But it's okay if you're writing for yourself!

One general comment: this is not Python, but Haskell. :) Unfortunately Python really does not want us to write functional code (two examples: no tail call optimizations, Python 3 hiding reduce() in functools). This is reflected in the culture and the community as well: nobody writes such code and you can't expect it to be liked by other Python programmers. But it's okay if you're writing for yourself!

One general comment: this is not Python, but Haskell. :) Unfortunately Python really does not want us to write functional code (two examples: no tail call optimizations, Python 3 hiding reduce() in functools). This is reflected in the culture and the community as well: nobody writes such code and you can't expect it to be liked by other Python programmers. But it's okay if you're writing for yourself!

Source Link
Quentin Pradet
  • 6.9k
  • 1
  • 24
  • 44

Great job, this code does a lot of things in few lines, is easy to read, and the results are quite impressive. But who's going to upvote me if only say this?

One general comment: this is not Python, but Haskell. :) Unfortunately Python really does not want us to write functional code (two examples: no tail call optimizations, Python 3 hiding reduce() in functools). This is reflected in the culture and the community as well: nobody writes such code and you can't expect it to be liked by other Python programmers. But it's okay if you're writing for yourself!

Another comment: Thank you for the docstrings, they're really useful. 👍

"""
Implementation of prefix-free compression and decompression.
"""

Use a one-liner docstring or expand the docstring (see PEP 257).

import doctest
from itertools import islice
from collections import Counter
import random
import json

INPUT_FILE = "commedia.txt"
COMPRESSED_OUTPUT_FILE = "commedia.pfc"
DICTIONARY_OUTPUT_FILE = "commedia.pfcd"

PEP 8: You need two spaces between those constants and the function below. Also, you don't use random. Consider using flake8 and yapf by integrating them into your text editor.

def binary_strings(s):
    """
    Given an initial list of binary strings `s`,
    yield all binary strings ending in one of `s` strings.

    >>> take(9, binary_strings(["010", "111"]))
    ['010', '111', '0010', '1010', '0111', '1111', '00010', '10010', '01010']
    """
    yield from s
    while True:
        s = [b + x for x in s for b in "01"]
        yield from s

Even though take is obvious for functional programmers, you should define it before binary_strings.

def take(n, iterable):
    """
    Return first n items of the iterable as a list.

    >>> take(5, range(10))
    [0, 1, 2, 3, 4]
    """
    return list(islice(iterable, n))

You're reversing the order of arguments of islice() for no real reason.

def prefix_free(generator):
    """
    Given a `generator`, yield all the items from it
    that do not start with any preceding element.

    >>> take(6, prefix_free(binary_strings(["00", "01"])))
    ['00', '01', '100', '101', '1100', '1101']
    """
    seen = []
    for x in generator:
        if not any(x.startswith(i) for i in seen):
            yield x
            seen.append(x)

This is elegant but inefficient: O(n^2) when you could probably do this in O(n). See Prime sieve in Haskell for a similar issue where the elegant code is slow.

Also, this algorithm cannot beat Huffman coding because the structure of the tree does not depend on the actual frequency of letters. However, on natural language it will probably be close to Huffman codes because of the Zipf law.

def build_translation_dict(text, starting_binary_codes=["000", "100", "111"]):
    """
    Builds a dict for `prefix_free_compression` where
       More common char -> More short binary strings
    This is compression as the shorter binary strings will be seen more times than
    the long ones.

    Univocity in decoding is given by the binary_strings being prefix free.

    >>> sorted(build_translation_dict("aaaaa bbbb ccc dd e", ["01", "11"]).items())
    [(' ', '001'), ('a', '01'), ('b', '11'), ('c', '101'), ('d', '0001'), ('e', '1001')]
    """
    binaries = sorted(list(take(
        len(set(text)), prefix_free(binary_strings(starting_binary_codes)))), key=len)
    frequencies = Counter(text)
    # char value tiebreaker to avoid non-determinism                     v
    alphabet = sorted(
        list(set(text)), key=(lambda ch: (frequencies[ch], ch)), reverse=True)
    return dict(zip(alphabet, binaries))

This is where we really miss Haskell. And I'm glad that I don't have to debug this. :)

Nothing to say about the functions that follow: they make sense and are very easy to read.

    with open(DICTIONARY_OUTPUT_FILE, "w") as f:
        f.write(json.dumps(d))

pickle is probably going to be more space efficient than JSON.

Now, for your next challenge, try to build an efficient language model with low perplexity on your input file. :)