Tukukkk

I'm trying to implement round-tripping from LinkedHashMaps to Clojure collections. The implementation below works for smaller collections, but larger collections raise OutOfMemoryError: GC overhead limit exceeded. Below is a generative test that passes, when the number of tests is small (~10), but exhausts memory for larger numbers of tests (~100).

I would appreciate any tips how to reduce the memory requirements of the implementation as well as general tips on how to improve the code's quality.

(import '[clojure.lang IPersistentMap IPersistentVector]

        '[java.util ArrayList LinkedHashMap])



(defprotocol LinkedHashMappable

  "Convert Clojure data structure to LinkedHashMap."

  (->linked-hash-map [data]))



(extend-protocol LinkedHashMappable

  IPersistentMap

  (->linked-hash-map [m]

    (let [lhm (LinkedHashMap.)]

      (doseq [[k v] m]

        (.put lhm

              (if (keyword? k) (name k) k)

              (->linked-hash-map v)))

      lhm))



  IPersistentVector

  (->linked-hash-map [v]

    (let [alist (ArrayList.)]

      (doseq [i v] (.add alist (->linked-hash-map i)))

      alist))



  Object

  (->linked-hash-map [o] o))



(defprotocol Clojurizable

  "Convert LinkedHashMap to Clojure data structure."

  (->clj [lhm]))



(extend-protocol Clojurizable

  LinkedHashMap

  (->clj [lhm]

    (reduce (fn [m [k v]]

              (assoc m (keyword k) (->clj v)))

            {}

            (iterator-seq (.. lhm entrySet iterator))))



  ArrayList

  (->clj [alist]

    (mapv ->clj alist))



  Object

  (->clj [o] o))



; ----- Test -----



(require '[clojure.test.check :as tc]

         '[clojure.test.check.generators :as gen]

         '[clojure.test.check.properties :as prop])



(def map-generator

  (gen/recursive-gen (fn [inner]

                       (gen/map gen/keyword (gen/one-of [(gen/map gen/keyword inner)

                                                         (gen/vector inner)])))

                     gen/string))



(def linked-hash-map-round-tripping

  (prop/for-all [m map-generator]

                (= (->clj (->linked-hash-map m)) m)))



; The test passes for smaller numbers of tests (~10), but exhausts memory for larger numbers (~100).

#_(tc/quick-check 100 linked-hash-map-round-tripping)

I think automatically creating keywords from string keys and string keys from keywords is not a good idea, for three reasons.

1. It would be surprising to a user to have values change classes when theoretically just pouring the contents of one map into another.


2. In the space of possible key transformations, keyword<-->string is just one point. As a user, I might want to do many other kinds of transformations while converting. For a more general approach, think about how transducers can be supplied to the four-argument form of into.


3. Keywords are interned. This may be a source of your memory trouble. Every new keyword gets stored permanently in the heap. That's how Clojure does fast keyword comparison by just checking object references. That means a property-based test that generates huge numbers of unique keywords is going to consume a lot of memory.