The ClojureWerkz Blog

News and updates about ClojureWerkz projects

Langohr 3.0.0 Is Released

TL;DR

Langohr is a Clojure RabbitMQ client that embraces AMQP 0.9.1 Model.

3.0.0 is a major release that introduces breaking API changes, JDK 8 compatibility, minor new features and major internal revamp.

Options as Maps

Functions that take options now require a proper Clojure map instead of pseudo keyword arguments:

1
2
3
4
5
6
7
8
9
10
11
12
;; in Langohr 2.x

(lq/declare ch q :durable true)
(lhcons/subscribe ch q (fn [_ _ _])
                        :consumer-tag ctag :handle-cancel-ok (fn [_]))
(lb/publish ch "" q "a message" :mandatory true)

;; in Langohr 3.x
(lq/declare ch q {:durable true})
(lhcons/subscribe ch q (fn [_ _ _])
                        {:consumer-tag ctag :handle-cancel-ok (fn [_])})
(lb/publish ch "" q "a message" {:mandatory true})

JDK 8 Compatibility

Langohr test suite now passes on JDK 8 (previously there was 1 failure in recovery test).

GH issue: #54.

Connection Recovery Performed by Java Client

Langohr no longer implements automatic connection recovery of its own. The feature is still there and there should be no behaviour changes but the functionality has now been pushed “upstream” in the Java client, so Langohr now relies on it to do all the work.

There is two public API changes:

  • com.novemberain.langohr.Recoverable is gone, langohr.core/on-recovery now uses com.rabbitmq.client.Recoverable instead in its signature.

  • Server-named queues will change after recovery. Use langohr.core/on-queue-recovery to register a listener for queue name change.

GH issue: #58.

RabbitMQ Java Client Upgrade

RabbitMQ Java client dependency has been updated to 3.4.2.

Custom Exception Handlers

langohr.core/exception-handler is a function that customizes default exception handler RabbitMQ Java client uses:

1
2
3
4
5
6
(require '[langohr.core :as rmq])

(let [(rmq/exception-handler :handle-consumer-exception-fn (fn [ch ex consumer
                                                               consumer-tag method-name]
                                                             ))]
  )

Valid keys are:

  • :handle-connection-exception-fn
  • :handle-return-listener-exception-fn
  • :handle-flow-listener-exception-fn
  • :handle-confirm-listener-exception-fn
  • :handle-blocked-listener-exception-fn
  • :handle-consumer-exception-fn
  • :handle-connection-recovery-exception-fn
  • :handle-channel-recovery-exception-fn
  • :handle-topology-recovery-exception-fn

GH issue: #47.

Clojure 1.7 Support

Clojure 1.7-specific compilation issues and warnings were eliminated.

clj-http Upgrade

clj-http dependency has been updated to 1.0.0.

ClojureWerkz Support Upgrade

clojurewerkz/support dependency has been updated to 1.1.0.

langohr.core/version is Removed

langohr.core/version was removed.

Change Log

Langohr change log is available on GitHub.

Langohr is a ClojureWerkz Project

Langohr is part of the group of libraries known as ClojureWerkz, together with

  • Elastisch, a minimalistic well documented Clojure client for ElasticSearch
  • Cassaforte, a Clojure Cassandra client built around CQL 3.0
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Langohr, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team

Cassaforte 2.0.0 Is Released

TL;DR

Cassaforte is a Clojure client for Apache Cassandra and DataStax Enterprise. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.

2.0.0 is a major release that introduces breaking API changes, new features and compatibility with Cassandra 2.1.

Changes between 1.3.x and 2.0.0

Cassaforte 2.0 has breaking API changes in most namespaces.

Client (Session) is Explicit Argument

All Cassaforte public API functions that issue requests to Cassandra now require a client (session) to be passed as an explicit argument:

1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/use-keyspace conn "cassaforte_keyspace"))
1
2
3
4
5
6
7
8
9
10
11
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]
            [clojurewerkz.cassaforte.query :refer :all]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/create-table conn "user_posts"
                (column-definitions {:username :varchar
                                     :post_id  :varchar
                                     :body     :text
                                     :primary-key [:username :post_id]})))
1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/insert conn "users" {:name "Alex" :age (int 19)}))

Hayt 2.0

Hayt dependency has been upgraded to 2.0 (GA).

Query DSL Taken From Hayt 2.0

Cassaforte no longer tries to support query condition DSLs for both Hayt 1.x and Hayt 2.0. Hayt 2.0 is the only supported flavour now and is the future.

Some examples of the changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
;; before
(where :name "Alex")

;; after
(where [[= :name "Alex"]])
(where {:name "Alex"})


;; before
(where :name "Alex" :city "Munich")

;; after
(where [[= :name "Alex"]
        [= :city "Munich"]])
(where {:name "Alex" :city "Munich"})


;; before
(where :name "Alex" :age [> 25])

;; after
(where [[= :name "Alex"]
        [> :age  25]])

;; before
(where :name "Alex" :city [:in ["Munich" "Frankfurt"]])

;; after
(where [[= :name "Alex"]
        [:in :city ["Munich" "Frankfurt"]]])

As it’s easy to see, the new condition style closer resembles Clojure itself and thus was a reasonable decision on behalf of Hayt developers.

Policy Namespace

Policy-related functions from clojurewerkz.cassaforte.client were extracted into clojurewerkz.cassaforte.policies:

1
2
3
(require '[clojurewerkz.cassaforte.policies :as cp])

(cp/exponential-reconnection-policy 100 1000)
1
2
3
4
(require '[clojurewerkz.cassaforte.policies :as cp])

(let [p (cp/round-robin-policy)]
  (cp/token-aware-policy p))

DataStax Java Driver Update

DataStax Java driver has been updated to 2.1.x.

Cassandra 2.1 Compatibility

Cassaforte 2.0 is compatible with Cassandra 2.1.

Cassandra Sessions Compatible with with-open

Session#shutdown was renamed to Session#close in cassandra-driver-core. Cassaforte needs to be adapted to that.

Contributed by Jarkko Mönkkönen.

TLS and Kerberos Support

Cassaforte now supports TLS connections and Kerberos authentication via DataStax CQL extensions.

The :ssl connection option now can be a map with two keys:

  • :keystore-path
  • :keystore-password

which provide a path and password to a JDK KeyStore on disk, created with keytool.

Optionally, an instance of SSLOptions can be provided via the :ssl-options connection option.

Contributed by Max Barnash.

GH issue: #60.

Support for overriding default SSL cipher suites

Providing a :cipher-suites key in the :ssl connection option allows to specify cipher suites that are enabled when connecting to a cluster with SSL. The value of this key is a Seq of Strings (e.g. a vector) where each item specifies a cipher suite:

1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

(cc/build-cluster {:ssl {:keystore-path "path/to/keystore"
                         :keystore-password "password"}})]
                         :cipher-suites ["TLS_RSA_WITH_AES_128_CBC_SHA"]}}

The :cipher-suites key is optional and may be omitted, in which case Datastax Java driver’s default cipher suites (com.datastax.driver.core.SSLOptions/DEFAULT_SSL_CIPHER_SUITES) are enabled.

This can be used to work around the need to install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files required by the default set of cipher suites. TLS_RSA_WITH_AES_128_CBC_SHA is a suite in the default set that works with the standard JCE. E.g. by specifying just that one, as in the code example, the standard JCE is enough.

Contributed by Juhani Hietikko.

GH issue: #61.

Cassandra Native Protocol v2 as Default

To preserve Cassandra 2.0 compatibility yet continue using the most recent Cassandra Java driver Cassaforte now uses native protocol v2 by default. v3 can be opted into using the :protocol-version connection option (with value of 3).

Double Licensed Under APL 2.0 and EPL 1.0

The project is now double-licensed under the Eclipse Public License 1.0 and Apache Public License 2.0. The codebase now includes APL 2.0 headers. License files for both APL 2.0 and EPL 1.0 are included in the distribution.

Fetch Size Support

(Internal to the client) automatic paging of result set rows now can be configured or disabled altogether, e.g. when running into problems similar to CASSANDRA-6722.

clojurewerkz.cassaforte.client/with-fetch-size is a macro that does that:

1
2
3
4
5
6
7
8
9
(require '[clojurewerkz.cassaforte.client :as cc])

;; alter page size
(cc/with-fetch-size 8192
  (comment "SELECT queries go here"))

;; disable internal client paging
(cc/with-fetch-size Integer/MAX_VALUE
  (comment "SELECT queries go here"))

Default fetch size is unaltered (Cassaforte relies on the Java driver default). This setting only makes sense for a certain subset of SELECT queries.

Clojure 1.4 and 1.5 Support Dropped

Cassaforte now requires Clojure 1.6.0.

Collections Converted to Clojure Data Structures

Cassandra maps, sets and lists are now automatically converted to their immutable Clojure counterparts.

Atomic Batches Support

Atomic batches are now easier to use with Cassaforte:

1
2
3
4
5
6
7
8
9
(require '[clojurewerkz.cassaforte.client :as client])
(require '[clojurewerkz.cassaforte.cql :as cql :refer :all])
(require '[clojurewerkz.cassaforte.query :refer :all])
(require '[qbits.hayt.dsl.statement :as hs])

(let [s (client/connect ["127.0.0.1"])]
  (cql/atomic-batch s (queries
                         (hs/insert :users (values {:name "Alex" :city "Munich" :age (int 19)}))
                         (hs/insert :users (values {:name "Fritz" :city "Hamburg" :age (int 28)})))))

clojurewerkz.cassandra.cql/iterate-table Now Terminates

clojurewerkz.cassandra.cql/iterate-table no longer produces an infinite sequence.

Keyspace as Option

It is now possible to choose keyspace via an option:

1
2
3
4
5
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

(let [conn (cc/connect {:hosts ["127.0.0.1"] :keyspace "a-keyspace"})]
  )

Contributed by Max Barnash (DataStax).

URI Connections

It is now possible to connect to a node and switch to a namespace using a URI string:

1
2
3
4
5
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

;; connects to node 127.0.0.1:9042 and uses "new_cql_keyspace" as keyspace
(cc/connect-with-uri "cql://127.0.0.1:9042/new_cql_keyspace")

Compression Option

:compression is a new option that can be used when connecting:

1
2
3
4
(require '[clojurewerkz.cassaforte.client :as client])

(let [s (client/connect ["127.0.0.1"] "my-keyspace" {:compression :snappy})]
  )

Valid compression values are:

  • :snappy
  • :lz4
  • :none (or nil)

Contirbuted by Max Barnash (DataStax).

Fixes Race Condition in Async Operations

Async database operations no longer suffer from a race condition between issueing them and definiting callbacks on the returned future value.

Contributed by Kirill Chernyshov.

Correct Deserialisation of Empty Strings

Empty string values are now correctly deserialised (previously they were returned as nil).

GH issue: #91.

Prepared Statement Cache Removed

Prepared statement cache was affecting client correctness in some cases and was removed.

Clojure 1.7.0-alpha2+ Compatibility

Cassaforte is now compatible with Clojure 1.7.0-alpha2 and later versions.

News and Updates

New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.

Cassaforte is a ClojureWerkz Project

Cassaforte is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • EEP, a Clojure library for stream (event) processing
  • Neocons, a Clojure client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Cassaforte, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team.

Buffy Reaches 1.0

Buffy The Byte Buffer Slayer

Buffy is a Clojure library to working with binary data, writing complete binary protocol implementations in Clojure, storing complex data structures in an off-heap cache, reading binary files and doing everything you would usually do with byte buffers.

Buffy is built on top of Netty byte buffers, which addresses many long standing issues with java.util.ByteBuffer.

Key Features

Buffy enables you to access buffers in the similar way you work with the regular Clojure data structures. You can get and set fields separately from one another, give names to different parts of binary representation and so on:

  • partial deserialization (read and deserialise parts of a byte buffer)
  • named access (access parts of your buffer by names)
  • composing/decomposing from key/value pairs
  • pretty hexdump
  • many useful default types that you can combine and extend easily

Data types used in Buffy buffers include:

  • primitive types, such as int32, boolean, byte, short, medium, float, and long
  • arbitrary-length string
  • byte arrays
  • composite types (combine multiple primitive values together)
  • repeated type (repeat any primitive arbitrary amount of times in payload)
  • enum type (for mapping between human-readable and binary representation of constants)

The 1.0 release

Starting with the 1.0 release Buffy API is considered to be moderately stable. Releases can be expected to follow a very SemVer-like change strategy. Most of the work now will go into minor improvements based on community feedback.

Buffy is a ClojureWerkz Project

Buffy is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Cassaforte, a Clojure Cassandra client built around CQL
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Meltdown, Clojure interface to Reactor
  • EEP, a Clojure event processing library
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Buffy, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

@michaelklishin on behalf of the ClojureWerkz Team

Validateur 2.4.2 Is Released

TL;DR

Validateur is a functional validations library inspired by Ruby’s ActiveModel. Validateur 2.4 is a minor feature release.

Changes Between 2.3.0 and 2.4.0

Clojure 1.4 Support Dropped

The project no longer tries to maintain Clojure 1.4 compatibility.

validate-some

validate-some tries any number of validators, short-circuiting at the first failed validator. This behavior is similar to or.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(require '[validateur.validation :refer :all])

(let [v (validate-some
         (presence-of :cake-count :message "missing_cake")
         (validate-by :cake-count odd? :message "even_cake"))]

  "Odd cake counts are valid."
  (v {:cake-count 1})
  ;;=> [true #{}]


  "Even cake counts only throw the second error, since the first
  validation passed."
  (v {:cake-count 2})
  ;;=> [false {:cake-count #{"even_cake"}}]

  "The second validation never gets called and never throws a NPE, as
  it would if we just composed them up."
  (v {})
  ;;=> [false {:cake-count #{"missing_cake"}}]
  )

Contributed by Sam Ritchie (PaddleGuru).

errors? and errors

Errors in validateur are vectors if keys are nested. If keys are only one layer deep – :cake, for example – the error can live at :cake or [:cake].

The errors function returns the set of errors for some key, nested or bare. :cake will return errors stored under [:cake] and vice-versa.

errors? is a boolean wrapper that returns true if some key has errors, false otherwise.

Contributed by Sam Ritchie (PaddleGuru).

Full Change Log

Validateur change log is available on GitHub.

Validateur is a ClojureWerkz Project

Validateur is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Cassaforte, a Clojure Cassandra client built around CQL
  • Neocons, a client for the Neo4J REST API
  • Welle, a Riak client with batteries included
  • Quartzite, a powerful scheduling library

and several others. If you like Validateur, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About The Author

Michael on behalf of the ClojureWerkz Team

The Next Year of ClojureWerkz

3 Years Old

ClojureWerkz has turned 3 last November. This is an age for an open source project at which the dust from initial development has begun to settle.

In this post we’d like to briefly outline what our plans are for the next year with ClojureWerkz.

Where We Are Today

We have close to 40 projects total (a few of them are now defunct, about 5 are incubating and not yet ready to be announced, some 30 have been around and used in production for at least half a year.

Our projects cover boring problems most developers face at some point:

  • Working with data stores
  • Working with message brokers (queues)
  • Data validation
  • Scheduling
  • Key derivation (stronger hashing)
  • Asynchronous and concurrent programming
  • Web crawling

We’ve also contributed to quite a few of our dependencies and related libraries, as well as clojure-doc.org, a community-developed set of Clojure documentation guides.

We’ve been true to our values:

  • Solving boring problems.
  • No bullshit, easy contribution process that includes people from all over the world.
  • Solid documentation.
  • Sane APIs.

which I’m personally quite fond of.

The Most Exciting Change

The most exciting change by far is that several of our projects now have [co-]maintainers outside of the original ClojureWerkz team. Neocons, Titanium, Ogre, Archimedes are now driven by folks who’ve joined us in the last year. Elastisch, Cassaforte, Validateur all have contributors that are active every month (if not every week).

Making it easy to join a project is incredibly important for the project’s sustainability. We are making progress on that end.

What’s Ahead

As the number of projects and users grows, we have to keep things more predictable. Changes that are likely to affect more users should be announced upfront and for all projects. A few of such changes have been considered before and now it’s time to begin implementing them:

  • Drop Clojure 1.4 and 1.5 support.
  • Drop JDK 6 support.
  • Dual-license projects under the Eclipse Public License 1.0 and Apache Software License 2.0, where possible.
  • Introduce a scale of project maturity.

Most of these are fairly self-explanatory but we’ll post more about them in upcoming weeks.

Helping You Help Us

Another thing that we’ve learnt over the last few years is: there are awesome people out there who would come out of the blue with contributions if original project authors make it easy.

We’ve been trying to make it easy, namely

  • We don’t require a contributor agreement
  • We accept pull requests on GitHub instead of asking for patches in a bottle
  • We try to make it easy to discover how to set up a test environment and run project tests
  • We use a hosted CI service

However, there’s one thing where we have a lot of work to do: making it easy to discover what to work on.

In upcoming weeks we are going through issue trackers of all of our actively developed projects and tagging issues suitable for beginners with “Low hanging fruit”. Leiningen has been doing this for a long time and it works well (they have 241 contributor at the time of writing, for science’s sake!).

We’ll also try to pipe some of those low hanging fruit issues into Clojure in the Open, an awesome initiative by Vic Goldfeld.

Legal Entity for ClojureWerkz

We are considering setting up a legal entity to make it easy to donate to ClojureWerkz and take us to the next level. Having such entity is going to take a lot of work but also going to open a lot of doors for us. If you have experience with setting up foundations in the UK or EU and would like to help, please contact us.

Wrapping Up

ClojureWerkz is growing. We’re still not attracted by esoteric problems, so you can expect more of what we’ve been doing all this time. It will be easier to help us.

Michael on behalf of the ClojureWerkz Team.

Monger 2.0.1 Is Released

TL;DR

Monger is an idiomatic Clojure MongoDB driver for a more civilized age. It has batteries included, offers powerful expressive query DSL, strives to support every MongoDB 2.0+ feature and has sane defaults. It also has solid documentation.

2.0.1 is a compatibility release.

Changes between 2.0.0 and 2.0.1

Clojure 1.7 Compatibility

Monger now compiles with Clojure 1.7.

MongoDB Java Driver Update

MongoDB Java driver dependency has been updated to 2.12.4.

Change Log

Monger change log is available on GitHub.

Monger is a ClojureWerkz Project

Monger is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Cassaforte, a Clojure Cassandra client built around CQL
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Monger, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

@michaelklishin on behalf of the ClojureWerkz Team

Elastisch 2.1.0 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

2.1.0 is feature release.

Changes between Elastisch 2.0.x and 2.1.0

Clojure 1.4 Support Dropped

Elastisch no longer officially supports Clojure 1.4. Most of the functionality still works well on that version but please don’t file bugs specific to that version.

Allow :index key in update-aliases (native)

clojurewerkz.elastisch.native.index/update-aliases expects indices to be added to be specified in the :indices key while the respective REST function uses :index. This can have unexpected results, namely the creation of the respective alias for all indices. It is now possible to supply either :index or :indices to the function.

GH issue: #108.

Contributed by Yannick Scherer (stylefruits)

Update with Partial Document via native API

clojurewerkz.elastisch.native.document/update-with-partial-doc is a new function in the Native Client (existed before in the REST API) that performs partial updates:

1
2
3
(require '[clojurewerkz.elastisch.native.document :as doc])

(doc/update-with-partial-doc conn "people" "person" "1" {:country "Sweden"})

Note that the REST API should now be called without wrapping the document in a :doc key. I.e. change {:doc {:field-to-update "Update"}} to {:field-to-update "Update"}.

Contributed by Sandeep Jagtap and Henrik Lundahl.

Ability to Specify Aliases In index.create-template

clojurewerkz.elastisch.rest.index.create-template now supports the :aliases option:

1
2
3
(require '[clojurewerkz.elastisch.rest.index :as idx])

(idx/create-template conn "accounts" {:template "account*" :settings {:index {:refresh_interval "60s"}} :aliases {:account-alias {}}})

Contributed by Jeffrey Erikson.

clj-http Update

clj-http dependency has been upgraded to version 1.0.x.

Allow Retry On Conflict Option

Updates and upserts now allow the retry-on-conflict option to be set. This helps to work around Elasticsearch version conflicts.

GH issue: #119.

Contributed by Michael Nussbaum (Braintree).

REST API Bulk Indexing Filters Out Operation Keys

clojurewerkz.elastisch.rest.bulk/bulk-index now filters out all operation/option keys so that they don’t get stored in the document body.

GH issue: #116.

Contributed by Michael Nussbaum (Braintree).

New Line in Multi-Search REST API

ElasticSearch Multi Search REST API endpoint is sensitive to the trailing new line. When it is missing, the response contains one result too few.

Elastisch now makes sure to append a new line to Multi Search request bodies.

Correct async-put in Native Client

Native client’s document/async-put no longer fails with an exception.

Contributed by Nikita Burtsev.

ElasticSearch Native Client Upgrade

Elastisch now depends on ElasticSearch native client version 1.4.x.

Single-Bucket Aggregation Fix in the Native Client

Child aggregations in single-bucket aggregations (i.e. “global”) are no longer silently dropped.

Contributed by Yannick Scherer (StyleFruits).

Aggregations Support in the Native Client

Native client now has support for aggregations.

The API is the same as in the REST client.

Note that ElasticSearch 1.2 has 26 aggregations. Currently only the most commonly used ones are supported but support for more types will be added eventually. The supported types are:

  • Avg
  • Max
  • Min
  • Sum
  • Stats
  • Extended stats
  • Cardinality, value count
  • Percentiles
  • Histogram
  • Date Histogram
  • Range
  • Date Range
  • Terms
  • Missing
  • Global

Multi-Search Support in the Native Client

Native client now has support for multi-search.

The API is the same as in the REST client except that the functions are in the clojurewerkz.elastisch.native.multi.

Extra Options on Upserts

clojurewerkz.elastisch.native.document/upsert now accepts a map of extra options, e.g. parent document ID:

1
(doc/upsert conn index-name index-type id doc {:parent parent-id})

Terms Query Helper

clojurewerkz.elastisch.query/terms is a newly added alias for clojurewerkz.elastisch.query/term when used with a collection.

Contributed by Martin Klepsch.

Remove Alias Now Works in Native Client

Bug fixed in native client for removing aliases from indices and improved inline documentation. See aliases in the guide.

GH issue: #98.

Highlighting Support in Native Client

Native client now supports (most of the) highlighting features the REST client does:

1
2
3
4
5
6
(require '[clojurewerkz.elastisch.native.document :as doc])
(require '[clojurewerkz.elastisch.query :as q])

(doc/search conn index type
            {:query (q/query-string :query "software" :default_field "summary")
             :highlight {:fields {:summary {}}}})

Per Connection clj-http Options in REST Client

It is now possible to specify clj-http options for REST API connections, e.g. to specify a timeout:

1
2
(esr/connect "http://127.0.0.1:9200/" {:conn-timeout 1000
                                       :basic-auth ["username" "pa$$w0rd"]})

Source Filtering Support in Native Client

Native client now supports source filtering just like the REST API client:

1
2
3
4
(doc/search conn index-name mapping-type
            :query   (q/match-all)
            :sort    {"first-name" "asc"}
            :_source ["first-name" "age"])
1
2
3
4
5
6
(doc/search conn index-name mapping-type
            :query   (q/match-all)
            :sort    {"first-name" "asc"}
            :_source {"exclude" ["title" "country"
                                 "planet" "biography"
                                 "last-name" "username"]})

GH issue: #73.

Search Can Return Fields and Source

Previously a search would return either the source document, or specific fields and not both. There are certain circumstances where having both are beneficial, for example when searching for a child document and you want to include the parent ID:

1
2
3
(require '[clojurewerkz.elastisch.native.document :as esd])

(esd/search conn "index" "child-type" :query (q/match-all) :fields ["_parent"])

The above would return the parent document ID in the :_parent field of each hit, but would not return the document itself. You can now have both by:

1
(esd/search conn "index" "child-type" :query (q/match-all) :fields ["_parent" "_source"])

Now the parent ID is in the :_parent field of each hit, and the matching document will be in :_source as per a normal search.

Contributed by Ben Ashford.

Full Change Log

Elastisch change log is available on GitHub.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Cassaforte, a Clojure Cassandra client
  • Titanium, a Clojure graph library
  • Neocons, a client for the Neo4J REST API
  • Welle, a Riak client with batteries included
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team

Cassaforte 2.0.0-rc2 Is Released

TL;DR

Cassaforte is a Clojure client for Apache Cassandra. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.

2.0.0-rc2 is a release candidate for 2.0.

Changes between 2.0.0-rc1 and 2.0.0-rc2

Fixes Race Condition in Async Operations

Async database operations no longer suffer from a race condition between issueing them and definiting callbacks on the returned future value.

Contributed by Kirill Chernyshov.

Compression Option

:compression is a new option that can be used when connecting:

1
2
3
4
(require '[clojurewerkz.cassaforte.client :as client])

(let [s (client/connect ["127.0.0.1"] "my-keyspace" {:compression :snappy})]
  )

Valid compression values are:

  • :snappy
  • :lz4
  • :none (or nil)

Contirbuted by Max Barnash (DataStax).

News and Updates

New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.

Cassaforte is a ClojureWerkz Project

Cassaforte is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • EEP, a Clojure library for stream (event) processing
  • Neocons, a Clojure client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Cassaforte, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team.

Cassaforte 2.0.0-rc1 Is Released

TL;DR

Cassaforte is a Clojure client for Apache Cassandra. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.

2.0.0-rc is a release candidate for 2.0.

Changes between 1.3.x and 2.0.0-rc1

Compared to 1.3.x, 2.0.0-rc1 has one major breaking API change.

Client (Session) is Explicit Argument

All Cassaforte public API functions that issue requests to Cassandra now require a client (session) to be passed as an explicit argument:

1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/use-keyspace conn "cassaforte_keyspace"))
1
2
3
4
5
6
7
8
9
10
11
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]
            [clojurewerkz.cassaforte.query :refer :all]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/create-table conn "user_posts"
                (column-definitions {:username :varchar
                                     :post_id  :varchar
                                     :body     :text
                                     :primary-key [:username :post_id]})))
1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]
            [clojurewerkz.cassaforte.cql    :as cql]))

(let [conn (cc/connect ["127.0.0.1"])]
  (cql/insert conn "users" {:name "Alex" :age (int 19)}))

Hayt Upgraded to 2.0

Hayt was upgraded to 2.0.

clojurewerkz.cassandra.cql/iterate-table Now Terminates

clojurewerkz.cassandra.cql/iterate-table no longer produces an infinite sequence.

URI Connections

It is now possible to connect to a node and switch to a namespace using a URI string:

1
2
3
4
5
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

;; connects to node 127.0.0.1:9042 and uses "new_cql_keyspace" as keyspace
(cc/connect-with-uri "cql://127.0.0.1:9042/new_cql_keyspace")

Cassandra 2.1 Compatibility

Cassaforte 2.0 is compatible with Cassandra 2.1.

Prepared Statement Cache Removed

Prepared statement cache was affecting client correctness in some cases and was removed.

Policy Namespace

Policy-related functions from clojurewerkz.cassaforte.client were extracted into clojurewerkz.cassaforte.policies:

1
2
3
(require '[clojurewerkz.cassaforte.policies :as cp])

(cp/exponential-reconnection-policy 100 1000)
1
2
3
4
(require '[clojurewerkz.cassaforte.policies :as cp])

(let [p (cp/round-robin-policy)]
  (cp/token-aware-policy p))

DataStax Java Driver Update

DataStax Java driver has been updated to 2.1.x.

Cassandra Sessions Compatible with with-open

Session#shutdown was renamed to Session#close in cassandra-driver-core. Cassaforte needs to be adapted to that.

Contributed by Jarkko Mönkkönen.

TLS and Kerberos Support

Cassaforte now supports TLS connections and Kerberos authentication via DataStax CQL extensions.

The :ssl connection option now can be a map with two keys:

  • :keystore-path
  • :keystore-password

which provide a path and password to a JDK KeyStore on disk, created with keytool.

Optionally, an instance of SSLOptions can be provided via the :ssl-options connection option.

Contributed by Max Barnash.

Clojure 1.4 and 1.5 Support Dropped

Cassaforte now requires Clojure 1.6.0.

Collections Converted to Clojure Data Structures

Cassandra maps, sets and lists are now automatically converted to their immutable Clojure counterparts.

Atomic Batches Support

Atomic batches are now easier to use with Cassaforte:

1
2
3
4
5
6
7
8
9
(require '[clojurewerkz.cassaforte.client :as client])
(require '[clojurewerkz.cassaforte.cql :as cql :refer :all])
(require '[clojurewerkz.cassaforte.query :refer :all])
(require '[qbits.hayt.dsl.statement :as hs])

(let [s (client/connect ["127.0.0.1"])]
  (cql/atomic-batch s (queries
                         (hs/insert :users (values {:name "Alex" :city "Munich" :age (int 19)}))
                         (hs/insert :users (values {:name "Fritz" :city "Hamburg" :age (int 28)})))))

Query DSL Taken From Hayt 2.0

Cassaforte no longer tries to support query condition DSLs for both Hayt 1.x and Hayt 2.0. Hayt 2.0 is the only supported flavour now and is the future.

Some examples of the changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
;; before
(where :name "Alex")

;; after
(where [[= :name "Alex"]])
(where {:name "Alex"})


;; before
(where :name "Alex" :city "Munich")

;; after
(where [[= :name "Alex"]
        [= :city "Munich"]])
(where {:name "Alex" :city "Munich"})


;; before
(where :name "Alex" :age [> 25])

;; after
(where [[= :name "Alex"]
        [> :age  25]])

;; before
(where :name "Alex" :city [:in ["Munich" "Frankfurt"]])

;; after
(where [[= :name "Alex"]
        [:in :city ["Munich" "Frankfurt"]]])

As it’s easy to see, the new condition style closer resembles Clojure itself and thus was a reasonable decision on behalf of Hayt developers.

Keyspace as Option

It is now possible to choose keyspace via an option:

1
2
3
4
5
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

(let [conn (cc/connect {:hosts ["127.0.0.1"] :keyspace "a-keyspace"})]
  )

Contributed by Max Barnash (DataStax).

Clojure 1.7.0-alpha2+ Compatibility

Cassaforte is now compatible with Clojure 1.7.0-alpha2 and later versions.

GH issue: #60.

Support for overriding default SSL cipher suites

Providing a :cipher-suites key in the :ssl connection option allows to specify cipher suites that are enabled when connecting to a cluster with SSL. The value of this key is a Seq of Strings (e.g. a vector) where each item specifies a cipher suite:

1
2
3
4
5
6
(ns cassaforte.docs
  (:require [clojurewerkz.cassaforte.client :as cc]))

(cc/build-cluster {:ssl {:keystore-path "path/to/keystore"
                         :keystore-password "password"}})]
                         :cipher-suites ["TLS_RSA_WITH_AES_128_CBC_SHA"]}}

The :cipher-suites key is optional and may be omitted, in which case Datastax Java driver’s default cipher suites (com.datastax.driver.core.SSLOptions/DEFAULT_SSL_CIPHER_SUITES) are enabled.

This can be used to work around the need to install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files required by the default set of cipher suites. TLS_RSA_WITH_AES_128_CBC_SHA is a suite in the default set that works with the standard JCE. E.g. by specifying just that one, as in the code example, the standard JCE is enough.

Contributed by Juhani Hietikko.

GH issue: #61.

News and Updates

New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.

Cassaforte is a ClojureWerkz Project

Cassaforte is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • EEP, a Clojure library for stream (event) processing
  • Neocons, a Clojure client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Cassaforte, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team.

Elastisch 2.1.0-rc1 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

2.1.0-rc1 is a release candidate of Elastisch 2.1, coming to you after 9 beta releases.

Changes between Elastisch 2.1.0-beta9 and 2.1.0-rc1

ElasticSearch Native Client Upgrade

Elastisch now depends on ElasticSearch native client version 1.4.x.

Clojure 1.4 Support Dropped

Elastisch no longer officially supports Clojure 1.4. Most of the functionality still works well on that version but please don’t file bugs specific to that version.

Allow :index key in update-aliases (native)

clojurewerkz.elastisch.native.index/update-aliases expects indices to be added to be specified in the :indices key while the respective REST function uses :index. This can have unexpected results, namely the creation of the respective alias for all indices. It is now possible to supply either :index or :indices to the function.

GH issue: #108.

Contributed by Yannick Scherer (stylefruits)

Update with Partial Document via native API

clojurewerkz.elastisch.native.document/update-with-partial-doc is a new function in the Native Client (existed before in the REST API) that performs partial updates:

1
2
3
(require '[clojurewerkz.elastisch.native.document :as doc])

(doc/update-with-partial-doc conn "people" "person" "1" {:country "Sweden"})

Note that the REST API should now be called without wrapping the document in a :doc key. I.e. change {:doc {:field-to-update "Update"}} to {:field-to-update "Update"}.

Contributed by Henrik Lundahl.

Changes between Elastisch 2.0.0 and 2.1.0-beta9

Ability to Specify Aliases In index.create-template

clojurewerkz.elastisch.rest.index.create-template now supports the :aliases option:

1
2
3
(require '[clojurewerkz.elastisch.rest.index :as idx])

(idx/create-template conn "accounts" {:template "account*" :settings {:index {:refresh_interval "60s"}} :aliases {:account-alias {}}})

Contributed by Jeffrey Erikson.

clj-http Update

clj-http dependency has been upgraded to version 1.0.x.

Allow Retry On Conflict Option

Updates and upserts now allow the retry-on-conflict option to be set. This helps to work around Elasticsearch version conflicts.

GH issue: #119.

Contributed by Michael Nussbaum (Braintree).

REST API Bulk Indexing Filters Out Operation Keys

clojurewerkz.elastisch.rest.bulk/bulk-index now filters out all operation/option keys so that they don’t get stored in the document body.

GH issue: #116.

Contributed by Michael Nussbaum (Braintree).

New Line in Multi-Search REST API

ElasticSearch Multi Search REST API endpoint is sensitive to the trailing new line. When it is missing, the response contains one result too few.

Elastisch now makes sure to append a new line to Multi Search request bodies.

Correct async-put in Native Client

Native client’s document/async-put no longer fails with an exception.

Contributed by Nikita Burtsev.

clj-time 0.8.0

clj-time dependency has been upgraded to version 0.8.0.

ElasticSearch Native Client Upgrade

Elastisch now depends on ElasticSearch native client version 1.3.x.

Single-Bucket Aggregation Fix in the Native Client

Child aggregations in single-bucket aggregations (i.e. “global”) are no longer silently dropped.

Contributed by Yannick Scherer (StyleFruits).

Aggregations Support in the Native Client

Native client now has support for aggregations.

The API is the same as in the REST client.

Note that ElasticSearch 1.2 has 26 aggregations. Currently only the most commonly used ones are supported but support for more types will be added eventually. The supported types are:

  • Avg
  • Max
  • Min
  • Sum
  • Stats
  • Extended stats
  • Cardinality, value count
  • Percentiles
  • Histogram
  • Date Histogram
  • Range
  • Date Range
  • Terms
  • Missing
  • Global

Multi-Search Support in the Native Client

Native client now has support for multi-search.

The API is the same as in the REST client except that the functions are in the clojurewerkz.elastisch.native.multi.

Extra Options on Upserts

clojurewerkz.elastisch.native.document/upsert now accepts a map of extra options, e.g. parent document ID:

1
(doc/upsert conn index-name index-type id doc {:parent parent-id})

Terms Query Helper

clojurewerkz.elastisch.query/terms is a newly added alias for clojurewerkz.elastisch.query/term when used with a collection.

Contributed by Martin Klepsch.

Remove Alias Now Works in Native Client

Bug fixed in native client for removing aliases from indices and improved inline documentation. See aliases in the guide.

GH issue: #98.

Highlighting Support in Native Client

Native client now supports (most of the) highlighting features the REST client does:

1
2
3
4
5
6
(require '[clojurewerkz.elastisch.native.document :as doc])
(require '[clojurewerkz.elastisch.query :as q])

(doc/search conn index type
            {:query (q/query-string :query "software" :default_field "summary")
             :highlight {:fields {:summary {}}}})

Per Connection clj-http Options in REST Client

It is now possible to specify clj-http options for REST API connections, e.g. to specify a timeout:

1
2
(esr/connect "http://127.0.0.1:9200/" {:conn-timeout 1000
                                       :basic-auth ["username" "pa$$w0rd"]})

Source Filtering Support in Native Client

Native client now supports source filtering just like the REST API client:

1
2
3
4
(doc/search conn index-name mapping-type
            :query   (q/match-all)
            :sort    {"first-name" "asc"}
            :_source ["first-name" "age"])
1
2
3
4
5
6
(doc/search conn index-name mapping-type
            :query   (q/match-all)
            :sort    {"first-name" "asc"}
            :_source {"exclude" ["title" "country"
                                 "planet" "biography"
                                 "last-name" "username"]})

GH issue: #73.

Search Can Return Fields and Source

Previously a search would return either the source document, or specific fields and not both. There are certain circumstances where having both are beneficial, for example when searching for a child document and you want to include the parent ID:

1
2
3
(require '[clojurewerkz.elastisch.native.document :as esd])

(esd/search conn "index" "child-type" :query (q/match-all) :fields ["_parent"])

The above would return the parent document ID in the :_parent field of each hit, but would not return the document itself. You can now have both by:

1
(esd/search conn "index" "child-type" :query (q/match-all) :fields ["_parent" "_source"])

Now the parent ID is in the :_parent field of each hit, and the matching document will be in :_source as per a normal search.

Contributed by Ben Ashford.

Update with Partial Document

clojurewerkz.elastisch.rest.document/update-with-partial-doc is a new function that performs partial updates:

1
2
3
(require '[clojurewerkz.elastisch.rest.document :as doc])

(doc/update-with-partial-doc conn "people" "person" "1" {:country "India"})

Contributed by Sandeep Jagtap.

Full Change Log

Elastisch change log is available on GitHub.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Cassaforte, a Clojure Cassandra client
  • Titanium, a Clojure graph library
  • Neocons, a client for the Neo4J REST API
  • Welle, a Riak client with batteries included
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

About the Author

Michael on behalf of the ClojureWerkz Team