The ClojureWerkz Blog

News and updates about ClojureWerkz projects

Elastisch 1.1.0 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

1.1.0 includes several new features, most notably a native client with the same API as the REST one Elastisch already has.

Supported ElasticSearch Versions

Elastisch 1.1’s native client requires ElasticSearch 0.90.0. ElasticSearch still evolves rapidly and binary protocol changes quite often, so the version match between the client and the server is currently a requirement for binary clients.

This restriction does not apply to REST API clients.

Changes in 1.1.0

Native Client

Elastisch 1.1.0 includes a major new feature: native ElasticSearch client. The client uses ElasticSearch’s Java API, and can be used with both transport and node clients.

Rationale

Native client is more bandwidth efficient. It also can use SMILE (binary JSON format) to be more efficient on the wire.

Namespace Layout

Native client API in Elastisch is nearly identical to that of the REST API client and resides in clojurewerkz.elastisch.native and clojurewerkz.elastisch.native.* namespaces (similarly to how clojurewerkz.elastisch.rest clojurewerkz.elastisch.rest.* namespaces are organized).

Connections

Transport client (used for TCP/remote connections) connections are set up using clojurewerkz.elastisch.native/connect!. Note that you need to provide node configuration that at least has cluster name in it:

1
2
3
4
5
6
(require '[clojurewerkz.elastisch.native :as es])

;; note that transport client uses port 9300 by default.
;; it also can connect to multiple cluster nodes
(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

Cluster name and transport node addresses can be retrieved via HTTP API, for example:

1
2
curl http://localhost:9200/_cluster/nodes
{"ok":true,"cluster_name":"elasticsearch_antares","nodes":...}}

Performing Operations

The Native client tries to be as close as possible to the existing REST client API. For example, document operation functions in clojurewerkz.elastisch.native.document, such as clojurewerkz.elastisch.native.document/create, follow clojurewerkz.elastisch.rest.document function signatures as closely as possible:

1
2
3
4
5
6
7
8
9
;; in the REPL
(require '[clojurewerkz.elastisch.native :as es])
(require '[clojurewerkz.elastisch.native.document :as doc])

(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

(doc/put index-name index-type id document)
(doc/get index-name index-type id)

The same with returned results. Note, however, that ES transport client does have (very) minor differences with the REST API and it is not always possible for Elastisch to completely cover such differences.

Async Operations

Native client offers a choice of synchronous (blocking calling thread until a response is received) and asynchronous (returns a future) versions of multiple API operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
;; in the REPL
(require '[clojurewerkz.elastisch.native :as es])
(require '[clojurewerkz.elastisch.native.document :as doc])

(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

(doc/put index-name index-type id document)

;; returns a response
(doc/get index-name index-type id)
;; returns a future that will eventually
;; contain a response
(doc/async-get index-name index-type id)

One notable exception to this is administrative operations (such as opening or closing an index). The rationale for this is that they are rarely executed on the hot code path (e.g. in tight loops), so convenience and better error visibility is more important for them.

GH issues: #17, #18, #20.

Clojure 1.5 By Default

Elastisch now depends on org.clojure/clojure version 1.5.0. It is still compatible with Clojure 1.3+ and if your project.clj depends on 1.3 or 1.4, it will be used, but 1.5 is the default now.

We encourage all users to upgrade to 1.5, it is a drop-in replacement for the majority of projects out there.

Bulk Request Support

Bulk requests are now supported. All the relevant code is in the clojurewerkz.elastisch.rest.bulk namespace. Here is a small example of bulk document indexing using this new API:

1
2
3
(require '[clojurewerkz.elastisch.rest.bulk :as eb])

(eb/bulk (eb/bulk-index doc1 doc2 doc3) :refresh true)

Contributed by Davie Moston.

Scroll Queries Support

Scroll queries are now easier to perform thanks to the new clojurewerkz.elastisch.rest.document/scroll function that takes a scroll id and amount of time retrieved documents and related information will be kept in memory for future retrieval. They are analogous to database cursors.

A short code example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(require '[clojurewerkz.elastisch.rest.document :as doc])
(require '[clojurewerkz.elastisch.query :as q])
(require '[clojurewerkz.elastisch.rest.response :refer [hits-from]])

(let [index-name   "articles"
      mapping-type "article"
      response     (doc/search index-name mapping-type
                               :query (q/query-string :query "*")
                               :search_type "scan"
                               :scroll "1m"
                               :size 1)
      scroll-id     (:_scroll_id response)
      scan-response (doc/scroll scroll-id :scroll "1m")
      scan-hits     (hits-from scan-response)]
  (println scan-hits))

Contributed by Davie Moston.

Cheshire Update

Cheshire dependency has been upgraded to version 5.1.1.

clj-http Update

clj-http dependency has been upgraded to version 0.7.2.

Count API No Longer Ignores Mapping Types

clojurewerkz.elastisch.rest.document/count no longer ignores mapping types.

GH issue: #6.

Count API now uses GET requests

clojurewerkz.elastisch.rest.document/count now correctly uses GET for requests without the query part andPOST for request that have it.

GH issue: #5.

Updates With Scripts

clojurewerkz.elastisch.rest.document/update-with-script is a new function that updates a document with a provided script:

1
2
3
4
5
6
7
8
9
10
(require '[clojurewerkz.elastisch.rest.document :as doc])

;; initializes a counter at 1
(doc/update-with-script index-name mapping-type "1"
      "ctx._source.counter = 1")

;; increments the counter by 4
(doc/update-with-script index-name mapping-type "1"
      "ctx._source.counter += inc"
      {"inc" 4})

clojurewerkz.elastisch.native.document/update-with-script is the native client counterpart that takes the same arguments.

Change Log

Elastisch change log is available on GitHub.

Thank You, Contributors

Kudos to Davie Moston, Andrew Jones, Jon Pither and Antonio Terreno who contributed to this release.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Langohr 1.0.0-beta14 Is Released

TL;DR

Langohr is a Clojure RabbitMQ client that embraces AMQP 0.9.1 Model.

1.0.0-beta14 is a development milestone release that includes a few minor features, mostly around error handling and recovery.

Changes between Langohr 1.0.0-beta13 and 1.0.0-beta14

Queueing Consumers

In its early days, Langohr has been using QueueingConsumer for langohr.queue/subscribe. It was later replaced by a DefaultConsumer implementation.

The key difference between the two is that

  • QueueingConsumer blocks the caller
  • with QueueingConsumer, deliveries are typically processed in the same thread

This implementation has pros and cons. As such, an implementation on top of QueueingConsumer is back with langohr.consumers/blocking-subscribe which is identical to langohr.consumers/subscribe in the signature but blocks the caller.

In addition, langohr.consumers/ack-unless-exception is a new convenience function that takes a delivery handler fn and will return a new function that explicitly acks deliveries unless an exception was raised by the original handler:

1
2
3
4
5
6
7
(require '[langohr.consumers :as lc])
(require '[langohr.basic     :as lb])

(let [f  (fn [metadata payload]
           (comment "Message delivery handler"))
      f' (lc/ack-unless-exception f)]
  (lb/consume ch q (lc/create-default :handle-delivery-fn f'))

Contributed by Ian Eure.

Shutdown Signal Functions

Several new functions in langohr.shutdown aid with shutdown signals:

  • langohr.shutdown/initiated-by-application?
  • langohr.shutdown/initiated-by-broker?
  • langohr.shutdown/reason-of
  • langohr.shutdown/channel-of
  • langohr.shutdown/connection-of

Clojure 1.5 By Default

Langohr now depends on org.clojure/clojure version 1.5.1. It is still compatible with Clojure 1.3 and if your project.clj depends on a different version, it will be used, but 1.5 is the default now.

We encourage all users to upgrade to 1.5, it is a drop-in replacement for the majority of projects out there.

Change Log

Langohr change log is available on GitHub.

Langohr is a ClojureWerkz Project

Langohr is part of the group of libraries known as ClojureWerkz, together with

  • Elastisch, a minimalistic well documented Clojure client for ElasticSearch
  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Langohr, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Elastisch 1.1.0-rc2 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature, provides both HTTP and native clients (as of 1.1) and has solid documentation.

1.1.0-rc2 is a preview release that improves native client performance.

Changes in 1.1.0-rc2

Native Client Performance Improvements

Native client is now over 50% faster on most commonly used operations thanks to much lower conversion overhead from ElasticSearch native client data structures to Clojure maps.

Elastisch change log is available on GitHub.

Thank You, Contributors

Kudos to Jon Pither and Antonio Terreno for doing the benchmarking, profiling and optimizations that made this substantial performance increase a reality.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Titanium, a powerful Clojure graph library
  • Neocons, a client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Titanium 1.0.0-beta1 Is Released

TL;DR

Titanium is a powerful Clojure graph library that is built on top of Aurelius Titan. It combines a Clojure-friendly API and graph processing DSL with the power of Titan.

beta1 is a major development milestone that has major public API changes, upgrades Titanium to Titan 0.3, provides a separate Clojure library for working with any Blueprints-compatible graph database and introduces a new powerful way of querying graphs: Ogre, a dialect of Gremlin.

Towards Making Clojure A Great Ecosystem For Graph Analysis

When we first introduced Titanium a few months ago, the response was very positive. People liked Titan features, swappable storage and straightforward transaction support.

Back then, Titanium wasn’t the only Titan-related Clojure project on the block. Zack Maril has been working on another project called Hermes for a while. Then Zack released Ogre, a Clojure library for querying Blueprints graphs.

Titanium and Hermes were largely solving the same problem and after discussing it for some time, we decided to join forces with Zack and continue working together on Titanium. Ogre and Archimedes are now ClojureWerkz projects and Titanium 1.0.0-beta1 is based on them.

We would like to thank Zack for considering joining the team and all the work he has been doing on Titanium and related Clojure graph libraries.

You should follow him on Twitter and GitHub.

Moving to Titan 0.3

Titan is still a young project and is actively being worked on. In version 0.3, Titan introduces several breaking API changes and new features such as

  • Full text search (indexing)
  • Better caching layer
  • Lots of optimizations
  • Properties on vertices can have properties on them which is very useful for version, timestamping, etc

Titanium closely follows these developments and 1.0.0-beta1 is based on Titan 0.3. Not all new Titan features are available in Titanium 1.0.0-beta1 but it won’t take long.

Improved Documentation

Titanium 1.0.0-beta1 includes a major documentation overhaul to adapt to the Titan 0.3 changes and separation into multiple libraries. In addition, Ogre documentation site is now also live and uses our standard documentation toolchain.

It’s not a ClojureWerkz project if it is not documented well!

Changes between Titanium 1.0.0-alpha2 and 1.0.0-beta1

Archimedes For Working With Blueprints Graphs

Titanium now relies a separate library called Archimedes for graph operations on any Blueprints-compatible graph database.

Ogre For Graph Querying

Titanium depends on Ogre, a powerful Clojure library for querying Blueprints-compatible graphs (a dialect of the Gremlin query language).

Ogre supercedes clojurewerkz.titanium.gpipe query DSL.

Titan Types Support

In Titan, edge and vertex properties can have custom types. Titanium now supports said custom types.

The documentation guide above provides more information.

Change Log

Titanium change log is available on GitHub.

News and Updates

New releases and updates are announced on Twitter. Titanium also has a mailing list, feel free to ask questions and report issues there.

How To Follow The Development

Titanium, Ogre and Archimedes are all available on GitHub.

If you have questions about these libraries, feel free to ask on the Titan mailing list or Titanium’s own mailing list.

Titanium is a ClojureWerkz Project

Titanium is part of the group of libraries known as ClojureWerkz, together with

  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Neocons, a Clojure client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Titanium, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team.

Scrypt 1.0.0 Is Released

TL;DR

ClojureWerkz Scrypt is a tiny Clojure library for working with scrypt, a key derivation function. Scrypt can be used to hash passwords and generate cryptographic keys. It is an alternative to Bcrypt and PBKDF2.

Why Use Scrypt

Values (e.g. passwords) encrypted with Scrypt are much more computationally hard to crack than with most other available key derivation functions:

Key Derivation Function Comparison

More can be found in the scrypt paper.

What’s In The Box

Scrypt is literally a couple of functions:

  • Encrypt a value (e.g. a password)
  • Verify a candidate against a hash (e.g. against a password used to authenticate)

Documentation

Scrypt is a small library and all of its documentation guide fits in the README.

Supported Clojure Versions

Scrypt supports Clojure 1.3+.

News and Updates

New releases and updates are announced on Twitter.

Scrypt is a ClojureWerkz Project

Scrypt is part of the group of libraries known as ClojureWerkz, together with

  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Titanium, a powerful graph library built on top of Titan
  • Neocons, a Clojure client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Scrypt, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team.

Elastisch 1.1.0-rc1 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

1.1.0-rc1 is a candidate release that has several new features, most notably a native client with the same API as the REST one Elastisch already has.

Supported ElasticSearch Versions

Elastisch 1.1’s native client requires ElasticSearch 0.90.0. ElasticSearch still evolves rapidly and binary protocol changes quite often, so the version match between the client and the server is currently a requirement for binary clients.

This restriction does not apply to REST API clients.

Changes in 1.1.0

Native Client

Elastisch 1.1.0 includes a major new feature: native ElasticSearch client. The client uses ElasticSearch’s Java API, and can be used with both transport and node clients.

Rationale

Native client is more bandwidth efficient. It also can use SMILE (binary JSON format) to be more efficient on the wire.

Namespace Layout

Native client API in Elastisch is nearly identical to that of the REST API client and resides in clojurewerkz.elastisch.native and clojurewerkz.elastisch.native.* namespaces (similarly to how clojurewerkz.elastisch.rest clojurewerkz.elastisch.rest.* namespaces are organized).

Connections

Transport client (used for TCP/remote connections) connections are set up using clojurewerkz.elastisch.native/connect!. Note that you need to provide node configuration that at least has cluster name in it:

1
2
3
4
5
6
(require '[clojurewerkz.elastisch.native :as es])

;; note that transport client uses port 9300 by default.
;; it also can connect to multiple cluster nodes
(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

Cluster name and transport node addresses can be retrieved via HTTP API, for example:

1
2
curl http://localhost:9200/_cluster/nodes
{"ok":true,"cluster_name":"elasticsearch_antares","nodes":...}}

Performing Operations

The Native client tries to be as close as possible to the existing REST client API. For example, document operation functions in clojurewerkz.elastisch.native.document, such as clojurewerkz.elastisch.native.document/create, follow clojurewerkz.elastisch.rest.document function signatures as closely as possible:

1
2
3
4
5
6
7
8
9
;; in the REPL
(require '[clojurewerkz.elastisch.native :as es])
(require '[clojurewerkz.elastisch.native.document :as doc])

(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

(doc/put index-name index-type id document)
(doc/get index-name index-type id)

The same with returned results. Note, however, that ES transport client does have (very) minor differences with the REST API and it is not always possible for Elastisch to completely cover such differences.

Async Operations

Native client offers a choice of synchronous (blocking calling thread until a response is received) and asynchronous (returns a future) versions of multiple API operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
;; in the REPL
(require '[clojurewerkz.elastisch.native :as es])
(require '[clojurewerkz.elastisch.native.document :as doc])

(es/connect! [["127.0.0.1" 9300]]
             {"cluster.name" "elasticsearch_antares" })

(doc/put index-name index-type id document)

;; returns a response
(doc/get index-name index-type id)
;; returns a future that will eventually
;; contain a response
(doc/async-get index-name index-type id)

One notable exception to this is administrative operations (such as opening or closing an index). The rationale for this is that they are rarely executed on the hot code path (e.g. in tight loops), so convenience and better error visibility is more important for them.

GH issues: #17, #18, #20.

Note that native ElasticSearch client currently relies on ElasticSearch 0.90.0.Beta1 client libraries and some operations will only work with that version.

Clojure 1.5 By Default

Elastisch now depends on org.clojure/clojure version 1.5.0. It is still compatible with Clojure 1.3+ and if your project.clj depends on 1.3 or 1.4, it will be used, but 1.5 is the default now.

We encourage all users to upgrade to 1.5, it is a drop-in replacement for the majority of projects out there.

Bulk Request Support

Bulk requests are now supported. All the relevant code is in the clojurewerkz.elastisch.rest.bulk namespace. Here is a small example of bulk document indexing using this new API:

1
2
3
(require '[clojurewerkz.elastisch.rest.bulk :as eb])

(eb/bulk (eb/bulk-index doc1 doc2 doc3) :refresh true)

Contributed by Davie Moston.

Scroll Queries Support

Scroll queries are now easier to perform thanks to the new clojurewerkz.elastisch.rest.document/scroll function that takes a scroll id and amount of time retrieved documents and related information will be kept in memory for future retrieval. They are analogous to database cursors.

A short code example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(require '[clojurewerkz.elastisch.rest.document :as doc])
(require '[clojurewerkz.elastisch.query :as q])
(require '[clojurewerkz.elastisch.rest.response :refer [hits-from]])

(let [index-name   "articles"
      mapping-type "article"
      response     (doc/search index-name mapping-type
                               :query (q/query-string :query "*")
                               :search_type "scan"
                               :scroll "1m"
                               :size 1)
      scroll-id     (:_scroll_id response)
      scan-response (doc/scroll scroll-id :scroll "1m")
      scan-hits     (hits-from scan-response)]
  (println scan-hits))

Contributed by Davie Moston.

Cheshire Update

Cheshire dependency has been upgraded to version 5.1.1.

clj-http Update

clj-http dependency has been upgraded to version 0.7.2.

Count API No Longer Ignores Mapping Types

clojurewerkz.elastisch.rest.document/count no longer ignores mapping types.

GH issue: #6.

Count API now uses GET requests

clojurewerkz.elastisch.rest.document/count now correctly uses GET for requests without the query part andPOST for request that have it.

GH issue: #5.

Change Log

Elastisch change log is available on GitHub.

We recommend all users to give 1.1.0-rc1 a try.

Thank You, Contributors

Kudos to Davie Moston and Andrew Jones who contributed to this release.

Road To 1.1

Elastisch 1.1 will be released once we update Elastisch documentation to cover all the new features.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Welle, a Riak client with batteries included
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Money 1.3.0 Is Released

Money is a Clojure library that deals with monetary amounts and currencies, built on top of Joda Money.

1.3.0 is a minor release that has no breaking API changes and several minor features.

Changes between Money 1.2.0 and 1.3.0

Hiccup Integration

clojurewerkz.money.hiccup, when compiled, will extend Hiccup’s HTML rendering protocol for monetary amounts and currency units. Internally it uses default formatter used by clojurewerkz.money.format: clojurewerkz.money.format/*default-formatter*, which can be rebound.

Formatting of Monetary Amounts

clojurewerkz.money.format/format is a new function that formats monetary amounts using default or provided locale:

1
2
3
4
5
6
7
8
9
10
(require '[clojurewerkz.money.currencies :as cu)
(require '[clojurewerkz.money.amounts :refer [amount-of])
(require '[clojurewerkz.money.format :refer :all])

(import java.util.Locale)

;; format using default system locale
(format (amount-of cu/GBP 20.0) Locale/UK) ;= GBP20,00
;; format using UK locale
(format (amount-of cu/GBP 20.0) Locale/UK) ;= £10.00

Custom Joda Money formatters are also supported.

Addition and Subtraction of Monetary Amounts

clojurewerkz.money.amounts/plus and clojurewerkz.money.amounts/minus now accept monetary amounts as well as doubles as 2nd argument.

Contributed by Ryan Neufeld.

Change Log

Money change log is available on GitHub.

Money is a ClojureWerkz Project

Money is part of the group of libraries known as ClojureWerkz, together with

  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Monger, a Clojure MongoDB driver for a more civilized age
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Money, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

@michaelklishin on behalf of the ClojureWerkz Team

IRC Channel for ClojureWerkz Project

ClojureWerkz IRC Channel on Freenode

Every once in a while we get a request to start an IRC channel for our projects where people can suggest ideas and ask questions.

We are happy to announce that we now have a channel: #clojurewerkz on irc.freenode.net. There are no strict “office hours” but it is much more likely that we will be around in European evening hours. Feel free to join and let us know how we can make ClojureWerkz projects better.

Michael.

How to Make Your Open Source Project Really Awesome

TL;DR

If you plan on releasing a library as open source, please make sure it has

  • Clear dependency/installation instructions
  • At least one brief documentation guide
  • A change log and tags in the repo
  • Some information about supported language/runtime/tool versions and project maturity
  • A mailing list where users can ask questions and help each other

Doing anything less than this will cause some of your users grief and anger. And likely some wasted time.

How To Make Your Open Source Library Really Awesome

Year after year, more and more people release libraries they’ve developed and would like to open source. Today we’d like to share a little bit of our experience about how to make sure users of your library are content with it.

There is one Rule of Thumb to this:

Don’t make your users mad

also known as

Don’t make your users want to break the computer

now lets see how specifically we can achieve that goal with a reasonable amount of effort.

Give It a Really Useful README

Even if your project has a nice site, it is likely that the first point of contact of potential users with it will be by reading the README file. We need to make sure it is awesome and contains useful information.

Provide Dependency Information

So you release open source software. This probably means you are kinda smart. Good for you. Unfortunately, not everybody is and some people are completely new to the language/ecosystem your project is in. This means that what is obvious to you is not at all obvious to many other people.

One such thing is missing dependency/installation information.

How do I even install this, was it so hard to make this clear?

is something that quickly leads to anger. Digging in the source to find our your artifact/gem/package name is annoying and some projects use really creative (read: stupid) names for artifacts that do not match code repository name, etc.

Relieve your users from having to deal with this crap. Make it dead obvious how to add a dependency on your project, ideally by copy-pasting a snippet.

See Welle if you need an example.

State Project Maturity Clearly

Have you been using it in production for months? Is it something you still believe is incomplete? Do you expect the API change drastically in the next version? Is this project mature and safe to use even for the most demanding and conservative projects?

State all that clearly. And next time you won’t waste a week of your time because you’ve made the wrong call on a library that did not provide any maturity information, you will realize how much difference a few sentences can make to real people.

Document Supported Runtime/Language/Tool Versions

Clojure has a very good track record when it comes to backwards compatibility. It is almost too good to be true. Besides the 1.2 to 1.3 migration, subsequent releases are pretty much a drop-in replacement for the vast majority of projects out there. As such, projects that have moved beyond 1.2 are generally on the most recent stable release

However, this won’t always be the case. At some point, a future Clojure release will have to break compatibility. How do we not make our users furious? By clearly stating what versions are supported in the README.

All it takes is writing one line of text. One less tweet on the week of your release and you save beginners potentially a lot of head scratching.

If you need an example, here’s one from Welle.

State What License You Use

You may not care about licenses much, but people who want to use your library at a BigBucks Inc do. They must know. And it may be a deal breaker for them when they want to adopt Clojure/Node.js/Scala/Go/whatever.

So state your license clearly. And please, use something business-friendly unless you have good reasons. Apache Public License 2.0) and Eclipse Public License) are good, friendly choices. Note that some licenses (e.g. MIT) are friendly and popular but do not offer any patent protection, which should not be ignored in the current legal climate.

Finally, remember that you can double-license, e.g. APL2/GPLv2 if you really want to be license-neutral. Then your users will pick what suits them.

When in doubt, refer to TL;DR Legal, open source licenses summarized & explained in plain English (but don’t consider it to be legal advice).

How To Fuck It Up

If you want to completely screw your users, do this:

  • Put a nearly blank README to your project’s root
  • Make it say “patches welcome” at the end
  • Invent your own license or use a completely unfamiliar one, such as WTFPL

Then your project will never have any users. Guaranteed.

Document Your Project

Documenting things is not easy and can be time consuming. However, documentation is the single best thing you can do to your users. Not only it saves them a lot of time, it makes them confident that your library is not abandonware.

Documentation helps your users accomplish things they decided to use your library in the first place. In the words of Rob Pike, it “empowers them”. It shows them that you care. It shows them that you are a real person, not a robot spitting code.

After about 2 years of working on ClojureWerkz, I can confidently say that there is nothing our users are more thankful for than documentation for our projects:

Writing great documentation takes time. Fortunately, modern tooling also helps you quite a bit and reduces the annoyances you have to deal with by a lot.

For ClojureWerkz projects, we’ve open sourced our Jekyll-based documentation site template. We are not great at CSS and the visual aspect of design, so we use Twitter Bootstrap. While our doc sites could look better, it is certainly pretty good compared to most open source projects out there. And you can use our template or create a similar one for your projects.

Better yet, if you open source your documentation site (and there is no reason to not do so), you will likely see people contributing minor improvements and corrections earlier than they will contribute code changes.

If you are still not sure whether documenting your project is worth it, watch this talk by Jacob Kaplan-Moss:

How To Fuck It Up

If you want to completely screw your users, do this:

  • Don’t write a single documentation guide, not even one example
  • Make sure your API reference is outdated by at least 3 months
  • Claim that people who are not willing to read the code to figure out even the most basic things are stupid and should flip burgers instead

Make It Easy to Upgrade

At some point, you are going to make another release of your project. This is another opportunity to either make your users happy that they’ve gone with your library or make them furious and waste their time.

Give a Fuck About Backwards Compatibility

One of the most infuriating things about software development is that moment when you upgrade a library and hundreds of tests fail. The only thing that angers me more is when I also have to rewrite half of my codebase because someone somewhere decided that breaking the public API without warning is fine.

So, commit to maintaining backwards compatibility. Of course, you don’t have to support projects from 15 years ago like OpenJDK does. But deprecate things before removing them and make it easy to discover what has changed.

How do you do that? By keeping a change log.

Have a Change Log

Every once in a while, your users will upgrade (more on this later in this post). The main questions they are going to ask themselves is

What has changed in this release?

and then

Will my code break? Will I have to rewrite it?

and finally,

Will Joe the ops guy hate me for upgrading?

All of these can be answered by having a change log. It’s like Twitter except it’s actually useful. Here’s how it works:

  • Every time you fix a bug, you write a brief entry to the change log
  • Every time you add a feature, you briefly mention it in the change log with a few code examples
  • Every time you make a breaking API change, you clearly state that in bold in the change log

That’s it. There is no step 4.

Change logs typically use reverse chronological order. Changes are grouped by versions. If you have multiple branches (e.g. master and 1.0.x), each one has a separate change log.

That is all. For example, here’s Welle’s change log.

Tag Releases

It’s that time again. You’ve bumped the version and about to publish artifacts. Stop and make another thing first: tag the commit that changes the version. Without tags it will be a royal pain to come up with a diff betwee nversions.

Some dependency (e.g. Bundler, Rebar) and configuration management tools can use tags and developers will expect tags to be available.

Use a consistent version scheme, e.g. v1.0.0-alpha1, v1.0.0, v1.1.2, etc. Having inconsistent tag names will guaratee ops people will hate your project till the end of days.

Announce Releases

Next thing to do after you push a release out is to write a blog post or simply a post to the mailing list of your project or a larger relevant mailing list (e.g. Clojure mailing list or RabbitMQ Discuss).

Make sure the subject starts with ANN or [ANN] to indicate that it is an announcement, e.g.

ANN Welle 1.5.0 is released

In your announcement make it clear what the project does and if the release is backwards-compatible or not, and link to the change log where users can find more details. That’s it.

Use Preview/Snapshot Versions For Development

Ever saw a project that has the same version, e.g. 0.2.1, for half a year? How do you know which version is actually 0.2.1? Is it a version in development? Did someone forget to bump it after a release? What’s going on?

This is driving all developers crazy. Don’t be that person. Use preview/snapshot versions for development and only “seal” versions when you are about to tag and cut a release. Then immediately bump the version to development.

Examples of development versions:

  • 1.1.0.pre1
  • 1.1.0-alpha1
  • 1.1.0-SNAPSHOT (for JVM languages)

Any other development version naming scheme is confusing and will guarantee your users a bad time.

How To Fuck It Up

If you want to completely screw your users, do this:

  • Break your public API randomly, ideally in subtle ways that your test suite won’t even reveal
  • Forget to bump development version
  • Never tag releases
  • Never announce releases

Use GitHub

I have no affiliation with GitHub and don’t assume Git is “the best” version control system out there. But it’s pretty good. And nearly everybody is on GitHub these days.

GitHub makes several things easy:

  • Discover your project
  • Browse and search code
  • Bring your attention to issues by filing issues or @mentioning you
  • Contribute trivial changes

Maybe the most importantly, GitHub is pretty friendly to less technical people. Yes, it is, and they are working hard on making it even better.

Using GitHub means you’ll have an extremely easy to use CI service: Travis CI.

You will get a lot more admiration from your users if you won’t make their life harder by asking them to deal with patches, look for your email across the Web to report issues and clone your 300 MB repository over poor 3G just to edit one typo.

Don’t be difficult.

Have a Place Where Users Can Get Support

If your project reaches any reasonable kind of popularity, you will have to answer questions about it. For that, set up a mailing list (a Google group will do) and if you frequent on IRC, start a channel.

Think you don’t have any time? The best part about having a mailing list is that users will quickly start helping themselves if you give them a way. So clearly indicate ways to get support in your project’s README.

Search Twitter for the name of your project every so often. You will discover all kinds of questions, criticism and praises. If you use Twitter actively, start a separate account for your project, like we have done with @ClojureWerkz.

It will help you build a community, learn about how people use your project and what can be improved. Finally, it will help you find people who will be able to help maintain your project. Not only it will save you time, it will encourage those people to spread the word about it.

If you need an example, Welle README has a section about community and support.

How To Fuck It Up

If you want to completely screw your users, do this:

  • Be hip and disable issues on your GitHub repository
  • Use a contributor agreement users have to mail in paper to Tanzania
  • Insist on using patches even for one line changes to the README
  • Put your repo into Darcs even if it’s a Ruby or JavaScript or Clojure project
  • Make it hard to find where the repository is

This will prevents others from contributing to your project and stealing some of the credit from you.

Pass It Over

At some point you may become disinterested in maintaining your project. Maybe you’ve moved on to a new job or no longer use your own project. Announce it on the mailing list, ask someone to take over your project. Sooner or later, someone will help. Being on GitHub helps with this, especially after they’ve introducted a feature that lets you transfer repository ownership.

No matter what you do, don’t turn your project into abandonware. This is the most sure way to want your current and future users to go on a kitten killing spree.

It’s always better to pass the project on than to make excuses later.

How To Fuck It Up

If you want to completely screw your users, do this:

  • Simply stop contributing to your project and answering mailing list questions without any explanation
  • Ignore pull requests, claim they should be called something else and are not really useful
  • Claim you simply are a Get Shit Done guy who loses interest once the problem is done

This will make sure your project will eventually be forked at least 300 times and an alternative will be created because figuring out which fork has what bugs fixed will be too annoying.

Final Thoughts

As you can see, it is not that hard to make your project approachable. Besides documentation guides, it does not take much to make sure your users are not mad and ops people don’t hate you.

Maintaining open source projects takes time and effort. But it pays off. I’ve been on GitHub since beta and can say that few other things have brought me more professional opportunities I’m blessed to have today than being active in the open source community.

And if you are not going to make something awesome, maybe do not release it in the first place.

Michael.

Welle 1.5.0 Is Released

TL;DR

Welle is an expressive Clojure client for Riak with batteries included.

1.5.0 is a minor release that is 100% backwards-compatible with 1.5.x and introduces several nice features:

  • Retriers for failing operations
  • Conflict resolvers
  • Easier to use update functions

This version has been tested against Riak 1.2.x and 1.3.x.

Changes between Welle 1.4.0 and 1.5.0

clojurewerkz.welle.kv/modify

clojurewerkz.welle.kv/modify is a new function that combines clojurewerkz.welle.kv/fetch and clojurewerkz.welle.kv/store with a user-provided mutation functions. The mutation function should take a single Riak object as an immutable map and return a modified one.

In case of siblings, a resolver should be used.

clojurewerkz.welle.kv/modify will update modification timestamp of the object.

clojurewerkz.welle.kv/modify takes the same options as clojurewerkz.welle.kv/fetch and clojurewerkz.welle.kv/store

Conflcit Resolvers

clojurewerkz.welle.kv/fetch, and clojurewerkz.welle.kv/store now accept a new option: :resolver. Resolvers are basically pure functions that take a collection of siblings and return a collection of Riak object maps.

Resolvers can be created using clojurewerkz.welle.conversion/resolver-from which takes a function that accepts a collection of deserialized (unless fetch was told otherwise) values and applies any conflict resolution logic necessary.

clojurewerkz.welle.kv/fetch-one now also supports resolvers via the :resolver option. It will raise an exception if siblings are detected and no resolver is provided.

Retriers

clojurewerkz.welle.kv/fetch, clojurewerkz.welle.kv/fetch-one, clojurewerkz.welle.kv/store, clojurewerkz.welle.kv/delete, and clojurewerkz.welle.kv/index-query now retry operations that fail due to a network issue or any other exception.

By default, the operations will be retrier 3 times. It is possible to provide a custom retrier using the :retrier option. Retriers can be created using clojurewerkz.welle.conversion/retrier-from which takes a function that accepts a callable (an operation that may need to be retried) and needs to invoke it, handling exceptions and applying any retrying logic needed.

clojurewerkz.welle.conversion/counting-retrier produces a retrier that will retry an operation given number of times. This is the kind of retrier Welle uses by default.

Skipping Deserialization for clojurewerkz.welle.kv/fetch

clojurewerkz.welle.kv/fetch supports a new boolean option :skip-deserialize that allows automatic deserialization to be skipped.

Contributed by Jonas Tehler.

Clojure 1.5 By Default

Welle now depends on org.clojure/clojure version 1.5.1. It is still compatible with Clojure 1.3+ and if your project.clj depends on a different version, it will be used, but 1.5 is the default now.

We encourage all users to upgrade to 1.5, it is a drop-in replacement for the majority of projects out there.

Change Log

We recommend all users to upgrade to 1.5.0 a try.

Welle change log is available on GitHub.

Welle is a ClojureWerkz Project

Welle is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Monger, a Clojure MongoDB client for a more civilized age
  • Neocons, a feature rich idiomatic Clojure client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Welle, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Donations

ClojureWerkz accepts donations. If you feel like our projects save you time, consider donating. Thanks.

Michael on behalf of the ClojureWerkz Team