The ClojureWerkz Blog

News and updates about ClojureWerkz projects

Monger 1.6.0 Is Released

TL;DR

Monger is an idiomatic Clojure MongoDB driver for a more civilized age. It has batteries included, offers powerful expressive query DSL, strives to support every MongoDB 2.0+ feature and has sane defaults. It also has solid documentation.

1.6.0 is a minor completely backwards-compatible release that makes it easier to use Monger with multiple databases and MongoDB servers.

Changes between 1.5.0 and 1.6.0

monger.multi.collection

monger.multi.collection is a new namespace with functions that are very similar to those in the monger.collection namespace but always take a database reference as an explicit argument.

They are supposed to be used in cases when relying on monger.core/*mongodb-database* is not enough.

Erik Bakstad contributed most of this work.

MongoDB Java Driver Update

MongoDB Java driver dependency has been updated to 2.11.2.

monger.core/drop-db

monger.core/drop-db is a new function that drops a database by name.

One More Cache Implementation

monger.cache/db-aware-monger-cache-factory will return a MongoDB-backed clojure.core.cache implementation that can use any database:

1
2
3
4
5
6
7
(require '[monger.core  :as mg])
(require '[monger.cache :as cache])

(let [db   (mg/get-db "altcache")
      coll "cache_entries"
      c    (cache/db-aware-monger-cache-factory db coll)]
  (comment "This cache instance will use the altcache DB"))

Ragtime changes

Bug fix: monger.ragtime/applied-migration-ids now returns a vector (instead of a set) in order to preserve the original creation order of the migrations.

Ragtime dependency has been updated to 0.3.3.

Change Log

Monger change log is available on GitHub.

Thank You, Contributors

Erik Bakstad contributed key features in this release.

Monger is a ClojureWerkz Project

Monger is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Cassaforte, a Clojure Cassandra client built around CQL
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Monger, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

@michaelklishin on behalf of the ClojureWerkz Team

Pantomime 2.0.0 Is Released

TL;DR

Pantomime is a tiny Clojure library for working with MIME types and file metadata.

Pantomime 2.0 introduces a new (natural) language detection feature built on top of Apache Tika and drops support for Clojure 1.3.

Future versions will likely introduce more functionality built on top of Tika, making Pantomime more than just a “tiny libarry for working with MIME types”).

Changes between Pantomime 1.8.0 and 2.0.0

Clojure 1.3 is No Longer Supported

Pantomime 2.0 drops support for Clojure 1.3.

Language Detection Support

pantomime.languages is a new namespace that provides functions for detecting natural languages:

1
2
3
4
5
6
7
(require '[pantomime.languages :as pl])

(pl/detect-language "this is English, it should not be hard to detect")
;= "en"

(pl/detect-language "parlez-vous Français")
;= "fr"

Change Log

We recommend all users to upgrade to 2.0.0.

Pantomime change log is available on GitHub.

Pantomime is a ClojureWerkz Project

Pantomime is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Cassaforte, a Clojure Cassandra client built around CQL
  • Monger, a Clojure MongoDB client for a more civilized age
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Pantomime, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Pantomime 1.8.0 Is Released

TL;DR

Pantomime is a tiny Clojure library that deals with MIME types (Internet media types, “content types”). Pantomime 1.8 upgrades Apache Tika to 1.4.

Changes between Pantomime 1.7.0 and 1.8.0

Apache Tika 1.4

Apache Tika dependency has been upgraded to version 1.4.

Change Log

We recommend all users to upgrade to 1.8.0.

Pantomime change log is available on GitHub.

Pantomime is a ClojureWerkz Project

Pantomime is part of the group of libraries known as ClojureWerkz, together with

  • Monger, a Clojure MongoDB client for a more civilized age
  • Welle, a Riak client with batteries included
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Neocons, a client for the Neo4J REST API
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Quartzite, a powerful scheduling library

and several others. If you like Pantomime, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Elastisch 1.1.1 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

1.1.1 is a bug fix release. We recommend all users to upgrade to it.

Changes in 1.1.1

Search Queries with a Subset of Fields are Converted Correctly

Search queries that only retrieve a subset of fields using the :fields option are now correctly converted to Clojure maps.

Contributed by Soren Macbeth.

We recommend all users to upgrade to 1.1.1 as soon as possible.

Elastisch change log is available on GitHub.

Thank You, Contributors

Kudos to Soren Macbeth for contributing a fix to this release.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Cassaforte, a Clojure Cassandra client
  • Titanium, a Clojure graph library
  • Neocons, a client for the Neo4J REST API
  • Welle, a Riak client with batteries included
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team

Carmine 2.0.0-beta1 Is Released

TL;DR

Carmine is fantastic Clojure client for Redis.

2.0.0 is a major release that introduces some breaking API changes, new features and another 20% performance improvement (now within less than 5% of the Java client).

Changes in Carmine 2.0

  • 20% better performance
  • Pluggable data compression and encryption based on Nippy 2.0
  • Improvements to message queueing and distributed locking features
  • Tundra, a way to plug a durable data store into Carmine for long-term data persistence

(sorry that the change log is so concise. We’ve asked Peter to keep a slightly more detailed one!)

Carmine 2.0 Migration Guide

Carmine users should get themselves familiar with the Carmine 2.0 Migration Guide before upgrading.

Carmine Has Sister Projects

Carmine is part of the group of libraries by Peter Taoussanis, together with

  • Timbre, a pure Clojure logging and profiling library
  • Tower, a pure Clojure library for i18n and l10n
  • Nippy, a fast drop-in replacement for the Clojure reader
  • Faraday, a Clojure DynamoDB client
  • Touchstone, a Clojure library for split testing

If you like these projects, you should follow Peter on Twitter.

@michaelklishin on behalf of the ClojureWerkz Team

Nippy 2.0.0-rc1 Is Released

TL;DR

Nippy is an attempt to provide a reliable, high-performance drop-in alternative to the Clojure reader.

2.0.0 is a major completely backwards-compatible release that includes several new features.

Changes in Nippy 2.0

Nippy 2.0 is a backwards-compatible release that introduces several new features:

  • Pluggable compression
  • Pluggable data encryption support
  • Much improved error messages
  • Easier integration with other libraries

(sorry that the change log is so concise. We’ve asked Peter to keep a slightly more detailed one!)

Nippy Has Sister Projects

Nippy is part of the group of libraries by Peter Taoussanis, together with

  • Carmine, a fantastic Clojure client for Redis
  • Faraday, a Clojure DynamoDB client
  • Timbre, a pure Clojure logging and profiling library
  • Tower, a Clojure library for split testing
  • Touchstone, a pure Clojure library for i18n and l10n

If you like these projects, you should follow Peter on Twitter.

@michaelklishin on behalf of the ClojureWerkz Team

Monger 1.6.0-beta3 Is Released

TL;DR

Monger is an idiomatic Clojure MongoDB driver for a more civilized age. It has batteries included, offers powerful expressive query DSL, strives to support every MongoDB 2.0+ feature and has sane defaults. It also has solid documentation.

1.6.0-beta3 is a milestone completely backwards-compatible release that includes updated dependencies, an alternative API that makes it easier to work with multiple databases and a few other minor features.

Changes between 1.5.0 and 1.6.0

monger.multi.collection

monger.multi.collection is a new namespace with functions that are very similar to those in the monger.collection namespace but always take a database reference as an explicit argument.

They are supposed to be used in cases when relying on monger.core/*mongodb-database* is not enough.

Erik Bakstad contributed most of this work.

MongoDB Java Driver Update

MongoDB Java driver dependency has been updated to 2.11.2.

monger.core/drop-db

monger.core/drop-db is a new function that drops a database by name.

One More Cache Implementation

monger.cache/db-aware-monger-cache-factory will return a MongoDB-backed clojure.core.cache implementation that can use any database:

1
2
3
4
5
6
7
(require '[monger.core  :as mg])
(require '[monger.cache :as cache])

(let [db   (mg/get-db "altcache")
      coll "cache_entries"
      c    (cache/db-aware-monger-cache-factory db coll)]
  (comment "This cache instance will use the altcache DB"))

Change Log

Monger change log is available on GitHub.

Monger is a ClojureWerkz Project

Monger is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Cassaforte, a Clojure Cassandra client
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Monger, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

@michaelklishin on behalf of the ClojureWerkz Team

Introducing Cassaforte

TL;DR

Cassaforte is a new Clojure client for Apache Cassandra 1.2+. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.

Another Brick in the Wall

ClojureWerkz projects are mostly known for our data store clients. Data processing is a sweet spot for Clojure and that’s what we primarily use it for.

Over time we’ve developed and released a number of data store clients:

and a few others.

Today we are adding another client to this family, Cassaforte.

Cassaforte Story

Cassaforte was started about 11 months ago as an experiment. Back then we needed a database well suited for time series data, and Cassandra is a good choice. However, dealing with existing clients, namely Hector and Astyanax, was quite a bit of a pain.

In the end, using Cassandra’s low level Thrift client worked well and we have been improving the codebase bit by bit, while working on an event collection and analytics system.

A few months ago, DataStax released a new Java driver for Cassandra that was a lot better suited to what Cassaforte needed to power it than any other alternative.

We were able to switch Cassaforte to it in a couple of weeks. About the same time, a great fellow Max Penet released a Clojure DSL for generating CQL, Hayt.

Hayt makes working with CQL as nice as the new clojure.java.jdbc DSL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(require '[clojurewerkz.cassaforte.cql :as cql])

(cql/prepared
   (cql/insert "users" {:name "Alex" :city "Munich")}))
(cql/prepared
   (cql/insert "users" {:name "Robert" :city "Berlin")}))

(cql/select "users" (where :name "Alex"))
;; => [{:name "Alex", :city "Munich"}]

(cql/select "users"
            (where :name [:in ["Alex" "Robert"]]))
;; => [{:name "Alex", :city "Munich"}
       {:name "Robert", :city "Berlin"}]
1
2
3
4
5
6
7
8
9
10
(require '[clojurewerkz.cassaforte.cql :as cql])

;; For clarity, we select :post_id column only
(cql/select "user_posts"
            (columns :post_id)
            (where :username "Alex")
            (order-by [:post_id :desc]))
;; => [{:post_id "post3"}
       {:post_id "post2"}
       {:post_id "post1"}]
1
2
3
4
5
6
7
8
(require '[clojurewerkz.cassaforte.cql :as cql])

(cql/select "user_posts"
            (columns :post_id)
            (where :username "Alex"
                   :post_id [> "post1"]
                   :post_id [< "post3"]))
;; => [{:post_id "post2"}]

Integrating Hayt took a few weeks as well. From there, it took some time to write initial documentation and add some polish.

Major props to the DataStax team for releasing such a nice and focused Java driver and Max for making querying Cassandra from Clojure such a good experience.

What’s In The Box

Cassaforte is young but already offers all the key features you’d expect from such a client:

  • Schema (keyspaces, tables/column families, indices) manipulation
  • All CQL operations
  • Inserts that work with Clojure data structures
  • CQL 3.0 queries, including queries with placeholders (?, a la JDBC)
  • Prepared statements
  • Counters
  • Multi-node (cluster) connections
  • Automatic deserialization of column names and values according to the schema
  • Lazy sequence-based queries over large result sets

Needless to say, as more features are added to Cassandra and the DataStax core Java driver, Cassaforte will soon to follow their lead.

Documentation

Cassaforte documentation is far from finished but it follows the “if it is not documented, it is not part of ClojureWerkz” rule. To get started, take a look at the Getting Started guide.

The rest of the guides are being worked on.

API reference is also available.

Thank You, Contributors

Cassaforte was started by Michael but ended up being a brain child of Alex.

We would like to thank Max Penet for providing a lot useful feedback, sharing Hayt with us and helping us define what we want Cassaforte to be and feel like.

News and Updates

New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.

Cassaforte is a ClojureWerkz Project

Cassaforte is part of the group of libraries known as ClojureWerkz, together with

  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Monger, a Clojure MongoDB client for a more civilized age
  • Elastisch, a minimalistic Clojure client for ElasticSearch
  • Welle, a Riak client with batteries included
  • Neocons, a Clojure client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Cassaforte, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team.

How We Document Our Projects With ClojureWerkz Docslate

This post will explain how our team was able to greatly reduce the friction in writing documentation guides by standardizing on a toolchain that other projects can use.

A Bit of History

Before there was ClojureWerkz, myself and Alex were fairly active in the open source community. I maintain several RabbitMQ clients and needed a new documentation site for one of them.

So I put together a doc site based on Jekyll and Alex helped me with styles. This toolchain has evolved a bit and when we were looking at writing documentation for Monger and Langohr in late 2011, we decided to just base them on what we already had and knew: that Jekyll site amqp gem was using.

ClojureWerkz grew and we kept reusing the same thing over and over again. By June 2013, ClojureWerkz has accumulated about a dozen of projects that have substantial documentation guides. Writing, editing and maintaining these docs took time and in the process, we’ve found ourselves repeating a lot of things between sites.

It’s easy to notice a lot of similarities between, say, Elastisch docs and Langohr docs. We also try to take good ideas from doc sites developed later and feed them back into older sites.

So we needed a standard toolchain that all these documentation projects could use. In particular, we wanted it to give us a very fast way of writing the first couple of guides before we would worry about finer details and editing.

Enter ClojureWerkz Docslate

Docslate is a relatively unknown project. We don’t write about it on this blog. It does not get updates very often. It does, however, play a major role in ClojureWerkz: powering our documentation guides and making writing them a more pleasant and straightforward process.

What Is Docslate

Docslate is a Jekyll site with predefined structure and integrated Twitter Bootstrap. Once you have it going, with just a few edits you are ready for the work that really matters: writing helpful documentation guides.

Why You Should Adopt Docslate For Your Project

Docslate lets you get to writing quickly without spending too much time on issues such as

  • Should I use Jekyll, Awestruct, hack my own thing with Pandoc or use a wiki?
  • What library should I use for code highlighting?
  • How do I make the docs look decent?
  • What licenses should I use for the site and content?

and so on.

It’s dead easy to put together a pretty good documentation site with Docslate and then work on copy and styling once your first version is ready to go live.

How To Get Started With Docslate

Since Docslate uses Jekyll, you’ll need Ruby (any version, although we typically use 2.0 or 1.9.3 these days) and RubyGems installed.

Then install Bundler with

gem install bundler

Clone Doclsate Git repository with

git clone git://github.com/clojurewerkz/docslate.git my-project-guides

Change to your local clone, remove .git directory and re-init the repo (Docslate is simply a starting point, your doc site should really be in a new repo):

cd my-project-guides
rm -rf ./.git
git init
git add .
git commit -m "Initial commit: Docslate"

Then install dependencies with Bundler:

bundle install --binstubs

and you are ready to run a development server with

./bin/jekyll serve --watch

then navigate to localhost:4000.

The site you’ll see is very similar in structure and styles to Elastisch docs, Monger docs, and so on.

How To Add Content

Docslate assumes your documentation guides have an index page, a ToC page and one or more guides. The index page is in HTML and can be found in index.html in the repository root.

The ToC page is in Markdown and can be found with individual guides under articles.

Docslate contains 3 pages for you to start with:

  guides.md
  getting_started.md
  community.md

guides.md is the ToC page. getting_started.md contains the Getting Started guide. Finally, community.md gives you a page to list information about the project, its mailing list, IRC channel, Twitter account, and other things you need to make your open source project really awesome.

Adding new guides is as straightforward as adding a Markdown file to the articles directory with the header (title, etc) Jekyll expects and linking to it from the ToC page.

For example, to add a guide on extending your library, create a new file at articles/extensions.md, add Jekyll header to it, make sure development server is started and navigate to localhost:4000/articles/extensions.html.

How To Customize Layout

Layout files can be found under _layouts. There are two layouts available:

  • Default layout
  • Article layout

Article layout is used to display individual articles (guides), default layout is used for everything else.

How To Customize Styles and Assets

Styleshsets and JavaScript files can be found under assets/stylesheets and assets/javascripts, respectively.

How To Generate The Site

Use

./bin/jekyll build

to generate the site for deployment. Site root then will be under _site. This directory is what you will deploy. It only contains static assets so deployment usually is as straightforward as rsync-ing files to your server and serving them with Nginx, Apache or any other Web server of your choice.

Where To Go From Here

Docslate is a starting point. It is meant to be customized for your project’s needs, styling and so on. Feel free to do it, add JavaScript libraries you need, upgrade and customize Twitter Bootstrap, and so on.

Wrapping Up

Docslate was born out of a dozen documentation sites we have for ClojureWerkz projects. It’s a battle tested, easy to use toolchain that provides you a good starting point and is completely open sourced under a reasonable license.

Now you don’t have an excuse to not document your open source project well!

Michael on behalf of the ClojureWerkz Team.

Elastisch 1.2.0-beta1 Is Released

TL;DR

Elastisch is a battle tested, small but feature rich and well documented Clojure client for ElasticSearch. It supports virtually every Elastic Search feature and has solid documentation.

1.2.0-beta1 is a development release that has improved sorting support in the native client.

Changes between Elastisch 1.1.0 and 1.2.0-beta1

Sort Improvements for Search Queries

clojurewerkz.elastisch.native.document/search now accepts maps as :search option values:

1
2
(doc/search index-name mapping-type :query (q/match-all)
                                    :sort  (array-map "title" "asc")

This is identical to how the option works with the REST client.

Elastisch is a ClojureWerkz Project

Elastisch is part of the group of libraries known as ClojureWerkz, together with

  • Monger, a Clojure MongoDB client for a more civilized age
  • Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
  • Cassaforte, an easy to use Clojure Cassandra client built around CQL3
  • Titanium, a powerful graph library on top of the Tinkerpop stack
  • Welle, a Riak client with batteries included
  • Neocons, a client for the Neo4J REST API
  • Quartzite, a powerful scheduling library

and several others. If you like Elastisch, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team