Friday, 20 November 2015

Document Modeling Basics

An often asked question of developers those are new to NoSQL is how to start with the document modeling. This article does not aim to give you answers to all document modeling related questions. It is more a starting point.

Flexible Schema

I am personally not a big fan of the word 'schema free'. My personal opinion is that if we talk about structured data, then we also talk about how to structure the data. (BTW: Couchbase also allows to store unstructured data as binaries. Also semi-structured is supported by e.g. embedding base64 encoded strings into JSON documents.) Couchbase Server does not enforce (on the database side) to follow a specific schema. This brings you more flexibility. Some documents might have a specific property, others might not have it. You don't have to specify upfront that a property might be there and then set it to a NULL value if it is not. So what you have is a flexible schema (or better data model), whereby the application is implicitly providing it. If your application/service is managing user profiles then you will find user documents whereby a user has a first name, last name ... and so on. So 'flexible data model' would be the better term.

Key Patterns

Best practice is to use meaningful key patterns. This helps you directly access a document based on it's context. A key could be a combination of a type and a unique attribute value or it can be also an artificial number. If possible, don’t use artificial numbers. This is indeed not every time possible. The following example shows the key of a user with the email address “”:

Pattern is: $type::$email

If you know that users are accessible via their email address then you can directly get the user without the need to perform a more complex query. (Whereby querying is e.g. supported via SQL like query language - N1QL - in Couchbase). 

A more interesting pattern would reflect a hierarchy. If you would assume that one employee belongs only to one company (but a company can have multiple employees) then you can reflect this ‘belongs to’ already in the key. The following shows an example of the key of a user who belongs to the company ‘Foo’ which has the domain ‘’.

Pattern is: $type::$domain::$id


We have already seen that the key pattern often has a type prefix. It is also best practice to store an extra type attribute. This allows you later to filter more specifically based on this type (e.g. to ask for all users). Here an example of a user:

“” : {
  “type” : “user”
  “first_name” : “Max”,
  “last_name” : “Mustermann”,
  “email” : “”

1:1 Relationships

In this case one entity X has a relationship to another one and vice versa. A one to one relationship can be modeled by embedding or by referencing documents. My recommendation would be to model such a relationship in the first step as a key reference and then embed if there are atomicity requirements, which means if there is a requirement to access the two entities most of the times together. The following shows an example of an user who has a session.

“” : {
  “type” : “user”,
  “first_name” : “Max”,
  “last_name” : “Mustermann”,
  “email” : “”,
  “session” : {
    “source” : “web”,
    “token” : “123456”
Embedded Document

The same example by expressing it now as a key reference:

“” : {
  “type” : “user”,
  “first_name” : “Max”,
  “last_name” : “Mustermann”,
  “email” : “”,
  “session” : “”

“” : {
    “type” : “session”,
    “source” : “web”,
    “token” : “123456”,
    “user” : “”
Explicitly Referenced Document

It’s easy to see that there is a direct relationship via the id of the user, which is the email address in this case. Because the two documents are anyway correlated via their keys we can in this case simplify it to:

“” : {
  “type” : “user”,
  “first_name” : “Max”,
  “last_name” : “Mustermann”,
  “email” : “”

“” : {
    “type” : “session”,
    “source” : “web”,
    “token” : “123456”
Documents those are implicitly referencing each other

When to embed or to reference is not like black or white. It's more transitioning with some grey values in between. Dependent on the access patterns it is indeed also possible to embed a part of the document and reference to another part.

1:Many Relationship

The one to many relationship means that one document references multiple other ones (1 up to n). A back reference from the referenced one might be suitable. Again, I would by default reference and would embed as an optimization step if there are any atomicity requirements. This is only my personal preference, you could also start with embedding documents and then externalize by optimize regarding cardinalities and data duplication. Here a company which has multiple employees:

“” : {
  “type” : “company”,
  “name” : “Some name”,
  “address” : “Some address”,
  “employees” : [ “”, “”]

“” : {
  “type” : “user”,
  “first_name” : “Bart”,
  “last_name” : “Simpson”,
  “email” : “”,
  “company” : “”

“” : {
  “type” : “user”,
  “first_name” : “Moe”,
  “email” : “”,
  “company” : “”
1-to-many via key references

Let’s now assume that we embed the users as subdocuments. Another alternative would be to embed them in an array. An array is better if you need only to iterate over the list of embedded documents. If you need to access specific sub-documents  by their id then embedding as nested documents would be preferred. It's identical to the question when you, as a developer, use a List or a Map in order to reflect the associations between your classes.

“” : {
  “type” : “company”,
  “name” : “Some name”,
  “address” : “Some address”,
  “employees” : {
        “” : {
            “type” : “user”,
            “first_name” : “Bart”,
            “last_name” : “Simpson”,
            “email” : “
        “” : {
            “type” : “user”,
            “first_name” : “Moe”,
            “email” : “
1-to-many as embedded document

What happens now if one employee works for multiple companies in the embedded case? Right, you get data duplicates because one user needs now to be fully embedded into 2 company documents. 

At the end it is a question of normalization. A completely normalized schema would contain a lot of relations whereby a de-normalized schema would in the worst case have everything in one table. As in relational databases, the truth is something in the middle. You would not embed everything into one document and you would normally also not express every property as an extra document and then use key references to glue them together. What works best depends on the actual requirements and how you need to access the data.

So you reference in order to avoid duplicates but you embed in order to have atomic access. Your documents are usually in average smaller if you reference but you then might have to perform client side joins (or server side ones via N1QL since Couchbase 4.0). 

The Many-Many relationship (via references) is quite similar to the 1-Many one. It just means that you have reference arrays (arrays of keys to express the relationships) on both sides.

Lookup Documents

A lookup document is a document which has only the purpose to provide you a direct reference to one or multiple other documents. Lookup documents are quite useful to maintain own indexes (alternative access paths) in Couchbase’s cache. Let’s assume you want to access a user profile via a customer number:

“customer_ref::12345” : {
  “ref” : “”

“” : {
  “type” : “user”,
  “first_name” : “Max”,
  “last_name” : “Mustermann”,
  “email” : “”

In order to get an user by his customer id, you can now perform 2 get operations. First you get the lookup document based on the customer id, then you read the ‘ref’ attribute which gives you the key of the associated user document. In the next step you can then access the user directly. This way of access is often more efficient than an exact match query because the index lookup is in this case independent from the number of documents in the bucket or entries in the index which is scanned as part of the query processing.

Atomic Counters

Couchbase allows you to increment counter values. This is a useful feature which helps you for instance to generate Id-s. So it’s similar to sequences in the relational world. The following shows some pseudo-code how to increment the counter value and then reuse it for the id generation.

//“count::user” : “0”
id = client.incr(“count::user”);
client.add(“user::” + id, doc);

A typical pattern would be to perform a multi-get based on a range (e.g. 0...count::user) by taking the counter value into account. You could then skip every non existent document by ignoring the ‘DocNotFound’ error messages. This is indeed prefered if you have evolving data with only a low amount of deletes.

We saw in the section ‘Key patterns’ that keys can reflect hierarchies. So you could easily reflect a 1:Many relationship this way by not using explicit references. A user document belongs to a company document if the corresponding key contains the company prefix. We can get all users of a company by knowing the number of users of the company. Here some pseudo code:

count = client.get(“”); //e.g. “563”

for ( i=0; i < count; i++ ) {
    doc = client.get(“” + i);

    if (! doc.err ) {


Basic document modeling techniques were presented here. Couchbase allows you a flexible data model. As mentioned, the way how to model your data is not always black or white. My personal preference is to:
  •  Start with the logical data model (e.g. derived from Object Oriented Analysis)
  •  Create a stupid and simple initial model (e.g. by using key references for 1:Many relationships all the time)
  •  Evolve and optimize it step by step regarding the requirements (unnecessary reference lists because the reference is clear via the key pattern, access patterns, atomicity, duplicates, ...). 
Here some useful rules/thoughts:
  • Use meaningful keys and speaking key patterns if possible!
  • Use counters for the key generation if there is a need to use artificial ids!
  • Maintain a type attribute!
  • Embed documents into others in order to allow to write/get them all together with the parent document. (Atomic access)
  • If not embedding and if possible (e.g. by using the counter value as part of the key, or by having correlated keys) then express the relationship via the key directly without having the need to reference via key arrays.
  • Reference in order to avoid data duplicates and in order to keep the average document size smaller. Often we just live with duplicates by having other advantages (atomic access, no client side joins). But on the other hand's side we might have such a high amount of data and such a high degree of connectivity that we can't duplicate all the time.
  • If the cardinalities (number of related documents) are too high and there is no requirement for atomicity then referencing would be preferred over embedding.
  • Externalize groups of properties from a document (by adding a 1:1 relationship) if you access this group of properties all the the time together and if the overhead of transferring all the other properties of the document the same time would be too high.
  • It might make sense to externalize reference arrays from a document if the number of references is very high and so you would like to avoid the overhead of transferring these arrays if you usually only access a few properties of the document. 

Monday, 2 November 2015

Using a Key-Value Store for Full Text Indexing and Search

Couchbase Server is a multi-purpose Database System. One of the purposes is to use it as a simple key-value store. A key-value store allows you to store/retrieve any value by its key. Such a value can be a JSON document (Couchbase allows you to index and query based on such JSON documents and so another purpose is the one as document database.), a small binary or a full text index entry. This article explains why such a key-value store can be also used for full text indexing purposes.

Let's explain how full text indexing works in general. A full text index is a so called inverted index. The table below shows how the following sentences would be indexed: 'Tim is sitting next to Bob' and 'Jim is sitting next to Bob'. The word 'Tim' is only existing in the first sentence and there is exactly one occurrence of it.

Term | Count | Reference
Tim | 1 | #1
is | 2 | #1, #2
sitting | 2 | #1, #2
next | 2 | #1, #2
to | 2 | #1, #2
Bob | 2 | #1, #2
Jim | 1 | #2

There are a bunch of specialized systems out there for full text indexing. Couchbase has for instance a very good integration with Elasticsearch. In the future Couchbase will also have it's own full text service which is called 'cbft'. However, this article is not about Elasticsearch and also not about 'cbft'. We want to use Couchbase's key-value store features for full text indexing here.

Let's define the data model first:

"fts::$field::$term" : { "count" : $numOf, "refs" : [ ...] }

It is actually quite simple. A full text search index entry does point back for a term to the original key-value pairs those are identified by their keys. The refs array contains these keys. The field is just the field on which we want to search. This could be for instance 'address' or 'message'. Let's say that the default field is called '_all'. So if no specific field is used then the '_all' field is used as the fallback field. 

In order to index based on a provided text, we can do the following:
  • Tokenize the text on which should be indexed. This means basically to break the text up into several words (terms). In our case we assume that our text contains the word 'fox'.
  • Check if the e.g. the key "fts::_all::fox" is already existing. If not then create the document by referencing back to the document id which contained the word 'fox'.
  • If the full text index entry was existent then check if the reference list does already contain the reference to the document which contains the text on which should be indexed.
  • If the reference list does not yet contain the key of this document then extend the reference list by adding the key of the document.
Now in order to search for the specific word 'fox', we just have to do the following:
  • Get "fts::_all::fox"
  • Perform a multi-get based on the keys in the array 'refs'
Some example source code (Node.js) can be found here: . The service is implemented here: . This application was created by using CEAN stack tools (Couchbase + Express + Angular + Node.js ). They are available here: .

Given that I already wrote this little demo application, it makes sense to try it out :-) . First let's add 2 sentences:
  • the_fox: The quick brown fox jumps over the lazy dog
  • the_cat: The quick brown fox jumps over the lazy cat

Now in the next step let's perform some searches. I implemented the word search in a way that you can enter any sequence of words (separated by white spaces). The following searches for 'cat':

As we can see, only the sentence with the id 'the_cat' contains the word 'cat'. Next lets's search for the word 'fox':

Both sentences contain the word 'fox'. Last but not least let's search for multiple words:

I think you get it ... :-) . The data which is stored in Couchbase looks as the following one:

This article explained how you can use Couchbase to store a full text search index. Such an index can be used for simple and basic text searches, which might be sufficient for some of your development projects. If you need more sophisticated text search or text analysis then a dedicated full text search service might be the better option.

Friday, 16 October 2015

Understanding Couchbase's Document Expiration

I just had to investigate Couchbase's T(ime)T(o)L(ive) feature a bit because I was asked how to react externally on document expirations via DCP.

The TTL feature allows you to create documents by specifying when the document should be automatically deleted. Here two example use cases:

  • Caching: The items in a cache should be only valid for a specific period of time. The TTL feature allows you to invalidate items in the cache automatically. Let's assume that you are caching some product details (as a retailer). Then the prices of some products might change daily (or even more often). It now would make sense to make sure that the product price gets automatically updated once a day. So if the item is expired in cache then it should be fetched again from the original source system. The newly fetched item then has the updated price information.
  • Session management: Couchbase is often used as a session cache/store. A session is just realized as a key value pair whereby the key is dependent on the id of the logged in user and the session value could be just a random hash value (e.g. an UUID). Let's expect that a session times out after 20 mins if the user is inactive. Which means that you just can create such a session document by using a TTL value of 20 mins. If the user shows some activity then you can touch the session document in order to extend the TTL.
So far we saw two examples for what the feature can be used. Let's now talk a bit about how it can be used. Let's create 1000 documents with an expiration value of 60 seconds:

       List<JsonDocument> docs = new ArrayList<>();

        for (int i = 0; i < 1000; i++) {

              docs.add(JsonDocument.create("key::" + i, 60, JsonObject.fromJson("{ \"content\" : \"This is the document #" + i  +"\"}")));    

        Observable.from(docs).flatMap(d -> bucket.upsert(d)).subscribe(

                res -> LOG.log(Level.INFO, "Created document with id {0}",

Instead using the relative number of seconds (since the creation) you could also use a Unix time stamp. For example, the value 1421454149 represents Saturday, 17 January 2015 at 00:22:29 UTC. The maximum value you can specify in seconds is the number of seconds in a month. Internally it is stored as a Unix time stamp.

So far so good. Let's now understand how this expiration works internally. There are actually multiple ways how to cause the actual removal of an expired document.
  • Lazily: If you try to access a document which is expired but not yet deleted then you would not get the document returned. Instead the document deletion would be performed. It's important to understand that this is only true for the document itself. If you access a Couchbase View then this would not cause a deletion. If you would retrieve the actual documents based on the View result (e.g. by performing an additional multi-get or by using the include_docs option) then this would cause it.
  • The expiry pager: This is a job which runs by default on every node in the cluster. You can set the interval by using the following command which sets it here to 60 seconds (for testing only) for a bucket with the name 'dcp'. Please be aware of that this setting is not persisted and so would not survive a node Couchbase restart: 
./cbepctl localhost:11210 set flush_param exp_pager_stime 60 -b dcp -p test
    • Compaction: Couchbase uses an append-only storage engine (Couchstore). So instead performing in-place updates, the updated data is appended to the end of the database files. The storage structure is a so called 'Append-only Btree'. It's easy to see that this leads to some fragmentation over the time. So compaction means to rewrite the database files (vBucket files) by skipping tombstone entries (e.g. old revisions, deleted documents or expired documents). Compaction is by default configured to run automatically, but you can also invoke it manually (for testing purposes):
    ./couchbase-cli bucket-compact -c localhost:8091 --bucket=dcp -u couchbase -p couchbase

    I wrote a little test program (for testing purposes only) which allows you to see which documents are expiring by consuming the D(atabase)C(hange)P(rotocol) stream of a Couchbase bucket. It also helps to show that each of the above mentioned expiration methods are causing DCP RemoveMessages. Regarding DCP, there is (as far as I can see) no way to distinguish an expiration event from another deletion one. Both are causing a RemoveMessage.

    • ExpirationHandler:
    package com.couchbase.example.dcp.handler;
    import com.couchbase.client.core.message.dcp.DCPRequest;
    import com.couchbase.client.core.message.dcp.RemoveMessage;
    import java.util.logging.Level;
    import java.util.logging.Logger;
     * A logging DCP handler for remove messages
     * @author David Maier <david.maier at>
    public class ExpirationHandler implements IHandler {
        private static final Logger LOG = Logger.getLogger(ExpirationHandler.class.getName());
        private int count = 0;
        public void handle(DCPRequest dcp) {
            if (dcp instanceof RemoveMessage) {
                RemoveMessage msg = (RemoveMessage) dcp;
                LOG.log(Level.INFO, "Removed document with bucket/vBucket/key: {0}/{1}/{2}", new Object[]{msg.bucket(), msg.partition(), msg.key()});
                LOG.log(Level.INFO, "So far {0} documents were deleted.", this.count);
    • ExpirationTest:
    package com.couchbase.example.dcp;
    import org.junit.BeforeClass;
    import org.junit.Test;
    import static com.couchbase.example.dcp.TestConstants.*;
    import com.couchbase.example.dcp.handler.ExpirationHandler;
    import java.util.ArrayList;
    import java.util.List;
    import java.util.logging.Level;
    import java.util.logging.Logger;
    import rx.Observable;
     * @author David Maier <david.maier at>
    public class ExpirationTest {
        private static final Logger LOG = Logger.getLogger(ExpirationTest.class.getName());
        private static Receiver r;
        private static AsyncBucket bucket;
        public ExpirationTest() {
            //Make sure that DCP is usable
            System.setProperty("com.couchbase.dcpEnabled", "true");
            bucket = CouchbaseCluster.create(new String[]{HOST}).openBucket(BUCKET, PWD).async();
            r = new Receiver(new String[]{HOST}, BUCKET, PWD, new ExpirationHandler());   
        public static void setUpClass() {
         * Create 1000 docs with an expiration time and then 
         * consume the DCP stream by handling remove messages
         * (1) Create an empty bucket
         * (2) Run this test
         * (2a) Wait until the expiry pager did run
         * (2b) After the expiration period access all keys
         * (2c) Run compaction
         * (3) You should see the line 'INFORMATION: So far 1.000 documents were deleted.'
         * in the output
        public void testReceiveDeletionStream() throws Exception {
  "-- testReceiveStream");
  "Creating some documents with expiration times ...");
            List<JsonDocument> docs = new ArrayList<>();
            for (int i = 0; i < 1000; i++) {
                  docs.add(JsonDocument.create("key::" + i, 60, JsonObject.fromJson("{ \"content\" : \"This is the document #" + i  +"\"}")));    
            Observable.from(docs).flatMap(d -> bucket.upsert(d)).subscribe(
                    res -> LOG.log(Level.INFO, "Created document with id {0}",
            //Wait a moment before streaming
  "Connecting ...");
  "Streaming ...");

    The full source code can be found here:

    Important to mention is that my test program uses Couchbase's Java core library. Couchbase does not support the usage of this library officially. Instead you should use one of the many Couchbase Connectors (Spark, Kafka, ...), which uses the core library behind the scenes, if you want to externally consume DCP messages.

    Monday, 28 September 2015

    Document Versioning in Couchbase

    Couchbase Server does out of the box not support document revisions but it would be quite simple to implement it on the application side. This article describes ways how to do this. The following topics are covered:

    • Handling concurrent access
    • Relevant attributes
    • One document per version
    • Embedded revision tree
    • Combined approaches 

    Handling concurrent access

    In a context of versioning multiple users/threads are creating new versions (and this maybe nearly the same time). So I think it makes sense to spot a light on concurrent access before we talk about versioning approaches. You will most probably need to combine concurrency handling with versioning.  Couchbase supports 2 ways of handling concurrent access to the same document. 
    • C(ompare) A(nd) S(wap): This is the optimistic approach. Each document has a built-in property which is the CAS-value. The CAS-value changes as soon as somebody updates the document. So the idea is to implement something like the following on the application side:
    1. Get the document and especially the CAS-value!
    2. Modify some document properties!
    3. Perform an update operation by passing the old CAS-value (from step 1.)!
    4. If somebody else updated the document meanwhile then a CAS mismatch error occurs because your client side CAS value is no longer identical to the server side one. If so then wait for a very short moment and then try again from step 1.!
    5. Multiple users/threads are accessing the same document. You will reach step 5. because you have the same chance as all others and so you will have the chance to update the document before someone else is doing it.
    • Locking: You can lock a document before you perform the changes and then release the lock. This is the pessimistic approach. A lock wait implementation is require in this case:
    1. Get the document and request a lock.
    2. If the document is locked then a lock error occurs!
    3. Wait until the document is released again and try from 1. !
    4. Update the document!
    5. Release the the document!

    Relevant Attributes

    Couchbase has some built-in attributes but you might want (dependent on your requirements) like to introduce some own versioning attributes.

    Here 2 relevant built-in attributes:
    • Revision number: Couchbase has the built-in attribute 'rev' which is accessible via the document's meta data (meta.rev). The revision number is increased for every update and is internally used for the conflict resolution is you use Couchbase's Cross Data Center Replication feature. A higher revision number means that a document was more often updated.
    • CAS value: This attribute was already explained in the previous chapter. It is used to determine if a document was changed since you accessed it the last time.
    Own attributes could be:
    • Update time stamp: A version could contain the update time stamp in order to determine who updated it last. You have to be careful here because your clients may not be time synchronized.
    • Custom revision number: Even if there is a built-in one, you can also introduce just a incrementing number as your revision number.
    • Updater: Person/service who/which updated the document.
    • Revision identifier: Another option would be to use an artificial id as the version number. So something like a UUID would be suitable.
    • Parent version: The previous revision.
    So the idea would be that you just embed the suitable versioning details into your document:

    One Document per Version

    The easiest implementation of versioning would just use the version as part of the key. So let's assume that we have a kind of Content Management use case. To avoid confusion regarding the terminology let's call the objects 'content items' (The term document would be overloaded in this case because we talk about JSON document in Couchbase but not about Word documents in ECMS-s.). So the key of a content item would be:
    • cnti::1abc-2def-3ghi-4jkl
    This follows the key pattern '$prefix::$id'. 

    What we need now is an additional atomic counter object to generate our revision numbers. Couchbase supports such counters. They can usually be incremented by using the 'incr' function of the SDK of your choice.
    • count::rev::cnti = 0
    The counter value is now used as part of the key of a content item:
    • cnti::1abc-2def-3ghi-4jkl::7
    This follows the key pattern '$prefix::$id::$rev'.

    It's easy to see that the counter also acts as a pointer to the latest revision. So the simplified approach would be:
    1. Increment the counter by generating a new revision id
    2. Get the old document which has the revision 'rev-1'
    3. Create a new document with the new revision id
    Because we create new revisions for every document there is no concurrent write access to the document itself but there is indeed concurrency regarding the counter.

    Embedded Revision Tree

    A more complex approach would be to embed the versions to the main document as a tree of changes. The disadvantage could be that you document size becomes quite big. So you should limit the number of revisions to embed. Couchbase's Sync Gateway (a synchronization endpoint for Couchbase Lite instances, whereby Couchbase Lite is a light weighted Couchbase which can run on your mobile device - Rev Tree Storage on Couchbase Server) uses this approach.

    The tree definition is quite simple. A tree has nodes. Each node, except the root node, has exactly one parent node. Each node in such a tree is representing one document revision. The tree describes which revision was derived from which other revision. The sub-tree from a specific node in the tree down to the leafs is called a branch.

    The picture above shows 6 revisions. Now your application has a lot of possibilities to use such a revision tree.

    • From which revision to fork?
    • Which revisions/branches to keep?
    • How to merge based on the revisions?
    • What should be the max. size of the revision tree?
    So as you can see this versioning approach is quite more complicated but provides you a lot of freedom.

    The idea is to have a head reference in the document which points to the current base revision.

    Combined Approaches

    I can see the following 2 main requirements for versioning:
    • Change History: Some compliance or security rules are enforcing that you have to be able to answer the question who changed what and when. For this approach the 'One Document per Version' approach would be sufficient.
    • Conflict Handling: Multiple users are creating several versions and you want to decide to be able to pick a winner or even merge several versions. For this the 'Revision Tree' approach would work best.
    Let's assume that you store only trees of a specific depth. The revision tree needs to be truncated in order to realize this but such a truncation would mean that you loose some older revisions. So an idea would be to archive the state of the tree as another revision:
    • If the revision tree becomes to big then
    1. Archive the current state of the revision tree by creating a new document for this version! An extra 'archive' bucket can be used for this purpose.
    2. Truncate the tree by setting a new head revision!


    Even if each document has a 'rev' attribute, Couchbase Server is not directly supporting document versioning. But you can use the described approaches to implement your own document versioning on the application side. Such an approach can be very simple or more complicated. Which approach should be used by you depends on your actual requirements.

    Friday, 25 September 2015

    How many Buckets?

    In Couchbase the equivalent of a database is called a bucket. A bucket is basically a data container which is split into 1024 partitions, the so called vBuckets. Each partition is stored as a Couchstore file on disk and the partitioning is also reflected in memory. Each configured replica leads to additional 1024 replica vBuckets. So if you have a bucket with 1 replica configured then this bucket has 1024 active vBuckets and 1024 replica vBuckets. The vBuckets are spread across the cluster nodes. So in a 2 node cluster each node would have 512 active vBuckets. In a n-node cluster each node would by default have nearly 1024/n vBuckets.

    An often asked question is 'How many Buckets should I create?'.

    There are multiple aspects to take into account here.
    • Max. possible number of Buckets
    • Logical data separation
    • Load separation
    • Physical data separation
    • Multi-tenancy

    Max. possible number of Buckets

    Before we start talking about why it might make sense to use multiple buckets, let's talk about how many buckets are possible in practice per cluster.

    As we will see a bucket is not just a logical container but also a physical one. Furthermore some cluster management is happening per bucket. So it would not be a good idea to create hundreds of buckets because it would lead to a management overhead.  The actual max. number is dependent on size, workload and so on. Couchbase is usually recommending a max. number of 10 buckets.

    Logical Data Separation

    Let's assume that you have multiple services. Regarding a service oriented approach (and especially micro services) those service would be decoupled by having disjunct responsibilities. An example would be:
    • Session Management Service: Create/validate session tokens for authorization purposes
    • Chat Service: Send/retrieve messages from/to users 
    The session management service is then indeed used by the chat service but both services have disjunct responsibilities.

    A logical data separation would mean that we create 2 buckets in order to make sure that there is a logical separation between sessions and messages. So you can co-locate some service by taking the scalability requirements into account. If these requirements are completely different then it would make sense to open a new cluster.

    Load Separation

    It's indeed the case that the workload pattern of the session management service is different from the chat service. Millions of users sending/reading messages would cause a lot of writes and reads whereby the token based authorization would cause mostly reads and a fewer number of writes for the log-ins.

    So it would be useful to create the session management bucket with other settings than the messaging one. For a session management use case it makes sense to have all of the sessions cached in order to reply quite quick to the question if a user is authorized to act in the context of a specific session.

    So the statement here is that it makes sense to have multiple buckets (or even multiple clusters) to support the individual load patterns better. Buckets with a high read throughput profit from higher memory quota settings. 

    Buckets with higher disk write throughput benefit from a higher disk I/O thread priority.

    Another aspect is load separation for indexing purposes. In 3.x this refers to the View indexing load. Couchbase Views are defined per bucket (one bucket can have multiple Design Documents, one Design Document contains multiple Views)  and by using map-reduce functions. Views are sharded across the cluster. So each node is responsible for indexing its data. They can have multiple roles:
    • Primary Index: Only emit a key of a KV-pair as the key into the View index
    • Secondary Index: Emit a property of the value of a KV-pair as the key into the View index
    • Materialized View: Emit a key and a value into the View index
    Views are using incremental Map-Reduce to be updated. This means that a creation/update/deletion of a document will be reflected in the View index. Let's assume that you have 20% of user documents in your bucket.  Your Views are defined in a way that they only work with user documents and so your map function is filtering them out. In this case the map function would be anyway executed for all documents by skipping 80% of them after checking if they have a type attribute which has the value 'User'. This is means some overhead. So for Views it would be a good idea to move these 20% of data into another bucket called 'users'. The Views are then only created for the 'users' bucket.

    In Couchbase Server 4.0, G(lobal) S(econdary) I(ndexes) are helping. As the name says a GSI is global and so it sits on a single server (or on multiple nodes for failover purposes). They are not sharded across multiple nodes. So the indexing node gets all the mutations streamed and then has to index also based on filter criteria, but the indexing load is separated from the data serving node and so does not affect it.

    CREATE INDEX def_name_type ON travel-sample(`name`) WHERE (`_type` = "User")
    So you win control regarding the indexing load separation by deciding which node serves which index.

    Physical Data Separation

    The previously mentioned load separation is closely related to a physical separation. 

    To store several GSI-s on several machines means to separate them physically. So Couchbase Server 4.0 and it's multi-dimensional scalability feature, whereby the dimensions are
    • Data Serving
    • Indexing
    • Querying
    allows you to physically scale-up per individual dimension. A node wich runs the query service can now have more CPU-s whereby a node which runs the data service can now have more RAM in order to support lighting fast reads.

    A physical data separation is also possible per bucket. Couchbase allows you to specify separated data and index directories. So it is already possible to keep the GSI-s or Views on faster SSD-s whereby (indeed dependent on performance requirements) your data can be stored on bigger slower disks. Each bucket is reflected as one folder on disk. So it's also possible to mount a specific bucket folders to a specific disks, which provides you a physical per bucket data separation.


    The multiple buckets question is also often asked in a context of multi-tenancy. So what if my service/application has to support multiple tenants? Should then every tenant have his own bucket?

    Let's answer this question by asking another one. Let's assume you still use a relational Database System. Would you create one database for every tenant? The answer is most often 'No' because the service/application is modeling the tenancy. So it would be more likely the case that there is a tenants table which is referenced by a user table (multiple users belong to one tenant). I saw only a very few Enterprise Applications in the wild which relied on features (like database side access control) of the underlying Database System for this purpose. 

    In Couchbase you have in theory the following ways to realize multi-tenancy:
    • One tenant per cluster: This would be suitable if you provide Couchbase as a Service. Then your customers have full access to Couchbase and they implement their own services/applications on top of it. In this case a cluster per tenant makes sense because the cluster needs to be scaled out dependent on the performance requirements of your customer's services/applications.
    • One tenant per bucket: As explained, there are two reasons why it is often not an option. The first one is the answer to the question above. The second one is the max. number of buckets per cluster.
    • One tenant per name space: As explained your application often already models multi-tenancy. So you can model a kind of name space per tenant by using key prefixes. ( Each document is stored as a KV-Pair in Couchbase. ) So every key which belongs to a specific tenant id belongs to a specific tenant. If an user with an email address of '' would log in, then he would only have access to documents those have the prefix ''. The key pattern for a message would be '$tenant::$type::$id', so for instance ''.


    Multiple buckets are making sense for Logical Data separation in a context of service oriented approaches. If the max. number of buckets per cluster would be exceeded or the scalability requirements of the buckets are too different then an additional cluster makes sense. An important reason for multiple buckets in 3.x is the load or physical data separation. 4.0 helps you to externalize the indexing load to other nodes which gives you more control regarding the balancing of the indexing load. Multi-tenancy should happen normally via name spaces (key prefixes) and not via multiple buckets.

    Thursday, 25 June 2015

    Using the Rexter 2.7 Graph Server with Couchbase Server

    One of my side projects at Couchbase is a Graph-API on top of it. This article explains how you can use the Rexter Graph Server with this implementation of the Blueprints 2.x API.

    There is a forked version of Rexter available here: .

    However, the modifications are quite simple.

    I disabled the enforcer plug-in in the main pom.xml file.

    The 'rexter-server/pom.xml' file contains additional dependencies.


    Also the file 'rexter-server/config/rexter.xml' was edited in order to use the Couchbase instance.


    To have this working it is expected that you have previously built CBGraph and that it sits in your local Maven cache. It can be found here:

    Your Couchbase connection is currently configured via the file '' which is part of the source code of CBGraph.

    ## Connection to Couchbase
    ## Views on side of Couchbase

    So the steps are:

    • Run 'mvn install' in the CBGraph project
    • Run 'mvn install' in the Rexter project

    Rexter's target folder now should contain a folder which is called 'rexster-server-2.7.0-SNAPSHOT-standalone'.
    • Execute 'bin/ -s'
    You should see a message like

    [INFO] Slf4JLogger - Connected to Node
    [INFO] Slf4JLogger - Opened bucket graph
    [INFO] Slf4JLogger - Connected to Node
    [INFO] Slf4JLogger - Connected to Node

    Now let's play a bit with the Rexter REST interface:
    You can also browse the Graph via Rexter's UI:  http://localhost:8182/doghouse 

    Here an example screen shot: