tag:blogger.com,1999:blog-27796106219283608242024-03-13T17:05:43.923+01:00nosqlgeek.orgNews by NoSQL GeeksDavid Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.comBlogger40125tag:blogger.com,1999:blog-2779610621928360824.post-17948116389060051112023-05-19T17:00:00.010+02:002023-05-24T11:13:44.979+02:00 How to use Redis as a Vector Database for Recommendations<h2 style="text-align: left;"><span style="white-space: pre-wrap;">Introduction</span></h2><div><span style="white-space: pre-wrap;">This blog post is the result of some preparation work for a recent meetup, where I introduced a bunch of recommendation engine algorithms. The idea of using vector similarity search for recommendations is quite simple:</span></div><div><span style="white-space: pre-wrap;"><br /></span></div><div><ol style="text-align: left;"><li>The interests of a user are expressed as a vector. Each component of the vector is associated with one category of interest.</li><li>If we know the interests of a specific user, we can search for the <b>K</b>-<b>N</b>(earest)<b>N</b>(eighbours) to find other users that share the same interests.</li><li>We can then inspect the behavior of these users (e.g., the purchase history) to recommend our user specific products.</li></ol>The following table shows the vector <span style="background-color: #f3f3f3;">[0.9, 0.7, 0.2]</span>:</div><div><br /></div><div>
<table border="1" cellpadding="10" cellspacing="1" style="background-color: #dfdfdf; border-collapse: collapse; border: 1px solid rgb(255, 255, 255); color: black; width: 100%;">
<tbody><tr>
<td><b style="text-align: center;"><i>books</i></b></td>
<td><b style="text-align: center;"><i>comics</i></b></td>
<td><b style="text-align: center;"><i>computers</i></b></td>
</tr>
<tr>
<td style="text-align: left;">0.9</td>
<td style="text-align: left;">0.7</td>
<td style="text-align: left;">0.2</td>
</tr>
</tbody></table>
</div><div><br /></div><div>Let's assume that a user is only interested in a specific category if the interest value is larger than the threshold of <span style="background-color: #f3f3f3;">0.4</span>, which means that our user is interested in 'books' and 'comics' but not in 'computers'.</div><div><br /></div><div><h2 style="text-align: left;">How to use Redis</h2>You can use Redis Stack's query and search capabilities to:<br /><ul style="text-align: left;"><li>Index vectors</li><li>Find similar vectors</li></ul>My source code example wraps the Redis commands by using a <span style="background-color: #f3f3f3;">VectorDB</span> class. Let's look at some of the methods I implemented for accessing Redis:</div><div><h3 style="text-align: left;"><a href="https://github.com/nosqlgeek/vss-rec/blob/01b310c67d2cdfb8d901b7304ba9e8d68b28bb8a/vss-rec/vector_db.py#L19" target="_blank">init</a></h3>The Redis connection is established within the constructor of the <span style="background-color: #f3f3f3;">VectorDB</span> class:</div><div><br /></div><div><script src="https://gist.github.com/nosqlgeek/73edc0e52dcbf24d5d244e24c0b9d485.js"></script><br /></div><div><h3 style="text-align: left;"><a href="https://github.com/nosqlgeek/vss-rec/blob/01b310c67d2cdfb8d901b7304ba9e8d68b28bb8a/vss-rec/vector_db.py#L44" target="_blank">create_index</a></h3>As the name indicates, this method creates a search index. The default schema has a <span style="background-color: #f3f3f3;">descr</span> text field, a <span style="background-color: #f3f3f3;">labels</span> tag field, a numeric field called <span style="background-color: #f3f3f3;">time</span>, and a vector field named <span style="background-color: #f3f3f3;">vec</span>. The relevant code within this method is equivalent to the following:<br /><br /></div><div><script src="https://gist.github.com/nosqlgeek/d85dd80f56060e0eb679519ea06ee6d2.js"></script><br /></div><h3 style="text-align: left;"><a href="https://github.com/nosqlgeek/vss-rec/blob/01b310c67d2cdfb8d901b7304ba9e8d68b28bb8a/vss-rec/vector_db.py#L83" target="_blank">add</a></h3><div>This method adds a vector with metadata to the database. I use a <a href="https://redis.io/docs/data-types/hashes/">Redis hash</a> in this case, but you can also store vectors within JSON with Redis Stack.</div><div><br /></div><div><script src="https://gist.github.com/nosqlgeek/385e0b090eb127c55ca8f1cfbb1862e0.js"></script><br /></div><div><br /></div><div>The data dictionary contains the fields <span style="background-color: #f3f3f3;">labels</span>, <span style="background-color: #f3f3f3;">descr</span>, <span style="background-color: #f3f3f3;">time</span>, and <span style="background-color: #f3f3f3;">vec</span>. The vector is stored as binary within the <span style="background-color: #f3f3f3;">vec</span> field. The library <span style="background-color: #f3f3f3;">numpy</span> is used to convert a more human-readable float vector (a Python list) to its byte string representation:</div><div><br /></div><div><script src="https://gist.github.com/nosqlgeek/f4d98526392a1e9eb1f6914e0de7cf01.js"></script><br /></div><div><br />Here is an example of such a data dictionary:<br /><br /></div><div><script src="https://gist.github.com/nosqlgeek/bd0f8657e2e4e10bc9b23fb05f2d5a6a.js"></script><br /></div><h3 style="text-align: left;"><a href="https://github.com/nosqlgeek/vss-rec/blob/01b310c67d2cdfb8d901b7304ba9e8d68b28bb8a/vss-rec/vector_db.py#LL108C9-L108C22" target="_blank">vector_search</a></h3><div>The <span style="background-color: #f3f3f3;">vector_search</span> method performs the vector similarity search. My implementation only returns the id and vector score. The query string has a few arguments:<br /><ul style="text-align: left;"><li> <b>Metadata query</b>: The variable <span style="background-color: #f3f3f3;">meta_data_query</span> is set to the query string that is executed to pre-filter based on the metadata, such as the description (<span style="background-color: #f3f3f3;">desc</span>) or the <span style="background-color: #f3f3f3;">labels</span>. The <span style="background-color: #f3f3f3;">=></span> operator means <span style="background-color: #f3f3f3;">execute before => execute after</span>. So, the metadata query is executed before the vector similarity search is performed.</li><li><b>Number of neighbours</b>: The value of <span style="background-color: #f3f3f3;">num_neighbours</span> is set to the KNN integer value.</li><li><b>Vector field</b>: This is the vector field that is used for the search. Redis can store multiple vector fields within an item (hash or JSON).</li></ul>Here is the source code that constructs the vector query string:</div><div><br /></div><div><script src="https://gist.github.com/nosqlgeek/5481bec91badc7f3d0d3659ec69d8bf5.js"></script><br /><br /></div><div>You can then query the database the following way:<br /><br /></div><div><script src="https://gist.github.com/nosqlgeek/e607f031de9cda19772fb95f1da0c4fb.js"></script><br /></div><div><br />For further details, please look at the <a href="https://redis.io/docs/stack/search/reference/vectors/">vector similarity search reference documentation</a>.<br /><br /></div><div><h2 style="text-align: left;">Putting it all together</h2>As explained, I decided to add a thin layer of abstraction by implementing <a href="https://github.com/nosqlgeek/vss-rec/blob/main/vss-rec/vector_db.py">this VectorDB</a> class. The following example shows how to use it:</div><div><ol style="text-align: left;"><li>Create an index</li><li>Add some vectors with metadata</li><li>Perform a simple query for users that are labeled with specific interests</li><li>Execute a vector similarity search for the two nearest neighbours</li></ol><br /></div><div>Here is the source code of the demo application:</div><div><br /></div><div><script src="https://gist.github.com/nosqlgeek/6551be7cc001ccf7da4e22727c847539.js"></script><br /><br /></div><div>The output of this program is:<br /><br /></div><div><script src="https://gist.github.com/nosqlgeek/025ae7760ea825acd5a890a9a248dde2.js"></script><br /></div><div><br /></div><div>It's important to understand that a lower score means that a vector is closer to the search vector. In my case, the result is ordered (ascending) by the vector field's score.</div><div><br /></div><div>I hope that you enjoyed this blog post. If you didn't find the time to read it entirely, then I also recorded a video walk-through.</div><div><br /></div><div><div style="padding:56.25% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/829726697?h=03eeeca992&badge=0&autopause=0&player_id=0&app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen style="position:absolute;top:0;left:0;width:100%;height:100%;" title="How to use Redis as a Vector Database for Recommendations"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script></div>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-34484725246774558682023-03-06T12:03:00.006+01:002023-03-06T12:20:00.642+01:00Gemeinsames Projekt zu 'AI-gestütztes Tool zur vereinfachten Erfassung von Objekten im Museum' mit der Hochschule Augsburg<p>Wir bei NoSQL Geeks freuen uns über eine weitere Zusammenarbeit mit dem Fachbereich Datenbanken der <a href="https://www.hs-augsburg.de/Informatik.html">Fakultät für Informatik der Hochschule Augsburg</a>. Wir wissen die Resonanz zu unserem Projektvorschlag <b>'</b><span style="font-family: Arial; font-size: 14.666667px; white-space: pre-wrap;"><b>AI-gestütztes Tool zur vereinfachten Erfassung von Objekten im Museum</b>' zu schätzen und blicken der Zusammenarbeit mit den Studierenden entgegen. Außerdem konnten wir das </span><a href="https://www.museum-krumbach.de">Mittelschwäbische Heimatmuseum Krumbach</a> als Anforderungsgeber, und Ansprechpartner zu fachlichen Fragen rund um die Dokumentation im Museum, gewinnen.</p><p><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">Hier eine kurze Projektbeschreibung: Museen jeder Größe stehen vor der Herausforderung, mit begrenzten personellen Mitteln Objekte aufzunehmen und zu dokumentieren. Die Beschreibung eines Objekts umfasst die Klassifizierung (z.B. Doppelhenkelvase), das Material (z.B. Porzellan), die Farbe, das Alter, die Herkunft, und viele weitere Eigenschaften. </span><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">Ziel des Projekts ist somit die Erstellung eines Open-Source-Tools zur Dokumentation im Museum, welches Merkmale wie Form, Farbe und Material erkennt und vorschlägt. </span><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">Die teilnehmenden Studenten werden die Gelegenheit haben, praktische Erfahrungen mit NoSQL-Datenmodellen, Vektoreinbettungen und „Vector Similarity Search“ zu sammeln.</span></p><p>Nähere Informationen sind außerdem auf der Github-Projektseite zu finden:</p><p></p><ul style="text-align: left;"><li><a href="https://github.com/nosqlgeek/ai-meets-museum">https://github.com/nosqlgeek/ai-meets-museum</a></li></ul><p></p>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-90375558628782276312023-03-06T11:33:00.010+01:002023-03-06T11:42:57.611+01:00Azure DevOps im Überblick<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjS3Qij6xv-fkcIguoJ4YE4n6QEiCphOMC5bZU36H0iCJ6xUz6sjxstG3mFaoAHDMtp4Up_Esg6Mzj_ZddY80w3P9Jaej0OYhsty0WhzdAp4xSWnrCKcHHfMu56KntRbfpE-HuxKw27D91iUn3_Kc_uSDQnfnTcSDWY1BuPc8GDflpk5nu90ntJl-XL" style="margin-left: 1em; margin-right: 1em; text-align: justify;"><img alt="" data-original-height="576" data-original-width="1024" height="180" src="https://blogger.googleusercontent.com/img/a/AVvXsEjS3Qij6xv-fkcIguoJ4YE4n6QEiCphOMC5bZU36H0iCJ6xUz6sjxstG3mFaoAHDMtp4Up_Esg6Mzj_ZddY80w3P9Jaej0OYhsty0WhzdAp4xSWnrCKcHHfMu56KntRbfpE-HuxKw27D91iUn3_Kc_uSDQnfnTcSDWY1BuPc8GDflpk5nu90ntJl-XL" width="320" /></a></div><div></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>Am 17.03.2023 treffen wir uns um 21:00 im <a href="https://www.google.com/maps/search/?api=1&query=48.243614%2C%2010.360803">Stückwerk</a>. Der Vortragende, Christian Linke, wird uns einen Überblick zum Thema 'DevOps mit und in der Azure Cloud' geben.</div><p></p><p>Nähere Informationen zum Event findet ihr in unserer Meetup-Gruppe hier:</p><p></p><ul style="text-align: left;"><li><a href="https://www.meetup.com/de-DE/nerdkram-mittelschwaben/events/292046891/">https://www.meetup.com/de-DE/nerdkram-mittelschwaben/events/292046891/</a></li></ul><div><br /></div><div><br /><br /></div><div><br /><br /></div><p></p>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-78537117041664492802022-12-02T09:30:00.000+01:002022-12-02T09:30:05.515+01:00Codecamp for Kids<p>NoSQL Geeks wird im Januar ein Codecamp für Kinder zwischen 12 und 16 Jahren organisieren. Nähere Details findet ihr hier: <a href="https://www.meetup.com/codecamps-by-nosql-geeks/events/290042192/">https://www.meetup.com/codecamps-by-nosql-geeks/events/290042192/</a>. Bei Fragen könnt ihr uns auch direkt kontaktieren. Alle Kontaktdaten finden ihr auf <a href="https://www.nosqlgeeks.de/de/index.html">https://www.nosqlgeeks.de</a>.</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw-N4zp0HrYrCNd6OgKDeGKQ45BnK9Pv-SUA-tryjOGw_XjUV0-hXt8tC9v9pJJZhXoqYM_s3J1Yhh115ZoQAeJWwVWX6f8f4T537sLy2AKZHeZNd8FZ9w68eWOAZ4vaM__OMUZHB-hBEfaEDp2Itz8pba_SJJlJwmeW-ysRlF_wpjALgX-QFWJRvj/s1388/55E62C55-49D0-48DE-ABB5-B09321DE60D4.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="1388" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw-N4zp0HrYrCNd6OgKDeGKQ45BnK9Pv-SUA-tryjOGw_XjUV0-hXt8tC9v9pJJZhXoqYM_s3J1Yhh115ZoQAeJWwVWX6f8f4T537sLy2AKZHeZNd8FZ9w68eWOAZ4vaM__OMUZHB-hBEfaEDp2Itz8pba_SJJlJwmeW-ysRlF_wpjALgX-QFWJRvj/s320/55E62C55-49D0-48DE-ABB5-B09321DE60D4.jpeg" width="320" /></a></div><br /><p><br /></p><p><br /></p>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-7394520101760704472022-12-02T09:20:00.000+01:002022-12-02T09:20:02.878+01:00New meet-up group in Mittelschwaben<p><br /></p><p>Wir freuen uns mitteilen zu können, dass wir eine Meet-up-Gruppe in Mittelschwaben organisieren werden. Themen sind u.a. IT, Softwareentwicklung und Datenverwaltung. Dazu werden wir uns regelmäßig in Krumbach (Schwaben) treffen. Nähere Informationen erhaltet ihr auf <a href="https://www.meetup.com/nerdkram-mittelschwaben">https://www.meetup.com/nerdkram-mittelschwaben/</a>. Lebt oder arbeitet Ihr nahe Mittelschwaben? Wollt ihr den Vorträgen zuhören, oder selbst präsentieren? Dann tretet doch einfach unserer Gruppe auf meetup.com bei!</p><p>/</p><p>We are happy to share that we will participate and sponsor a meet-up group around IT, Software Development, and Data Management in Mittelschwaben. The idea is to meet frequently in Krumbach (Schwaben). Further details can be found here: <a href="https://www.meetup.com/nerdkram-mittelschwaben">https://www.meetup.com/nerdkram-mittelschwaben/</a>. </p><p>Are you located in this area of the world? Do you want to participate, either as guest or as presenter? Then please join our group on meetup.com!</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRMKV3VuLw1z9uEms4iBBBIYNlIsT1I3ASL0O_1D0Lu9jTn4pQCLHybDwymHr36oR8cARKjti4wSFrWFUmcdDtI0Gc8PJ4ZyZCbMtZcQF9Pm5eSQUweENGyk7GfkCUOW_-OE5ABLM2N_yWidhCVUm_YE7xhbf3241Fh0tAnfr1jx5J1NUTfkTlGKM8/s1080/6B30A414-4801-4857-A2FF-1C8AC961DACC.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="878" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRMKV3VuLw1z9uEms4iBBBIYNlIsT1I3ASL0O_1D0Lu9jTn4pQCLHybDwymHr36oR8cARKjti4wSFrWFUmcdDtI0Gc8PJ4ZyZCbMtZcQF9Pm5eSQUweENGyk7GfkCUOW_-OE5ABLM2N_yWidhCVUm_YE7xhbf3241Fh0tAnfr1jx5J1NUTfkTlGKM8/s320/6B30A414-4801-4857-A2FF-1C8AC961DACC.jpeg" width="260" /></a></div><br /><p><br /></p>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-30164783052095996782022-11-21T10:31:00.004+01:002023-03-06T12:25:11.567+01:00Talk at the University of Applied Sciences in Augsburg about practical use cases of NoSQLIt was a pleasure visiting the University of Applied Sciences in Augsburg last week. Prof. Dr. Michael Predeschly invited me to give a talk about practical uses cases of NoSQL. It was amazing to see all those interested students and to answer their questions about polyglot persistence, the right usage of NoSQL databases, and practical use cases.
<div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7o4UlIFb9bvAlUNE5ND-Yjo0uuqgI6Wj2DZa00hX_XsIOxFLxXy2c79qOvi8O4rtTlV9-gY4zzk3i-loa2vSyW4PhtN-cZ_5OkuLKW9xTI1PbXWU8L5shVvURITUf0l_KpbN0hMyhMQ0KBdSJ18Z3IF9tilbV1O9e7RPes_xQXyLPQMhxi41Ha-43/s2204/Bildschirmfoto%202022-11-21%20um%2010.27.19.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1238" data-original-width="2204" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7o4UlIFb9bvAlUNE5ND-Yjo0uuqgI6Wj2DZa00hX_XsIOxFLxXy2c79qOvi8O4rtTlV9-gY4zzk3i-loa2vSyW4PhtN-cZ_5OkuLKW9xTI1PbXWU8L5shVvURITUf0l_KpbN0hMyhMQ0KBdSJ18Z3IF9tilbV1O9e7RPes_xQXyLPQMhxi41Ha-43/s320/Bildschirmfoto%202022-11-21%20um%2010.27.19.png" width="320" /></a></div><div><br /></div>Do you want to know more? Then reach out to me. My contact details can be found here: <a href="https://www.nosqlgeeks.de">https://www.nosqlgeeks.de</a> .<br /><div><br /></div>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-40746064330509301612022-11-20T16:03:00.010+01:002022-11-21T10:16:22.025+01:00NoSQL Geeks is part of the Stückwerk Community The Stückwerk Community is a project of the "Kult E.V." in Krumbach (the home town of the company NoSQL Geeks). The idea is to create a place where culture meets social initiatives. There are frequent events like art exhibitions or intercultural meet-and-greet-s. NoSQL Geeks will present during this year's XMas market that happens in the Stückwerk building in the town center. More about NoSQL Geeks can be found here: <a href="https://www.nosqlgeeks.de">https://www.nosqlgeeks.de</a>.
<br/>
/
<br/>
Das Stückwerk ist ein Projekt des Kult-Vereins in Krumbach (der Firmensitz der NoSQL Geeks). Die Idee ist es einen Platz der Begegnung von Kultur und sozialen Initiativen zu schaffen. So gibt es z.B. Veranstaltungen wie Kunstaustellungen oder interkulturelle Treffen. NoSQL Geeks wird sich wärend des diesjährigen Weihnachtsmarkts im Stückwerksgebäude im Stadzentrum vorstellen. Mehr über NoSQL Geeks erfahren Sie hier: <a href="https://www.nosqlgeeks.de">https://www.nosqlgeeks.de</a>.
<div><br /></div><div><div style="padding:56.25% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/773013927?h=f09d861f5e&badge=0&autopause=0&player_id=0&app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen style="position:absolute;top:0;left:0;width:100%;height:100%;" title="NoSQL Geeks im St&uuml;ckwerk Krumbach"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script><br /><div></div><div><a href="https://vimeo.com/773013927">https://vimeo.com/773013927</a></div></div>@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-27050192569543221752019-10-25T16:13:00.004+02:002022-11-20T18:15:09.238+01:00So what exactly is an Event Loop?<u><b>Introduction </b></u><br />
<br />
Most of you are knowing that I am working a lot with Redis. And some of you might also know that Redis OSS 'standalone' is mainly single-threaded. The reason why it can achieve that high throughputs on a single instance is that it uses an event loop. But what the hell is an event loop and how does it work?<br />
<br />
First of all, let me tell you the story behind this article. It all started last weekend. For some reason, I found the time to read a Kotlin book. Not sure why I did, guess I just had the feeling that I am too long disconnected from actual development tasks and wanted to explore one of the comparable new programming languages. The book was great and I had the impression that Kotlin is actually quite nice. Then I went back to my main task (helping to enable the Technical Field at Redis Labs) and had to work on a slide about the Redis event loop. A look at the source code (and the following article https://redis.io/topics/internals-rediseventlib), raised a question for me: Why is there a TimeEvent? The answer might finally sound simple, but just the fact that I dug a bit deeper into the event loop topic caused the idea to just implement a very simple event loop in Kotlin. So I did a bit more research and found also very good explanations of how the Node.js event loop works.<br />
<br />
So this article has the intention to share my learnings by using this light-weighted event loop that I implemented as an academic example<br />
<br />
<br />
<u><b>What is kEventLib?</b></u><br />
<br />
So kEventLib is a light-weighted event loop implementation in Kotlin. As said, this
project has more academical character. The idea is to illustrate how an event loop works by giving me the chance to play a bit around with Kotlin.<br />
<br />
<br />
<u><b>What's an Event?</b></u><br />
<br />
We are defining a generic event as something which can happen at a specific time, has a type and a payload:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/fa172dd53a05c9c869d5a1be9c301619.js"></script><br />
<br />
More specific events were derived from <code>Event</code>:<br />
<ul>
<li><b>SimpleEvent</b>: An event without a specific time. It doesn't matter exactly when such an event should be executed</li>
<li><b>TimedEvent</b>: An event which allows passing a delay, which means that the event should not be executed before this time is over</li>
</ul>
Timed events are having a lower priority than non-timed events. So we
will execute non-timed events first, but we are considering that timed
events are deferred to be executed in the future.<br />
An excellent example of timed events would be 'disk write' events in
Redis or async calls in Node.js.<br />
<br />
<br />
<u><b>Why do we need an Event Queue and Event Buffer?</b></u><br />
<br />
I decided to implement two different structures, dependent on if it is about a non-timed event or a timed event.<br />
<ul>
<li><b>EventQueue</b>: We are using the event queue to process the events in the order of their appearance. Node.js is using a stack
instead of a queue because calls can be nested, and so a call-stack
makes more sense.</li>
<li><b>EventBuffer</b>: This structure is used to buffer timed events. My naive event-loop works in a way that timed events are only processed after all non-timed events are processed. You could indeed think of more sophisticated scheduling approaches.</li>
</ul>
<br />
<u><b>What does the Event Loop?</b></u><br />
<br />
The event loop is a ... loop which runs a function call in a single thread:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/fae4fe0417a61357c382dae369932e73.js"></script>
<div class="highlight highlight-source-kotlin">
<pre><span class="pl-k"></span>
</pre>
</div>
Processing an event means to check first if the event queue is empty.
If not, then we are processing one of the queued events. If it is
empty, then we start processing the buffered events. All buffered events
that are in the past will be processed, whereby the event with the
minimum timestamp (the one which happened earliest) will be processed
first.<br />
It can happen that no event can be processed. Then an <code>EmptyEvent</code> is returned. It can also happen that an error occurs when submitting an event to the loop. This will return an <code>ErrorEvent</code>.
Such an error is caused by the fact that either the event loop's queue or buffer is fully utilized. The 'submitter' would then need to implement a back-pressure mechanism.<br />
<br />
<br />
<b><u>Show me an Example!</u></b><br />
<br />
Here some example code:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/0bea974485230dc8fa3dfbf38bde2238.js"></script><br />
<br />
The execution output (handling an event just prints some details about it) looks like:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/4db66c084fe88d174a169c7d4f7bed77.js"></script><br />
<br />
Events that are printed with the prefix '-1' are non-timed events.
Otherwise, the prefix is the timestamp (to which the event was deferred to).<br />
<br />
You can see that the non-timed events were executed first. Then we didn't submit non-timed for a while (due to the sleep after every submission of a timed event) which is the reason why some of the timed events are executed. Then we are executing the non-timed events again and finally the deferred timed ones. <br />
<br />
I hope you enjoyed reading this post. The full source code can be found here: <a href="https://github.com/nosqlgeek/kEventLib">https://github.com/nosqlgeek/kEventLib</a> . <br />
<br />@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-62132425557952079632019-05-10T10:42:00.002+02:002022-11-20T18:15:42.262+01:00A simple but special Redis Web ClientIt was a while ago when I wrote my last blog post. So let me take the chance to write something about a very simple piece of software which started as a fun project of mine. Let's do it a bit different this time. I will show you the results first and then I will explain what's special about this application:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://raw.github.com/nosqlgeek/ng-ipadprodemo/master/screenshot/Exec2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="654" data-original-width="800" height="326" src="https://raw.github.com/nosqlgeek/ng-ipadprodemo/master/screenshot/Exec2.jpg" width="400" /></a></div>
<br />
This looks very basic, right? What's special about it? So here is the story behind it:<br />
<br />
It all started a few weeks ago when I decide to buy an iPad Pro (11 inches). The motivation was indeed not to use it as a development machine. My current role requires to draw some diagrams and to explain stuff a bit more vizually. So the iPad Pro seemed to be a nice device for such a purpose.<br />
Being a techie, I wondered a bit which kind of development can be done on it and I started to install some tools for experimenting with them:<br />
<ul>
<li>Working Copy: A Git client</li>
<li>Pythonista: A Python IDE</li>
<li>StaSh: A shell for Pythonista which allows you to use packages via 'pip'</li>
<li>Blink: A CLI with an SSH client</li>
<li>VNC Viewer: Access the screen of your computer (which is running a VNC server)</li>
</ul>
<br />
Especially Pythonista is a great tool. Here a screen shot of it:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://raw.github.com/nosqlgeek/ng-ipadprodemo/master/screenshot/IDE.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="559" data-original-width="800" height="278" src="https://raw.github.com/nosqlgeek/ng-ipadprodemo/master/screenshot/IDE.jpg" width="400" /></a></div>
<br />
<br />
It all went a bit "crazy" when I went to my barber to get my beard cut. Now my Turkish barber is a very good one which means that he is very busy (seems this is a pattern across all industries - let it be IT-specialists or barbers). So I had to wait for about 2 hours to get a shave. As I already expected some waiting time, I took my iPad Pro with me (for i.e. reading a book). Holding it in my hands, I was then thinking why not trying to drive this 'How to develop on this device?' idea forward. But which kind of application should I develop? So one of the thoughts was to be able to demonstrate Redis (<a href="https://redis.io/" rel="nofollow">https://redis.io/</a>) itself on the iPad. Wouldn't it be cool to<br />
<ul>
<li>Just connect a small device with a Pen to a big screen (i.e. projector via USB-C, screen sharing)</li>
<li>Explain some concepts by just drawing like on a whiteboard</li>
<li>Demonstrate some Redis basics by being connected to a Redis Cloud instance (You can get a 30MB database for free here: <a href="https://redislabs.com/redis-enterprise/essentials/" rel="nofollow">https://redislabs.com/redis-enterprise/essentials/</a>)</li>
</ul>
?<br />
<br />
Given the fact that I invested some time to write this article, my opinion is clearly: "Yes, it would be cool!". However, if you are still wondering "Why the hell should I want to develop on an iPad Pro?" then I guess the answer is: "Because you can!" ;-) So this is more a fun project, whereby it might be a good basic example for:<br />
<ul>
<li>Understanding some Redis basics</li>
<li>Using a very popular Redis Python client (<a href="https://github.com/andymccurdy/redis-py">https://github.com/andymccurdy/redis-py</a>)</li>
<li>Understanding some Jinja2 (<a href="http://jinja.pocoo.org/docs/2.10/" rel="nofollow">http://jinja.pocoo.org/docs/2.10/</a>) HTML templating basics</li>
<li>Learning how to use some Flask basics (<a href="http://flask.pocoo.org/" rel="nofollow">http://flask.pocoo.org/</a>) for building a simple web application</li>
</ul>
<br />
So have fun!<br />
<br />
Ah before I forget it: If you are searching for a very basic Redis Web Client or if you are interested in to take a look at the source code, then the source code can be found <a href="https://www.blogger.com/u/1/for%20now:%20https://github.com/nosqlgeek/ng-ipadprodemo/edit/master/README.md">here</a> for now.<br />
<br />@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-86981827889499073502019-02-08T15:22:00.002+01:002022-11-20T18:15:58.780+01:00Building a Recommendation Engine with RedisWhen I was asked which topic I would like to present at this year's OOP conference, I was out of the box thinking about 'Something with Machine Learning' involved. It was years ago at the university when I had a secondary focus on 'Artificial Intelligence and Neural Networks' and I think that's fair to say that the topic was not as 'hot' as it is today. The algorithms were the same as today but the frameworks were not that commodity and calculations happened either on paper, with MatLab or with some very specialized software for neural network training. However, the actual discipline stayed fascinating and even if I would not call myself a Data Scientist (I sticked more with my primary focus which was Database Implementation Techniques - so I am more a database guy :-) ) I am really amazed of the adoption and number of arising frameworks in the field of Machine Learning and Artifical Intelligence.<br />
<br />
Machine Learning or Artificial Intelligence is quite a wide field and so I concluded to go with something more specific which has touch points to Machine Learning and AI. The topic with which I finally went is:<br />
<ul>
<li> Redis Modules for Recommender Systems</li>
</ul>
Most of you might know Redis already but it's maybe worth to mention what Redis actually is:<br />
<blockquote class="tr_bq">
Redis is an open source (BSD licensed), in-memory data structure store,
used as a database, cache and message broker. It supports data
structures such as strings, hashes, lists, sets, sorted sets with range
queries, bitmaps, hyperloglogs, geospatial indexes with radius queries
and streams. Redis has built-in replication, Lua scripting, LRU
eviction, transactions and different levels of on-disk persistence.</blockquote>
Redis is very popular. Here a ranking of the most popular database systems (just to highlight
how popular Redis is):<br />
<ul>
<li><a href="https://db-engines.com/en/ranking">https://db-engines.com/en/ranking</a></li>
</ul>
Redis is modular, which means that it can be extended by modules (like plug-ins). A list of Redis modules can be found here.<br />
<ul>
<li> <a href="https://redislabs.com/community/redis-modules-hub/">https://redislabs.com/community/redis-modules-hub/ </a></li>
</ul>
<u><b>Agenda</b></u><br />
<br />
As I am already seeing that this article is becoming a bit longer than a blog article should be, here an outlook regarding the topics. I hope this will motivate the one or the other to continue reading ... . <br />
<ol>
<li>Data Model</li>
<li>Preparations</li>
<li>Content Based Filtering (using Sets)</li>
<li>Collaborative Filtering (using Sets)</li>
<li>Ratings based Collaborative Filtering (using Sorted Sets)</li>
<li>Social Collaborative Filtering (using RedisGraph)</li>
<li>Content Relevance via Full Text Search (using RediSearch)</li>
<li>Probabilistic Data Structures (using the built-in HyperLogLog structure + ReBloom)</li>
<li>Machine Learning for Classifications and Predictions (using Redis-ML and other AI modules)</li>
</ol>
<ul>
</ul>
<u><b>Data Model</b></u><br />
<br />
Let's discuss which problem needs to be solved. Therefore the following simplified data model might be interesting:<br />
<ul>
<li><b>User</b>: A real-life person which is interacting with a system by showing some interests in items. Users can be classified.</li>
<li><b>Item</b>: The thing users can be interested in. Items can be classified.</li>
<li>Interest <b>Classification</b>: Classifications can happen based on item properties, the user attributes, the relationship of users to existing items or the relationship of users to other users and their items. A classification can be for instance expressed as a simple 'Class membership' or as a number which is telling how likely something belongs to which class.</li>
<li><b>Recommendation</b>: The actual recommendation, so which items could be interesting for a user is derived from some classifications. </li>
</ul>
Our example code will basicall use the following terms:<br />
<ul>
<li>Users are just named users</li>
<li>We are looking at specific items, which means Comic books</li>
<li>Several algorithms and approaches are using different kinds of classifications</li>
</ul>
<br />
<b><u>Preparations</u></b><br />
<b><u><br /></u></b>
If you want to follow the code samples of this blog post then you can also find a Jupyter notbook here.<br />
<ul>
<li> <a href="https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb">https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb</a></li>
</ul>
I basically prepared the following Redis instances (one per module): <br />
<ul>
<li>Redis (r) </li>
<li>Redis + Machine Learning (r_m)</li>
<li>Redis (Bloom Filters) + HyperLogLog (r_b)</li>
<li>Redis + Graph (r_g)</li>
<li>Redis + Full Text Search (r_s)</li>
</ul>
<br />
<br />
<script src="https://gist.github.com/nosqlgeek/dbe7e3ff3a5a052d895a95fe0760c405.js"></script>
<i>Python Prep Script</i><br />
<br />
<br />
<u><b>Content Based Filtering</b></u><br />
<br />
The idea is to look at what a specific user is interested in and then to
recommend things those are similar (i.e. having the same class) as
other things the user is liking.<br />
<br />
Content based filtering can be done based on 'real set' operations. Redis is coming with a 'Set' data structure which is allowing to perform membership checks, scans, intersections, unions and so on.<br />
<br />
Let's look at the following example: <br />
<br />
<script src="https://gist.github.com/nosqlgeek/1e45caea7803879619a05153f6df2c40.js"></script>
<i>Python Script</i><br />
<u><b></b></u><br />
The output is:<br />
<blockquote class="tr_bq">
David could be also interested in: { 'Fantastic Four', 'Wonder Woman', 'Batman', 'Dragon Age', 'Avatar', 'Valerian', 'Spiderman'}</blockquote>
<br />
<br />
<u><b>Collarborative Filtering</b></u><br />
<u><b><br /></b></u>
The
underlying idea is that if person A likes the same things as person B,
then person B might also like the other items those are liked by person
A. So it's mandatory to have details about many other users collected for a proper classification:<br />
<br />
We are using again Redis' Set data structure and especially the union and diff operations.<br />
<br />
Let's add some demo data:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/6db875122d23ae21cde60ee3047301bd.js"></script>
<i>Python Demo Data Script </i><br />
<br />
Now let's look at the following example:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/ab49315c873bb5239f7b65c429d0332c.js"></script>
<i>Python Script</i><br />
<br />
We can now see which other users B could be interested in the same items as A and then derive a recommendation for the given user A based on the other interests of users B:<br />
<blockquote class="tr_bq">
Users interested in the same items as David: {'david', 'pieter'}<br />
David is interested in: {'Spiderman', 'Batman'}<br />
David could be also interested in: {'Wonder Woman'} </blockquote>
<br />
<u><b>Ratings based Collaborative Filtering</b></u><br />
<br />
We are talking about collaborative filtering again. So the approach is to derive a recommendation from similarities to other users. In addition we are now interested in 'How much
does a user like an item?'. This allows us i.e. to find out if two or more users
are liking similar things. Items those are liked by user B but
not yet liked by user A could be also interesting for user A.<br />
<br />
The Redis structure which is used is a 'Sorted Set'. An element in a sorted has a score. We will use this score as our rating value (1-5, i.e. stars). A very cool feature of sorted sets is that set operations are allowing to aggregate scores. By default, the resulting score of an element is the sum of the scores across the considered sorted sets. You can combine aggregations with weights. Weights are multiplicators for scores. The weight (1,-1) means to subtract the second score value from the first score value.<br />
<br />
We will first find users B those rated the same items as a specific user A. The idea is then to leverage this aggregation feature in order to calculate the distance vector of ratings between user A and the previously identified users B. We will then use RMS in order to calculate the average distance as a scalar. The R(oot) M(ean) S(quare) value of a set of values is the square root of the arithmetic mean of the squares of the values. Only users with an average rating distance less or equal to 1 (which means that the users rating was very similar) will be considered. Finally we will recommend items of users B to user A, whereby we are only considering items with a score of at least 4.<br />
<br />
I added some helper functions to the following script. I am bascially not pasting them here again by hoping that their functionality is self-explaining. The full source code can be found here: <a href="https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb">https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb</a> .<br />
<br />
<i><br /></i>
<script src="https://gist.github.com/nosqlgeek/d22412e22c98a7062afba7a21d26aefd.js"></script>
<i>Python Demo Data Script</i><br />
<br />
<br />
Let's first find users B those are liking the same things as A:<br />
<br />
<br />
<script src="https://gist.github.com/nosqlgeek/9936ae13adfbba3840f811059026a7de.js"></script>
<i>Python Script</i><br />
<br />
The output is:<br />
<blockquote class="tr_bq">
The following users rated David's items: ['pieter', 'david']</blockquote>
Now let's calculate the similarities by then proposing the highest rated items of users B:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/db86a93bad4ab08c1379f7331100ceb4.js"></script>
<i>Python Script</i><br />
<br />
The result is that Pieter has a matching rating distance and so 'Aqua Man' is a highly recommended Comic to David (whatever tells this about Pieter ;-) ):<br />
<blockquote class="tr_bq">
The rating distance to pieter is [('batman', 1.0), ('superman', -1.0)]<br />
The average distance (RMS) to pieter is 1.0<br />
The following is highly recommended: [('aqua_man', 5.0)]</blockquote>
<br />
<br />
<u><b>Social Collaborative Filtering</b></u><br />
<u><b><br /></b></u>
The previous examples used Sets and Sorted Sets. We are now exploring
how to use Graphs. Our example is taking a social ('friend of') aspect
into account. A mathematical Graph is described as a set of vertices V and a set of Edges E, whereby E is a subset of VxV. Graph database systems are extending this mathematical definition to a so called 'Property Graph Model' which means that vertices and edges can have properties (KV pairs) associated.<br />
<br />
Our idea is to find all comics of friends B of a given user A those are interested in a specific comic category (Super Heros). Comic books that are liked more often by the friends of user A are more relevant and should be recommended.<br />
<br />
I am again skipping the helper functions in order to avoid to blow this article even more up. If you are interested, then the full source code can be found here: <a href="https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb">https://github.com/nosqlgeek/rl-recsys/blob/master/notebooks/Redis_for_Recommendations.ipynb</a>. The function names are hopefully self-explaining.<br />
<br />
<br />
<script src="https://gist.github.com/nosqlgeek/666d77cc8600fb8686b36411b98327ed.js"></script>
<i>Python Demo Data Script</i><br />
<br />
Here the actual Graph Query:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/7ad6d20019292bb60004ae3f956613d0.js"></script>
<i>Python Script</i><br />
<br />
As 'Wonder Woman' is liked by 2 of David's friends it is more relevant than the other comics:<br />
<blockquote class="tr_bq">
David has the following friends: [[['name'], ['Pieter'], ['Vassilis'], ['Katrin']]]<br />
David likes [[['name'], ['Spiderman'], ['Batman']]]<br />
Comic 'Wonder Woman' with relevance 2.000000<br />
Comic 'Batman' with relevance 1.000000<br />
Comic 'Superman' with relevance 1.000000</blockquote>
<br />
<u><b>Content Relevance via Full Text Search</b></u><br />
<br />
RediSearch is a search engine module for Redis. It comes with multiple built-in scoring functions. We will look at T(erm)F(requency)I(inverse)D(ocument)F(requency). It takes the following aspects into account:<br />
<ol>
<li>Term Frequency: How often does a specific term appear?</li>
<li>Inverse Document Frequency: An inverse document frequency factor is
incorporated which diminishes the weight of terms that occur very
frequently in the document set and increases the weight of terms that
occur rarely (i.e. the relevance of the word 'the'</li>
</ol>
Furhter details about scoring can be found here:<br />
<ul>
<li><a href="https://oss.redislabs.com/redisearch/Scoring/">https://oss.redislabs.com/redisearch/Scoring/</a></li>
</ul>
We are trying to identify how likely something belongs to a specific class/category by performing a text search for terms those are associated to this category. <br />
<br />
Helper functions are again skipped in this article but can be found in the source code repo.<br />
<br />
<br />
<script src="https://gist.github.com/nosqlgeek/c415c5e45e3ca417f6a7ee861da63e8c.js"></script>
<i>Python Script</i><br />
<br />
Spiderman is more likely a super hero than Batman:<br />
<blockquote class="tr_bq">
[2, 'spiderman', 0.10000000000000001,['name', 'Spiderman'], 'batman', 0.035714285714285712,['name', 'Batman']]</blockquote>
<br />
<u><b>Probabilistic Data Structures</b></u><br />
<u><b><br /></b></u>
Probabilistic data structures are characterized in the follwoing way: They ...<br />
<ul>
<li>use hash functions for randamization purposes</li>
<li>return an approximated result</li>
<li>the error is under a specific threshold</li>
<li>are much more space efficient than deterministic approaches</li>
<li>provide a constant query time</li>
</ul>
You would use them because sometimes …<br />
<ul>
<li>speed is more important than correctness</li>
<li>compactness is more important than correctness</li>
<li>you only need certain data guarantees</li>
</ul>
It's possible to combine them with deterministic approaches (i.e. HLL + det. counter for discovering counter manipulations).<br />
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<br />
We will take a look at the following two structures:<br />
<ul>
<li>
<b>HyperLogLog</b>: Cardinality estimation of a set, i.e. unique visits</li>
<li>
<b>Bloom Filter</b>: Check if an item is contained in a set whereby false-positves are possible</li>
</ul>
Our example will not use 'unique visits' but we are more interested in how many unique users 'touched' a specific comic. Just imagine a real-life comic book store. A bunch of nerds (including myself ...) are hanging around and they are browsing for comics. Interesting comics will be removed from the shelf in order to take a closer look. This is what I mean with 'touched'. A comic which is more often touched can be considered as more interesting. We can count these unique touches quite space efficently by using a HyperLogLog:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/25a70b8b562a8f07ca18b2337b27bfde.js"></script>
<i>Python Script </i><br />
<br />
The output is:<br />
<blockquote class="tr_bq">
HLL initial size: 31<br />
Approx. count: 4<br />
Please wait ...<br />
Final HLL size: 10590 bytes<br />
Approx. count: 99475 </blockquote>
<br />
The bloom filter can be used to check if a user is interested a specific comic category without storing the users per category in a set:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/07d95dd64e3f0b1bcf6dedad6200cd3f.js"></script>
<i>Python Script</i><br />
<br />
We are also printing the sizes out in order to demonstrate how space efficient Bloom filters are. The output of this script is:<br />
<blockquote class="tr_bq">
BF size: 115 bytes<br />
BF size: 76 bytes<br />
Is Katrin interested in Fantasy?: 1<br />
Is Katrin interested in Super Heros?: 0<br />
Is David interested in Super Heros?: 1<i><br /></i></blockquote>
<br />
<u><b>Machine Learning for Classifications and Predictions</b></u><br />
<br />
We are closing this blog article by circling back to the introduction of it. Classifications were so far often seen as a given (i.e. a comic book belonging to the 'Fantasy' category). Others could be derived by taking the existing user interests into account. We also mentioned in the section 'Data Model' that classifications might be derived from user attributes or item properties. Now, Machine Learning is providing us ways to describe a more complex models by taking such attributes (=features) into account. Such features can be represented by structured data (i.e. the comic name, ...) or unstructured data (i.e. the images within a comic book, the used colors within comic book). Feature vectors could be for instance derived from the bitmap of a scanned cover of a comic book. At the end, you can consider every ML approach as a way to approximate a function F(x) -> y, whereby x is a feature vector and y is the output vector. The idea is to create/train a model based on the known values y for given vectors x. These given vectors are called the training features. The idea is to derive a model which is able to approximate/predict a 'good' output vector y for an unknown input vector x.<br />
<br />
Here 2 examples for such models:
<br />
<ul>
<li>
Decision Tree ensembles (random forests). The idea is to conduct a
forest of decision trees at training time. RedisML can be used for the
Model Serving by leveraging these decision trees for i.e. classification
purposes. The class which appears most often will be the winner.<br />
</li>
<li>
Neural networks: Train the weighted connections between neurons by
using a learning algorithm (i.e. Backpropagation).<br />
</li>
</ul>
The following example leverages 2 very small decision trees:<br />
<ul>
<li>Users with an age <=20 are liking Manga comics</li>
<li>Users with more than 1000 are not liking Manga comics</li>
</ul>
</div>
<div class="text_cell_render border-box-sizing rendered_html">
<script src="https://gist.github.com/nosqlgeek/8818523be82900987e720b615a36eb69.js"></script>
<i>Python Script</i><br />
<br />
I am feeling that this article could tell much more about 'Neural Networks and Artificial Intelligence', but I am also hoping that it's understandable that this is a very wide field and so I am thinking that it is worth to write a dedicated article about Redis for Machine Learning and Artificial Intelligence at a later point in time.<br />
<br />
Finally, here some attitional Redis modules those didn't make it into one of the earlier sections:<br />
<br />
<ul>
<li><b>Neural Redis</b>: Is a Redis module that implements feed
forward neural networks as a native data type for Redis. The project
goal is to provide Redis users with an extremely simple to use machine
learning experience.</li>
<li><b>Countminsketch</b>: An apporximate frequency counter</li>
<li><b>Topk</b>:
An almost deterministic top k elements counte</li>
<li><b>Redis-tdigest</b>: T-digest data structure wich can be used for accurate online
accumulation of rank-based statistics such as quantiles and cumulative
distribution at a point.
</li>
</ul>
<ul>
</ul>
Thanks for reading! Feedback regarding this article is very welcome! </div>
</div>
</div>
@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-89473029792606892482018-06-19T18:21:00.005+02:002022-11-20T18:16:06.208+01:00Asynchronous Operation Execution with Netty on RedisNetty got my attention a while back and I just wanted to play a bit around with it. Given the fact that I am already fallen in love with Redis, what would be more fun than implementing a low level client for Redis based on Netty?<br />
<br />
Let's begin to answer the question "What the hell is Netty?". Netty is an asynchronous (Java based) event-driven network application framework. It is helping you to develop high performance protocol servers and clients.<br />
<br />
We are obviously more interested in the client part here, meaning that this article is focusing on how to interact with a Redis Server.<br />
<br />
Netty is already coming with <a href="https://redis.io/topics/protocol">RESP</a> support. The package 'io.netty.handler.codec.redis' contains several Redis message formats:<br />
<br />
<br />
<ul>
<li><b>RedisMessage</b>: A general Redis message</li>
<li><b>ArrayRedisMessages</b>: An implementation of the RESP Array message</li>
<li><b>SimpleRedisStringMessage</b>: An implementation of a RESP Simple String message</li>
<li>...</li>
</ul>
<br />
So all we need to do is to:<br />
<br />
<ol>
<li><b>Boostrap a channel</b>: A channel is a nexus to a network socket or component which is capable of I/O operations. Bootstrapping means to assign the relevant components to the channel (Event loop group, handlers, listeners, ...) and to establish the socket connection. An example class can be found <a href="https://github.com/nosqlgeek/jRxRedis/blob/master/src/main/java/org/nosqlgeek/jrxredis/core/netty/RedisClientBootstrap.java">here</a>.</li>
<li><b>Define a channel pipeline</b>: We are using an initialization handler in order to add several other handlers to the channel's pipeline. The pipeline is a list of channel handlers, whereby each handler handles or intercepts inbound events or outbound operations. Our channel pipeline is having the following handlers: RedisDecoder (Inbound handler that decodes into a RedisMessage), RedisBulkStringAggregator (Inbound handler that aggregates an BulkStringHeaderRedisMessage and its following BulkStringRedisContents into a single FullBulkStringRedisMessage), RedisArrayAggregator (Aggregates RedisMessage parts into an ArrayRedisMessage) and RedisEncoder (This outbound handler encodes RedisMessage into <br />bytes by following the RESP (REdis Serialization Protocol). Netty will first apply the outbound handlers to the passed in value. Then it will put the encoded message on the socket. When the response will be received then it will apply the inbound handlers. The last handler is then able to work with the decoded (pre-handled) message. An example for such a pipeline definition can be found <a href="https://github.com/nosqlgeek/jRxRedis/blob/master/src/main/java/org/nosqlgeek/jrxredis/core/netty/RedisChannelInit.java">here</a>.</li>
<li><b>Add a custom handler</b>: We are also adding a custom duplex handler to the pipeline. It is used in order to execute custom logic when a message is received (channelRead) or sent (write). We are not yet planning to execute business logic based on the RedisMessage but instead want to just fetch it, which means that our handler just allows to retrieve the result. My handler is providing an asynchronous method to do so. The method 'sendAsyncMessage' returns a Future. It's then possible to check if the Future is completed. When it is completed then you can get the RedisMessage from it. This handler is buffering the futures until they are completed. The source code of my example handler can be found <a href="https://github.com/nosqlgeek/jRxRedis/blob/master/src/main/java/org/nosqlgeek/jrxredis/core/netty/RedisClientHandler.java">here</a>. </li>
</ol>
BTW: It's also possible to attach listeners to a channel. Whereby I found it initially to be a good idea to use listeners in order to react on new messages, I had to realize that channel listeners are invoked before the last handler (the last one is usually your custom one), which means that you face the issue that your received message did not go through the channel pipeline when the listener is invoked. So my conclusion is that channel listeners are more used for side tasks (inform someone that something was received, log a message out, ...) instead of the message processing itself, whereby handlers are designed in order to be used to process the received messages. So if you want to use listeners then a better way is to let the handler work with promises and then attach the listener to the promise of a result.<br />
<div>
<br /></div>
<div>
In addition the following classes were implemented for demoing purposes:</div>
<div>
<ul>
<li><b>GetMsg and SetMsg</b>: Are extending the class ArrayRedisMessage by defining how a GET and a SET message are looking like.</li>
<li><b>AsyncRedisMessageBuffer</b>: A message buffer which uses a blocking queue in order to buffer outgoing and incoming messages. The Redis Client Handler (my custom handler) is doing the following: Sending a message causes that the Future is put into the buffer. When the response arrives then the Future is updated and removed from the buffer. Whoever called the 'sendAsyncMessage' method has hopefully still a reference to the just dequeued Future. I used '<a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/LinkedBlockingDeque.html">LinkedBlockingDeque</a>' which means that the implementation should be thread safe.</li>
</ul>
<div>
Here a code example how to use the handler in order to execute an asynchronous GET operation:</div>
<div>
<br />
<script src="https://gist.github.com/nosqlgeek/d76154ce3d7fbd87713fe7d109917b51.js"></script><br />
<br /></div>
<div>
Hope you enjoyed reading this blog post! Feedback is welcome.</div>
<div>
<br /></div>
</div>
<div>
<div>
<br /></div>
</div>
@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-60675116291799444042018-06-06T16:09:00.000+02:002018-06-06T20:25:37.575+02:00Data Encryption at RestData security and protection is currently a hot topic. It seems that we reached the point when the pendulum is swinging back again. After years of voluntary openness by sharing personal information freely with social networks, people are getting more and more concerned about how their personal data is used in order to profile or influence them. Social network vendors are getting currently bad press, but maybe we should ask ourself the fair question "Didn't we know all the time that their services are not for free and that we are paying them with our data?". Maybe not strictly related to prominent (so called) 'data scandals' but at least following the movement of the pendulum is the new European GDPR regulation around data protection. Even if I think that it tends to 'overshoot the mark' (as we would say in German) and leaves data controllers and processors sometimes in the dark (unexpected rhyme ...), it is a good reason for me address some security topics again from a technical point of view. So this article has the subject of 'Data Encryption at Rest' on Linux servers.<br />
<br />
To be more accurate this article is mainly focusing on how to ensure that folders are encrypted under Linux. Linux provides the following ways to encrypt your data:<br />
<ul>
<li><b>Partition level</b>: This allows you to define an encrypted partition on a hard drive. I think that this is the most commonly seen way of encrypting data with Linux. Most Linux distributions are providing this option already during the installation</li>
<li><b>Folder level</b>: Allows you to encrypt specific folders by i.e. mounting them under as specific path. The 'ecryptfs' solution can be used for such a purpose.</li>
<li><b>File level</b>: It is also possible to encrypt single files. The PGP (Pretty Good Privacy) tools can be used for this purpose.</li>
</ul>
<div>
<br /></div>
<div>
This article focuses on the 'Folder level' encryption. It has the advantage that you can define encrypted folders on-demand without the need to repartition your drives. It also doesn't just work on single files but allows you to mount your folder directly. Each file which is stored in the encrypted folder is encrypted separately. This is especially useful if you want to encrypt only specific data by providing only specific users unencrypted access. One use case would be to only allow your CIFS service (File Server service) unencrypted access to the folder. I can also easily see that database systems could leverage this feature, whereby I didn't test which performance implication might be seen when using folder level encryption with DBMS.</div>
<div>
<br /></div>
<div>
<ul>
<li><b>Step 0 - Install 'ecryptfs'</b></li>
</ul>
<div>
<div>
<div>
<table border="2px" cellpadding="10" style="width: 100%;"><tbody>
<tr><td><span style="background-color: white; color: #c33720; font-family: "menlo"; font-size: 11px;"><b>apt</b></span><span style="background-color: white; font-family: "menlo"; font-size: 11px;">-get -y install ecryptfs-utils</span></td></tr>
</tbody></table>
</div>
</div>
</div>
</div>
<div>
<ul>
<li><b>Step 1 - Create a hidden directory</b>: Let's assume that we have a folder /mnt/data which is the mount point of your main data partition (in my case an EXT4 partition on a RAD1 of 2 spinning HDD-s. We create a hidden folder named encrypted there:</li>
</ul>
<div>
<table border="2px" cellpadding="10" style="width: 100%;">
<tbody>
<tr><td><span style="color: #c33720; font-family: "menlo"; font-size: 11px;"><b>mkdir</b></span><span style="font-family: "menlo"; font-size: 11px;"> /mnt/data/.encrypted</span></td></tr>
</tbody></table>
</div>
<div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<br /></div>
</div>
</div>
<div>
<ul>
<li><b>Step 2 - Create a second folder: </b>This folder is used as our mount point. All access needs to happen via this second folder.</li>
</ul>
<div>
<table border="2px" cellpadding="10" style="width: 100%;"><tbody>
<tr><td><span style="background-color: white; color: #c33720; font-family: "menlo"; font-size: 11px;"><b>mkdir</b></span><span style="background-color: white; font-family: "menlo"; font-size: 11px;"> /mnt/encrypted</span></td></tr>
</tbody></table>
</div>
<ul>
<li><b>Step 2 - Mount the hidden folder as an encrypted one:</b> Let's assume we want to access our encryption folder under /mnt/encrypted. This means that each write to the newly mounted folder is involving the encryption of the written data. Here a small script which does the job:</li>
</ul>
</div>
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<table border="2px" cellpadding="10" style="width: 100%;"><tbody>
<tr><td><div style="margin: 0px;">
<span style="background-color: white; color: #5230e1; font-family: "menlo"; font-size: 11px;">#!/bin/bash</span></div>
<div>
<div style="background-color: white; color: #33bbc8; font-family: menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="color: black; font-variant-ligatures: no-common-ligatures;">mount </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-t</span><span style="color: black; font-variant-ligatures: no-common-ligatures;"> ecryptfs\</span><br />
<span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-o </span><span style="font-variant-ligatures: no-common-ligatures;">rw,relatime,ecryptfs_fnek_sig</span><span style="color: black; font-variant-ligatures: no-common-ligatures;">=</span>82028e5be8a0a05b,\<br />
<span style="font-variant-ligatures: no-common-ligatures;">ecryptfs_sig</span><span style="color: black; font-variant-ligatures: no-common-ligatures;">=</span><span style="font-variant-ligatures: no-common-ligatures;">5</span>5<span style="font-variant-ligatures: no-common-ligatures;">028e0be5a0a0</span>8<span style="font-variant-ligatures: no-common-ligatures;">a,ecryptfs_cipher</span><span style="color: black; font-variant-ligatures: no-common-ligatures;">=</span><span style="font-variant-ligatures: no-common-ligatures;">aes,\</span><br />
<span style="font-variant-ligatures: no-common-ligatures;">ecryptfs_key_bytes</span><span style="color: black; font-variant-ligatures: no-common-ligatures;">=</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">16</span><span style="color: black; font-variant-ligatures: no-common-ligatures;">,ecryptfs_unlink_sigs\ </span><span style="color: black;">/mnt/data/.encrypted /mnt/encrypted</span></div>
</div>
</td></tr>
</tbody></table>
</div>
<div>
<div>
<br /></div>
</div>
The mount command will ask your for the passphrase. The passphrase will be used for every remount.<br />
<br />
WARNING: If you loose your passphrase, then you will no longer be able to read your previously encrypted data.<br />
<br />
This is what's stored in your mounted folder:<br />
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div>
</div>
</div>
<br />
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<table border="2px" cellpadding="10" style="width: 100%;"><tbody>
<tr><td><div style="margin: 0px;">
<span style="background-color: white; color: #34bc26; font-family: "menlo"; font-size: 11px;"><b>root@ubuntu-server</b></span><span style="background-color: white; font-family: "menlo"; font-size: 11px;">:</span><span style="background-color: white; color: #5230e1; font-family: "menlo"; font-size: 11px;"><b>/mnt/encrypted</b></span><span style="background-color: white; font-family: "menlo"; font-size: 11px;"># ls</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">hello2.txt hello.txt</span></div>
<div style="background-color: white; color: #34bc26; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"><b>root@ubuntu-server</b></span><span style="color: black; font-variant-ligatures: no-common-ligatures;">:</span><span style="color: #5230e1; font-variant-ligatures: no-common-ligatures;"><b>/mnt/encrypted</b></span><span style="color: black; font-variant-ligatures: no-common-ligatures;"># cat hello.txt </span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">Hello world!</span></div>
</td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<div>
Whereby the original folder contains the encrypted data:<br />
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<table border="2px" cellpadding="10"><tbody>
<tr><td><div style="margin: 0px;">
</div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
</div>
<div style="background-color: white; color: #5230e1; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="color: #34bc26; font-variant-ligatures: no-common-ligatures;"><b>root@ubuntu-server</b></span><span style="color: black; font-variant-ligatures: no-common-ligatures;">:</span><span style="font-variant-ligatures: no-common-ligatures;"><b>/mnt/data/.encrypted</b></span><span style="color: black; font-variant-ligatures: no-common-ligatures;"># ls</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">ECRYPTFS_FNEK_ENCRYPTED.FWYW-ctPtO0USURgl98vtKSoykT9hmQROUa3cBMaMT0UyWKbxkF7KQOiU--- ECRYPTFS_FNEK_ENCRYPTED.FWYW-ctPtO0USURgl98vtKSoykT9hmQROUa3TeggyUTAxFqhqUkBB.a-Bk--</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="color: #34bc26; font-variant-ligatures: no-common-ligatures;"><b>root@ubuntu-server</b></span><span style="font-variant-ligatures: no-common-ligatures;">:</span><span style="color: #5230e1; font-variant-ligatures: no-common-ligatures;"><b>/mnt/data/.encrypted</b></span><span style="font-variant-ligatures: no-common-ligatures;"># cat ECRYPTFS_FNEK_ENCRYPTED.FWYW-ctPtO0USURgl98vtKSoykT9hmQROUa3cBMaMT0UyWKbxkF7KQOiU---</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">??tY?ì</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">?"3DUfw`n6?</span></div>
<div dir="rtl" style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"> ?3ﯙY7?_?_CONSOLE"?[堠zx?ŷZ?G??铅?Lj*?9?.fEN??`????R?:??83?F???{???</span></div>
<div style="background-color: white; font-family: Menlo; font-size: 11px; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"> ??_Z&tx?,?2!?w</span></div>
<div>
<span style="font-variant-ligatures: no-common-ligatures;"><br /></span></div>
</td></tr>
</tbody></table>
</div>
<br />
Access to the decrypted data is possible under the following circumstances:<br />
<br />
<ul>
<li>The logged-in user has permission to read or write the folder /mnt/encrypted</li>
<li>The folder /mnt/data/.encrypted was mounted to /mnt/encrypted by providing the passphrase</li>
</ul>
<div>
It's especially no longer possible to read the unencrypted data after removing the hard disk physically from a machine. As said, it's necessary to know the passphrase in order to decrypt the data of this folder again.</div>
<div>
<br /></div>
<div>
Hope this article is helpful :-) .</div>
</div>
<br />@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-18861006188006826982018-01-22T21:13:00.002+01:002022-11-20T18:17:38.632+01:00To PubSub or not to PubSub, that is the question<br />
<h3>
Introduction </h3>
The PubSub pattern is quite simple:<br />
<ul>
<li>Publishers can publish messages to channels</li>
<li>Subscribers of these channels are able to receive the messages from them</li>
</ul>
There is no knowledge of the publisher about the functionality of any of the subscribers. So they are acting independently. The only thing which glues them together is a message within a channel.<br />
<br />
Here a very brief example with Redis:<br />
<ul>
<li>Open a session via 'redis-cli' and enter the following command in order to subscribe to a channel with the name 'public' </li>
</ul>
<script src="https://gist.github.com/nosqlgeek/623cdf75dea2915ff64a3b758be562e3.js"></script><br />
<ul>
<li> In another 'redis-cli' session enter the following command in order to publish the message 'Hello world' to the 'public' channel: </li>
</ul>
<script src="https://gist.github.com/nosqlgeek/f39ab12b2dfc0987c45a352af075c032.js"></script> <br />
<br />
The result in the first session is:<br />
<ul>
</ul>
<br />
<script src="https://gist.github.com/nosqlgeek/29534201a0cfda245e97da789dd7b869.js"></script><br />
<br />
BTW: It's also possible to subscribe to a bunch of channels by using patterns, e.g. `PSUBSCRIBE pub*` <br />
<h3>
</h3>
<h3>
</h3>
<h3>
Fire and Forget </h3>
If we would start additional subscribers after our experiment then they won't receive the previous messages. So we can see that we can only receive messages when we are actively subscribed. Meaning that we can't retrieve missed messages afterwards. In other words:<br />
<ul>
<li>Only currently listening subscribers are retrieving messages</li>
<li>A message is retrieved by all active subscribers of a channel</li>
<li>If a subscriber dies and comes back later then it might have missed messages</li>
</ul>
PubSub is completely independent from the key space. So whatever is published to a channel will not directly affect the data in your database. Published messages are not persisted and there are no delivery guarantees. However, you can indeed use it in order to notify subscribers that something affected your key space (e.g. The value of item 'hello:world' has changed, you might fetch the change!).
So what's the purpose of PubSub then? It's about message delivery and notifications. Each of the subscribers can decide by himself how to handle the received message. Because all subscribers of a channel receive the same message, it's obviously not about scaling the workload itself. This is an important difference in comparision to message queue use cases.<br />
<h3>
</h3>
<h3>
Message Queues</h3>
Message queues on the other's hand side are intended to scale the workload. A list of messages is processed by a pool of workers. As the pool of workers is usually limited in size, it's important that messages are buffered until a worker is free in order to process it. Redis (Enterprise) features like <br />
<ul>
<li>Persistency</li>
<li>High Availability </li>
</ul>
are quite more important for such a queuing scenario. So such a queue should surrive a node failure or a node restart.
Redis fortunately comes with the LIST data structure for simple queues or a Sorted Set structure for priortiy queues.<br />
<br />
It's important to state that there are already plenty of libraries and solutions out there for this purpose. Here two examples:<br />
<ul>
<li><a href="https://github.com/resque/resque)">Resque</a></li>
<li><a href="https://github.com/mperham/sidekiq/wiki">Sidekiq</a></li>
</ul>
A very simple queue implementation would use a list. Because entries of the list are strings, it would be good to encode messages into e.g. JSON if they have a more complex structure. <br />
<ul>
<li>Create a queue and inform the scheduler that a new queue is alive:</li>
</ul>
<script src="https://gist.github.com/nosqlgeek/d4ede6d916af2165c5a055bf7e80124a.js"></script> <br />
<ul>
</ul>
<ul>
<li>Add 2 messages to the queue:</li>
</ul>
<ul>
</ul>
<script src="https://gist.github.com/nosqlgeek/10c201ee9443a8023c3a3a50e31c6241.js"></script><br />
<br />
<ul>
<li>Schedule the workers: We could indeed use a more complex scheduling apporach. However, the simplest and stupidest would be to just assign the next worker of the pool to the next message. So in order to dequeue a message we can just use `LPOP`:</li>
</ul>
<br />
<script src="https://gist.github.com/nosqlgeek/839d922671262a66f5a2b319a382b324.js"></script><br />
<br />
BTW: If our queue would be initially empty then there is a way to wait for a while until something arrives by using the `BLPOP` command.<br />
<br />
Using PubSub is actually optional for our message queue example. It's easy to see that the scheduler could also assign workers without getting notified because it can at any time access the queues and messages. However, I found it a bit more dynamic to combine our queue example with PubSub: <br />
<ul>
<li>The scheduler gets notified when new work needs to be assigned to the workers</li>
<li>As these notifications are fire and forget, it would be also possible for the scheduler to check from time to time if there is something to do</li>
<li>If the scheduler dies then another instance can be started which can access the database in order to double check which work was already done by the workers and which work still needs to be done. An interuppted job can be restarted based on such state information. </li>
</ul>
<h3>
</h3>
<h3>
</h3>
<h3>
Summary </h3>
Redis' PubSub is 'Fire and forget'. It's intended to be used to deliver messages from many (publishers) to many (subscribers). It's indeed a useful feature for notification purposes. However, it's important to understand the differences between a messaging and a message processing use case.<br />
<br />
The way how we used it was to inform a single scheduler that some work needs to be done. The scheduler would then hand over to a pool of worker threads in order to process the actual queue. The entire state of the queue was stored in our database as list because PubSub alone is not intended to be used for message queuing use cases. In fact the usage of PubSub for our queuing example was optional.@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-23478222629807176542017-07-27T11:29:00.004+02:002022-11-20T18:17:19.893+01:00Indexing with RedisIf you follow my news on Twitter then you might have realized that I just started to work more with Redis. Redis (=Remote Dictionary Server) is known as a Data Structure Store. This means that we can not just deal with Key-Value Pairs (called Strings in Redis) but in addition with data structures as Hashes (Hash-Maps), Lists, Sets or Sorted Sets. Further details about data structures can be found here:<br />
<div>
<ul>
<li><a href="https://redis.io/topics/data-types-intro">https://redis.io/topics/data-types-intro</a></li>
</ul>
<h3>
</h3>
<h3>
</h3>
<h3>
<br /></h3>
<h3>
Indexing in Key-Value Stores</h3>
<div>
With a pure Key-Value Store, you would typically maintain your index structures manually by applying some KV-Store patterns. Here some examples:</div>
<div>
<br /></div>
<div>
<ul>
<li><b>Direct access via the primary key:</b> The key itself is semantically meaningful and so you can access a value directly by knowing how the key is structured (by using key patterns). An example would be to access an user profile by knowing the user's id. The key looks like 'user::<uid>'.</li>
<li><b>Exact match by a secondary key:</b> The KV-Store itself can be seen as a huge Hash-Map, which means that you can use lookup items in order to reference other ones. This gives you a kind of Hash Index. An example would be to find a user by his email address. The lookup item has the key 'email::<email_addr>', whereby the value is the key of the user. In order to fetch the user with a specific email address you just need to do a Get operation on the key with the email prefix and then another one on the key with the user prefix. </li>
<li><b>Range by a secondary key:</b> This is where it is getting a bit more complicated with pure KV-Stores. Most of them allow you to retrieve a list of all keys, but doing a full 'key space scan' is not efficient (complexity of O(n), n=number of keys). You can indeed build your own tree structure by storing lists as values and by referencing between them, but maintaining these search trees on the application side is really not what you usually want to do.</li>
</ul>
<h3>
</h3>
<h3>
</h3>
<h3>
<br /></h3>
<h3>
The Redis Way</h3>
<div>
So how is Redis addressing these examples? We are leveraging the power of data structures as Hashes and Sorted Sets.</div>
<div>
<br /></div>
<h4>
Direct Access via the Primary Key</h4>
<div>
A Get operation already has a complexity of O(1). This is the same for Redis.</div>
</div>
</div>
<div>
<br /></div>
<h4>
Exact Match by a Secondary Key</h4>
<div>
Hashes (as the name already indicates) can be directly used to build a hash index in order to support exact match 'queries'. The complexity of accessing an entry in a Redis Hash is indeed O(1). Here an example:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/nosqlgeek/aaad48777dd922ba560d219e86b11ab0.js"></script>
</div>
<div>
In addition Redis Hashes are supporting operations as HSCAN. This provides you a cursor based approach to scan hashes. Further information can be found here:<br />
<br />
<ul>
<li><a href="https://redis.io/commands/scan">https://redis.io/commands/scan</a></li>
</ul>
<br />
Here an example:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/630bfa65e4b1b54c855760da3816beb1.js"></script>
<br />
<h4>
Range By a Secondary Key</h4>
</div>
<div>
Sorted Sets can be used to support range 'queries'. The way how this works is that we use the value for which we are searching as the score (order number). To scan such a Sorted Set has then a complexity of O(log(n)+m) whereby n is the number of elements in the set and m is the result set size.<br />
<br />
Here an example:<br />
<br />
<script src="https://gist.github.com/nosqlgeek/3d78cb196f1fa976a45dfc17f3b07628.js"></script></div>
If you add 2 elements with the same score then they are sorted lexicographically. This is interesting for non-numeric values. The command ZRANGEBYLEX allows you to perform range 'queries' by taking the lexicographic order into account.<br />
<h3>
</h3>
<h3>
<br /></h3>
<h3>
Modules</h3>
<div>
Redis supports now Modules (since v4.0). Modules are allowing you to extend Redis' functionality. One module which perfectly matches the topic of this blog post is RediSearch. RediSearch is basically providing Full Text Indexing and Searching capabilities to Redis. It uses an Inverted Index behind the scenes. Further details about RediSearch can be found here:</div>
<div>
<ul>
<li> <a href="http://redisearch.io/">http://redisearch.io</a></li>
</ul>
</div>
<div>
<br /></div>
<div>
Here a very basic example from the RediSearch documentation:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/nosqlgeek/5bac6f740a4fc22e80e578022b8d43a8.js"></script></div>
<div>
<br />
As usual, I hope that you found this article useful and informative. Feedback is very welcome!</div>
<div>
<br /></div>
<div>
<br /></div>
@nosqlgeekhttp://www.blogger.com/profile/12422139674678796017noreply@blogger.comtag:blogger.com,1999:blog-2779610621928360824.post-85884176034797865612017-04-06T16:44:00.000+02:002017-04-06T16:58:27.474+02:00Kafka Connect with Couchbase<h3>
About Kafka</h3>
Apache Kafka is a distributed persistent message queuing system. It is used in order to realize publish-subscribe use cases, process streams of data in real-time and store a stream of data safely in a distributed replicated cluster. That said Apache Kafka is not a database system but can stream data from a database system in near-real-time. The data is represented as a message stream with Kafka. Producers put messages in a so called message topic and Consumers take messages out of it for further processing. There is a variety of connectors available. A short introduction to Kafka can be found here: <a href="https://www.youtube.com/watch?v=fFPVwYKUTHs">https://www.youtube.com/watch?v=fFPVwYKUTHs</a> . This video explains the basic concepts and how Producers and Consumers are looking like. However, Couchbase supports 'Kafka Connect' since version 3.1 of it's connector. The Kafka documentation says "Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka.". Kafka provides a common framework for Kafka connectors. It can run in a distributed or standalone mode and it distributed and scalable by default.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-XEGdUQctXgM/WOX16ogXpkI/AAAAAAAAFiY/8NM09HtXkS8zndypaQhYpuBMvz6Qk1WGgCLcB/s1600/kafka-cb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="https://3.bp.blogspot.com/-XEGdUQctXgM/WOX16ogXpkI/AAAAAAAAFiY/8NM09HtXkS8zndypaQhYpuBMvz6Qk1WGgCLcB/s320/kafka-cb.png" width="320" /></a></div>
<h3>
Setup</h3>
Kafka uses Apache Zookeeper. Zookeeper is a cluster management service. The documentation states that "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications ... ZooKeeper aims at distilling the essence of these different services into a very simple interface to a centralized coordination service."<br />
<br />
After downloading and extracting the standard distribution of Apache Kafka, you can start a local Zookeeper instance by using the default configuration the following way:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/08a18a4cd4297602f89821f17d260e34.js"></script>
<br />
The next step is to configure 3 Kafka message broker nodes. We will run these services just on the same host for demoing purposes but it's obvious that they can also run more distributed. In order to so we need to create configurations for the broker servers. So copy the config/server.properties file to server-1.properties and server-2.properties and then edit it. The file 'sever.properties' has the following settings:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/35c030d1fd21c9a29c48d8f1d078742d.js"></script>
<br />
Let's assume that $i is the id of the broker. So the first broker has id '0', listens on port 9092 and logs to 'kafka-logs-0'. The second broker has the id '1', listens on port 9093 and logs to 'kafka-logs-1'. The third broker configuration is self-explaining.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/2a6f7918125ccbc1d3d8cca2d4e9f23d.js"></script>
<br />
The next step is to download and install and Couchbase Plug-in. Just copy the related libraries to the libs sub-folder and the configuration files to the config sub-folder of your Kafka installation.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/06dd1d05071dcb10715bf3bdc5cd70f3.js"></script>
<br />
<h3>
Streaming data from Couchbase</h3>
Before we can stream data from Couchbase we need to create a topic to which we want to stream to. So let's create a topic which is named 'test-cb'.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/6eb32793d8d3dffb62190696d54f4b7c.js"></script>
<br />
You can then describe this topic by using the following command:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/e51c0c52d5a5798aaea2c4be15a8a877.js"></script>
<br />
The topic which we created has 3 partitions. Each node is the leader for 1 partition. The leader is the node responsible for all reads and writes for the given partition. The Replicas is the list of nodes that replicate the log for this partition.<br />
<br />
Now let's create a configuration file for distributed workers under 'config/couchbase-distributed.properties':<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/f5110d183fae0b558b59668d3bec21db.js"></script>
<br />
The Connect settings are more or less the default ones. Now we also have to provide the connector settings. If using the distributed mode then the settings have to be provided by registering the connector via the Connect REST service:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/365d3a05b1f8d690fac4c650dd917a97.js"></script>
<br />
The configuration file 'couchbase-distributed.json' has a name attribute and an embedded object with the configuration settings:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/a1626d6408b6e893e95f2d0fcd6ef973.js"></script>
<br />
The Couchbase settings refer to a Couchbase bucket and the topic name to which we want to stream DCP messages out of Couchbase. In order to run the Connect workers in distributed mode, we can now execute:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/7260f213d28487ed1faa567ceea6e6ec.js"></script>
<br />
The log file contains information about the tasks. We configured 2 tasks to run. The output contains the information which task is responsible for which Couchbase shards (vBuckets):<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/cc6b1dc147f6dd927cf9c35976e7636d.js"></script>
<br />
For now let's just consume the 'test-cb' messages by using a console logging consumer:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/f00100152939e37eb1124a6cf834f8c2.js"></script>
<br />
One entry looks as the following one:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/80b78d601b67f98996c4ca0fe07decbf.js"></script>
<br />
We just used the standard value converter. The value is in reality a JSON document but represented as Base64 encoded string in this case.<br />
<br />
Another article will explain how to use Couchbase via Kafka Connect as the sink for messages.<br />David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-84372011341051709332016-09-12T20:29:00.002+02:002016-09-13T23:56:34.599+02:00Visualizing time series data from Couchbase with GrafanaGrafana is a quite popular tool for querying and visualizing time series data and metrics. If you follow my blog then you might have seen my earlier post about how to use Couchbase Server for managing time series data:<br />
<br />
<br />
<ul>
<li><a href="http://nosqlgeek.blogspot.de/2016/08/time-series-data-management-with.html">http://nosqlgeek.blogspot.de/2016/08/time-series-data-management-with.html</a></li>
</ul>
<br />
<br />
This blog is now about extending this idea by providing a Grafana Couchbase plug-in for visualizing purposes.<br />
<br />
After you installed Grafana (I installed it on Ubuntu, but there are installation guides available <a href="http://docs.grafana.org/installation/" target="_blank">here</a> for several platforms), you are asked to configure a data source. Before we will use Grafana's 'SimpleJson' data source, it's relevant how the backend of such a data source looks like.<br />
<br />
<br />
<ul>
<li><b>'/'</b>: Returns any successful response in order to test if the data source is available</li>
<li><b>'/search</b>': Returns the available metrics. We will just return 'dax' in our example.</li>
<li><b>'/annotations'</b>: Returns an array of annotations. Such an annotation has a title, a time where it would occur, a text and a tag. We just return an empty array in our example. But you can easily see that it would be possible to create an annotation if a specific value is exceeded or a specific time is reached.</li>
<li><b>'/query'</b>: The request is containing a time range and a target metric. The result is an array which has an entry for every target metric and each of these entries has an array of data points. Each data point is a tuple of the metric value and the time stamp.</li>
</ul>
<div>
<br /></div>
<div>
We will just extend our example from before with an Grafana endpoint and then point Grafana's generic JSON data source plug-in to it, but I can already see a project on the horizon which standardizes the time series management in Couchbase via a standard REST service which can then be used by a dedicated Grafana Couchbase plug-in.<br />
<br />
First let's look at our backend implementation:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/dcbe915285cfb45659a67b2675ff7ce3.js"></script><br />
<br />
<br />
As usual, the full code can be found here: <a href="https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/grafana.js">https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/grafana.js</a><br />
<br />
Here how we implemented the backend:<br />
<br />
<ul>
<li><b>'/'</b>: As you can see we just return a 'success:true' if the backend is accessible.</li>
<li><b>'/search'</b>: The only metric which our backend provides is the 'dax' one. </li>
<li><b>'/annotations'</b>: Only an example annotation is returned in this case. </li>
<li><b>'/query'</b>: We just check if the requested metric is the 'dax' one. In this first example, we don't take the aggregation documents into account. Instead we just request the relevant data points by using a multi-get based on the time range. Because Grafana expects the datapoints in time order, we have to finally sort them by time. Again, this code will be extended in order to take the several aggregation levels into account (Year->Month->Day->Hour).</li>
</ul>
<br />
<br />
Now back to Grafana! Let's assume that you successfully installed the 'SimpleJson' data source:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-dMTCAD68n7I/V9bv-l02OOI/AAAAAAAAFWE/5i0zViXcW1sHKRgrcEEE6-5-dtkx9MqPQCLcB/s1600/Bildschirmfoto%2B2016-09-12%2Bum%2B19.11.19.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="284" src="https://1.bp.blogspot.com/-dMTCAD68n7I/V9bv-l02OOI/AAAAAAAAFWE/5i0zViXcW1sHKRgrcEEE6-5-dtkx9MqPQCLcB/s320/Bildschirmfoto%2B2016-09-12%2Bum%2B19.11.19.png" width="320" /></a></div>
<br />
Then the only thing you need to do is to add a new data source to Grafana by pointing to our backend service (To run the backend service, just execute 'node app.js' after you checked out the <a href="https://github.com/dmaier-couchbase/cb-ts-demo" target="_blank">full repository</a> and installed all necessary dependencies.):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-4tuq89Je8LA/V9bzM01xCuI/AAAAAAAAFWQ/wyV1MWRmtdEJjzi9DznbWqHi0i1OcBjDQCLcB/s1600/Bildschirmfoto%2B2016-09-12%2Bum%2B19.12.58.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://4.bp.blogspot.com/-4tuq89Je8LA/V9bzM01xCuI/AAAAAAAAFWQ/wyV1MWRmtdEJjzi9DznbWqHi0i1OcBjDQCLcB/s320/Bildschirmfoto%2B2016-09-12%2Bum%2B19.12.58.png" width="320" /></a></div>
<br />
In this example I actually, just loaded a bit of random data for testing purposes by using the <a href="http://demo_data.js/">demo_data.js</a> script.<br />
<br />
Then all you have to do is to create a Dashboard an place a panel on it:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-UGn_ouc8rg8/V9bz7jkFHpI/AAAAAAAAFWU/l0zhDKdo0IQ1K0aaZFv_PZqrExLZpLx0ACLcB/s1600/Bildschirmfoto%2B2016-09-12%2Bum%2B19.28.46.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="146" src="https://1.bp.blogspot.com/-UGn_ouc8rg8/V9bz7jkFHpI/AAAAAAAAFWU/l0zhDKdo0IQ1K0aaZFv_PZqrExLZpLx0ACLcB/s320/Bildschirmfoto%2B2016-09-12%2Bum%2B19.28.46.png" width="320" /></a></div>
<br />
<br />
The rest should work more or less the same as with any other Grafana data source. :-)</div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-935612181382586792016-08-26T09:27:00.001+02:002016-08-26T11:20:00.890+02:00Time series data management with Couchbase ServerCouchbase Server is a Key Value store and Document database. The combination of being able to store time series entries as KV pairs with the possibilities to aggregate data automatically in the background via Map-Reduce and the possibility to dynamically query the data via the query language N1QL makes Couchbase Server a perfect fit for time series management use cases.<br />
<div>
<br /></div>
<div>
The high transaction volume seen in time series use cases is meaning that relational database systems are often not a good fit. A single Couchbase Cluster on the other hand side might support hundreds of thousands (up to millions) of operations per second (indeed dependent on the node and cluster size).</div>
<div>
<br /></div>
<div>
Time series use cases seen with Couchbase are for instance:<br />
<div>
<ul>
<li><b>Activity tracking</b>: Track the activity of a user whereby each data point is a vector of activity measurement values (e.g location, ...)</li>
<li><b>Internet of things:</b> Frequently gathering data points of internet connected devices (as cars, alarm systems, home automation devices, ...), storing them as a time series and aggregate them in order monitor and analyse the device behavior</li>
<li><b>Financial</b>: Store currency or stock courses as time series in order to analyse (e.g. predictive analysis) based on this data. A course diagram is typically showing a time series.</li>
<li><b>Industrial Manufacturing</b>: Getting measurement values from machine sensors in order to analyse the quality of parts.</li>
</ul>
<div>
<div>
<div>
<br /></div>
<div>
But before we start digging deeper into an example, let's talk a bit about the background of time series data management:</div>
</div>
</div>
</div>
</div>
<div>
<br /></div>
<div>
A time series is a series of data points in time order. So mathematically spoken a time series is expressed as a diskrete function with (simplified) two dimensions. The first dimension (x-axis) is the time. The second dimension (y-axis) is the data point value, whereby a data point value can be again a vector (which makes it actually 1+n dimensional, whereby n is the vector size). Most commonly the values on the time-axis are on an equidistant grid, which means that the distance between any x values x_1 and x_2 is equal.</div>
<div>
<br /></div>
<div>
So what to do with such a time series?</div>
<div>
<ul>
<li><b>Analyse the past</b>: Statistics, reporting, ...</li>
<li><b>Real-time analysis</b>: Monitor current activities, find anomalies, ...</li>
<li><b>Predictive analysis</b>: Forecast, estimate, extrapolate, classify, ...</li>
</ul>
<div>
<br /></div>
</div>
<div>
Good, time to look at an example. First we need a data source which is frequently providing changing data. Such data could be financial courses, a sensor measurement, a human heart beat and so on.</div>
<div>
<br /></div>
<div>
Let's take a financial course. Google is providing such information via 'Google Finance'. So in order to get the current course of the DAX (this might tell you where I am living ;-) ), you just have to open up <a href="https://www.google.com/finance?q=INDEXDB%3A.DAX">https://www.google.com/finance?q=INDEXDB%3A.DAX</a>. In order to get the same information as JSON you can just use <a href="https://www.google.com/finance/info?q=INDEXDB%3ADAX">https://www.google.com/finance/info?q=INDEXDB%3ADAX</a> .</div>
<div>
<br /></div>
<div>
What we get by accessing this API is:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/dmaier-couchbase/41a52c31e916debe560d2b7c9fd89fa4.js"></script>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
So far so good. Now let's write a litte Node.js application (by using <a href="http://www.ceanjs.org/">http://www.ceanjs.org</a>) which is polling every minute for the current course and then writes it into Couchbase. To be more accurate: we actually fetch every 30 seconds in order to reach the granularity of a minute. In this example we decided for the minute granularity but it would work in a similar way for an e.g. seconds granularity. We also just expect that the last fetched value for a minute is the minute value. An even more sophisticated approach would be to store the max. 2 gathered values in an array in our minutes document and already aggregate on those two (avg as the minute value instead the last one). It's a question of accuracy. The key of such a data point is indeed dependent on the time stamp. We are just interested in the course value 'l', the difference 'c' and the time stamp 'lt_dts'. The job logic then looks as the following one:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/a5b349c897e5ae9cc50f8ec1080daa01.js"></script><br />
<br />
BTW: The full source code can be found here: <a href="https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/course_retrieval_job.js">https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/course_retrieval_job.js</a><br />
<br />
This looks then as the following in Couchbase.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-WCQcXIInQlo/V77MruGoJqI/AAAAAAAAFUg/3o5X2gufkoIBGTAKDQYEWbnXRSQ42ys2gCLcB/s1600/Bildschirmfoto%2Bvom%2B2016-08-25%2B12%253A46%253A45.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="77" src="https://1.bp.blogspot.com/-WCQcXIInQlo/V77MruGoJqI/AAAAAAAAFUg/3o5X2gufkoIBGTAKDQYEWbnXRSQ42ys2gCLcB/s400/Bildschirmfoto%2Bvom%2B2016-08-25%2B12%253A46%253A45.png" width="400" /></a></div>
<br />
<br />
Fine, so what's next? Let's start with direct access to time series values. In order to fetch all values for a given range, you don't need any index structure because:<br />
<br />
<ul>
<li>The discrete time value is part of the key. So our time-axis is directly expressed via the key space.</li>
</ul>
It's also easy to see that JSON document value is more or less a vector (as defined above)<br />
<br />
So let's write a little service which takes a start time stamp and an end time stamp as a parameter in order to provide you all the requested values.<br />
<br />
The service code could look like this:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/4c9351a9d2cc3691b51ed1e44d017948.js"></script><br />
<br />
<br />
<br />
The full code can be found here: <a href="https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/by_time.js">https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/by_time.js</a><br />
<br />
It just takes the start and end time in the format following format:<br />
<br />
<br />
<ul>
<li>http://localhost:9000/service/by_time?start=2016-08-25T13:15&end=2016-08-25T13:20 </li>
</ul>
<br />
The output looks as the following one:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/2880ff18f2b14545639d130d603464f8.js"></script><br />
<br />
<br />
Let's next calculate some statistics based on these values. Therefore we will create some aggregate documents. As you might already imagine, we will aggregate based on the time. The resulting time dimension for these aggregates will be 'Year -> Month -> Day -> Hour'. So their will be:<br />
<br />
<ul>
<li><b>An hour aggregate:</b> It aggregates based on the minutes time series. There are 60 minutes per hour to aggregate. </li>
<li><b>An day aggregate:</b> It aggregates based on the hour aggregates. There are 24 hours per day.</li>
<li><b>A month aggregate:</b> It aggregates based on the day aggregates. There are between 28 and 31 days per month.</li>
<li><b>A year aggregate:</b> It aggregates based on the month aggregates. There 12 months per year.</li>
</ul>
<div>
I guess you got it :-) ...</div>
<div>
<br /></div>
<div>
So how to build these aggregates? There are multiple ways to do it. Here just some of them:</div>
<div>
<ul>
<li>Use the built-in views and write the view results for a specific time range back to Couchbase</li>
<li>Execute a N1QL query by using aggregate functions</li>
<li>Do the calculations on the client side by fetching the data and write the results back</li>
<li>Load or stream the data into Spark in order to do the necessary calculations there and write the results back to Couchbase</li>
</ul>
</div>
<br />
Let's have a look at Views first. Views provide built-in map-reduce. We want to calculate the following statistic values:<br />
<br />
<ul>
<li>The average value of the course</li>
<li>The maximum value of the course</li>
<li>The minimum value of the course</li>
</ul>
<div>
We will just create one View for this. The following map and reduce functions are created on the Couchbase Server side:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-d-eVfAry-Tc/V79hnYwQ1VI/AAAAAAAAFUw/vOQ0O-pmqDgh9LthVxqZgyC7q-v8N3THwCLcB/s1600/Bildschirmfoto%2Bvom%2B2016-08-25%2B23%253A22%253A19.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="140" src="https://4.bp.blogspot.com/-d-eVfAry-Tc/V79hnYwQ1VI/AAAAAAAAFUw/vOQ0O-pmqDgh9LthVxqZgyC7q-v8N3THwCLcB/s400/Bildschirmfoto%2Bvom%2B2016-08-25%2B23%253A22%253A19.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The request parameters for aggregating directly for one hour are looking like:</div>
<div>
<ul>
<li><a href="http://localhost:8092/ts/_design/time/_view/stats?stale=false&startkey=[2016,8,25,14,0,0]&endkey=[2016,8,25,15,0,0]&inclusive_end=false&reduce=true&group_level=4">http://localhost:8092/ts/_design/time/_view/stats?stale=false&startkey=[2016,8,25,14,0,0]&endkey=[2016,8,25,15,0,0]&inclusive_end=false&reduce=true&group_level=4</a></li>
</ul>
<div>
The result is the hour aggregation based on the time series data points:</div>
</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/dmaier-couchbase/a1bfc746e70555f2de34a6555af939bf.js"></script></div>
<div>
<br /></div>
<div>
It's easy to see that it also allows us to directly access the time function which has the hour (and no longer the minute) as the distance on time axis. The data points are then the aggregation values. The same View can be used to get the monthly and the yearly aggregation values. The trick is to set the range parameters and the group level in the right way. In the example above 'group_level=4' was used because the hour information is at the fourth position of the date array which was emitted as the search key. In order to get the daily aggregation, just use a query like this:<br />
<br />
<ul>
<li><a href="http://localhost:8092/ts/_design/time/_view/stats?stale=false&startkey=[2016,8,25,0,0,0]&endkey=[2016,8,26,0,0,0]&inclusive_end=false&reduce=true&group_level=3">http://localhost:8092/ts/_design/time/_view/stats?stale=false&startkey=[2016,8,25,0,0,0]&endkey=[2016,8,26,0,0,0]&inclusive_end=false&reduce=true&group_level=3</a></li>
</ul>
<br />
<br />
Now let's create an aggregation service which is using this View result in order to return the aggregation for a specific hour. It queries the aggregate for a given hour and stores the aggregation result as an aggregate document if the hour is already a full hour (so if it has 60 data points). In reality you could also run a job in order to make sure that the aggregates are built upfront. In this demo application we just build them at access time. The next time they will be not accessed from the View, but directly from the KV store.<br />
<br />
Following the code of the service:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/b2b18f341f62241cd0fb20665c337f59.js"></script></div>
<div>
<br /></div>
<div>
<br />
The full code can be found here: <a href="https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/agg_by_hour.js">https://github.com/dmaier-couchbase/cb-ts-demo/blob/master/routes/agg_by_hour.js</a><br />
<br />
The result in Couchbase would be then:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-6gQt0xx9Qu4/V7_rdfRtI5I/AAAAAAAAFVA/YN_c3eOKfhE9lGCidTqn_upYPDEbBvu8wCLcB/s1600/Bildschirmfoto%2Bvom%2B2016-08-26%2B08%253A59%253A47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="108" src="https://3.bp.blogspot.com/-6gQt0xx9Qu4/V7_rdfRtI5I/AAAAAAAAFVA/YN_c3eOKfhE9lGCidTqn_upYPDEbBvu8wCLcB/s320/Bildschirmfoto%2Bvom%2B2016-08-26%2B08%253A59%253A47.png" width="320" /></a></div>
<br />
<br />
Their might be the question in your head 'What's if I want to aggregate by taking a specific aggregation level into account, but also need to have the last minutes (highest granularity in our example) into account?'. The answer is to combine the approaches of accessing the minute data points directly and the lower granularity aggregates. Here an example: If you want to access everything since 14:00 until 15:02, whereby 15:00 is not yet a full hour, then you can do this by using the following formula.<br />
<br />
<ul>
<li> Agg(14:00) + Agg(t_15:00, t_15:01, t_15:02)</li>
</ul>
<br />
It's easy to see that you can derive additional formulas for other scenarios.<br />
<br />
A related question is how long you should keep the highest granularity values. One year has 525600 minutes. And so we would get every year 525600 minute documents. So for this use case we could decide to remove the minute documents (Couchbase even comes with a TTL feature in order to let them expire automatically) because it's unlikely the case that someone is interested in more than the daily course after one year. How long you keep the finest granularity data points indeed depends on your requirement and how fine your finest granularity actually is.<br />
<br />
OK, so this blog article is already getting quite long. Another one will follow which then will cover the following topics:<br />
<br />
<br />
<ul>
<li>Visualizing time series data</li>
<li>How query time series data with N1QL</li>
<div>
</div>
<li>Predictive analysis of time series data with Couchbase and Apache Spark</li>
</ul>
</div>
</div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-20735078540977826592016-07-01T13:47:00.003+02:002016-07-01T14:00:10.199+02:00Caching in JavaEE with Couchbase<div>
One of Couchbase Server's typical use cases is caching. As you might know it is a KV store. The value of a KV pair can be JSON document. Not only the fact that Couchbase Server can store JSON documents makes it a document database, more the fact that you can index and query on JSON data defines it's characteristic as a JSON document database. Back to the KV store: If you you configure the built-in managed cache in a way that all your data is fitting into memory then Couchbase Server is used as a highly available distributed cache.</div>
<div>
<br /></div>
If you are a Java developer, then one of your questions might be if it makes sense to use Couchbase as a cache for your applications. I had several projects, where EhCache was replaced by Couchbase because of the Garbage Collection implications. The performance was often quite better with a centralized, low-latency (sub-milliseconds) cache than with one which was colocated with the application instances. This indeed depends on several factors (size of the cache entries, number of cache entries, access throughput). The next questions might be how to best integrate such a cache into your application. A typical pattern is:<br />
<div>
<ul>
<li>Try to read the data from the cache</li>
<li>If it is there, then use it</li>
<li>If is not there then get the data from the source system (e.g. relational DBMS)</li>
<li>Put it into the cache</li>
<li>The next time when you try to access the same data, then it will be most probably in the cache</li>
</ul>
<div>
Couchbase's Java SDK is quite simple for CRUD operations:</div>
</div>
<div>
<ul>
<li>C: Insert</li>
<li>R: Get</li>
<li>U: Update, Replace</li>
<li>D: Remove</li>
</ul>
<div>
So as soon as you established a Bucket (a data container) connection, you can use it as a cache. However, this is involving implementation work on your side. </div>
</div>
<div>
<br /></div>
<div>
I just looked at the Java standard JCache and also used the chance to play a bit around with CDI (Dependency Injection). JCache is implemented by several providers and look at that there is already a Developer Preview of a Couchbase implementation available (http://blog.couchbase.com/jcache-dp2).</div>
<div>
<br /></div>
<div>
<i><b>Side note:</b> The Couchbase JCache implementation is not yet officially released. Couchbase has also a good Spring Data integration which also comes with cashing support.</i></div>
<div>
<br /></div>
<div>
So let's get started. First we need to have a cache instance which we can use for caching purposes.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/dmaier-couchbase/4173d770862c28969eea144594ee59e3.js"></script>
</div>
<div>
<br /></div>
<div>
As you can see, we are creating a CacheProvider, then a Config and finally a CacheManager in order to access the Cache. Our cache is an object cache, whereby objects are stored by a string key. The Factory ensures that we have only a single instance of our ObjectCache. It's not using CDI and so can be also used with JavaSE. In a real world you would probably not use constants for the factory configuration, but it seemed to be sufficient for this example.<br />
<br />
Now let's use the factory. Actually we misuse it a bit here because we use it in a Producer. In a pure CDI world, you would just use the code for initializing the cache in the producer method. So the producer is actually your factory, whereby the producer method acts as a source of objects to be injected. The annotation 'CBObjectCache' was bound to the producer.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/c05989d9732277f0c2e6cb13a6609869.js"></script><br />
<br />
Now that we have a producer, we can just inject CBObjectCache somewhere else. Let's do this in an Interceptor. We will use this interceptor later in order to cache objects automatically when a method is called. The annotation 'Cached' is bound to our interceptor.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/b2c0edb868b810bfc0a568a9933d2dc5.js"></script><br />
<br />
Now in order to use our interceptor, we just have to annotate a method which should cache the passed data. The example below shows that the 'createHelloMessage' is annotated with 'Cached'. So before the actual method code is executed, the value of the variable 'name' will be cached in Couchbase. In order proof this, the value is fetched in the method body again to be printed out by the 'HelloWorldServlet'.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/b1d3f94a18b3d3d7e93780e52f317bdd.js"></script><br />
<br />
Before I forget it, here how it is looking like in Couchbase:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-bR1GKdbNMQs/V3ZbGnrYWjI/AAAAAAAAFPw/_7vDZZwp7I8xmclewrALMg9Mjm2LgHatQCLcB/s1600/Bildschirmfoto%2B2016-07-01%2Bum%2B13.58.30.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="72" src="https://4.bp.blogspot.com/-bR1GKdbNMQs/V3ZbGnrYWjI/AAAAAAAAFPw/_7vDZZwp7I8xmclewrALMg9Mjm2LgHatQCLcB/s320/Bildschirmfoto%2B2016-07-01%2Bum%2B13.58.30.png" width="320" /></a></div>
<br />
Hope this small introduction to JCache and CDI was interesting for you. :-)<br />
<br />
The full source code can be found here: <a href="https://github.com/dmaier-couchbase/cb-jboss/tree/master/hello-jcache-cdi">https://github.com/dmaier-couchbase/cb-jboss/tree/master/hello-jcache-cdi</a> .</div>
<div>
<br /></div>
<div>
<br /></div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-53288686556056473452016-06-01T11:07:00.000+02:002016-06-01T11:07:14.549+02:00How to build Couchbase ServerCouchbase Server is Open Source under Apache2 license and even if an user would normally not build it from the source code (in fact the custom built versions are not officially supported by Couchbase), you might want to participate in the Couchbase Community by providing some lines of code. The first thing you need is to be able to build Couchbase Server from the source code.<br />
<br />
The Couchbase Server source code is not just in one repository. Instead it is spread over multiple Git repositories. A tool which can be used in order to abstract the access to these multiple Git repositories is 'repo'. So 'repo' is a repository management tool on top of Git. It's also by Google for Android and so a short documentation can be found here: <a href="https://source.android.com/source/using-repo.html">https://source.android.com/source/using-repo.html</a> . The installation instructions are available at <a href="http://source.android.com/source/downloading.html#installing-repo">http://source.android.com/source/downloading.html#installing-repo</a> .<br />
<br />
Here some 'repo' commands:<br />
<ul>
<li><b>repo init</b>: Installs the repository to the current directory</li>
<li><b>repo sync</b>: Downloads the new changes and updates the working files in the local directory</li>
<li><b>repo start</b>: Begins a new branch for development, starting from the revision specified in the manifest</li>
</ul>
Repo is using manifest files. The Couchbase manifest files can be found here: <a href="https://github.com/couchbase/manifest">https://github.com/couchbase/manifest</a> . Let's take a look into one of these files (e.g. /released/4.5.0-beta.xml):<br />
<br />
<pre class="prettyprint" style="background: rgb(247, 247, 247); border: 1px solid rgb(221, 221, 221); font-stretch: normal; margin-bottom: 1em; overflow: auto; padding: 1em;"><span style="line-height: 14px;"><remote name="couchbase" fetch="git://github.com/couchbase/" review="review.couchbase.org" /></span>
...
<default remote="couchbase" revision="master" />
<project name="bleve" remote="blevesearch" revision="760057afb67ba9d8d7ad52f49a87f2bf9d31a945" path="godeps/src/github.com/blevesearch/bleve"/>
...</pre>
<br />
As you can see, the manifest includes the Git repos those are containing Couchbase dependencies. By default the master branch was referenced here. Each dependency can be provided with a specific Git Hash or branch name in order to make sure that you build based on the right version of the dependent library.<br />
<br />
Before we build it's required to have at least make and cmake installed on your build box. If not the build will fail by telling you what's required. I already had a C development environment, python and Go installed on my computer. The build of Couchbase is actually quite simple:<br />
<br />
<pre class="prettyprint" style="background: rgb(247, 247, 247); border: 1px solid rgb(221, 221, 221); font-stretch: normal; margin-bottom: 1em; overflow: auto; padding: 1em;">cd --
mkdir -p src/couchbase
cd src/couchbase
repo init -u git://github.com/couchbase/manifest.git -m <branch_name>
repo sync
make
</branch_name></pre>
<br />
The built version of Couchbase is then available in the sub-folder 'install'.<br />
<br />David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-41031517823735648012016-05-13T13:12:00.001+02:002016-05-13T20:33:54.882+02:00Couchbase Server 4.5's new Sub-Document API<h3>
Introduction</h3>
The Beta version of Couchbase Server 4.5 has just been released, so let's try it out! A complete overview of all the great new features can be found here: <a href="http://developer.couchbase.com/documentation/server/4.5/introduction/intro.html">http://developer.couchbase.com/documentation/server/4.5/introduction/intro.html</a>. This article will highlight the new Sub-Document API feature.<br />
<br />
What's a sub-document? The following document contains a sub-document which is accessible via the field 'tags':<br />
<div>
</div>
<br />
<script src="https://gist.github.com/dmaier-couchbase/d5edc01ab0a3d41b17d7248ee54cd277.js"></script><br />
<h3>
So far</h3>
With earlier Couchbase versions (<4.5) the update of a document had to follow the following pattern:<br />
<br />
<ul>
<li>Get the whole document which needs to be updated</li>
<li>Update the documents on the client side (e.g. by only updating a few properties)</li>
<li>Write the whole document back</li>
</ul>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-cKS2iQ2eQ2g/VzWpH7A1y-I/AAAAAAAAFOY/2dMH7CuXHRUSijpAL4noT3ZMJlzq5_i4wCLcB/s1600/pingpong.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="301" src="https://4.bp.blogspot.com/-cKS2iQ2eQ2g/VzWpH7A1y-I/AAAAAAAAFOY/2dMH7CuXHRUSijpAL4noT3ZMJlzq5_i4wCLcB/s320/pingpong.png" width="320" /></a></div>
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
A simple Java code example would be:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/f174de40229aa5b07eeec18b3deccb35.js"></script>
<br />
<h3>
Now with 4.5</h3>
<div>
The new sub-document API is a server side feature which allows you to (surprise, surprise ...) only get or modify a sub-document of an existing document in Couchbase. The advantages are:<br />
<br />
<ul>
<li> Better usability on the client side</li>
<ul>
<li>CRUD operations can be performed based on paths</li>
<li>In cases where the modification doesn't rely on the previous value, you can update a document without the need to fetch it upfront</li>
<li>You can easier maintain key references between documents</li>
</ul>
<li> Improved performance</li>
<ul>
<li>It saves network bandwidth and has a improved latency because you don't need to transfer the whole document over the wire</li>
</ul>
</ul>
<br />
<br />
The sub-document API also allows you to get or modify inner values or arrays of a (sub-)document.<br />
<ul>
<li><b>Lookup operations:</b> Queries the document for a specific path, e.g. GET, EXISTS</li>
<li><b>Mutation operations:</b> Modify one or multiple paths in a document, e.g. UPSERT, ARRAY_APPEND, COUNTER</li>
</ul>
<div>
A more detailed description of the API can be found in the Couchbase documentation: <a href="http://developer.couchbase.com/documentation/server/4.5-dp/sub-doc-api.html">http://developer.couchbase.com/documentation/server/4.5-dp/sub-doc-api.html</a> .</div>
<div>
<br /></div>
</div>
<div>
The update of a document can now follow the following pattern:<br />
<br />
<ul>
<li>Update directly a property or subdocument by specifying the path under which it can be found</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-RQawqeSbz10/VzWoi_xkFpI/AAAAAAAAFOQ/mSBToE0krxAgJBPXscpsqxTvPY0kYLr_gCLcB/s1600/ping.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="284" src="https://4.bp.blogspot.com/-RQawqeSbz10/VzWoi_xkFpI/AAAAAAAAFOQ/mSBToE0krxAgJBPXscpsqxTvPY0kYLr_gCLcB/s320/ping.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Our Java example would now be simplified to:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/dmaier-couchbase/49afdbabfa83fd424afa31088f1d64b2.js"></script>
</div>
<div>
<br /></div>
<div>
<h3>
Optimistic "locking"</h3>
<div>
Couchbase Server does not have a built-in transaction manager, but if you talk about transactional behavior, the requirements are quite often less than what a ACID transaction manager would provide (e.g. handling just concurrent access instead of being fully ACID compliant). In Couchbase a document has a so called C(ompare) A(nd) S(wap) value. This value changes as soon as a document is modified on the server side.</div>
<div>
<ul>
<li>Get a document with a specific CAS value</li>
<li>Change the properties on the client side</li>
<li>Try to replace the document by passing the old CAS value. If the CAS value changed in between on the server side then you know that someone else modified the document in between and so you can retry to apply your changes.</li>
</ul>
<div>
So CAS is used for an optimistic locking approach. It's optimistic because you expect that you can apply your changes and you handle the case that this wasn't possible because someone else changed it before. A pessimistic approach would be to lock the document upfront and so no one else can write it until this lock will be released again.</div>
</div>
<div>
<br /></div>
<div>
You could now ask the following question:</div>
<div>
<ul>
<li>What happens if I modify a sub-document and someone else updates the same or another sub-document of the same document?</li>
</ul>
<div>
Sub-document operations are atomic. Atomicity means all or nothing. So if you update a sub-document by not retrieving an error message then you can be sure that the update was performed on the server side. This means if 5 clients are appending an element to an embedded array, then you can be sure that all 5 values were appended. However, atomicity isn't meaning consistency regarding the state. So it isn't telling you about conflicts. So if 2 clients are updating the same sub-document then both updates will be performed but in order to find out if their was a conflict regarding these updates you would still need the CAS value (or use pessimistic locking instead). However, if you are sure that the clients act on different sub-documents then you know that there will be no conflict and then the CAS value would be not required.</div>
</div>
<div>
<br />
<script src="https://gist.github.com/dmaier-couchbase/a2b5e4e757400b0bae3657a6b59e6a9d.js"></script>
</div>
<div>
<h3>
Summary</h3>
</div>
<div>
The new Sub-Document API is one of the new great features of Couchbase 4.5 (Beta). It allows you to avoid to fetch the whole document in order to read/modify only a part of it. This means a better usability from a client side point of view. One of the main advantages is that it improves the performance, especially if working with bigger documents.</div>
</div>
</div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-48880284192918626372016-05-06T13:35:00.002+02:002016-05-06T13:35:48.699+02:00Microservices and Polyglot Persistence<h3>
TOC</h3>
<ul>
<li>Introduction</li>
<li>Why Microservices?</li>
<li>Polyglot character</li>
<li>What's happening with my Database?</li>
<li>Summary</li>
</ul>
<div>
<h3>
Introduction</h3>
The idea behind Microservices is already described by it's name. In summary it means to use multiple smaller self-contained services to build up a system, instead of using one monolithic one. This explanation does sound simple, doesn't it? We will see that it is not because breaking up one single big system in several services has quite a lot implications.<br />
<br />
<h3>
Why Microservices?</h3>
A monolithic system would be a system which has only one main component. One of the disadvantages is usually that you have to deploy changes in a way that they affect the deployment of the whole system. A today's system is actually not completely monolithic at all, because it normally already consists of several sub-components. Often other decomposition mechanisms are already used. One way would be to build your system modular. Such a module might be actually a good candidate for a microservice, whereby it should optimally have business domain specific functionality and not a pure technical one.<br />
<br />
Another aspect, you should be already familiar with as an object oriented developer, is de-coupling (loose coupling). Actually one component should live in a way for it's own. Sure there are well defined dependencies to other components. De-coupling allows you to ensure that you can replace one component of your system without the need to rewrite the a majority system again.<br />
<br />
If splitting up a monolithic application into several parts, whereby specific functionalities are provided as services, you end up with a distributed system because each service is deployed by it's own. The idea is exactly to be able to scale these services independently out.<br />
<br />
So Microservices are in a way not a complete new invention. Microservices are often just a consequence of what we already know or target regarding software architectures. Service oriented designs are also not completely new for us.<br />
<br />
<h3>
Polyglot character</h3>
One system made of multiple smaller services:<br />
<br />
<ul>
<li><u style="font-weight: bold;">Can have a variety of communication protocols:</u> About 10 years ago, I remember to have discussions about SOAP vs. REST. I actually liked SOAP because it was well defined and so your service client could be created just by the service definition. It has message based communication and there was a kind of standard message format (dependent on the binding). REST on the other hand's side had the charm to be less chatty and resource based. The protocol how 2 parties are communicating (which resources are accessed in exactly which way) was not out of the box predefined. Indeed, you also define what a REST service exposes. But it seemed to happen more often that the service did no longer talk exactly the same language as the client. Actually, it was more like it was speaking partially a weird dialect which could no longer be understood by the client and so the client had to learn it as well. There are libs and frameworks those are helping you (e.g annotations in JAX-RS). However, I'm pretty sure that a today's green field solutions would rely on RESTFul services. Sometimes you don't come from a green field and so you might still need to integrate a variety of different kinds of services. </li>
<li><b><u>Can be implemented by using several programming languages and frameworks</u></b>: It's just relevant for another component of your system how to communicate with a specific service. The actual implementation is completely hidden from the other components of your system. So one service might be implemented in Node.js but another one might be implemented in Python. There are sometimes good reasons to develop one part of a system in e.g. C but others with maybe less effort e.g. in Node.js. Not every component might have the same resource and efficiency requirements (e.g. Garbage collection vs. manual disposal)</li>
<li><b><u>Can be developed by different kinds of developers:</u></b> This is indeed related to the different programming languages and frameworks point. From my personal experience I would say that a C and a Java developer are really speaking different languages. Not just regarding the programming language, also how a specific problem would be addressed or how the tool chains are looking like. There is no good or bad, it's just different. So given that different functionalities might be developed by multiple different and independent teams, this point especially makes sense if these teams already got skill sets around specific programming languages and frameworks. </li>
<li><u style="font-weight: bold;">Polyglot persistence:</u> A modern application consists usually of 3 tiers: interface, service and persistence. Given that we split the service tier up into multiple smaller services, there is the fair question what happens with your database/storage tier. We will discuss this a bit later in depth. Important is that the several services can use different database/storage back-ends. One service might need to write content items and stores the content itself in a Blob Store and the meta data in a Document Database. Another service might need to store the information who knows whom and so uses a Graph Database. A third service might handle user sessions and so uses a KV-store. This is what what polyglot persistence means.</li>
</ul>
<br />
<br />
<h3>
What's happening with my Database?</h3>
This is actually quite interesting. Even if your system used before several modules, you will quite often see that the modules integrate with each other on the database level. The reasons is that the rules for your schema consolidation (regarding the good database schema design) might conflict with the de-coupling requirements. At the end each independent service should use it's own database. Instead of integrating service functionality on the database tier, the services should talk with each other. Let's use a very simple and stupid example. We talk about orders and customers. Let's assume that your shopping service is independent from the user profile service. Sure, shopping needs to know who the customer is, but not at the same level of detail as the user profile service knows. In a monolithic application you would have a 1-many relationship between customers and orders. So getting all the orders of a customer would be JOIN query. In a more service oriented world, you would ask for the directory service for the customer (by e.g. his email address) and then you would ask the shopping service for all the orders of this customer. If e.g. a new order should be processed then the shopping service would also need to talk with the payment service in order to fulfill the payment. The payment service also only knows the relevant information about the customer and not the complete user profile. Again, a very simple example, but the point is clear. A Microservices approach leads to distributed system, made of several services, which leads to split up databases as well.<br />
<div>
<br /></div>
Now, the relational database was so far gluing your data focused operations together by talking about transactions. Doesn't the service based approach mean that I loose these transactions on the database tier? Exactly! Given that you decoupled your services by no longer integrating on the database tier, you know have also to take care about the transactions on the service level. Relational database systems vendors are talking since decades about ACID and you got the impression that you absolutely need it? From my experience it's quite often the case that you anyway give up on ACID for performance reasons (weaker isolation level - e.g. read uncommitted) and we tend to rely such a lot on the DB's transaction management (by accidentally tolerating it's overhead) that we forgot that we often don't need ACID but only handling concurrent access to specific data items. The NoSQL system Couchbase Server for instance, doesn't come with a built-in transaction manager, but it comes with a framework which helps you on the client side to handle transactional behavior. You can e.g. lock specific documents (JSON documents or KV-pairs) and so somebody else has to wait until it is released again. Or you can be more optimistic and use C(ompare)A(nd)S(wap). A write operation is then successful if the CAS value for your document is still the same. This means if nobody else did change the document since you have fetched it. Otherwise you can just try it again with the updated document until you are the winner. Sure, there are also strictly transactional cases out there. They can be addressed by using a service side transaction manager (e.g. implementing 2-phase commit).<br />
<br />
Not to use one single and big database is also a chance. We already talked about that you want to be able to scale your services out (adding new service instances behind a load balancer - so web scale) independently. Scaling out the service tier is only half of the story. More and more service instances might also raise the need to scale out on the storage/database tier. So instead doing all with your non-scalable relational DBMS, you can now follow the polyglot persistence idea and use the right database for the job, which means that you might introduce a highly scalable NoSQL database system for some of your service.<br />
<br />
<h3>
Summary</h3>
</div>
<div>
As explained Microservices are self-contained services those are providing business domain specific functionality. A system which uses Microservices is per definition a distributed system, with all it's advantages and disadvantages. Getting your system more scalable is easier possible, whereby distributed transactions are harder. Polyglot persistence is one benefit. You can now use the right storage or database system for the job, dependent on the requirements of the specific service. </div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-21927352109600536972016-03-26T09:38:00.004+01:002016-03-26T09:38:53.318+01:00CBGraph now supports edge list compression<b><u>About CBGraph</u></b><br />
<br />
CBGraph (<a href="https://github.com/dmaier-couchbase/cb-graph">https://github.com/dmaier-couchbase/cb-graph</a>) is a Graph API for the NoSQL database system Couchbase Server.<br />
<div class="p1">
<span class="s1"><br /></span>
<b><u>Adjacency list compression</u></b><br />
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">The latest version of CBGraph (v0.9.1) supports now adjacency list compression. An adjacency list is the list of neighbors of a vertex in a Graph.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
So far the adjacency lists were stored directly at the vertices but vertices can become quite big if they have a huge amount of incoming or outgoing edges (such a vertex is called a supernode). One of the limitations which such a supernode introduces is that it just takes longer to transfer a e.g. a 10MB vertex over the wire than e.g. a 1KB one. In order support such supernodes better by reducing the network latency, two optimization steps were introduced for CBGraph.</div>
<div class="p1">
</div>
<div class="p1">
</div>
<ol>
<li>Compress the adjacency list by still storing it at the vertex (as base64 string). The base64 encoding causes that the lists are taking a bit more space for small vertices but you save a lot (saw up to 50% with UUID-s as vertex id-s) for supernodes.</li>
<li>Externalize and compress the adjacency list as a binary</li>
</ol>
<div>
The used compression algorithm is GZIP.<br />
<br />
There are the following switches in the graph.properties file in order to controll the compression mode.</div>
<ul class="ul1">
<li class="li1"><span class="s1">graph.compression.enabled</span></li>
<li class="li1"><span class="s1">graph.compression.binary</span></li>
</ul>
<div>
The first property controls if compression is used. The second one controls if the compressed adjacency list is stored as a separated binary (As Couchbase Server is a KV store and a document database, you can directly store binaries as KV pairs).</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><u>Compression disabled</u></b></div>
<div>
<br /></div>
The following setting is used in order to disable compression:<br />
<ul>
<li>graph.compression.enabled=false</li>
</ul>
The document model which is used in Couchbase looks then as the following one:<br />
<br />
<div>
<script src="https://gist.github.com/dmaier-couchbase/9fc55fdb0da67f96dad9.js"></script>
</div>
<b><u>Compression enabled by embedding the adjacency list</u></b><br />
<b><u><br /></u></b>
The following setting is used in order to enable compression:<br />
<ul>
<li>graph.compression.enabled=true</li>
<li>graph.compression.enabled=false</li>
</ul>
<div>
The document model that is used in Couchbase looks then as the following one:<br />
<br /></div>
<div>
<script src="https://gist.github.com/dmaier-couchbase/820837064b2d58ded749.js"></script>
</div>
<br />
<b><u>Compression enabled by storing the adjacency list as a separated binary</u></b><br />
<br />
The following setting is used in order to enable the storage of the adjacency list as compressed binary:<br />
<ul>
<li>graph.compression.enabled=true</li>
<li>graph.compression.enabled=true</li>
</ul>
<div>
The document model which is used looks then as the following one:</div>
<div>
<br /></div>
<script src="https://gist.github.com/dmaier-couchbase/80613874a77587656b33.js"></script>
<br />
<div>
<b><u>Conclusion</u></b><br />
<b><u><br /></u></b>
Edge list compression helps in order to handle supernodes. The advantages are obviously that you can store more edges at a vertex, but also that the average size of a node is lower and so the latency behavior is improved because the network latency for getting a vertex from Couchbase to CBGraph is a function of the size of a vertex. The overall performance for bigger graphs was improved via this feature. As you can see, the underlying model looks different dependent on the compression mode. So the compression mode is a life time decision for a Graph. You can't access an uncompressed Graph via a CBGraph instance which is configured to use compression and vice versa.</div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-87620641009585883382016-02-09T16:02:00.001+01:002016-02-10T14:28:47.092+01:00Large-scale data processing with Couchbase Server and Apache SparkI just had the chance to work a bit with Apache Spark. Apache Spark is a distributed computation framework. So the idea is to spread computation tasks to many machines in a computation cluster. The idea here is to load data from Couchbase, process it in Spark and store the results back to Couchbase. Couchbase is the perfect companion for Spark because it is capable to handle huge amounts of data, provides a high performance (hundreds of thousands ops per second / sub-milliseconds latency) and is horizontally scalable by also being fault tolerant (replica copies, failover, ...).<br />
<br />
You might already know Hadoop for this purpose. Sparks approach is similar but different ;-). In Hadoop you typically load everything into the Hadoop distributed file system and then let process it 'co-located' in parallel. In Spark each worker node is processing the data by default in memory. Your data is described by a R(esilient) D(istributed) D(ataset). Such an RDD is in the first step not the data itself. It is more describing from where the data is coming and which kind of data is expected to be processed. The RDD API is also describing how data can be processed (actions, transformations). In the next step the data is retrieved based on this description, whereby it is distributed across multiple workers (executors). RDD-s are not just sharded across the cluster, it is also possible to replicate them for fault tollerance.<br />
<br />
Just in case that you don't know Spark, here the components of the Spark stack:<br />
<br />
<ul>
<li><b>Spark Core</b>: Handle RDD-s from several sources (and store them to several targets); So you could easily combine the data from several data sources with the data in Couchbase in order to derive new data which can then be stored back to your Couchbase bucket.</li>
<li><b>Spark SQL</b>: Handle data frames (RDD-s with a schema) whereby retrieving them (e.g. from a database system) by using a SQL like query syntax; Couchbase has it's own SQL like query language (N1QL) and so the integration works like a charm.</li>
<li><b>Spark Streaming</b>: Handle D(iscrete)Streams (RDD-s as micro batches); Couchbase allows you to consume the D(atabase) C(hange) P(rotocol) stream in order to react on changes in a bucket.</li>
</ul>
This blog post is focusing on these 3 main components, but there are also the following ones:<br />
<br />
<ul>
<li> MLib: An algorithm package for machine learning (to solve e.g. classification-, cluster-, regression- problems)</li>
<li>GraphX: Graph processing and graph-parallel computations</li>
</ul>
<br />
The following diagram shows the main architecture of a Spark cluster:<br />
<br />
<br />
<a href="http://3.bp.blogspot.com/-wMZo1PdLH7w/Vrs67HbxrlI/AAAAAAAAFHE/wJU2DBPxduI/s1600/spark.png" imageanchor="1"><img border="0" height="127" src="https://3.bp.blogspot.com/-wMZo1PdLH7w/Vrs67HbxrlI/AAAAAAAAFHE/wJU2DBPxduI/s400/spark.png" width="400" /></a><br />
<br />
<ul>
<li><b>Driver Program</b>: Creates the Spark context, declares the transformation and actions on RDD-s of data and submits them to the Master</li>
<li><b>Cluster Manager</b>: Acquires executors on worker nodes in the cluster.</li>
<li><b>Workers and their Executors</b>: Execute tasks and return results to the driver</li>
</ul>
<br />
So what can we do with Couchbase? The following simple code example (Scala) shows how to read some keys from the database, filter based on the country, log them to the console and count. Finally the driver is storing back the result to Couchbase.<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/a87b5fafdb854a20c290.js"></script>
You can also store the RDD-s directly to Couchbase by using 'rdd.saveToCouchbase()'. Here an example from Couchbase's documentation:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/874e502ff6a04191a166.js"></script>
The following very simple example shows how to retrieve data via SparkSQL:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/a81051400b0f45b22c3d.js"></script>
And finally, here a streaming example which retrieves a micro-batch every second:<br />
<br />
<script src="https://gist.github.com/dmaier-couchbase/7563d31ec5f83be3df2c.js"></script>
The example source can be found here: <a href="https://github.com/dmaier-couchbase/cb-spark-example">https://github.com/dmaier-couchbase/cb-spark-example</a> . It also provides the helper 'Contexts' which I used in order to initialize and access the Spark context.<br />
<br />
It's clear that this blog post was just a very brief introduction to Spark. Further, more use case focused, articles will follow. :-)<br />
<br />
Further examples can be found in Couchbase's documentation: <a href="http://developer.couchbase.com/documentation/server/4.1/connectors/spark-1.0/spark-intro.html">http://developer.couchbase.com/documentation/server/4.1/connectors/spark-1.0/spark-intro.html</a> .<br />
<br />
<br />David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-3838512812953868782015-11-20T10:03:00.002+01:002015-11-20T10:24:41.614+01:00Document Modeling BasicsAn often asked question of developers those are new to NoSQL is how to start with the document modeling. This article does not aim to give you answers to all document modeling related questions. It is more a starting point.<br />
<br />
<h3>
Flexible Schema</h3>
<div>
I am personally not a big fan of the word 'schema free'. My personal opinion is that if we talk about structured data, then we also talk about how to structure the data. (BTW: Couchbase also allows to store unstructured data as binaries. Also semi-structured is supported by e.g. embedding base64 encoded strings into JSON documents.) Couchbase Server does not enforce (on the database side) to follow a specific schema. This brings you more flexibility. Some documents might have a specific property, others might not have it. You don't have to specify upfront that a property might be there and then set it to a NULL value if it is not. So what you have is a flexible schema (or better data model), whereby the application is implicitly providing it. If your application/service is managing user profiles then you will find user documents whereby a user has a first name, last name ... and so on. So 'flexible data model' would be the better term.</div>
<div>
<br /></div>
<h3>
Key Patterns</h3>
<div>
<div>
Best practice is to use meaningful key patterns. This helps you directly access a document based on it's context. A key could be a combination of a type and a unique attribute value or it can be also an artificial number. If possible, don’t use artificial numbers. This is indeed not every time possible. The following example shows the key of a user with the email address “mmustermann@domain.com”:</div>
</div>
<div>
<span id="docs-internal-guid-dd2c3e59-23dc-486b-36ab-162818e7b50f"><br /></span>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com”</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Pattern is: $type::$email</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23dc-486b-36ab-162818e7b50f">
</span></div>
<div>
<br /></div>
<div>
<div>
If you know that users are accessible via their email address then you can directly get the user without the need to perform a more complex query. (Whereby querying is e.g. supported via SQL like query language - N1QL - in Couchbase). </div>
<div>
<br /></div>
<div>
A more interesting pattern would reflect a hierarchy. If you would assume that one employee belongs only to one company (but a company can have multiple employees) then you can reflect this ‘belongs to’ already in the key. The following shows an example of the key of a user who belongs to the company ‘Foo’ which has the domain ‘foo.org’.</div>
</div>
<div>
<br /></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::foo.org::12345”</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Pattern is: $type::$domain::$id</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23dc-bea4-c697-6d95bda94f22">
</span></div>
<div>
<br /></div>
<h3>
Types</h3>
<div>
We have already seen that the key pattern often has a type prefix. It is also best practice to store an extra type attribute. This allows you later to filter more specifically based on this type (e.g. to ask for all users). Here an example of a user:</div>
<div>
<br /></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Max”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Mustermann”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23de-16a9-693b-5854aba8df46">
</span></div>
<div>
<br /></div>
<h3>
1:1 Relationships</h3>
<div>
<div>
In this case one entity X has a relationship to another one and vice versa. A one to one relationship can be modeled by embedding or by referencing documents. My recommendation would be to model such a relationship in the first step as a key reference and then embed if there are atomicity requirements, which means if there is a requirement to access the two entities most of the times together. The following shows an example of an user who has a session.</div>
</div>
<div>
<br /></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Max”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Mustermann”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “mmustermann@domain.com”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “session” : { </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “source” : “web”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “token” : “123456”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> }</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Embedded Document</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23e0-1eb3-dce8-2ec2d63fc8fe">
</span></div>
<div>
<br /></div>
<div>
The same example by expressing it now as a key reference:</div>
<div>
<span id="docs-internal-guid-dd2c3e59-23e0-619a-a7a7-5ea38332c9bd"><br /></span>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Max”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Mustermann”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “mmustermann@domain.com”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “session” : “session::mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
<br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“session::mmustermann@domain.com” : { </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “session”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “source” : “web”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “token” : “123456”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “user” : “user::mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Explicitly Referenced Document</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23e0-619a-a7a7-5ea38332c9bd">
</span></div>
<div>
<br /></div>
<div>
<div>
It’s easy to see that there is a direct relationship via the id of the user, which is the email address in this case. Because the two documents are anyway correlated via their keys we can in this case simplify it to:</div>
</div>
<div>
<span id="docs-internal-guid-dd2c3e59-23e0-f902-f64a-bf01ca2f425c"><br /></span>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Max”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Mustermann”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
<br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“session::mmustermann@domain.com” : { </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “session”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “source” : “web”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “token” : “123456”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Documents those are implicitly referencing each other</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23e0-f902-f64a-bf01ca2f425c">
</span></div>
<div>
<br /></div>
<div>
When to embed or to reference is not like black or white. It's more transitioning with some grey values in between. Dependent on the access patterns it is indeed also possible to embed a part of the document and reference to another part.<br />
<span style="line-height: 1.38;"><br /></span>
<br />
<h3>
<span style="line-height: 1.38;">1:Many Relationship</span></h3>
<span style="line-height: 1.38;">The one to many relationship means that one document references multiple other ones (1 up to n). A back reference from the referenced one might be suitable. Again, I would by default reference and would embed as an optimization step if there are any atomicity requirements. This is only my personal preference, you could also start with embedding documents and then externalize by optimize regarding cardinalities and data duplication. Here a company which has multiple employees:</span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"></span>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<br />
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“company::domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “company”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “name” : “Some name”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “address” : “Some address”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “employees” : [ “user::bart.simpson@domain.com”, “user::moe@domain.com”]</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
<br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::bart.simpson@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Bart”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Simpson”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “</span><a href="mailto:bart.simpson@domain.com" style="text-decoration: none;"><span style="color: #1155cc; font-family: "arial"; font-size: 14.6667px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">bart.simpson@domain.com</span></a><span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “company” : “company::domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
<br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::moe@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”, </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Moe”, </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “</span><a href="mailto:moe@domain.com" style="text-decoration: none;"><span style="color: #1155cc; font-family: "arial"; font-size: 14.6667px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">moe@domain.com</span></a><span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “company” : “company::domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">1-to-many via key references</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><span id="docs-internal-guid-dd2c3e59-23e6-199e-0c7b-0275ce939491">
</span></span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">Let’s now assume that we embed the users as subdocuments. Another alternative would be to embed them in an array. An array is better if you need only to iterate over the list of embedded documents. If you need to access specific sub-documents by their id then embedding as nested documents would be preferred. It's identical to the question when you, as a developer, use a List or a Map in order to reflect the associations between your classes.</span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“company::domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “company”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “name” : “Some name”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “address” : “Some address”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “employees” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “user::bart.simpson@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Bart”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Simpson”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “</span><a href="mailto:bart.simpson@domain.com" style="text-decoration: none;"><span style="color: #1155cc; font-family: "arial"; font-size: 14.6667px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">bart.simpson@domain.com</span></a><span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> },</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “user::moe@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”, </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Moe”, </span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “</span><a href="mailto:moe@domain.com" style="text-decoration: none;"><span style="color: #1155cc; font-family: "arial"; font-size: 14.6667px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">moe@domain.com</span></a><span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> }</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> }</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">1-to-many as embedded document</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><span id="docs-internal-guid-dd2c3e59-23ea-5d0a-c21b-11e9f8033e3f">
</span></span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
</span>
<div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">What happens now if one employee works for multiple companies in the embedded case? Right, you get data duplicates because one user needs now to be fully embedded into 2 company documents. </span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">At the end it is a question of normalization. A completely normalized schema would contain a lot of relations whereby a de-normalized schema would in the worst case have everything in one table. As in relational databases, the truth is something in the middle. You would not embed everything into one document and you would normally also not express every property as an extra document and then use key references to glue them together. What works best depends on the actual requirements and how you need to access the data.</span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">So you reference in order to avoid duplicates but you embed in order to have atomic access. Your documents are usually in average smaller if you reference but you then might have to perform client side joins (or server side ones via N1QL since Couchbase 4.0). </span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0"><br /></span></div>
<div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">The Many-Many relationship (via references) is quite similar to the 1-Many one. It just means that you have reference arrays (arrays of keys to express the relationships) on both sides.</span></div>
</div>
<span id="docs-internal-guid-4275e662-23e1-51bc-f0d0-adc3124516e0">
<div>
<br /></div>
<h3>
Lookup Documents</h3>
<div>
A lookup document is a document which has only the purpose to provide you a direct reference to one or multiple other documents. Lookup documents are quite useful to maintain own indexes (alternative access paths) in Couchbase’s cache. Let’s assume you want to access a user profile via a customer number:</div>
<div>
<br /></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“customer_ref::12345” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “ref” : “mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
<br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">“user::mmustermann@domain.com” : {</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “type” : “user”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “first_name” : “Max”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “last_name” : “Mustermann”,</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> “email” : “mmustermann@domain.com”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23f2-a8c5-86d4-b9ae93e37259">
</span></div>
<div>
<br /></div>
<div>
In order to get an user by his customer id, you can now perform 2 get operations. First you get the lookup document based on the customer id, then you read the ‘ref’ attribute which gives you the key of the associated user document. In the next step you can then access the user directly. This way of access is often more efficient than an exact match query because the index lookup is in this case independent from the number of documents in the bucket or entries in the index which is scanned as part of the query processing.</div>
<div>
<br /></div>
<h3>
Atomic Counters</h3>
<div>
<div>
Couchbase allows you to increment counter values. This is a useful feature which helps you for instance to generate Id-s. So it’s similar to sequences in the relational world. The following shows some pseudo-code how to increment the counter value and then reuse it for the id generation.</div>
</div>
<div>
<span id="docs-internal-guid-dd2c3e59-23fc-6787-1a12-e18943477f4d"><br /></span>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">//“count::user” : “0”</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">id = client.incr(“count::user”);</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">client.add(“user::” + id, doc);</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-dd2c3e59-23fc-6787-1a12-e18943477f4d">
</span></div>
<div>
<div>
<br />
A typical pattern would be to perform a multi-get based on a range (e.g. 0...count::user) by taking the counter value into account. You could then skip every non existent document by ignoring the ‘DocNotFound’ error messages. This is indeed prefered if you have evolving data with only a low amount of deletes.</div>
<div>
<br /></div>
<div>
We saw in the section ‘Key patterns’ that keys can reflect hierarchies. So you could easily reflect a 1:Many relationship this way by not using explicit references. A user document belongs to a company document if the corresponding key contains the company prefix. We can get all users of a company by knowing the number of users of the company. Here some pseudo code:</div>
</div>
<div>
<br /></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 400px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">count = client.get(“count::foo.org::user”); //e.g. “563”</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">for ( i=0; i < count; i++ ) {</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> doc = client.get(“user::foo.org::” + i);</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> if (! doc.err ) {</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> result.add(doc);</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> }</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">}</span></div>
</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-7d943ef6-23fd-1d46-3fcf-33e7f5b1f0de">
</span></div>
<div>
<br /></div>
<h3>
Conclusion</h3>
<div>
Basic document modeling techniques were presented here. Couchbase allows you a flexible data model. As mentioned, the way how to model your data is not always black or white. My personal preference is to:</div>
<div>
<ul>
<li> Start with the logical data model (e.g. derived from Object Oriented Analysis)</li>
<li> Create a stupid and simple initial model (e.g. by using key references for 1:Many relationships all the time)</li>
<li> Evolve and optimize it step by step regarding the requirements (unnecessary reference lists because the reference is clear via the key pattern, access patterns, atomicity, duplicates, ...). </li>
</ul>
</div>
<div>
Here some useful rules/thoughts:</div>
<div>
<ul>
<li>Use meaningful keys and speaking key patterns if possible!</li>
<li>Use counters for the key generation if there is a need to use artificial ids!</li>
<li>Maintain a type attribute!</li>
<li>Embed documents into others in order to allow to write/get them all together with the parent document. (Atomic access)</li>
<li>If not embedding and if possible (e.g. by using the counter value as part of the key, or by having correlated keys) then express the relationship via the key directly without having the need to reference via key arrays.</li>
<li>Reference in order to avoid data duplicates and in order to keep the average document size smaller. Often we just live with duplicates by having other advantages (atomic access, no client side joins). But on the other hand's side we might have such a high amount of data and such a high degree of connectivity that we can't duplicate all the time.</li>
<li>If the cardinalities (number of related documents) are too high and there is no requirement for atomicity then referencing would be preferred over embedding.</li>
<li>Externalize groups of properties from a document (by adding a 1:1 relationship) if you access this group of properties all the the time together and if the overhead of transferring all the other properties of the document the same time would be too high.</li>
<li>It might make sense to externalize reference arrays from a document if the number of references is very high and so you would like to avoid the overhead of transferring these arrays if you usually only access a few properties of the document. </li>
</ul>
</div>
</span></div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0tag:blogger.com,1999:blog-2779610621928360824.post-7104650222056242022015-11-02T21:08:00.000+01:002015-11-02T21:47:26.748+01:00Using a Key-Value Store for Full Text Indexing and SearchCouchbase Server is a multi-purpose Database System. One of the purposes is to use it as a simple key-value store. A key-value store allows you to store/retrieve any value by its key. Such a value can be a JSON document (Couchbase allows you to index and query based on such JSON documents and so another purpose is the one as document database.), a small binary or a full text index entry. This article explains why such a key-value store can be also used for full text indexing purposes.<br />
<br />
Let's explain how full text indexing works in general. A full text index is a so called inverted index. The table below shows how the following sentences would be indexed: 'Tim is sitting next to Bob' and 'Jim is sitting next to Bob'. The word 'Tim' is only existing in the first sentence and there is exactly one occurrence of it.<br />
<table class="highlight tab-size js-file-line-container" data-tab-size="8" style="background: rgb(255, 255, 255); border-collapse: collapse; border: 0px; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; line-height: 1.4; margin: 0px; padding: 0px;"><tbody>
<tr><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC8" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"><br class="Apple-interchange-newline" /></td></tr>
<tr><td class="blob-num js-line-number" data-line-number="9" id="file-inverted_index-L9" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC9" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Term | Count | Reference</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="10" id="file-inverted_index-L10" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC10" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">------------------------</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="11" id="file-inverted_index-L11" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC11" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Tim | 1 | #1</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="12" id="file-inverted_index-L12" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC12" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">is | 2 | #1, #2</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="13" id="file-inverted_index-L13" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC13" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">sitting | 2 | #1, #2</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="14" id="file-inverted_index-L14" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC14" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">next | 2 | #1, #2</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="15" id="file-inverted_index-L15" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC15" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">to | 2 | #1, #2</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="16" id="file-inverted_index-L16" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC16" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Bob | 2 | #1, #2</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="17" id="file-inverted_index-L17" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC17" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Jim | 1 | #2</td></tr>
</tbody></table>
<br />
<br />
There are a bunch of specialized systems out there for full text indexing. Couchbase has for instance a very good integration with Elasticsearch. In the future Couchbase will also have it's own full text service which is called 'cbft'. However, this article is not about Elasticsearch and also not about 'cbft'. We want to use Couchbase's key-value store features for full text indexing here.<br />
<br />
Let's define the data model first:<br />
<br />
<br class="Apple-interchange-newline" />
<table class="highlight tab-size js-file-line-container" data-tab-size="8" style="background: rgb(255, 255, 255); border-collapse: collapse; border: 0px; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; line-height: 1.4; margin: 0px; padding: 0px;"><tbody>
<tr><td class="blob-num js-line-number" data-line-number="9" id="file-inverted_index-L9" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC9" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">"fts::$field::$term" : { "count" : $numOf, <span style="line-height: 16.8px;">"refs" : [ ...] </span><span style="background-color: transparent; line-height: 1.4;">}</span>
</td></tr>
<tr><td class="blob-num js-line-number" data-line-number="10" id="file-inverted_index-L10" style="-webkit-user-select: none; background: transparent; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; color: rgba(0, 0, 0, 0.298039); cursor: pointer; line-height: 18px; min-width: inherit; padding: 1px 10px !important; text-align: right; vertical-align: top; white-space: nowrap; width: 14px;"></td><td class="blob-code blob-code-inner js-file-line" id="file-inverted_index-LC10" style="background: transparent; border: 0px; overflow: visible; padding: 1px 10px !important; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"><br /></td></tr>
</tbody></table>
<br />
<div>
It is actually quite simple. A full text search index entry does point back for a term to the original key-value pairs those are identified by their keys. The refs array contains these keys. The field is just the field on which we want to search. This could be for instance 'address' or 'message'. Let's say that the default field is called '_all'. So if no specific field is used then the '_all' field is used as the fallback field. </div>
<div>
<br /></div>
<div>
In order to index based on a provided text, we can do the following:</div>
<div>
<ul>
<li>Tokenize the text on which should be indexed. This means basically to break the text up into several words (terms). In our case we assume that our text contains the word 'fox'.</li>
<li>Check if the e.g. the key "fts::_all::fox" is already existing. If not then create the document by referencing back to the document id which contained the word 'fox'.</li>
<li>If the full text index entry was existent then check if the reference list does already contain the reference to the document which contains the text on which should be indexed.</li>
<li>If the reference list does not yet contain the key of this document then extend the reference list by adding the key of the document.</li>
</ul>
</div>
<div>
Now in order to search for the specific word 'fox', we just have to do the following:</div>
<div>
<ul>
<li>Get "fts::_all::fox"</li>
<li>Perform a multi-get based on the keys in the array 'refs'</li>
</ul>
<div>
Some example source code (Node.js) can be found here: <a href="https://github.com/dmaier-couchbase/cb-node-fts">https://github.com/dmaier-couchbase/cb-node-fts</a> . The service is implemented here: <a href="https://github.com/dmaier-couchbase/cb-node-fts/blob/master/routes/fts.js">https://github.com/dmaier-couchbase/cb-node-fts/blob/master/routes/fts.js</a> . This application was created by using CEAN stack tools (Couchbase + Express + Angular + Node.js ). They are available here: <a href="http://www.ceanjs.org/">http://www.ceanjs.org</a> .</div>
</div>
<div>
<br /></div>
<div>
Given that I already wrote this little demo application, it makes sense to try it out :-) . First let's add 2 sentences:</div>
<div>
<ul>
<li>the_fox: The quick brown fox jumps over the lazy dog</li>
<li>the_cat: The quick brown fox jumps over the lazy cat</li>
</ul>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-y6DEMNi4Kfs/Vje804H66RI/AAAAAAAAFCQ/JzqWwRvreMg/s1600/added_the_fox.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-y6DEMNi4Kfs/Vje804H66RI/AAAAAAAAFCQ/JzqWwRvreMg/s400/added_the_fox.png" width="400" /></a></div>
<div>
<br /></div>
<div>
Now in the next step let's perform some searches. I implemented the word search in a way that you can enter any sequence of words (separated by white spaces). The following searches for 'cat':</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-BEa_PdFpdUw/Vje9aDHlnSI/AAAAAAAAFCY/GeC-aMqNaqk/s1600/result_cat.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="http://1.bp.blogspot.com/-BEa_PdFpdUw/Vje9aDHlnSI/AAAAAAAAFCY/GeC-aMqNaqk/s400/result_cat.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
As we can see, only the sentence with the id 'the_cat' contains the word 'cat'. Next lets's search for the word 'fox':</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-uZvhwoznZBk/Vje-XDHJtKI/AAAAAAAAFCk/s4fmea15ywg/s1600/result_fox.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="327" src="http://1.bp.blogspot.com/-uZvhwoznZBk/Vje-XDHJtKI/AAAAAAAAFCk/s4fmea15ywg/s400/result_fox.png" width="400" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
Both sentences contain the word 'fox'. Last but not least let's search for multiple words:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-KsJzFdHCPlc/Vje-pkBtiII/AAAAAAAAFCs/Gzhsg75SB-Q/s1600/result_dog_cat.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="327" src="http://2.bp.blogspot.com/-KsJzFdHCPlc/Vje-pkBtiII/AAAAAAAAFCs/Gzhsg75SB-Q/s400/result_dog_cat.png" width="400" /></a></div>
<div>
<br /></div>
<div>
I think you get it ... :-) . The data which is stored in Couchbase looks as the following one:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-YVeCcU2hPmU/Vje_MiiU0NI/AAAAAAAAFC4/-2pA_dvA578/s1600/couchbase.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="http://1.bp.blogspot.com/-YVeCcU2hPmU/Vje_MiiU0NI/AAAAAAAAFC4/-2pA_dvA578/s400/couchbase.png" width="400" /></a></div>
<div>
<br /></div>
<div>
This article explained how you can use Couchbase to store a full text search index. Such an index can be used for simple and basic text searches, which might be sufficient for some of your development projects. If you need more sophisticated text search or text analysis then a dedicated full text search service might be the better option.</div>
David Maierhttp://www.blogger.com/profile/13397846517622289056noreply@blogger.com0