Lisp Web Tales

My attempts at learning web development, foolishly using common lisp, and even more foolishly, writing about it in public

By Pavel Penev

Get the PDF/EPUP/MOBI version from leanpub.com

Get the source from Github

Table of Contents

  1. Preface
  2. Introduction
  3. Why Lisp
  4. Whats in the book
  5. Who is this for
  6. What you need to get started
  7. Typographic conventions
  8. 1 The basics
  9. Raw example
  10. A simple blog
  11. The source code
  12. Source walk-through
  13. Conclusion
  14. 2 Setting up a project
  15. Systems
  16. Quicklisp and manual installation
  17. restas-project
  18. Setting up a hello-world project
  19. Running the project
  20. Conclusion
  21. 3 HTML generation and templating
  22. Exploring all the options
  23. closure-template
  24. 4 Putting it together: Updated blog example
  25. Creating the project
  26. Setting up defmodule.lisp
  27. Updating the templates
  28. Rewriting the routes
  29. Post chapter refactoring
  30. Running the app
  31. 5 Persistence part I: PostgreSQL
  32. Introduction
  33. Setting up PostgreSQL
  34. What is a policy?
  35. Creating the project
  36. The schema
  37. Connecting
  38. Defining the tables.
  39. Defining our interface
  40. Exporting the interface
  41. Conclusion
  42. 6 Persistence part II: The UI
  43. Introduction
  44. Set up
  45. The templates
  46. The routes
  47. Getting it to run
  48. 7 Persistence part III: Redis
  49. Introduction
  50. A note on redis
  51. The “schema” or lack there of
  52. The implementation
  53. Handling upvotes
  54. Handling links
  55. Conclusion
  56. 8 Modules as reusable components part I: restas-directory-publisher
  57. Introduction
  58. Handling static files in restas
  59. Mounting modules
  60. Adding a stylesheet to linkdemo
  61. Contexts
  62. Conclusion
  63. 9 Modules as reusable components part II: Implementing a reusable module
  64. Introduction
  65. The interface
  66. The templates
  67. The routes
  68. Using authdemo in an application
  69. Conclusion
  70. Appendix A: Getting started
  71. Linux
  72. Windows
  73. Appendix B: Recomended reading
  74. Online tutorials
  75. Cliki: The Common Lisp wiki
  76. IRC

7 Persistence part III: Redis

Introduction

We have already implemented a mostly complete application using PostgreSQL, now we’ll implement the datastore backend using redis. Because of the separation between interface and implementation that CLOS and restas policies provide us we won’t have to touch any of the application code outside of adding redis as a dependency and actually implementing the policy to use it.

Redis is a very simple key-value store, it has support for strings, sets, lists and hash tables, and uses a very simple protocol and api. It has no query language, and no way to define a schema, which both simplifies and complicates things.

I assume redis is installed and is running on the local machine. On debian-like systems, this is just an apt-get install redis-server away. You can see if redis is running by starting the console client redis-cli and typeing ping in the prompt, it should respond with “PONG”.

Next, let’s add cl-redis, the lisp client library for redis to our list of dependencies in linkdemo.asd. Also, lets add a file named redis-datastore.lisp to the project. This is how our system definition looks like:

 1 (asdf:defsystem #:linkdemo
 2   :serial t
 3   :description "Your description here"
 4   :author "Your name here"
 5   :license "Your license here"
 6   :depends-on (:RESTAS :SEXML :POSTMODERN :ironclad :babel :cl-redis)
 7   :components ((:file "defmodule")
 8                (:file "pg-datastore")
 9                (:file "redis-datastore")
10                (:file "util")
11                (:file "template")
12                (:file "linkdemo")))

In defmodule.lisp we must add a package definition for our implementation of the redis backend:

1 (defpackage #:linkdemo.redis-datastore
2   (:use #:cl #:redis #:linkdemo.policy.datastore)
3   (:export #:redis-datastore))

Like linkdemo.pg-datastore this package uses the cl package, the redis package which contains the redis api and the policy package, where the names of the methods we must implement reside. We export the symbol redis-datastore, which names the class we use to access the datastore.

A note on redis

Redis commands are fairly simple, but some of them conflict with Common Lisp names, so they are all prefixed with red-. For instance getting the value of the key “foo” is done with (red-get "foo"). Connecting to the datastore is done with the connect function which takes :host and :port parameters or with the with-connection macro which takes the same parameters. Like with postmodern, we’ll be using with-connection.

The “schema” or lack there of

Because redis is schemaless, we must decide on a way to structure our data in redis. After some research and experimentation this is what I came up with:

A user record will be stored as a string, containing the printed representation of a plist. We’ll use the common lisp read/print consistency feature to ensure that what we print into that string will be read back consistently as a lisp list. Essentially, we’re serializing data into strings. Each such record will be kept under a key named USER:{id}, where id is an integer denoting the id of the record. Since keys are strings, we’ll have to generate them with a function using string concatenation. We’ll also need another record called USER-IDS which is an integer we’ll increment every time we add a user, and use the value as a new id.

Because we’ll also need to lookup users by their usernames, we’ll add another key to the datastore, called USERNAME:{username}, where username is the username string. This key will hold the id we’ll use to lookup the whole record for that user. This is a sort of reverse lookup if you will.

Posts we’ll store in a similar way, we’ll have a POSTS-IDS key with the id count, and we’ll have a POST:{id} key holding the serialized plist record for that post.

Upvotes are interesting because we can just use sets to store the set of user names that have upvoted a given post. We’ll store these sets in keys named UPVOTE:{post-id}.

So for example if we have one user, named user, the datastore will contain this information:

1 "USER-IDS" : 1
2 "USER:1" : "(:id 1 :username \"user\" :password \"...\" :salt \"...\")"
3 "USERNAME:user" : 1:

If he posts a link, it would look like this:

1 "POST-IDS" : 1
2 "POST:1" : "(:id 1 :url \"http://google.com\" :title \"google\" :submitter-id 1)"

And the corresponding upvote:

1 "upvote:1" : {"user", }

The implementation

In redis-datastore.lisp first lets define our datastore access class. Redis connections take two arguments, a host and a port, by default they are the local host, and 6379, Here is the class:

1 (in-package #:linkdemo.redis-datastore)
2 
3 (defclass redis-datastore ()
4   ((host :initarg :host :initform #(127 0 0 1) :accessor host)
5    (port :initarg :port :initform 6379 :accessor port)))

Note that the host is given as a vector denoting the ip address 127.0.0.1. Since there is nothing to initialize, the datastore-init method is empty:

1 (defmethod datastore-init ((datastore redis-datastore)))

Convenience functions

Next are a couple of convenience functions we’ll need, first are the familiar hash-password and check-password from the pg-datastore.lisp file. Take it as an exercises to move these functions to a separate package in a separate file and eliminate the duplication, if it bugs you:

 1 (defun hash-password (password)
 2   (multiple-value-bind (hash salt)
 3       (ironclad:pbkdf2-hash-password (babel:string-to-octets password))
 4     (list :password-hash (ironclad:byte-array-to-hex-string hash)
 5           :salt (ironclad:byte-array-to-hex-string salt))))
 6 
 7 (defun check-password (password password-hash salt)
 8   (let ((hash (ironclad:pbkdf2-hash-password
 9                (babel:string-to-octets password)
10                :salt (ironclad:hex-string-to-byte-array salt))))
11     (string= (ironclad:byte-array-to-hex-string hash)
12              password-hash)))

In order to store lisp lists in redis, we need to print and read them consistently. Fortunately for us, lisp is itself a kind of serialization format. Lisp objects like symbols, keywords, strings, lists and numbers can be printed to a string, and then read back as lisp objects. Here are two functions that will do that for us:

1 (defun serialize-list (list)
2   (with-output-to-string (out)
3     (print list out)))
4 
5 (defun deserialize-list (string)
6   (let ((read-eval nil))
7     (read-from-string string)))

And finally, we need a way to generate keys like “USER:1” and “USER:2” and so on, the function make-key takes a keyword and a string or number and generates a key for us:

1 (defun make-key (prefix suffix)
2   (format nil "~a:~a" (symbol-name prefix) suffix))

Handling users

In order to get a user record by username, first we lookup the user id, We do this simply with a red-get command and a key in the form of "USERNAME:{username}", generated with make-key. Next we use the id to retrieve the user record and convert it to a list with deserialize-list:

1 (defmethod datastore-find-user ((datastore redis-datastore) username)
2   (with-connection (:host (host datastore)
3                     :port (port datastore))
4     (let ((user-id (red-get (make-key :username username))))
5       (when user-id
6         (deserialize-list (red-get (make-key :user user-id)))))))

Authenticating the user is done with almost identical code to the postmodern example:

1 (defmethod datastore-auth-user ((datastore redis-datastore) username password)
2   (let ((user (datastore-find-user datastore username)))
3     (when (and user
4                (check-password password 
5                                (getf user :password)
6                                (getf user :salt)))
7       username)))

Registering the user on the other hand is a bit more involved. First we must create a new id by using the red-incr command, which increments the value of the USER-IDS key. Then, we must use this id to generate a new “USERS:{id}” key, and set it to the serialized plist containing the user information. We must then add the id as a value to the “USERNAME:{username}” key. And finally, we return the username. Not to forget also hashing the password, and checking if such a user already exists:

 1 (defmethod datastore-register-user ((datastore redis-datastore) username password)
 2   (with-connection (:host (host datastore)
 3                     :port (port datastore))
 4     (unless (datastore-find-user datastore username)
 5       (let* ((password-salt (hash-password password))
 6              (id (red-incr :user-ids))
 7              (record (list :id id
 8                            :username username
 9                            :password (getf password-salt :password-hash)
10                            :salt (getf password-salt :salt))))
11         (red-set (make-key :user id) (serialize-list record))
12         (red-set (make-key :username username) id)
13         username))))

Handling upvotes

Checking if a user has upvoted a link is as easy as checking to see if that user is in the set of upvoters for that link. Sets in redis are accessed with the red-sismember command(which I assume stands for “set is member”). It simply takes a key and a value and checks to see if that value is in the set denoted by the key:

1 (defmethod datastore-upvoted-p ((datastore redis-datastore) link-id user)
2   (with-connection (:host (host datastore)
3                     :port (port datastore))
4     (red-sismember (make-key :upvote link-id) user)))

Upvoting a post is also a fairly simple task. First we must check if the user exists, and that the link isn’t upvoted, and then we simply add the username to the set of users who have upvoted this link. Adding an element to a set is done with the red-sadd command:

1 (defmethod datastore-upvote ((datastore redis-datastore) link-id user)
2   (with-connection (:host (host datastore)
3                     :port (port datastore))
4     (when (and (datastore-find-user datastore user)
5                (not (datastore-upvoted-p datastore link-id user)))
6       (when (red-sadd (make-key :upvote link-id) user)
7         link-id))))

Posting a link involves first getting the user id of the submitter, generating a new id for the link, and then setting the key “POST:{id}” to the serialized plist of the record. After that we upvote the link:

 1 (defmethod datastore-post-link ((datastore redis-datastore) url title user)
 2   (with-connection (:host (host datastore)
 3                     :port (port datastore))
 4     (let* ((submitter-id (getf (datastore-find-user datastore user) :id))
 5            (id (red-incr :posts-ids))
 6            (link (list :id id
 7                        :url url
 8                        :title title
 9                        :submitter-id submitter-id)))
10       (red-set (make-key :post id) (serialize-list link))
11       (datastore-upvote datastore (getf link :id) user))))

Extracting all the links is a bit interesting. Somehow we bust get a list of all keys that start with “POST:” and then extract them all. We’re in luck, since redis has a command red-keys that returns a list of keys matching a pattern, we simply pass it “POST:*” and we’ll get them all. Then we “get” the keys and deserialize their values:

1 (defun get-all-links/internal ()
2   (let ((keys (red-keys (make-key :post "*"))))
3     (loop for key in keys
4          collect (deserialize-list (red-get key)))))

Getting the upvote count is as easy as counting the elements of a set, and fortunate for us, redis has such a command, red-scard, which I can never remember, and always have to lookup :)

1 (defmethod datastore-upvote-count ((datastore redis-datastore) link-id)
2   (with-connection (:host (host datastore)
3                     :port (port datastore))
4     (red-scard (make-key :upvote link-id))))

The functions add-vote-count, sort-links and datastore-get-all-links are almost the same:

 1 (defun add-vote-count (datastore links username)
 2   (loop
 3      for link in links
 4      for id = (getf link :id)
 5      collect (list* :votes (datastore-upvote-count datastore id)
 6                     :voted-p (datastore-upvoted-p datastore id username)
 7                     link)))
 8 
 9 (defun sort-links (links)
10   (sort links #'>
11         :key #'(lambda (link) (getf link :votes))))
12 
13 (defmethod datastore-get-all-links ((datastore redis-datastore) &optional username)
14   (with-connection (:host (host datastore)
15                     :port (port datastore))
16     (sort-links
17      (add-vote-count datastore
18                      (get-all-links/internal)
19                      (or username "")))))

Conclusion

And we’re done. Save the file, reload the code and now we can start our redis backed app with:

1 * (linkdemo:start-linkdemo 
2     :datastore 'linkdemo.redis-datastore:redis-datastore)

And in fact, if we swap out the value of *datastore* with an instance of redis-datastore or pg-datastore we can switch the backend database while the app is running. This is pretty cool IMHO.

Further reading: