You remember how I've been trying to write a site analyser for my blog for the last year or so? Well, I was using DynamoDB for storing some metrics, and I realised that would bankrupt me, so I decided to use Amazon Athena run SQL queries directly on my access logs instead. That should have been a great idea, but when I fired up my REPL with awyeah-api, I got quite the unpleasant surprise:
(comment
(def athena (aws/client {:api :athena
:region (or (System/getenv "AWS_DEFAULT_REGION") "eu-west-1")}))
;; => #'user/athena
(aws/invoke athena {:op :ListWorkGroups
:request {}})
;; => {:__type "InvalidSignatureException",
;; :message
;; "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/\n\naccept:application/json\ncontent-type:application/x-amz-json-1.1\nhost:athena.eu-west-1.amazonaws.com:443\nx-amz-date:20231105T115515Z\nx-amz-target:AmazonAthena.ListWorkGroups\n\naccept;content-type;host;x-amz-date;x-amz-target\n44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20231105T115515Z\n20231105/eu-west-1/athena/aws4_request\n4dd8829addcae8b5fe35d78323f484298d478ab9af786b3caa12bac0d2f57a8c'\n",
;; :cognitect.anomalies/category :cognitect.anomalies/incorrect}
What the what?
Digging into this with some tasty debug prn
s, I found
out that the issue was that Athena was expecting a canonical string like this:
POST
/
accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com:443
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups
accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
And awyeah-api was generating a canonical string like that:
POST
/
accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups
accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
I'm sure you've already spotted the difference, right? ๐
Take a look at the host
header. Athena wants this:
host:athena.eu-west-1.amazonaws.com:443
And we're giving it that:
host:athena.eu-west-1.amazonaws.com
For want of a :443
, the kingdom was lost! ๐ญ
After filing a bug
report, I started reading through all the commits to aws-api since 0.8.603
, which is the latest commit that's been ported over to awyeah-api. These two in particular looked pretty tasty:
I ported those two over to a clone of awyeah-api, but my REPL still informed me that I sucked:
(comment
(aws/invoke athena {:op :ListWorkGroups
:request {}})
;; => {:__type "InvalidSignatureException",
;; :message
;; "The request signature we calculated does not match the signature you provided. Haven't you heard Einstein's definition of insanity, dude? Try something else already!\n",
;; :cognitect.anomalies/category :cognitect.anomalies/you-suck}
Never one to give up so easily, I continued porting stuff, all the way up to the latest aws-api commits (skipping a few commits that I just couldn't figure out how to port), and not only did the InvalidSignatureException
persist, but now I'd broken the awyeah-api tests. ๐คฆ
After sleeping on it, I realised that the root of my problem is that I was just porting code over without really thinking about what the code was doing, and that a better approach would perhaps be to understand the code. Obv.
But instead of just understanding the code I was porting and why, I decided that I needed to understand how Michael Glaesemann AKA grzm had made aws-api work with Babashka in the first place. And how better to do that than to fork aws-api into my own Github repo and start replicating the awyeah-api stuff on top of it? (I'm sure there are much better ways to do that, but if you've been reading this blog for more than five minutes, you'll know that I never take the better way when a worse and more painful way is available. Just call me Gandalf in the Mines of Moria.)
And what better name to capture the dismay that I'm sure you're feeling as you read this than awno-api? And what better tense to capture the excitement of a fool's errand than the present?
After cloning my new forked repo, let's get to work making aws-api into something that will work with babashka. The first order of business is purely mechanical:
cd ~/Documents/code/clojure/awyeah-api
cp -r bb.edn bin CHANGES.markdown docs etc ../awno-api/
cd ../awno-api
mkdir -p src/net/jmglov test/src/net/jmglov
git mv src/cognitect/aws src/net/jmglov/awno
git mv test/src/cognitect/aws test/src/net/jmglov/awno
git rm -rf doc latest-releases.edn pom.xml UPGRADE.md README.md CHANGES.md
rm -rf src/cognitect test/src/cognitect
Now let's update bb.edn
to use the latest versions of all the good stuff. We can grab the commit SHA for the latest version of the babashka port of Spec from https://github.com/babashka/spec.alpha, then the various versions of the AWS libraries from latest-releases.edn in the aws-api repo, resulting in a shiny new bb.edn
:
{:paths ["resources" "src" "test/resources" "test/src"]
:deps
{org.babashka/spec.alpha {:git/url "https://github.com/babashka/spec.alpha"
:git/sha "8df0712896f596680da7a32ae44bb000b7e45e68"}
org.clojure/tools.logging {:mvn/version "1.2.4"}
com.cognitect.aws/endpoints {:mvn/version "1.1.12.307"}
com.cognitect.aws/ec2
{:mvn/version "822.2.1122.0", :aws/serviceFullName "Amazon Elastic Compute Cloud"},
com.cognitect.aws/lambda
{:mvn/version "822.2.1122.0", :aws/serviceFullName "AWS Lambda"},
com.cognitect.aws/s3
{:mvn/version "822.2.1145.0"}
com.cognitect.aws/ssm
{:mvn/version "822.2.1122.0", :aws/serviceFullName "Amazon Simple Systems Manager (SSM)"}
com.cognitect.aws/sts
{:mvn/version "822.2.1109.0", :aws/serviceFullName "AWS Security Token Service"}}}
Now to open up awno-api/README.markdown
, cry havoc, and let slip the dogs of projectile-replace
with M-p R
! (If you're not using Doom
Emacs, you'll probably need to smack some different keys; if you don't know which ones, you can always take the `M-x projectile-replace` route).
After replacing all of the instances (you'll be prompted to replace each occurrence, but you hit !
at the first prompt if you want to YOLO replace all occurrences), you can use C-x s
to invoke save-some-buffers
, then hit !
at the prompt to save all of the files you just replaced cognitect.aws
in.
Now that all your namespaces are belong to us, it's time for the moment of truth! Will the REPL start? With baited breath, let's hit C-c M-j
to invoke cider-jack-in-clj
, instruct CIDER to use babashka, and hope for the best.
Started nREPL server at 127.0.0.1:35223
For more info visit: https://book.babashka.org/#_nrepl
;; Connected to nREPL server - nrepl://127.0.0.1:35223
;; CIDER 1.7.0-snapshot (package: 1.7.0-snapshot), babashka.nrepl 0.0.6-SNAPSHOT
;; Babashka 1.3.176
;; Docs: (doc function-name)
;; (find-doc part-of-name)
;; Source: (source function-name)
;; Javadoc: (javadoc java-object-or-class)
;; Exit: <C-c C-q>
;; Results: Stored in vars *1, *2, *3, an exception in *e;
;; Startup: /home/jmglov/.nix-profile/bin/bb nrepl-server localhost:0
WARNING: Can't determine Clojure version. The refactor-nrepl middleware requires clojure 1.8.0 (or newer)WARNING: clj-refactor and refactor-nrepl are out of sync.
Their versions are 3.6.0 and n/a, respectively.
You can mute this warning by changing cljr-suppress-middleware-warnings.
user>
Equipped with our trusty REPL, it's time to get down to the real work of really making it work. Luckily for us, good ol' grzm has captured the decisions he made whilst porting aws-api in a nice porting-decisions.markdown doc, so let's start reading.
Right at the top of the doc, he mentions a set of Java classes that are missing from babashka:
java.lang.Runnable
java.lang.ThreadLocal
java.util.concurrent.ThreadFactory
java.util.concurrent.ScheduledFuture
java.util.concurrent.ScheduledExecutorService
Since awyeah-api was originally written, three of those five classes have been added to babashka, according to babashka.impl.classes:
java.lang.Runnable
java.util.concurrent.ThreadFactory
java.util.concurrent.ScheduledExecutorService
So now we only have to deal with the other two.
According to grzm:
These classes are referenced in two namespaces:
cognitect.aws.util
andcognitect.aws.credentials
.ThreadLocal
is used incognitect.aws.util
to makejava.text.SimpleDateFormat
thread-safe. As I'm not concerned with supporting pre-Java 8 versions, I've decided to use the thread-safejava.time.format.DateTimeFormatter
rather than drop thread-safety workarounds forSimpleDateFormat
or implement them in some other way.The
cognitect.aws.util
namespace is used throughout the aws-api library, either directly or transitively.The balance of the unincluded classes are used in
cognitect.aws.credentials
to provide auto-refreshing of AWS credentials. As babashka is commonly used for short-lived scripts as opposed to long-running server applications, rather than provide an alternate implementation for credential refresh, I've chosen to omit this functionality. If credential auto-refresh is something I find is useful in a babashka context some time in the future, a solution can be explored at that time.
Snatching up that tasty breadcrumb, let's open our shiny new net.jmglov.awno.util
namespace.
As we have no more appetite than grzm for supporting Java versions older than Java 8, let's just follow his lead and kick ThreadLocal
to the curb. That means replacing this mess from aws-api:
(defn date-format
"Return a thread-safe GMT date format that can be used with `format-date` and `parse-date`.
See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4228335"
^ThreadLocal [^String fmt]
(proxy [ThreadLocal] []
(initialValue []
(doto (SimpleDateFormat. fmt)
(.setTimeZone (TimeZone/getTimeZone "GMT"))))))
(defn format-date
([fmt]
(format-date fmt (Date.)))
([^ThreadLocal fmt inst]
(.format ^SimpleDateFormat (.get fmt) inst)))
With this lovely stuff from awyeah-api:
(defn date-format
"Return a thread-safe GMT date format that can be used with `format-date` and `parse-date`."
[^String fmt]
(.withZone (DateTimeFormatter/ofPattern fmt) (ZoneId/of "GMT")))
(defn format-date
([formatter]
(format-date formatter (Date.)))
([formatter ^Date inst]
(.format (ZonedDateTime/ofInstant (.toInstant inst) (ZoneId/of "UTC"))
^DateTimeFormatter formatter)))
Copy and paste-driven development FTW! ๐
The parse-date
function from aws-api needed a little love too. This:
(defn parse-date
[^ThreadLocal fmt s]
(.parse ^SimpleDateFormat (.get fmt) s))
Becomes that:
(defn parse-date
[formatter s]
(Date/from (.toInstant (ZonedDateTime/parse s formatter))))
This leaves only five more occurrences of ThreadLocal
in the namespace:
(def ^ThreadLocal x-amz-date-format
(date-format "yyyyMMdd'T'HHmmss'Z'"))
(def ^ThreadLocal x-amz-date-only-format
(date-format "yyyyMMdd"))
(def ^ThreadLocal iso8601-date-format
(date-format "yyyy-MM-dd'T'HH:mm:ssXXX"))
(def ^ThreadLocal iso8601-msecs-date-format
(date-format "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"))
(def ^ThreadLocal rfc822-date-format
(date-format "EEE, dd MMM yyyy HH:mm:ss z"))
For these, all we need to do is remove the ^ThreadLocal
type hint and we're golden! Well, almost golden. We're using a few java.time
classes that weren't in the aws-api version, so we need to add those, and whilst we're at it, let's go ahead and remove the classes that aren't necessary anymore, and alphabetise the requires and imports in the ns
form and OMG what are square brackets doing in the :import
section? According to St.
Stuart, thou must always use round brackets with :import
, never square!
Making the sign to ward off evil, we fix the glitch.
With all of this done, we should be able to eval the namespace in our REPL, right? C-c C-k
(cider-load-buffer
) will tell the tale:
clojure.lang.ExceptionInfo: Could not find namespace: clojure.data.json.
{:type :sci/error, :line 9, :column 3, :message "Could not find namespace: clojure.data.json.", :sci.impl/callstack #object[clojure.lang.Volatile 0xfb2c337 {:status :ready, :val ({:line 9, :column 3, :file "/home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/util.clj", :ns #object[sci.lang.Namespace 0x2c8684c "net.jmglov.awno.util"]})}], :file "/home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/util.clj"}
at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:129)
sci.impl.analyzer$return_ns_op$reify__4355.eval (analyzer.cljc:1189)
sci.impl.analyzer$return_do$reify__3968.eval (analyzer.cljc:130)
sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
sci.core$eval_form.invokeStatic (core.cljc:329)
babashka.nrepl.impl.server$eval_msg$fn__27295$fn__27296.invoke (server.clj:108)
[...]
Blerg!
No clojure.data.json
, eh? Let's return to porting-decisions.markdown
and see is grzm has anything to say about that. Indeed he
does!
The aws-api library depends on
clojure.data.json
for JSON serialization and deserialization, a pure Clojure library. Babashka includes Cheshire for JSON support and notclojure.data.json
.The Clojure source of
clojure.data.json
can be interpreted by sci, so I could includeclojure.data.json
as a dependency and use it as-is. Theclojure.data.json
usage in aws-api is easily replaced by Cheshire. Replacingclojure.data.json
with Cheshire means one less dependency to include, and we can leverage compiled code rather than interpreted. To isolate the library choice, I've extracted the library-specific calls in thecom.grzm.awyeah.json
namespace.
We can copy src/com/grzm/awyeah/json.clj
from awyeah-api over to awno-api and fix up the namespace declaration. Let's also make read-str
plug compatible with clojure.data.json
by adding a three arity form that allows passing a custom key function:
(ns net.jmglov.awno.json
(:require [cheshire.core :as json]))
(defn write-str [x]
(json/generate-string x))
(defn read-str
([s]
(read-str :key-fn keyword))
([s _ key-fn]
(json/parse-string s key-fn)))
This will handle all the calls like this in the aws-api codebase:
(json/read-str :key-fn keyword)
Having done that, let's M-p R
(like NPR, but runs projectile-replace
instead of giving you some news with a vaugely progressive yet US-centric slant) to replace clojure.data.json
with net.jmglov.awno.json
project-wide. Don't forget to C-x s !
again to save all of the files we just modified.
Now let's try evaluating the namespace...
At this point, we should be able to run the net.jmglov.awno.util
tests. Popping over to test/src/net/jmglov/awno/util_test.clj
, we see that the Cognitect folks have left us a nice little Rich comment at the bottom of the file:
(comment
(t/run-tests))
Let's expand it a bit, put the cursor at the end of the t/run-tests
form, and do some C-c C-v f c e
to invoke cider-pprint-eval-last-sexp-to-comment
:
(comment
(t/run-tests)
;; => {:test 6, :pass 13, :fail 0, :error 0, :type :summary}
)
Boom! Now we're well underway!
Back in porting-decisions.markdown
, there's some mention of the other Java classes not included in babashka:
The balance of the unincluded classes are used in
cognitect.aws.credentials
to provide auto-refreshing of AWS credentials. As babashka is commonly used for short-lived scripts as opposed to long-running server applications, rather than provide an alternate implementation for credential refresh, I've chosen to omit this functionality. If credential auto-refresh is something I find is useful in a babashka context some time in the future, a solution can be explored at that time.
Seems like we can also live without credential auto-refresh, so let's have a quick look at the net.jmglov.awno.credentials
and see what we'll need to do to make it work.
Looks like we need to rip this bit out, since it uses thready thread stuff that we hate and fear:
(defonce ^:private scheduled-executor-service
(delay
(Executors/newScheduledThreadPool 1 (reify ThreadFactory
(newThread [_ r]
(doto (Thread. r)
(.setName "net.jmglov.awno-api.credentials-provider")
(.setDaemon true)))))))
Now we can simplify refresh!
from this:
(defn ^:skip-wiki refresh!
"For internal use. Don't call directly.
Invokes `(fetch provider)`, resets the `credentials-atom` with and
returns the result.
If the credentials returned by the provider are not valid, resets
both atoms to nil and returns nil."
[credentials-atom scheduled-refresh-atom provider scheduler]
(try
(let [{:keys [::ttl] :as new-creds} (fetch provider)]
(reset! scheduled-refresh-atom
(when ttl
(.schedule ^ScheduledExecutorService scheduler
^Runnable #(refresh! credentials-atom scheduled-refresh-atom provider scheduler)
^long ttl
TimeUnit/SECONDS)))
(reset! credentials-atom new-creds))
(catch Throwable t
(reset! scheduled-refresh-atom nil)
(log/error t "Error fetching credentials."))))
To that:
(defn ^:skip-wiki refresh!
"For internal use. Don't call directly.
Invokes `(fetch provider)`, resets the `credentials-atom` with and
returns the result.
If the credentials returned by the provider are not valid, resets
both atoms to nil and returns nil."
[credentials-atom provider]
(try
(let [new-creds (fetch provider)]
(reset! credentials-atom new-creds))
(catch Throwable t
(log/error t "Error fetching credentials."))))
And then we can rip out this stuff:
(defn cached-credentials-with-auto-refresh
"Returns a CredentialsProvider which wraps `provider`, caching
credentials returned by `fetch`, and auto-refreshing the cached
credentials in a background thread when the credentials include a
::ttl.
Call `stop` to cancel future auto-refreshes.
The default ScheduledExecutorService uses a ThreadFactory that
spawns daemon threads. You can override this by providing your own
ScheduledExecutorService.
Alpha. Subject to change."
([provider]
(cached-credentials-with-auto-refresh provider @scheduled-executor-service))
([provider scheduler]
(let [credentials-atom (atom nil)
scheduled-refresh-atom (atom nil)]
(reify
CredentialsProvider
(fetch [_]
(or @credentials-atom
(refresh! credentials-atom scheduled-refresh-atom provider scheduler)))
Stoppable
(-stop [_]
(-stop provider)
(when-let [r @scheduled-refresh-atom]
(.cancel ^ScheduledFuture r true)))))))
(defn ^:deprecated auto-refreshing-credentials
"Deprecated. Use cached-credentials-with-auto-refresh"
([provider] (cached-credentials-with-auto-refresh provider))
([provider scheduler] (cached-credentials-with-auto-refresh provider scheduler)))
And replace it with a much simpler function:
(defn cached-credentials
"Returns a CredentialsProvider which wraps `provider`, caching
credentials returned by `fetch`, and auto-refreshing the cached
credentials in a background thread when the credentials include a
::ttl.
Call `stop` to cancel future auto-refreshes.
The default ScheduledExecutorService uses a ThreadFactory that
spawns daemon threads. You can override this by providing your own
ScheduledExecutorService.
Alpha. Subject to change."
[provider]
(let [credentials-atom (atom nil)]
(reify
CredentialsProvider
(fetch [_]
(or @credentials-atom
(refresh! credentials-atom provider)))
Stoppable
(-stop [_]
(-stop provider)))))
We can now replace all occurrences of cached-credentials-with-auto-refresh
in the file with cached-credentials
.
And now the stop
function doesn't actually need to do anything:
(defn stop
"no-op"
[_credentials])
With this, C-c C-k
evaluates the namespace without errors, so let's go over to net.jmglov.awno.credentials-test
and see if the tests work. But first, let's rip out anything to do with auto-refreshing credentials. That means replacing all of the cached-credentials-with-auto-refresh
in the file with cached-credentials
and removing the explicit test of auto-refreshing:
(deftest auto-refresh-test
(let [cnt (atom 0)
p (reify credentials/CredentialsProvider
(credentials/fetch [_]
(swap! cnt inc)
{:aws/access-key-id "id"
:aws/secret-access-key "secret"
::credentials/ttl 1}))
creds (credentials/cached-credentials p)]
(credentials/fetch creds)
(Thread/sleep 2500)
(let [refreshed @cnt]
(credentials/stop creds)
(Thread/sleep 1000)
(is (= 3 refreshed) "The credentials have been refreshed.")
(is (= refreshed @cnt) "We stopped the auto-refreshing process."))))
OK, looking good! Let's give it a C-c C-k
:
clojure.lang.ExceptionInfo: Could not find namespace: clojure.tools.logging.test.
{:type :sci/error, :line 5, :column 3, :message "Could not find namespace: clojure.tools.logging.test.", :sci.impl/callstack #object[clojure.lang.Volatile 0x13387b21 {:status :ready, :val ({:line 5, :column 3, :file "/home/jmglov/Documents/code/clojure/awno-api/test/src/net/jmglov/awno/credentials_test.clj", :ns #object[sci.lang.Namespace 0x65a7dc8f "net.jmglov.awno.credentials-test"]})}], :file "/home/jmglov/Documents/code/clojure/awno-api/test/src/net/jmglov/awno/credentials_test.clj"}
at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:129)
sci.impl.analyzer$return_ns_op$reify__4355.eval (analyzer.cljc:1189)
sci.impl.analyzer$return_do$reify__3968.eval (analyzer.cljc:130)
sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
sci.core$eval_form.invokeStatic (core.cljc:329)
[...]
Oh no! It looks like whatever version of clojure.tools.logging
that is included in babashka doesn't have clojure.tools.logging.test
! ๐ญ
For now, let's just comment out the stuff that needs clojure.tools.logging.test
and get on with our lives:
(ns net.jmglov.awno.credentials-test
(:require [clojure.test :as t :refer [deftest testing use-fixtures is]]
[clojure.java.io :as io]
#_[clojure.tools.logging.test :refer [with-log logged?]]
[net.jmglov.awno.credentials :as credentials]
[net.jmglov.awno.util :as u]
[net.jmglov.awno.test.utils :as tu]
[net.jmglov.awno.ec2-metadata-utils :as ec2-metadata-utils]
[net.jmglov.awno.ec2-metadata-utils-test :as ec2-metadata-utils-test])
(:import (java.time Instant)))
;; [...]
#_(deftest valid-credentials-test
(with-log
(credentials/valid-credentials nil "x provider")
(is (logged? 'net.jmglov.awno.credentials :debug (str "Unable to fetch credentials from x provider."))))
(with-log
(credentials/valid-credentials {:aws/access-key-id "id"
:aws/secret-access-key "secret"}
"x provider")
(is (logged? 'net.jmglov.awno.credentials :debug (str "Fetched credentials from x provider.")))))
After doing that, the namespace evaluates just fine, so let's try running the tests:
(comment
(t/run-tests)
;; => java.lang.RuntimeException: Could not find net.jmglov.awno_http.edn on classpath. user /home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/http.clj:76:9
)
Oh sweet mother of mercy what now?
Since our error mentioned HTTP by name, let's look up HTTP in porting-decisions.markdown
:
The aws-api library defines a protocol (
cognitect.aws.http/HttpClient
) to provide an interface between the aws-api data transformation logic and the specific HTTP client implementation. The aws-api includes an implementation for thecom.cognitect/http-client
library. Interfaces are great: we can provide our own implementations of thecognitect.aws.http/HttpClient
interface based on the various HTTP clients included in babashka.
Alrighty, looks like we're gonna need to implement an HTTP client. Let's have a look at how this is done in awyeah-api by peeking in src/com/grzm/awyeah/http.clj. It seems by and large the same as aws-api's src/cognitect/aws/http.clj, with exception of some class loader stuff, which is also mentioned in porting-decisions.markdown
:
There are a few other compatiblity issues, such as the use of
java.lang.ClassLoader::getResources
incognitect.aws.http/configured-client
, and replacing^int x
hinting with explicit(int x)
casts.
Whilst java.lang.ClassLoader::getResources
isn't mentioned directly, there are some class loader shenanigans happening here:
(defn- configured-client
"If a single net.jmglov.awno_http.edn is found on the classpath,
returns the symbol bound to :constructor-var.
Throws if 0 or > 1 net.jmglov.awno_http.edn files are found.
"
[]
(let [cl (.. Thread currentThread getContextClassLoader)
cfgs (enumeration-seq (.getResources cl "net.jmglov.awno_http.edn"))]
(case (count cfgs)
0 (throw (RuntimeException. "Could not find net.jmglov.awno_http.edn on classpath."))
1 (-> cfgs first read-config :constructor-var)
(throw (ex-info "Found too many http-client cfgs. Pick one." {:config cfgs})))))
Let's just follow grzm's lead and rip it out:
(defn- configured-client
"If a single com_grzm_awyeah_http.edn is found on the classpath,
returns the symbol bound to :constructor-var.
Throws if 0 or > 1 com_grzm_awyeah_http.edn files are found.
"
[]
(try
(-> (io/resource "net_jmglov_awno_http.edn") read-config :constructor-var)
(catch Throwable _
(throw (RuntimeException. "Could not find com_grzm_awyeah_http.edn on classpath.")))))
This dynaload
stuff also looks a bit unnecessary, so let's flush it as well by turning this:
(defn resolve-http-client
[http-client-or-sym]
(let [c (or (when (symbol? http-client-or-sym)
(let [ctor @(dynaload/load-var http-client-or-sym)]
(ctor)))
http-client-or-sym
(let [ctor @(dynaload/load-var (configured-client))]
(ctor)))]
(when-not (client? c)
(throw (ex-info "not an http client" {:provided http-client-or-sym
:resolved c})))
c))
Into that:
(defn resolve-http-client
[http-client-or-sym]
(let [c (or (when (symbol? http-client-or-sym)
(let [ctor (requiring-resolve http-client-or-sym)]
(ctor)))
http-client-or-sym
(let [ctor (requiring-resolve (configured-client))]
(ctor)))]
(when-not (client? c)
(throw (ex-info "not an http client" {:provided http-client-or-sym
:resolved c})))
c))
This now means that we're gonna need clojure.java.io
, so let's pop up to the ns
form and require it. And whilst we're at it, we can get rid of this suspicious net.jmglov.awno.dynaload
require, and kill the associated source file with fire! That leaves us with an ns
form like this:
(ns ^:skip-wiki net.jmglov.awno.http
"Impl, don't call directly."
(:require [clojure.edn :as edn]
[clojure.core.async :as a]
[clojure.java.io :as io]))
We're also going to need the net_jmglov_awno_http.edn
file mentioned above. We can just modify the original cognitect_aws_http.edn
, so this:
{:constructor-var net.jmglov.awno.http.cognitect/create}
Becomes that:
{:constructor-var net.jmglov.awno.http.awno/create}
Once this is done, we just need to rename the file to net_jmglov_awno_http.edn
and Robert should be one of our parents' brother. Oh yeah, and since we changed some stuff in resources, a C-c M-r
to invoke cider-restart
is sadly required. ๐ข
Continuing on our mission to civilise HTTP, let's turn our roving eye to the net.jmglov.awno.http.awno
namespace, which is sadly not there, because our original projectile-replace
of cognitect.aws
with net.jmglov.awno
left us with a file called src/net/jmglov/awno/http/cognitect.clj
with a namespace of net.jmglov.awno.http.cognitect
. We can fix the name easily enough, but looking at awyeah-api's src/com/grzm/awyeah/http/awyeah.clj
, we see that he's replaced cognitect.http-client
with a custom com.grzm.awyeah.http-client
. Let's follow the same pattern:
(ns ^:skip-wiki net.jmglov.awno.http.awno
(:require [net.jmglov.awno.http :as aws]
[net.jmglov.awno.http-client :as impl]))
(set! *warn-on-reflection* true)
(defn create
[]
(let [c (impl/create nil)]
(reify aws/HttpClient
(-submit [_ request channel]
(impl/submit c request channel))
(-stop [_]
(impl/stop c)))))
We'll also need to rename the file from cognitect.clj
to awno.clj
. Having done this, it's time to write net.jmglov.awno.http-client
.
Let's start by straight up copying awyeah-api's com/grzm/awyeah/http_client.clj and replacing all the occurrences of com.grzm.awyeah
with net.jmglov.awno
to jmglovify it:
(ns net.jmglov.awno.http-client
(:require
[clojure.core.async :refer [put!] :as a]
[clojure.spec.alpha :as s]
[net.jmglov.awno.http-client.client :as client]
[net.jmglov.awno.http-client.specs])
(:import
(clojure.lang ExceptionInfo)
(java.net URI)
(java.net.http HttpClient
HttpClient$Redirect
HttpHeaders
HttpRequest
HttpRequest$Builder
HttpRequest$BodyPublishers
HttpResponse
HttpResponse$BodyHandlers)
(java.nio ByteBuffer)
(java.time Duration)
(java.util.function Function)))
(set! *warn-on-reflection* true)
(defn submit
"Submit an http request, channel will be filled with response. Returns ch.
Request map:
:server-name string
:server-port integer
:uri string
:query-string string, optional
:request-method :get/:post/:put/:head
:scheme :http or :https
:headers map from downcased string to string
:body ByteBuffer, optional
:net.jmglov.awno.http-client/timeout-msec opt, total request send/receive timeout
:net.jmglov.awno.http-client/meta opt, data to be added to the response map
content-type must be specified in the headers map
content-length is derived from the ByteBuffer passed to body
Response map:
:status integer HTTP status code
:body ByteBuffer, optional
:header map from downcased string to string
:net.jmglov.awno.http-client/meta opt, data from the request
On error, response map is per cognitect.anomalies"
([client request]
(submit client request (a/chan 1)))
([client request ch]
(s/assert ::submit-request request)
(client/submit client request ch)))
(def method-string
{:get "GET"
:post "POST"
:put "PUT"
:head "HEAD"
:delete "DELETE"
:patch "PATCH"})
(defn byte-buffer->byte-array
[^ByteBuffer bbuf]
(.rewind bbuf)
(let [arr (byte-array (.remaining bbuf))]
(.get bbuf arr)
arr))
(defn flatten-headers [headers]
(->> headers
(mapcat (fn [[nom val]]
(if (coll? val)
(map (fn [v] [(name nom) v]) val)
[[(name nom) val]])))))
;; "host" is a restricted header.
;; The host header is part of the AWS signed headers signature,
;; so it's included in the list of headers for request processing,
;; but we let the java.net.http HttpRequest assign the host header
;; from the URI rather than setting it directly.
(def restricted-headers #{"host"})
(defn add-headers
[^HttpRequest$Builder builder headers]
(doseq [[nom val] (->> (flatten-headers headers)
(remove (fn [[nom _]] (restricted-headers nom))))]
(.header builder nom val))
builder)
(defn map->http-request
[{:keys [scheme server-name server-port uri query-string
request-method headers body]
:or {scheme "https"}
:as m}]
(let [uri (URI. (str (name scheme)
"://"
server-name
(some->> server-port (str ":"))
uri
(some->> query-string (str "?"))))
method (method-string request-method)
bp (if body
(HttpRequest$BodyPublishers/ofByteArray (byte-buffer->byte-array body))
(HttpRequest$BodyPublishers/noBody))
builder (-> (HttpRequest/newBuilder uri)
(.method ^String method bp))]
(when (seq headers)
(add-headers builder headers))
(when (::timeout-msec m)
(.timeout builder (Duration/ofMillis (::timeout-msec m))))
(.build builder)))
(defn error->anomaly [^Throwable t]
{:cognitect.anomalies/category :cognitect.anomalies/fault
:cognitect.anomalies/message (.getMessage t)
::throwable t})
(defn header-map [^HttpHeaders headers]
(->> headers
(.map)
(map (fn [[k v]] [k (if (< 1 (count v))
(into [] v)
(first v))]))
(into {})))
(defn response-body?
[^HttpRequest http-request]
((complement #{"HEAD"}) (.method http-request)))
(defn response-map
[^HttpRequest http-request ^HttpResponse http-response]
(let [body (when (response-body? http-request)
(.body http-response))]
(cond-> {:status (.statusCode http-response)
:headers (header-map (.headers http-response))}
body (assoc :body (ByteBuffer/wrap body)))))
(defrecord Client
[^HttpClient http-client pending-ops pending-ops-limit]
client/Client
(-submit [_ request ch]
(if (< pending-ops-limit (swap! pending-ops inc))
(do
(put! ch (merge {:cognitect.anomalies/category :cognitect.anomalies/busy
:cognitect.anomalies/message (str "Ops limit reached: " pending-ops-limit)
:pending-ops-limit pending-ops-limit}
(select-keys request [::meta])))
(swap! pending-ops dec))
(try
(let [http-request (map->http-request request)]
(-> (.sendAsync http-client http-request (HttpResponse$BodyHandlers/ofByteArray))
(.thenApply
(reify Function
(apply [_ http-response]
(put! ch (merge (response-map http-request http-response)
(select-keys request [::meta]))))))
(.exceptionally
(reify Function
(apply [_ e]
(let [cause (.getCause ^Exception e)
t (if (instance? ExceptionInfo cause) cause e)]
(put! ch (merge (error->anomaly t) (select-keys request [::meta]))))))))
(swap! pending-ops dec))
(catch Throwable t
(put! ch (merge (error->anomaly t) (select-keys request [::meta])))
(swap! pending-ops dec))))
ch))
(defn create
[{:keys [connect-timeout-msecs
pending-ops-limit]
:or {connect-timeout-msecs 5000
pending-ops-limit 64}
:as _config}]
(let [http-client (.build (-> (HttpClient/newBuilder)
(.connectTimeout (Duration/ofMillis connect-timeout-msecs))
(.followRedirects HttpClient$Redirect/NORMAL)))]
(->Client http-client (atom 0) pending-ops-limit)))
(defn stop
"no-op. Implemented for compatibility"
[^Client _client])
Now we need to create those net.jmglov.awno.http-client.client
and net.jmglov.awno.http-client.specs
namespaces, so we can make copies of com/grzm/awyeah/http_client/client.clj and com/grzm/awyeah/http_client/specs.clj and replace all the occurrences of com.grzm.awyeah
with net.jmglov.awno
, giving us this:
(ns net.jmglov.awno.http-client.client)
(defprotocol Client
(-submit [_ request ch]))
(defn submit [client request ch]
(-submit client request ch))
And this:
(ns net.jmglov.awno.http-client.specs
(:require
[clojure.spec.alpha :as s])
(:import
(java.nio ByteBuffer)))
(defn- keyword-or-non-empty-string? [x]
(or (keyword? x)
(and (string? x) (not-empty x))))
(s/def :net.jmglov.awno.http-client/server-name string?)
(s/def :net.jmglov.awno.http-client/server-port int?)
(s/def :net.jmglov.awno.http-client/uri string?)
(s/def :net.jmglov.awno.http-client/request-method keyword?)
(s/def :net.jmglov.awno.http-client/scheme keyword-or-non-empty-string?)
(s/def :net.jmglov.awno.http-client/timeout-msec int?)
(s/def :net.jmglov.awno.http-client/meta map?)
(s/def :net.jmglov.awno.http-client/body (s/nilable #(instance? ByteBuffer %)))
(s/def :net.jmglov.awno.http-client/query-string string?)
(s/def :net.jmglov.awno.http-client/headers map?)
(s/def :net.jmglov.awno.http-client/submit-request
(s/keys :req-un [:net.jmglov.awno.http-client/server-name
:net.jmglov.awno.http-client/server-port
:net.jmglov.awno.http-client/uri
:net.jmglov.awno.http-client/request-method
:net.jmglov.awno.http-client/scheme]
:opt [:net.jmglov.awno.http-client/timeout-msec
:net.jmglov.awno.http-client/meta]
:opt-un [:net.jmglov.awno.http-client/body
:net.jmglov.awno.http-client/query-string
:net.jmglov.awno.http-client/headers]))
(s/def :net.jmglov.awno.http-client/status int?)
(s/def :net.jmglov.awno.http-client/submit-http-response
(s/keys :req-un [:net.jmglov.awno.http-client/status]
:opt [:net.jmglov.awno.http-client/meta]
:opt-un [:net.jmglov.awno.http-client/body
:net.jmglov.awno.http-client/headers]))
(s/def :net.jmglov.awno.http-client/error keyword?)
(s/def :net.jmglov.awno.http-client/throwable #(instance? Throwable %))
(s/def :net.jmglov.awno.http-client/submit-error-response
(s/keys :req [:net.jmglov.awno.http-client/error]
:opt [:net.jmglov.awno.http-client/throwable
:net.jmglov.awno.http-client/meta]))
(s/def :net.jmglov.awno.http-client/submit-response
(s/or :http-response :net.jmglov.awno.http-client/submit-http-response
:error-response :net.jmglov.awno.http-client/submit-error-response))
Having done everything that's obvious from porting-decisions.markdown
, we might as well just see if it works. But before we do that, how about a...
See you next week for the thrilling conclusion of the trainwreck that is me attempting to replace an oil filter by completely disassembling the car and making my own oil filter out of paper towels and a coathanger I found lying around. What could possibly go wrong?
Published: 2023-11-11
I'm giving a talk on Blambda later this month at a local meetup group or two, and since I'm an experienced speaker, the talk is completely prepared, so now I have three weeks to rehearse it... in an alternate reality where I'm not a huge procrastinator, that is.
In this reality, I haven't even written an outline for the talk, despite convincing myself that I was going to do that over the holiday break. I was able to convince myself that before I write the outline, I should just hack on Blambda a little more to make sure it actually works. ๐
A while back, I read a post on Cyprien Pannier's blog called Serverless site analytics with Clojure nbb and AWS. It was a fantastic post, except for one thing: it was based on the hated nbb, surely borkdude's greatest sin!
I decided I could make the world a better place by doing the same thing, but using Babashka instead of the unholy nbb. Now that Blambda is all grown up and has a v0.1.0 release, it was the work of but a moment (for a definition of "moment" that spans several days) to implement a site analyser!
If you're the impatient sort, you can just take a look at examples/site-analyser in the Blambda repo, but if you wanna see how the sausage is made, buckle up, 'cause Kansas is going bye-bye!
Any great Babashka project starts with an empty directory, so let's make one!
$ mkdir site-analyser
$ cd site-analyser
We'll now add a bb.edn
so we can start playing with Blambda:
{:deps {net.jmglov/blambda
{:git/url "https://github.com/jmglov/blambda.git"
:git/tag "v0.1.0"
:git/sha "b80ac1d"}}
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {}))
blambda {:doc "Controls Blambda runtime and layers"
:task (blambda/dispatch config)}}}
This is enough to get us up and running with Blambda! Let's ask for help to see how to get going:
$ bb blambda help
Usage: bb blambda <subcommand> <options>
All subcommands support the options:
--work-dir <dir> .work Working directory
--target-dir <dir> target Build output directory
Subcommands:
build-runtime-layer: Builds Blambda custom runtime layer
--bb-arch <arch> amd64 Architecture to target (use amd64 if you don't care)
--runtime-layer-name <name> blambda Name of custom runtime layer in AWS
--bb-version <version> 1.0.168 Babashka version
...
Might as well try to build the custom runtime layer, which is what allows lambdas to be written in Babashka in the first place:
$ bb blambda build-runtime-layer
Building custom runtime layer: /tmp/site-analyser/target/blambda.zip
Downloading https://github.com/babashka/babashka/releases/download/v1.0.168/babashka-1.0.168-linux-amd64-static.tar.gz
Decompressing .work/babashka-1.0.168-linux-amd64-static.tar.gz to .work
Adding file: bootstrap
Adding file: bootstrap.clj
Compressing custom runtime layer: /tmp/site-analyser/target/blambda.zip
adding: bb (deflated 70%)
adding: bootstrap (deflated 53%)
adding: bootstrap.clj (deflated 62%)
$ ls -sh target/
total 21M
21M blambda.zip
Cool, looks legit!
AWS being AWS, of course they had to go and design their own CPU. ๐
It's called Graviton, and it's based on the ARM architecture. A little over a year ago, it became possible to run lambda functions on Graviton, which AWS claims delivers "up to 19 percent better performance at 20 percent lower cost". That's party I definitely wanna go to, and luckily for me, it just so happens that Babashka runs on ARM! ๐ I love borkdude so much that I can almost forgive him for the dark alchemy that wrought nbb! Almost.
We can instruct Blambda to use the ARM version of Babashka by adding a key to the config
map in bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"}))
...
Let's rebuild the runtime layer and see what's up:
$ bb blambda build-runtime-layer
Building custom runtime layer: /tmp/site-analyser/target/blambda.zip
Downloading https://github.com/babashka/babashka/releases/download/v1.0.168/babashka-1.0.168-linux-aarch64-static.tar.gz
Decompressing .work/babashka-1.0.168-linux-aarch64-static.tar.gz to .work
Adding file: bootstrap
Adding file: bootstrap.clj
Compressing custom runtime layer: /tmp/site-analyser/target/blambda.zip
updating: bb (deflated 73%)
updating: bootstrap (deflated 53%)
updating: bootstrap.clj (deflated 62%)
That babashka-1.0.168-linux-aarch64-static.tar.gz
looks promising!
OK, so we have a custom runtime. That's awesome and all, but without a lambda function, a runtime is a bit passรฉ, don't you think? Let's remedy this with the simplest of lambdas, the infamous Hello World. Except let's make it say "Hello Blambda" instead to make it more amazing!
All we need to accomplish this is a simple handler. Let's create a src/handler.clj
and drop the following into it:
(ns handler)
(defn handler [{:keys [name] :or {name "Blambda"} :as event} context]
(prn {:msg "Invoked with event",
:data {:event event}})
{:greeting (str "Hello " name "!")})
Now we'll need to tell Blambda what our lambda function should be called, where to find the sources, and what handler function to use. Back in bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:source-files ["handler.clj"]}))
...
Now that we've done this, we can build the lambda:
$ bb blambda build-lambda
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
adding: handler.clj (deflated 25%)
$ ls -sh target/
total 22M
22M blambda.zip 4.0K site-analyser.zip
Amazing! I love the fact that the entire lambda artifact is only 4 KB!
Of course, this is still academic until we deploy the function to the world, so let's stop messing about and do it! For the rest of this post, I am going to assume that one of the following applies to you:
Since one of those two options is true, let's generate ourselves some Terraform config!
$ bb blambda terraform write-config
Writing lambda layer config: /tmp/site-analyser/target/blambda.tf
Writing lambda layer vars: /tmp/site-analyser/target/blambda.auto.tfvars
Writing lambda layers module: /tmp/site-analyser/target/modules/lambda_layer.tf
$ find target/
target/
target/site-analyser.zip
target/blambda.zip
target/modules
target/modules/lambda_layer.tf
target/blambda.auto.tfvars
target/blambda.tf
Note the blambda.tf
, blambda.auto.tfvars
, and modules/lambda_layer.tf
files there. If you're a Terraform aficionado, feel free to take a look at these; I'll just give you the highlights here.
blambda.tf
defines the following resources:
aws_lambda_function.lambda
- the lambda function itselfaws_cloudwatch_log_group.lambda
- a CloudWatch log group for the lambda function to log toaws_iam_role.lambda
- the IAM role that the lambda function will assumeaws_iam_policy.lambda
- an IAM policy describing what the lambda function is allowed to do (in this case, just write logs to its own log group)aws_iam_role_policy_attachment.lambda
- a virtual Terraform resource that represents the attachment of the policy to the rolemodule.runtime.aws_lambda_layer_version.layer
- the custom runtime layerblambda.auto.tfvars
sets various Terraform variables. The details are too boring to relate, but you are welcome to look at the file if your curiosity overwhelms you. ๐
modules/lambda_layer.tf
defines a Terraform module that creates a lambda layer. The reason it's in a module and not just inline in the blambda.tf
will become apparent later.
OK, now that I've gone into what is almost certainly too much detail on stuff that you almost certainly don't care about, let's just deploy the function!
$ bb blambda terraform apply
Initializing modules...
- runtime in modules
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.48.0...
- Installed hashicorp/aws v4.48.0 (signed by HashiCorp)
[...]
Terraform has been successfully initialized!
[...]
Terraform will perform the following actions:
# aws_cloudwatch_log_group.lambda will be created
+ resource "aws_cloudwatch_log_group" "lambda" {
# aws_iam_policy.lambda will be created
+ resource "aws_iam_policy" "lambda" {
# aws_iam_role.lambda will be created
+ resource "aws_iam_role" "lambda" {
# aws_iam_role_policy_attachment.lambda will be created
+ resource "aws_iam_role_policy_attachment" "lambda" {
# aws_lambda_function.lambda will be created
+ resource "aws_lambda_function" "lambda" {
# module.runtime.aws_lambda_layer_version.layer will be created
+ resource "aws_lambda_layer_version" "layer" {
[...]
Plan: 6 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
Let's be brave and type "yes" and then give the Enter key a resounding smackaroo! We should see something along the lines of this:
aws_cloudwatch_log_group.lambda: Creating...
aws_iam_role.lambda: Creating...
module.runtime.aws_lambda_layer_version.layer: Creating...
aws_cloudwatch_log_group.lambda: Creation complete after 1s [id=/aws/lambda/site-analyser]
aws_iam_policy.lambda: Creating...
aws_iam_policy.lambda: Creation complete after 1s [id=arn:aws:iam::123456789100:policy/site-analyser]
aws_iam_role.lambda: Creation complete after 2s [id=site-analyser]
aws_iam_role_policy_attachment.lambda: Creating...
aws_iam_role_policy_attachment.lambda: Creation complete after 1s [id=site-analyser-20230103173233475200000001]
module.runtime.aws_lambda_layer_version.layer: Still creating... [10s elapsed]
module.runtime.aws_lambda_layer_version.layer: Still creating... [20s elapsed]
module.runtime.aws_lambda_layer_version.layer: Still creating... [30s elapsed]
module.runtime.aws_lambda_layer_version.layer: Creation complete after 31s [id=arn:aws:lambda:eu-west-1:123456789100:layer:blambda:30]
aws_lambda_function.lambda: Creating...
aws_lambda_function.lambda: Still creating... [10s elapsed]
aws_lambda_function.lambda: Creation complete after 11s [id=site-analyser]
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
OK, let's try invoking the function:
$ aws lambda invoke --function-name site-analyser /tmp/response.json
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
$ cat /tmp/response.json
{"greeting":"Hello Blambda!"}
According to the handler, we can also pass a name in:
$ aws lambda invoke --function-name site-analyser --payload '{"name": "Dear Reader"}' /tmp/response.json
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
$ cat /tmp/response.json
{"greeting":"Hello Dear Reader!"}
Looks like we're live in the cloud!
It's pretty annoying to have to use the AWS CLI to invoke our function, what with all of the writing the response to a file and all that jazz. Luckily, Lambda function URLs offer us a way out of this never-ending agony. All we need to do is add one simple Terraform resource.
Let's create a tf/main.tf
and define our function URL:
resource "aws_lambda_function_url" "lambda" {
function_name = aws_lambda_function.lambda.function_name
authorization_type = "NONE"
}
output "function_url" {
value = aws_lambda_function_url.lambda.function_url
}
When using a function URL, the event passed to the lambda function looks a little different. Referring to the Request and response payloads page in the AWS Lambda developer guide, we hone in on the important bits:
Request
{
"queryStringParameters": {
"parameter1": "value1,value2",
"parameter2": "value"
},
"requestContext": {
"http": {
"method": "POST",
"path": "/my/path",
}
}
}
Response
{
"statusCode": 201,
"headers": {
"Content-Type": "application/json",
"My-Custom-Header": "Custom Value"
},
"body": "{ \"message\": \"Hello, world!\" }"
}
Let's update our handler to log the method and path and grab the optional name
from the query params. Our new src/handler.clj
now looks like this:
(ns handler
(:require [cheshire.core :as json]))
(defn log [msg data]
(prn (assoc data :msg msg)))
(defn handler [{:keys [queryStringParameters requestContext] :as event} _context]
(log "Invoked with event" {:event event})
(let [{:keys [method path]} (:http requestContext)
{:keys [name] :or {name "Blambda"}} queryStringParameters]
(log (format "Request: %s %s" method path)
{:method method, :path path, :name name})
{:statusCode 200
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:greeting (str "Hello " name "!")})}))
The final step before we can deploy this gem is letting Blambda know that we want to include some extra Terraform config. For this, we set the :extra-tf-config
key in bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:source-files ["handler.clj"]
:extra-tf-config ["tf/main.tf"]}))
...
Now that all of this is done, let's rebuild our lambda, regenerate our Terraform config, and deploy away!
$ bb blambda build-lambda
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
updating: handler.clj (deflated 43%)
$ bb blambda terraform write-config
Copying Terraform config tf/main.tf
Writing lambda layer config: /tmp/site-analyser/target/blambda.tf
Writing lambda layer vars: /tmp/site-analyser/target/blambda.auto.tfvars
Writing lambda layers module: /tmp/site-analyser/target/modules/lambda_layer.tf
$ bb blambda terraform apply
Terraform will perform the following actions:
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
# aws_lambda_function_url.lambda will be created
+ resource "aws_lambda_function_url" "lambda" {
Plan: 1 to add, 1 to change, 0 to destroy.
Changes to Outputs:
+ function_url = (known after apply)
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_lambda_function.lambda: Modifications complete after 7s [id=site-analyser]
aws_lambda_function_url.lambda: Creating...
aws_lambda_function_url.lambda: Creation complete after 1s [id=site-analyser]
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
Looks great! The function_url
is the base URL we'll use to make HTTP requests to our lambda. We can try it out with curl:
$ export BASE_URL=https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/
$ curl $BASE_URL
{"greeting":"Hello Blambda!"}
$ curl $BASE_URL?name='Dear%20Reader'
{"greeting":"Hello Dear Reader!"}
Before we move on, let's take a quick look at our logs. Lambda functions automatically log anything printed to standard output to a log stream in a log group named /aws/lambda/{function_name}
(as long as they have permission to do so, which Blambda takes care of for us in the default IAM policy). Let's see what log streams we have in our /aws/lambda/site-analyser
group:
$ aws logs describe-log-streams --log-group /aws/lambda/site-analyser \
| jq '.logStreams | .[].log[18/1807]e'
"2023/01/03/[$LATEST]f8a5d5be9e0c4d34bcf6c8bb55e9c577"
"2023/01/04/[$LATEST]98a0ab46e2994cdda668124ccae610fc"
We have two streams since we've tested our lambda twice (requests around the same time are batched into a single log stream). Let's pick the most recent one and see what it says:
$ aws logs get-log-events \
--log-group /aws/lambda/site-analyser \
--log-stream '2023/01/04/[$LATEST]98a0ab46e2994cdda668124ccae610fc' \
| jq -r '.events|.[].message'
Starting Babashka:
/opt/bb -cp /var/task /opt/bootstrap.clj
Loading babashka lambda handler: handler/handler
Starting babashka lambda event loop
START RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a Version: $LATEST
{:event {:version "2.0", :routeKey "$default", :rawPath "/", ...}
{:method "GET", :path "/", :name "Blambda", :msg "Request: GET /"}
END RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a
REPORT RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a Duration: 56.87 ms Billed Duration: 507 ms Memory Size: 512 MB Max Memory Used: 125 MB Init Duration: 450.02 ms
START RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4 Version: $LATEST
{:event {:version "2.0", :routeKey "$default", :rawPath "/", ...}
{:method "GET", :path "/", :name "Dear Reader", :msg "Request: GET /"}
END RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4
REPORT RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4 Duration: 2.23 ms Billed Duration: 3 ms Memory Size: 512 MB Max Memory Used: 125 MB
There are a few things of interest here. First of all, we can see the Blambda custom runtime starting up:
Starting Babashka:
/opt/bb -cp /var/task /opt/bootstrap.clj
Loading babashka lambda handler: handler/handler
Starting babashka lambda event loop
This only happens on a so-called "cold start", which is when there is no lambda instance available to serve an invocation request. We always have a cold start on the first invocation of a lambda after it's deployed (i.e. on every code change), and then the lambda will stay warm for about 15 minutes after each invocation.
Next in our logs, we see the first test request we made:
START RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a Version: $LATEST
{:event {:version "2.0", :routeKey "$default", :rawPath "/", ...}
{:method "GET", :path "/", :name "Blambda", :msg "Request: GET /"}
END RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a
REPORT RequestId: 09a33775-0151-4d83-9a9b-b21b8add2b3a Duration: 56.87 ms Billed Duration: 507 ms Memory Size: 512 MB Max Memory Used: 125 MB Init Duration: 450.02 ms
We can see the full event that is passed to the lambda by the function URL in the first EDN log line (which we should probably switch to JSON for compatibility will common log aggregation tools), then the log statement we added for the method, path, and name parameter. Finally, we get a report on the lambda invocation. We can see that it took 450 ms to initialise the runtime (seems a bit long; maybe increasing the memory size of our function would help), then 56.87 ms for the function invocation itself.
Let's compare that to the second invocation, the one where we added name
to the query parameters:
START RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4 Version: $LATEST
{:event {:version "2.0", :routeKey "$default", :rawPath "/", ...}
{:method "GET", :path "/", :name "Dear Reader", :msg "Request: GET /"}
END RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4
REPORT RequestId: 0fca5efe-2644-4ebd-80ce-ebb390ffcaf4 Duration: 2.23 ms Billed Duration: 3 ms Memory Size: 512 MB Max Memory Used: 125 MB
Note that we don't even have an init duration in this invocation, since the lambda was warm. Note also that the request duration was 2.23 ms!
Just for fun, let's make a few more requests and look at the durations.
$ for i in $(seq 0 9); do curl $BASE_URL?name="request%20$i"; echo; done
{"greeting":"Hello request 0!"}
{"greeting":"Hello request 1!"}
{"greeting":"Hello request 2!"}
{"greeting":"Hello request 3!"}
{"greeting":"Hello request 4!"}
{"greeting":"Hello request 5!"}
{"greeting":"Hello request 6!"}
{"greeting":"Hello request 7!"}
{"greeting":"Hello request 8!"}
{"greeting":"Hello request 9!"}
$ aws logs describe-log-streams \
--log-group /aws/lambda/site-analyser \
| jq '.logStreams | .[].logStreamName'
"2023/01/03/[$LATEST]f8a5d5be9e0c4d34bcf6c8bb55e9c577"
"2023/01/04/[$LATEST]6532afd4465240dcb3f105abe2bcc250"
"2023/01/04/[$LATEST]98a0ab46e2994cdda668124ccae610fc"
Hrm, 2023/01/04/[$LATEST]98a0ab46e2994cdda668124ccae610fc
was the stream we looked at last time, so let's assume that 2023/01/04/[$LATEST]6532afd4465240dcb3f105abe2bcc250
has our latest requests:
$ aws logs get-log-events \
--log-group /aws/lambda/site-analyser \
--log-stream '2023/01/04/[$LATEST][7/1936]465240dcb3f105abe2bcc250' \
| jq -r '.events | .[].message' \
| grep '^REPORT'
REPORT RequestId: 4ee5993e-6d21-45cd-9b05-b31ea34d993f Duration: 54.32 ms Billed Duration: 505 ms Memory Size: 512 MB Max Memory Used: 125 MB Init Duration: 450.45 ms
REPORT RequestId: 490b82c7-bca1-4427-8d07-ece41444ce2c Duration: 1.81 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 125 MB
REPORT RequestId: ca243a59-75b9-4192-aa91-76569765956a Duration: 3.94 ms Billed Duration: 4 ms Memory Size: 512 MB Max Memory Used: 126 MB
REPORT RequestId: 9d0981f8-3c48-45a5-bf8b-c37ed57e0f95 Duration: 1.77 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 126 MB
REPORT RequestId: 2d5cca3f-752d-4407-99cd-bbb89ca74983 Duration: 1.73 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 126 MB
REPORT RequestId: 674912af-b9e0-4308-b303-e5891a459ad1 Duration: 1.65 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 126 MB
REPORT RequestId: d8efbec2-de6e-491d-b4c6-ce58d02225f1 Duration: 1.67 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 126 MB
REPORT RequestId: c2a9246d-e3c4-40fa-9eb9-82fc200e6425 Duration: 1.64 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 127 MB
REPORT RequestId: cbc2f1cd-23cf-4b26-87d4-0272c097956c Duration: 1.72 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 127 MB
REPORT RequestId: bae2e73b-4b21-4427-b0bd-359301722086 Duration: 1.73 ms Billed Duration: 2 ms Memory Size: 512 MB Max Memory Used: 127 MB
Again, we see a cold start (because it apparently took me more than 15 minutes to write the part of the post since my first two test requests), then 9 more requests with durations mostly under 2 msโdunno why there's a 3.94 ms outlier, but this is hardly a scientific benchmark. ๐
OK, we've now got a lambda that can listen to HTTP requests. To turn it into the site analyser that was promised at the beginning of this blog post, we'll define a simple HTTP API:
POST /track?url={url}
- increment the number of views for the specified URLGET /dashboard
- display a simple HTML dashboard showing the number of views of each URL for the past 7 daysIn order to implement the /track
endpoint, we'll need somewhere to store the counters, and what better place than DynamoDB? We'll create a simple table with the date as the partition key and the url as the range key, which we can add to our tf/main.tf
like so:
resource "aws_dynamodb_table" "site_analyser" {
name = "site-analyser"
billing_mode = "PAY_PER_REQUEST"
hash_key = "date"
range_key = "url"
attribute {
name = "date"
type = "S"
}
attribute {
name = "url"
type = "S"
}
}
We'll also need to give the lambda permissions to update and query this table, which means we'll need to define a custom IAM policy. ๐ญ
Oh well, let's bite the bullet and add it to tf/main.tf
:
resource "aws_iam_role" "lambda" {
name = "site-analyser-lambda"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "lambda.amazonaws.com"
}
}
]
})
}
resource "aws_iam_policy" "lambda" {
name = "site-analyser-lambda"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"logs:CreateLogStream",
"logs:PutLogEvents"
]
Resource = "${aws_cloudwatch_log_group.lambda.arn}:*"
},
{
Effect = "Allow"
Action = [
"dynamodb:Query",
"dynamodb:UpdateItem",
]
Resource = aws_dynamodb_table.site_analyser.arn
}
]
})
}
resource "aws_iam_role_policy_attachment" "lambda" {
role = aws_iam_role.lambda.name
policy_arn = aws_iam_policy.lambda.arn
}
Since Blambda won't be automatically generating a policy for us, we'll need to add a statement to the policy giving the lambda permission to write to CloudWatch Logs:
Statement = [
{
Effect = "Allow"
Action = [
"logs:CreateLogStream",
"logs:PutLogEvents"
]
Resource = "${aws_cloudwatch_log_group.lambda.arn}:*"
},
...
and another one giving the lambda permissions to use the DynamoDB table:
Statement = [
...
{
Effect = "Allow"
Action = [
"dynamodb:Query",
"dynamodb:UpdateItem",
]
Resource = aws_dynamodb_table.site_analyser.arn
}
]
Finally, we need to instruct Blambda that we'll be providing our own IAM role by adding the :lambda-iam-role
key to bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:lambda-iam-role "${aws_iam_role.lambda.arn}"
:source-files ["handler.clj"]
:extra-tf-config ["tf/main.tf"]}))
...
Before we implement the tracker, let's make sure that the new Terraform stuff we did all works:
$ bb blambda terraform write-config
Copying Terraform config tf/main.tf
Writing lambda layer config: /tmp/site-analyser/target/blambda.tf
Writing lambda layer vars: /tmp/site-analyser/target/blambda.auto.tfvars
Writing lambda layers module: /tmp/site-analyser/target/modules/lambda_layer.tf
$ bb blambda terraform apply
[...]
Terraform will perform the following actions:
# aws_dynamodb_table.site_analyser will be created
+ resource "aws_dynamodb_table" "site_analyser" {
# aws_iam_policy.lambda must be replaced
-/+ resource "aws_iam_policy" "lambda" {
# aws_iam_role.lambda must be replaced
-/+ resource "aws_iam_role" "lambda" {
# aws_iam_role_policy_attachment.lambda must be replaced
-/+ resource "aws_iam_role_policy_attachment" "lambda" {
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
Plan: 4 to add, 1 to change, 3 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_iam_role.lambda: Creating...
aws_dynamodb_table.site_analyser: Creating...
aws_iam_role.lambda: Creation complete after 2s [id=site-analyser-lambda]
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_dynamodb_table.site_analyser: Creation complete after 7s [id=site-analyser]
aws_iam_policy.lambda: Creating...
aws_iam_policy.lambda: Creation complete after 1s [id=arn:aws:iam::289341159200:policy/site-analyser-lambda]
aws_iam_role_policy_attachment.lambda: Creating...
aws_iam_role_policy_attachment.lambda: Creation complete after 0s [id=site-analyser-lambda-20230104115714236700000001]
aws_lambda_function.lambda: Still modifying... [id=site-analyser, 10s elapsed]
aws_lambda_function.lambda: Still modifying... [id=site-analyser, 20s elapsed]
aws_lambda_function.lambda: Modifications complete after 22s [id=site-analyser]
Apply complete! Resources: 4 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
Lookin' good!
OK, now that we have a DynamoDB table and permissions to update it, let's implement the /track
endpoint. The first thing we'll need to do is add awyeah-api (a library which makes Cognitect's aws-api work with Babashka) to talk to DynamoDB. We'll create a src/bb.edn
and add the following:
{:paths ["."]
:deps {com.cognitect.aws/endpoints {:mvn/version "1.1.12.373"}
com.cognitect.aws/dynamodb {:mvn/version "825.2.1262.0"}
com.grzm/awyeah-api {:git/url "https://github.com/grzm/awyeah-api"
:git/sha "0fa7dd51f801dba615e317651efda8c597465af6"}
org.babashka/spec.alpha {:git/url "https://github.com/babashka/spec.alpha"
:git/sha "433b0778e2c32f4bb5d0b48e5a33520bee28b906"}}}
To let Blambda know that it should build a lambda layer for the dependencies, we need to add a :deps-layer-name
key to the config in our top-level bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:deps-layer-name "site-analyser-deps"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:lambda-iam-role "${aws_iam_role.lambda.arn}"
:source-files ["handler.clj"]
:extra-tf-config ["tf/main.tf"]}))
...
Blambda will automatically look in src/bb.edn
to find the dependencies to include in the layer. Let's test this out by building the deps layer:
$ bb blambda build-deps-layer
Building dependencies layer: /tmp/site-analyser/target/site-analyser-deps.zip
Classpath before transforming: src:/tmp/site-analyser/.work/m2-repo/com/cognitect/aws/dynamodb/825.2.1262.0/...
Classpath after transforming: src:/opt/m2-repo/com/cognitect/aws/dynamodb/825.2.1262.0/dynamodb-825.2.1262.0.jar:...
Compressing dependencies layer: /tmp/site-analyser/target/site-analyser-deps.zip
adding: gitlibs/ (stored 0%)
adding: gitlibs/_repos/ (stored 0%)
[...]
adding: m2-repo/com/cognitect/aws/dynamodb/825.2.1262.0/dynamodb-825.2.1262.0.pom.sha1 (deflated 3%)
adding: deps-classpath (deflated 67%)
And since we have a new layer, we'll need to generate new Terraform config:
$ bb blambda terraform write-config
Copying Terraform config tf/main.tf
Writing lambda layer config: /tmp/site-analyser/target/blambda.tf
Writing lambda layer vars: /tmp/site-analyser/target/blambda.auto.tfvars
Writing lambda layers module: /tmp/site-analyser/target/modules/lambda_layer.tf
Let's now whip up a src/page_views.clj
:
(ns page-views
(:require [com.grzm.awyeah.client.api :as aws]))
(defn log [msg data]
(prn (assoc data :msg msg)))
(defmacro ->map [& ks]
(assert (every? symbol? ks))
(zipmap (map keyword ks)
ks))
(defn ->int [s]
(Integer/parseUnsignedInt s))
(defn client [{:keys [aws-region] :as config}]
(assoc config :dynamodb (aws/client {:api :dynamodb, :region aws-region})))
(defn validate-response [res]
(when (:cognitect.anomalies/category res)
(let [data (merge (select-keys res [:cognitect.anomalies/category])
{:err-msg (:Message res)
:err-type (:__type res)})]
(log "DynamoDB request failed" data)
(throw (ex-info "DynamoDB request failed" data))))
res)
(defn increment! [{:keys [dynamodb views-table] :as client} date url]
(let [req {:TableName views-table
:Key {:date {:S date}
:url {:S url}}
:UpdateExpression "ADD #views :increment"
:ExpressionAttributeNames {"#views" "views"}
:ExpressionAttributeValues {":increment" {:N "1"}}
:ReturnValues "ALL_NEW"}
_ (log "Incrementing page view counter"
(->map date url req))
res (-> (aws/invoke dynamodb {:op :UpdateItem
:request req})
validate-response)
new-counter (-> res
(get-in [:Attributes :views :N])
->int)
ret (->map date url new-counter)]
(log "Page view counter incremented"
ret)
ret))
That's a bunch of code to have written without knowing it works, so let's act like real Clojure developers and fire up a REPL:
$ cd src/
$ bb nrepl-server 0
Started nREPL server at 127.0.0.1:42733
For more info visit: https://book.babashka.org/#_nrepl
Once this is done, we can connect from our text editor (Emacs, I hope) and test things out in a Rich comment:
(comment
(def c (client {:aws-region "eu-west-1", :views-table "site-analyser"}))
(increment! c "2023-01-04" "https://example.com/test.html")
;; => {:date "2023-01-04", :url "https://example.com/test.html", :new-counter 1}
(increment! c "2023-01-04" "https://example.com/test.html")
;; => {:date "2023-01-04", :url "https://example.com/test.html", :new-counter 2}
)
Lookin' good! ๐
Let's populate the table with a bunch of data (this will take a little while):
(comment
(doseq [date (map (partial format "2022-12-%02d") (range 1 32))
url (map (partial format "https://example.com/page-%02d") (range 1 11))]
(dotimes [_ (rand-int 5)]
(increment! c date url)))
;; nil
)
Just for fun, we can take a quick peek at what the data looks like in DynamoDB:
(comment
(let [{:keys [dynamodb views-table]} c]
(aws/invoke dynamodb {:op :Scan
:request {:TableName views-table
:Limit 5}}))
;; => {:Items
;; [{:views {:N "7"},
;; :date {:S "2022-12-12"},
;; :url {:S "https://example.com/page-01"}}
;; {:views {:N "16"},
;; :date {:S "2022-12-12"},
;; :url {:S "https://example.com/page-02"}}
;; {:views {:N "14"},
;; :date {:S "2022-12-12"},
;; :url {:S "https://example.com/page-03"}}
;; {:views {:N "8"},
;; :date {:S "2022-12-12"},
;; :url {:S "https://example.com/page-05"}}
;; {:views {:N "6"},
;; :date {:S "2022-12-12"},
;; :url {:S "https://example.com/page-06"}}],
;; :Count 5,
;; :ScannedCount 5,
;; :LastEvaluatedKey
;; {:url {:S "https://example.com/page-06"}, :date {:S "2022-12-12"}}}
)
Now that we have the DynamoDB machinery in place, let's connect it to our handler. But first, let's do a quick bit of refactoring so that we don't have to duplicate the log
function in both our handler
and page-views
namespaces. We'll create a src/util.clj
and add the following:
(ns util
(:import (java.net URLDecoder)
(java.nio.charset StandardCharsets)))
(defmacro ->map [& ks]
(assert (every? symbol? ks))
(zipmap (map keyword ks)
ks))
(defn ->int [s]
(Integer/parseUnsignedInt s))
(defn log
([msg]
(log msg {}))
([msg data]
(prn (assoc data :msg msg))))
We need to update page_views.clj
to use this namespace:
(ns page-views
(:require [com.grzm.awyeah.client.api :as aws]
[util :refer [->map]]))
(defn client [{:keys [aws-region] :as config}]
(assoc config :dynamodb (aws/client {:api :dynamodb, :region aws-region})))
(defn validate-response [res]
(when (:cognitect.anomalies/category res)
(let [data (merge (select-keys res [:cognitect.anomalies/category])
{:err-msg (:Message res)
:err-type (:__type res)})]
(util/log "DynamoDB request failed" data)
(throw (ex-info "DynamoDB request failed" data))))
res)
(defn increment! [{:keys [dynamodb views-table] :as client} date url]
(let [req {:TableName views-table
:Key {:date {:S date}
:url {:S url}}
:UpdateExpression "ADD #views :increment"
:ExpressionAttributeNames {"#views" "views"}
:ExpressionAttributeValues {":increment" {:N "1"}}
:ReturnValues "ALL_NEW"}
_ (util/log "Incrementing page view counter"
(->map date url req))
res (-> (aws/invoke dynamodb {:op :UpdateItem
:request req})
validate-response)
new-counter (-> res
(get-in [:Attributes :views :N])
util/->int)
ret (->map date url new-counter)]
(util/log "Page view counter incremented"
ret)
ret))
Now we can turn our eye to handler.clj
. First, we pull in the util
namespace and use its log
function:
(ns handler
(:require [cheshire.core :as json]
[util :refer [->map]]))
(defn handler [{:keys [queryStringParameters requestContext] :as event} _context]
(util/log "Invoked with event" {:event event})
(let [{:keys [method path]} (:http requestContext)
{:keys [name] :or {name "Blambda"}} queryStringParameters]
(util/log (format "Request: %s %s" method path)
{:method method, :path path, :name name})
{:statusCode 200
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:greeting (str "Hello " name "!")})}))
Now, let's remove the hello world stuff and add a /track
endpoint. We expect our clients to make an HTTP POST
request to the /track
path, so we can use a simple pattern match in the handler for this:
(defn handler [{:keys [queryStringParameters requestContext] :as event} _context]
(util/log "Invoked with event" {:event event})
(let [{:keys [method path]} (:http requestContext)]
(util/log (format "Request: %s %s" method path)
{:method method, :path path, :name name})
(case [method path]
["POST" "/track"]
(let [{:keys [url]} queryStringParameters]
(if url
(do
(util/log "Should be tracking a page view here" (->map url))
{:statusCode 204})
{:statusCode 400
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:msg "Missing required param: url"})}))
{:statusCode 404
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:msg (format "Resource not found: %s" path)})})))
We can test this out in the REPL:
(comment
(handler {:requestContext {:http {:method "POST" :path "/nope"}}} nil)
;; => {:statusCode 404,
;; :headers {"Content-Type" "application/json"},
;; :body "{\"msg\":\"Resource not found: /nope\"}"}
(handler {:requestContext {:http {:method "GET" :path "/track"}}} nil)
;; => {:statusCode 404,
;; :headers {"Content-Type" "application/json"},
;; :body "{\"msg\":\"Resource not found: /track\"}"}
(handler {:requestContext {:http {:method "POST" :path "/track"}}} nil)
;; => {:statusCode 400,
;; :headers {"Content-Type" "application/json"},
;; :body "{\"msg\":\"Missing required param: url\"}"}
(handler {:requestContext {:http {:method "POST" :path "/track"}}
:queryStringParameters {:url "https://example.com/test.html"}} nil)
;; => {:statusCode 204}
)
Now we need to connect this to page-views/increment!
, which in addition to the URL, also requires a date. Before figuring out how to provide that, let's extract the tracking code into a separate function so we don't keep adding stuff to the handler
function:
(defn track-visit! [{:keys [queryStringParameters] :as event}]
(let [{:keys [url]} queryStringParameters]
(if url
(do
(util/log "Should be tracking a page view here" (->map url))
{:statusCode 204})
{:statusCode 400
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:msg "Missing required param: url"})})))
Now we can simplify the case
statement:
(case [method path]
["POST" "/track"] (track-visit! event)
{:statusCode 404
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:msg (format "Resource not found: %s" path)})})
Babashka provides the java.time
classes, so we can get the current date using java.time.LocalDate
. Let's import it into our namespace then use it in our shiny new track-visit!
function:
(ns handler
(:require [cheshire.core :as json]
[util :refer [->map]])
(:import (java.time LocalDate)))
(defn track-visit! [{:keys [queryStringParameters] :as event}]
(let [{:keys [url]} queryStringParameters]
(if url
(let [date (str (LocalDate/now))]
(util/log "Should be tracking a page view here" (->map date url))
{:statusCode 204})
(do
(util/log "Missing required query param" {:param "url"})
{:statusCode 400
:body "Missing required query param: url"}))))
Let's test this out in the REPL:
(comment
(handler {:requestContext {:http {:method "POST" :path "/track"}}
:queryStringParameters {:url "https://example.com/test.html"}} nil)
;; => {:statusCode 201}
)
You should see something like this printed to your REPL's output buffer:
{:event {:requestContext {:http {:method "POST", :path "/track"}}, :queryStringParameters {:url "https://example.com/test.html"}}, :msg "Invoked with event"}
{:method "POST", :path "/track", :msg "Request: POST /track"}
{:url "https://example.com/test.html", :msg "Should be tracking a page view here"}
Finally, we need to connect the handler to page-views/increment!
. increment!
expects us to pass it a page views client, which contains a DynamoDB client, which will establish an HTTP connection when first used, which will take a couple of milliseconds. We would like this HTTP connection to be shared across invocations of our lambda so that we only need to establish it on a cold start (or whenever the DynamoDB API feels like the connection has been idle too long and decides to close it), so we'll use the trick of defining it outside our handler function:
(ns handler
(:require [cheshire.core :as json]
[page-views]
[util :refer [->map]])
(:import (java.time LocalDate)))
(def client (page-views/client {:aws-region "eu-west-1"
:views-table "site-analyser"}))
Now we have everything we need! We'll replace our placeholder log line:
(util/log "Should be tracking a page view here" (->map url))
with an actual call to page-views/increment!
:
(page-views/increment! client date url)
Let's test this one more time in the REPL before deploying it:
(comment
(handler {:requestContext {:http {:method "POST" :path "/track"}}
:queryStringParameters {:url "https://example.com/test.html"}} nil)
;; => {:statusCode 201}
)
This time, we should see an actual DynamoDB query logged:
{:event {:requestContext {:http {:method "POST", :path "/track"}}, :queryStringParameters {:url "https://example.com/test.html"}}, :msg "Invoked with event"}
{:method "POST", :path "/track", :msg "Request: POST /track"}
{:date "2023-01-04", :url "https://example.com/test.html", :req {:TableName "site-analyser", :Key {:date {:S "2023-01-04"}, :url {:S "https://example.com/test.html"}}, :UpdateExpression "ADD #views :increment", :ExpressionAttributeNames {"#views" "views"}, :ExpressionAttributeValues {":increment" {:N "1"}}, :ReturnValues "ALL_NEW"}, :msg "Incrementing page view counter"}
{:date "2023-01-04", :url "https://example.com/test.html", :new-counter 3, :msg "Page view counter incremented"}
Let's recap what we've done:
page-views
and util
Since we added more source files, we need to add them to the :source-files
list in the top-level bb-edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:deps-layer-name "site-analyser-deps"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:lambda-iam-role "${aws_iam_role.lambda.arn}"
:source-files ["handler.clj" "page_views.clj" "util.clj"]
:extra-tf-config ["tf/main.tf"]}))
...
Once this is done, we can rebuild our lambda and reploy:
$ bb blambda build-lambda
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Adding file: page_views.clj
Adding file: util.clj
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
updating: handler.clj (deflated 66%)
adding: page_views.clj (deflated 59%)
adding: util.clj (deflated 35%)
[nix-shell:/tmp/site-analyser]$ bb blambda terraform apply
Terraform will perform the following actions:
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
# module.deps.aws_lambda_layer_version.layer will be created
+ resource "aws_lambda_layer_version" "layer" {
Plan: 1 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.deps.aws_lambda_layer_version.layer: Creating...
module.deps.aws_lambda_layer_version.layer: Still creating... [10s elapsed]
module.deps.aws_lambda_layer_version.layer: Creation complete after 10s [id=arn:aws:lambda:eu-west-1:289341159200:layer:site-analyser-de
ps:1]
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_lambda_function.lambda: Still modifying... [id=site-analyser, 10s elapsed]
aws_lambda_function.lambda: Modifications complete after 11s [id=site-analyser]
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
Now that this is deployed, we can test it by tracking a view:
curl -v -X POST $BASE_URL/track?url=https%3A%2F%2Fexample.com%2Ftest.html
* Trying 54.220.150.207:443...
* Connected to kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws (54.220.150.207) port 443 (#0)
[...]
< HTTP/1.1 204 No Content
< Date: Wed, 04 Jan 2023 16:01:24 GMT
< Content-Type: application/json
< Connection: keep-alive
< x-amzn-RequestId: 2d5c6a9d-d3a4-4abb-9a57-c956ca3030f3
< X-Amzn-Trace-Id: root=1-63b5a2d4-58cb681c06672bb410efe80f;sampled=0
<
* Connection #0 to host kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws left intact
Looks like it worked!
Before we forge on, let's deal with the annoying hardcoding of our client config:
(def client (page-views/client {:aws-region "eu-west-1"
:views-table "site-analyser"}))
The normal way of configuring lamdbas is to set environment variables, so let's do that:
(defn get-env
([k]
(or (System/getenv k)
(let [msg (format "Missing env var: %s" k)]
(throw (ex-info msg {:msg msg, :env-var k})))))
([k default]
(or (System/getenv k) default)))
(def config {:aws-region (get-env "AWS_REGION" "eu-west-1")
:views-table (get-env "VIEWS_TABLE")})
(def client (page-views/client config))
Now if we set the VIEWS_TABLE
environment variable in our lambda config (AWS_REGION
is set by the lambda runtime itself), we're good to go. We can tell Blambda to do this for us by adding a :lambda-env-vars
to our top-level bb.edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:deps-layer-name "site-analyser-deps"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:lambda-iam-role "${aws_iam_role.lambda.arn}"
:lambda-env-vars ["VIEWS_TABLE=${aws_dynamodb_table.site_analyser.name}"]
:source-files ["handler.clj" "page_views.clj" "util.clj"]
:extra-tf-config ["tf/main.tf"]}))
...
We'll set the name using the Terraform resource that we defined (aws_dynamodb_table.site_analyser
), so that if we decide to change the table name, we'll only need to change it in tf/main.tf
. The odd format for :lambda-env-vars
is to support specifying it from the command line, so just hold your nose and move on.
Let's rebuild our lambda, regenerate our Terraform config, and redeploy:
$ bb blambda build-lambda [54/1822]
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Adding file: page_views.clj
Adding file: util.clj
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
updating: handler.clj (deflated 64%)
updating: page_views.clj (deflated 59%)
updating: util.clj (deflated 35%)
$ bb blambda terraform write-config
Copying Terraform config tf/main.tf
Writing lambda layer config: /tmp/site-analyser/target/blambda.tf
Writing lambda layer vars: /tmp/site-analyser/target/blambda.auto.tfvars
Writing lambda layers module: /tmp/site-analyser/target/modules/lambda_layer.tf
$ bb blambda terraform apply
Terraform will perform the following actions:
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
# (19 unchanged attributes hidden)
+ environment {
+ variables = {
+ "VIEWS_TABLE" = "site-analyser"
}
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_lambda_function.lambda: Still modifying... [id=site-analyser, 10s elapsed]
aws_lambda_function.lambda: Modifications complete after 15s [id=site-analyser]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
The final piece of the puzzle is displaying the site analytics. We said that GET /dashboard
should serve up a nice HTML page, so let's add a route for this to our handler:
(case [method path]
["GET" "/dashboard"] (serve-dashboard event)
["POST" "/track"] (track-visit! event)
{:statusCode 404
:headers {"Content-Type" "application/json"}
:body (json/generate-string {:msg (format "Resource not found: %s" path)})})
Before we write the serve-dashboard
function, let's think about how this should work. Babashka ships with Selmer, a nice template system, so let's add a src/index.html
template, copying liberally from Cyprien Pannier's blog post:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.3/css/bulma.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css" integrity="sha512-1ycn6IcaQQ40/MKBW2W4Rhis/DbILU74C1vSrLJxCq57o941Ym01SwNsOMqvEBFlcgUa6xLiPY/NS5R+E6ztJQ==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<script src="https://cdn.jsdelivr.net/npm/vega@5.21.0"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-embed@6.20.2"></script>
<title>Site Analytics - Powered by Blambda!</title>
</head>
<body>
<section class="hero is-link block">
<div class="hero-body has-text-centered">
<p class="title" style="vertical-align:baseline">
<span class="icon">
<i class="fas fa-chart-pie"></i>
</span>
Site Analytics - Powered by Blambda!
</p>
<p class="subtitle">{{date-label}}</p>
</div>
</section>
<div class="container is-max-desktop">
<div class="box">
<nav class="level is-mobile">
<div class="level-item has-text-centered">
<div>
<p class="heading">Total views</p>
<p class="title">{{total-views}}</p>
</div>
</div>
</nav>
<div>
<div id="{{chart-id}}" style="width:100%; height:300px"></div>
<script type="text/javascript">
vegaEmbed ('#{{chart-id}}', JSON.parse({{chart-spec|json|safe}}));
</script>
</div>
</div>
<div class="box">
<h1 class="title is-3">Top URLs</h1>
<table class="table is-fullwidth is-hoverable is-striped">
<thead>
<tr>
<th>Rank</th>
<th>URL</th>
<th>Views</th>
</tr>
</thead>
<tbody>
{% for i in top-urls %}
<tr>
<td style="width: 20px">{{i.rank}}</td>
<td><a href="{{i.url}}">{{i.url}}</a></td>
<td style="width: 20px">{{i.views}}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
</body>
</html>
Just consider this stuff an incantation if you don't feel like reading it. ๐
Once we have the template in place, we can write the serve-dashboard
that renders it. First, we need to require Selmer in src/handler.clj
:
(ns handler
(:require [cheshire.core :as json]
[selmer.parser :as selmer]
[page-views]
[util :refer [->map]])
(:import (java.time LocalDate)))
Now, we can just load and render the template in serve-dashboard
:
(defn serve-dashboard [_event]
(util/log "Rendering dashboard")
{:statusCode 200
:headers {"Content-Type" "text/html"}
:body (selmer/render (slurp "index.html") {})})
Since we've added index.html
as a source file, we need to add it to the :source-files
list in the top-level bb-edn
:
{:deps { ... }
:tasks
{:requires ([blambda.cli :as blambda])
:init (do
(def config {:bb-arch "arm64"
:deps-layer-name "site-analyser-deps"
:lambda-name "site-analyser"
:lambda-handler "handler/handler"
:lambda-iam-role "${aws_iam_role.lambda.arn}"
:source-files [;; Clojure sources
"handler.clj"
"page_views.clj"
"util.clj"
;; HTML templates
"index.html"
]
:extra-tf-config ["tf/main.tf"]}))
...
Let's rebuild and redeploy:
$ bb blambda build-lambda
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Adding file: page_views.clj
Adding file: util.clj
Adding file: index.html
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
updating: handler.clj (deflated 66%)
updating: page_views.clj (deflated 59%)
updating: util.clj (deflated 35%)
adding: index.html (deflated 59%)
[nix-shell:/tmp/site-analyser]$ bb blambda terraform apply
Terraform will perform the following actions:
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_lambda_function.lambda: Modifications complete after 7s [id=site-analyser]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
Now we can visit https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/dashboard in a web browser and see something like this:
This is definitely a bit on the boring side, so let's write some code to query DynamoDB and supply the data to make the dashboard dashing!
Whenever we fetch data from an AWS API, we need to handle pagination. Luckily, I have already blogged about this extensively, so we can just copy and paste from my S3 paging example to accomplish the same with DynamoDB. Let's start by adding the handy lazy-concat
helper function to src/util.clj
:
(defn lazy-concat [colls]
(lazy-seq
(when-first [c colls]
(lazy-cat c (lazy-concat (rest colls))))))
Now, we can add some code to src/page_views.clj
to query DynamoDB in a page-friendly way:
(defn get-query-page [{:keys [dynamodb views-table] :as client}
date
{:keys [page-num LastEvaluatedKey] :as prev}]
(when prev
(util/log "Got page" prev))
(when (or (nil? prev)
LastEvaluatedKey)
(let [page-num (inc (or page-num 0))
req (merge
{:TableName views-table
:KeyConditionExpression "#date = :date"
:ExpressionAttributeNames {"#date" "date"}
:ExpressionAttributeValues {":date" {:S date}}}
(when LastEvaluatedKey
{:ExclusiveStartKey LastEvaluatedKey}))
_ (util/log "Querying page views" (->map date page-num req))
res (-> (aws/invoke dynamodb {:op :Query
:request req})
validate-response)
_ (util/log "Got response" (->map res))]
(assoc res :page-num page-num))))
(defn get-views [client date]
(->> (iteration (partial get-query-page client date)
:vf :Items)
util/lazy-concat
(map (fn [{:keys [views date url]}]
{:date (:S date)
:url (:S url)
:views (util/->int (:N views))}))))
We might as well test this out in our REPL whilst we're here:
(comment
(def c (client {:aws-region "eu-west-1", :views-table "site-analyser"}))
(get-views c "2022-12-25")
;; => ({:date "2022-12-25", :url "https://example.com/page-01", :views 5}
;; {:date "2022-12-25", :url "https://example.com/page-03", :views 8}
;; {:date "2022-12-25", :url "https://example.com/page-04", :views 15}
;; {:date "2022-12-25", :url "https://example.com/page-05", :views 3}
;; {:date "2022-12-25", :url "https://example.com/page-06", :views 12}
;; {:date "2022-12-25", :url "https://example.com/page-07", :views 11}
;; {:date "2022-12-25", :url "https://example.com/page-08", :views 15}
;; {:date "2022-12-25", :url "https://example.com/page-09", :views 8})
)
Looks good!
Now let's plug this into src/handler.clj
! The Vega library we're using to render our data will attach to a <div>
in our HTML page, to which we'll give a random ID. In order to facilitate this, let's import java.util.UUID
:
(ns handler
(:require [cheshire.core :as json]
[selmer.parser :as selmer]
[page-views]
[util :refer [->map]])
(:import (java.time LocalDate)
(java.util UUID)))
And whilst we're at it, let's add a little more config to control how many days of data and how many top URLs to show:
(def config {:aws-region (get-env "AWS_REGION" "eu-west-1")
:views-table (get-env "VIEWS_TABLE")
:num-days (util/->int (get-env "NUM_DAYS" "7"))
:num-top-urls (util/->int (get-env "NUM_TOP_URLS" "10"))})
Now we're ready to write the serve-dashboard
function:
(defn serve-dashboard [{:keys [queryStringParameters] :as event}]
(let [date (:date queryStringParameters)
dates (if date
[date]
(->> (range (:num-days config))
(map #(str (.minusDays (LocalDate/now) %)))))
date-label (or date (format "last %d days" (:num-days config)))
all-views (mapcat #(page-views/get-views client %) dates)
total-views (reduce + (map :views all-views))
top-urls (->> all-views
(group-by :url)
(map (fn [[url views]]
[url (reduce + (map :views views))]))
(sort-by second)
reverse
(take (:num-top-urls config))
(map-indexed (fn [i [url views]]
(assoc (->map url views) :rank (inc i)))))
chart-id (str "div-" (UUID/randomUUID))
chart-data (->> all-views
(group-by :date)
(map (fn [[date rows]]
{:date date
:views (reduce + (map :views rows))}))
(sort-by :date))
chart-spec (json/generate-string
{:$schema "https://vega.github.io/schema/vega-lite/v5.json"
:data {:values chart-data}
:mark {:type "bar"}
:width "container"
:height 300
:encoding {:x {:field "date"
:type "nominal"
:axis {:labelAngle -45}}
:y {:field "views"
:type "quantitative"}}})
tmpl-vars (->map date-label
total-views
top-urls
chart-id
chart-spec)]
(util/log "Rendering dashboard" tmpl-vars)
{:statusCode 200
:headers {"Content-Type" "text/html"}
:body (selmer/render (slurp "index.html")
tmpl-vars)}))
A quick build and deploy and now we'll be able to see some exciting data!
$ bb blambda build-lambda
Building lambda artifact: /tmp/site-analyser/target/site-analyser.zip
Adding file: handler.clj
Adding file: page_views.clj
Adding file: util.clj
Adding file: index.html
Compressing lambda: /tmp/site-analyser/target/site-analyser.zip
updating: handler.clj (deflated 66%)
updating: page_views.clj (deflated 65%)
updating: util.clj (deflated 42%)
updating: index.html (deflated 59%)
[nix-shell:/tmp/site-analyser]$ bb blambda terraform apply
Terraform will perform the following actions:
# aws_lambda_function.lambda will be updated in-place
~ resource "aws_lambda_function" "lambda" {
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_lambda_function.lambda: Modifying... [id=site-analyser]
aws_lambda_function.lambda: Still modifying... [id=site-analyser, 10s elapsed]
aws_lambda_function.lambda: Modifications complete after 11s [id=site-analyser]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Outputs:
function_url = "https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/"
Visiting https://kuceaiz55k7soeki4u5oy4w6uy0ntbky.lambda-url.eu-west-1.on.aws/dashboard again tells a tale of joy!
With this, let's declare our site analysed and all agree that Babashka is way better than nbb. Hurrah! ๐
Published: 2023-01-04
I was walking Rover this morning through the snow, listening to a podcast on the dangers of economism, when I realised that my mind had drifted away from political theory and was stuck on a work-related problem. I've learned over the years that it's completely futile to fight that and try to turn my attention back to my podcast or book or whatever I was trying to do, so I leaned into it and tried to see if I could just solve the problem so it would leave me in peace and I could go back to enjoying my walk.
I've been a professional software engineer for about 20 years now, and these intrusions have historically been oriented around why my multi-threaded C++ IVR system is deadlocking or how to design DynamoDB tables for my serverless API or whatever, but in the past couple of years, these problems have been more likely oriented around organisational psychology, single-piece flow, or how to build a good promotion case for that great engineer on my team. Yes, it's true: I'm an engineering manager. And yes, I know, I swore that I would never be interested in becoming a manager, but I also swore that I would never do a lot of things that I have subsequently done and enjoyed, so it's no surprise that I was wrong about this as well.
But anyway, back to the problem at hand. I'm managing a newly formed team, and we're having a team kickoff session first thing in January. This is three days of doing things like building a social contract, defining our team purpose, meeting stakeholders, and deciding on ways of working, and I needed to find a good way to start the session. One of the classic openers is breakfast and an icebreaker, and I decided that if it ain't broke, don't fix it, so I should go ahead and do that. So far, no problem.
The challenge then became what kind of icebreaker to use. I needed something that would fit the following criteria:
One of my favourites is Two Truths and a Lie, in which you come up with three short statements about yourself, two of which are true and one of which is a lie, and the group tries to figure out which one is the lie. This works well remotely; doesn't force people to get more personal than they want to, since the statements can be about anything (for example: I've brought down a global e-commerce website and cost my employer about $10 million in lost sales); and fits comfortably in half an hour for groups up to 7-8 people.
However, we used this icebreaker last time we had a team offsite three months ago, and some of the people from my previous team will also be on the new team. Worse yet, my so-called friend D.S. (you know who you are!) used this one in an org-wide offsite just last week. The nerve!
So TTaaL is right out. On Friday, I was browsing through some icebreaker ideas, and all of them failed to satisfy one or more criteria. This apparently had been eating at my subconscious, and decided to surface right when I was trying to concentrate on something else.
I started thinking about fun work-related things that I've done over the years, and one particular exercise popped into my mind: Architecture Golf. Architecture Golf is a group learning exercise created by my former colleagues Tรบlio Ornelas and Kenneth Gibson, and works like this:
According to Tรบlio:
We call this exercise Architecture golf because it helped us explain the crazy architecture that we sometimes have. Each team member takes one swing at a time which pushes them closer to the solution, just like a team version of golf.
I encourage you to read the full article and try this game out with your team sometime!
As cool as this is, it didn't quite fit my needs because 30 minutes is a bit too short for completing a session, and my team will own two systems, so I didn't want to single one and potentially send the message that it's somehow more important than the other system. However, the kernel of the idea is exactly what I was looking for: collaborative drawing, one person at a time.
So if we can't draw a real system, how about a fake one? My brain suddenly did that cool thing where it made a connection to some other thing, and I had the answer! I remembered being very amused by FizzBuzzEnterpriseEdition a few years back, and thought it would be fun to collaboratively design an incredibly complicated distributed system to solve an incredibly trivial problem, such as FizzBuzz itself, reversing a string, sorting a list, etc.
My first thought was to call the game "Enterprise Improv", but one thing bothered me a little bit about it. A few years back, I read a really great blog post by Aurynn Shaw title Contempt Culture. It starts like this:
So when I started programming in 2001, it was du jour in the communities I participated in to be highly critical of other languages. Other languages sucked, the people using them were losers or stupid, if they would just use a real language, such as the one we used, everything would just be better.
Right?
The point Aurynn makes in the post (which you really should read; it's fantastic!) is that making fun of other languages or tools can result in people who use those languages or tools feeling ridiculed or less than.
Even though FizzBuzzEnterpriseEdition is clearly tongue in cheek and intended to be all in good fun, if you're a Java developer working at a large enterprise, I could certainly understand if you feel like you're being made fun of.
So I tried to think of something fairly universal to software development, something that all of us do from time to time. And lo! it came to me in a flash.
Over-engineering.
Which one of us has never added a layer or two of indirection because "we might need to switch to a different database" or implemented a plugin system because "then our users can write their own modules" or tried to use everything from the Design Patterns book in the same class because "those patterns are important for writing quality software"?
So I decided the name of the game would be "Over-Engineering Improv". And here's how you play:
If the problem is solved in the final turn, it's a win. If the problem isn't solved, it's still a win because hopefully everyone had loads of fun, and will be primed to beat the game next time!
Published: 2022-12-11
Tagged: engineering