jmglov's blog
A blog about stuff but also things.
A blog about stuff but also things.
The fun thing about having a blog which is built with a static site
generator is that you get to waste spend time customising it. In today's instalment of "Hacking the blog", we'll see how to add a "favicon", which is that little icon thingy on your tab title.
My first order of business was to figure out what I wanted to use for my favicon. I decided that I really love this drawing that my friend Sebastian did on a whiteboard way back in 2016, so why not use it?
The first step (after I remembered what a "favicon" was actually called) was making a clean version of this image that would scale down nicely to the various sizes used by browsers. Courtesy of a really thorough answer to a question on Stack Overflow, I found a really cool site called RealFaviconGenerator, which would take an image (recommended to be at least 260x206 pixels) and spit out a zipfile containing a bunch of files:
These files, if placed at the root of your website and combined with a chunk of HTML in your <head> section, would do the right thing for All the Browsers and All the Smartphones.
So I opened up my image in the GIMP, cropped it so only my head was visible, and removed the speech bubble, resulting in the following:
I fed this image into RealFaviconGenerator, which gave me back the zipfile described above and the following HTML fragment:
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
The next order of business was to get the favicon onto my site. If you remember from the Actually blogging with Clojure post, the way the blog works is this:
bb.edn which defines a render task that looks like this: render {:doc "Render blog"
:task (load-file "render.clj")}render.clj does stuff like converting Markdown to HTML, then uses the Selmer templating system to shove blog posts into the templates/base.html templaterender.clj then copies the resulting HTML files to a public/ directorypublish in bb.edn that uses the AWS CLI to sync everything in the public/ to the S3 bucket that contains my blog: publish {:doc "Publish to jmglov.net"
:depends [render]
:task (shell "aws s3 sync –delete public/ s3://jmglov.net/blog/")}So getting the favicon injected into every page was as simple as blasting the HTML fragment into templates/base.html.
Almost.
You see, I also need to put the content being referenced in all of those <link> tags up on the website. My website itself uses the exact same machinery as the blog, meaning I need to add the HTML fragment to the top-level templates/base.html, and in order to get the favicon stuff onto the website, I need to hack up the top-level render.clj.
Looking at the way the existing render.clj (which I stole with pride from borkdude’s blog) handles images and CSS is most instructive:
(def out-dir "public")
;;;; Sync images and CSS
(def asset-dir (fs/create-dirs (fs/file out-dir "assets")))
(fs/copy-tree "assets" asset-dir {:replace-existing true})
(spit (fs/file out-dir "style.css")
(slurp "templates/style.css"))Since the deploy task will sync everything from the public/ directory to my website, I just need to put the contents of the favicon zipfile in public/ and I win!
I unzipped the favicon zipfile into a top-level favicon/ directory, then added the following to render.clj:
;;;; Sync favicon
(def favicon-dir (fs/create-dirs (fs/file out-dir)))
(fs/copy-tree "favicon" favicon-dir {:replace-existing true})And that's all it took to display my little cartoon face on your browser's tab! 🏆
If you're interested, you can check out this commit to see everything I did, all wrapped up into one cute little package.
Discuss this post here.
Published: 2022-07-05
Today I finally tried to use Blambda! for something real: a log parser for my HTTP access logs that S3 and Cloudfront write to my logs bucket. You can follow along with my fun on Github: jmglov/s3-log-parser.
Here's what I'm trying to do:
I'm sure this will change quite drastically as I go, but it seems like a fun problem that will find the rough edges with Blambda!
Here's how I proceeded.
In my project, I created a simple handler in src/s3_log_parser.clj:
(ns s3-log-parser)
(defn handler [event context]
(prn {:msg "Invoked with event",
:data {:event event}})
{})This will just log the event it was invoked with and then return an empty map (or JSON object, if you must).
Since this lambda will eventually interact with S3, I decided to bite the bullet and include the babashka-aws pod. I created a src/bb.edn like this:
{:paths ["."]
:pods {org.babashka/aws {:version "0.1.2"}}}This bb.edn will be picked up by Babashka when it is executing my lambda, since Blambda! runs bb from the directory where my lambda archive is unpacked ($LAMBDA_TASK_ROOT, for those of you familiar with building custom runtimes).
I created a top-level bb.edn, which is used for defining Babashka tasks to build and deploy my function (not to be confused with src/bb.edn, which will be used at lambda runtime). The build task looks like this:
build {:doc "Builds lambda artifact"
:requires ([clojure.java.shell :refer [sh]])
:task (let [{:keys [target-dir work-dir]} (th/parse-args)
work-dir (str work-dir "/lambda")
src-dir "src"
lambda-zipfile (th/target-file target-dir "function.zip")]
(doseq [dir [target-dir work-dir]]
(fs/create-dirs dir))
(doseq [f ["bb.edn" "s3_log_parser.clj"]]
(println "Adding file" f)
(fs/delete-if-exists (format "%s/%s" work-dir f))
(fs/copy (format "%s/%s" src-dir f) work-dir))
(println "Compressing lambda archive:" lambda-zipfile)
(let [{:keys [exit err]}
(sh "zip" "-r" lambda-zipfile "."
:dir work-dir)]
(when (not= 0 exit)
(println "Error:" err))))}You can read the source for the th namespace in task_helper.clj if you like, but basically, what the build task is doing is:
bb.edn and s3_log_parser.clj files from the src directory to the work directorytarget/function.zipMy deploy task is pretty straightforward:
deploy {:doc "Deploys lambda using babashka-aws."
:depends [build]
:requires ([pod.babashka.aws :as aws])
:task (let [{:keys [target-dir] :as args}
(th/parse-args)
lambda-zipfile (th/target-file target-dir "function.zip")
zipfile (fs/read-all-bytes lambda-zipfile)]
(th/create-or-update-lambda (assoc args :zipfile zipfile)))}It reads in target/function.zip and passes it along to task-helper/create-or-update-lambda, which is a little more interesting:
(defn create-or-update-lambda [{:keys [aws-region bb-arch
lambda-handler lambda-name lambda-role
runtime-layer zipfile]
:as args}]
(let [lambda (aws/client {:api :lambda
:region aws-region})
_ (println "Checking to see if lambda exists:" lambda-name)
lambda-exists? (-> (aws/invoke lambda {:op :GetFunction
:request {:FunctionName lambda-name}})
(contains? :Configuration))]
(if lambda-exists?
(update-lambda lambda args)
(create-lambda lambda args))))If no lambda with the name we've specified exists, we call create-lambda:
(defn create-lambda [lambda
{:keys [aws-region bb-arch
lambda-handler lambda-name lambda-role
runtime-layer zipfile]}]
(let [lambda-arch (if (= "amd64" bb-arch) "x86_64" "arm64")
runtime (if (= "amd64" bb-arch) "provided" "provided.al2")
sts (aws/client {:api :sts
:region aws-region})
account-id (-> (aws/invoke sts {:op :GetCallerIdentity}) :Account)
layer-arns (->> [runtime-layer]
(map #(format "arn:aws:lambda:%s:%s:layer:%s"
aws-region account-id
(latest-layer-version lambda %))))
role-arn (format "arn:aws:iam::%s:role/%s"
account-id lambda-role)
req {:FunctionName lambda-name
:Runtime runtime
:Role role-arn
:Code {:ZipFile zipfile}
:Handler lambda-handler
:Layers layer-arns
:Architectures [lambda-arch]}
_ (println "Creating lambda:" (pr-str req))
res (aws/invoke lambda {:op :CreateFunction
:request req})]
(when (contains? res :cognitect.anomalies/category)
(println "Error:" (pr-str res)))))If you're interested in this aws/client and aws/invoke stuff, this is the aws-api library provided by the babashka-aws pod.
latest-layer-version is a simple function that checks if our layer name includes a version (like blambda:5), or if not, uses Lambda's ListLayerVersions API to select the latest version:
(defn latest-layer-version [lambda layer-name]
(if (string/includes? layer-name ":")
layer-name
(let [latest-version (->> (aws/invoke lambda {:op :ListLayerVersions
:request {:LayerName layer-name}})
:LayerVersions
(sort-by :Version)
last
:Version)]
(format "%s:%s" layer-name latest-version))))I can now deploy this by running bb deploy (I've pre-baked the IAM role required for this lambda, but it's basically the same as the one from the Blambda! example).
The problem is, when I invoke this lambda using a test event in the console, I get an error:
Test Event Name
test
Response
{
"errorMessage": "RequestId: 8292cbcf-2862-4189-97c2-757bc58a4ed8 Error: Runtime exited with error: exit status 1",
"errorType": "Runtime.ExitError"
}
Function Logs
ashka.pods.impl.resolver$download.invokeStatic(resolver.clj:105)
at babashka.pods.impl.resolver$pod_manifest.invokeStatic(resolver.clj:123)
at babashka.pods.impl.resolver$resolve.invokeStatic(resolver.clj:175)
at babashka.pods.impl$resolve_pod.invokeStatic(impl.clj:327)
[...]
Exception in thread "main" java.io.FileNotFoundException: /home/sbx_user1051/.babashka/pods/repository/org.babashka/aws/0.1.2/manifest.edn (No such file or directory)
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST_FileNotFoundException_constructor_970c509c6abfd3f98898b9a7521945418b90b270(JNIJavaCallWrappers.java:0)
[...]
END RequestId: 8292cbcf-2862-4189-97c2-757bc58a4ed8
REPORT RequestId: 8292cbcf-2862-4189-97c2-757bc58a4ed8 Duration: 594.19 ms Billed Duration: 595 ms Memory Size: 128 MB Max Memory Used: 21 MB
RequestId: 8292cbcf-2862-4189-97c2-757bc58a4ed8 Error: Runtime exited with error: exit status 1
Runtime.ExitError
Request ID
8292cbcf-2862-4189-97c2-757bc58a4ed8
Oops! Babashka is trying to load the babashka-aws pod that I specified in my src/bb.edn, but I haven't provided that pod. I could use babashka.pods/load-pod at runtime to grab the pod, but that would mean that my lambda would have a slow cold start. A better idea is to pub the pod on the lambda instance's filesystem, but how can we do that?
The hint is in this line:
Exception in thread "main" java.io.FileNotFoundException: /home/sbx_user1051/.babashka/pods/repository/org.babashka/aws/0.1.2/manifest.edn (No such file or directory)
If I can create a .babashka directory in the lambda instance's home directory, Babashka should find any pods I put there. Of course, Lambda doesn't let you do that, but it does let you put stuff in /opt, by using a layer. Searching on Clojurians Slack yielded borkdude mentioning an environment variable, $BABASHKA_PODS_DIR, which Babashka will use for the pods repository.
Now I have all the pieces I need. The first step is...
I added a build-pods task to my top-level bb.edn:
build-pods {:doc "Builds pods layer"
:requires ([clojure.java.shell :refer [sh]])
:task (let [{:keys [target-dir work-dir]} (th/parse-args)
work-dir (str work-dir "/pods")
pods-dir (str (fs/home) "/.babashka/pods")
pods-zipfile (th/target-file target-dir "pods.zip")]
(doseq [dir [target-dir work-dir]]
(fs/create-dirs dir))
(doseq [pod ["org.babashka/aws/0.1.2"]
:let [dst (format "%s/.babashka/pods/repository/%s" work-dir pod)]]
(when-not (fs/exists? dst)
(println "Adding pod" pod)
(fs/copy-tree (format "%s/repository/%s" pods-dir pod) dst)))
(println "Compressing pods layer" pods-zipfile
"from dir:" work-dir)
(let [{:keys [exit err]}
(sh "zip" "-r" pods-zipfile "."
:dir work-dir)]
(when (not= 0 exit)
(println "Error:" err))))}What it's doing is copying the ~/.babashka/pods/repository/org.babashka/aws/0.1.2 directory into the work dir, then adding it to target/pods.zip.
Deploying the layer looks like this:
deploy-pods {:doc "Deploys pods layer using babashka-aws."
:depends [build-pods]
:requires ([pod.babashka.aws :as aws])
:task (let [{:keys [pods-layer aws-region target-dir]} (th/parse-args)
pods-zipfile (th/target-file target-dir "pods.zip")
client (aws/client {:api :lambda
:region aws-region})
zipfile (fs/read-all-bytes pods-zipfile)
_ (println "Publishing layer version for layer" pods-layer)
res (aws/invoke client {:op :PublishLayerVersion
:request {:LayerName pods-layer
:Content {:ZipFile zipfile}}})]
(if (:cognitect.anomalies/category res)
(prn "Error:" res)
(println "Published layer" (:LayerVersionArn res))))}Since I'm already using the Blambda! layer in my lambda, adding a new layer only requires making minor changes to task-helper/create-lambda:
pods-layer to the function arguments (defn create-lambda [lambda
{:keys [aws-region bb-arch
lambda-handler lambda-name lambda-role
pods-layer runtime-layer zipfile]}]
;; ...
)pods-layer to layer-arns (let [layer-arns (->> [runtime-layer]
(map #(format "arn:aws:lambda:%s:%s:layer:%s"
aws-region account-id
(latest-layer-version lambda %))))
;; ...
]
;; ...
)The only problem left is having Blambda! set BABASHKA_PODS_DIR when starting bb. This is a simple matter of updating the bootstrap script in Blambda! itself:
#!/bin/sh
export BABASHKA_PODS_DIR=/opt/.babashka/pods
cd $LAMBDA_TASK_ROOT
/opt/bb -cp $LAMBDA_TASK_ROOT /opt/bootstrap.clj
Now when I test my lambda, I get a much more satisfying result:
Test Event Name
test
Response
{}
Function Logs
START RequestId: e0d573c2-5afa-4567-af74-592f12efa094 Version: $LATEST
Loading babashka lambda handler: s3-log-parser/handler
Starting babashka lambda event loop
{:msg "Invoked with event", :data {:event {:key1 "value1", :key2 "value2", :key3 "value3"}}}
END RequestId: e0d573c2-5afa-4567-af74-592f12efa094
REPORT RequestId: e0d573c2-5afa-4567-af74-592f12efa094 Duration: 109.60 ms Billed Duration: 559 ms Memory Size: 128 MB Max Memory Used: 117 MB Init Duration: 448.61 ms
Request ID
e0d573c2-5afa-4567-af74-592f12efa094
There is so much wrong and gross here:
bb.ednStay tuned for more thrilling adventures as I eat my own dogfood!
Discuss this post here.
Published: 2022-07-04
A couple of weeks ago, I made a todo list for the summer. One of the items on there was to create an AWS Lambda custom runtime for Babashka. I actually accomplished that a few days later, and today I want to walk through the what, the why, and the how of that project.
Let's start by answering the question of what a custom runtime is. For anyone not familiar with AWS Lambda, it is basically a way to run a function in the AWS cloud without having to worry about how or where the function is actually executed. For me, cloud functions are the next natural step along the path of computing without caring about machines. First came servers that you had to host yourself, then came VMs that you could run on servers you hosted yourself, then came EC2, which gave you a VM hosted by somebody else, then came containers, then came managed container environments like Kubernetes, then came function as a service, which allowed you to provide a zip file that just ran somewhere. You can nitpick the order if you want, but this is more or less accurate. I'm also not claiming that serverless is right for all workloads, but when it is, it's pretty great.
So having explained what Lambda is, I'll crack on with explaining custom runtimes. Lambda comes out of the box with runtimes that support a great variety of programming languages: Python, Golang, .Net, Ruby, JavaScript, and Java. Those last two are of interest to Clojure programmers, since they allow us to write ClojureScript functions and execute them on the NodeJS runtime, or Clojure proper on the Java runtime (you could technically also execute Clojure programs on the .Net runtime using Clojure CLR, but I doubt many people are doing that). This is great, unless you need predictably low-ish latency, because the first time you invoke a lambda function, AWS need to spin up an execution environment, then execute your function. This is called a "cold start", and for the JVM, it can take a few thousand milliseconds, and that's before starting the Clojure runtime, which can take a few thousand more.
Clojure programmers have long known about this JVM startup delay, of course, which is why we tend not to write command line utilities in Clojure, since it is quite annoying for your utility to take 2-3 seconds just to tell you that you've misspelled one of the options (was it --dry-run or --dryrun?). And of course we've had ways around that for awhile as well, mostly based on ClojureScript (Lumo is one example). So one could write lambdas in ClojureScript and run them on the NodeJS runtime and not have to wait around for the JVM to start up (the NodeJS runtime has a cold start of a few hundred milliseconds instead of a few thousand for the JVM), though there are a few drawbacks with that as well:
println statements locally and then compile and then upload and then try again and then realise you need another println somewhere else... ugh! To be fair, JVM Clojure has the same issue.Luckily, AWS has provided the ability to specify a custom runtime, which can be written in any language and just needs to be executable on a Linux system and implement a simple local invocation API. This means that you can now write lambda functions in any language!
Getting back to Clojure, the wonderful borkdude realised that if one could compile a Clojure program using GraalVM, it would start up fast, thus enabling command line programs in Clojure that didn't make you want to pull your hair out. "But why stop there?" borkdude presumably thought to himself. "Writing shell scripts in Bash kinda sucks, so what about writing them in Clojure instead? All I'd have to do is write a program that can interpret Clojure and compile it with GraalVM and then it could execute Clojure scripts or Clojure code passed on the command line and then my life would be complete." And this magical program, my friends, is called Babashka.
Since Babashka starts fast and can evaluate Clojure source code, we can build a custom runtime for Lambda that uses Babashka to execute our lambda functions, thus gaining the ability to edit source code in the lambda console and use Clojure instead of ClojureScript, both of which are very important to me.
This was the motivation behind building Blambda!, which is a custom runtime that can be deployed as a Lambda layer. Let's talk about how it works.
A custom runtime requires only one thing: an executable named bootstrap in the root level of your lambda function's archive. When your function is invoked for the first time, the Lambda runtime executes the bootstrap function, which is then expected to call Lambda's next invocation
API, which returns the request body of whatever called your lambda function, which the runtime customarily hands off to the actual code implementing your lambda function and then feeds the return value of that to the Lambda invocation
response
API, and then waits for the next request and does the same thing all over again.
In the case of Blambda!, the custom runtime consists of three parts:
bootstrap.clj, that implements the request handling loop described above. I borrowed this from an existing Babashka runtime, bb-lambda, which I decided not to use because it uses Docker, which makes me almost as sad as NodeJS. ;)bootstrap shell script which uses Babashka to evaluate the above Clojure programPackaging this as a layer is as simple as downloading Babashka, then zipping it into an archive with the other two files, which you can see in the build task of Blambda!'s bb.edn.
To use Blambda!, you can build and deploy the custom runtime layer by cloning the repo and running:
bb deploy
You then create a lambda function that uses the "provided" runtime, includes the "blambda" layer that was created by the bb deploy command, and sets the handler to whatever function in your namespace that will handle function invocations. For example, if you have a namespace like this:
(ns hello)
(defn hello [{:keys [name] :or {name "Blambda"} :as event} context]
(prn {:msg "Invoked with event",
:data {:event event}})
{:greeting (str "Hello " name "!")})you can create a lambda function like this:
and then test it with an event like this:
{
"name": "jmglov"
}
The Lambda console will display something like this:
Test Event Name
hello
Response
{
"greeting": "Hello jmglov!"
}
Function Logs
START RequestId: 4288f5e7-f4c9-41b2-a26f-b5d688c146ec Version: $LATEST
Loading babashka lambda handler: hello/hello
Starting babashka lambda event loop
{:msg "Invoked with event", :data {:event {:name "jmglov"}}}
END RequestId: 4288f5e7-f4c9-41b2-a26f-b5d688c146ec
REPORT RequestId: 4288f5e7-f4c9-41b2-a26f-b5d688c146ec Duration: 240.41 ms Billed Duration: 669 ms Memory Size: 128 MB Max Memory Used: 97 MB Init Duration: 427.73 ms
Request ID
4288f5e7-f4c9-41b2-a26f-b5d688c146ec
Stay tuned for future posts on Blambda! when I try to actually use it for something real. ;)
Discuss this post here.
Published: 2022-07-03