jmglov's blog
A blog about stuff but also things.
A blog about stuff but also things.
In addition to spending far too much of my time doing silly things with Clojure and then even farther too much of my time writing about doing silly things with Clojure, I spend some of my time thinking about, talking about, and participating in labour organising here in Sweden. As I was talking about unions and such to Ray one day, no doubt six tangents into one of my usual rambling explorations of an idea, he interrupted my flow. "Stop!" he said, "for I have a plan so cunning you could pin a tail on it and call it a weasel!" Curiosity piqued, I enquired as to the nature of said plan. "We should make a podcast," he continued, "and on this podcast, we should talk about tech workers and why it makes sense for them to unionise. And we should focus on Sweden, since it's a fairly unique labour market, plus you know interesting people who we could interview."
"Ray," I rejoined, "that truly is a plan of weasel-grade cunning. I have but one suggestion that will turn this good idea into a great one." "And what," quoth he, "pray tell, is that suggestion?" "Babashka! Scittle! Clojure!" I exclaimed, so full of excitement I was having troubling supporting my proper nouns with clauses of explanatory power. "We could use all of this amazing technology for all of the heavy lifting around making a podcast! We could build a website using S3 static hosting, then we could use an approach similar to how I built my blog to create pages for episodes with show notes and transcripts and all that good stuff!"
So it was agreed, and thus Organising Tech in Sweden came to be.
As with any of my recent projects, my first step is always to create a directory and drop a Scittle-enabled bb.edn in it:
: ~; mkdir ~/code/orgtech-se
: ~; cd !$
bb.edn
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:git/sha "e203166a020509d126149ff8046489857ce5c89f"}}
:tasks
{http-server {:doc "Starts http server for serving static files"
:requires ([babashka.http-server :as http])
:task (do (http/serve {:port 1341 :dir "public"})
(println "Serving static assets at http://localhost:1341"))}
browser-nrepl {:doc "Start browser nREPL"
:requires ([sci.nrepl.browser-server :as bp])
:task (bp/start! {})}
-dev {:depends [http-server browser-nrepl]}
dev {:task (do (run '-dev {:parallel true})
(deref (promise)))}}}
Since this is a static website, we can create a static public/index.html for it, with the usual favicon and social
sharing stuff:
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Organising Tech in Sweden Podcast</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width" />
<link rel="stylesheet" href="/css/main.css">
<!-- Favicon from https://realfavicongenerator.net/ -->
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
<!-- Social sharing (Facebook, Twitter, LinkedIn, etc.) -->
<meta name="title" content="Organising Tech in Sweden">
<meta name="twitter:title" content="Organising Tech in Sweden">
<meta property="og:title" content="Organising Tech in Sweden">
<meta property="og:type" content="website">
<meta name="description" content="A limited podcast series exploring union organising in Swedish tech companies">
<meta name="twitter:description"
content="A limited podcast series exploring union organising in Swedish tech companies">
<meta property="og:description" content="A limited podcast series exploring union organising in Swedish tech companies">
<meta name="twitter:url" content="https://orgtech.se/">
<meta property="og:url" content="https://orgtech.se/">
<meta name="twitter:image" content="https://orgtech.se/img/orgtech-se-preview.jpg">
<meta name="twitter:card" content="summary_large_image">
<meta property="og:image" content="https://orgtech.se/img/orgtech-se-preview.jpg">
<meta property="og:image:alt"
content="Podcast logo: 'Organising Tech in Sweden' superimposed on raised fists with a Swedish flag with a circuit board pattern in the background">
</head>
<body>
<div id="wrapper">
<div id="left-side">
<div id="cover-image">
<img src="/img/orgtech-se-cover.jpg"
title="Organising Tech in Sweden"
alt="Podcast logo: 'Organising Tech in Sweden' superimposed on raised fists with a Swedish flag with a circuit board pattern in the background" />
</div>
<div id="aggregators-1">
<div id="apple">
<a class="apple-button"
href="https://podcasts.apple.com/us/podcast/organising-tech-in-sweden/id1766442275?itsct=podcast_box_badge&itscg=30200&ls=1">
<img src="https://tools.applemediaservices.com/api/badges/listen-on-apple-podcasts/badge/en-us?size=250x83&releaseDate=1725494400"
title="Listen on Apple Podcasts"
alt="Listen on Apple Podcasts"
class="apple-button">
</a>
</div>
<div id="spotify">
<a href="https://open.spotify.com/show/53psoLoX187axvmgb80l1x">
<img src="/img/spotify-podcast-badge-blk-grn-330x80.svg"
title="Listen on Spotify"
alt="Listen on Spotify">
</a>
</div>
</div>
<div id="aggregators-2">
<div id="podbean">
<a href="https://www.podbean.com/podcast-detail/2r2tz-31b053/Organising-Tech-in-Sweden-Podcast"
rel="noopener noreferrer" target="_blank">
<img src="https://pbcdn1.podbean.com/fs1/site/images/badges/w600_1.png"
title="Listen on Podbean"
alt="Listen on Podbean">
</a>
</div>
</div>
</div>
<div id="main">
<div id="header">
<h1 id="title" class="header">Episode 1 is out now!</h1>
<!-- <h1 id="title" class="header"><a href="episodes/">Episodes</a></h1> -->
<div id="socials">
<a href="https://x.com/orgtech_se">
<img src="/img/twitter-color-svgrepo-com.svg"
title="Follow us on Twitter!"
alt="Twitter logo" />
</a>
<a href="https://bsky.app/profile/orgtech-se.bsky.social">
<img src="/img/bluesky-logo.svg"
title="Follow us on Bluesky!"
alt="Bluesky logo" />
</a>
</div>
</div>
<div class="text">
<p>
Organising Tech in Sweden is a limited podcast series exploring union
organising in Swedish tech companies. Join us as we sit down with some
of the people involved in the campaigns to win collective bargaining
rights at two of Sweden's tech unicorns, Klarna and Spotify.
</p>
<div id="production-info">
<div>
<p>
Listen to our latest episode:<br />
🔊 <a href="/episodes/ep01-klarna-part1">Organising Klarna - Part 1</a>
</p>
<p>
Produced by Hakuna Matata Produktion
</p>
<p>
Cover art by <a href="https://anyakjordan.com/">Anya K. Jordan</a>
<a href="https://bsky.app/profile/anyakjordan.bsky.social">@anyakjordan.bsky.social</a>
</p>
<p>
Theme music by <a href="https://soundcloud.com/ptzery">Ptzery</a>
</p>
</div>
<div id="hmp-logo">
<img src="/img/hakuna-matata-produktion.png"
title="Hakuna Matata Produktion"
alt="Hakuna Matata Produktion logo">
</div>
</div>
</div>
</div>
</div>
<div id="news">
<h1>News</h1>
<h2>Episode 1 is out!</h2>
<p>🔊 <a href="/episodes/ep01-klarna-part1">Organising Klarna - Part 1</a></p>
<p>
We kick off Organising Tech in Sweden in style by recounting the story of
how a collective bargaining agreement (CBA) was won at Klarna, a major
Swedish fintech. In fact, Klarna was the first unicorn in Sweden to be
unionised (and probably the first unicorn in Europe as well)!
</p>
<p>
To hear all about how this went down, your co-hosts Josh and Ray are joined by
Thomas, the founder of the Klarna Unionen Club (a union "local", to use
terminology that might be more familiar to US listeners); Sen, the chair of
the club who won the bargaining agreement against the odds; and Kim, a former
Klarna employee with extensive knowledge of Swedish labour law and market
policy.
</p>
</div>
</body>
</html>
We can grab all the nice images and such from the interwebs:
: ~/code/orgtech-se; curl \
https://orgtech.se/orgtech-se-favicon-and-img.tar.gz \
| tar xvz -C public
And of course we need to make it nice and responsive so it looks good both on a computer screen and a mobile phone screen. Let's create public/css/main.css and drop some stylish styles therein:
body {
font:
1.2em Helvetica,
Arial,
sans-serif;
margin: 20px;
padding: 0;
}
body > div {
max-width: 100%;
margin-left: auto;
margin-right: auto;
}
img {
max-width: 100%;
}
a {
text-decoration: none;
&:hover {
text-decoration: underline;
}
}
#header {
display: flex;
justify-content: space-between;
align-items: center;
gap: 10px;
}
h1 {
font-weight: bold;
font-size: larger;
}
#socials {
display: flex;
gap: 10px;
}
#socials img {
max-width: 32px;
&:hover {
transform: scale(1.1);
}
}
@media screen and (min-width: 600px) {
body > div {
max-width: 800px;
margin-top: 1em;
}
#wrapper {
display: flex;
}
#cover-image {
margin-right: 20px;
max-width: 40%;
}
}
Now we can fire up a local webserver:
: ~/code/orgtech-se; bb dev
Serving assets at http://localhost:1341
Serving static assets at http://localhost:1341
nREPL server started on port 1339...
Websocket server started on 1340...
Gaze ye now upon the glories of http://localhost:1341!
We of course have already registered a domain and done the intricate dance of setting up S3 static website hosting and CloudFront and all of that, so all we need to do to publish our website is copy some files into our S3 bucket. And of course, what better way to do this than with a Babashka task?
As avid REPL-drivers, we want to use our REPL for task development as well, so the first thing we do is create a tasks.clj with a boring publish function in it:
(ns tasks)
(defn publish [{:keys [website-bucket out-dir] :as opts}]
(println (format "Publishing %s/ to s3://%s/"
out-dir website-bucket)))
Now we need to hook that up to bb.edn by setting the classpath appropriately, pulling in our new tasks namespace, defining some options, and adding a publish task:
{:deps { ... }
:paths ["."]
:tasks
{:requires ([tasks])
:init (def opts
{:website-bucket "orgtech.se"
:out-dir "public"})
;; ...
publish (tasks/publish opts)}}
We can now test this:
: ~/code/orgtech-se; bb publish
Publishing public/ to s3://orgtech.se/
Jumping back to tasks.clj, we fire up a trusty CIDER REPL with a C-c M-j (cider-jack-in-clj) flourish, followed by C-c C-k (cider-load-buffer) to evaluate the buffer (readers following along with an inferior text editor will have to perform whatever complex ritual necessary to start a REPL and connect to it and then evaluate the "file" or whatever your text editor calls the thing you're editing).
Thus equipped, we can open up a Rich
comment, define some opts, and evaluate our publish function:
(comment
(def opts {:website-bucket "orgtech.se"
:out-dir "public"}) ; C-c C-v f c e
;; => #'tasks/opts
(publish opts) ; C-c C-e
)
Our REPL buffer now looks something like this:
Started nREPL server at 127.0.0.1:44571
For more info visit: https://book.babashka.org/#_nrepl
;; Connected to nREPL server - nrepl://127.0.0.1:44571
;; CIDER 1.12.0 (Split), babashka.nrepl 0.0.6-SNAPSHOT
;; Babashka 1.3.188
;; Docs: (doc function-name)
;; (find-doc part-of-name)
;; Source: (source function-name)
;; Javadoc: (javadoc java-object-or-class)
;; Exit: <C-c C-q>
;; Results: Stored in vars *1, *2, *3, an exception in *e;
;; Startup: /home/jmglov/.nix-profile/bin/bb nrepl-server localhost:0
Publishing public/ to s3://orgtech.se/
user>
OK, now it's time to figure out how to do the actual copying of files to S3. We could of course use the spectacular awyeah-api to do stuff to AWS right from our Clojure code, but that smacks of effort. 🤔
Fortunately, we remember that Babashka was originally conceived as a replacement for Bash shell scripting (I mean, the "bash" is right there in the name, so that's kind of a major clue), and we know that there's an AWS command line tool that knows how to sync stuff from a local directory to S3:
: ~/code/orgtech-se; aws s3 sync help
SYNC() SYNC()
NAME
sync -
DESCRIPTION
Syncs directories and S3 prefixes. Recursively copies new and updated
files from the source directory to the destination. Only creates fold-
ers in the destination if they contain one or more files.
SYNOPSIS
sync
<LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
[...]
EXAMPLES
The following sync command syncs objects from a local diretory to the
specified prefix and bucket by uploading the local files to s3. A lo-
cal file will require uploading if the size of the local file is dif-
ferent than the size of the s3 object, the last modified time of the
local file is newer than the last modified time of the s3 object, or
the local file does not exist under the specified bucket and prefix.
In this example, the user syncs the bucket mybucket to the local cur-
rent directory. The local current directory contains the files
test.txt and test2.txt. The bucket mybucket contains no objects:
aws s3 sync . s3://mybucket
Output:
upload: test.txt to s3://mybucket/test.txt
upload: test2.txt to s3://mybucket/test2.txt
[...]
This looks like just the thing we need, so let's use the power of babashka.process to invoke aws s3 sync:
(ns tasks
(:require [babashka.process :as p]))
(defn publish [{:keys [website-bucket out-dir] :as opts}]
(let [sync-cmd ["aws s3 sync"
(format "%s/" out-dir)
(format "s3://%s/" website-bucket)]]
(apply println sync-cmd)
(apply p/shell sync-cmd)))
(comment
(def opts {:website-bucket "orgtech.se"
:out-dir "public"}) ; C-c C-v f c e
;; => #'tasks/opts
(publish opts) ; C-c C-e
)
After a brief delay, our REPL buffer now helpfully tells us:
aws s3 sync public/ s3://orgtech.se/
user>
And if we have a look in that there bucket, we see some files:
: ~/code/orgtech-se; aws s3 ls --recursive s3://orgtech.se/
2024-08-23 10:40:05 102613 android-chrome-192x192.png
2024-08-23 10:40:05 337153 android-chrome-512x512.png
2024-08-23 10:40:05 96934 apple-touch-icon.png
2024-08-23 10:40:05 246 browserconfig.xml
2024-08-23 10:40:05 720 css/main.css
2024-08-23 10:40:05 47189 favicon-16x16.png
2024-08-23 10:40:05 48597 favicon-32x32.png
2024-08-23 10:40:05 12014 favicon.ico
2024-08-23 10:40:05 745 img/bluesky-logo.svg
2024-08-23 10:40:05 3231594 img/orgtech-se-cover.jpg
2024-08-23 10:40:05 513884 img/orgtech-se-preview.jpg
2024-08-23 10:40:05 1943 img/twitter-color-svgrepo-com.svg
2024-08-23 10:40:05 1933 img/volume.png
2024-08-23 10:40:05 3074 index.html
2024-08-23 10:40:05 33084 mstile-150x150.png
2024-08-23 10:40:05 426 site.webmanifest
And now browsing to https://orgtech.se/ reveals a lovely little website that looks just like the one on http://localhost:1341. 🎉
Let's change the header in public/index.html to test out the syncing:
<h1 id="title" class="header">Coming Thursday, 12 September!</h1>
Before we YOLO eval our publish function again, we notice that aws s3 sync has a lovely little --dryrun option, which doesn't actually do the stuff but rather prints out what stuff it would do. Let's implement this!
tasks.clj
(defn publish [{:keys [website-bucket out-dir dryrun]
:as opts}]
(let [sync-cmd (concat ["aws s3 sync"]
(when dryrun ["--dryrun"])
[(format "%s/" out-dir)
(format "s3://%s/" website-bucket)])]
(apply println sync-cmd)
(apply p/shell sync-cmd)))
(comment
(publish (assoc opts :dryrun true)) ; C-c C-e
)
The REPL window helpfully says:
aws s3 sync --dryrun public/ s3://orgtech.se/
user>
but we don't see the output of the aws s3 sync command itself. This is due to the REPL not capturing stdout for the subprocess, I guess. We can handle this thusly:
(ns tasks
(:require [babashka.process :as p]))
(defn shell [& args]
(let [p (apply p/shell {:out :string
:err :string
:continue true}
args)]
(println (:out p))
(when-not (zero? (:exit p))
(println (:err p)))
p))
(defn publish [{:keys [website-bucket out-dir dryrun]
:as opts}]
(let [sync-cmd (concat ["aws s3 sync"]
(when dryrun ["--dryrun"])
[(format "%s/" out-dir)
(format "s3://%s/" website-bucket)])]
(apply println sync-cmd)
(apply shell sync-cmd)))
(comment
(publish (assoc opts :dryrun true)) ; C-c C-e
)
And now the REPL sez:
aws s3 sync --dryrun public/ s3://orgtech.se/
(dryrun) upload: public/index.html to s3://orgtech.se/index.html
user>
This is what we expect to see: only index.html will be uploaded, since it's the only thing that has changed.
It would be nice to run this from the command line, but we currently have no way of passing the dryrun option through short of adding it to the opts map in bb.edn. Fortunately for us, there's babashka-cli, which does all sorts of awesome command-line parsing! Let's put it to work:
(ns tasks
(:require [babashka.cli :as cli]
[babashka.process :as p]))
;; ...
(comment
(cli/parse-opts ["--website-bucket" "orgtech.se"
"--out-dir" "public"
"--dryrun"])
;; => {:website-bucket "orgtech.se", :out-dir "public", :dryrun true}
)
Now we can use parse-opts in our publish function like so:
(defn publish [default-opts]
(let [{:keys [website-bucket out-dir dryrun]
:as opts} (merge default-opts
(cli/parse-opts *command-line-args*))
sync-cmd (concat ["aws s3 sync"]
(when dryrun ["--dryrun"])
[(format "%s/" out-dir)
(format "s3://%s/" website-bucket)])]
(apply println sync-cmd)
(apply shell sync-cmd)))
Running this from the command line, we get the desired result:
: ~/code/orgtech-se; bb publish --dryrun
aws s3 sync --dryrun public/ s3://orgtech.se/
(dryrun) upload: public/index.html to s3://orgtech.se/index.html
And if we omit the --dryrun arg:
: ~/code/orgtech-se; bb publish --dryrun
aws s3 sync --dryrun public/ s3://orgtech.se/
upload: public/index.html to s3://orgtech.se/index.html
Amazing!
If we open https://orgtech.se/index.html in a browser and view source, however, we get a nasty surprise: the lovely newline we added at the end of the file isn't there! What in the world is going on here?
Well, it turns out that one of the primary functions of a CDN (Content Distribution Network) like CloudFront is to cache responses so every request that hits an endpoint doesn't have to go all the way back to the origin (in this case, our S3 bucket) to serve the response. So we've fallen prey to #2 in the list of the 4 hardest problems in Computer Science:
What to do, what to do?
Luckily for us, CloudFront gives us a way to invalidate the cache so the first request for a given endpoint re-fetches from the origin. Even more luckily for us, the AWS CLI surfaces this:
: ~/code/orgtech; aws cloudfront create-invalidation help
CREATE-INVALIDATION() CREATE-INVALIDATION()
NAME
create-invalidation -
DESCRIPTION
Create a new invalidation.
See also: AWS API Documentation
SYNOPSIS
create-invalidation
--distribution-id <value>
[--paths <value>]
[...]
OPTIONS
--distribution-id (string) The distribution's id.
--paths (string) The space-separated paths to be invalidated.
[...]
So what we can do is create an invalidation right after syncing to the S3 bucket in our publish function. In order to do this, we'll need a distribution ID. Let's ask CloudFront about the distributions we have:
: ~/code/orgtech; aws cloudfront list-distributions \
| bb -i '(let [ds (-> (str/join "\n" *input*)
(json/parse-string true)
(get-in [:DistributionList :Items]))]
(map (juxt #(get-in % [:Aliases :Items 0]) :Id) ds))'
(["www.jmglov.net" "F2ABC12UVWXYZ9"]
["politechspod.com" "F7E33IJKLMN0P6"]
["www.orgtech.se" "FDCBA42RSTUV3"])
This looks like the one we're after:
["www.orgtech.se" "FDCBA42RSTUV3"]
Let's go ahead and add the distribution ID to our bb.edn:
{ ; ...
{:requires ([tasks])
:init (def opts
{:website-bucket "orgtech.se"
:out-dir "public"
:distribution-id "FDCBA42RSTUV3"})
;; ...
}}
Now we can use this in tasks.clj:
(defn publish [default-opts]
(let [{:keys [website-bucket out-dir distribution-id dryrun]
:as opts} (merge default-opts
(cli/parse-opts *command-line-args*))
sync-cmd (concat ["aws s3 sync"]
(when dryrun ["--dryrun"])
[(format "%s/" out-dir)
(format "s3://%s/" website-bucket)])
invalidate-cmd ["aws cloudfront create-invalidation"
"--distribution-id" distribution-id
"--paths" :???]]
;; ...
))
OK, now where can we get our paths? Well, recall that aws s3 sync --dryrun helpfully outputs what is to be done:
aws s3 sync --dryrun public/ s3://orgtech.se/
(dryrun) upload: public/index.html to s3://orgtech.se/index.html
Let's consume this from Babashka to grab the paths! First, we'll dirty the dishes:
: orgtech-se; touch public/index.html public/css/main.css
: orgtech-se; aws s3 sync --dryrun public/ s3://orgtech.se/
(dryrun) upload: public/css/main.css to s3://orgtech.se/css/main.css
(dryrun) upload: public/index.html to s3://orgtech.se/index.html
And then parse that output in our tasks.clj:
(ns tasks
(:require ; ...
[clojure.string :as str]))
(comment
(def default-opts {:website-bucket "orgtech.se"
:out-dir "public"
:distribution-id "FDCBA42RSTUV3"}) ; C-c C-v f c e
;; => #'tasks/default-opts
(->> (shell "aws s3 sync --dryrun public/ s3://orgtech.se/")
:out
str/split-lines
(map #(str/replace % #"^[(]dryrun[)] upload: public(/\S+) to .+$" "$1")))
;; => ("/css/main.css" "/index.html")
)
Now that we know how to determine which files have changed, let's plug this into our publish function to add to the aws cloudfront create-invalidation command:
(defn publish [default-opts]
(let [{:keys [website-bucket out-dir distribution-id dryrun]
:as opts} (merge default-opts
(cli/parse-opts *command-line-args*))
sync-cmd (concat ["aws s3 sync"]
(when dryrun ["--dryrun"])
[(format "%s/" out-dir)
(format "s3://%s/" website-bucket)])
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
paths-re (re-pattern (format "^[(]dryrun[)] upload: %s(/\\S+) to .+$"
out-dir))
invalidate-cmd (concat ["aws cloudfront create-invalidation"
"--distribution-id" distribution-id
"--paths"]
(->> (apply shell (concat sync-cmd ["--dryrun"]))
:out
str/split-lines
(map #(str/replace % paths-re "$1"))))
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
]
(apply println sync-cmd)
(apply shell sync-cmd)
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇
(apply println invalidate-cmd)
(when-not dryrun
(apply shell invalidate-cmd))
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆
))
(comment
(publish (assoc default-opts :dryrun true)) ; C-c C-e
)
Our REPL buffer duly notes:
aws s3 sync --dryrun public/ s3://orgtech.se/
(dryrun) upload: public/css/main.css to s3://orgtech.se/css/main.css
(dryrun) upload: public/index.html to s3://orgtech.se/index.html
aws cloudfront create-invalidation --distribution-id FDCBA42RSTUV3
--paths /css/main.css /index.html
Looks good, so let's try it for realz:
: ~/code/orgtech; bb publish
aws s3 sync public/ s3://orgtech.se/
aws cloudfront create-invalidation --distribution-id FDCBA42RSTUV3
--paths /css/main.css /index.html
{
"Location": "https://cloudfront.amazonaws.com/2020-05-31/distribution/FDCBA42RSTUV3/invalidation/ICECSBHVIW089I89RLYODUBMXI",
"Invalidation": {
"Id": "ICECSBHVIW089I89RLYODUBMXI",
"Status": "InProgress",
"CreateTime": "2024-08-25T07:14:55.130Z",
"InvalidationBatch": {
"Paths": {
"Quantity": 2,
"Items": [
"/css/main.css",
"/index.html"
]
},
"CallerReference": "cli-1724570094-253923"
}
}
}
This is promising. Let's refill our coffee and then check to see if the invalidation has finishing invalidating:
: ~/code/orgtech; aws cloudfront get-invalidation \
--distribution-id FDCBA42RSTUV3 \
--id ICECSBHVIW089I89RLYODUBMXI
{
"Invalidation": {
"Id": "ICECSBHVIW089I89RLYODUBMXI",
"Status": "Completed",
"CreateTime": "2024-08-25T07:14:55.130Z",
"InvalidationBatch": {
"Paths": {
"Quantity": 2,
"Items": [
"/css/main.css",
"/index.html"
]
},
"CallerReference": "cli-1724570094-253923"
}
}
}
If we now Shift-reload the page in our browser, we'll see the wonderful new header! 🎉
Now that we have an amazing website and a way to publish it, let's record an episode and then make a nice trailer to get people pumped up! We'll use Zencastr to do this, which produces a lovely MP3 for us as well as a transcript. For now, we have the following files on disk:
: organising-tech-in-sweden; tree
.
├── bb.edn
├── ep00-trailer
│ ├── otis-ep00-trailer.mp3
│ └── otis-ep0-trailer_transcription.txt
├── ep01-klarna-part1
│ ├── otis-ep01-klarna-part1.mp3
│ └── otis-ep01-klarna-part1_transcription.txt
├── public
│ ├── ...
│ └── index.html
└── tasks.clj
Zencastr's transcripts are, um, functional, shall we say, but any machine transcription tool will require a human who speaks the actual language being transcribed (in this case, English) to clean things up. Luckily, there's an amazing free (as in source and as in beer!) browser-based tool called oTranscribe that lets us listen to our lovely audio whilst editing the transcript, with keyboard shortcuts for pausing and resuming playback, rewinding and fast forwarding, adjusting playback speed, etc.
To unlock all this goodness, we'll need to convert our boring Zencastr transcripts, which look like this:
00:02.00
jmglov
Already and we are live now. So welcome everyone to organizing tech in Sweden
I am here my name is Josh I'm here with a ah. Cast of characters that will
delight in a maze and I will introduce them here in a minute but before we get
to the cool people. Let me introduce. My co-host Ray joining us all the way from
Belgium Ray you want to say hey.
00:30.61
Ray
Yeah on uncool Belgium hello everyone? Well it's a bit warmer than Sweden now.
But okay I yeah you were talking about being oh yeah, okay fine.
00:42.78
jmglov
Yeah, all right? So we are here like I said to talk about organizing tech in
Sweden and um, basically what we want to do is introduce. Folks who might not
know much about Sweden other than Ekea is from here and chocolate. Oh no wait.
That's Switzerland for some reason and the us. Oh you know Belgium sure sure.
Sure. Um.
[...]
into amazing OTR (oTranscribe's file format) ones, which look like some HTML stuffed into some JSON.
To do this converting, we could use Transcribble, which I wrote a while back and forgot to blog about. Or we could just open up https://otranscribe.com/ in our browser and click the big blue "Start transcribing" button, then click the "Choose audio (or video) file" button and choose our ~/code/orgtech-se/ep01-klarna-part1/otis-ep01-klarna-part1.mp3 file, and then paste in our Zencastr transcript, warts and all. If we click the Play button (or hit Esc, which is oTranscribe's play/pause keyboard shortcut), our episode will start playing, and we can hit Ctrl+J to add a timestamp to the transcript when we hear me say "So, welcome everyone to Organising Tech in Sweden". After much listening and editing, which we will just handwave away here, we now have a pristine transcript!
We'll now click the "Export" button to pop up the "Download transcript as..." dialog, select "oTranscribe format (.otr)", and save as a new ~/code/orgtech-se/ep01-klarna-part1/otis-ep01-klarna-part1.otr file.
Now that we have some files, we need to stuff those in a feed. Luckily, we have some experience with podcast feeds. Using Selmer to write the feed worked out pretty nicely then, so let's elect to do the same thing again. In fact, since we already did the hard work of creating code that knows how to write an RSS file for a music album, why don't we see if we can modify it a bit to support podcasts as well?
Let's pop over to ~/code/soundcljoud/processor/main.clj and remind ourselves how we turned an album into an RSS feed:
(defn process-album [opts dir]
(let [info (album-info opts dir)
tmpdir (fs/create-temp-dir {:prefix "soundcljoud."})
info (update info :tracks (partial map #(process-track % tmpdir)))]
(spit (fs/file tmpdir "album.rss") (rss/album-feed opts info))
(assoc info :out-dir tmpdir)))
In this case, we fetched Discogs metadata for the album in the album-info function, created a temporary directory, did some transcoding in process-track, then used rss/album-feed to apply a Selmer template to our album metadata. Opening up the soundcljoud.rss namespace, we see that the album-feed function is extremely specific to music albums:
(defn album-feed [opts album-info]
(let [template (-> (io/resource "album-feed.rss") slurp)]
(selmer/render template
(-> album-info
(update :tracks
(partial map #(update % :mp3-filename
fs/file-name)))
(assoc :date (now))))))
Whilst there's no obvious way to repurpose it, we can follow the same basic pattern:
Let's sketch out a podcast-feed function:
(defn podcast-feed [opts podcast-info] ;; ❓ podcast-info how?
(let [template :???] ;; ❓ where do we get this?
(->> podcast-info
;; ❓ maybe some massaging here?
(selmer/render template))))
The first question is where we get the podcast-info. We got album-info from Discogs, but since Discogs presumably knows nothing about our podcast (and why would it?), let's create a static ~/code/orgtech/podcast.edn file instead, fill it with whatever data our podcast feed will need (I guess it's time to rhyme), and read in the EDN before calling this function.
Having made that decision, we must now ask ourselves where we will get our podcast feed template from. In the case of albums, we provided a template as a resource directly from Soundcljoud, so why don't we do that again?
(defn podcast-feed [opts podcast-info]
(let [template (-> (io/resource "podcast-feed.rss") slurp)]
(->> podcast-info
;; ❓ maybe some massaging here?
(selmer/render template))))
In order to know what if any massaging podcast-info will need, we'll need to create the template and the podcast.edn file and see where the gaps are. Let's consult Apple's handy A Podcaster’s Guide to
RSS and start writing resources/podcast-feed.rss. First, we need the standard feed skeleton:
<?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0"
xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<!-- TBD: some stuff here -->
</channel>
</rss>
Now to start populating the contents <channel> tag. According to Apple, we need the following:
| Show tags | Usage | Parent tag |
|---|---|---|
<title> | The show title. | <channel> |
<description> | The show description. | <channel> |
<itunes:image> | The artwork for the show. | <channel> |
<language> | The language spoken on the show. | <channel> |
<itunes:explicit> | The podcast parental advisory information. | <channel> |
<itunes:category> | The show category information. | <channel> |
This is straightforward enough (except for <itunes:category>, which we'll come back to):
<channel>
<title>{{podcast.title}}</title>
<description>{{podcast.description|safe}}</description>
<itunes:image href="{{base-url}}{{podcast.image}}"/>
<language>{{podcast.language}}</language>
<itunes:explicit>{{podcast.explicit}}</itunes:explicit>
</channel>
By the way, that {{podcast.description|safe}} thingy is a Selmer filter that exempts the variable from being
HTML-escaped. Since our description text goes in the body of the <description> tag, we don't want things like "rock & roll" getting rendered as "rock & roll", because that would be yucky.
Now we need to add that data to our podcast.edn:
{:base-url "https://orgtech.se"
:podcast {:title "Organising Tech in Sweden"
:description "Organising Tech in Sweden is a limited podcast series exploring union organising in Swedish tech companies. Join us as we sit down with some of the people involved in the campaigns to win collective bargaining rights at two of Sweden's tech unicorns, Klarna and Spotify."
:image "/img/orgtech-se-cover.jpg"
:language "en"
:explicit true}}
As we were writing podcast.edn, we realised that podcast-info was actually the data in the EDN file under the :podcast key, so we are really just providing an opts:
(defn podcast-feed [opts]
(let [template (-> (io/resource "podcast-feed.rss") slurp)]
(->> opts
;; ❓ maybe some massaging here?
(selmer/render template))))
Let's give this a go in our REPL:
(comment
(require '[clojure.edn :as edn])
;; => nil
(def opts (-> (slurp "/home/jmglov/code/orgtech-se/podcast.edn")
(edn/read-string)))
;; => #'soundcljoud.rss/opts
opts
;; => {:base-url "https://orgtech.se",
;; :podcast
;; {:title "Organising Tech in Sweden",
;; :description
;; "Organising Tech in Sweden is a...",
;; :image "/img/orgtech-se-cover.jpg",
;; :language "en",
;; :explicit true}}
(podcast-feed opts)
;; => "<?xml version='1.0' encoding='UTF-8'?>
;; <rss version=\"2.0\"
;; xmlns:itunes=\"http://www.itunes.com/dtds/podcast-1.0.dtd\"
;; xmlns:atom=\"http://www.w3.org/2005/Atom\">
;; <channel>
;; <title>Organising Tech in Sweden</title>
;; <description>Organising Tech in Sweden is a...</description>
;; <itunes:image href=\"https://orgtech.se/img/orgtech-se-cover.jpg\"/>
;; <language>en</language>
;; <itunes:explicit>true</itunes:explicit>
;; </channel>
;; </rss>"
)
Let's now come back to that tricky <itunes:category> tag. Referring back to the Apple docs, we see:
For a complete list of categories and subcategories, see Apple Podcast categories.
Select the category that best reflects the content of your show. If available, you can also define a subcategory.
Single category:
<itunes:category text="History" />Category with subcategory:
<itunes:category text="Society & Culture"><itunes:category text="Documentary" /></itunes:category>
We can add categories to our podcast-feed.rss template using Selmer's for tag:
<channel>
<!-- ... -->
{% for category in podcast.categories %}
<itunes:category text="{{category.text}}">
{% for subcategory in category.subcategories %}
<itunes:category text="{{subcategory.text}}" />
{% endfor %}
</itunes:category>
{% endfor %}
</channel>
And now we need to pick a category or two and add them to our podcast.edn. The Apple Podcasts
categories page lists the options, of which we choose:
Expressing this in EDN, we get:
{:base-url "https://orgtech.se"
:podcast { ; ...
:categories [{:text "Technology"}
{:text "News"
:subcategories [{:text "Politics"}]}]}}
And our REPL shows us what we'd expect to see:
(comment
(def opts (-> (slurp "/home/jmglov/code/orgtech-se/podcast.edn")
(edn/read-string)))
;; => #'soundcljoud.rss/opts
(podcast-feed opts)
;; => "<?xml version='1.0' encoding='UTF-8'?>
;; <rss version=\"2.0\"
;; xmlns:itunes=\"http://www.itunes.com/dtds/podcast-1.0.dtd\"
;; xmlns:atom=\"http://www.w3.org/2005/Atom\">
;; <channel>
;; ...
;; <itunes:category text=\"Technology\">
;; </itunes:category>
;; <itunes:category text=\"News\">
;; <itunes:category text=\"Politics\" />
;; </itunes:category>
;; </channel>
;; </rss>"
)
Having sorted our required tags, let's take a look at Apple's recommended and "situational" tags (which we'll just treat as "recommended"):
| Show tags | Usage | Parent tag |
|---|---|---|
<itunes:author> | The group responsible for creating the show. | <channel> |
<link> | The website associated with a podcast. | <channel> |
<itunes:title> | The show title specific for Apple Podcasts. | <channel> |
<itunes:type> | The type of show. Its values can be one of the following: | |
| • Episodic. Episodes are intended to be consumed without any specific order. | ||
| • Serial. Episodes are intended to be consumed in sequential order. | ||
<copyright> | The show copyright details. | <channel> |
Again, this is quite straightforward to add to our template:
<channel>
<!-- ... -->
<itunes:author>{{podcast.author}}</itunes:author>
<link>{{base-url}}</link>
<itunes:title>{{podcast.title}}</itunes:title>
<itunes:type>{{podcast.type}}</itunes:type>
<copyright>{{podcast.copyright}}</copyright>
</channel>
and to our podcast.edn:
{:base-url "https://orgtech.se"
:podcast { ; ...
:author "Organising Tech in Sweden"
:type "Serial"
:copyright "All rights reserved, Organising Tech in Sweden"}}
Testing things out in our REPL, we see what we expect to see. 🙂
Now it's time to add some episodes! Here are the Apple Podcast required, recommended, and situational tags for episodes:
| Show tags | Usage | Parent tag |
|---|---|---|
<title> | An episode title. | <item> |
<enclosure> | The episode content, file size, and file type information. The <enclosure> tag has three attributes: | <item> |
| • URL. The URL attribute points to your podcast media file. | ||
| • Length. The length attribute is the file size in bytes. | ||
| • Type. The type attribute provides the correct category for the type of file. | ||
<guid> | The episode’s globally unique identifier (GUID) | <item> |
<pubDate> | The date and time when an episode was released. Format the date using the RFC 2822 specifications. For example: Sat, 01 Apr 2023 19:00:00 GMT. | <item> |
<description> | An episode description. | <item> |
<itunes:duration> | The duration of an episode. Different duration formats are accepted however it is recommended to convert the length of the episode into seconds. | <item> |
<link> | An episode link URL. | <item> |
<itunes:explicit> | The podcast parental advisory information. | <item> |
<itunes:title> | The show title specific for Apple Podcasts. | <item> |
<itunes:episode> | An episode number. | <item> |
<itunes:episodeType> | The episode type. | <item> |
| • Full. Specify full when you are submitting the complete content of your show. | ||
| • Trailer. Specify trailer when you are submitting a short, promotional piece of content that represents a preview of your current show. | ||
| • Bonus. Specify bonus when you are submitting extra content for your show (for example, behind the scenes information or interviews with the cast) or cross-promotional content for another show. | ||
<itunes:transcript> | A link to the episode transcript in the Closed Caption format. | <item> |
Unfortunately, Transcribble doesn't yet support VTT or SRT transcripts, so we can't provide the transcript directly in iTunes. What we will do instead is display the OTR transcript that we previously prepared in oTranscribe on our episode page (which is yet to be written, but we'll get there in the end). In order to do this, let's add a custom <transcriptUrl> tag.
Let's start with our template as usual:
<channel>
<!-- ... -->
{% for episode in episodes %}
<item>
<title>{{episode.title}}</title>
<enclosure
url="{{base-url}}{{episode.path}}/{{episode.audio-file}}"
length="{{episode.audio-filesize}}"
type="{{episode.mime-type}}" />
<guid>{{base-url}}{{episode.path}}/{{episode.audio-file}}</guid>
<pubDate>{{episode.date}}</pubDate>
<description><![CDATA[{{episode.description|safe}}]]></description>
<itunes:duration>{{episode.duration}}</itunes:duration>
<link>{{base-url}}{{episode.path}}</link>
<itunes:title>{{episode.title}}</itunes:title>
{% if episode.number %}<itunes:episode>{{episode.number}}</itunes:episode>{% endif %}
<itunes:episodeType>{{episode.type}}</itunes:episodeType>
<transcriptUrl>{{base-url}}{{episode.path}}/{{episode.transcript-file}}</transcriptUrl>
</item>
{% endfor %}
</channel>
And now we know what episodes need to look like in our podcast.edn file:
{ ; ...
:episodes
[{:number 0
:date "Thu, 5 Sep 2024 00:00:00 +0000"
:type "Trailer"
:title "Trailer"
:summary "Union organising seems to be in the air these days, as tech workers wake up and realise that they are, in fact, workers."
:description "
<p>
Union organising seems to be in the air these days, as tech workers wake up and
realise that they are, in fact, workers. Here in Sweden, it's no exception.
Join us as we sit down with some of the people involved in organising two of
Sweden's foremost tech unicorns, Klarna and Spotify. This is Organising Tech in
Sweden.
</p>
<p class=\"soundcljoud-hidden\">
To view full show notes, including transcripts, please visit the
<a href=\"{{base-url}}{{episode.path}}/\">episode page</a>.
</p>
<p>
Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>
<a href=\"https://bsky.app/profile/anyakjordan.bsky.social\">@anyakjordan.bsky.social</a>
</p>
<p>
Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>
</p>"
:path "/episodes/ep00-trailer"
:audio-file "otis-ep00-trailer.mp3"
:transcript-file "otis-ep00-trailer.otr"
:explicit false
:mime-type "audio/mpeg"}
{:number 1
:date "Thu, 12 Sep 2024 00:00:00 +0000"
:type "Full"
:title "Organising Klarna - Part 1"
:summary "A conversation with three of the organisers behind the successful campaign to win a Collective Bargaining Agreement at Klarna"
:description "
<p>
We kick off Organising Tech in Sweden in style by recounting the story of how
a collective bargaining agreement (CBA) was won at Klarna, a major Swedish
fintech. In fact, Klarna was the first unicorn in Sweden to be unionised (and
probably the first unicorn in Europe as well)!
</p>
<p>
To hear all about how this went down, your co-hosts Josh and Ray are joined by
Thomas, the founder of the Klarna Unionen Club (a union \"local\", to use
terminology that might be more familiar to US listeners); Sen, the chair of
the club who won the bargaining agreement against the odds; and Kim, a former
Klarna employee with extensive knowledge of Swedish labour law and market
policy.
</p>
<p>
This is part 1 of the conversation, which will be concluded in Episode 2.
</p>
<p class=\"soundcljoud-hidden\">
To view full show notes, including transcripts, please visit the
<a href=\"{{base-url}}{{episode.path}}/\">episode page</a>.
</p>
<p>
Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>
<a href=\"https://bsky.app/profile/anyakjordan.bsky.social\">@anyakjordan.bsky.social</a>
</p>
<p>
Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>
</p>"
:path "/episodes/ep01-klarna-part1"
:audio-file "otis-ep01-klarna-part1.mp3"
:transcript-file "otis-ep01-klarna-part1.otr"
:explicit false
:mime-type "audio/mpeg"}
{:preview? true
:number 2
:date "Thu, 19 Sep 2024 00:00:00 +0000"
:type "Full"
:title "Organising Klarna - Part 2"
:summary "The conclusion of our conversation with three of the organisers behind the successful campaign to win a Collective Bargaining Agreement at Klarna"
:description "
<p>
We finish our conversation with Sen, Thomas, and Kim about how a collective
bargaining agreement (CBA) was won at Klarna. In this episode, we cover the
impact of immigrant workers on organising, the impact of organising on
organisers, and the impact of strikes on negotiations. All of this and a happy
ending too!
</p>
<p class=\"soundcljoud-hidden\">
To view full show notes, including transcripts, please visit the
<a href=\"{{base-url}}{{episode.path}}/\">episode page</a>.
</p>
<p>
Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>
<a href=\"https://bsky.app/profile/anyakjordan.bsky.social\">@anyakjordan.bsky.social</a>
</p>
<p>
Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>
</p>"
:path "/episodes/ep02-klarna-part2"
:audio-file "otis-ep02-klarna-part2.mp3"
:transcript-file "otis-ep02-klarna-part2.otr"
:explicit false
:mime-type "audio/mpeg"}]}
Testing this in our REPL...
(comment
(def opts (-> (slurp "/home/jmglov/code/orgtech-se/podcast.edn")
(edn/read-string)))
;; => #'soundcljoud.rss/opts
(podcast-feed opts)
;; => java.lang.NullPointerException soundcljoud.rss /home/jmglov/code/soundcljoud/processor/src/soundcljoud/rss.clj:34:26
)
...we get an unpleasant surprise. 😮
This is a bit annoying to debug, but we can surmise that one of the template variables in the episode template must be missing. Doing a little visual inspection identifies the culprit:
<enclosure
url="{{base-url}}{{episode.path}}/{{episode.audio-file}}"
length="{{episode.audio-filesize}}"
type="{{episode.mime-type}}" />
We don't have audio-filesize in our episode data structure. 😢
All is not lost, however. Let's cast our minds back to the definition of the podcast-feed function:
(defn podcast-feed [opts]
(let [template (-> (io/resource "podcast-feed.rss") slurp)]
(->> opts
;; ❓ maybe some massaging here?
(selmer/render template))))
The answer to the question "maybe some massaging here?" now reveals itself to be "Yes. Yes! A thousand times yes!" We also know at least one massage technique we're going to need to use, namely setting the audio-filesize key for each episode. Let's start out by giving ourselves a way to update episodes:
(defn update-episode [opts episode]
episode)
(defn update-episodes [opts]
(update opts :episodes #(map (partial update-episode opts) %)))
(defn podcast-feed [opts]
(let [template (-> (io/resource "podcast-feed.rss") slurp)]
(->> opts
update-episodes
(selmer/render template))))
Now we can figure out how to add the filesize to each episode. As usual, Babashka's got us covered! Checking out the babashka.fs API documentation, we find a function called babashka.fs/size:
size
(size f)Returns the size of a file (in bytes).
Let's mess around a bit in the REPL:
(comment
(def base-dir "/home/jmglov/code/orgtech-se")
;; => #'soundcljoud.rss/base-dir
(def opts (-> (slurp (fs/file base-dir "podcast.edn"))
(edn/read-string)
(assoc :base-dir base-dir)))
(let [episode (-> opts :episodes first)
filename (format "%s%s/%s"
base-dir (:path episode) (:audio-file episode))]
(fs/size filename))
;; => java.nio.file.NoSuchFileException:
;; /home/jmglov/code/orgtech-se/episodes/ep00-trailer/otis-ep00-trailer.mp3
;; /home/jmglov/code/soundcljoud/processor/src/soundcljoud/rss.clj:4:5
)
Oops! Seems like we've traded one problem for another. 😬
On disk, the files are laid out like this:
: organising-tech-in-sweden; tree
.
├── bb.edn
├── ep00-trailer
│ ├── otis-ep00-trailer.mp3
│ └── otis-ep0-trailer_transcription.txt
├── ep01-klarna-part1
│ ├── otis-ep01-klarna-part1.mp3
│ └── otis-ep01-klarna-part1_transcription.txt
├── ep02-klarna-part2
│ ├── otis-ep02-klarna-part2.mp3
│ └── otis-ep02-klarna-part2_transcription.txt
├── public
│ ├── ...
│ └── index.html
└── tasks.clj
But we are looking for the audio file in the path in which it should exist on the server, which makes sense from an RSS feed perspective, which should use paths corresponding to the published site. Our publish task uses aws s3 sync to publish everything in our public/ directory, so if we drop the MP3s there, they will get put in the correct place on the S3 website. For now, let's cheat by using our REPL to put the files where they need to go:
(comment
(def opts (-> (slurp (fs/file base-dir "podcast.edn"))
(edn/read-string)
(assoc :base-dir base-dir
:out-dir "public")))
;; => #'soundcljoud.rss/opts
(doseq [episode (:episodes opts)
:let [filename (format "%s/%s%s/%s"
(:base-dir opts) (:out-dir opts)
(:path episode) (:audio-file episode))
src-filename (fs/file dir
(fs/file-name (:path episode))
(:audio-file episode))]]
(when-not (fs/exists? filename)
(fs/create-dirs (fs/parent filename))
(fs/copy src-filename filename)))
;; => nil
)
OK, this will do for now. Let's grab this code, clean it up a bit, and shove it into our update-episode function:
(defn update-episode [{:keys [base-dir out-dir] :as opts}
{:keys [audio-file path] :as episode}]
(assoc episode :audio-filesize
(fs/size (format "%s/%s%s/%s" base-dir out-dir path audio-file))))
Before testing this out in the REPL, we should add the :out-dir key to our podcast.edn so we don't rely on the caller to add it to opts:
{:base-url "https://orgtech.se"
:podcast { ... }
:episodes [ ... ]}
OK, now we're ready to give it a spin in the REPL:
(comment
(podcast-feed (assoc opts :out-dir "public"))
;; => "<?xml version='1.0' encoding='UTF-8'?>\n
;; <rss version=\"2.0\"\n
;; xmlns:itunes=\"http://www.itunes.com/dtds/podcast-1.0.dtd\"\n
;; xmlns:atom=\"http://www.w3.org/2005/Atom\">\n
;; <channel>\n
;; <title>Organising Tech in Sweden</title>\n
;; <description>Organising Tech in Sweden is a...</description>\n
;; <itunes:image href=\"https://orgtech.se/img/orgtech-se-cover.jpg\"/>\n
;; <language>en</language>\n
;; <itunes:explicit>true</itunes:explicit>\n\n
;; <itunes:category text=\"Technology\">\n\n
;; </itunes:category>\n\n
;; <itunes:category text=\"News\">\n\n
;; <itunes:category text=\"Politics\" />\n\n
;; </itunes:category>\n\n
;; <itunes:author>Organising Tech in Sweden</itunes:author>\n
;; <link>https://orgtech.se</link>\n
;; <itunes:title>Organising Tech in Sweden</itunes:title>\n
;; <itunes:type>Serial</itunes:type>\n
;; <copyright>All rights reserved</copyright>\n\n
;; <item>\n
;; <title>Trailer</title>\n
;; <enclosure\n
;; url=\"https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.mp3\"\n
;; length=\"1016937\"\n
;; type=\"audio/mpeg\" />\n
;; <guid>https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.mp3</guid>\n
;; <pubDate>Thu, 5 Sep 2024 00:00:00 +0000</pubDate>\n
;; <description><![CDATA[\n
;; <p>\n Union organising seems to be in the air these days...</p>\n
;; <p class=\"soundcljoud-hidden\">\n
;; To view full show notes, including transcripts, please visit the\n
;; <a href=\"{{base-url}}{{episode.path}}/\">episode page</a>.\n
;; </p>\n
;; <p>\n
;; Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>\n
;; </p>\n
;; <p>\n
;; Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>\n</p>
;; ]]></description>\n
;; <itunes:duration></itunes:duration>\n
;; <link>https://orgtech.se/episodes/ep00-trailer</link>\n
;; <itunes:title>Trailer</itunes:title>\n
;; <itunes:episode>0</itunes:episode>\n
;; <itunes:episodeType>Trailer</itunes:episodeType>\n
;; <transcriptUrl>
;; https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.otr
;; </transcriptUrl>\n
;; </item>\n\n
;; ...
;; </channel>\n
;; </rss>\n"
)
This looks like a good start, but a few things jump out at us:
<itunes:duration> tag is emptyLet's tackle the duration issue first, because we already have the tools to fix that in the soundcljoud.processor code that we wrote for Garth:
(ns soundcljoud.audio
(:require [babashka.fs :as fs]
[babashka.process :as p]
[cheshire.core :as json]
[clojure.string :as str]))
(defn mp3-duration [filename]
(-> (p/shell {:out :string}
"ffprobe -v quiet -print_format json -show_format -show_streams"
filename)
:out
(json/parse-string keyword)
:streams
first
:duration
(str/replace #"[.]\d+$" "")))
;; ...
Let's pull soundcljoud.audio into our namespace and then grab the duration in update-episode:
(ns soundcljoud.rss
(:require ; ...
[soundcljoud.audio :as audio])
(:import ...))
;; ...
(defn update-episode [{:keys [base-dir src-dir] :as opts}
{:keys [audio-file path] :as episode}]
(let [filename (format "%s/%s%s/%s" base-dir src-dir path audio-file)]
(assoc episode
:audio-filesize (fs/size filename)
:duration (audio/mp3-duration filename))))
;; ...
(comment
(podcast-feed opts)
;; => "<?xml version='1.0' encoding='UTF-8'?>\n
;; <rss version=\"2.0\" ...>\n
;; <channel>\n
;; <title>Organising Tech in Sweden</title>\n
;; <description>Organising Tech in Sweden is a...</description>\n
;; ...
;; <item>\n
;; <title>Trailer</title>\n
;; ...
;; <itunes:duration>42</itunes:duration>\n
;; ...
;; </item>\n\n
;; ...
;; </channel>\n
;; </rss>\n"
)
This looks good, so let's turn our roving eye to the last remaining problem.
After rendering our RSS feed template, we somehow still have unrendered Selmer in our output:
<description>
<![CDATA[
<p>
Union organising seems to be in the air these days...
</p>
<p class="soundcljoud-hidden">
To view full show notes, including transcripts, please visit the
<a href="{{base-url}}{{episode.path}}/">episode page</a>.
</p>
<p>
Cover art by <a href="https://anyakjordan.com/">Anya K. Jordan</a>
</p>
<p>
Theme music by <a href="https://soundcloud.com/ptzery">Ptzery</a>
</p>]]>
</description>
Let's see what's going on in our podcast-feed.rss template for episodes:
{% for episode in episodes %}
<item>
<title>{{episode.title}}</title>
...
<description><![CDATA[{{episode.description|safe}}]]></description>
...
</item>
{% endfor %}
So we're plugging episode.description into the template. Let's see what that looks like in our podcast.edn:
{ ; ...
:episodes
[{:number 0
:title "Trailer"
;; ...
:description "
<p>
Union organising seems to be in the air these days, as tech workers wake up and
realise that they are, in fact, workers. Here in Sweden, it's no exception.
Join us as we sit down with some of the people involved in organising two of
Sweden's foremost tech unicorns, Klarna and Spotify. This is Organising Tech in
Sweden.
</p>
<p class=\"soundcljoud-hidden\">
To view full show notes, including transcripts, please visit the
<a href=\"{{base-url}}{{episode.path}}/\">episode page</a>.
</p>
<p>
Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>
<a href=\"https://bsky.app/profile/anyakjordan.bsky.social\">@anyakjordan.bsky.social</a>
</p>
<p>
Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>
</p>"
;; ...
}
;; ...
]}
Ah-ha! The value of episode.description itself contains some templating. So it looks like we need to render that as well.
(defn update-episode [{:keys [base-dir src-dir] :as opts}
{:keys [audio-file path] :as episode}]
(let [filename (format "%s/%s%s/%s" base-dir src-dir path audio-file)]
(assoc episode
:audio-filesize (fs/size filename)
:duration (audio/mp3-duration filename)
:description (selmer/render (:description episode)
(assoc opts :episode episode)))))
;; ...
(comment
(podcast-feed opts)
;; => "<?xml version='1.0' encoding='UTF-8'?>\n
;; <rss version=\"2.0\" ...>\n
;; <channel>\n
;; <title>Organising Tech in Sweden</title>\n
;; <description>Organising Tech in Sweden is a...</description>\n
;; ...
;; <item>\n
;; <title>Trailer</title>\n
;; <description><![CDATA[\n
;; <p>\n Union organising seems to be in the air these days...</p>\n
;; <p class=\"soundcljoud-hidden\">\n
;; To view full show notes, including transcripts, please visit the\n
;; <a href=\"https://orgtech.se/episodes/ep00-trailer/\">episode page</a>.\n
;; </p>\n
;; <p>\n
;; Cover art by <a href=\"https://anyakjordan.com/\">Anya K. Jordan</a>\n
;; </p>\n
;; <p>\n
;; Theme music by <a href=\"https://soundcloud.com/ptzery\">Ptzery</a>\n</p>
;; ]]></description>\n
;; ...
;; </item>\n\n
;; ...
;; </channel>\n
;; </rss>\n"
)
OK, this looks much better! And in fact, it looks so much better that we can declare victory and move on to figuring out how to write this beautiful feed to disk!
To do that, let's jump back to our orgtech-se/bb.edn and add a task for rendering the feed. We'll need to add the Soundcljoud processor to our deps, then we can pretend we have a tasks/render function and call it:
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:git/sha "e203166a020509d126149ff8046489857ce5c89f"}
;; You can always depend on Soundcljoud!
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
io.github.jmglov/soundcljoud
{:local/root "/home/jmglov/code/soundcljoud/processor"}
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
io.github.jmglov/transcribble
{:local/root "/home/jmglov/code/transcribble/cli"}}
:paths ["."]
:tasks
{
;; ...
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
render {:doc "Create webpages from templates"
:task (tasks/render opts)}
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
;; ...
}}
And now we pop over to tasks.clj to implement the task. Sadly, we need to restart our REPL since we added a new dependency and I'm too lazy to learn how to use the new clojure.repl.deps.add-lib from Clojure 1.12 (added to Babashka in version 1.4.192). In Emacs, we can do this with C-c C-z (cider-switch-to-repl-buffer) to jump to the REPL buffer, then C-c C-q (cider-quit) to stop the REPL, then **C-c M-j** (cider-jack-in-clj) to start a new REPL. Easy peasy!
Thus armed with a new REPL, let's pull in the namespaces required to load our podcast.edn and then actually load our podcast.edn:
(ns tasks
(:require [babashka.cli :as cli]
[babashka.process :as p]
;; Pull in some new namespaces
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇
[babashka.fs :as fs]
[clojure.edn :as edn]
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆
))
(comment
(def default-opts {:website-bucket "orgtech.se"
:out-dir "public"
:distribution-id "FDCBA42RSTUV3"}) ; C-c C-v f c e
;; => #'tasks/default-opts
(def opts
(let [base-dir (str (fs/cwd))]
(merge default-opts
(-> (fs/file base-dir "podcast.edn")
slurp
edn/read-string
(assoc :base-dir base-dir)))))
;; => #'tasks/opts
opts
;; => {:website-bucket "orgtech.se/blog",
;; :out-dir "public",
;; :distribution-id "EPTUS11MTYJF7",
;; :base-url "https://orgtech.se",
;; :src-dir "public",
;; :podcast { ... },
;; :episodes [ ... ],
;; :base-dir "/home/jmglov/code/orgtech-se"}
)
To set ourselves up for success with soundcljoud.rss/podcast-feed, we know that we need our MP3 files in the right place. And in fact, we cheated a bit in our REPL to copy those files to the right place, which means we have some code lying around that we can use! And whilst we're at it, we should also copy the transcript files, since we're referring to them in the rendered feed.
(comment
(doseq [episode (:episodes opts)
file (map episode [:audio-file :transcript-file])
:let [filename (format "%s/%s%s/%s"
(:base-dir opts) (:src-dir opts)
(:path episode) file)
src-filename (fs/file (:base-dir opts)
(fs/file-name (:path episode))
file)]]
(when-not (fs/exists? filename)
(fs/create-dirs (fs/parent filename))
(fs/copy src-filename filename)))
;; => nil
(->> (fs/glob (fs/file (:base-dir opts) (:src-dir opts)) "episodes/**")
(map #(-> (str %)
(str/replace (:base-dir opts) ""))))
;; => ("/public/episodes/ep01-klarna-part1"
;; "/public/episodes/ep01-klarna-part1/otis-ep01-klarna-part1.otr"
;; "/public/episodes/ep01-klarna-part1/otis-ep01-klarna-part1.mp3"
;; "/public/episodes/ep02-klarna-part2"
;; "/public/episodes/ep02-klarna-part2/otis-ep02-klarna-part2.otr"
;; "/public/episodes/ep02-klarna-part2/otis-ep02-klarna-part2.mp3"
;; "/public/episodes/ep00-trailer"
;; "/public/episodes/ep00-trailer/otis-ep00-trailer.otr"
;; "/public/episodes/ep00-trailer/otis-ep00-trailer.mp3")
)
Now that the files are, well, filed, let's see about rendering the podcast feed.
(ns tasks
(:require ; ...
[soundcljoud.rss :as rss]))
(comment
(let [feed-file (fs/file (:src-dir opts) "feed.rss")]
(println (format "Writing RSS feed %s" feed-file))
(->> (rss/podcast-feed opts)
(spit feed-file)))
;; => nil
(slurp "public/feed.rss")
;; => "<?xml version='1.0' encoding='UTF-8'?>\n
;; <rss version=\"2.0\"\n
;; xmlns:itunes=\"http://www.itunes.com/dtds/podcast-1.0.dtd\"\n
;; xmlns:atom=\"http://www.w3.org/2005/Atom\">\n
;; <channel>\n
;; <title>Organising Tech in Sweden</title>\n
;; <description>Organising Tech in Sweden is a...</description>\n
;; <itunes:image href=\"https://orgtech.se/img/orgtech-se-cover.jpg\"/>\n
;; <language>en</language>\n
;; <itunes:explicit>true</itunes:explicit>\n
;; <itunes:category text=\"Technology\">\n
;; </itunes:category>\n
;; <itunes:category text=\"News\">\n
;; <itunes:category text=\"Politics\" />\n
;; </itunes:category>\n
;; <itunes:author>Organising Tech in Sweden</itunes:author>\n
;; <link>https://orgtech.se</link>\n
;; <itunes:title>Organising Tech in Sweden</itunes:title>\n
;; <itunes:type>Serial</itunes:type>\n
;; <copyright>All rights reserved, Organising Tech in Sweden</copyright>\n
;; <item>\n
;; <title>Trailer</title>\n
;; <enclosure\n
;; url=\"https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.mp3\"\n
;; length=\"1016937\"\n
;; type=\"audio/mpeg\" />\n
;; <guid>https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.mp3</guid>\n
;; <pubDate>Thu, 5 Sep 2024 00:00:00 +0000</pubDate>\n
;; <description><![CDATA[\n
;; <p>\n
;; Union organising seems to be in the air these days...
;; </p>]]>
;; </description>\n
;; <itunes:duration>42</itunes:duration>\n
;; <link>https://orgtech.se/episodes/ep00-trailer</link>\n
;; <itunes:title>Trailer</itunes:title>\n
;; <itunes:episode>0</itunes:episode>\n
;; <itunes:episodeType>Trailer</itunes:episodeType>\n
;; <transcriptUrl>https://orgtech.se/episodes/ep00-trailer/otis-ep00-trailer.otr</transcriptUrl>\n
;; </item>\n
;; ...
;; </channel>
;; </rss>
)
OK, now we have everything we need to write our render function, so let's get to it:
(defn render [default-opts]
(let [base-dir (str (fs/cwd))
{:keys [episodes src-dir] :as opts}
(merge default-opts
(cli/parse-opts *command-line-args*)
(-> (fs/file base-dir "podcast.edn")
slurp
edn/read-string
(assoc :base-dir base-dir)))
feed-file (fs/file src-dir "feed.rss")]
(doseq [{:keys [path] :as episode} (:episodes opts)
file (map episode [:audio-file :transcript-file])
:let [filename (format "%s/%s%s/%s" base-dir src-dir path file)
src-filename (fs/file base-dir (fs/file-name path) file)]]
(when-not (fs/exists? filename)
(fs/create-dirs (fs/parent filename))
(fs/copy src-filename filename)))
(println (format "Writing RSS feed %s" feed-file))
(->> (rss/podcast-feed opts)
(spit feed-file))))
(comment
(render default-opts)
;; => nil
)
We should now be able to aim our web browser at http://localhost:1341/feed.rss and see a lovely podcast feed.
As lovely as this loveliness is, our eye is inexorably and tragically drawn to one thing which we do not love:
<item>
...
<link>https://orgtech.se/episodes/ep00-trailer</link>
...
</item>
This page, dear reader, does not exist!
Where does this <link> thingy come from, and why do we need it anyway? Well, if we refer back to A Podcaster's Guide to
RSS, we see:
<link>An episode link URL. This is used when an episode has a corresponding webpage.
Ah, so it's an episode page we need, eh? Well, we have a bunch of info about the episode in our podcast.edn file, and some code that loops over episodes and does stuff in tasks/render, and a deep and abiding love for Selmer, so let's whip up an episode page template, then plug some stuff in whilst we're looping over episodes. We'll start with the template, which we'll drop into a new templates/episode-page.html file:
<!doctype html>
<html class="no-js" lang="">
<head>
<title>
{{podcast.title}} Episode {{episode.number}} - {{episode.title}}
</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<link rel="stylesheet" href="/css/main.css">
<!-- Favicon from https://realfavicongenerator.net/ -->
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
<!-- Social sharing (Facebook, Twitter, LinkedIn, etc.) -->
<meta name="title" content="{{podcast.title}} Episode {{episode.number}} - {{episode.title}}">
<meta name="twitter:title" content="{{podcast.title}} Episode {{episode.number}} - {{episode.title}}">
<meta property="og:title" content="{{podcast.title}} Episode {{episode.number}} - {{episode.title}}">
<meta property="og:type" content="website">
<meta name="description" content="{{episode.summary}}">
<meta name="twitter:description" content="{{episode.summary}}">
<meta property="og:description" content="{{episode.summary}}">
<meta name="twitter:url" content="{{base-url}}{{episode.path}}/index.html">
<meta property="og:url" content="{{base-url}}{{episode.path}}/index.html">
<meta name="twitter:image" content="{{base-url}}{{preview-image}}">
<meta name="twitter:card" content="summary_large_image">
<meta property="og:image" content="{{base-url}}{{preview-image}}">
<meta property="og:image:alt" content="{{podcast.image-alt}}">
</head>
<body>
<div id="wrapper">
<div id="left-side">
<img id="cover-image" src="{{podcast.image}}" alt="{{podcast.image-alt}}" />
<div id="aggregators-1">
<div id="apple">
<a class="apple-button"
href="https://podcasts.apple.com/us/podcast/organising-tech-in-sweden/id1766442275?itsct=podcast_box_badge&itscg=30200&ls=1">
<img src="https://tools.applemediaservices.com/api/badges/listen-on-apple-podcasts/badge/en-us?size=250x83&releaseDate=1725494400"
title="Listen on Apple Podcasts" alt="Listen on Apple Podcasts" class="apple-button">
</a>
</div>
<div id="spotify">
<a href="https://open.spotify.com/show/53psoLoX187axvmgb80l1x">
<img src="/img/spotify-podcast-badge-blk-grn-330x80.svg" title="Listen on Spotify"
alt="Listen on Spotify">
</a>
</div>
</div>
<div id="aggregators-2">
<div id="podbean">
<a href="https://www.podbean.com/podcast-detail/2r2tz-31b053/Organising-Tech-in-Sweden-Podcast"
rel="noopener noreferrer" target="_blank">
<img src="https://pbcdn1.podbean.com/fs1/site/images/badges/w600_1.png"
title="Listen on Podbean" alt="Listen on Podbean">
</a>
</div>
</div>
</div>
<div id="main">
<nav id="header">
<h1 id="title">{{episode.title}}</h1>
<div id="socials">
{% for social in socials %}
<a href="{{social.url}}">
<img src="{{social.image}}" alt="{{social.image-alt}}" />
</a>
{% endfor %}
</div>
</nav>
<div id="description">{{episode.description|safe}}</div>
</div>
</div>
<div id="transcript">
<h1>Transcript</h1>
<div id="transcript-body">{{episode.transcript-html|safe}}</div>
</div>
</body>
</html>
We should also sprinkle a little extra CSS into our public/css/main.css:
/* ... */
#aggregators-1 {
display: flex;
justify-content: space-between;
margin-top: 10px;
}
#aggregators-1 img {
width: 175px;
}
#apple a {
display: inline-block;
overflow: hidden;
}
.apple-button {
border-radius: 13px;
}
#aggregators-2 {
display: flex;
justify-content: space-between;
margin-top: 10px;
}
#podbean img {
height: 42px;
}
/* Some paragraphs in the description shouldn't be displayed on the episode page */
p.soundcljoud-hidden {
display: none;
}
#transcript {
background-color: #e4f1fe;
border: solid 1px;
padding-left: 1em;
padding-right: 1em;
margin-top: 1em;
}
#transcript-body > br {
display: none;
}
span.timestamp {
margin-right: 5px;
color: blue;
cursor: pointer;
&:hover {
text-decoration: underline;
}
}
@media screen and (min-width: 600px) {
/* ... */
#aggregators-1 img {
width: 155px;
}
#podbean img {
height: 37px;
}
#podbean img {
margin-top: 0px;
}
}
We'll need the following template vars:
Most of this we already have, but there are a couple new things. Let's take the easiest two first.
We'll add some alt text for our podcast cover image and a social preview image to podcast.edn. The alt text is just a description of the cover image, and it turns out that the preview image is one of the many things we hardcoded into our public/index.html way back when.
{ ; ...
:preview-image "/img/orgtech-se-preview.jpg"
;; ...
:podcast { ; ...
:image-alt "Organising Tech in Sweden superimposed on raised fists with a Swedish flag with a circuit board pattern in the background"
;; ...
}
;; ...
}
Let's turn next to socials. This is how we refer to it in the template:
{% for social in socials %}
<a href="{{social.url}}">
<img src="{{social.image}}" alt="{{social.image-alt}}" />
</a>
{% endfor %}
This means that it needs to be a list, and each list item should be a map containing three keys:
Let's add the following to our podcast.edn:
{ ; ...
:preview-image "/img/orgtech-se-preview.jpg"
;; ...
:socials [{:name "Twitter"
:url "https://x.com/orgtech_se"
:image "/img/twitter-color-svgrepo-com.svg"
:image-alt "Twitter logo"}
{:name "BlueSky"
:url "https://bsky.app/profile/orgtech-se.bsky.social"
:image "/img/bluesky-logo.svg"
:image-alt "BlueSky logo"}]
;; ...
}
Finally, we need to conjure up one last key for each episode:
Episodes already have a :transcript-file key, which refers to an OTR file. Let's have a quick look at one of those and see what is contained therein:
{
"text": "<p>[Theme music begins]</p><p><span class=\"timestamp\" data-timestamp=\"12.111684\">00:12</span><b>Josh</b>: ... </p>",
"media": "otis-ep01-klarna-part1.mp3",
"media-time": 1315.629803
}
What we have here is a JSON file with a thin veneer of metadata around a looooong HTML string. Let's deal with this back in podcast.rss/update-episode:
(defn update-episode [{:keys [base-dir src-dir] :as opts}
;; 👇👇👇👇👇👇👇
{:keys [audio-file transcript-file path] :as episode}]
;; 👆👆👆👆👆👆👆
(let [filename (format "%s/%s%s/%s" base-dir src-dir path audio-file)
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
transcript (format "%s/%s%s/%s" base-dir src-dir path transcript-file)]
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
(assoc episode
:audio-filesize (fs/size filename)
:duration (audio/mp3-duration filename)
:description (selmer/render (:description episode)
(assoc opts :episode episode))
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
:transcript-html (-> (slurp transcript)
(json/parse-string keyword)
:text))))
Having satisfied ourselves that we have all of the data we need for our template, let's get to rendering. In our tasks/render function, we're looping over episodes in order to copy the MP3 and OTR files into the right place. Sadly, we can't just pop selmer/render into that doseq and be done with it, because we need to ensure the audio file is in the right place before we can use update-episode. No problem, we'll just add a new bit after spitting our podcast feed:
(ns tasks
(:require ; ...
[selmer.parser :as selmer]))
(defn render [default-opts]
(let [...]
;; ...
(println (format "Writing RSS feed %s" feed-file))
(->> (rss/podcast-feed opts)
(spit feed-file))
(let [template (slurp "templates/episode-page.html")
opts (rss/update-episodes opts)]
(doseq [{:keys [path] :as episode} (:episodes opts)
:let [filename (format "%s/%s%s/%s"
base-dir out-dir path "index.html")]]
(println "Writing episode page" filename)
(->> (selmer/render template (assoc opts :episode episode))
(spit filename))))))
And now, the moment of truth!
: orgtech-se; bb render
Writing RSS feed public/feed.rss
Writing episode page ~/code/orgtech-se/public/episodes/ep00-trailer/index.html
Writing episode page ~/code/orgtech-se/public/episodes/ep01-klarna-part1/index.html
Writing episode page ~/code/orgtech-se/public/episodes/ep02-klarna-part2/index.html
And now if we visit (for example) http://localhost:1341/episodes/ep01-klarna-part1, we should see an amazing webpage:
Astute observers may have noticed one issue with the episode page: there's no way to play the episode. 🤦🏼
Fear not! In the next instalment, we'll look at playing a podcast with ClojureScript, perhaps even using our own friend Soundcljoud!
Last time on "Soundcljoud gets more cloudy", I found myself deeply saddened that the eternal truths I was seeking in the music of Garth Brooks remained elusive due to my attempts to seek forward in a track were rebuffed by my browser, instead abruptly returning me to the beginning of the track. 😳
Appropriately chastened, I popped the bonnet and had a look at what my user agent was doing on my behalf. When I loaded a track, I saw a request like this:
GET /Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3 HTTP/1.1
Range: bytes=0-
and a response like this:
HTTP/1.1 200 OK
Content-length: 5943424
Content-Type: audio/mpeg
Server: http-kit
with a bunch of bytes in the body. In fact, a bountiful buffet of beautiful bytes, five whole million of them! And another 943,424 thrown in for dessert.
Herein lies the rub. What the browser wants back is some indication that the server knows how to return a range of bytes, because the browser doesn't want to fetch the entire damned file every time the user starts playing a track. After all, the user might be trying to remember if the track entitled "The Old Stuff" contains the amazing homage to a "worn out tape of Chris LeDoux" (spoiler: it does not), and just listening to the first few seconds to determine this, then, disappointed, moving on to another track to sample the first few seconds of that one.
And how, you might ask, does the server indicate its range savviness? Well, according to our good friends over at the Mozilla Developer Network, by returning a response such as this:
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-length: 1048576
Content-Range: bytes 0-1048575/5943424
Content-Type: audio/mpeg
Let's refresh our memory a bit by firing up Soundcljoud:
cd ~/code/soundcljoud/player
bb dev
Now we can pop over to http://localhost:1341/, open up the soundcljoud.cljs in Emacs (or whatever inferior text editor you choose to inflict upon yourself), hit C-c l C (cider-connect-cljs) to start a REPL connected to localhost port 1339 (REPL type nbb), and finally evaluate load-ui! to get things going:
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
)
Opening the network tab, we see exactly what the browser asked for and exactly what the server responded:
First, the browser asks for some bytes, starting at the beginning of the file:
Range: bytes=0-
Since the end of the byte range isn't specified, the server is free to decide how many bytes to send back. Let's say we'll send back 1 MB (1048576 bytes). Our response should start by indicating that we're not returning the entire file, but rather just a part of it:
HTTP/1.1 206 Partial Content
Now we need to say which bytes we're returning, out of the total number of bytes in the file, as well as the length of the response, in bytes:
Content-Range: bytes 0-1048575/6062208
Content-length: 1048576
Note that the byte range is zero-indexed and inclusive on the end, meaning that the last byte we return is at index 1048575, whilst the content length is the number of bytes in the response body.
Finally, we need to let the client know what kind of range requests we support. We'll limit this to bytes:
Accept-Ranges: bytes
We must now flip Hegel on his head, as the saying goes, and move from lofty ideas to dirty, inconvenient material reality. In other words, we gotta implement range requests in our actual webserver.
Let's cast our minds back to what happens when we type
bb dev
in our terminal. According to our bb.edn:
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:git/sha "b38c1f16ad2c618adae2c3b102a5520c261a7dd3"}}
:tasks
{http-server
{:doc "Starts http server for serving static files"
:requires ([babashka.http-server :as http])
:task (do (http/serve {:port 1341 :dir "public"})
(println "Serving static assets at http://localhost:1341"))}
browser-nrepl
{:doc "Start browser nREPL"
:requires ([sci.nrepl.browser-server :as bp])
:task (bp/start! {})}
-dev
{:depends [http-server browser-nrepl]}
dev
{:task (do (run '-dev {:parallel true})
(deref (promise)))}}}
OK, so it looks like io.github.babashka/http-server is the thing serving up our content. Let's go ahead and clone that so we can start digging through the code:
cd ~/code
git clone git@github.com:babashka/http-server.git
Tracing through bb.edn, we see that the webserver is started by calling babashka.http-server/serve with a config map containing the port and directory:
{ ;; ...
:tasks
{http-server
{:requires ([babashka.http-server :as http])
:task (do (http/serve {:port 1341 :dir "public"})
(println "Serving static assets at http://localhost:1341"))}
;; ...
}}
Let's see what's going on thereabouts in the http-server source code. Opening src/babashka/http_server.clj:
(defn serve
"Serves static assets using web server.
Options:
* `:dir` - directory from which to serve assets
* `:port` - port
* `:headers` - map of headers {key value}"
[{:keys [port]
:or {port 8090}
:as opts}]
(let [dir (or (:dir opts) ".")
opts (assoc opts :dir dir :port port)
dir (fs/path dir)]
(assert (fs/directory? dir) (str "The given dir `" dir "` is not a directory."))
(binding [*out* *err*]
(println (str "Serving assets at http://localhost:" (:port opts))))
(server/run-server (file-router dir (opts :headers)) opts)))
we see a bunch of ceremony before server/run-server is called with a file-router (whatever that is) and some opts; basically the port and directory we passed in from bb.edn. But what, pray tell, is this mystical server namespace?
(ns babashka.http-server
(:require [babashka.fs :as fs]
[clojure.string :as str]
#_[clojure.tools.cli :refer [parse-opts]]
[hiccup2.core :as html]
[babashka.cli :as cli]
[org.httpkit.server :as server])
(:import [java.net URLDecoder URLEncoder]))
Aha! 'Tis none other than http-kit, a "minimalist and efficient Ring-compatible HTTP client+server for Clojure". Looking at the documentation for
run-server, we see that the file-router thingy must return a Ring
handler, which is nothing more than a function that takes a request map as its argument and returns a response map. This function will be called by http-kit upon every request.
start-server returns a function that we can call to stop the
server.
Using this knowledge, let's dig into the file-router handler function:
(defn file-router [dir headers]
(fn [{:keys [uri]}]
(let [f (fs/path dir (str/replace-first (URLDecoder/decode uri) #"^/" ""))
index-file (fs/path f "index.html")]
(update (cond
(and (fs/directory? f) (fs/readable? index-file))
(body index-file)
(fs/directory? f)
(index dir f)
(fs/readable? f)
(body f)
(and (nil? (fs/extension f)) (fs/readable? (with-ext f ".html")))
(body (with-ext f ".html") headers)
:else
{:status 404 :body (str "Not found `" f "` in " dir)})
:headers (fn [response-headers]
(merge headers response-headers))))))
OK, what's going on here? Well, we're returning a function (i.e. the Ring handler) that basically grabs the path part of the URI (which will be relative to the directory named by our :dir option; in other words, soundcljoud/player/public) and asks a series of questions in a cond form:
index.html that is readable by the webserver?index.html)?.html extension on the end, is a file that is readable by the webserver?Let's think for a second about which case we're interested in. Our browser is requesting /Garth Brooks/Fresh Horses/Garth Brooks - The Old Stuff.mp3, which is going to hit condition #3 in the list:
(fs/readable? f)
(body f)
Let's see what's going on with this body. And yes, I am aware that sounds like the title of a Pitbull) collabo with Nicki Minaj.
(defn- body
([path]
(body path {}))
([path headers]
{:headers (merge {"Content-Type" (ext-mime-type (fs/file-name path))} headers)
:body (fs/file path)}))
The only thing happening here is that the MIME
type of the file is being looked up using its extension and added as the Content-Type header, then the path itself is turned into a java.io.File with the babashka.fs/file function and added to the response map under the :body key. Presumably, http-kit will then take that java.io.File object and send the bytes back as the response body.
This looks very similar to what we will need to do, with the exception that instead of sending back all of the bytes in the file, we'll just want to send back those that were asked for.
Now that we know more or less where to start, let's fire up a REPL and start playing!
The first thing we need to do is Ctrl-c our bb dev process, since we won't be able to start a webserver on port 1341 with that one in the way.
Next, let's open up http-server/src/babashka/http_server.clj in Emacs and start a REPL with C-c M-j (cider-jack-in-clj), choosing babashka as the command to start the REPL. Now, we load the buffer with C-c C-k (cider-load-buffer), and sign in relief as we're back in the REPL again.
For our first order of business, let's try starting a server from the REPL to serve up the files in the soundcljoud/player/public directory on port 1341, just like we had before:
(comment
(def dir "../soundcljoud/player/public") ; C-c C-v f c e
;; => #'babashka.http-server/dir
(def server
(server/run-server (file-router dir {})
{:dir dir, :port 1341}))
;; => #'babashka.http-server/server
)
OK, so we maybe have a webserver running. Let's try fetching a file to be sure:
: jmglov@alhana; curl http://localhost:1341/site.webmanifest
{
"name": "Soundcljoud",
"short_name": "Soundcljoud",
"icons": [
{
"src": "icons/android-chrome-192x192.png",
"sizes": "192x192",
"type": "image/png"
},
{
"src": "icons/android-chrome-512x512.png",
"sizes": "512x512",
"type": "image/png"
}
],
"theme_color": "#ffffff",
"background_color": "#ffffff",
"display": "standalone"
}
Looks good!
The next step is making Soundcljoud use our local HTTP server instead of starting a new one. Back in soundcljoud/player, we open up bb.edn. Let's go ahead and change the deps first:
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:git/sha "b38c1f16ad2c618adae2c3b102a5520c261a7dd3"}}
;; ...
}
For the io.github.babashka/http-server dep, we can change the value from a Git reference to a local directory like this:
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:local/root "../../http-server"}}
;; ...
}
Next, we'll need to figure out how to start just the browser REPL. Let's take a look at the existing dev task that we've been using:
{ ;; ...
:tasks {http-server
{:doc "Starts http server for serving static files"
:requires ([babashka.http-server :as http])
:task (do
(http/serve {:port 1341 :dir "public"})
(println "Serving static assets at http://localhost:1341"))}
browser-nrepl
{:doc "Start browser nREPL"
:requires ([sci.nrepl.browser-server :as bp])
:task (bp/start! {})}
-dev
{:depends [http-server browser-nrepl]}
dev
{:task (do (run '-dev {:parallel true})
(deref (promise)))}}}
So dev just runs the -dev task in parallel, then derefs an empty promise to avoid exiting (calling deref on a promise will block the calling thread until the promise delivers, which an empty promise never will). The -dev task itself depends on http-server and browser-nrepl, but does nothing on its own.
Let's create a new task that follows this pattern but only starts the browser NREPL:
{ ;; ...
:tasks { ;; ...
dev
{:task (do (run '-dev {:parallel true})
(deref (promise)))}
browser
{:task (do (run 'browser-nrepl {:parallel true})
(deref (promise)))}}}
Now let's fire it up and see what happens:
: jmglov@alhana; bb browser
nREPL server started on port 1339...
Websocket server started on 1340...
Cool! If we now open http://localhost:1341/ in the browser, switch back to our open soundcljoud.cljs buffer and hit C-c l C, we see some very welcome log messages in our terminal:
nREPL server started on port 1339...
:msg "{:versions {\"scittle-nrepl\" {\"major\" \"0\", ..."
With baited breath, we evaluate the load-ui! form and... see the good ol' Eye of Garth! 🎉
This means Soundcljoud is using the http-server we're running from our REPL.
Switching back to the http-server/src/babashka/http_server.clj buffer, let's figure out how to do some REPL-driven development to implement handling range requests.
The first order of business might be giving ourselves a way to log the requests we're getting from the client. Let's create an atom at the top of the file for this very purpose:
(defonce state (atom {:requests [], :log []}))
I'm using defonce instead of plain 'ol def here because I tend to hit C-c C-k quite often whilst editing code, which not only causes the buffer to be re-evaluated, but also causes Emacs to ask me if I want to save my changes to the file, which is useful to keep code that's running in the system from drifting away from the code that's written in the source file. If I used def instead of defonce, my state atom would be reset every time I re-evaluate the buffer.
Now, we know that the function returned by file-router is a Ring handler, so let's jump there and see about how we can shove each request into our state atom:
(defn file-router [dir headers]
(fn [{:keys [uri]}]
;; ...
))
OK, at the moment, the handler function only cares about the :uri key in the request. Let's bind the entire request and then add it to the atom:
(defn file-router [dir headers]
(fn [{:keys [uri] :as req}]
(swap! state update :requests conj req)
;; ...
))
In order to test this, we need to restart the server since we made a change to the anonymous function returned by file-router. To do this, we stop the server by calling the function that server/run-server returned when we evaluated it, then evaluate the server/run-server expression again:
(comment
(server)
;; => nil
(def server
(server/run-server (file-router dir {})
{:dir dir, :port 1341}))
;; => #'babashka.http-server/server
)
Now, let's curl the manifest file again:
: jmglov@alhana; curl http://localhost:1341/site.webmanifest
{
"name": "Soundcljoud",
...
}
If we look at our state atom now, we can see that the request was successfully logged:
(comment
(:requests @state)
;; => [{:remote-addr "0:0:0:0:0:0:0:1",
;; :start-time 1004192760289113,
;; :headers
;; {"accept" "*/*", "host" "localhost:1341", "user-agent" "curl/8.4.0"},
;; :async-channel
;; #object[org.httpkit.server.AsyncChannel 0x44d028e7 "/[0:0:0:0:0:0:0:1]:1341<->/[0:0:0:0:0:0:0:1]:45890"],
;; :server-port 1341,
;; :content-length 0,
;; :websocket? false,
;; :content-type nil,
;; :character-encoding "utf8",
;; :uri "/site.webmanifest",
;; :server-name "localhost",
;; :query-string nil,
;; :body nil,
;; :scheme :http,
;; :request-method :get}]
)
OK, now that we've got some basic logging in place, let's get back to thinking about range requests. A good place to start is by looking at the requests we get from Soundcljoud when it loads a file, so let's pop back over to that browser window and click on a track.
Once we've done that, we can look at the request in our http-server REPL:
(comment
(->> @state
:requests
(map #(select-keys % [:start-time :headers :uri]))
last)
;; => {:start-time 1006716878994472,
;; :headers
;; {"range" "bytes=0-",
;; "sec-fetch-site" "same-origin",
;; "sec-ch-ua-mobile" "?0",
;; "host" "localhost:1341",
;; "user-agent"
;; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
;; "sec-ch-ua"
;; "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\"",
;; "sec-ch-ua-platform" "\"Linux\"",
;; "referer" "http://localhost:1341/",
;; "connection" "keep-alive",
;; "accept" "*/*",
;; "accept-language" "en-GB,en",
;; "sec-fetch-dest" "audio",
;; "accept-encoding" "identity;q=1, *;q=0",
;; "sec-fetch-mode" "no-cors",
;; "sec-gpc" "1"},
;; :uri
;; "/Garth%20Brooks/Fresh%20Horses/01%20-%20Garth%20Brooks%20-%20The%20Old%20Stuff.mp3"}
)
The interesting bit is this header right here, which is the thing that tells us that what we're dealing with here is a range request:
;; :headers
;; {"range" "bytes=0-",
Remember those 5 questions we asked back in file-router?
index.html that is readable by the webserver?index.html)?.html extension on the end, is a file that is readable by the webserver?Well, let's insert a new question in there as #3, and bump the rest down:
index.html that is readable by the webserver?index.html)?range header in our request?.html extension on the end, is a file that is readable by the webserver?Let's write that in Clojure instead of English:
(defn file-router [dir headers]
(fn [{:keys [uri] :as req}]
(swap! state update :requests conj req)
(let [f (fs/path dir (str/replace-first (URLDecoder/decode uri) #"^/" ""))
index-file (fs/path f "index.html")]
(update (cond
(and (fs/directory? f) (fs/readable? index-file))
(body index-file)
(fs/directory? f)
(index dir f)
;; 👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
(and (fs/readable? f) (contains? (:headers req) "range"))
(do
(swap! state update :log conj "Handling range request")
(body f))
;; 👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
(fs/readable? f)
(body f)
(and (nil? (fs/extension f)) (fs/readable? (with-ext f ".html")))
(body (with-ext f ".html") headers)
:else
{:status 404 :body (str "Not found `" f "` in " dir)})
:headers (fn [response-headers]
(merge headers response-headers))))))
Now we can try this out. Unfortunately, we need to restart the server again to have it pick up the new code. The issue is that file-handler is returning an anonymous function, so when we edit the code and re-evaluate the buffer, we're not updating the copy of the function that http-kit is using as the request handler, we're updating file-handler itself, so the next time it's called, it will return a new handler function. In the writing of this blog, I did try pulling the anonymous function out and giving it a name, which I expected to fix this issue, but that didn't work, for reasons that aren't clear to me (maybe because http-kit is running the server on a different thread?). Yell at me in the Clojurians Slack thread if you know how to do this. 😅
Anyway, let's stop the server as usual:
(comment
(server)
;; => nil
)
And now, since we know we're going to need to do this dance every time we make changes to the code, let's write a little convenience function:
(comment
(defn restart-server []
(when (:server @state)
((:server @state)))
(reset! state
{:requests []
:log []
:server
(server/run-server (file-router dir {})
{:dir dir, :port 1341})}))
;; => #'babashka.http-server/restart-server
)
Now we can just call restart-server whenever we need to, well, restart the server. Let's do so now:
(comment
(restart-server)
;; => {:requests [],
;; :server
;; #object[clojure.lang.AFunction$1 0x3ece031a "clojure.lang.AFunction$1@3ece031a"]}
)
Having done this, let's pop back over to Soundcljoud and click on another track, then inspect the log to make sure we see the message we expect:
(comment
(->> @state
:log
last)
;; => "Handling range request"
)
Looks good! Except for the fact that we're still returning the entire file in the request body, of course. Still, the key to REPL-driven development is rapidly iterating, so let's take that next iteration now!
Since we've captured the request, let's go ahead and pull the range header out so we can play with it:
(comment
(-> (:requests @state) last (get-in [:headers "range"]))
;; => "bytes=0-"
(def range-header *1)
;; => #'babashka.http-server/range-header
(let [[start end] (-> range-header
(str/replace #"^bytes=" "")
(str/split #"-"))]
[start end])
;; => ["0" nil]
)
The header parsing thing looks like a good thing to make into a function:
(defn- parse-range-header [range-header]
(map #(when % (Long/parseLong %))
(-> range-header
(str/replace #"^bytes=" "")
(str/split #"-"))))
(comment
(parse-range-header range-header)
;; => (0)
)
OK, now let's shift gears and figure out how to return a specific byte range from a file. After much searching, I found a magical way to seek to an arbitrary location in a file in Java (and hence Clojure, through the magic of interop). Every FileInputStream has an associated FileChannel, and this FileChannel has a helpful position() instance method, which sets the position in the FileChannel for subsequent read operations on the channel.
Now, how to perform a read operation on a FileInputStream? Looking at the documentation, this method looks quite useful:
public int read(byte[] b)
throws IOException
Reads up to b.length bytes of data from this input stream into an array of bytes. This method blocks until some input is available.
And how do we create a byte[] array of an arbitrary size in Clojure? Why, by using the aptly-named byte-array function, naturally! 😀
Let's try this out, using our helpful site.webmanifest file:
(comment
(let [arr (byte-array 32)]
(with-open [is (java.io.FileInputStream. manifest-file)]
(-> is .getChannel (.position 0))
(.read is arr))
(String. arr))
;; => "{\n \"name\": \"Soundcljoud\",\n "
(let [arr (byte-array 16)]
(with-open [is (java.io.FileInputStream. manifest-file)]
(-> is .getChannel (.position 14))
(.read is arr))
(String. arr))
;; => "\"Soundcljoud\",\n "
)
Now we're cooking with gas! 💥
Let's see if we can make a nice function out of this:
(defn- read-bytes [f [start end]]
(let [arr (byte-array (- end start))]
(with-open [is (java.io.FileInputStream. f)]
(-> is .getChannel (.position start))
(.read is arr))
arr))
(comment
(-> (read-bytes manifest-file [0 31])
(String.))
;; => "{\n \"name\": \"Soundcljoud\",\n "
(-> (read-bytes manifest-file [14 29])
(String.))
;; => "\"Soundcljoud\",\n "
)
There's one issue remaining, though. Remember the range header we got from Soundcljoud?
(comment
range-header
;; => "bytes=0-"
(parse-range-header range-header)
;; => (0)
)
We have a start, but not an end. 😱
Let's think about what we want to do in this case. The client is effectively saying, "give me as many bytes as you feel inclined to do, starting at this offset in the file". So how many bytes are we inclined to hand out willy-nilly? I dunno, how about 1 mega of them bytes?
(defn- read-bytes [f [start end]]
(let [end (or end (dec (+ start (* 1024 1024)))
arr (byte-array (- end start))]
(with-open [is (java.io.FileInputStream. f)]
(-> is .getChannel (.position start))
(.read is arr))
arr))
(comment
(-> (read-bytes manifest-file [0 31])
(String.)) ; ⚠ OMG wait don't evaluate this for the love of Pete!
)
Yeah, so you really don't want to evaluate that last read-bytes expression. "And why's that," you might ask? "Well," I might answer, "cast your mind back to the Java documentation":
public int read(byte[] b)
throws IOException
Reads up to b.length bytes of data from this input stream into an array of bytes. 👉 This method blocks until some input is available. 👈
"And how do you know this is a problem?" you might query. "Well," I might respond, "um, just 'cuz? I mean... I certainly didn't evaluate this and hang my REPL process and then have to forcibly kill Emacs or anything, because that would be a rookie mistake. Haha." And then I might laugh nervously and quickly change the subject. "So, how 'bout them Yankees?" I might mutter, maybe even looking at my shoes.
So blerg, what to do, what to do?
Well, we do know (or at least can know) how many bytes are in the file, so maybe we don't read past the end of the file? Amazing insights you get in this here blog, innit?
(defn- read-bytes [f [start end]]
(let [end (or end (dec (min (fs/size f)
(+ start (* 1024 1024)))))
arr (byte-array (- end start))]
(with-open [is (java.io.FileInputStream. f)]
(-> is .getChannel (.position start))
(.read is arr))
arr))
(comment
(let [f manifest-file
end nil
end (or end (dec (min (fs/size f) (* 1024 1024))))]
end)
;; => 457
;; Should be safe to do this... 🙈
(-> (read-bytes manifest-file [0 31])
(String.))
;; => "{\n \"name\": \"Soundcljoud\",\n "
(-> (read-bytes manifest-file [14 29])
(String.))
;; => "\"Soundcljoud\",\n "
;; Never in doubt... 😌
)
OK, we're making some progress here. In fact, it seems that we have most of the pieces we'll need to actually fulfil a range request, so let's see about sticking them together in a reasonable way.
Let's review what the response to a range request is supposed to look like:
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-Length: 1048576
Content-Range: bytes 0-1048575/5943424
Content-Type: audio/mpeg
At the moment, we're just using the body function to respond to range requests:
(defn file-router [dir headers]
;; ...
(cond
;; ...
(and (fs/readable? f) (contains? (:headers req) "range"))
(do
(swap! state update :log conj "Handling range request")
(body f))
;; ...
)
;; ...
)
And body just chucks the file into a map with some headers:
(defn- body
([path]
(body path {}))
([path headers]
{:headers (merge {"Content-Type" (ext-mime-type (fs/file-name path))} headers)
:body (fs/file path)}))
Let's follow suit. Since http-kit is so magical and wonderful, we'll go out on a limb and make the assumption that if we just stuff our byte array into the response body, http-kit will do The Right Thing™.
(defn- byte-range
([path request-headers]
(byte-range path request-headers {}))
([path request-headers response-headers]
(let [f (fs/file path)
[start end
:as requested-range] (parse-range-header (request-headers "range"))
arr (read-bytes f requested-range)
num-bytes-read (count arr)]
{:status 206
:headers (merge {"Content-Type" (ext-mime-type (fs/file-name path))
"Accept-Ranges" "bytes"
"Content-Length" num-bytes-read
"Content-Range" (format "bytes %d-%d/%d"
start
(+ start num-bytes-read)
(fs/size f))}
response-headers)
:body arr})))
(comment
(byte-range manifest-file {"range" "bytes=0-"})
;; => {:status 206,
;; :headers
;; {"Content-Type" nil,
;; "Accept-Ranges" "bytes",
;; "Content-Length" 458,
;; "Content-Range" "bytes 0-457/458"},
;; :body
;; [123, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 83, 111, 117,
;; 110, 100, 99, 108, 106, 111, 117, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115,
;; 104, 111, 114, 116, 95, 110, 97, 109, 101, 34, 58, 32, 34, 83, 111, 117, 110,
;; 100, 99, 108, 106, 111, 117, 100, 34, 44, 10, 32, 32, 32, 32, 34, 105, 99,
;; 111, 110, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 114, 99, 34, 58, 32,
;; 34, 105, 99, 111, 110, 115, 47, 97, 110, 100, 114, 111, 105, 100, 45, 99,
;; 104, 114, 111, 109, 101, 45, 49, 57, 50, 120, 49, 57, 50, 46, 112, 110, 103,
;; 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 105,
;; 122, 101, 115, 34, 58, 32, 34, 49, 57, 50, 120, 49, 57, 50, 34, 44, 10, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 121, 112, 101, 34, 58,
;; 32, 34, 105, 109, 97, 103, 101, 47, 112, 110, 103, 34, 10, 32, 32, 32, 32,
;; 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 114, 99, 34, 58, 32, 34,
;; 105, 99, 111, 110, 115, 47, 97, 110, 100, 114, 111, 105, 100, 45, 99, 104,
;; 114, 111, 109, 101, 45, 53, 49, 50, 120, 53, 49, 50, 46, 112, 110, 103, 34,
;; 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 105, 122,
;; 101, 115, 34, 58, 32, 34, 53, 49, 50, 120, 53, 49, 50, 34, 44, 10, 32, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 121, 112, 101, 34, 58, 32,
;; 34, 105, 109, 97, 103, 101, 47, 112, 110, 103, 34, 10, 32, 32, 32, 32, 32,
;; 32, 32, 32, 125, 10, 32, 32, 32, 32, 93, 44, 10, 32, 32, 32, 32, 34, 116,
;; 104, 101, 109, 101, 95, 99, 111, 108, 111, 114, 34, 58, 32, 34, 35, 102, 102,
;; 102, 102, 102, 102, 34, 44, 10, 32, 32, 32, 32, 34, 98, 97, 99, 107, 103,
;; 114, 111, 117, 110, 100, 95, 99, 111, 108, 111, 114, 34, 58, 32, 34, 35, 102,
;; 102, 102, 102, 102, 102, 34, 44, 10, 32, 32, 32, 32, 34, 100, 105, 115, 112,
;; 108, 97, 121, 34, 58, 32, 34, 115, 116, 97, 110, 100, 97, 108, 111, 110, 101,
;; 34, 10, 125, 10]}
)
That looks fairly reasonable. Let's now complete the plumbing so when we turn on the tap of range requests, we get a delicious stream of ice cold, alpine spring fed responses flowing back:
(defn file-router [dir headers]
;; ...
(cond
;; ...
(and (fs/readable? f) (contains? (:headers req) "range"))
(do
(swap! state update :log conj "Handling range request")
(byte-range f (:headers req)))
;; ...
)
;; ...
)
(comment
((file-router dir {}) {:headers {"range" "bytes=0-"}
:uri "/site.webmanifest"})
;; => {:status 206,
;; :headers
;; {"Content-Type" nil,
;; "Accept-Ranges" "bytes",
;; "Content-Length" 458,
;; "Content-Range" "bytes 0-457/458"},
;; :body
;; [123, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 83, 111, 117,
;; 110, 100, 99, 108, 106, 111, 117, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115,
;; 104, 111, 114, 116, 95, 110, 97, 109, 101, 34, 58, 32, 34, 83, 111, 117, 110,
;; 100, 99, 108, 106, 111, 117, 100, 34, 44, 10, 32, 32, 32, 32, 34, 105, 99,
;; 111, 110, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 114, 99, 34, 58, 32,
;; 34, 105, 99, 111, 110, 115, 47, 97, 110, 100, 114, 111, 105, 100, 45, 99,
;; 104, 114, 111, 109, 101, 45, 49, 57, 50, 120, 49, 57, 50, 46, 112, 110, 103,
;; 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 105,
;; 122, 101, 115, 34, 58, 32, 34, 49, 57, 50, 120, 49, 57, 50, 34, 44, 10, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 121, 112, 101, 34, 58,
;; 32, 34, 105, 109, 97, 103, 101, 47, 112, 110, 103, 34, 10, 32, 32, 32, 32,
;; 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 114, 99, 34, 58, 32, 34,
;; 105, 99, 111, 110, 115, 47, 97, 110, 100, 114, 111, 105, 100, 45, 99, 104,
;; 114, 111, 109, 101, 45, 53, 49, 50, 120, 53, 49, 50, 46, 112, 110, 103, 34,
;; 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 115, 105, 122,
;; 101, 115, 34, 58, 32, 34, 53, 49, 50, 120, 53, 49, 50, 34, 44, 10, 32, 32,
;; 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 121, 112, 101, 34, 58, 32,
;; 34, 105, 109, 97, 103, 101, 47, 112, 110, 103, 34, 10, 32, 32, 32, 32, 32,
;; 32, 32, 32, 125, 10, 32, 32, 32, 32, 93, 44, 10, 32, 32, 32, 32, 34, 116,
;; 104, 101, 109, 101, 95, 99, 111, 108, 111, 114, 34, 58, 32, 34, 35, 102, 102,
;; 102, 102, 102, 102, 34, 44, 10, 32, 32, 32, 32, 34, 98, 97, 99, 107, 103,
;; 114, 111, 117, 110, 100, 95, 99, 111, 108, 111, 114, 34, 58, 32, 34, 35, 102,
;; 102, 102, 102, 102, 102, 34, 44, 10, 32, 32, 32, 32, 34, 100, 105, 115, 112,
;; 108, 97, 121, 34, 58, 32, 34, 115, 116, 97, 110, 100, 97, 108, 111, 110, 101,
;; 34, 10, 125, 10]}
)
Looks great... except for the Content-Type: nil bit, since our server has no clue what a .webmanifest extension portends, but who cares about such trivial details, since we're not gonna be getting range requests for non-media files anyway. Plus, a standard request for that file does the same thing:
(comment
((file-router dir {}) {:headers {}
:uri "/site.webmanifest"})
;; => {:headers {"Content-Type" nil},
;; :body
;; #object[java.io.File 0x659969c9 "../soundcljoud/player/public/site.webmanifest"]}
)
🤷
Before we break out the 🍾 though, let's try this in the wild. And before we try this in the wild, it probably behoves us—at least, I feel rather behoved, and it's my blog, so I'm going to follow this deep sense of behoval where it leads—to log responses as well as requests, so let's make one last minor change to good 'ol file-router:
(defn file-router [dir headers]
(fn [{:keys [uri] :as req}]
;; 👉 Move the state swappage from here...
(let [f (fs/path dir (str/replace-first (URLDecoder/decode uri) #"^/" ""))
index-file (fs/path f "index.html")
res
(update (cond
(and (fs/directory? f) (fs/readable? index-file))
(body index-file)
(fs/directory? f)
(index dir f)
(and (fs/readable? f) (contains? (:headers req) "range"))
(do
(swap! state update :log conj "Handling range request")
(byte-range f (:headers req)))
(fs/readable? f)
(body f)
(and (nil? (fs/extension f)) (fs/readable? (with-ext f ".html")))
(body (with-ext f ".html") headers)
:else
{:status 404 :body (str "Not found `" f "` in " dir)})
:headers (fn [response-headers]
(merge headers response-headers)))]
;; ...to here 👇
(swap! state
update :requests
conj {:request req, :response (dissoc res :body)})
res)))
Casting our minds back to the last post in this potentially infinite sequence
of posts, we recall that Soundcljoud was unable to seek in the audio file. Let's repeat this experience by jumping over to soundcljoud/player/public/soundcljoud.cljs:
(comment
(-> (get-el "audio")
(.-seekable)
(.-length))
;; => 1
(let [s (-> (get-el "audio")
(.-seekable))]
[(.start s 0) (.end s 0)])
;; => [0 0]
)
This is what we expected, since we haven't restarted the server to apply our changes. Let's do that now (back in our http-server REPL):
(comment
(restart-server)
;; => {:requests [],
;; :log [],
;; :server
;; #object[clojure.lang.AFunction$1 0x2d75d828 "clojure.lang.AFunction$1@2d75d828"]}
)
Now we can click on another track in Soundcljoud and see what happens. 😬
OK, nothing blew up. Let's look at the request in the http-server logs:
(comment
(->> (:requests @state)
(filter #(str/ends-with? (get-in % [:request :uri]) ".mp3"))
(map (fn [{:keys [request response]}]
{:request {:uri (:uri request)
:headers (select-keys (:headers request)
["range"])}
:response response})))
;; => ({:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=0-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 1048576,
;; "Content-Range" "bytes 0-1048575/3426432"}}})
)
So far, so good. But if we now seek, can we find? Let's ask in our Soundcljoud REPL:
(comment
(let [seekable (-> (get-el "audio") (.-seekable))]
(->> (.-length seekable)
range
(map (fn [i]
[(.start seekable i) (.end seekable i)]))))
;; => ([0 142.654694])
)
And if we actually click play? OMG we hear the sweet sweet sounds of a steel guitar! And if we seek forward in the track? Garth sings! Let's just check in with http-server one last time to see what it thinks:
(comment
(->> (:requests @state)
(filter #(str/ends-with? (get-in % [:request :uri]) ".mp3"))
(map (fn [{:keys [request response]}]
{:request {:uri (:uri request)
:headers (select-keys (:headers request)
["range"])}
:response response})))
;; => ({:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=0-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 1048575,
;; "Content-Range" "bytes 0-1048575/3426432"}}}
;; {:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=0-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 1048575,
;; "Content-Range" "bytes 0-1048575/3426432"}}}
;; {:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=1048575-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 1048575,
;; "Content-Range" "bytes 1048575-2097150/3426432"}}}
;; {:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=2097150-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 1048575,
;; "Content-Range" "bytes 2097150-3145725/3426432"}}}
;; {:request
;; {:uri
;; "/Garth%20Brooks/Fresh%20Horses/Garth%20Brooks%20-%20It%27s%20Midnight%20Cinderella.mp3",
;; :headers {"range" "bytes=3145725-"}},
;; :response
;; {:status 206,
;; :headers
;; {"Content-Type" "audio/mpeg",
;; "Accept-Ranges" "bytes",
;; "Content-Length" 280706,
;; "Content-Range" "bytes 3145725-3426431/3426432"}}})
)
Now that, my friends, smells like the sweet sweet smell of...
Ah... it's been a while since I've been able to use that lovely image. 🌅
Well... the fact that I'm asking this rhetorical question points to the answer likely being "no". 😅
And in fact it isn't the end, because I feel (perhaps arrogantly so) that this range support could be useful to others using babashka.http-server, so I should probably open up a pull request for the borkiest of
dudes to review. I'll quickly fork http-server on Github, then update my remotes in magit to make origin point to git@github.com:jmglov/http-server.git and upstream point to git@github.com:babashka/http-server.git, stash my changes, create a range-requests branch, then pop the stash.
I doubt Señor Borkdude will be terribly impressed by my Rich comment and state atom, so I'd better go ahead and remove that nonsense before committing. I'll open a feature request on the Github project as well, since I know this is how Borkdude prefers to work.
With this, I have a fairly minimal commit that I'm ready to subject to the slings and arrows of outrageous fortune that are part of any Borkdude code review:
range-requests a87a841e02d362ae8dc346153b166d28882c3c6e
Author: Josh Glover <jmglov@jmglov.net>
AuthorDate: Tue Aug 13 14:18:47 2024 +0200
Commit: Josh Glover <jmglov@jmglov.net>
CommitDate: Tue Aug 13 17:08:31 2024 +0200
Support range requests
2 files changed, 42 insertions(+)
CHANGELOG.md | 4 ++++
src/babashka/http_server.clj | 38 ++++++++++++++++++++++++++++++++++++++
modified CHANGELOG.md
@@ -2,6 +2,10 @@
[Http-server](https://github.com/babashka/http-server): Serve static assets with [babashka](https://babashka.org/)
+## Unreleased
+
+- [#16](https://github.com/babashka/http-server/issues/16): support range requests
+
## 0.1.13
- [#13](https://github.com/babashka/http-server/issues/13): add an ending slash to the dir link, and don't encode the slashes ([@KDr2](https://github.com/KDr2))
modified src/babashka/http_server.clj
@@ -165,6 +165,41 @@
{:headers (merge {"Content-Type" (ext-mime-type (fs/file-name path))} headers)
:body (fs/file path)}))
+(defn- parse-range-header [range-header]
+ (map #(when % (Long/parseLong %))
+ (-> range-header
+ (str/replace #"^bytes=" "")
+ (str/split #"-"))))
+
+(defn- read-bytes [f [start end]]
+ (let [end (or end (dec (min (fs/size f)
+ (+ start (* 1024 1024)))))
+ arr (byte-array (- end start))]
+ (with-open [is (java.io.FileInputStream. f)]
+ (-> is .getChannel (.position start))
+ (.read is arr))
+ arr))
+
+(defn- byte-range
+ ([path request-headers]
+ (byte-range path request-headers {}))
+ ([path request-headers response-headers]
+ (let [f (fs/file path)
+ [start end
+ :as requested-range] (parse-range-header (request-headers "range"))
+ arr (read-bytes f requested-range)
+ num-bytes-read (count arr)]
+ {:status 206
+ :headers (merge {"Content-Type" (ext-mime-type (fs/file-name path))
+ "Accept-Ranges" "bytes"
+ "Content-Length" num-bytes-read
+ "Content-Range" (format "bytes %d-%d/%d"
+ start
+ (+ start num-bytes-read)
+ (fs/size f))}
+ response-headers)
+ :body arr})))
+
(defn- with-ext [path ext]
(fs/path (fs/parent path) (str (fs/file-name path) ext)))
@@ -179,6 +214,9 @@
(fs/directory? f)
(index dir f)
+ (and (fs/readable? f) (contains? (:headers req) "range"))
+ (byte-range f (:headers req))
+
(fs/readable? f)
(body f)
Wish me well, folks! If I'm not heard from again, you'll know that my pull request was found to be sub-par and I was sent to Java Jail to work on an enterprise workflow management system. 😭
Soundcljoud has clearly now implemented the critical functionality of Soundcloud, so I could call it a day, but I'm loathe to do that when I could instead extend it to be the best podcast player that ever was! Maybe I'll rebrand it OverClj... or better yet, CljerCast! VCs, get your wallets ready and stay posted for the next instalment of the exciting Soundcljoud series, right here on jmglov.net!
Previously on Soundcljoud:
What's Going on with that Body cover art:
Last time on "Soundcljoud, or a young man's Soundcloud clonejure", I promised to clone Soundcloud, but then got bogged down in telling the story of my life and never got around to the actual cloning part. 😬
To be fair to myself, I did do a bunch of stuff to prepare for cloning, so now we can get to it with no further ado! (Skipping the ado bit is very out of character for me, I know. I'll just claim this parenthetical as my ado and thus fulfil your expectations of me as the most verbose writer in the Clojure community. You're welcome!)
If you've followed along with any of my other cloning adventures, you'll know where I'm going with this: straight to Scittle Town!
I'll start by creating a player directory and dropping a bb.edn into it:
{:deps {io.github.babashka/sci.nrepl
{:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
io.github.babashka/http-server
{:git/sha "b38c1f16ad2c618adae2c3b102a5520c261a7dd3"}}
:tasks {http-server {:doc "Starts http server for serving static files"
:requires ([babashka.http-server :as http])
:task (do (http/serve {:port 1341 :dir "public"})
(println "Serving static assets at http://localhost:1341"))}
browser-nrepl {:doc "Start browser nREPL"
:requires ([sci.nrepl.browser-server :as bp])
:task (bp/start! {})}
-dev {:depends [http-server browser-nrepl]}
dev {:task (do (run '-dev {:parallel true})
(deref (promise)))}}}
In short, what's happening here is I'm setting up a Babashka project with a dev task that starts a webserver on port 1341 serving up the files in the public/ directory, starts an nREPL server on port 1339 that we can connect to with Emacs (or any inferior text editor of your choosing), and a websocket server on port 1340 that is connected to the nREPL server on one end and waiting for a ClojureScript app to connect to the other end.
Speaking of the public/ directory, I need a public/index.html file to serve up:
<!doctype html>
<html class="no-js" lang="">
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<title>Soundcljoud</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="style.css">
<script src="https://cdn.jsdelivr.net/npm/scittle@0.6.15/dist/scittle.js" type="application/javascript"></script>
<script src="https://cdn.jsdelivr.net/npm/scittle@0.6.15/dist/scittle.promesa.js" type="application/javascript"></script>
<script>var SCITTLE_NREPL_WEBSOCKET_PORT = 1340;</script>
<script src="https://cdn.jsdelivr.net/npm/scittle@0.6.15/dist/scittle.nrepl.js"
type="application/javascript"></script>
<script type="application/x-scittle" src="soundcljoud.cljs"></script>
</head>
<body>
<h1>Soundcljoud</h1>
<div id="wrapper" style="display: none;">
<div id="player">
<div class="cover-image">
<img src="" alt="" />
</div>
<div id="controls">
<audio controls src=""></audio>
<div id="tracks" style=""></div>
</div>
</div>
</div>
</body>
</html>
The index.html file loads three JavaScript scripts:
Once this JavaScript is in place, index.html loads the soundcljoud.cljs ClojureScript file, which we'll come to in just a second.
For a (much) more detailed explanation, refer to the Popping in a Scittle section of my cljcastr, or a young man's Zencastr clonejure blog post.
The body of index.html is all about setting up a basic HTML page with this structure:
+----------------------+
| Soundcljoud |
+-------+--------------+ <---
| Album | Audio player | }
| cover +--------------+ } <div id="wrapper">
| image | Tracks list | }
+-------+--------------+ <---
Note that everything inside the wrapper div is hidden from the start:
<div id="wrapper" style="display: none;">
We don't know anything about the album we want to display yet, and there's no point in showing a bunch of empty divs until we do.
Let's drop a public/style.css in as well:
body {
font:
1.2em Helvetica,
Arial,
sans-serif;
margin: 20px;
padding: 0;
}
img {
max-width: 100%;
}
#wrapper {
max-width: 960px;
margin: 2em auto;
}
#controls {
display: flex;
flex-direction: column;
gap: 5px;
}
#tracks {
display: flex;
flex-direction: column;
gap: 3px;
}
@media screen and (min-width: 900px) {
#wrapper {
display: flex;
}
#player {
display: flex;
gap: 3%;
}
#cover-image {
margin-right: 5%;
max-width: 60%;
}
#controls {
width: 25%;
}
}
All of this stuff is about using screen real estate effectively. The first chunk of CSS applies universally, but the bit inside this:
@media screen and (min-width: 900px) {
/* ... */
}
only applies to windows at least 900px wide. So our page defaults to a layout that's appropriate for phones (or really narrow browser windows), but then adjusts to move more content "above the fold" so you can probably see the entire UI without scrolling if you're viewing the page on a standard computer.
Now that we have all of the HTML and CSS plumbing in place, let's add a public/soundcljoud.cljs file to get started with some ClojureScripting:
(ns soundcljoud
(:require [promesa.core :as p]))
Before we can start REPL-driving, we need to put the key in the ignition and give it a right twist! In other words, we open up a terminal in the top-level player/ directory and invoke Babashka:
: jmglov@alhana; bb dev
Serving static assets at http://localhost:1341
nREPL server started on port 1339...
Websocket server started on 1340...
If we now connect to http://localhost:1341/, we'll be rewarded with a simple webpage:
This by itself is of course monumentally boring, so let's inject some excitement into our lives by jumping into soundcljoud.cljs and pressing C-c l C (cider-connect-cljs), selecting localhost, port 1339, and nbb for the REPL type (assuming you're in Emacs; if you're using some other editor, perform the incantations necessary to connect your ClojureScript REPL to localhost:1339).
If everything went according to plan, you should see something like this in your terminal window:
:msg "{:versions
{\"scittle-nrepl\"
{\"major\" \"0\", \"minor\" \"0\", \"incremental\" \"1\"}},
:ops
{\"complete\" {}, \"info\" {}, \"lookup\" {}, \"eval\" {},
\"load-file\" {}, \"describe\" {}, \"close\" {}, \"clone\" {},
\"eldoc\" {}},
:status [\"done\"],
:id \"3\",
:session \"3264dc1e-1b46-48a6-b11a-f606fea032b7\",
:ns \"soundcljoud\"}"
:msg "{:value \"nil\",
:id \"5\",
:session \"3264dc1e-1b46-48a6-b11a-f606fea032b7\",
:ns \"soundcljoud\"}"
:msg "{:status [\"done\"],
:id \"5\",
:session \"3264dc1e-1b46-48a6-b11a-f606fea032b7\",
:ns \"soundcljoud\"}"
And something like this in your editor's REPL window:
;; Connected to nREPL server - nrepl://localhost:1339
;; CIDER 1.12.0 (Split)
;;
;; ClojureScript REPL type: nbb
;;
nil>
Let's prove that it works by evaluating the buffer with C-c C-k (cider-load-buffer), adding a Rich
comment, putting some ClojureScript in there that grabs our wrapper div, positioning our cursor at the end of the form, and evaluating that sucker with C-c C-v f c e (cider-pprint-eval-last-sexp-to-comment):
(ns soundcljoud
(:require [promesa.core :as p]))
(comment
(js/document.querySelector "#wrapper")
;; => #object[HTMLDivElement [object HTMLDivElement]]
)
We've proven that we can evaluate ClojureScript code in the running browser process from our REPL buffer, which is nifty for sure, but our page still bores us, and the result of evaluating that code is pretty useless:
#object[HTMLDivElement [object HTMLDivElement]]
Let's actually do something with the div we've pulled down, and whilst we're at it, provide a useful way of logging stuff:
(ns soundcljoud
(:require [promesa.core :as p]))
(defn log
([msg]
(log msg nil))
([msg obj]
(if obj
(js/console.log msg obj)
(js/console.log msg))
obj))
(comment
(let [div (js/document.querySelector "#wrapper")]
(set! (.-style div) "display: flex")
(log "All is revealed!" div))
;; => #object[HTMLDivElement [object HTMLDivElement]]
)
Fantastic! By using js/document.log (by the way, that js/ prefix is the way you instruct ClojureScript to do some JavaScript interop; it's basically saying "look for the next symbol in the top-level scope in JavaScript land"), we now get the fancy inspection tools in the browser's JavaScript console so we can expand parts of the object and drill down to see stuff we're interested in.
Now that we've established a baseline, we can get stuck in and do some real work. 💪🏻
Do you remember the MP3 files and RSS
feed we prepared in the previous blog post? Let's plop those down in our public/ directory so we can access them from the webapp we're slowly constructing:
: jmglov@alhana; mkdir -p 'public/Garth Brooks/Fresh Horses'
: jmglov@alhana; cp /tmp/soundcljoud.12524185230907219576/*.{rss,mp3} !$
: jmglov@alhana; ls -1 !$
album.rss
'Garth Brooks - Cowboys and Angels.mp3'
'Garth Brooks - Ireland.mp3'
"Garth Brooks - It's Midnight Cinderella.mp3"
"Garth Brooks - Rollin'.mp3"
"Garth Brooks - She's Every Woman.mp3"
"Garth Brooks - That Ol' Wind.mp3"
'Garth Brooks - The Beaches of Cheyenne.mp3'
'Garth Brooks - The Change.mp3'
'Garth Brooks - The Fever.mp3'
'Garth Brooks - The Old Stuff.mp3'
Now that our files are in place, let's see about loading the RSS feed from ClojureScript:
(comment
(def base-path "/Garth+Brooks/Fresh+Horses")
;; => #'soundcljoud/base-path
(p/->> (js/fetch (js/Request. (str base-path "/album.rss")))
(.text)
(log "Fetched XML:"))
;; => #<Promise[~]>
)
In our console, we can see what we fetched:
Fetched XML: <?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0"
xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<atom:link
href="http://localhost:1341/Garth+Brooks/Fresh+Horses/album.rss"
rel="self"
type="application/rss+xml"/>
<title>Garth Brooks - Fresh Horses</title>
<link>https://api.discogs.com/masters/212114</link>
<pubDate>Sun, 01 Jan 1995 00:00:00 +0000</pubDate>
<itunes:subtitle>Album: Garth Brooks - Fresh Horses</itunes:subtitle>
<itunes:author>Garth Brooks</itunes:author>
<itunes:image href="https://i.discogs.com/0eLXmM1tK1grkH8cstgDT6eV2TlL0NvgWPZBoyScJ_8/rs:fit/g:sm/q:90/h:600/w:600/czM6Ly9kaXNjb2dz/LWRhdGFiYXNlLWlt/YWdlcy9SLTY4NDcx/Ny0xNzE3NDU5MDIy/LTMxNjguanBlZw.jpeg"/>
<item>
<itunes:title>The Old Stuff</itunes:title>
<title>The Old Stuff</title>
<itunes:author>Garth Brooks</itunes:author>
<enclosure
url="http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3"
length="5943424" type="audio/mpeg" />
<pubDate>Sun, 01 Jan 1995 00:00:00 +0000</pubDate>
<itunes:duration>252</itunes:duration>
<itunes:episode>1</itunes:episode>
<itunes:episodeType>full</itunes:episodeType>
<itunes:explicit>false</itunes:explicit>
</item>
...
<item>
<itunes:title>Ireland</itunes:title>
<title>Ireland</title>
<itunes:author>Garth Brooks</itunes:author>
<enclosure
url="http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+Ireland.mp3"
length="6969472" type="audio/mpeg" />
<pubDate>Sun, 01 Jan 1995 00:00:00 +0000</pubDate>
<itunes:duration>301</itunes:duration>
<itunes:episode>10</itunes:episode>
<itunes:episodeType>full</itunes:episodeType>
<itunes:explicit>false</itunes:explicit>
</item>
</channel>
</rss>
That looks quite familiar! That also looks like a bunch of text, which is not the nicest thing to extract data from. Luckily, that's a bunch of structured text, and more luckily, it's XML (XML is great, and don't let anyone tell you otherwise! And don't get me started on how we've reinvented XML but poorly with JSON Schema and all of this other nonsense we've built up around JSON because we realised that things like data validation are important when exchanging data between machines. 🤦🏼♂️), and most luckily of all, browsers know how to parse XML (which makes sense, as modern HTML is in fact XML):
(defn parse-xml [xml-str]
(.parseFromString (js/window.DOMParser.) xml-str "text/xml"))
(comment
(p/->> (js/fetch (js/Request. (str base-path "/album.rss")))
(.text)
parse-xml
(log "Fetched XML:"))
;; => #<Promise[~]>
)
Let's do the right thing and make a function out of this:
(defn fetch-xml [path]
(p/->> (js/fetch (js/Request. path))
(.text)
parse-xml
(log "Fetched XML:")))
Now that we know how to fetch and parse XML, let's see how to extract useful information from it. Looking at the log output, we can see that the parsed XML is of type #document, just like our good friend js/document (the current webpage that the browser is displaying). That's right, we have a Document Object Model, which means we can use all the tasty DOM functions we're used to, such as document.querySelector() to grab a node using an XPATH query.
Let's start with the album title:
(comment
(p/let [title (p/-> (fetch-xml (str base-path "/album.rss"))
(.querySelector "title")
(.-innerHTML))]
(set! (.-innerHTML (js/document.querySelector "h1")) title))
;; => #<Promise[~]>
)
Cool! We now see "Garth Brooks - Fresh Horses" as our page heading! Let's see about grabbing the album art next:
(comment
(p/let [xml (fetch-xml (str base-path "/album.rss"))
title (p/-> xml
(.querySelector "title")
(.-innerHTML))
image (p/-> xml
(.querySelector "image")
(.getAttribute "href"))]
(set! (.-innerHTML (js/document.querySelector "h1")) title)
(set! (.-src (js/document.querySelector ".cover-image > img")) image)
(set! (.-style (js/document.querySelector "#wrapper")) "display: flex;"))
;; => #<Promise[~]>
)
Before we go any further, let's create some functions from this big blob of code. At the moment, we're complecting two things:
Let's do the functional programming thing and create a purely functional core and a mutable shell. Instead of extracting and updating, we'll create a function that transforms the XML DOM representation of an album into a ClojureScript representation:
(defn xml-get [el k]
(-> el
(.querySelector k)
(.-innerHTML)))
(defn xml-get-attr [el k attr]
(-> el
(.querySelector k)
(.getAttribute attr)))
(defn ->album [xml]
{:title (xml-get xml "title")
:image (xml-get-attr xml "image" "href")})
(defn load-album [path]
(p/-> (fetch-xml path) ->album))
(comment
(p/let [{:keys [title image] :as album} (load-album (str base-path "/album.rss"))]
(set! (.-innerHTML (js/document.querySelector "h1")) title)
(set! (.-src (js/document.querySelector ".cover-image > img")) image)
(set! (.-style (js/document.querySelector "#wrapper")) "display: flex;"))
;; => #<Promise[~]>
)
Now that we have a nice ClojureScript data structure to represent our album, let's tackle the DOM mutations we need to do to display the album:
(defn get-el [selector]
(if (instance? js/HTMLElement selector)
selector ; already an element; just return it
(js/document.querySelector selector)))
(defn set-styles! [el styles]
(set! (.-style el) styles))
(defn display-album! [{:keys [title image] :as album}]
(let [header (get-el "h1")
cover (get-el ".cover-image > img")
wrapper (get-el "#wrapper")]
(set! (.-innerHTML header) title)
(set! (.-src cover) image)
(set-styles! wrapper "display: flex;")
album))
(comment
(p/-> (load-album (str base-path "/album.rss")) display-album!)
;; => #<Promise[~]>
)
Displaying the album title and cover art is all well and good, but in order to complete our Soundcloud clone, we need some way of actually listening to the music on the album. If you recall, our RSS feed contains a series of <item> tags representing the tracks:
<item>
<itunes:title>The Old Stuff</itunes:title>
<title>The Old Stuff</title>
<itunes:author>Garth Brooks</itunes:author>
<enclosure
url="http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3"
length="5943424" type="audio/mpeg" />
<pubDate>Sun, 01 Jan 1995 00:00:00 +0000</pubDate>
<itunes:duration>252</itunes:duration>
<itunes:episode>1</itunes:episode>
<itunes:episodeType>full</itunes:episodeType>
<itunes:explicit>false</itunes:explicit>
</item>
...
<item>
<itunes:title>Ireland</itunes:title>
<title>Ireland</title>
<itunes:author>Garth Brooks</itunes:author>
<enclosure
url="http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+Ireland.mp3"
length="6969472" type="audio/mpeg" />
<pubDate>Sun, 01 Jan 1995 00:00:00 +0000</pubDate>
<itunes:duration>301</itunes:duration>
<itunes:episode>10</itunes:episode>
<itunes:episodeType>full</itunes:episodeType>
<itunes:explicit>false</itunes:explicit>
</item>
What we need from each item in order to display and play the track is:
Let's write an aspirational function that assumes it will be called with a DOM element representing an <item> and transforms it into a ClojureScript map, just as we did for the item itself:
(defn ->track [item-el]
{:artist (xml-get item-el "author")
:title (xml-get item-el "title")
:number (-> (xml-get item-el "episode") js/parseInt)
:src (xml-get-attr item-el "enclosure" "url")})
For the track number, we need to convert it to an integer, since the text contents of an XML elements are, well, text, and we'll want to sort our tracks numerically.
Now that we have a function to convert an <item> into a track, let's plug that into our ->album function to add a list of tracks to the album:
(defn ->album [xml]
{:title (xml-get xml "title")
:image (xml-get-attr xml "image" "href")
:tracks (->> (.querySelectorAll xml "item")
(map ->track)
(sort-by :number))})
OK, we have data representing a list of tracks, so we need to consider how we want to display it. If we cast our mind back to our HTML, we have a div where the tracks should go:
<body>
...
<div id="wrapper" style="display: none;">
<div id="player">
...
<div id="controls">
...
<div id="tracks" style=""></div>
</div>
</div>
</div>
</body>
What we can do is create a <span> for each track, something like this:
<span>1. The Old Stuff</span>
Let's go ahead and write that function:
(defn track->span [{:keys [number artist title] :as track}]
(let [span (js/document.createElement "span")]
(set! (.-innerHTML span) (str number ". " title))
span))
(comment
(p/->> (load-album (str base-path "/album.rss"))
:tracks
first
track->span
(log "The first track is:"))
;; => #<Promise[~]>
)
In the JavaScript console, we see:
The first track is: <span>1. The Old Stuff</span>
This is cool, because the track->span function is still pure—there's no mutation occurring there. We have one and only one place where that's doing mutation, and that's display-album!, which is where we can hook into our functional core and display the tracks. In order to do that, we'll take our list of tracks, turn them into a list of <span> elements, and then set them as the children of the #tracks div.
(defn set-children! [el children]
(.replaceChildren el)
(doseq [child children]
(.appendChild el child))
el)
(defn display-album! [{:keys [title image tracks] :as album}]
(let [header (get-el "h1")
cover (get-el ".cover-image > img")
wrapper (get-el "#wrapper")]
(set! (.-innerHTML header) title)
(set! (.-src cover) image)
(->> tracks
(map track->span)
(set-children! (get-el "#tracks")))
(set-styles! wrapper "display: flex;")
album))
(comment
(p/-> (load-album (str base-path "/album.rss")) display-album!)
;; => #<Promise[~]>
)
This is fantastic... if all we want to do is know what's on an album. But of course my initial problem was wanting to listen to Garth and not having a way to do that. Now I have written much Clojure and ClojureScript, and still cannot listen to Garth. 🤔
Of course what I do have is an HTML <audio> element and an MP3 file with a source URL, and I bet if I can just put these two things together, my ears will soon be filled with the sweet sweet sounds of 90s country music.
Let's start out with the simplest thing we can do, which is to activate the first track on the album once it's loaded. Since display-album! returns the album, we can just add some code to the end of the pipeline:
(comment
(def base-path "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #'soundcljoud/base-path
(p/->> (load-album (str base-path "/album.rss"))
display-album!
:tracks
first
:src
(set! (.-src (get-el "audio"))))
;; => #<Promise[~]>
)
As soon as we evaluate this code, the <audio> element comes to life, displaying a duration and activating the play button. Pressing the play button, we do in fact hear some Garth! 🎉
However, our UX is quite poor, since there's no visual representation of which track is playing. We can fix this by emboldening the active track:
(comment
(p/let [{:keys [number src] :as track}
(p/->> (load-album (str base-path "/album.rss"))
display-album!
:tracks
first)]
(-> (get-el "#tracks")
(.-children)
seq
(nth (dec number))
(set-styles! "font-weight: bold;"))
(set! (.-src (get-el "audio")) src))
;; => #<Promise[~]>
)
Speaking of UX, though, one would imagine that they'd be able to change to a track by clicking on it. At the moment, clicking does nothing, but that's easy enough to fix by adding an event handler to our span for each track that activates the track. Let's create a function and shovel our track activating code in there:
(defn activate-track! [{:keys [number src] :as track}]
(log "Activating track:" (clj->js track))
(let [track-spans (seq (.-children (get-el "#tracks")))]
(-> track-spans
(nth (dec number))
(set-styles! "font-weight: bold;")))
(set! (.-src (get-el "audio")) src)
track)
By the way, that clj->js function takes a ClojureScript data structure (in this case, our track map) and recursively transforms it into a JavaScript object so it can be printed nicely in the JS console.
OK, now that we have activate-track! as a function, we can use it in a click handler:
(defn track->span [{:keys [number title] :as track}]
(let [span (js/document.createElement "span")]
(set! (.-innerHTML span) (str number ". " title))
(.addEventListener span "click" (partial activate-track! track))
span))
(comment
(p/-> (load-album (str base-path "/album.rss"))
display-album!
:tracks
first
activate-track!)
;; => #<Promise[~]>
)
Evaluating this code activates the first track on the album as before, and then clicking another track highlights it in bold and loads it into the <audio> element. That's good, but what isn't so good is that the first track stays bold. 😬
Luckily, there's an easy fix for this. All we need to do is reset the weight of all the track spans before bolding the active one in activate-track!:
(defn activate-track! [{:keys [number src] :as track}]
(log "Activating track:" (clj->js track))
(let [track-spans (seq (.-children (get-el "#tracks")))]
(doseq [span track-spans]
(set-styles! span "font-weight: normal;"))
(-> track-spans
(nth (dec number))
(set-styles! "font-weight: bold;")))
(set! (.-src (get-el "audio")) src)
track)
Amazing!
Whilst we're ticking off UX issues, let's think about what should happen when our user clicks on a different track. At the moment, we load the track into the player and then the user has to click the play button to start listening to it. That is perfectly reasonable when first loading the album, but if I'm listening to a track and then select another one, I would kinda expect the new track to start playing automatically instead of me having to click play manually.
Let's see how we can do this. According to the HTMLMediaElement documentation, our <audio> element should have paused attribute telling us whether playback is happening. Let's try it out:
(comment
(p/-> (load-album (str base-path "/album.rss"))
display-album!
:tracks
first
activate-track!)
;; => #<Promise[~]>
(-> (get-el "audio")
(.-paused))
;; => true
)
Now if we click the play button and check the value of the paused attribute again:
(comment
(-> (get-el "audio")
(.-paused))
;; => false
)
Excellent! Now let's see how we programatically start playing a newly loaded track. Referring back to the documentation, we discover a HTMLMediaElement.play() method. Let's try that out:
(comment
(p/-> (load-album (str base-path "/album.rss"))
display-album!
:tracks
second
activate-track!)
;; => #<Promise[~]>
(-> (get-el "audio")
(.play))
;; => #<Promise[~]>
)
Evaluating this code results in "Cowboys and Angels" starting to play!
Now we can use what we've learned to teach activate-track! to start playing the track when appropriate:
(defn activate-track! [{:keys [number src] :as track}]
(log "Activating track:" (clj->js track))
(let [track-spans (seq (.-children (get-el "#tracks")))
audio-el (get-el "audio")
paused? (.-paused audio-el)]
(doseq [span track-spans]
(set-styles! span "font-weight: normal;"))
(-> track-spans
(nth (dec number))
(set-styles! "font-weight: bold;"))
(set! (.-src audio-el) src)
(when-not paused?
(.play audio-el)))
track)
(comment
(p/-> (load-album (str base-path "/album.rss"))
display-album!
:tracks
first
activate-track!)
;; => #<Promise[~]>
)
When the album loads, the first track is activated but doesn't start playing. Clicking on another track activates it but doesn't start playing it. However, if we click the play button and start listening to the active track, then click on another track, the new track is activated and immediately starts playing.
This, my friends, is some seriously good UX! Of course, we can improve it further.
The next UX nit that we should pick is the fact that when a track ends, our poor user has to manually click on the next track and then manually click the play button just to keep listening to the album. This seems a bit mean of us, so let's see what we can do in order to be the nice people that we know we are, deep down inside.
Our good friend HTMLMediaElement has a bunch of
events that tell us useful things about what's happening with the media, and one of these events is ended:
Fired when playback stops when end of the media (
This seems like it will fit the bill quite nicely. Hopping back in our hammock for a minute, we think about what should happen when the end of a track is reached:
We can of course add a ended event listener to the <audio> element every time a new track is activated, but this is problematic because we would then want to remove the previous event listener, and it turns out that removing event
listeners is a bit
complicated. What if we instead had an event listener that knew what track was currently playing, where that track comes in the album, and what track (if any) is next? Then we'd only have to attach a listener once, right after we load the album. Let's think through how we could do that.
So far, we've been relying on the state of the DOM to tell us things like if the track is paused. A much more functional approach would be to control the state ourselves using immutable data structures and so on. A nice side effect of this (sorry, Haskell folks, Clojurists are just fine with uncontrolled side effects) is that it actually makes REPL-driven development easier as well! 🤯
Let's start by extracting a function to handle the tedium of loading the album, displaying it, and then activating the first track:
(defn load-ui! [dir]
(p/->> (load-album (str dir "/album.rss"))
display-album!
:tracks
first
activate-track!))
Now that we have this, we'll define a top-level atom to hold the state, then update our load-ui! function to stuff the album into the atom once it's loaded:
(def state (atom nil))
(defn load-ui! [dir]
(p/->> (load-album (str dir "/album.rss"))
display-album!
(assoc {} :album)
(reset! state)
:album
:tracks
first
activate-track!))
What we're doing here is creating a map to hold the state, then assoc-ing the loaded album into the map under the :album key, then putting that map into the state atom with reset!, which returns the new value saved in the atom, which is the one we just put in there, which will look like this:
{:title "Garth Brooks - Fresh Horses",
:image "https://i.discogs.com/.../LTMxNjguanBlZw.jpeg",
:tracks
({:artist "Garth Brooks",
:title "The Old Stuff",
:number 1,
:src "http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3"}
...
{:artist "Garth Brooks",
:title "Ireland",
:number 10,
:src "http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+Ireland.mp3"}), :paused? true}
We'll then grab the album back out of the map and proceed as before to activate the first track. This is a little gross, but we'll clean it up as we go.
Oh yeah, and remember when I promised this would make debugging easier? Check this out:
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
@state
;; => {:album {:title "Garth Brooks - Fresh Horses",
;; :image "https://i.discogs.com/.../LTMxNjguanBlZw.jpeg",
;; :tracks
;; ({:artist "Garth Brooks",
;; :title "The Old Stuff",
;; :number 1,
;; :src "http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3"}
;; ...
;; {:artist "Garth Brooks",
;; :title "Ireland",
;; :number 10,
;; :src "http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+Ireland.mp3"})}}
)
That's right, we no longer have to rely on logging stuff to the JS console in our promise chains!
OK, but we haven't really changed anything other than making the load-ui! function more complicated. Let's add a little more to our state atom so we can actually tackle the problem of auto-advancing tracks. First, we'll add a :paused? key:
(defn load-ui! [dir]
(p/->> (load-album (str dir "/album.rss"))
display-album!
(assoc {:paused? true} :album)
(reset! state)
:album
:tracks
first
activate-track!))
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
@state
;; => {:paused? true, :album {...}}
)
Now let's add an event listener to the <audio> element that updates the state when the play button is pressed, doing a little cleanup of the load-ui! function whilst we're at it:
(defn load-ui! [dir]
(p/let [album (load-album (str dir "/album.rss"))]
(display-album! album)
(reset! state {:paused? true, :album album})
(->> album
:tracks
first
activate-track!)
(.addEventListener (get-el "audio") "play"
#(swap! state assoc :paused? false))))
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
(:paused? @state)
;; => true
;; Click the play button and...
(:paused? @state)
;; => false
)
If you're not familiar with swap!, it takes an atom and a function which will be called with the current value of the atom, then sets the next value of the atom to whatever the function returns, just like update does for plain old maps. And also just like update, it has a shorthand form so that instead of writing this:
(swap! state #(assoc % :paused? false))
you can write this:
(swap! state assoc :paused? false)
in which case swap! will treat the arg after the atom as a function which will be called with the current value first, then the rest of the args to swap!. You can imagine that swap! is written something like this:
(defn swap!
([atom f]
(reset! atom (f @atom)))
([atom f & args]
(reset! atom (apply f @atom args))))
It's obviously not written like that, even though that would technically probably maybe work. It's actually written like this:
(defn swap!
"Atomically swaps the value of atom to be:
(apply f current-value-of-atom args). Note that f may be called
multiple times, and thus should be free of side effects. Returns
the value that was swapped in."
{:added "1.0"
:static true}
([^clojure.lang.IAtom atom f] (.swap atom f))
([^clojure.lang.IAtom atom f x] (.swap atom f x))
([^clojure.lang.IAtom atom f x y] (.swap atom f x y))
([^clojure.lang.IAtom atom f x y & args] (.swap atom f x y args)))
But you get the point.
Aaaaaanyway, I seem to have digressed—which is firmly on brand for this blog, so I apologise for nothing!
But yeah, at this point, we're back to the functionality that we had before. If we click on a track whilst the player is paused, the new track is selected but doesn't start playing, and if we click on a new track whilst the player is playing, the player plays on by playing the new track. Got it?
However, activate-track! is still relying on the DOM to keep track of whether the player is paused. Let's fix this by checking the state atom instead:
(defn activate-track! [{:keys [number src] :as track}]
(log "Activating track:" (clj->js track))
(let [track-spans (seq (.-children (get-el "#tracks")))
audio-el (get-el "audio")
;; Instead of this 👇
;; paused? (.-paused audio-el)
;; Do this! 👇
{:keys [paused?]} @state]
;; ...
)
track)
Next, let's write a function to advance to the next track:
(defn advance-track! []
(let [{:keys [active-track album]} @state
{:keys [tracks]} album
last-track? (= active-track (count tracks))]
(when-not last-track?
(activate-track! (nth tracks active-track)))))
Oops, this is relying on :active-track being present in the state atom. Let's put it there in activate-track!
(defn activate-track! [{:keys [number src] :as track}]
(log "Activating track:" (clj->js track))
(let [track-spans (seq (.-children (get-el "#tracks")))
audio-el (get-el "audio")
{:keys [paused?]} @state]
;; ...
)
;; Swappity swap swap! 👇
(swap! state assoc :active-track number)
track)
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
@state
;; => {:paused? true,
;; :active-track 1,
;; :album {...}}
)
Now we should be able to actually call advance-track! to, well, advance the track:
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
(:active-track @state)
;; => 1
(advance-track!)
;; => {:artist "Garth Brooks", :title "Cowboys And Angels", :number 2, :src "http://localhost:1341/Garth+Brooks/Fresh+Horses/Garth+Brooks+-+Cowboys+and+Angels.mp3"}
(:active-track @state)
;; => 2
)
We will have also seen the highlighted track change when we evaluated the (advance-track!) form! 🎉
What we're building up to is of course the ability to play our album continuously. When one track ends, the next should begin. And our good friend <audio> has just what we need, in the form of the ended event. If we add one line of code to register advance-track! as the listener for the ended event:
(defn load-ui! [dir]
(p/let [album (load-album (str dir "/album.rss"))]
(display-album! album)
(reset! state {:paused? true, :album album})
(->> album
:tracks
first
activate-track!)
(.addEventListener (get-el "audio") "play"
#(swap! state assoc :paused? false))
(.addEventListener (get-el "audio") "ended"
advance-track!)))
(comment
(load-ui! "http://localhost:1341/Garth+Brooks/Fresh+Horses")
;; => #<Promise[~]>
;; Click ▶️ and witness the glory!
)
We win!
Winners who have won before and know how to win will of course know that the best thing to do after winning is to stride triumphantly to the podium, receive your 🥇, wave to your adoring public, soak up the applause like warm sunshine on a July day (unless you're in the southern hemisphere, in which case the warm sunshine is best appreciated in December, unless you're close enough to the equator to appreciate warm sunshine whenever you damn well please, unless you're too close to the equator and that sunshine is too warm to appreciate because you're sweating like wild), and then head home, find a comfy chair and open a bottle of champagne or fizzy water or tasty whiskey or whatever.
I, of course, am no such winner, so instead of retiring to my comfy chair with a glass of Lagavulin, I want to jump ahead in a track, so I confidently reach for the audio control and click ahead in the timeline, and... nothing happens WTF?
Reading more documentation, I discover that I can see the current time in seconds in the track by reading its currentTime property, and I can seek to an arbitrary time by setting currentTime, so let's give that a try, shall we? (Spoiler: we shall.)
(comment
(.-currentTime (get-el "audio"))
;; => 37.010544
(set! (.-currentTime (get-el "audio")) 50)
;; => nil
;; Why did my track start over? 🤬
(.-currentTime (get-el "audio"))
;; => 2.006649
)
To make a long story short, this all boils down to how the browser actually implements seeking. When it first loads the audio track, it issues a request like this:
GET /Garth+Brooks/Fresh+Horses/Garth+Brooks+-+The+Old+Stuff.mp3 HTTP/1.1
Range: bytes=0-
and expects a response like this:
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-length: 5943424
Content-Range: bytes 0-1024000/5943424
Content-Type: audio/mpeg
It will then buffer the bytes it got back and make the track seekable within those bytes, as described here. You can peer under the hood by inspecting the buffered and seekable properties of the <audio> element:
audio.buffered.length; // returns 2
audio.buffered.start(0); // returns 0
audio.buffered.end(0); // returns 5
audio.buffered.start(1); // returns 15
audio.buffered.end(1); // returns 19
But if we do this in our player, we experience a deep feeling of melancholy:
(comment
(let [b (-> (get-el "audio")
(.-buffered))]
[(.start b 0) (.end b 0)])
;; => [0 144.758]
(-> (get-el "audio")
(.-seekable)
(.-length))
;; => 1
(let [s (-> (get-el "audio")
(.-seekable))]
[(.start s 0) (.end s 0)])
;; => [0 0]
)
The buffering looks fine, but it seems that we can only seek between 0 seconds and 0 seconds in the track, which kinda explains why attempting to set currentTime to any number that isn't 0 results in seeking back to 0. 😭
Seeking apparently only works if we get that blessed 206 Partial Content response from the webserver, so the browser knows how to make subsequent range requests to buffer more data, and unfortunately, the built-in babashka.http-server that we're using to serve up files in public/ responds like this:
HTTP/1.1 200 OK
Content-length: 5943424
Content-Type: audio/mpeg
Server: http-kit
No partial content?
We may attempt to fix this next time on "Soundcljoud, or a young man's Soundcloud clonejure", that is if there is a next time.