jmglov's blog
A blog about stuff but also things.
A blog about stuff but also things.
You remember how I've been trying to write a site analyser for my blog for the last year or so? Well, I was using DynamoDB for storing some metrics, and I realised that would bankrupt me, so I decided to use Amazon Athena run SQL queries directly on my access logs instead. That should have been a great idea, but when I fired up my REPL with awyeah-api, I got quite the unpleasant surprise:
(comment
(def athena (aws/client {:api :athena
:region (or (System/getenv "AWS_DEFAULT_REGION") "eu-west-1")}))
;; => #'user/athena
(aws/invoke athena {:op :ListWorkGroups
:request {}})
;; => {:__type "InvalidSignatureException",
;; :message
;; "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/\n\naccept:application/json\ncontent-type:application/x-amz-json-1.1\nhost:athena.eu-west-1.amazonaws.com:443\nx-amz-date:20231105T115515Z\nx-amz-target:AmazonAthena.ListWorkGroups\n\naccept;content-type;host;x-amz-date;x-amz-target\n44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20231105T115515Z\n20231105/eu-west-1/athena/aws4_request\n4dd8829addcae8b5fe35d78323f484298d478ab9af786b3caa12bac0d2f57a8c'\n",
;; :cognitect.anomalies/category :cognitect.anomalies/incorrect}
What the what?
Digging into this with some tasty debug prns, I found
out that the issue was that Athena was expecting a canonical string like this:
POST
/
accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com:443
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups
accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
And awyeah-api was generating a canonical string like that:
POST
/
accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups
accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
I'm sure you've already spotted the difference, right? 🙄
Take a look at the host header. Athena wants this:
host:athena.eu-west-1.amazonaws.com:443
And we're giving it that:
host:athena.eu-west-1.amazonaws.com
For want of a :443, the kingdom was lost! 😭
After filing a bug
report, I started reading through all the commits to aws-api since 0.8.603, which is the latest commit that's been ported over to awyeah-api. These two in particular looked pretty tasty:
I ported those two over to a clone of awyeah-api, but my REPL still informed me that I sucked:
(comment
(aws/invoke athena {:op :ListWorkGroups
:request {}})
;; => {:__type "InvalidSignatureException",
;; :message
;; "The request signature we calculated does not match the signature you provided. Haven't you heard Einstein's definition of insanity, dude? Try something else already!\n",
;; :cognitect.anomalies/category :cognitect.anomalies/you-suck}
Never one to give up so easily, I continued porting stuff, all the way up to the latest aws-api commits (skipping a few commits that I just couldn't figure out how to port), and not only did the InvalidSignatureException persist, but now I'd broken the awyeah-api tests. 🤦
After sleeping on it, I realised that the root of my problem is that I was just porting code over without really thinking about what the code was doing, and that a better approach would perhaps be to understand the code. Obv.
But instead of just understanding the code I was porting and why, I decided that I needed to understand how Michael Glaesemann AKA grzm had made aws-api work with Babashka in the first place. And how better to do that than to fork aws-api into my own Github repo and start replicating the awyeah-api stuff on top of it? (I'm sure there are much better ways to do that, but if you've been reading this blog for more than five minutes, you'll know that I never take the better way when a worse and more painful way is available. Just call me Gandalf in the Mines of Moria.)
And what better name to capture the dismay that I'm sure you're feeling as you read this than awno-api? And what better tense to capture the excitement of a fool's errand than the present?
After cloning my new forked repo, let's get to work making aws-api into something that will work with babashka. The first order of business is purely mechanical:
cd ~/Documents/code/clojure/awyeah-api
cp -r bb.edn bin CHANGES.markdown docs etc ../awno-api/
cd ../awno-api
mkdir -p src/net/jmglov test/src/net/jmglov
git mv src/cognitect/aws src/net/jmglov/awno
git mv test/src/cognitect/aws test/src/net/jmglov/awno
git rm -rf doc latest-releases.edn pom.xml UPGRADE.md README.md CHANGES.md
rm -rf src/cognitect test/src/cognitect
Now let's update bb.edn to use the latest versions of all the good stuff. We can grab the commit SHA for the latest version of the babashka port of Spec from https://github.com/babashka/spec.alpha, then the various versions of the AWS libraries from latest-releases.edn in the aws-api repo, resulting in a shiny new bb.edn:
{:paths ["resources" "src" "test/resources" "test/src"]
:deps
{org.babashka/spec.alpha {:git/url "https://github.com/babashka/spec.alpha"
:git/sha "8df0712896f596680da7a32ae44bb000b7e45e68"}
org.clojure/tools.logging {:mvn/version "1.2.4"}
com.cognitect.aws/endpoints {:mvn/version "1.1.12.307"}
com.cognitect.aws/ec2
{:mvn/version "822.2.1122.0", :aws/serviceFullName "Amazon Elastic Compute Cloud"},
com.cognitect.aws/lambda
{:mvn/version "822.2.1122.0", :aws/serviceFullName "AWS Lambda"},
com.cognitect.aws/s3
{:mvn/version "822.2.1145.0"}
com.cognitect.aws/ssm
{:mvn/version "822.2.1122.0", :aws/serviceFullName "Amazon Simple Systems Manager (SSM)"}
com.cognitect.aws/sts
{:mvn/version "822.2.1109.0", :aws/serviceFullName "AWS Security Token Service"}}}
Now to open up awno-api/README.markdown, cry havoc, and let slip the dogs of projectile-replace with M-p R! (If you're not using Doom
Emacs, you'll probably need to smack some different keys; if you don't know which ones, you can always take the `M-x projectile-replace` route).
After replacing all of the instances (you'll be prompted to replace each occurrence, but you hit ! at the first prompt if you want to YOLO replace all occurrences), you can use C-x s to invoke save-some-buffers, then hit ! at the prompt to save all of the files you just replaced cognitect.aws in.
Now that all your namespaces are belong to us, it's time for the moment of truth! Will the REPL start? With baited breath, let's hit C-c M-j to invoke cider-jack-in-clj, instruct CIDER to use babashka, and hope for the best.
Started nREPL server at 127.0.0.1:35223
For more info visit: https://book.babashka.org/#_nrepl
;; Connected to nREPL server - nrepl://127.0.0.1:35223
;; CIDER 1.7.0-snapshot (package: 1.7.0-snapshot), babashka.nrepl 0.0.6-SNAPSHOT
;; Babashka 1.3.176
;; Docs: (doc function-name)
;; (find-doc part-of-name)
;; Source: (source function-name)
;; Javadoc: (javadoc java-object-or-class)
;; Exit: <C-c C-q>
;; Results: Stored in vars *1, *2, *3, an exception in *e;
;; Startup: /home/jmglov/.nix-profile/bin/bb nrepl-server localhost:0
WARNING: Can't determine Clojure version. The refactor-nrepl middleware requires clojure 1.8.0 (or newer)WARNING: clj-refactor and refactor-nrepl are out of sync.
Their versions are 3.6.0 and n/a, respectively.
You can mute this warning by changing cljr-suppress-middleware-warnings.
user>
Equipped with our trusty REPL, it's time to get down to the real work of really making it work. Luckily for us, good ol' grzm has captured the decisions he made whilst porting aws-api in a nice porting-decisions.markdown doc, so let's start reading.
Right at the top of the doc, he mentions a set of Java classes that are missing from babashka:
java.lang.Runnablejava.lang.ThreadLocaljava.util.concurrent.ThreadFactoryjava.util.concurrent.ScheduledFuturejava.util.concurrent.ScheduledExecutorServiceSince awyeah-api was originally written, three of those five classes have been added to babashka, according to babashka.impl.classes:
java.lang.Runnablejava.util.concurrent.ThreadFactoryjava.util.concurrent.ScheduledExecutorServiceSo now we only have to deal with the other two.
According to grzm:
These classes are referenced in two namespaces:
cognitect.aws.utilandcognitect.aws.credentials.ThreadLocalis used incognitect.aws.utilto makejava.text.SimpleDateFormatthread-safe. As I'm not concerned with supporting pre-Java 8 versions, I've decided to use the thread-safejava.time.format.DateTimeFormatterrather than drop thread-safety workarounds forSimpleDateFormator implement them in some other way.The
cognitect.aws.utilnamespace is used throughout the aws-api library, either directly or transitively.The balance of the unincluded classes are used in
cognitect.aws.credentialsto provide auto-refreshing of AWS credentials. As babashka is commonly used for short-lived scripts as opposed to long-running server applications, rather than provide an alternate implementation for credential refresh, I've chosen to omit this functionality. If credential auto-refresh is something I find is useful in a babashka context some time in the future, a solution can be explored at that time.
Snatching up that tasty breadcrumb, let's open our shiny new net.jmglov.awno.util namespace.
As we have no more appetite than grzm for supporting Java versions older than Java 8, let's just follow his lead and kick ThreadLocal to the curb. That means replacing this mess from aws-api:
(defn date-format
"Return a thread-safe GMT date format that can be used with `format-date` and `parse-date`.
See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4228335"
^ThreadLocal [^String fmt]
(proxy [ThreadLocal] []
(initialValue []
(doto (SimpleDateFormat. fmt)
(.setTimeZone (TimeZone/getTimeZone "GMT"))))))
(defn format-date
([fmt]
(format-date fmt (Date.)))
([^ThreadLocal fmt inst]
(.format ^SimpleDateFormat (.get fmt) inst)))
With this lovely stuff from awyeah-api:
(defn date-format
"Return a thread-safe GMT date format that can be used with `format-date` and `parse-date`."
[^String fmt]
(.withZone (DateTimeFormatter/ofPattern fmt) (ZoneId/of "GMT")))
(defn format-date
([formatter]
(format-date formatter (Date.)))
([formatter ^Date inst]
(.format (ZonedDateTime/ofInstant (.toInstant inst) (ZoneId/of "UTC"))
^DateTimeFormatter formatter)))
Copy and paste-driven development FTW! 🎉
The parse-date function from aws-api needed a little love too. This:
(defn parse-date
[^ThreadLocal fmt s]
(.parse ^SimpleDateFormat (.get fmt) s))
Becomes that:
(defn parse-date
[formatter s]
(Date/from (.toInstant (ZonedDateTime/parse s formatter))))
This leaves only five more occurrences of ThreadLocal in the namespace:
(def ^ThreadLocal x-amz-date-format
(date-format "yyyyMMdd'T'HHmmss'Z'"))
(def ^ThreadLocal x-amz-date-only-format
(date-format "yyyyMMdd"))
(def ^ThreadLocal iso8601-date-format
(date-format "yyyy-MM-dd'T'HH:mm:ssXXX"))
(def ^ThreadLocal iso8601-msecs-date-format
(date-format "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"))
(def ^ThreadLocal rfc822-date-format
(date-format "EEE, dd MMM yyyy HH:mm:ss z"))
For these, all we need to do is remove the ^ThreadLocal type hint and we're golden! Well, almost golden. We're using a few java.time classes that weren't in the aws-api version, so we need to add those, and whilst we're at it, let's go ahead and remove the classes that aren't necessary anymore, and alphabetise the requires and imports in the ns form and OMG what are square brackets doing in the :import section? According to St.
Stuart, thou must always use round brackets with :import, never square!
Making the sign to ward off evil, we fix the glitch.
With all of this done, we should be able to eval the namespace in our REPL, right? C-c C-k (cider-load-buffer) will tell the tale:
clojure.lang.ExceptionInfo: Could not find namespace: clojure.data.json.
{:type :sci/error, :line 9, :column 3, :message "Could not find namespace: clojure.data.json.", :sci.impl/callstack #object[clojure.lang.Volatile 0xfb2c337 {:status :ready, :val ({:line 9, :column 3, :file "/home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/util.clj", :ns #object[sci.lang.Namespace 0x2c8684c "net.jmglov.awno.util"]})}], :file "/home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/util.clj"}
at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:129)
sci.impl.analyzer$return_ns_op$reify__4355.eval (analyzer.cljc:1189)
sci.impl.analyzer$return_do$reify__3968.eval (analyzer.cljc:130)
sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
sci.core$eval_form.invokeStatic (core.cljc:329)
babashka.nrepl.impl.server$eval_msg$fn__27295$fn__27296.invoke (server.clj:108)
[...]
Blerg!
No clojure.data.json, eh? Let's return to porting-decisions.markdown and see is grzm has anything to say about that. Indeed he
does!
The aws-api library depends on
clojure.data.jsonfor JSON serialization and deserialization, a pure Clojure library. Babashka includes Cheshire for JSON support and notclojure.data.json.The Clojure source of
clojure.data.jsoncan be interpreted by sci, so I could includeclojure.data.jsonas a dependency and use it as-is. Theclojure.data.jsonusage in aws-api is easily replaced by Cheshire. Replacingclojure.data.jsonwith Cheshire means one less dependency to include, and we can leverage compiled code rather than interpreted. To isolate the library choice, I've extracted the library-specific calls in thecom.grzm.awyeah.jsonnamespace.
We can copy src/com/grzm/awyeah/json.clj from awyeah-api over to awno-api and fix up the namespace declaration. Let's also make read-str plug compatible with clojure.data.json by adding a three arity form that allows passing a custom key function and of course supporting the deprecated json-str function (because it's sometimes used in aws-api instead of write-str):
(ns net.jmglov.awno.json
(:require [cheshire.core :as json]))
(defn write-str [x]
(json/generate-string x))
;; Implement deprecated function as well
(def json-str write-str)
(defn read-str
([s]
(read-str s :key-fn keyword))
([s _ key-fn]
(json/parse-string s key-fn)))
This will handle all the calls like this in the aws-api codebase:
(json/read-str :key-fn keyword)
Having done that, let's M-p R (like NPR, but runs projectile-replace instead of giving you some news with a vaugely progressive yet US-centric slant) to replace clojure.data.json with net.jmglov.awno.json project-wide. Don't forget to C-x s ! again to save all of the files we just modified.
Now let's try evaluating the namespace...
At this point, we should be able to run the net.jmglov.awno.util tests. Popping over to test/src/net/jmglov/awno/util_test.clj, we see that the Cognitect folks have left us a nice little Rich comment at the bottom of the file:
(comment
(t/run-tests))
Let's expand it a bit, put the cursor at the end of the t/run-tests form, and do some C-c C-v f c e to invoke cider-pprint-eval-last-sexp-to-comment:
(comment
(t/run-tests)
;; => {:test 6, :pass 13, :fail 0, :error 0, :type :summary}
)
Boom! Now we're well underway!
Back in porting-decisions.markdown, there's some mention of the other Java classes not included in babashka:
The balance of the unincluded classes are used in
cognitect.aws.credentialsto provide auto-refreshing of AWS credentials. As babashka is commonly used for short-lived scripts as opposed to long-running server applications, rather than provide an alternate implementation for credential refresh, I've chosen to omit this functionality. If credential auto-refresh is something I find is useful in a babashka context some time in the future, a solution can be explored at that time.
Seems like we can also live without credential auto-refresh, so let's have a quick look at the net.jmglov.awno.credentials and see what we'll need to do to make it work.
Looks like we need to rip this bit out, since it uses thready thread stuff that we hate and fear:
(defonce ^:private scheduled-executor-service
(delay
(Executors/newScheduledThreadPool 1 (reify ThreadFactory
(newThread [_ r]
(doto (Thread. r)
(.setName "net.jmglov.awno-api.credentials-provider")
(.setDaemon true)))))))
Now we can simplify refresh! from this:
(defn ^:skip-wiki refresh!
"For internal use. Don't call directly.
Invokes `(fetch provider)`, resets the `credentials-atom` with and
returns the result.
If the credentials returned by the provider are not valid, resets
both atoms to nil and returns nil."
[credentials-atom scheduled-refresh-atom provider scheduler]
(try
(let [{:keys [::ttl] :as new-creds} (fetch provider)]
(reset! scheduled-refresh-atom
(when ttl
(.schedule ^ScheduledExecutorService scheduler
^Runnable #(refresh! credentials-atom scheduled-refresh-atom provider scheduler)
^long ttl
TimeUnit/SECONDS)))
(reset! credentials-atom new-creds))
(catch Throwable t
(reset! scheduled-refresh-atom nil)
(log/error t "Error fetching credentials."))))
To that:
(defn ^:skip-wiki refresh!
"For internal use. Don't call directly.
Invokes `(fetch provider)`, resets the `credentials-atom` with and
returns the result.
If the credentials returned by the provider are not valid, resets
both atoms to nil and returns nil."
[credentials-atom provider]
(try
(let [new-creds (fetch provider)]
(reset! credentials-atom new-creds))
(catch Throwable t
(log/error t "Error fetching credentials."))))
And then we can rip out this stuff:
(defn cached-credentials-with-auto-refresh
"Returns a CredentialsProvider which wraps `provider`, caching
credentials returned by `fetch`, and auto-refreshing the cached
credentials in a background thread when the credentials include a
::ttl.
Call `stop` to cancel future auto-refreshes.
The default ScheduledExecutorService uses a ThreadFactory that
spawns daemon threads. You can override this by providing your own
ScheduledExecutorService.
Alpha. Subject to change."
([provider]
(cached-credentials-with-auto-refresh provider @scheduled-executor-service))
([provider scheduler]
(let [credentials-atom (atom nil)
scheduled-refresh-atom (atom nil)]
(reify
CredentialsProvider
(fetch [_]
(or @credentials-atom
(refresh! credentials-atom scheduled-refresh-atom provider scheduler)))
Stoppable
(-stop [_]
(-stop provider)
(when-let [r @scheduled-refresh-atom]
(.cancel ^ScheduledFuture r true)))))))
(defn ^:deprecated auto-refreshing-credentials
"Deprecated. Use cached-credentials-with-auto-refresh"
([provider] (cached-credentials-with-auto-refresh provider))
([provider scheduler] (cached-credentials-with-auto-refresh provider scheduler)))
And replace it with a much simpler function:
(defn cached-credentials
"Returns a CredentialsProvider which wraps `provider`, caching
credentials returned by `fetch`, and auto-refreshing the cached
credentials in a background thread when the credentials include a
::ttl.
Call `stop` to cancel future auto-refreshes.
The default ScheduledExecutorService uses a ThreadFactory that
spawns daemon threads. You can override this by providing your own
ScheduledExecutorService.
Alpha. Subject to change."
[provider]
(let [credentials-atom (atom nil)]
(reify
CredentialsProvider
(fetch [_]
(or @credentials-atom
(refresh! credentials-atom provider)))
Stoppable
(-stop [_]
(-stop provider)))))
We can now replace all occurrences of cached-credentials-with-auto-refresh in the file with cached-credentials.
And now the stop function doesn't actually need to do anything:
(defn stop
"no-op"
[_credentials])
With this, C-c C-k evaluates the namespace without errors, so let's go over to net.jmglov.awno.credentials-test and see if the tests work. But first, let's rip out anything to do with auto-refreshing credentials. That means replacing all of the cached-credentials-with-auto-refresh in the file with cached-credentials and removing the explicit test of auto-refreshing:
(deftest auto-refresh-test
(let [cnt (atom 0)
p (reify credentials/CredentialsProvider
(credentials/fetch [_]
(swap! cnt inc)
{:aws/access-key-id "id"
:aws/secret-access-key "secret"
::credentials/ttl 1}))
creds (credentials/cached-credentials p)]
(credentials/fetch creds)
(Thread/sleep 2500)
(let [refreshed @cnt]
(credentials/stop creds)
(Thread/sleep 1000)
(is (= 3 refreshed) "The credentials have been refreshed.")
(is (= refreshed @cnt) "We stopped the auto-refreshing process."))))
OK, looking good! Let's give it a C-c C-k:
clojure.lang.ExceptionInfo: Could not find namespace: clojure.tools.logging.test.
{:type :sci/error, :line 5, :column 3, :message "Could not find namespace: clojure.tools.logging.test.", :sci.impl/callstack #object[clojure.lang.Volatile 0x13387b21 {:status :ready, :val ({:line 5, :column 3, :file "/home/jmglov/Documents/code/clojure/awno-api/test/src/net/jmglov/awno/credentials_test.clj", :ns #object[sci.lang.Namespace 0x65a7dc8f "net.jmglov.awno.credentials-test"]})}], :file "/home/jmglov/Documents/code/clojure/awno-api/test/src/net/jmglov/awno/credentials_test.clj"}
at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:129)
sci.impl.analyzer$return_ns_op$reify__4355.eval (analyzer.cljc:1189)
sci.impl.analyzer$return_do$reify__3968.eval (analyzer.cljc:130)
sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
sci.core$eval_form.invokeStatic (core.cljc:329)
[...]
Oh no! It looks like whatever version of clojure.tools.logging that is included in babashka doesn't have clojure.tools.logging.test! 😭
For now, let's just comment out the stuff that needs clojure.tools.logging.test and get on with our lives:
(ns net.jmglov.awno.credentials-test
(:require [clojure.test :as t :refer [deftest testing use-fixtures is]]
[clojure.java.io :as io]
#_[clojure.tools.logging.test :refer [with-log logged?]]
[net.jmglov.awno.credentials :as credentials]
[net.jmglov.awno.util :as u]
[net.jmglov.awno.test.utils :as tu]
[net.jmglov.awno.ec2-metadata-utils :as ec2-metadata-utils]
[net.jmglov.awno.ec2-metadata-utils-test :as ec2-metadata-utils-test])
(:import (java.time Instant)))
;; [...]
#_(deftest valid-credentials-test
(with-log
(credentials/valid-credentials nil "x provider")
(is (logged? 'net.jmglov.awno.credentials :debug (str "Unable to fetch credentials from x provider."))))
(with-log
(credentials/valid-credentials {:aws/access-key-id "id"
:aws/secret-access-key "secret"}
"x provider")
(is (logged? 'net.jmglov.awno.credentials :debug (str "Fetched credentials from x provider.")))))
After doing that, the namespace evaluates just fine, so let's try running the tests:
(comment
(t/run-tests)
;; => java.lang.RuntimeException: Could not find net.jmglov.awno_http.edn on classpath. user /home/jmglov/Documents/code/clojure/awno-api/src/net/jmglov/awno/http.clj:76:9
)
Oh sweet mother of mercy what now?
Since our error mentioned HTTP by name, let's look up HTTP in porting-decisions.markdown:
The aws-api library defines a protocol (
cognitect.aws.http/HttpClient) to provide an interface between the aws-api data transformation logic and the specific HTTP client implementation. The aws-api includes an implementation for thecom.cognitect/http-clientlibrary. Interfaces are great: we can provide our own implementations of thecognitect.aws.http/HttpClientinterface based on the various HTTP clients included in babashka.
Alrighty, looks like we're gonna need to implement an HTTP client. Let's have a look at how this is done in awyeah-api by peeking in src/com/grzm/awyeah/http.clj. It seems by and large the same as aws-api's src/cognitect/aws/http.clj, with exception of some class loader stuff, which is also mentioned in porting-decisions.markdown:
There are a few other compatiblity issues, such as the use of
java.lang.ClassLoader::getResourcesincognitect.aws.http/configured-client, and replacing^int xhinting with explicit(int x)casts.
Whilst java.lang.ClassLoader::getResources isn't mentioned directly, there are some class loader shenanigans happening here:
(defn- configured-client
"If a single net.jmglov.awno_http.edn is found on the classpath,
returns the symbol bound to :constructor-var.
Throws if 0 or > 1 net.jmglov.awno_http.edn files are found.
"
[]
(let [cl (.. Thread currentThread getContextClassLoader)
cfgs (enumeration-seq (.getResources cl "net.jmglov.awno_http.edn"))]
(case (count cfgs)
0 (throw (RuntimeException. "Could not find net.jmglov.awno_http.edn on classpath."))
1 (-> cfgs first read-config :constructor-var)
(throw (ex-info "Found too many http-client cfgs. Pick one." {:config cfgs})))))
Let's just follow grzm's lead and rip it out:
(defn- configured-client
"If a single com_grzm_awyeah_http.edn is found on the classpath,
returns the symbol bound to :constructor-var.
Throws if 0 or > 1 com_grzm_awyeah_http.edn files are found.
"
[]
(try
(-> (io/resource "net_jmglov_awno_http.edn") read-config :constructor-var)
(catch Throwable _
(throw (RuntimeException. "Could not find com_grzm_awyeah_http.edn on classpath.")))))
This dynaload stuff also looks a bit unnecessary, so let's flush it as well by turning this:
(defn resolve-http-client
[http-client-or-sym]
(let [c (or (when (symbol? http-client-or-sym)
(let [ctor @(dynaload/load-var http-client-or-sym)]
(ctor)))
http-client-or-sym
(let [ctor @(dynaload/load-var (configured-client))]
(ctor)))]
(when-not (client? c)
(throw (ex-info "not an http client" {:provided http-client-or-sym
:resolved c})))
c))
Into that:
(defn resolve-http-client
[http-client-or-sym]
(let [c (or (when (symbol? http-client-or-sym)
(let [ctor (requiring-resolve http-client-or-sym)]
(ctor)))
http-client-or-sym
(let [ctor (requiring-resolve (configured-client))]
(ctor)))]
(when-not (client? c)
(throw (ex-info "not an http client" {:provided http-client-or-sym
:resolved c})))
c))
This now means that we're gonna need clojure.java.io, so let's pop up to the ns form and require it. And whilst we're at it, we can get rid of this suspicious net.jmglov.awno.dynaload require, and kill the associated source file with fire! That leaves us with an ns form like this:
(ns ^:skip-wiki net.jmglov.awno.http
"Impl, don't call directly."
(:require [clojure.edn :as edn]
[clojure.core.async :as a]
[clojure.java.io :as io]))
We're also going to need the net_jmglov_awno_http.edn file mentioned above. We can just modify the original cognitect_aws_http.edn, so this:
{:constructor-var net.jmglov.awno.http.cognitect/create}
Becomes that:
{:constructor-var net.jmglov.awno.http.awno/create}
Once this is done, we just need to rename the file to net_jmglov_awno_http.edn and Robert should be one of our parents' brother. Oh yeah, and since we changed some stuff in resources, a C-c M-r to invoke cider-restart is sadly required. 😢
Continuing on our mission to civilise HTTP, let's turn our roving eye to the net.jmglov.awno.http.awno namespace, which is sadly not there, because our original projectile-replace of cognitect.aws with net.jmglov.awno left us with a file called src/net/jmglov/awno/http/cognitect.clj with a namespace of net.jmglov.awno.http.cognitect. We can fix the name easily enough, but looking at awyeah-api's src/com/grzm/awyeah/http/awyeah.clj, we see that he's replaced cognitect.http-client with a custom com.grzm.awyeah.http-client. Let's follow the same pattern:
(ns ^:skip-wiki net.jmglov.awno.http.awno
(:require [net.jmglov.awno.http :as aws]
[net.jmglov.awno.http-client :as impl]))
(set! *warn-on-reflection* true)
(defn create
[]
(let [c (impl/create nil)]
(reify aws/HttpClient
(-submit [_ request channel]
(impl/submit c request channel))
(-stop [_]
(impl/stop c)))))
We'll also need to rename the file from cognitect.clj to awno.clj. Having done this, it's time to write net.jmglov.awno.http-client.
Let's start by straight up copying awyeah-api's com/grzm/awyeah/http_client.clj and replacing all the occurrences of com.grzm.awyeah with net.jmglov.awno to jmglovify it:
(ns net.jmglov.awno.http-client
(:require
[clojure.core.async :refer [put!] :as a]
[clojure.spec.alpha :as s]
[net.jmglov.awno.http-client.client :as client]
[net.jmglov.awno.http-client.specs])
(:import
(clojure.lang ExceptionInfo)
(java.net URI)
(java.net.http HttpClient
HttpClient$Redirect
HttpHeaders
HttpRequest
HttpRequest$Builder
HttpRequest$BodyPublishers
HttpResponse
HttpResponse$BodyHandlers)
(java.nio ByteBuffer)
(java.time Duration)
(java.util.function Function)))
(set! *warn-on-reflection* true)
(defn submit
"Submit an http request, channel will be filled with response. Returns ch.
Request map:
:server-name string
:server-port integer
:uri string
:query-string string, optional
:request-method :get/:post/:put/:head
:scheme :http or :https
:headers map from downcased string to string
:body ByteBuffer, optional
:net.jmglov.awno.http-client/timeout-msec opt, total request send/receive timeout
:net.jmglov.awno.http-client/meta opt, data to be added to the response map
content-type must be specified in the headers map
content-length is derived from the ByteBuffer passed to body
Response map:
:status integer HTTP status code
:body ByteBuffer, optional
:header map from downcased string to string
:net.jmglov.awno.http-client/meta opt, data from the request
On error, response map is per cognitect.anomalies"
([client request]
(submit client request (a/chan 1)))
([client request ch]
(s/assert ::submit-request request)
(client/submit client request ch)))
(def method-string
{:get "GET"
:post "POST"
:put "PUT"
:head "HEAD"
:delete "DELETE"
:patch "PATCH"})
(defn byte-buffer->byte-array
[^ByteBuffer bbuf]
(.rewind bbuf)
(let [arr (byte-array (.remaining bbuf))]
(.get bbuf arr)
arr))
(defn flatten-headers [headers]
(->> headers
(mapcat (fn [[nom val]]
(if (coll? val)
(map (fn [v] [(name nom) v]) val)
[[(name nom) val]])))))
;; "host" is a restricted header.
;; The host header is part of the AWS signed headers signature,
;; so it's included in the list of headers for request processing,
;; but we let the java.net.http HttpRequest assign the host header
;; from the URI rather than setting it directly.
(def restricted-headers #{"host"})
(defn add-headers
[^HttpRequest$Builder builder headers]
(doseq [[nom val] (->> (flatten-headers headers)
(remove (fn [[nom _]] (restricted-headers nom))))]
(.header builder nom val))
builder)
(defn map->http-request
[{:keys [scheme server-name server-port uri query-string
request-method headers body]
:or {scheme "https"}
:as m}]
(let [uri (URI. (str (name scheme)
"://"
server-name
(some->> server-port (str ":"))
uri
(some->> query-string (str "?"))))
method (method-string request-method)
bp (if body
(HttpRequest$BodyPublishers/ofByteArray (byte-buffer->byte-array body))
(HttpRequest$BodyPublishers/noBody))
builder (-> (HttpRequest/newBuilder uri)
(.method ^String method bp))]
(when (seq headers)
(add-headers builder headers))
(when (::timeout-msec m)
(.timeout builder (Duration/ofMillis (::timeout-msec m))))
(.build builder)))
(defn error->anomaly [^Throwable t]
{:cognitect.anomalies/category :cognitect.anomalies/fault
:cognitect.anomalies/message (.getMessage t)
::throwable t})
(defn header-map [^HttpHeaders headers]
(->> headers
(.map)
(map (fn [[k v]] [k (if (< 1 (count v))
(into [] v)
(first v))]))
(into {})))
(defn response-body?
[^HttpRequest http-request]
((complement #{"HEAD"}) (.method http-request)))
(defn response-map
[^HttpRequest http-request ^HttpResponse http-response]
(let [body (when (response-body? http-request)
(.body http-response))]
(cond-> {:status (.statusCode http-response)
:headers (header-map (.headers http-response))}
body (assoc :body (ByteBuffer/wrap body)))))
(defrecord Client
[^HttpClient http-client pending-ops pending-ops-limit]
client/Client
(-submit [_ request ch]
(if (< pending-ops-limit (swap! pending-ops inc))
(do
(put! ch (merge {:cognitect.anomalies/category :cognitect.anomalies/busy
:cognitect.anomalies/message (str "Ops limit reached: " pending-ops-limit)
:pending-ops-limit pending-ops-limit}
(select-keys request [::meta])))
(swap! pending-ops dec))
(try
(let [http-request (map->http-request request)]
(-> (.sendAsync http-client http-request (HttpResponse$BodyHandlers/ofByteArray))
(.thenApply
(reify Function
(apply [_ http-response]
(put! ch (merge (response-map http-request http-response)
(select-keys request [::meta]))))))
(.exceptionally
(reify Function
(apply [_ e]
(let [cause (.getCause ^Exception e)
t (if (instance? ExceptionInfo cause) cause e)]
(put! ch (merge (error->anomaly t) (select-keys request [::meta]))))))))
(swap! pending-ops dec))
(catch Throwable t
(put! ch (merge (error->anomaly t) (select-keys request [::meta])))
(swap! pending-ops dec))))
ch))
(defn create
[{:keys [connect-timeout-msecs
pending-ops-limit]
:or {connect-timeout-msecs 5000
pending-ops-limit 64}
:as _config}]
(let [http-client (.build (-> (HttpClient/newBuilder)
(.connectTimeout (Duration/ofMillis connect-timeout-msecs))
(.followRedirects HttpClient$Redirect/NORMAL)))]
(->Client http-client (atom 0) pending-ops-limit)))
(defn stop
"no-op. Implemented for compatibility"
[^Client _client])
Now we need to create those net.jmglov.awno.http-client.client and net.jmglov.awno.http-client.specs namespaces, so we can make copies of com/grzm/awyeah/http_client/client.clj and com/grzm/awyeah/http_client/specs.clj and replace all the occurrences of com.grzm.awyeah with net.jmglov.awno, giving us this:
(ns net.jmglov.awno.http-client.client)
(defprotocol Client
(-submit [_ request ch]))
(defn submit [client request ch]
(-submit client request ch))
And this:
(ns net.jmglov.awno.http-client.specs
(:require
[clojure.spec.alpha :as s])
(:import
(java.nio ByteBuffer)))
(defn- keyword-or-non-empty-string? [x]
(or (keyword? x)
(and (string? x) (not-empty x))))
(s/def :net.jmglov.awno.http-client/server-name string?)
(s/def :net.jmglov.awno.http-client/server-port int?)
(s/def :net.jmglov.awno.http-client/uri string?)
(s/def :net.jmglov.awno.http-client/request-method keyword?)
(s/def :net.jmglov.awno.http-client/scheme keyword-or-non-empty-string?)
(s/def :net.jmglov.awno.http-client/timeout-msec int?)
(s/def :net.jmglov.awno.http-client/meta map?)
(s/def :net.jmglov.awno.http-client/body (s/nilable #(instance? ByteBuffer %)))
(s/def :net.jmglov.awno.http-client/query-string string?)
(s/def :net.jmglov.awno.http-client/headers map?)
(s/def :net.jmglov.awno.http-client/submit-request
(s/keys :req-un [:net.jmglov.awno.http-client/server-name
:net.jmglov.awno.http-client/server-port
:net.jmglov.awno.http-client/uri
:net.jmglov.awno.http-client/request-method
:net.jmglov.awno.http-client/scheme]
:opt [:net.jmglov.awno.http-client/timeout-msec
:net.jmglov.awno.http-client/meta]
:opt-un [:net.jmglov.awno.http-client/body
:net.jmglov.awno.http-client/query-string
:net.jmglov.awno.http-client/headers]))
(s/def :net.jmglov.awno.http-client/status int?)
(s/def :net.jmglov.awno.http-client/submit-http-response
(s/keys :req-un [:net.jmglov.awno.http-client/status]
:opt [:net.jmglov.awno.http-client/meta]
:opt-un [:net.jmglov.awno.http-client/body
:net.jmglov.awno.http-client/headers]))
(s/def :net.jmglov.awno.http-client/error keyword?)
(s/def :net.jmglov.awno.http-client/throwable #(instance? Throwable %))
(s/def :net.jmglov.awno.http-client/submit-error-response
(s/keys :req [:net.jmglov.awno.http-client/error]
:opt [:net.jmglov.awno.http-client/throwable
:net.jmglov.awno.http-client/meta]))
(s/def :net.jmglov.awno.http-client/submit-response
(s/or :http-response :net.jmglov.awno.http-client/submit-http-response
:error-response :net.jmglov.awno.http-client/submit-error-response))
Having done everything that's obvious from porting-decisions.markdown, we might as well just see if it works. But before we do that, how about a...
See you next week for the thrilling conclusion of the trainwreck that is me attempting to replace an oil filter by completely disassembling the car and making my own oil filter out of paper towels and a coathanger I found lying around. What could possibly go wrong?