clickr, or a young man's Flickr clonejure

A space station

Way back when, before I got my first Macbook (the white 2006 model, so cool!) and thus had no iPhoto to keep all of my photos for me, I used some open source photo album thingy on Linux (it was so long ago that I don't even remember what it was called) on a server running under my desk with a fixed IP. Using this amazing setup, I could create photo albums and share them with family who wanted to keep up with my thrilling life. When we moved to Tokyo in 2005, I got rid of the Linux server (and the desk it was under and the apartment the desk was in) and brought only a laptop with me. When we first arrived, we didn't even have internet, so I borrowed wifi from someone in the next building who had helpfully eschewed a password, and I thought it would be a bit cheeky to locate that person and ask them to purchase a fixed IP so that I could run a webserver so I could share photo albums with my family.

Luckily for me, two people named Stewart Butterfield and Caterina Fake felt my pain and started a company called Ludicorp and tried to create a web-based massively multiplayer online game called Game Neverending, which they failed to do but made some pretty cool tools for assets which they then decided could be turned into an online photo sharing service which they named Flickr.

Which is a very long-winded way of saying—and unless you're very new to this blog, you will know that I am a very long-winded person—that I signed up for Flickr for my photo-sharing needs.

A few years later, I bought the aforementioned Macbook, and started using iPhoto, which was much nicer than copying photos from a memory card onto my hard drive and then uploading them to Flickr, so I got kinda lazy about creating albums on Flickr, until my son was born and my family started demanding to see pictures of him.

Flash forward to many years later, and now we all have smartphones and stuff (even my mom!), so we just take pictures which automatically go to iCloud and iPhoto and iDon'tKnowWhereElse and then share them with each other in Signal and Telegram and WhatsApp and 95 other smartphone messaging apps like normal people, so we don't really use Flickr any more. In fact, I only remember we have Flickr when they email me once a year to remind me they're about to charge my credit card another $100 or whatever.

My reluctance to pay for something that I don't use is exceeded only by my reluctance to spend time on manually migrating off that thing, so I didn't do anything about this until a couple of years ago, when I just so happened to get the annual renewal email right before my winter vacation started, so I was like "Oh, now I'll have some free time and I'll probably be bored so I should probably see if Flickr has an API that I could use to download my photos and put them onto S3 like the good lord intended."

Flickr did in fact have an API, and better yet, they had a Java client, which I wrapped up in some Clojure and named clickr because it's like Flickr in Clojure ha ha ha I'm so clever. I got the listing of albums and downloading of photos and uploading of them to S3 working pretty easily, but for some reason didn't actually back up the albums to S3 except for the first one, probably because I evaluated some code in the REPL to start the back up and then switched tabs to read a blog or something and then forgot what I was doing and closed my laptop or something similarly absentminded.

Flash forward to a couple of weeks ago, when I once again got the famous email from Flickr and remembered that I really should get around to backing up those albums so I can stop paying Flickr and pay AWS a few more cents a year to store them for me. Since I now hate Clojure and only love Babashka, I decided that instead of going back to the Clojure wrapper around the Java library, I'd whip up a quick HTTP client in Babashka instead and then just use awyeah-api for the S3-ing. I got a few hours into struggling with signing requests to get an OAuth access token which I could then get the user's (me) authorization to exchange for an access token—you know, as you do—when I decided that my life was way to short for taking a URL such as

https://www.flickr.com/services/oauth/request_token
?oauth_nonce=89601180
&oauth_timestamp=1305583298
&oauth_consumer_key=653e7a6ecc1d528c516cc8f92cf98611
&oauth_signature_method=HMAC-SHA1
&oauth_version=1.0
&oauth_callback=http%3A%2F%2Fwww.example.com

and creating a base string such as

GET&https%3A%2F%2Fwww.flickr.com%2Fservices%2Foauth%2Frequest_token&oauth_callback%3Dhttp%253A%252F%252Fwww.example.com%26oauth_consumer_key%3D653e7a6ecc1d528c516cc8f92cf98611%26oauth_nonce%3D95613465%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1305586162%26oauth_version%3D1.0

and then generating the following signature

7w18YS2bONDPL%2FzgyzP5XTr5af4%3D

and then wondering why in the hell Flickr's API kept giving me a signature incorrect error when the damn signature was obviously correct because you know...

A diagram showing the three step OAuth flow for Flickr

I rapidly came to the conclusion that I should retreat to the safe embrace of JVM Clojure and just let the Java client do all the nasty work of authenticating so I could do the fun work of Clojuring.

Onwards and upwards!

I cloned my trusty clickr repo, then screamed as I saw a project.clj and realised that I didn't even have Leiningen installed because I roll with tools.deps now and I can't remember any lein commands and OMG it's nearly 2024 (this was a few weeks ago, remember?) and I'd better generate a deps.edn like a normal person:

{:paths ["src" "dev"]
 :deps {com.flickr4java/flickr4java {:mvn/version "3.0.1"}}}

Having done this, I could open src/clickr/flickr.clj and try and remember how to use some code I wrote a million couple years ago and didn't document in any way, shape, or form because I'm a bad boy and life's too short for documentation!

According to Flickr4Java:

To use the API just construct an instance of the class com.flickr4java.flickr.test.Flickr and request the interfaces which you need to work with.

String apiKey = "YOUR_API_KEY";
String sharedSecret = "YOUR_SHARED_SECRET";
Flickr f = new Flickr(apiKey, sharedSecret, new REST());

OK, so we can start our namespace out by importing the Flickr client:

(ns clickr.flickr
  (:import (com.flickr4java.flickr Flickr
                                   REST)))

And then C-c M-j (cider-jack-in-clj) to start a REPL, and then C-c C-k (cider-load-buffer) to evaluate the buffer, then it seems like I need an API key and the corresponding secret, which I can grab from https://www.flickr.com/services/apps/by/jmglov. Lemme just drop that in a map for safe-keeping and do a little C-v f c e (cider-pprint-eval-last-sexp-to-comment, obv):

(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"})
  ;; => #'clickr.flickr/config

  )

OK, so creating a client should be as simple as this:

  (def flickr (Flickr. (:api-key config) (:secret config) (REST.)))
  ;; => #'clickr.flickr/flickr

And with this, it should be easy to figure out how to get a list of albums, right? Right?

What the Flickr?

Popping over to Flickr's API reference, I'll just do a quick search for album and... 0 occurrences? Wait what now?

OK, time to scan the page for something that looks album-ish. 🙄

Ah, photosets.getList looks promising. Searching the Flickr4Java repo for photoset turns up a src/examples/java/Backup.java, which looks quite helpful:

public class Backup {
    private final String nsid;
    private final Flickr flickr;
    private AuthStore authStore;

    public Backup(String apiKey, String nsid, String sharedSecret, File authsDir) throws FlickrException {
        flickr = new Flickr(apiKey, sharedSecret, new REST());
        this.nsid = nsid;

        if (authsDir != null) {
            this.authStore = new FileAuthStore(authsDir);
        }
    }

    // ...

    public static void main(String[] args) throws Exception {
        if (args.length < 4) {
            System.out.println("Usage: java " + Backup.class.getName() + " api_key nsid shared_secret output_dir");
            System.exit(1);
        }
        Backup bf = new Backup(args[0], args[1], args[2], new File(System.getProperty("user.home") + File.separatorChar + ".flickrAuth"));
        bf.doBackup(new File(args[3]));
    }
}

If we look at the doBackup() method, it looks like there's some authorisation we need to do above and beyond the API key and secret, which aligns with all the Flickr API docs said about "every request must be signed OAuth blah blah blah":

RequestContext rc = RequestContext.getRequestContext();

if (this.authStore != null) {
    Auth auth = this.authStore.retrieve(this.nsid);
    if (auth == null) {
        this.authorize();
    } else {
        rc.setAuth(auth);
    }
}

Assuming we have no Auth object from the start, let's have a look at what the authorize() method does:

private void authorize() throws IOException, FlickrException {
    AuthInterface authInterface = flickr.getAuthInterface();
    OAuth1RequestToken requestToken = authInterface.getRequestToken();

    String url = authInterface.getAuthorizationUrl(requestToken, Permission.READ);
    System.out.println("Follow this URL to authorise yourself on Flickr");
    System.out.println(url);
    System.out.println("Paste in the token it gives you:");
    System.out.print(">>");

    String tokenKey = new Scanner(System.in).nextLine();

    OAuth1Token accessToken = authInterface.getAccessToken(requestToken, tokenKey);

    Auth auth = authInterface.checkToken(accessToken);
    RequestContext.getRequestContext().setAuth(auth);
    this.authStore.store(auth);
    System.out.println("Thanks.  You probably will not have to do this every time.  Now starting backup.");
}

OK, translating this to Clojure is pretty straight-forward. I'll start with my usual approach when dealing with API clients of creating a "context" which contains my config, various clients, and any other data that I might need to pass around. Let's create an init-client function that takes my config as an argument and returns such a context:

(defn init-client [{:keys [api-key secret] :as ctx}]
  (assoc ctx :flickr {:client (Flickr. api-key secret (REST.))}))

We can call that and get a context back containing the client:

(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"})
  ;; => #'clickr.flickr/config

  (init-client config)
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :flickr
  ;;     {:client
  ;;      #object[com.flickr4java.flickr.Flickr 0x12246439 "com.flickr4java.flickr.Flickr@12246439"]}}

  )

Now I have a client, but I need to perform the authorisation dance from the Java code. In the constructor of the Java class, they create a FileAuthStore in the authsDir, which is actually the directory called output_dir in the main() function. 🙄

Let's be a good citizen by creating this auth store in the user's home directory. In order to do this, let's add the awesome babashka.fs library to our deps:

{:paths ["src" "dev"]
 :deps {babashka/fs {:mvn/version "0.4.19"}
        com.flickr4java/flickr4java {:mvn/version "3.0.1"}}}

babashka.fs contains a bunch of really useful utility functions that work in both Babashka and JVM Clojure!

Tragically, after adding a new dep, I need to restart my REPL. It is possible to hotload dependencies into a running REPL process, which should have gotten easier in Clojure 1.12, and of course borkdude has a deps.add-lib project which should make it even easier, but I haven't used this stuff yet, so I'll just do the primitive thing and restart (swearing under my breath), of course.

Now that we have babashka.fs, creating a FileAuthStore in the ~/.flickrAuth directory is as simple as this:

(ns clickr.flickr
  (:require [babashka.fs :as fs])
  (:import (com.flickr4java.flickr Flickr
                                   REST)
           (com.flickr4java.flickr.util FileAuthStore)))

(defn make-auth-store []
  (FileAuthStore. (fs/file (fs/home) ".flickrAuth")))

Let's create the auth store in init-client and add it to the context along with the Flickr client itself:

(defn init-client [{:keys [api-key secret] :as ctx}]
  (let [client (Flickr. api-key secret (REST.))
        auth-store (make-auth-store)]
    (assoc ctx :flickr {:client client, :auth-store auth-store})))

Now that we have a way to create an auth store, let's write an authorise function to exchange a request token for an access token:

(ns clickr.flickr
  (:require [babashka.fs :as fs])
  (:import (com.flickr4java.flickr Flickr
                                   REST)
           (com.flickr4java.flickr.auth Permission)
           (com.flickr4java.flickr.util FileAuthStore)))

(defn authorise [{:keys [flickr] :as ctx}]
  (let [{:keys [client]} flickr
        auth-interface (.getAuthInterface client)
        req-token (.getRequestToken auth-interface)
        url (.getAuthorizationUrl auth-interface req-token Permission/READ)]
    (update ctx :flickr assoc :url url, :req-token req-token)))

Let's try it out:

(comment

  (def ctx (init-client config))
  ;; => #'clickr.flickr/ctx

  (authorise ctx)
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :flickr
  ;;     {:client
  ;;      #object[com.flickr4java.flickr.Flickr 0x12246439 "com.flickr4java.flickr.Flickr@12246439"]
  ;;      :auth-store
  ;;      #object[com.flickr4java.flickr.util.FileAuthStore 0x6b1f411f "com.flickr4java.flickr.util.FileAuthStore@6b1f411f"]
  ;;      :url
  ;;      "https://www.flickr.com/services/oauth/authorize?oauth_token=72157720906512072-49cebfb020e7fb06&perms=read"
  ;;      :req-token
  ;;      #object[com.github.scribejava.core.model.OAuth1RequestToken 0x2617e72c "com.github.scribejava.core.model.OAuth1RequestToken@43e76617"]}}

  )

If I visit that URL in a web browser where we're signed into Flickr, I'm prompted to authorise the app corresponding to the API key I previously created:

A prompt to authorize an app to my Flickr account

If I click "OK, I'll authorize it", I get a code that I should "type into the application". Let's set that up by modifying the authorise function to accept a code, which it will use to exchange the request token for an access token, then store that access token in the auth store that we previously created.

(ns clickr.flickr
  (:require [babashka.fs :as fs])
  (:import (com.flickr4java.flickr Flickr
                                   RequestContext
                                   REST)
           (com.flickr4java.flickr.auth Permission)
           (com.flickr4java.flickr.util FileAuthStore)))

(defn authorise [{:keys [flickr] :as ctx}]
  (let [{:keys [client auth-store req-token token-key]} flickr
        auth-interface (.getAuthInterface client)]
    (if (and req-token token-key)
      (let [access-token (.getAccessToken auth-interface req-token token-key)
            auth (.checkToken auth-interface access-token)]
        (.setAuth (RequestContext/getRequestContext) auth)
        (.store auth-store auth)
        (update ctx :flickr dissoc :req-token :token-key :url))
      (let [req-token (.getRequestToken auth-interface)
            url (.getAuthorizationUrl auth-interface req-token Permission/READ)]
        (update ctx :flickr assoc :url url, :req-token req-token)))))

The idea here is that if you pass a request token and token code in the Flickr context, you want to exchange a request token for an access token, and if you don't, you want to obtain a request token and the URL to authorise it.

OK, let's give it a whirl:

(comment

  (def ctx (authorise ctx))
  ;; => #'clickr.flickr/ctx

  (get-in ctx [:flickr :url])
  ;; => "https://www.flickr.com/services/oauth/authorize?oauth_token=72157720906550331-8cf736dfbfe34398&perms=read"

  (authorise (assoc-in ctx [:flickr :token-key] "123-456-789"))
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :flickr
  ;;     {:client
  ;;      #object[com.flickr4java.flickr.Flickr 0x67f9df60 "com.flickr4java.flickr.Flickr@67f9df60"],
  ;;      :auth-store
  ;;      #object[com.flickr4java.flickr.util.FileAuthStore 0x237de92 "com.flickr4java.flickr.util.FileAuthStore@237de92"]}}

  )

In theory, we should now have a Flickr4Java request context with an access token, and that access token stored in our auth store. Let's check out what's in the auth store:

(comment

  (.retrieveAll (get-in ctx [:flickr :auth-store]))
  ;; => [#object[com.flickr4java.flickr.auth.Auth 0x7737c4f2 "com.flickr4java.flickr.auth.Auth@7737c4f2"]]

  )

Ooh, exciting!

If we cast our mind back to that Java code:

Auth auth = this.authStore.retrieve(this.nsid);
if (auth == null) {
    this.authorize();
} else {
    rc.setAuth(auth);
}

we see that the authorize() method is only called if we don't have an access token (the Auth object). If we have one, we just stuff it in the request context and move on with our life. Let's make one final change to our authorise function to do the same thing:

(defn authorise [{:keys [flickr] :as ctx}]
  (let [{:keys [client auth-store req-token token-key]} flickr
        auth-interface (.getAuthInterface client)
        auth (-> auth-store .retrieveAll first)]
    (cond
      ;; We have a valid access token from the auth store
      auth
      (do
        (.setAuth (RequestContext/getRequestContext) auth)
        (update ctx :flickr assoc :auth auth))

      ;; We have a request token and a token key, so exchange the request
      ;; token for an access token
      (and req-token token-key)
      (let [access-token (.getAccessToken auth-interface req-token token-key)
            auth (.checkToken auth-interface access-token)]
        (.setAuth (RequestContext/getRequestContext) auth)
        (.store auth-store auth)
        (update ctx :flickr dissoc :req-token :token-key :url))

      ;; We don't have any tokens, so grab a request token and the URL to
      ;; authorise it
      :default
      (let [req-token (.getRequestToken auth-interface)
            url (.getAuthorizationUrl auth-interface req-token Permission/READ)]
        (update ctx :flickr assoc :url url, :req-token req-token)))))

We can see if it works by creating a brand new client and calling authorise on it:

(comment

  (-> (init-client config) authorise)
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :flickr
  ;;     {:client
  ;;      #object[com.flickr4java.flickr.Flickr 0x15c343a9 "com.flickr4java.flickr.Flickr@15c343a9"],
  ;;      :auth-store
  ;;      #object[com.flickr4java.flickr.util.FileAuthStore 0x6df7251f "com.flickr4java.flickr.util.FileAuthStore@6df7251f"]
  ;;      :auth
  ;;      #object[com.flickr4java.flickr.auth.Auth 0x4d1d0eb0 "com.flickr4java.flickr.auth.Auth@4d1d0eb0"]}}

  )

That calls for a celebration!

A woman on a beach at sunrise with her head thrown back, saying

Given that you only need to go through the rigmarole of exchanging a request token for an auth token once, I feel like it's reasonable to have init-client go ahead and authorise the client. This will work just fine for the rigmarole case as well, since you can just call authorise on the client yourself if it has a :url key set, so there's no harm in doing this.

(defn init-client [{:keys [api-key secret] :as ctx}]
  (let [client (Flickr. api-key secret (REST.))
        auth-store (make-auth-store)]
    (-> ctx
        (assoc :flickr {:client client, :auth-store auth-store})
        authorise)))

OK, so what about those albums again?

Before we got sidetracked by all of this annoying security stuff, the actual goal was to get a list of my photo albums, which we learned are apparently called "photosets" in the Flickr API. Let's have a look at how the example Java code deals with them:

PhotosetsInterface pi = flickr.getPhotosetsInterface();
Iterator sets = pi.getList(this.nsid).getPhotosets().iterator();

The nsid thingy turns out to just be my Flickr user ID, which I can grab from the Auth object. Let's grab an album and see what it's all about!

(comment

  (def album
    (let [ps-interface (.getPhotosetsInterface (get-in ctx [:flickr :client]))
          user-id (.. (get-in ctx [:flickr :auth]) getUser getId)]
      (-> ps-interface (.getList user-id) .getPhotosets first)))
  ;; => #'clickr.flickr/album

  album
  ;; => #object[com.flickr4java.flickr.photosets.Photoset 0x726a87d4 "com.flickr4java.flickr.photosets.Photoset@726a87d4"]

  )

Cool, I guess. Looking at the Flickr API reference for photosets.getList, we can see an example XML response:

<photosets page="1" pages="1" perpage="30" total="2" cancreate="1">
  <photoset id="72157626216528324" primary="5504567858" secret="017804c585" server="5174" farm="6" photos="22" videos="0" count_views="137" count_comments="0" can_comment="1" date_create="1299514498" date_update="1300335009">
    <title>Avis Blanche</title>
    <description>My Grandma's Recipe File.</description>
  </photoset>
  <photoset id="72157624618609504" primary="4847770787" secret="6abd09a292" server="4153" farm="5" photos="43" videos="12" count_views="523" count_comments="1" can_comment="1" date_create="1280530593" date_update="1308091378">
    <title>Mah Kittehs</title>
    <description>Sixty and Niner. Born on the 3rd of May, 2010, or thereabouts. Came to my place on Thursday, July 29, 2010.</description>
  </photoset>
</photosets>

Let's make some assumptions about what getters the Photoset class has and see if we can find out anything interesting about our album:

(comment

  (.getId album)
  ;; => "72177720314024335"

  (.getTitle album)
  ;; => "clickr demo"

  (.getDescription album)
  ;; => "Photo album demo for my clickr blog post"

  )

Nice! Given this, let's write a function that turns a Photoset into a plain old map (or POM—that's what that means, right?):

(defn ->album [photoset]
  {:id (.getId photoset)
   :title (.getTitle photoset)
   :description (.getDescription photoset)
   :object photoset})

(comment

  (->album album)
  ;; => {:id "72177720314024335",
  ;;     :title "clickr demo",
  ;;     :description "Photo album demo for my clickr blog post",
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photosets.Photoset 0x726a87d4 "com.flickr4java.flickr.photosets.Photoset@726a87d4"]}

  )

And now that we have this, why not a function that grabs all of my albums?

(defn get-albums [{:keys [flickr] :as ctx}]
  (let [user-id (.. (:auth flickr) getUser getId)]
    (->> (-> (:client flickr)
             (.getPhotosetsInterface)
             (.getList user-id)
             (.getPhotosets))
         (map ->album))))

(comment

  (->> (get-albums ctx)
       count)
  ;; => 165

  (->> (get-albums ctx)
       (take 2))
  ;; => ({:id "72177720314024335",
  ;;      :title "clickr demo",
  ;;      :description "Photo album demo for my clickr blog post",
  ;;      :object
  ;;      #object[com.flickr4java.flickr.photosets.Photoset 0x9d8b7ae "com.flickr4java.flickr.photosets.Photoset@9d8b7ae"]}
  ;;     {:id "72157706528674711",
  ;;      :title "Kai's 8th Birthday",
  ;;      :description
  ;;      "April 2015. Kaiche's first year at SIS. Treating his classmates to muffins in class. Sushi and cake with mommy and daddy. ",
  ;;      :object
  ;;      #object[com.flickr4java.flickr.photosets.Photoset 0x1bb741fe "com.flickr4java.flickr.photosets.Photoset@1bb741fe"]})

  )

Alright, it looks like I have quite a few albums to deal with here. But an album should contain some photos, right? Let's see if we can figure out how to grab them.

In Java, that apparently looks like this:

Photoset set = (Photoset) sets.next();
PhotoList photos = pi.getPhotos(set.getId(), 500, 1);

Let's try that in Clojure:

(comment

  (def album (-> (get-albums ctx) first))
  ;; => #'clickr.flickr/album

  (def ps-interface (.getPhotosetsInterface (get-in ctx [:flickr :client])))
  ;; => #'clickr.flickr/ps-interface

  (def photos (.getPhotos ps-interface (:id album) 500 1))
  ;; => #'clickr.flickr/photos

  (count photos)
  ;; => 8

  (first photos)
  ;; => #object[com.flickr4java.flickr.photos.Photo 0xd254585 "com.flickr4java.flickr.photos.Photo@14ea992b"]
  )

According to the Flickr API documentation for photosets.getPhotos, the 500 and the 1 are the number of photos to return per page and the page number, both of which are the default values. Seems fine.

The docs are a little sparse on what's in a photo, though:

<photoset id="4" primary="2483" page="1" perpage="500" pages="1" total="2">
  <photo id="2484" secret="123456" server="1" title="my photo" isprimary="0" />
  <photo id="2483" secret="123456" server="1" title="flickr rocks" isprimary="1" />
</photoset>

Let's ask the JVM what getters the photo class has:

  (def photo (first photos))
  ;; => #'clickr.flickr/photo

  (->> photo class .getDeclaredMethods
       (map #(.getName %))
       (filter #(str/starts-with? % "get"))
       sort)
  ;; => ("getBaseImageUrl"
  ;;     "getComments"
  ;;     "getCountry"
  ;;     "getCounty"
  ;;     "getDateAdded"
  ;;     "getDatePosted"
  ;;     "getDateTaken"
  ;;     "getDescription"
  ;;     "getEditability"
  ;;     "getFarm"
  ;;     "getGeoData"
  ;;     "getHdMp4"
  ;;     "getHdMp4Url"
  ;;     "getIconFarm"
  ;;     "getIconServer"
  ;;     "getId"
  ;;     "getImage"
  ;;     "getImageAsStream"
  ;;     "getLarge1600Size"
  ;;     "getLarge1600Url"
  ;;     "getLarge2048Size"
  ;;     "getLarge2048Url"
  ;;     "getLargeAsStream"
  ;;     "getLargeImage"
  ;;     "getLargeSize"
  ;;     "getLargeUrl"
  ;;     "getLastUpdate"
  ;;     "getLicense"
  ;;     "getLocality"
  ;;     "getMedia"
  ;;     "getMediaStatus"
  ;;     "getMedium640Size"
  ;;     "getMedium640Url"
  ;;     "getMedium800Size"
  ;;     "getMedium800Url"
  ;;     "getMediumAsStream"
  ;;     "getMediumImage"
  ;;     "getMediumSize"
  ;;     "getMediumUrl"
  ;;     "getMobileMp4"
  ;;     "getMobileMp4Url"
  ;;     "getNotes"
  ;;     "getOriginalAsStream"
  ;;     "getOriginalBaseImageUrl"
  ;;     "getOriginalFormat"
  ;;     "getOriginalHeight"
  ;;     "getOriginalImage"
  ;;     "getOriginalImage"
  ;;     "getOriginalImageAsStream"
  ;;     "getOriginalSecret"
  ;;     "getOriginalSize"
  ;;     "getOriginalUrl"
  ;;     "getOriginalWidth"
  ;;     "getOwner"
  ;;     "getPathAlias"
  ;;     "getPermissions"
  ;;     "getPhotoUrl"
  ;;     "getPlaceId"
  ;;     "getPublicEditability"
  ;;     "getRegion"
  ;;     "getRotation"
  ;;     "getSecret"
  ;;     "getServer"
  ;;     "getSiteMP4Size"
  ;;     "getSiteMP4Url"
  ;;     "getSizes"
  ;;     "getSmall320Size"
  ;;     "getSmall320Url"
  ;;     "getSmallAsInputStream"
  ;;     "getSmallImage"
  ;;     "getSmallSize"
  ;;     "getSmallSquareAsInputStream"
  ;;     "getSmallSquareImage"
  ;;     "getSmallSquareUrl"
  ;;     "getSmallUrl"
  ;;     "getSquareLargeSize"
  ;;     "getSquareLargeUrl"
  ;;     "getSquareSize"
  ;;     "getStats"
  ;;     "getTags"
  ;;     "getTakenGranularity"
  ;;     "getThumbnailAsInputStream"
  ;;     "getThumbnailImage"
  ;;     "getThumbnailSize"
  ;;     "getThumbnailUrl"
  ;;     "getTitle"
  ;;     "getUrl"
  ;;     "getUrls"
  ;;     "getUsage"
  ;;     "getVideoOriginalSize"
  ;;     "getVideoOriginalUrl"
  ;;     "getVideoPlayerSize"
  ;;     "getVideoPlayerUrl"
  ;;     "getViews")

Let's write a ->photo function to POM-ify this suckah!

(defn ->photo [photo]
  {:id (.getId photo)
   :title (.getTitle photo)
   :description (.getDescription photo)
   :date-taken (.getDateTaken photo)
   :width (.getOriginalWidth photo)
   :height (.getOriginalHeight photo)
   :geo-data (.getGeoData photo)
   :rotation (.getRotation photo)
   :object photo})

(comment

  (->photo photo)
  ;; => {:description nil,
  ;;     :date-taken nil,
  ;;     :geo-data nil,
  ;;     :rotation -1,
  ;;     :width 0,
  ;;     :title "sean-hargreaves-phoenix-new-5-final-a",
  ;;     :id "53460147147",
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photos.Photo 0xd254585 "com.flickr4java.flickr.photos.Photo@14ea992b"],
  ;;     :height 0}

  )

And now that we have this, let's add the photos to the the album map:

(defn ->album [{:keys [flickr] :as ctx} photoset]
  (let [ps-interface (.getPhotosetsInterface (:client flickr))]
    {:id (.getId photoset)
     :title (.getTitle photoset)
     :description (.getDescription photoset)
     :photos (->> (.getPhotos ps-interface (:id album) 500 1)
                  (map (partial ->photo ctx)))
     :object photoset}))

Note that in order to list the photos in an album, we need the PhotosetInterface, which we get from the Flickr client in the context, so we need to add the context as an argument to ->album. In fact, let's make it a convention that ctx is always the first argument to functions in this namespace, and add it to ->photo as well:

(defn ->photo [_ctx photo]
  {:id (.getId photo)
   :title (.getTitle photo)
   :description (.getDescription photo)
   :date-taken (.getDateTaken photo)
   :width (.getOriginalWidth photo)
   :height (.getOriginalHeight photo)
   :geo-data (.getGeoData photo)
   :rotation (.getRotation photo)
   :object photo})

We don't actually need it in this function, so we'll prepend an underscore to the arg name so CIDER or LSP or clj-kondo or whatever tooling you're using won't yell at us that it's unused. This is a convention that I first discovered in Erlang, and I think it's a cool way to say that "this is the context, which we don't need here (yet)".

Finally, we need to pass the context along in get-albums as well:

(defn get-albums [{:keys [flickr] :as ctx}]
  (let [user-id (.. (:auth flickr) getUser getId)]
    (->> (-> (:client flickr)
             (.getPhotosetsInterface)
             (.getList user-id)
             (.getPhotosets))
         (map (partial ->album ctx)))))

Making off with the goods

Now that we have a way to list albums and the photos within them, let's see how to download said photos. Reading on in Backup.java, I encounter this good stuff:

Photo p = (Photo) setIterator.next();
String url = p.getLargeUrl();
URL u = new URL(url);
String filename = u.getFile();
filename = filename.substring(filename.lastIndexOf("/") + 1, filename.length());
System.out.println("Now writing " + filename + " to " + setDirectory.getCanonicalPath());
BufferedInputStream inStream = new BufferedInputStream(photoInt.getImageAsStream(p, Size.LARGE));
File newFile = new File(setDirectory, filename);

FileOutputStream fos = new FileOutputStream(newFile);

int read;

while ((read = inStream.read()) != -1) {
    fos.write(read);
}
fos.flush();
fos.close();
inStream.close();

That looks like one way to do it. 😬

Let's see if we can clean this up a little. This whole mess:

while ((read = inStream.read()) != -1) {
    fos.write(read);
}

can be replaced with clojure.java.io/copy, so let's update the ns form to require in clojure.java.io, and import the other Java classes mentioned in this code snippet whilst we're at it:

(ns clickr.flickr
  (:require [babashka.fs :as fs]
            [clojure.java.io :as io])
  (:import (com.flickr4java.flickr Flickr
                                   RequestContext
                                   REST)
           (com.flickr4java.flickr.auth Permission)
           (com.flickr4java.flickr.photos Size)
           (com.flickr4java.flickr.util FileAuthStore)
           (java.io BufferedInputStream
                    FileOutputStream)))

I also find this stuff to get the filename of the photo to be a bit annoying:

String url = p.getLargeUrl();
URL u = new URL(url);
String filename = u.getFile();
filename = filename.substring(filename.lastIndexOf("/") + 1, filename.length());

Let's make the filename the photo's ID plus its format. We can do this in our ->photo function like so:

(defn ->photo [_ photo]
  (let [id (.getId photo)
        extension (.getOriginalFormat photo)
        filename (format "%s.%s" id extension)]
    {:id id
     :filename filename
     :title (.getTitle photo)
     :description (.getDescription photo)
     :date-taken (.getDateTaken photo)
     :width (.getOriginalWidth photo)
     :height (.getOriginalHeight photo)
     :geo-data (.getGeoData photo)
     :rotation (.getRotation photo)
     :object photo}))

Now we can write a function to download a photo nicely. To be good citizens, let's put the file in the tmp directory (/tmp on Linux and MacOS, who knows where on Windows). Luckily for us, babashka.fs has a handy temp-dir function that can do this! 🎉

(defn download-photo! [{:keys [flickr] :as ctx}
                       {:keys [filename] :as photo}]
  (let [p-interface (.getPhotosInterface (:client flickr))]
    (with-open [in (BufferedInputStream. (.getImageAsStream p-interface (:object photo) Size/LARGE))
                out (FileOutputStream. (fs/file (fs/temp-dir) filename))]
      (io/copy in out))))

Now let's try calling it and seeing if it works!

(comment

  (def album (-> (get-albums ctx) first))
  ;; => #'clickr.flickr/album

  (def photo (-> album :photos first))
  ;; => #'clickr.flickr/photo

  photo
  ;; => {:description nil,
  ;;     :date-taken nil,
  ;;     :geo-data nil,
  ;;     :rotation -1,
  ;;     :width 0,
  ;;     :title "sean-hargreaves-phoenix-new-5-final-a",
  ;;     :filename "53460147147.jpg",
  ;;     :id "53460147147",
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photos.Photo 0x5d288375 "com.flickr4java.flickr.photos.Photo@14ea992b"],
  ;;     :height 0}

  (download-photo! ctx photo)
  ;; => nil

  (fs/exists? (fs/file (fs/temp-dir) (:filename photo)))
  ;; => true

  )

Sure enough, we now have a /tmp/53460147147.jpg file!

A space station

Backing things up

Now that we've proven that we can download a photo, let's think about how we want the backup process to work. What we probably want is to create a "folder" in our S3 bucket for each album, and then put all of the photos for that album inside it. Let's write a function to download an entire album.

The first order of business is to create a directory to hold the photos in the album that we're about to download. Let's use the album ID as the name of the directory and drop it in the tmp directory:

(comment

  (fs/file (fs/temp-dir) (:id album))
  ;; => #object[java.io.File 0x4e8e72c2 "/tmp/72177720314024335"]

  )

We can now use create-dirs to create the directory:


(comment

  (->> (fs/file (fs/temp-dir) (:id album)) fs/create-dirs)
  ;; => #object[sun.nio.fs.UnixPath 0x1fbb7d68 "/tmp/72177720314024335"]

  )

OK, so we have a directory to hold the photos, but we're going to have to update download-photo! to use that directory instead of dropping stuff straight into /tmp. Easy enough:

(defn download-photo! [{:keys [flickr out-dir] :as ctx}
                       {:keys [filename] :as photo}]
  (let [p-interface (.getPhotosInterface (:client flickr))]
    (with-open [in (BufferedInputStream. (.getImageAsStream p-interface (:object photo) Size/LARGE))
                out (FileOutputStream. (fs/file out-dir filename))]
      (io/copy in out))))

(comment

  (def album-dir (->> (fs/file (fs/temp-dir) (:id album)) fs/create-dirs))
  ;; => #'clickr.flickr/album-dir

  (download-photo! (assoc ctx :out-dir album-dir) photo)
  ;; => nil

  (fs/exists? (fs/file album-dir (format "%s.jpg" (:id photo))))
  ;; => true

  )

Having thus tamed download-photo!, we can write download-album!:

(defn download-album! [ctx {:keys [id photos] :as album}]
  (let [album-dir (->> id (fs/file (fs/temp-dir)) fs/create-dirs fs/file)]
    (->> photos
         (map (partial download-photo! (assoc ctx :out-dir album-dir)))
         doall)))

(comment

  (download-album! ctx album)
  ;; => (nil nil nil nil nil nil nil nil)

  (fs/glob album-dir "*")
  ;; => [#object[sun.nio.fs.UnixPath 0x72466b2f "/tmp/72177720314024335/53461405604.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x788d40e5 "/tmp/72177720314024335/53460163402.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x26f7ff12 "/tmp/72177720314024335/53460161007.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x250e8e92 "/tmp/72177720314024335/53460147147.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x17fa6f15 "/tmp/72177720314024335/53461214223.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x5e28f67 "/tmp/72177720314024335/53461091151.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x1cf4ffa6 "/tmp/72177720314024335/53460151727.jpg"]
  ;;     #object[sun.nio.fs.UnixPath 0x4ddf88b4 "/tmp/72177720314024335/53461088046.jpg"]]

  )

Fantastic! Though I have to admit that I don't find the list of nils very friendly. Let's make one last change to download-photo so that it returns the photo, with the location of the downloaded file added to it:

(defn download-photo! [{:keys [flickr out-dir] :as ctx}
                       {:keys [filename] :as photo}]
  (let [p-interface (.getPhotosInterface (:client flickr))
        out-file (fs/file out-dir filename)]
    (with-open [in (BufferedInputStream. (.getImageAsStream p-interface (:object photo) Size/LARGE))
                out (FileOutputStream. out-file)]
      (io/copy in out))
    (assoc photo :out-file out-file)))

(comment

  (download-photo! (assoc ctx :out-dir album-dir) photo)
  ;; => {:description nil,
  ;;     :date-taken nil,
  ;;     :geo-data nil,
  ;;     :rotation -1,
  ;;     :width 0,
  ;;     :title "sean-hargreaves-phoenix-new-5-final-a",
  ;;     :filename "53460147147.jpg",
  ;;     :id "53460147147",
  ;;     :out-file
  ;;     #object[java.io.File 0x44f5abfa "/tmp/72177720314024335/53460147147.jpg"],
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photos.Photo 0x4d361782 "com.flickr4java.flickr.photos.Photo@14ea992b"],
  ;;     :height 0}

  )

Much nicer! We can now do the same thing to download-album!:

(defn download-album! [ctx {:keys [id photos] :as album}]
  (let [album-dir (->> id (fs/file (fs/temp-dir)) fs/create-dirs fs/file)
        photos (->> photos
                    (map (partial download-photo! (assoc ctx :out-dir album-dir)))
                    doall)]
    (assoc album :out-dir album-dir, :photos photos)))

(comment

  (download-album! ctx album)
  ;; => {:id "72177720314024335",
  ;;     :title "clickr demo",
  ;;     :description "Photo album demo for my clickr blog post",
  ;;     :photos
  ;;     ({:description nil,
  ;;       :date-taken nil,
  ;;       :geo-data nil,
  ;;       :rotation -1,
  ;;       :width 0,
  ;;       :title "sean-hargreaves-phoenix-new-5-final-a",
  ;;       :filename "53460147147.jpg",
  ;;       :id "53460147147",
  ;;       :out-file
  ;;       #object[java.io.File 0x163fa069 "/tmp/72177720314024335/53460147147.jpg"],
  ;;       :object
  ;;       #object[com.flickr4java.flickr.photos.Photo 0x4d361782 "com.flickr4java.flickr.photos.Photo@14ea992b"],
  ;;       :height 0}
  ;; [...]
  ;;      {:description nil,
  ;;       :date-taken nil,
  ;;       :geo-data nil,
  ;;       :rotation -1,
  ;;       :width 0,
  ;;       :title "daniel-jennings-img-7554",
  ;;       :filename "53460151727.jpg",
  ;;       :id "53460151727",
  ;;       :out-file
  ;;       #object[java.io.File 0x289e2344 "/tmp/72177720314024335/53460151727.jpg"],
  ;;       :object
  ;;       #object[com.flickr4java.flickr.photos.Photo 0x805e360 "com.flickr4java.flickr.photos.Photo@436e36e8"],
  ;;       :height 0}),
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photosets.Photoset 0x19d70721 "com.flickr4java.flickr.photosets.Photoset@19d70721"],
  ;;     :out-dir #object[java.io.File 0x67b12ec4 "/tmp/72177720314024335"]}

  )

Uploading to S3

We're most of the way there now. The final piece of the puzzle is taking our lovely directory full of downloaded files and putting it on S3, as described above. To accomplish this, let's avail ourselves of the Cognitect aws-api library that has served us so well in the past. First, let's add it to deps.edn:

{:paths ["src" "dev"]
 :deps {babashka/fs {:mvn/version "0.4.19"}
        com.cognitect.aws/api {:mvn/version "0.8.686"}
        com.cognitect.aws/endpoints {:mvn/version "1.1.12.504"}
        com.cognitect.aws/s3 {:mvn/version "848.2.1413.0"}
        com.flickr4java/flickr4java {:mvn/version "3.0.1"}}}

Sadly, we'll now need to restart our REPL. I really need to get hot reloading working! Sounds like a project for another day, though I'm sure it will be incredibly simple...

In any case, having now restarted our REPL, let's create a clickr.s3 namespace for ourselves:

(ns clickr.s3
  (:require [cognitect.aws.client.api :as aws]
            [clojure.string :as str]
            [clojure.java.io :as io])
  (:import (java.io ByteArrayInputStream)))

Now we'll need an S3 client. Let's follow the same pattern we used for our Flickr client:

(defn init-client [{:keys [aws-region] :as ctx}]
  (let [client (aws/client {:api :s3, :region aws-region})]
    (assoc ctx :s3 {:client client})))

Now we can actually use the same config as we used before! 🤯 We just need to add the AWS region in there, then we can call init-client:


(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"
               :aws-region "eu-west-1"})
  ;; => #'clickr.s3/config

  (def ctx (init-client config))
  ;; => #'clickr.s3/ctx

  (-> ctx :s3 :client)
  ;; => #object[cognitect.aws.client.impl.Client 0x6b5a7704 "cognitect.aws.client.impl.Client@6b5a7704"]

  )

With a working client in hand, we can write a function to upload a photo. Let's follow the same pattern as download-photo!: do the side-effecting thing and then return the photo, assoc-ing in the S3 key where we uploaded it.

(defn upload-photo! [{:keys [s3 s3-bucket] :as ctx}
                     {:keys [out-file] :as photo}]
  (let [s3-key :???]
    (aws/invoke (:client s3)
                {:op :PutObject
                 :request {:Bucket s3-bucket
                           :Key s3-key
                           :Body :???}})
    (assoc photo :s3-key s3-key)))

OK, a couple things here. First, we need to add the S3 bucket to our context. That's easy enough:


(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"
               :aws-region "eu-west-1"
               :s3-bucket "photos.jmglov.net"})
  ;; => #'clickr.s3/config

  (def ctx (init-client config))
  ;; => #'clickr.s3/ctx

  ctx
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :aws-region "eu-west-1",
  ;;     :s3-bucket "photos.jmglov.net",
  ;;     :s3
  ;;     {:client
  ;;      #object[cognitect.aws.client.impl.Client 0xcd319cb "cognitect.aws.client.impl.Client@cd319cb"]}}

  )

Next, we need a way to read in the photo file. babashka.fs comes to the rescue again with a function called read-all-bytes! Let's require in babashka.fs:

(ns clickr.s3
  (:require [babashka.fs :as fs]
            [cognitect.aws.client.api :as aws]))

And now we can use that in our upload-photo! function:

(defn upload-photo! [{:keys [s3 s3-bucket] :as ctx}
                     {:keys [out-file] :as photo}]
  (let [s3-key :???]
    (aws/invoke (:client s3)
                {:op :PutObject
                 :request {:Bucket s3-bucket
                           :Key s3-key
                           :Body (fs/read-all-bytes out-file)}})
    (assoc photo :s3-key s3-key)))

Finally, we need to figure out what the S3 key should be. Using the convention from download-album!, we know the photo will be in a directory corresponding to the album ID. Let's use that as the key and prepend clickr/ to it so that we don't pollute the top-level of the S3 bucket with a bunch of nonsense. We can grab the directory and filename using our old friend babashka.fs, of course:

(comment

  (def photo {:out-file "/tmp/72177720314024335/53460151727.jpg"})
  ;; => #'clickr.s3/photo

  (-> photo :out-file fs/file-name)
  ;; => "53460151727.jpg"

  (-> photo :out-file fs/parent fs/file-name)
  ;; => "72177720314024335"

  )

Cool! With this, we have enough to construct the S3 key:

(defn upload-photo! [{:keys [s3 s3-bucket] :as ctx}
                     {:keys [out-file] :as photo}]
  (let [s3-key (format "%s/%s/%s"
                       "clickr"
                       (-> photo :out-file fs/parent fs/file-name)
                       (-> photo :out-file fs/file-name))]
    (aws/invoke (:client s3)
                {:op :PutObject
                 :request {:Bucket s3-bucket
                           :Key s3-key
                           :Body (fs/read-all-bytes out-file)}})
    (assoc photo :s3-key s3-key)))

We can try this out:

(comment

  (upload-photo! ctx photo)
  ;; => {:out-file "/tmp/72177720314024335/53460151727.jpg",
  ;;     :s3-key "clickr/72177720314024335/53460151727.jpg"}

  )

And one should always trust but verify, right?

: jmglov@laurana; aws s3 ls s3://photos.jmglov.net/clickr/72177720314024335/53460151727.jpg
2024-01-17 12:06:17      90743 53460151727.jpg

Wow, that was surprisingly painless!

One thing that's bothering me a little, though, is that hardcoded "clickr" in there. Let's move that to the config like we did with the S3 bucket:

(defn upload-photo! [{:keys [s3 s3-bucket s3-prefix] :as ctx}
                     {:keys [out-file] :as photo}]
  (let [s3-key (format "%s/%s/%s"
                       s3-prefix
                       (-> photo :out-file fs/parent fs/file-name)
                       (-> photo :out-file fs/file-name))]
    (aws/invoke (:client s3)
                {:op :PutObject
                 :request {:Bucket s3-bucket
                           :Key s3-key
                           :Body (fs/read-all-bytes out-file)}})
    (assoc photo :s3-key s3-key)))

(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"
               :aws-region "eu-west-1"
               :s3-bucket "photos.jmglov.net"
               :s3-prefix "clickr"})
  ;; => #'clickr.s3/config

  (def ctx (init-client config))
  ;; => #'clickr.s3/ctx

  ctx
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :aws-region "eu-west-1",
  ;;     :s3-bucket "photos.jmglov.net",
  ;;     :s3-prefix "clickr",
  ;;     :s3
  ;;     {:client
  ;;      #object[cognitect.aws.client.impl.Client 0x2dc1f1c2 "cognitect.aws.client.impl.Client@2dc1f1c2"]}}

  (upload-photo! ctx photo)
  ;; => {:out-file "/tmp/72177720314024335/53460151727.jpg",
  ;;     :s3-key "clickr/72177720314024335/53460151727.jpg"}

  )

Having done this, writing a function to upload all the photos in an album is quite straightforward:

(defn upload-album! [ctx {:keys [photos] :as album}]
  (update album :photos #(doall (map (partial upload-photo! ctx) %))))

(comment

  (def album-dir "/tmp/72177720314024335")
  ;; => #'clickr.s3/album-dir

  (def album {:id "72177720314024335"
              :out-dir album-dir
              :photos (->> (fs/glob album-dir "*")
                           (map (fn [out-file] {:out-file (str out-file)})))})
  ;; => #'clickr.s3/album

  (upload-album! ctx album)
  ;; => {:id "72177720314024335",
  ;;     :out-dir "/tmp/72177720314024335",
  ;;     :photos
  ;;     ({:out-file "/tmp/72177720314024335/53461405604.jpg",
  ;;       :s3-key "clickr/72177720314024335/53461405604.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53460163402.jpg",
  ;;       :s3-key "clickr/72177720314024335/53460163402.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53460161007.jpg",
  ;;       :s3-key "clickr/72177720314024335/53460161007.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53460147147.jpg",
  ;;       :s3-key "clickr/72177720314024335/53460147147.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53461214223.jpg",
  ;;       :s3-key "clickr/72177720314024335/53461214223.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53461091151.jpg",
  ;;       :s3-key "clickr/72177720314024335/53461091151.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53460151727.jpg",
  ;;       :s3-key "clickr/72177720314024335/53460151727.jpg"}
  ;;      {:out-file "/tmp/72177720314024335/53461088046.jpg",
  ;;       :s3-key "clickr/72177720314024335/53461088046.jpg"})}

  )

Tying it all together

All of this looks reasonably reasonable, but our REPL-driven development of the clickr.s3 namespace was pretty mocktacular, which fills me with a vague sense of unease. Let's make sure we can upload an honest to goodness album!

To make sure we don't have any cruft lying around in our REPL, let's create a new namespace and require in the flickr and s3 stuff:

(ns user
  (:require [clickr.flickr :as flickr]
            [clickr.s3 :as s3]))

We'll need some config, which we can just copy and paste straight from our experiments in the s3 namespace:

(comment

  (def config {:api-key "beefface5678910"
               :secret "facecafe1234"
               :aws-region "eu-west-1"
               :s3-bucket "photos.jmglov.net"
               :s3-prefix "clickr"})
  ;; => #'user/config

  )

With this config firmly in hand, we can create flickr and s3 clients:

(comment

  (def ctx (-> config flickr/init-client s3/init-client))
  ;; => #'user/ctx

  ctx
  ;; => {:api-key "beefface5678910",
  ;;     :secret "facecafe1234",
  ;;     :aws-region "eu-west-1",
  ;;     :s3-bucket "photos.jmglov.net",
  ;;     :s3-prefix "clickr",
  ;;     :flickr
  ;;     {:client
  ;;      #object[com.flickr4java.flickr.Flickr 0x2c4181a2 "com.flickr4java.flickr.Flickr@2c4181a2"],
  ;;      :auth-store
  ;;      #object[com.flickr4java.flickr.util.FileAuthStore 0x5368861c "com.flickr4java.flickr.util.FileAuthStore@5368861c"],
  ;;      :auth
  ;;      #object[com.flickr4java.flickr.auth.Auth 0x7c8ff0d6 "com.flickr4java.flickr.auth.Auth@7c8ff0d6"]},
  ;;     :s3
  ;;     {:client
  ;;      #object[cognitect.aws.client.impl.Client 0x5b35646d "cognitect.aws.client.impl.Client@5b35646d"]}}

  )

Download an album:

(comment

  (def album (->> (flickr/get-albums' ctx)
                  first
                  (flickr/download-album!' ctx)))
  ;; => #'user/album

  (->> album :photos (map :out-file))
  ;; => (#object[java.io.File 0x1c2cd5ae "/tmp/72177720314024335/53460147147.jpg"]
  ;;     #object[java.io.File 0x1678a01 "/tmp/72177720314024335/53461405604.jpg"]
  ;;     #object[java.io.File 0x4f708220 "/tmp/72177720314024335/53461091151.jpg"]
  ;;     #object[java.io.File 0x75723234 "/tmp/72177720314024335/53461088046.jpg"]
  ;;     #object[java.io.File 0x6cace2fd "/tmp/72177720314024335/53460163402.jpg"]
  ;;     #object[java.io.File 0x51265aeb "/tmp/72177720314024335/53460161007.jpg"]
  ;;     #object[java.io.File 0x1a0ca36d "/tmp/72177720314024335/53461214223.jpg"]
  ;;     #object[java.io.File 0x7e8bc18a "/tmp/72177720314024335/53460151727.jpg"])

  )

And upload it to S3!

(comment

  (s3/upload-album! ctx album)
  ;; => {:id "72177720314024335",
  ;;     :title "clickr demo",
  ;;     :description "Photo album demo for my clickr blog post",
  ;;     :photos
  ;;     ({:description nil,
  ;;       :date-taken nil,
  ;;       :geo-data nil,
  ;;       :rotation -1,
  ;;       :width 0,
  ;;       :title "sean-hargreaves-phoenix-new-5-final-a",
  ;;       :filename "53460147147.jpg",
  ;;       :id "53460147147",
  ;;       :s3-key "clickr/72177720314024335/53460147147.jpg",
  ;;       :out-file
  ;;       #object[java.io.File 0x1c2cd5ae "/tmp/72177720314024335/53460147147.jpg"],
  ;;       :object
  ;;       #object[com.flickr4java.flickr.photos.Photo 0x3bbff9e1 "com.flickr4java.flickr.photos.Photo@14ea992b"],
  ;;       :height 0}
  ;; [...]
  ;;      {:description nil,
  ;;       :date-taken nil,
  ;;       :geo-data nil,
  ;;       :rotation -1,
  ;;       :width 0,
  ;;       :title "daniel-jennings-img-7554",
  ;;       :filename "53460151727.jpg",
  ;;       :id "53460151727",
  ;;       :s3-key "clickr/72177720314024335/53460151727.jpg",
  ;;       :out-file
  ;;       #object[java.io.File 0x7e8bc18a "/tmp/72177720314024335/53460151727.jpg"],
  ;;       :object
  ;;       #object[com.flickr4java.flickr.photos.Photo 0x1e20bd4c "com.flickr4java.flickr.photos.Photo@436e36e8"],
  ;;       :height 0}),
  ;;     :object
  ;;     #object[com.flickr4java.flickr.photosets.Photoset 0x6411aa2d "com.flickr4java.flickr.photosets.Photoset@6411aa2d"],
  ;;     :out-dir #object[java.io.File 0x377eb990 "/tmp/72177720314024335"]}

  )

And check that all is right with the world:

: jmglov@laurana; aws s3 ls s3://photos.jmglov.net/clickr/72177720314024335/
2024-01-17 12:48:26     125643 53460147147.jpg
2024-01-17 12:48:29      90743 53460151727.jpg
2024-01-17 12:48:28      88417 53460161007.jpg
2024-01-17 12:48:28     185725 53460163402.jpg
2024-01-17 12:48:27     178392 53461088046.jpg
2024-01-17 12:48:27     106074 53461091151.jpg
2024-01-17 12:48:29      88013 53461214223.jpg
2024-01-17 12:48:26      98035 53461405604.jpg

A woman on a beach at sunrise with her head thrown back, saying

But wait just a second here...

The unbearable lightness of being dissatisfied

A person sitting alone, holding their face in their hands

Much like Angelica Schuyler, I'll never be satisfied, because having to browse my albums with the S3 console is kind of a let down, not to mention that this:

The Flickr site, displaying the photos in the clickr demo album

looks way better than this:

The AWS S3 console, displaying the files we uploaded as a table of text

What is a young man to do? Well, you'll have to stick around for the next instalment of this exciting series, which I promise I'll actually write, because I've started writing it already and... um... trust me?

Part 2: clickr goes frontend

🏷 clojure aws clickr clonejure
📝 Published: 2024-01-17
📣 Discuss this post here
Awno! Here we go!
clickr goes frontend