(Updated: )
/ #node #serverless #javascript 

Super-powered newsletter content with Pocket and Netlify Lambda

An example Netlify Lambda to fetch all “newsletter” posts from Pocket.

Pocket is an application and web service for managing a reading list of articles from the Internet. It’s quite widely used and tightly integrated into the Firefox browser. I find that I use it extensively to save articles (usually about development).

For the Enterprise Node.js and JavaScript newsletter I have a “From the Web” section, which is populated from links from the web. Since most links that end up there are at one point or another stored on Pocket, I built a lambda that fetches posts tagged with “newsletter” from Pocket. I then consume that lambda from a Hugo newsletter file generator.

See the lambda code at src/lambda/newsletter.js in the repository github.com/HugoDF/pocket-newsletter-lambda.

Run it at https://pocket-newsletter-lambda.netlify.com/, or even deploy your own on Netlify.

The application is styled using TailwindCSS, you can see a starter project for that at github.com/HugoDF/netlify-lambda-tailwind-static-starter.

Table of Contents

Pocket Fetching logic

The bulk of the Pocket-specific logic is the fetchBookmarks function, it does the following:

  • fetch from Pocket API using consumer key and access token
    • passes state: 'all' in order to get both archived and unarchived posts
    • uses tag: 'newsletter' to fetch only posts tagged with newsletter
    • detailType: 'complete' means the API returns more complete data
  • convert the response to a flat list of { title, url, excerpts, authors } (all of those fields are strings)
  • return the list

See the code (full source at github.com/HugoDF/pocket-newsletter-lambda).

async function fetchBookmarks(consumerKey, accessToken) {
  const res = await axios.post('https://getpocket.com/v3/get', {
    consumer_key: consumerKey,
    access_token: accessToken,
    tag: 'newsletter',
    state: 'all',
    detailType: 'complete'
  });
  const {list} = res.data;
  // List is a key-value timestamp->entry map
  const entries = Object.values(list);
  return entries.map(
    ({
      given_title,
      given_url,
      resolved_url,
      resolved_title,
      excerpt,
      authors,
      rest
    }) => ({
      ...rest,
      title: given_title || resolved_title,
      url: given_url || resolved_url,
      excerpt,
      authors: authors
        ? Object.values(authors)
            .map(({name}) => name)
            .filter(Boolean)
            .join(',')
        : ''
    })
  );
}

Netlify Lambda request validation and body-parsing

HTTP verb and payload presence validation

The lambda only supports POST with a body, hence:

if (event.httpMethod !== 'POST') {
  return {
    statusCode: 404,
    body: 'Not Found'
  };
}

if (!event.body) {
  return {
    statusCode: 400,
    body: 'Bad Request'
  };
}

Parsing POST request bodies from form submissions and AJAX/JSON requests

We support both URL-encoded form POST requests (done eg. when JS is disabled on the demo page) and JSON requests.

The body arrives either base64 encoded (if using a URL-encoded form body request) or not. This is denoted by the isBase64Encoded flag on the event.

Parsing a base64-encoded string in Node is done using Buffer.from(event.body, 'base64').toString('utf-8).

To convert the body from URL-encoded form into an object, the following function is used, which works for POSTs with simple fields.

function parseUrlEncoded(urlEncodedString) {
  const keyValuePairs = urlEncodedString.split('&');
  return keyValuePairs.reduce((acc, kvPairString) => {
    const [k, v] = kvPairString.split('=');
    acc[k] = v;
    return acc;
  }, {});
}

Here’s the functionality in the lambda:

const {
    pocket_consumer_key: pocketConsumerKey,
    pocket_access_token: pocketAccessToken
  } = event.isBase64Encoded
    ? parseUrlEncoded(Buffer.from(event.body, 'base64').toString('utf-8'))
    : JSON.parse(event.body);

If the consumer key or access token are missing we send a 400:

if (!pocketConsumerKey || !pocketAccessToken) {
  return {
    statusCode: 400,
    body: 'Bad Request'
  };
}

Sending appropriate responses on failure

We attempt to fetchBookmarks, this functionality has been broken down in “Pocket Fetching logic”.

When the Pocket API fails on a request error we want to send back that response’s information to the client. If the failure can’t be identified, 500. When fetchBookmarks succeeds send a 200 with data.

Thankfully when axios fails, it has a response property on the error. This means that our “Proxy back Pocket API errors” use-case and the other 2 cases are easily fulfilled with:

try {
  const bookmarks = await fetchBookmarks(pocketConsumerKey, pocketAccessToken);

  return {
    statusCode: 200,
    body: JSON.stringify(bookmarks)
  };
} catch(e) {
  if (e.response) {
    return {
      statusCode: e.response.statusCode,
      body: `Error while connecting to Pocket API: ${e.response.statusText}`
    }
  }
  return {
    statusCode: 500,
    body: e.message
  }
}

Sample response

See github.com/HugoDF/pocket-newsletter-lambda#sample-response or try out the application yourself at pocket-newsletter-lambda.netlify.com/.

A Pocket-driven newsletter in the real-world

On codewithhugo.com, the lambda doesn’t read the access token and consumer key from the request. Instead it’s a GET endpoints that reads the token and key from environment variables.

It’s actually simpler to do that. We set POCKET_CONSUMER_KEY and POCKET_ACCESS_TOKEN in the Netlify build configuration. Then update the lambda to the following:

const {POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKEN} = process.env;

// keep fetchBookmarks as is

export async function handler(event) {
  if (event.httpMethod !== 'GET') {
    return {
      statusCode: 404,
      body: 'Not Found'
    };
  }

  const bookmarks = await fetchBookmarks(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKEN);
  return {
    statusCode: 200,
    body: JSON.stringify(bookmarks)
  };
}

codewithhugo.com runs on Hugo. To generate a new newsletter file, I leverage Hugo custom archetypes (ie. a content type that has a template to generate it).

The archetype looks something like the following:

{{- $title := replace (replaceRE `[0-9]{4}-[0-9]{2}-[0-9]{2}-` "" .Name) "-" " " | title -}}
---
title: {{ $title }} - Code with Hugo
---

{{- $newsletterBookmarks := getJSON "https://codewithhugo.com/.netlify/functions/newsletter" }}
{{ range $newsletterBookmarks }}
[{{ .title }}]({{.url}}) by {{ .authors }}: {{ .excerpt }}
{{ end }}

Once the newsletter has been generated and edited, it gets added to buttondown.email, you can see the result of this approach at the Enterprise Node.js and JavaScript Archives.

unsplash-logoMr Cup / Fabien Barral

Author

Hugo Di Francesco

Co-author of "Professional JavaScript", "Front-End Development Projects with Vue.js" with Packt, "The Jest Handbook" (self-published). Hugo runs the Code with Hugo website helping over 100,000 developers every month and holds an MEng in Mathematical Computation from University College London (UCL). He has used JavaScript extensively to create scalable and performant platforms at companies such as Canon, Elsevier and (currently) Eurostar.

Get The Jest Handbook (100 pages)

Take your JavaScript testing to the next level by learning the ins and outs of Jest, the top JavaScript testing library.