Page pre-cache proposal

[mingard]

Overview

To increase page load, and to reduce load-spikes on web and api during peak period, the previously discussed pre-caching pages concept could be achieved by the following method.

• Reduce parameters
• Evaluate datasource requestParams
• Fetch datasource results
• Filter results based on requestParams and routes
• Pre-cache updated pages
• Refresh cache on non-updated pages

Reduce page parameters

For each page, evaluate the page routes and reduce to a list of parameters

{
 "path": "/news/:category/:title?"
}

Results:

[{
  "key": "category",
  "optional": false
},
{
  "key": "title",
  "optional": true
}]

Evaluate datasource requestParams

"requestParams": [
  {
    "param": "category",
    "field": "categoryHandle"
  },
  {
    "param": "title",
    "field": "handle"
  }
]

Fetch datasource results

Fetch only updatedAt, and filterable field from all datasources attached to all pages. In the example given, this would be categoryHandle, title and of course, updatedAt.

Filter results based on requestParams and routes

Taking all possible combinations of required parameters, e.g. category (categoryHandle) to create a list of page queries, noting that some documents can appear under multiple queries. In this example, the article Article about sport exists under sports and all.

/news/sports/article-about-sport
/news/all/article-about-sport

Pre-cache updated pages

For those page queries that have one or more documents with a updatedAt later than the last time the page was cached, re-cache the page

Refresh cache on non-updated pages

For those page queries that have no documents with an updatedAt later than the last time the page was cached, extend the cache

I may have missed something here, feedback appreciated.