DEV Community: Lilou Artz

Auto Indexing of URLs via Google API

Lilou Artz — Tue, 27 Aug 2024 08:15:30 +0000

Pillser aggregates information about 18,000+ supplements from multiple sources. One of the challenges is to ensure that up to date information is surfaced to users when they search for supplements using Google. In this post, I will show you how I use Google APIs to index content immediately after it is updated.

Google

I am really not sure why this was so hard to figure out (I could not find any code examples), but Google APIs Node.js Client provides a way to interact with the Indexing API and Indexing API allows web developers to notify Google about state changes in the URLs they own.

Here is the code that I used to submit URLs using the Indexing API:

import { config } from '#app/config.server';
import { google } from 'googleapis';

const submitToGoogleSearchConsole = async (url: string) => {
  const auth = new google.auth.JWT(
    config.GOOGLE_CLOUD_CLIENT_EMAIL,
    undefined,
    config.GOOGLE_CLOUD_PRIVATE_KEY,
    ['https://www.googleapis.com/auth/indexing'],
    undefined,
  );

  await auth.authorize();

  const indexing = google.indexing({
    auth,
    version: 'v3',
  });

  await indexing.urlNotifications.publish({
    requestBody: {
      type: 'URL_UPDATED',
      url,
    },
  });
};

I call this function whenever I update a supplement product or when someone asks a public question about supplements.

Acquiring Credentials

The code provided above uses a service account for authentication with the Google API. You will need to acquire credentials for the service account. Here is the documentation.

Once you have the service account, you will need to add the email address of the service account to the Google Search Console as an owner of the site.

What can be indexed?

Google Indexing API documentation has a note about what can be indexed:

The Indexing API allows any site owner to directly notify Google when pages are added or removed. This allows Google to schedule pages for a fresh crawl, which can lead to higher quality user traffic. Currently, the Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.

I only saw this note after I had already implemented the Indexing API. However, despite the note, I discovered that the content I submitted to the Indexing API was indexed by Google almost immediately, i.e. it appears like the API can be used to index any content despite the note.

Bonus: IndexNow

I am also calling IndexNow API. Here is what it is:

IndexNow is an easy way for websites owners to instantly inform search engines about latest content changes on their website. In its simplest form, IndexNow is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.

Submitting your URLs to IndexNow is going to inform search engines like Microsoft Bing, Naver, Seznam.cz, Yandex, and Yep about the latest content changes on your website. However, it is not going to inform Google Search. Therefore, you still need to use the Indexing API to notify Google about the changes.

import { httpClient } from '#app/services/httpClient.server';

// generate a key at https://www.bing.com/indexnow/getstarted
// A ${key}.txt file is also placed in the public/ folder.
const key = '...';

export const submitUrlsToIndexNow = async (urls: string[]) => {
  await httpClient('https://api.indexnow.org/IndexNow', {
    headers: {
      'Content-Type': 'application/json',
    },
    json: {
      host: 'pillser.com',
      key,
      keyLocation: `https://pillser.com/${key}.txt`,
      urlList: urls,
    },
    method: 'POST',
  });
};

Tracking Indexing Progress

I use Google Search Console to monitor the progress of indexing.

Speeding Up Your Website Using Fastify and Redis Cache

Lilou Artz — Mon, 26 Aug 2024 07:52:59 +0000

Less than 24 hours ago, I wrote a post about how to speed up your website using Cloudflare cache. However, I've since moved most of the logic to a Fastify middleware using Redis. Here is why and how you can do it yourself.

Cloudflare Cache Issues

I ran into two issues with Cloudflare cache:

Page navigation broke after enabling caching of the responses. I raised an issue about this in the Remix forum a while back, but as of writing this, it is still unresolved. It is not clear why caching the response is causing the page navigation to break, but it only happens when the response is cached by Cloudflare.
I could not get Cloudflare to perform Serve Stale Content While Revalidating as described in the original post. Looks like it is not a feature that is available.

There were a few other issues that I ran into (like not being able to purge the cache using pattern matching), but those were not critical to my use case.

Therefore, I decided to move the logic to a Fastify middleware using Redis.

[!NOTE]
I left Cloudflare cache for image caching. In this case, Cloudflare cache effectively functions as a CDN.

Fastify Middleware

What follows is an annotated version of the middleware that I wrote to cache responses using Fastify.

const isCacheableRequest = (request: FastifyRequest): boolean => {
  // Do not attempt to use cache for authenticated visitors.
  if (request.visitor?.userAccount) {
    return false;
  }

  if (request.method !== 'GET') {
    return false;
  }

  // We only want to cache responses under /supplements/.
  if (!request.url.includes('/supplements/')) {
    return false;
  }

  // We provide a mechanism to bypass the cache.
  // This is necessary for implementing the "Serve Stale Content While Revalidating" feature.
  if (request.headers['cache-control'] === 'no-cache') {
    return false;
  }

  return true;
};

const isCacheableResponse = (reply: FastifyReply): boolean => {
  if (reply.statusCode !== 200) {
    return false;
  }

  // We don't want to cache responses that are served from the cache.
  if (reply.getHeader('x-pillser-cache') === 'HIT') {
    return false;
  }

  // We only want to cache responses that are HTML.
  if (!reply.getHeader('content-type')?.toString().includes('text/html')) {
    return false;
  }

  return true;
};

const generateRequestCacheKey = (request: FastifyRequest): string => {
  // We need to namespace the cache key to allow an easy purging of all the cache entries.
  return 'request:' + generateHash({
    algorithm: 'sha256',
    buffer: stringifyJson({
      method: request.method,
      url: request.url,
      // This is used to cache viewport specific responses.
      viewportWidth: request.viewportWidth,
    }),
    encoding: 'hex',
  });
};

type CachedResponse = {
  body: string;
  headers: Record<string, string>;
  statusCode: number;
};

const refreshRequestCache = async (request: FastifyRequest) => {
  await got({
    headers: {
      'cache-control': 'no-cache',
      'sec-ch-viewport-width': String(request.viewportWidth),
      'user-agent': request.headers['user-agent'],
    },
    method: 'GET',
    url: pathToAbsoluteUrl(request.originalUrl),
  });
};

app.addHook('onRequest', async (request, reply) => {
  if (!isCacheableRequest(request)) {
    return;
  }

  const cachedResponse = await redis.get(generateRequestCacheKey(request));

  if (!cachedResponse) {
    return;
  }

  reply.header('x-pillser-cache', 'HIT');

  const response: CachedResponse = parseJson(cachedResponse);

  reply.status(response.statusCode);
  reply.headers(response.headers);
  reply.send(response.body);
  reply.hijack();

  setImmediate(() => {
    // After the response is sent, we send a request to refresh the cache in the background.
    // This effectively serves stale content while revalidating.
    // Therefore, this cache does not reduce the number of requests to the origin;
    // The goal is to reduce the response time for the user.
    refreshRequestCache(request);
  });
});

const readableToString = (readable: Readable): Promise<string> => {
  const chunks: Uint8Array[] = [];

  return new Promise((resolve, reject) => {
    readable.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
    readable.on('error', (err) => reject(err));
    readable.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
  });
};

app.addHook('onSend', async (request, reply, payload) => {
  if (reply.hasHeader('x-pillser-cache')) {
    return payload;
  }

  if (!isCacheableRequest(request) || !isCacheableResponse(reply) || !(payload instanceof Readable)) {
    // Indicate that the response is not cacheable.
    reply.header('x-pillser-cache', 'DYNAMIC');

    return payload;
  }

  const content = await readableToString(payload);

  const headers = omit(reply.getHeaders(), [
    'content-length',
    'set-cookie',
    'x-pillser-cache',
  ]) as Record<string, string>;

  reply.header('x-pillser-cache', 'MISS');

  await redis.setex(
    generateRequestCacheKey(request),
    getDuration('1 day', 'seconds'),
    stringifyJson({
      body: content,
      headers,
      statusCode: reply.statusCode,
    } satisfies CachedResponse),
  );

  return content;
});

The comments walk through the code, but here are some key points:

Caching Criteria:
- Requests:
- Do not cache responses for authenticated users.
- Only cache GET requests.
- Only cache responses for URLs that include "/supplements/".
- Bypass cache if the request header contains cache-control: no-cache.
- Responses:
- Only cache successful responses (statusCode is 200).
- Do not cache responses already served from the cache (x-pillser-cache: HIT).
- Only cache responses with content-type: text/html.
Cache Key Generation:
- Use SHA-256 hash of a JSON representation containing request method, URL, and viewport width.
- Prefix the cache key with 'request:' for easy namespacing and purging.
Request Handling:
- Hook into the onRequest lifecycle to check if a request has a cached response.
- Serve the cached response if available, marking it with x-pillser-cache: HIT.
- Start a background task to refresh the cache after sending a cached response, implementing "Serve Stale Content While Revalidating".
Response Handling:
- Hook into the onSend lifecycle to process and cache responses.
- Convert readable streams to string for simpler caching.
- Exclude specific headers (content-length, set-cookie, x-pillser-cache) from the cache.
- Mark non-cacheable responses as x-pillser-cache: DYNAMIC.
- Cache responses with a TTL (Time To Live) of one day, marking new entries with x-pillser-cache: MISS.

Results

I ran latency tests from several locations and captured the slowest response time for each URL. The results are below:

URL	Country	Origin Response Time	Cloudflare Cached Response Time	Fastify Cached Response Time
https://pillser.com/vitamins/vitamin-b1	us-west1	240ms	16ms	40ms
https://pillser.com/vitamins/vitamin-b1	europe-west3	320ms	10ms	110ms
https://pillser.com/vitamins/vitamin-b1	australia-southeast1	362ms	16ms	192ms
https://pillser.com/supplements/vitamin-b1-3254	us-west1	280ms	10ms	38ms
https://pillser.com/supplements/vitamin-b1-3254	europe-west3	340ms	12ms	141ms
https://pillser.com/supplements/vitamin-b1-3254	australia-southeast1	362ms	14ms	183ms

Compared to Cloudflare cache, Fastify cache is slower. That's because the cached content is still served from the origin, whereas Cloudflare cache is served from regional edge locations. However, I found that these response times are plenty to achieving good user experience.

Speeding Up Your Website Using Cloudflare Cache

Lilou Artz — Sun, 25 Aug 2024 13:50:45 +0000

Performance is critical for websites to rank in Google search results. Pillser implements a number of techniques to load and render pages quickly. However, nothing beats caching. In this post, I will share my experience with Cloudflare cache.

Cloudflare Cache

I chose Cloudflare Cache because I am already using Cloudflare for other things.

To use Cloudflare cache, I needed to:

Tiered Cache and Cache Reserve are not strictly necessary, but they enable more reliable and faster cache hits.

When you enable Cache Reserve, you are able to cache gigabytes of data. Meanwhile, Tiered Cache reduces the amount of servers that Cloudflare needs to hop through to serve your website, which improves performance, e.g. I saw cached response times go from 100ms to under 10ms when I enabled Tiered Cache.

Finally, you need to add Cache Rules to define which pages should be cached. For example, I only want to cache pages that are accessed by non-authenticated users (identified by the presence of a user_account cookie), and I only want to cache pages matching a specific URL pattern. Here is a rule that does just that:

(
  not http.cookie contains "user_account" and (
    http.request.uri.path eq "/" or
    starts_with(http.request.uri.path, "/supplements") or
    starts_with(http.request.uri.path, "/probiotics") or
    starts_with(http.request.uri.path, "/vitamins") or
    starts_with(http.request.uri.path, "/minerals") or
    starts_with(http.request.uri.path, "/brands")
  )
)

I love that Cloudflare cache is so flexible. Their rules language is very powerful.

Cache by Device Type

You can enable options like Cache by device type if you are serving different content to different devices. Example: Pillser will render a different number of supplements per page depending on whether the user is on a mobile device or a desktop.

Once enabled, Cloudflare sends a CF-Device-Type HTTP header to your origin with a value of either mobile, tablet, or desktop for every request to specify the visitor's device type.

Utilize Strong ETags

The ETag HTTP response header is an identifier for a specific version of a resource.

Enable Respect strong ETags to ensure that the cache is invalidated when the content changes.

For this to function, you need to add a ETag header to the response. Fastify ecosystem has a plugin to automatically generate strong ETags.

Serve Stale Content While Revalidating (Not Working as Expected)

This is the only thing that I was not able to figure out.

My ideal behavior would be to cache products for a short period of time (e.g., 1 hour) and then serve stale content while revalidating.

I have therefore configured Edge TTL to Ignore cache-control header and use this TTL and set the TTL to 1 hour. This ensures that the cache becomes stale after 1 hour.

I have then left Do not serve stale content while updating disabled. This is supposed to make Cloudflare serve stale content while revalidating, but it does not seem to work.

I am still occasionally seeing content being served directly from the origin with cf-cache-status set to MISS. I would expect this to not happen, as the revalidation should happen in the background while the cache is being served. If you happen to know how to fix this, please let me know.

Lacking Features: Max Age for Stale Content

Another thing that I noticed is that Cloudflare will sometimes expire cached content based on Cache-Control headers. However, in the example of wanting to serve stale content while revalidating, I would expect that there would be a setting that allows me to explicitly say how long the content should be cached for regardless of the Cache-Control header, i.e., I would want to set max-age to several days, but require that Cloudflare revalidates the content every hour.

Effectively, I want to force Cloudflare to retain the cache beyond the TTL.

Purging Cache

Last but not least, I needed a way to purge the cache. Cloudflare provides several ways to purge the cache. However, I found that the API approach is the easiest to use:

import { config } from '#app/config.server';
import Cloudflare from 'cloudflare';

const cloudflare = new Cloudflare({
  apiEmail: config.CLOUDFLARE_API_EMAIL,
  apiKey: config.CLOUDFLARE_API_KEY,
});

const response = await cloudflare.cache.purge({
  files: ['https://pillser.com/'],
  zone_id: config.CLOUDFLARE_ZONE_ID,
});

This allows me to automate the purging of individual product cache, e.g. when a product is updated.

Results

I ran latency tests from several locations and captured the slowest response time for each URL. The results are below:

URL	Country	Origin Response Time	Cached Response Time
https://pillser.com/vitamins/vitamin-b1	us-west1	240ms	16ms
https://pillser.com/vitamins/vitamin-b1	europe-west3	320ms	10ms
https://pillser.com/vitamins/vitamin-b1	australia-southeast1	362ms	16ms
https://pillser.com/supplements/vitamin-b1-3254	us-west1	280ms	10ms
https://pillser.com/supplements/vitamin-b1-3254	europe-west3	340ms	12ms
https://pillser.com/supplements/vitamin-b1-3254	australia-southeast1	362ms	14ms

The results are consistent across multiple regions. It is clear that Cloudflare cache hugely improves the performance of the website, especially for users further away from the origin (US).

Automated ways to security audit your website

Lilou Artz — Sun, 28 Jul 2024 10:16:32 +0000

One of the goals of Pillser is to become HIPAA compliant. In short, HIPAA is a set of national standards that govern the privacy and security of individuals' health information. The compliance involves assessment of the security, privacy, and administrative safeguards. It's complicated and time-consuming, and if I've learned anything from my experience dealing with similar security standards (like PCI-DSS and SOC-2), it is that the more self-auditing you do, the more you can simplify the cost and effort of compliance when a third-party is involved.

This post is focused on the security aspect of Pillser, specifically, the automated checklists that anyone can use to assess their own website's security. It is not meant to be a comprehensive security audit, but rather the lowest hanging fruit that you need to clear before diving deeper into the more complex aspects of security.

Headers

Perhaps the simplest item on the checklist is to ensure that your website has proper HTTP headers.

A few tools that can help you with this:

Tools like SecurityHeaders.com and Mozilla's Observatory scan your website to identify any missing security headers.

Both audits have a brief explanation of what the headers do and why they are important.

Content Security Policy (CSP)

content-security-policy header was arguably the hardest to implement (particularly nonce). It is a policy that allows you to define what resources can be loaded on your website. This is important because it prevents malicious scripts from being injected on your website. For Pillser, it was important because we host user generated content (e.g., Ask AI, reviews, etc.). We wanted to make sure that even if a user was able to successfully inject malicious code, we would be able to detect it, block it, and remove it.

To my surprise, implementing CSP identified that Cloudflare was injecting a script into our website. This script (which is not malicious) was injected by Cloudflare to obfuscate email addresses. It is not something that we have a use case for, so we decided to remove it. However, this issue report turned out to be a good validation of the value of CSP.

DNSSEC

DNSSEC is a standard that specifies how to validate the DNS records of a domain. Implementing DNSSEC is important because it ensures that DNS queries are not spoofed.

To check if DNSSEC is enabled, you can use the DNSSEC debugger.

Email Security

Email is a critical part of our business. We use it to communicate with our customers, share information about our products, and to send out newsletters.

To ensure that our email sender cannot be spoofed, we implemented SPF and DKIM. SPF is a standard that specifies how to validate the sender of an email, while DKIM specifies how to validate the domain of an email.

We are using Cloudflare's DMARC Management to understand if messages sent from our domain are passing DMARC authentication, DomainKeys Identified Mail (DKIM) authentication, and Sender Policy Framework (SPF) policies.

Outside of Cloudflare, you may also try MXToolbox, which is a free tool to (among other things) check DMARC records.

SSL Server Test

The SSL auditing tools ensure that your website is serving over HTTPS and using secure ciphers. The SSL is necessary to prevent man-in-the-middle attacks.

For this audit, we used SSL Labs. It is a free service that performs a series of tests on your website to ensure that it is secure.

Because we are using Cloudflare, we expected to pass with flying colors. However, SSL test identified that we were allowing deprecated ciphers. If you are using Cloudflare, you may want to set the Minimum TLS Version to 1.3. Use a tool like caniuse.com to determine what percentage of your visitors support TLS 1.3.

HTTP Strict Transport Security

HTTP Strict Transport Security (HSTS) is a security feature that informs browsers that the site can only be accessed using HTTPS. This is more secure than simply configuring a HTTP-to-HTTPS (301) redirect on your server, where the initial HTTP connection is still vulnerable to a man-in-the-middle attack.

To implement HSTS, you need to add the Strict-Transport-Security header to your website, e.g. Strict-Transport-Security: max-age=31536000; includeSubDomains; preload. When the browser receives this header, it will remember that the current domain should only be accessed over HTTPS, i.e. even if user types http://pillser.com in the address bar, the browser will automatically redirect to https://pillser.com.

As an extra layer of precaution, you can also opt-in to preloading the HSTS list. This is a list of domains that are known to support HSTS. If you are a site owner, you can submit your domain to the list at https://hstspreload.org.

Technology Lookup

The next audit is to identify what technologies are being used on your website, as identified based on clues like the presence of specific scripts, headers, cookies, and other indicators. The goal is to reduce the attack surface by limiting the information about your technology stack.

To perform this audit, you can use Wappalyzer. It is a free tool that identifies the technologies used on your website.

Domain Expiry

Use a tool like What's My DNS? to determine the expiry date of your domain.

We keep our domain's expiration date set to 5 years in the future. This is to ensure that we do not accidentally expire our domain, which could become a security risk.

Security.txt

security.txt is a standard that specifies how to disclose security contact information. It is a file that lives at the root of your website, e.g. https://pillser.com/.well-known/security.txt.

We use security.txt to disclose our security contact information.

Application-Level Security Audit

Last but not least, we perform an application-level security audit. This is an autonomous process that attempts to identify known vulnerabilities in the application. It relies on recognizing patterns in the application (e.g. sign up form, login form, etc.), and then attempts to exploit those patterns by submitting malicious requests.

There are many tools available for this, e.g. Burp Suite, ZAP, etc. We've evaluated a few and found Probely to be the most comprehensive. They have a trial, so your first few scans will be free. After each scan, you will get a report that includes a list of all findings and a recommendation on how to fix them. You will also get a PCI-DSS and OWASP compliance report.

A few things to consider before performing an application-level security audit:

Consider if you want to disable Sentry for the duration of the audit. This is because errors triggered by malformed requests might exhaust your monthly limits.
Consider what forms might trigger side-effects. For example, if you have forms that send emails, consider how you want to handle those if there is a sudden spike in attempts to submit those forms.

Closing Thoughts

Security is not about setting it and forgetting it. For example, this post is written based on my notes from over a month ago. I was surprised to see that recent changes caused minor regressions, and used the opportunity to reapply the necessary security measures. In large part, this post is a reminder to myself of the regular audits that we need to do to ensure that our website is secure.

This post is the first in a series of posts that will cover the security and compliance aspects of Pillser. In the future, we will cover internal audits, dependency management, and other security measures. We will also cover the best practices for security and privacy at the application-level, such as automatic logoff, encryption, event logging, etc. Join our Discord server if you are interested in learning more.

Smaller Documents for Smaller Screens using Sec-CH-Viewport-Width

Lilou Artz — Thu, 27 Jun 2024 18:43:54 +0000

If you have been following my engineering blog, you know that I am obsessive about performance. Pillser lists a lot of data about supplements and research papers, and I want to make sure that the website is fast and responsive. One of the ways I've done it is by using Sec-CH-Viewport-Width to determine the width of the viewport and serve smaller documents to mobile devices.

What is `Sec-CH-Viewport-Width`?

Sec-CH-Viewport-Width is a Client Hints (CH) header to convey the viewport width of a client's display in CSS pixels. This header allows web servers to adapt their responses based on the actual size of the user's viewport, enabling better optimization of resources like images and layout.

However, by default, the header is not sent by the browser. To enable it, you need to send HTTP response headers with Accept-CH: Sec-CH-Viewport-Width. This will instruct the browser to send the Sec-CH-Viewport-Width header in the subsequent requests.

How does Pillser use `Sec-CH-Viewport-Width`?

If you look at pages like the supplement search or a specific supplement category page, you will notice that (on desktop devices) there is a lot of tabular data being displayed. This data provides valuable information for someone researching supplements, but it is not very readable on mobile devices and it accounts for a lot of the page's weight.

To solve this problem, Pillser uses Sec-CH-Viewport-Width to determine the width of the viewport and serve smaller documents to mobile devices. It works just like CSS media queries, but instead of deciding which content to display on a device, it makes the decision on the server. Here is the implementation of useViewportWidth:

import { usePublicAppGlobal } from './usePublicAppGlobal';
import { useEffect, useState } from 'react';

export const useViewportWidth = () => {
  const publicAppGlobal = usePublicAppGlobal();

  const [width, setWidth] = useState<number | null>(
    publicAppGlobal.visitor.viewportWidth,
  );

  useEffect(() => {
    const handleResize = () => {
      setWidth(window.innerWidth);
    };

    window.addEventListener('resize', handleResize);

    return () => {
      window.removeEventListener('resize', handleResize);
    };
  }, []);

  return width;
};

On the server, I parse the Sec-CH-Viewport-Width header and populate the visitor.viewportWidth field in the public app global. This field is then used by the useViewportWidth hook to determine the width of the viewport. Here is the server-side logic:

let viewportWidth: number | null;

try {
  viewportWidth = z
    .number({ coerce: true })
    .min(1)
    .parse(request.headers.get('sec-ch-viewport-width'));
} catch {
  viewportWidth = null;
}

And that's really all there is to it. The Sec-CH-Viewport-Width header is sent by the browser, Pillser parses it, and uses the result to determine the width of the viewport. This allows Pillser to serve smaller documents to mobile devices, improving the user experience and reducing the page weight.

Gotchas

Two gotchas to be aware of: browser support and the initial render.

Today, Client Hints are supported by 76% of browsers. The primary browsers that do not support Client Hints are Safari and Firefox. Regarding, Safari iOS, since we are defaulting to the smallest size in absence of the header (see the next section), it is not a problem. As for Safari desktop and Firefox, the website will still work as expected, but it will need to recalculate the content on the client-side. That's a fine trade-off if it means that the majority of visitors will get improved experience.

(You can also add support to Safari and Firefox by implementing pseudo-Client Hints by using cookies to set the viewport width.)

The other gotcha to be aware of is that the browser will only send the Sec-CH-Viewport-Width header in subsequent requests, not in the response. This means that the first time a user visits a page, their viewport width will not be known. To fix this, I default to always using the smallest breakpoint when the viewport width is unknown. This way, the mobile devices will still get the correct content, but the desktop UI will be updated upon recalculating the viewport using client-side logic.

Optimizing Image Loading with AVIF Placeholders for Enhanced Performance

Lilou Artz — Thu, 20 Jun 2024 13:24:53 +0000

It's no secret that page load times have a big impact on user experience, bounce rates, and SEO.

Meanwhile, some of the Pillser pages load a lot of data, e.g.,

https://pillser.com/brands/absolute-nutrition (13 products)
https://pillser.com/brands/21st-century (215 products) (takes a few seconds to load)

(The topic of why I am not using pagination is for another day.)

Therefore, I need to squeeze out every last bit of performance to maximize the user experience.

LQIP

Low Quality Image Placeholder (LQIP) is a technique that allows us to serve a low quality placeholder image to the browser while the actual image is being loaded.

Example:

The challenge though is that because the image needs to be visible immediately, we need to inline the actual image in the HTML, i.e. every bit counts towards the page size.

Here is what it looks like for the image above:



<div style="border: 1px solid #eee; width: 320px; aspect-ratio: 2469/1606; background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAXCAMAAABd273TAAAAIGNIUk0AAHomAACAhAAA+gAAAIDoAAB1MAAA6mAAADqYAAAXcJy6UTwAAAC0UExURfz///v///j8//r+//b6/fP3+vL2+fX5/PT4+/f7/vH1+O/z9vD09/n9/+zw8+ru8e3x9Ovv8u7y9ent8Ofr7uTo6+Pn6uXp7Obq7eDk597i5d/j5tzg49vf4t3h5OHl6Nnd4Nfb3tXZ3NTY29PX2tre4ejs79ba3dLW2c/T1s3R1M7S1dDU19jc39HV2MzQ08nN0MjMz8rO0cvP0sXJzMTIy8fLzsbKzcPHysLGyeLm6f///w5wCeoAAAABYktHRDs5DvRsAAAACXBIWXMAAAsSAAALEgHS3X78AAAAB3RJTUUH6AYUDCQTGrRfOwAAAAFvck5UAc+id5oAAAIZSURBVCjPTZLrdqowEIVTrXeLVRAEEsi14ZKEm4LH93+wM9rV1ebXXsle32T2DEIIvc1mc/Q8b++L5WL2kmi+Wqx/5Ga12rwc8/V2t119mzfL/Xb9AeJjA7e77Xo2n8/WO+/gbd+/3z8Px9Ni9oa2O88PAt/bL5d7LziHkX9ab2ar/fFyBrlcoSAK4ySJw8D3g3OKSZYH3n5/8sMEZ/nleEIJpoxzRpI8TwkTkqs0+vIvOWZcq/QcIC2KsqqNYEpZ7uqqFDQNw5jowjiukhyZumm7rq8LzoVp2nYwmqQptkXdVE6TBDVDd73dxqEsClf149RWkuJMcTP0A9AwGtrpdr9PfW0cAKapqwpLMBV1D1hpya/BOVP9GIiVVTu2VcEIanooce2ghHR1P8KtfBqKqpu6xjGF6qrvxrGvnIB2hu6bq5gbxuvUG26RK6u+bRsjtX518TQopU1/vV/bUjAkIYahgS6t1UU9tPB1qygv29u/e1dLjbhwZf0Miiomy2ZoDKdE/TEwLZ0xhaaEUEgSWFph8ipxgxIWwSQKVwhLsowAojSCZklGi2a8jsOzTWW1kFLTLE0zCl7JSJrHmJcQVAmpIwII8TtNASMMozClsm5KQeIzwsRqrimOH484o0xb2IKvywPDNKVNIx9lgNBMZXl4fqTEWtiBr4MfpZQLjUP/ExZGWWZJ8jjDamEKqOjgHYOcACyJvN1/yWxdxmlLsYwAAAC0ZVhJZklJKgAIAAAABgASAQMAAQAAAAEAAAAaAQUAAQAAAFYAAAAbAQUAAQAAAF4AAAAoAQMAAQAAAAIAAAATAgMAAQAAAAEAAABphwQAAQAAAGYAAAAAAAAASAAAAAEAAABIAAAAAQAAAAYAAJAHAAQAAAAwMjEwAZEHAAQAAAABAgMAAKAHAAQAAAAwMTAwAaADAAEAAAD//wAAAqAEAAEAAAAgAAAAA6AEAAEAAAAXAAAAAAAAAB72jjsAAAAldEVYdGRhdGU6Y3JlYXRlADIwMjQtMDYtMjBUMTI6MzY6MTkrMDA6MDApQGkQAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDI0LTA2LTIwVDEyOjM2OjE5KzAwOjAwWB3RrAAAACh0RVh0ZGF0ZTp0aW1lc3RhbXAAMjAyNC0wNi0yMFQxMjozNjoxOSswMDowMA8I8HMAAAAVdEVYdGV4aWY6Q29sb3JTcGFjZQA2NTUzNTN7AG4AAAAgdEVYdGV4aWY6Q29tcG9uZW50c0NvbmZpZ3VyYXRpb24ALi4uavKhZAAAABN0RVh0ZXhpZjpFeGlmT2Zmc2V0ADEwMnNCKacAAAAVdEVYdGV4aWY6RXhpZlZlcnNpb24AMDIxMLh2VngAAAAZdEVYdGV4aWY6Rmxhc2hQaXhWZXJzaW9uADAxMDAS1CisAAAAF3RFWHRleGlmOlBpeGVsWERpbWVuc2lvbgAzMoisZxcAAAAXdEVYdGV4aWY6UGl4ZWxZRGltZW5zaW9uADIzOya/RQAAABd0RVh0ZXhpZjpZQ2JDclBvc2l0aW9uaW5nADGsD4BjAAAAAElFTkSuQmCC);background-size:100% 100%"></div>

This LQIP has been generated using ThumbHash. Compared to other implementations of LQIP (like BlurHash or Potato WebP), this one encodes more details in the same space.

However, the above image representation still consumes 2,050 bytes of data, which adds up to ~440 KB for a page with 215 images (like the 21st Century brand page). That's a lot!

AVIF

The realization that I had was that, just how I use AVIF for product images themselves (because the file size is smaller), I can use AVIF to reduce the size of the LQIP. Here is what the same image looks like in AVIF:



<div style="border: 1px solid #eee; width: 320px; aspect-ratio: 2469/1606; background-image: url(data:image/avif;base64,AAAAHGZ0eXBhdmlmAAAAAGF2aWZtaWYxbWlhZgAAAc1tZXRhAAAAAAAAACFoZGxyAAAAAAAAAABwaWN0AAAAAAAAAAAAAAAAAAAAAA5waXRtAAAAAAABAAAARmlsb2MAAAAAREAAAwACAAAAAAHxAAEAAAAAAAAAFAABAAAAAAIFAAEAAAAAAAAAJgADAAAAAAIrAAEAAAAAAAAAvgAAAE1paW5mAAAAAAADAAAAFWluZmUCAAAAAAEAAGF2MDEAAAAAFWluZmUCAAAAAAIAAGF2MDEAAAAAFWluZmUCAAABAAMAAEV4aWYAAAAA12lwcnAAAACxaXBjbwAAABNjb2xybmNseAACAAIABoAAAAAMYXYxQ4EAHAAAAAAUaXNwZQAAAAAAAAAgAAAAFwAAAA5waXhpAAAAAAEIAAAAOGF1eEMAAAAAdXJuOm1wZWc6bXBlZ0I6Y2ljcDpzeXN0ZW1zOmF1eGlsaWFyeTphbHBoYQAAAAAMYXYxQ4EgAgAAAAAUaXNwZQAAAAAAAAAgAAAAFwAAABBwaXhpAAAAAAMICAgAAAAeaXBtYQAAAAAAAAACAAEEgYYHiAACBIIDhIUAAAAoaXJlZgAAAAAAAAAOYXV4bAACAAEAAQAAAA5jZHNjAAMAAQABAAABAG1kYXQSAAoFGBE/ZhUyCRgAAAEACQwfcxIACgU4ET9mCTIbGAAAAEBJ5HdzPsgdh/pW4sdXS55ZvLbaUbb4AAAABkV4aWYAAElJKgAIAAAABgASAQMAAQAAAAEAAAAaAQUAAQAAAFYAAAAbAQUAAQAAAF4AAAAoAQMAAQAAAAIAAAATAgMAAQAAAAEAAABphwQAAQAAAGYAAAAAAAAASAAAAAEAAABIAAAAAQAAAAYAAJAHAAQAAAAwMjEwAZEHAAQAAAABAgMAAKAHAAQAAAAwMTAwAaADAAEAAAD//wAAAqAEAAEAAAAgAAAAA6AEAAEAAAAXAAAAAAAAAA==);background-size:100% 100%"></div>

The above is now 1,019 bytes (or 50% of the original LQIP).

Using sharp to convert PNG to AVIF

thumbhash defaults to producing png images. Perhaps, this is to support a broader range of browsers (AVIF has 93.62% browser support). However, I made a conscious decision that it is an acceptable trade-off to use AVIF for the placeholder images if it means that I can reduce the image size by 50%.

Therefore, I am using thumbhash to generate the LQIP, and then using sharp to convert the PNG to AVIF. Here is the underlying code:



import sharp from 'sharp';
import { rgbaToThumbHash, thumbHashToDataURL } from 'thumbhash';

const dataUrlToBuffer = (dataUrl: string) => {
  const match = dataUrl.match(/^data:[^;]+;base64,([^"]+)/u);

  if (!match) {
    throw new Error('Invalid data URL');
  }

  const [, base64] = match;

  return Buffer.from(base64, 'base64');
};


export const generateThumbHashDataUrl = async (image: Buffer) => {
  const smallImage = await sharp(image).resize(100);

  const { data, info } = await smallImage
    .ensureAlpha()
    .raw()
    .toBuffer({ resolveWithObject: true });

  const dataUrl = thumbHashToDataURL(
    rgbaToThumbHash(info.width, info.height, data),
  );

  return `data:image/avif;base64,${(
    await sharp(dataUrlToBuffer(dataUrl)).avif().toBuffer()
  ).toString('base64')}`;
};

The conversion happens when uploading the image to the database, therefore the overhead does not impact the user experience.

And that's it! Using this simple technique I was able to significantly reduce the page size when there are a lot of images.

Designing a Website Without 404s

Lilou Artz — Tue, 11 Jun 2024 02:24:03 +0000

When I started working on Pillser, I knew I am not going to get everything right the first time.

This is particularly true about the information architecture of the website.

The idea behind Pillser is simple: associate research with supplements. However, as there aren't man
y anologous websites, I have to come up with how to present this information. As such, I wanted to reduce lock-in to URL architecture as much as possible. One of the ways I've done it is by using fuzzy matching to redirect users to the right place when they make a typo, a product is renamed, or the logic used to generate the URL changes. (The latter has happened enough times already that I am glad that I've implemented this feature!)

Here is what it looks like from the user's perspective:

/supplements/spore-probiotic-6066 This is the correct URL for the supplement.
/supplements/spore-probiotic This URL is missing the ID. However, the website will redirect the user to the correct page.
/supplements/youtheory-spore-probiotic This URL includes the brand name. However, since disregarding the brand name "spore probiotic" is unique enough to identify the supplement, the website will redirect the user to the correct page.

At the moment, I've applied this logic only to the supplement pages, but I am planning to extend it to the rest of the website.

In practice, the implementation is so simple that I am surprised that more websites do not implement it. It is basically a fallback mechanism that queries the database to find the closest match to the URL.

SELECT
  id,
  slug
FROM supplement
ORDER BY similarity(slug, ${slug}) DESC
LIMIT 1

The similarity function is a PostgreSQL pg_tgrm extension that calculates the similarity between two strings. The closer the value is to 1, the more similar the strings are. Since my goal is to find the closest match, I order the results by the similarity in descending order and pick the first one.

In the backend, I log whenever such a redirect happens. This way, I can manually override the redirect logic if I discover that the chosen supplement is not the correct one or not the most relevant substitute.

The primary goal of this is to provide a better user experience. I am unsure what are the implications for SEO, but I am hoping that it will be positive. I will keep you updated on this.

Building an Alternative to Examine.com: A Challenging Journey!

Lilou Artz — Sun, 02 Jun 2024 20:40:01 +0000

Many of you might already know about Examine.com, a leading resource for supplement research. Examine focuses on specific health areas, scouts for related research papers, and then describes the link to different supplements. This meticulous process is understandably labor-intensive, which is why they charge a fee for accessing their data. However, with the advent of AI, many parts of this process can be automated, and that's the solution I'm working on.

For instance, here's a compilation of research linked to Reduced Body Weight:

https://pillser.com/health-outcomes/reduced-body-weight-158

I've indexed thousands of research papers and extracted insights that connect these studies to various supplements on the market. My long-term goal is to create a pioneering supplement store where every product is linked to research. Unlike Examine, I plan to make all research summaries public and instead focus on earning affiliate revenue from related supplement sales.

The biggest challenge is ensuring data accuracy. Given the complexity of these topics, I am currently limiting insights to demonstrate a link between the study, health outcome, and substance. Users are then directed to the actual research papers to build confidence in their decisions. However, as AI models evolve, I aim to expand this into a comprehensive insight engine.

AI and large language models (LLMs) are crucial in this process. Finding research papers is relatively straightforward with resources like PubMed. I use a combination of API services, varying in cost and speed, to scout for relevant mentions in the research (using fast and cheap models), validate the relevance of these mentions (using top models), and finally ensure the accuracy of the summary versus the excerpt (using another model). The idea is that the first model may make mistakes, the second filters out false positives, and the third acts as a final safeguard.

I am deeply fascinated by this problem domain and, more broadly, by data normalization and the applications of LLMs in solving these problems. While this project is currently a hobby that I anticipate will be a money-losing activity for a long time, I believe there is a significant chance that Pillser could become a preferred site for supplement buyers due to its unique combination of scientific backing and extensive inventory.

I am probably a few months away from completing the database, but I wanted to share this for early feedback. The website is called Pillser, and you can already search for different health goals and associated research directly from the landing page.

DEV Community: Lilou Artz

Auto Indexing of URLs via Google API

Google

Acquiring Credentials

What can be indexed?

Bonus: IndexNow

Tracking Indexing Progress

Speeding Up Your Website Using Fastify and Redis Cache

Cloudflare Cache Issues

Fastify Middleware

Results

Speeding Up Your Website Using Cloudflare Cache

Cloudflare Cache

Cache by Device Type

Utilize Strong ETags

Serve Stale Content While Revalidating (Not Working as Expected)

Lacking Features: Max Age for Stale Content

Purging Cache

Results

Automated ways to security audit your website

Headers

Content Security Policy (CSP)

DNSSEC

Email Security

SSL Server Test

HTTP Strict Transport Security

Technology Lookup

Domain Expiry

Security.txt

Application-Level Security Audit

Closing Thoughts

Smaller Documents for Smaller Screens using Sec-CH-Viewport-Width

What is Sec-CH-Viewport-Width?

How does Pillser use Sec-CH-Viewport-Width?

Gotchas

Optimizing Image Loading with AVIF Placeholders for Enhanced Performance

LQIP

AVIF

Using sharp to convert PNG to AVIF

Designing a Website Without 404s

Building an Alternative to Examine.com: A Challenging Journey!

What is `Sec-CH-Viewport-Width`?

How does Pillser use `Sec-CH-Viewport-Width`?