DEV Community: Pieter D

Tired of your Docker builds taking so long?

Pieter D — Fri, 11 Nov 2022 10:45:46 +0000

I have some good news: Docker has a cache feature that can help you speed up your builds, and it is enabled out of the box. But the way you write your Dockerfiles has a big influence on how cacheable your builds are. If you rewrite a few parts of your Dockerfile, cache can help you cut precious time from your build.

How does Docker cache work? What makes a Dockerfile more cacheable? And how do you optimize the build process? This hands-on video takes a slow Dockerfile and optimizes it step by step:

Convert any address on Earth to coordinates with this free API

Pieter D — Mon, 19 Sep 2022 18:36:08 +0000

Whenever you enter an address into Google Maps, it will instantly figure out where on Earth it's located. We sometimes take it for granted, and it's not always perfect, but it remains a cool software engineering achievement. This process of converting addresses to a pair of latitude/longitude coordinates is known as geocoding.

There are some interesting use cases for geocoding beyond Google Maps. A food delivery service may want to calculate delivery costs based on the distance between the delivery address and the restaurant, which requires resolving the addresses of both to coordinates. While Google Maps provides a geocoding API for third-party applications, it'll cost you about $5 per 1,000 requests.

But there's no need to pay. Positionstack has a generous free tier allowing 25,000 API requests per month.

To start using their geolocation service, first create an account. When you finish signing up, you'll receive an API key.

Unfortunately, Positionstack does not allow you to issue additional API keys for each of your projects. Per-project keys are a common security practice to make it easier to revoke access in case one of your keys leak. Since Positionstack is a free service, the worst thing that can happen is that your account gets suspended if anyone abuses your key. While better than a massive bill, it's still a good reason to keep your key safe.

With your key in hand, let's try a request! I'm using the excellent command-line tool HTTPie below, but Postman or cURL will work just as well.



❯ http -b get 'http://api.positionstack.com/v1/forward?access_key=[your API key here]&limit=1&query=3%20abbey%20road,%20london'
{
    "data": [
        {
            "administrative_area": null,
            "confidence": 1,
            "continent": "Europe",
            "country": "United Kingdom",
            "country_code": "GBR",
            "county": null,
            "label": "3 Abbey Road, London, England, United Kingdom",
            "latitude": 51.53218,
            "locality": "London",
            "longitude": -0.177866,
            "name": "3 Abbey Road",
            "neighbourhood": "St. John's Wood",
            "number": "3",
            "postal_code": "NW8 9AY",
            "region": "Greater London",
            "region_code": null,
            "street": "Abbey Road",
            "type": "address"
        }
    ]
}

A couple of observations here. In addition to the latitude and longitude, you'll receive a number of other attributes from which you can derive a cleaned version of the address you looked up. These attributes may not be relevant now, but if you ever need to split unstructured addresses into street, number, postal code, locality, and country, they come in handy.

You will get up to 10 results by default. The limit parameter will let you reduce or increase the cap depending on your needs.

We just used Positionstack's forward geocoding service. If you have a latitude/longitude pair that you'd like to resolve to an address, that's possible too. We can use their reverse geocoding API for this purpose. Here's an example for 42.8130,-1.6475.



❯ http -b get 'http://api.positionstack.com/v1/reverse?access_key=[your API key]&limit=1&query=42.8130,-1.6475'
{
    "data": [
        {
            "administrative_area": "Pamplona/Iruña",
            "confidence": 0.8,
            "continent": "Europe",
            "country": "Spain",
            "country_code": "ESP",
            "county": null,
            "distance": 0.033,
            "label": "Entrada al Parking del Baluarte, Pamplona, NA, Spain",
            "latitude": 42.813258,
            "locality": "Pamplona",
            "longitude": -1.647702,
            "name": "Entrada al Parking del Baluarte",
            "neighbourhood": null,
            "number": null,
            "postal_code": null,
            "region": "Navarre",
            "region_code": "NA",
            "street": null,
            "type": "venue"
        }
    ]
}

One major drawback of Positionstack is that the free version does not allow HTTPS requests. The lack of HTTPS support has privacy implications for the addresses you submit. Alternatively, Bing Maps has a developer tier that supports HTTPS, but its usage limit does not reset. In other words: Bing's alternative is more of a trial account than a free tier. On the other hand, Bing has the advantage of being licensed for commercial use - which Positionstack's free version is not. Another free option worth considering is OpenWeatherMap's geocoder, which appears to work up to the municipality level but not down to street level.

There are a few niche features in Positionstack's API that I haven't covered above, so do check their documentation for some interesting nuggets. If you ever needed to, you could even use Positionstack to retrieve the currency or calling code of the location you looked up. Happy geocoding!

Overwhelmed by Docker? Start with this easy 55-second animation

Pieter D — Thu, 21 Apr 2022 20:17:49 +0000

Let's be real: getting into software engineering is hard. Even when you're focused on one specific technology you want to learn, there's so much depth to everything. If you've tried getting into Docker before but felt disoriented, here's a short animation that can help you see the forest through the trees.

It touches on Docker's purpose, Docker images, containers, and several frequently used Docker commands. This summary will send you on your way and bridge the gap towards the more detailed resources you find all over the internet.

One of these two apple.com domains is fake

Pieter D — Tue, 12 Apr 2022 19:36:15 +0000

Five years ago, a security researcher registered a fake domain that is indistinguishable from apple.com. Chrome released a patch so people would be able to tell the difference. After an intense discussion and careful consideration, Firefox decided not to fix it. That's why the current version of Firefox still displays both domain as identical. What happened?

If you are reading this - and I have a hunch that you are - you know the basic Latin alphabet. What you might not think about every day is that lots of different alphabets are used across the world. Even within the Latin script, there are different alphabet flavors containing characters like ü, à, î, ł or ñ. Similarly, there is the Cyrillic script. It also has a lot of regional variations.

Latin and Cyrillic have some overlap. In some fonts, the lowercase Latin A happens to look exactly like the lowercase Cyrillic A. Same with the lowercase Latin P and the lowercase Cyrillic R. That's how I can write apple.com and аррӏе.com with different characters, and have them look identical in some fonts.

To help people recognize the difference, Chrome wrote a patch. Since then, domain names under Latin-character TLDs that consist only of Cyrillic lookalike characters now look different than the Latin domain. It now renders the fake apple.com domain name as xn--80ak6aa92e.com.

Firefox took a stance against the fix. Their reasoning is explained in the video, and they do have a point. The gist of it is that they want to treat Latin and Cyrillic as equals, and automatically treating either the Latin or Cyrillic variant as suspicious would go against that. They also argue that it's the .com registry's job to prevent the registration of lookalike domain names, which the registry isn't doing. Plus, Firefox still has Phishing Protection as a final line of defense.

So that's why five years later, you can still access the fake domain name in Firefox. Meanwhile, the fake domain hasn't been claimed by Apple, nor has it been taken offline by its domain registrar. Presumably, it's because it isn't actively trying to scam anyone. But it serves as a reminder that we have a lot of alphabets on this planet and that sometimes, this causes sticky situations.

When does your domain name need glue records?

Pieter D — Sat, 05 Feb 2022 13:35:09 +0000

Let's say you own example.com, and you want to host your name servers at ns1.example.com and ns2.example.com.

When a DNS client looks up ns1.example.com, the root servers would tell the client to ask the name servers of the .com TLD, which would tell it to ask the name servers of example.com. Hmm. You need to ask the example.com name servers where the name servers of example.com are.

See the chicken-and-egg problem? It's a maze without an exit. But you can create one - if you provide glue records. Basically, you have to ensure the TLD's name servers reveal your own name servers' IP addresses through additional DNS records. Here's an animated video that explains how it works.

Are glue records dangerous?

When used incorrectly, glue records may cause your website to become unreachable. In essence, you're asking your TLD's name servers to cache the IP address translation of your name servers. If you rely on a third-party DNS service like CloudFlare for your domain's name servers, their IP addresses can change at any time without your knowledge.

If you're using ruth.ns.cloudflare.com and west.ns.cloudflare.com as your name servers, and you pushed glue records for them, your website may become unavailable if at least one of its name servers gets a new IP address.

Long story short: glue records are a crucial element of DNS, but only use them when strictly necessary. Be aware that you will need to accommodate any IP address changes in your glue records to ensure your website remains available.

Why Windows uses \r\n newlines instead of \n

Pieter D — Sun, 14 Nov 2021 13:44:57 +0000

You may have noticed that Windows has a different way of storing newlines than Mac and Linux. A newline on Windows is encoded as \r\n, which symbolizes a Carriage Return (CR) followed by a Line Feed (LF). By contrast, Mac and Linux simply use a \n.

Why does Windows do its own thing, and why does it use two characters to signal the start of a new line?

To understand why Windows does what it does, we have to travel back in time to typewriters. Think of them as printers that have a keyboard. Whenever you typed a letter, a hammer would smash ink against paper. In the early 20th century - before the internet - people figured out how to connect typewriters over a wired telephone network so that companies could send each other direct messages across the country. One of these connected typewriter brands was Teletype.

In the 1960s, a new Teletype model came out that could be controlled from a computer. Problem was, if you let your computer send a \n character to the Teletype, the typewriter didn't always return to the start of the next line fast enough for the next character to be printed. That could lead to smudged letters being printed halfway your previous line.

The workaround was to always send a \r character before sending a \n character. The \r would return to the start of the current line while the \n pushed the paper up one line. This sequence of two characters gave the typewriter enough time to catch up before the first character of the next line had to be processed.

Our second player in this story is CP/M, a popular operating system from around that time. CP/M wanted to make it easy to interact with Teletype machines, and device drivers weren't a thing yet, so its engineers adopted Teletype's newline sequence as CP/M native newline sequence for compatibility reasons. Microsoft's competitor MS-DOS, in its turn, wanted to be compatible with CP/M. Therefore, MS-DOS also sided with the \r\n. MS-DOS would later evolve into Windows over the 80s and 90s, and the rest is history.

Microsoft could choose to migrate Windows files to \n. Mac actually did the same thing around in 2001, when it successfully retired its \r newline sequence in favor of \n with the launch of Mac OS X 10.0. Nowadays, Teletype is out of the picture and device drivers are mature. There doesn't seem to be much reason to continue using two newline characters – other than wasting disk space. Then again... if it ain't broke, does it need to be fixed?

Make your shell scripts more bugproof with a few simple habits

Pieter D — Sat, 23 Oct 2021 11:14:04 +0000

Your terminal can do some really cool things. The | operator was a stroke of genius. It lets you take the output of one command and feed it to another, chaining together a few simple tools to perform a bigger automated task.

Shell scripts are a natural extension of your terminal's power. Creating ready-to-run recipes for later use? Sweet. But once you start using these scripts for more advanced things, you run into some unfortunate Bash quirks. For example: when a part of your script produces an error, the rest of the script will still be executed, making its behavior unpredictable.

There are some habits that you can adopt right now to make writing shell scripts easier.

Here's the gist of it if you don't need further explanation:

Add set -euo pipefail to improve how your script deals with undeclared variables and non-zero exit codes.
Add IFS=$'\n\t' so that array elements are split in a predictable way.
When defining functions, return data with echo instead of return. The return keyword doesn't do the same thing as in other languages.
Returning arrays from functions is not really possible. As a bad but easy workaround, you can pass array return values via a global variable.
Keep your whitespace in mind. Confusingly, if [ FOO=='bar' ] will overwrite the value of $FOO. if [ FOO = 'bar' ] will perform a string equality check.

Alternatively, for complex tasks, you'll probably have an easier time using a different scripting language like Python, Ruby or Perl.

Additional reading

Unofficial bash strict mode
Your bash scripts are rubbish, use another language

TCP, UDP or QUIC? Read this before you choose.

Pieter D — Sun, 30 May 2021 12:28:48 +0000

If you're writing an application that sends data over a network, you have an important choice to make. How will it communicate? In some cases you can - and probably should! - reuse an existing application-layer protocol like HTTP. But maybe you need something specialized to your needs.

In this case you'll be writing your own application-layer protocol. Your new protocol will build on either TCP, UDP or QUIC. Which one should you pick and why?

For a detailed look at the differences between the three protocols, you might want to watch this animated video:

What your best choice is depends on the nature of your application. Here are some important considerations.

TCP and UDP are well-established protocols that are pretty mature. QUIC is a relatively new alternative to TCP that may outperform it, but its development is still ongoing.
TCP and QUIC provide reliable data transfer. This means you can be reasonably sure that what you send comes out correctly on the receiver's end. UDP provides no such guarantees. Data may be delivered in a different order, parts of your message may get lost, or data may be delivered to your application in a corrupted state.
UDP provides the lowest possible lag because it doesn't have to provide any reliability guarantees. TCP and QUIC negotiate a connection before sending data, and may delay sending new data as they await delivery confirmations from the receiver.
Unlike TCP, UDP is able to support multicast and broadcast transmissions.
QUIC provides encryption out of the box. For TCP connections you can use TLS 1.3 with some additional setup. UDP transmissions can use DTLS.
If your application maintains open connections over a longer period, they may get disrupted when your users switch between WiFi, Ethernet and cell towers. QUIC has an edge here: its connection migration feature can handle those network switches.

Turbocharge your Linux terminal productivity with these 12 tips

Pieter D — Fri, 30 Apr 2021 16:32:37 +0000

The Linux terminal is an incredibly powerful tool to get stuff done quickly. How fast can you go? That greatly depends on how much hotkeys and other power user tricks you know. Here are 12 tips to add to your arsenal.

For details and visual demos, see video below. Read on for the text version.

Let your terminal type filenames for you. Whenever you're typing the name of a file, folder or command, only type a little bit of it and press Tab. Your shell will try to figure out what you're after and complete the name for you. If multiple matches exist, Tab again to see the options, type a few more characters, then Tab once more.
Quickly jump between words. Forgot to add a flag at the start of your command? Alt+Left and Alt+Right let you do word jumps.
Faster backspacing. If you need to erase a part of your command, pressing Backspace is kinda slow. Alt+Backspace erases one word at a time.
Repeating past commands. Ctrl+R allows you to easily recall commands you've done before. Hit the shortcut, then type a part of a past command, then Enter once you see it.
Go back a directory. Entering cd - will bring you right back into the directory you were in before your last cd command. Similarly, you can jump back to your previous Git branch with git checkout -.
Split your terminal. You can have multiple shell prompts on your screen at the same time using Terminator (easiest, mouse-based) or tmux (more powerful, keyboard-based).
Alias long commands. Often need to repeat a long command? Create an alias for it so that you can just execute myAlias next time. Details in the video. An alias looks like alias myAlias="echo Hello world!".
Send output to clipboard. On X with xclip installed, you can pipe any output to xclip -selection clipboard to send it to your system clipboard. Bonus points if you alias it for quick access. Say goodbye to tedious manual text selection!
Make reusable functions. If you want to create an alias but some of its parameters should be variable, this is what you need. A Bash function looks like this:
```
function resizeToWidth() {
    convert -resize $1 $2 $2.$1w.png
}
```
Once defined, it works like a regular command, e.g. resizeToWidth 500 myImage.png.
Get automatic suggestions. Thanks to the Zsh plugin zsh-autosuggestions your terminal can suggest commands you've previously used as you type – even if you forgot to press Ctrl+R. You can accept suggestions with the Right arrow, or just keep typing if it's not the command you need.
Let your terminal prompt display your current Git branch. Many Zsh themes like robbyrussel's will add an indication of your current Git branch name to your prompt if you're in a Git repository. You won't need git status as much and you'll be less likely to commit to the wrong branch.
Syntax highlighting. You already know syntax highlighting from your IDE, but you can also get that in your shell. This is very useful if you're typing up a command that uses strings, variables or advanced features that go beyond your average ls ~/Desktop. It also shows a visual difference between commands that exist and commands that don't.

What productivity tricks do you use? Share your tips in the comments!