Photo by: Paul Esch-Laurent (original)

Using Go Vanity URLs with Hugo

Go vanity import URLs are useful tools for maintaining a consistent import path for public Go packages, even if one changes code repository location. In this article, I show how to set up vanity URLs in a website built by Hugo and deployed on Netlify for Go packages hosted on a third-party site such as GitLab or GitHub.

Go Package Import Paths

Ordinarily Go packages are identified by the URL of source code repository (often git but could also be other supported version control systems) where the package source code resides. While this works well, there are cases where associating a package to a the repository URL is undesirable. The most prevalent disadvantage of direct import URLs is the difficulty in migrating source from one host to another. If one decides to change hosting providers, one must change all the import statements in dependent code to point to the new URL. This is especially troublesome in the case of a public package that is used by other people’s code as their import statements need to be fixed as well.

Vanity Import URLs

The common solution to this migration problem is to use Go’s vanity import URLs. A vanity URL provides a fixed URL for identifying and importing a package that is independent of the repository hosting the code. Once a vanity URL is set up, the backend repository can be changed (path or host) without impacting downstream code.

Beyond migrations, vanity URLs also allow developers a way to associate their packages with their own domains/websites without worrying about hosting git (or other) repositories themselves.

That said, if uses vanity URLs hosted on a private domain, the vanity URL host can be a failure point. If the host is down, the packages won’t be accessible, even if the actual code repository is accessible. While Go’s recent inclusion of package proxy and cache system will help in this regard, one should still be aware of the potential for access failures, especially if the URLs are self-hosted.

How Vanity Import URLs Work

Vanity URL support is built directly into the Go toolchain, primarily the go get command. When fetching a package, Go examines the import path to determine if it matches one of a handful of predefined code hosting sites (e.g. github.com, bitbucket.org, etc…). If no match is found, the go get command makes an HTTP(S) GET request to the import URL. The tooling examines the <head> section of the returned document looking for a <meta> tag named go-import containing the following space-separated fields within its content attribute:

For example, a vanity URL for the package, “example.org/mypackage”, might have the following <meta> tag in its HTML <head>:

<meta name="go-import" content="example.org/mypackage git https://github.com/exampleUser/mypackage">

In addition to the go-import <meta> element above, Go also supports a go-source element that is used by the Go documentation tools (eg. go doc) for linking to files and particular lines in a file. By placing the following data in the content attribute, one can support on-line documentation resources such as godoc.org and its successor, pkg.go.dev:

For github.com hosted repositories, the go-source element would look like:

<meta name="go-source" content="example.org/mypackage
  https://github.com/exampleUser/mypackage
  https://github.com/exampleUser/mypackage/tree/master{/dir}
  https://github.com/exampleUser/mypackage/blob/master{/dir}/{file}#L{line}">

See https://github.com/golang/gddo/wiki/Source-Code-Links for additional details.

Implementing in Hugo / Netlify

I was interested in setting up vanity URLs for some of my public projects using my existing blog which is built with Hugo and hosted on Netlify. Specifically, I was looking for the following behavior:

As part of my investigation into vanity URLs, I read some very good posts by Nate Finch and Márk Sági-Kazár. While their approach wasn’t fully suitable for what I had in mind, they are excellent resources which were quite useful to my work.

As I decided to have the default (non-Go tool) webpage associated with a package be a locally hosted project page, I did not need the Hugo aliases based redirects approach that Nate outlined. Initially I played with Netlify redirect rules to redirect requests from the Go tooling to a specific page containing only the <meta> tags, while allowing non-tooling requests to go to the project page. However, this proved to be an unnecessary complexity as the I could include the Go tooling <meta> tags directly in the project page itself.

Generating <meta> Tags

The first step of implementing vanity URLs for my site was to create a partial template, goimports.html with the following content:

<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta name="go-import" content="{{ index .Params.packages 0 }} git {{ .Params.repo }}">
<meta name="go-source" content="{{ index .Params.packages 0 }} {{ .Params.repo }} {{ .Params.repo }}/tree/master{/dir} {{.Params.repo}}/blob/master{/dir}/{file}#L{line}">

This template requires two parameters, packages, a list of packages associated with the project, and repo, the URL of the repository hosting the packages. These parameters are stored in the front matter of individual project pages similar to the following:

---
name: "cereal"
description: "A Go library for reading Python cerealizer archives"
repo: "https://jbowen.dev/cereal"
packages: ["jbowen.dev/cereal"]
---

Next I add a reference to the partial template in my main head.html template, wrapping it within a conditional check on the packages parameter as follows:

{{ if .Params.packages }}{{ partial "goimports.html" . }}{{ end -}}

One potential caveat to the above approach is to be careful with the location of the <meta> tags within the generated HTML file. In order to minimize parsing errors, the Go docs on remote import paths state:

The meta tag should appear as early in the file as possible. In particular, it should appear before any raw JavaScript or CSS, to avoid confusing the go command’s restricted parser.

URL Redirects

In order to use a short import path such as "jbowen.dev/cereal" we need to establish a redirect from https://jbowen.dev/cereal to the actual project page at https://blog.jbowen.dev/projects/cereal. My hosting provider, Netlify, provides a means for defining HTTP redirects using a _redirects file. By placing the following in a _redirects file in the root of the web directory, one can handle redirecting all requests for files in the base level of the web site to the projects directory.

/* /projects/:splat

While this works, it does not handle my needs completely as I already have some root level URLs that I do not want to be redirected to the projects folder. As such, I need to instead add a individual redirect command for each package:

/package1 /projects/package1
/package2 /projects/package2

With this last piece in place, everything should be working. However, we have to remember to update our _redirects file every time we add a package. What if instead we could automatically generate the file for every project we have defined?

Creating Redirects File with Pipes

Hugo Pipes allow one to generate resource files based on input files, known as assets (and stored, by default, in the assets directory in the Hugo site). Different functions can operate on asset files, including resources.ToCSS to convert SASS/SCSS files to CSS and resources.Minify to minify the size of a resource. For my case, I used the resources.ExecuteAsTemplate function to autogenerate the redirects file based on a template and using the combined list of all packages from the front matter of all the projects page as the template data.

First, I wrote a short assets/redirects.tmpl file with the following:

# Redirect all defined packages from /pkg to /projects/pkg
# Also ensure any "subpackages" redirect to same /project/pkg
# Only generated for projects with a defined parameter packages
{{- range . -}}
{{- if .Params.packages -}}
{{- $dir := trim .File.Dir "/" -}}
{{ range .Params.packages }}
{{ strings.TrimPrefix "jbowen.dev" . }}        /{{ $dir }}
{{- end -}}
{{- end -}}
{{- end -}}

This template first ranges over all the passed page data and looks for those pages with a packages parameter in its front matter. For these matches it determines the current directory (eg. projects/cereal) and then reads all the values from packages and creates redirect entries for each package path to the common project page.

In order to execute the template, we need to add the following to the layouts/projects/list.html template:

<!--
    Generate _redirects file based on project pages

    {{ $redirects := resources.Get "redirects.tmpl" | resources.ExecuteAsTemplate "_redirects" .Data.Pages }}
    {{ $redirects.Permalink }}
-->

Here we first fetch the template via resources.Get then pipe into resources.ExecuteAsTemplate with the additional parameters of the output file name (relative to document root) and the data source for our template (the list of all pages in our projects section). Importantly, simply running the resources.ExecuteAsTemplate function by itself does not actually create the output file as Hugo first checks that there is a reference to it elsewhere. This is provided via the reference to $redirects.Permalink which returns the URL to the output file (and triggers the creation of said file).

Savvy readers might note that I included both template statements within HTML comment tags, <!-- ... -->. There is an interesting reason for this. Unlike a CSS file or other resource files, there are no files in our Hugo site code that actually need to use the _redirects file directly; it simply needs to be present when uploaded to Netlify. Enclosing our template code in an HTML comment allows it to be evaluated but the .Permalink output is not actually included in the final list index file, since Hugo strips HTML comments from its output files by default.

Wrapping Up

With the changes outlined above, one should have everything set up to access packages from your vanity URLs. All one should need to do is push the changes to production and then test with a simple go get command:

$ go get -v -u jbowen.dev/cereal
get "jbowen.dev/cereal": found meta tag get.metaImport{Prefix:"jbowen.dev/cereal", VCS:"git", RepoRoot:"https://github.com/jamesbo13/cereal"} at //jbowen.dev/cereal?go-get=1
jbowen.dev/cereal (download)

As the verbose (-v) output shows, go get has successfully fetched our meta tag on cereal project page and decoded successfully to the GitHub repository where it was able to download the package source. Success!

References

Remote Time Machine for Multiple Macs