Go Web Starter Kit

This is a fairly complete web development starter kit in Go. It tries to be as simple as possible without leaving out modern features. The goal is to create a setup that a normal person can learn and study core web technologies like the DOM and web servers. I would classify it as, "Old school with modern things."

The Stack

Currently I'm using the following components in a tastefully crafted stack similar to a Crepe Cake:

• goose -- This manages the database migrations.
• sqlx -- This is the database driver.
• squirrel -- This is the SQL query generator.
• Fiber -- This is the main web API and does most everything you need.
• tailwind -- This makes your sites perty and is easy to use, but I reject the @apply slander and use @apply to give you nice things.
• ssgod -- A static site generator I wrote that only does static site generation and nothing else...unlike Hugo.
• ozai -- A process manager that replaced Air because Air went crazy like Hugo.
• chromedp -- This does your automated testing for you.

I then add a few little "sweetener" APIs on top of this for simple things like, making the tests nicer or making it easier to quickly return a page from a view.

Getting Started

First, install a couple tools that'll be used everywhere, and that have huge dependencies you don't actually need in your project:

go install golang.org/x/pkgsite/cmd/pkgsite@latest
go install github.com/pressly/goose/v3/cmd/goose@latest

To start your own project do this:

git clone https://lcthw.dev/go/go-web-starter-kit.git my-project
cd my-project

Then on Linux/OSX you want to delete the .git with this:

rm -rf .git
mv LICENSE LICENSE.old # it's MIT

And on Windows you use the ever more clear and totally easier:

rm -recurse -force .git
mv LICENSE LICENSE.old # it's MIT

Once you have that you can make it your own:

git init .
go mod tidy
cp config_example.json config.json
make migrate_up
sqlite3 db.sqlite3 ".read tools/pragmas.sql"
make
make dev

This gets the site running and ready for development. You only need to do this once.. The next day that you want to work, cd to your project and only run make dev:

make dev

Linux Bug?

I found that on Ubuntu the tailwindcss command "wouldn't run." Turns out when tailwindcss forks it does the right thing and closes all possible open file descriptors, but Ubuntu has set the hard limit to 1 billion:

$ ulimit -Hn
1073741816

This is a lot of files to try to close, so tailwindcss is actually stuck doing that. You can fix this with:

ulimit -n 65536

You can pick any reasonable number, and then tailwindcss works like expected.

Configuration

There's 3 basic moving parts to the whole system: webapp, ssgod, tailwind, and ozai.

You can configure the webapp using the config.json which you copied over from config_example.json. The options are fairly self-explanatory.

You can configure ssgod using the ssgod.json file. You'll notice that there's two layouts with this setup, one for webapp and another for ssgod. You can just point ssgod at your views/layouts/main.html file if you want, but it seems people want two different layouts for their static site vs. "the app."

You can configure tailwind using the static/input_style.css and editing the Makefile to have it run how you want. I use the @apply heavily because I'm a programmer and no programmer would ever tell someone they should sprinkle repetitive bullshit all over their HTML. Resist the authority. Use @apply.

Finally, you can configure other processes to run with Ozai in the .ozai.json file. Why so many different formats? I'm still learning which is best to use and while I like TOML I think I may go with .json files to reduce dependencies. You tell me though.

Ozai Autorestarts

Ozai is a simpler tool that doesn't do auto restart. Instead it will create a little webserver at http://127.0.0.1:9999 with endpoints you can hit with curl to force a restart. Look in the .ozai.json file to see what's configured. This makes it easy to simple add a curl call in your build process and kick everything over. Look in the Makefile for how I do this.

This works better than the way Air does it for a few reasons:

1. You can run your build like normal in your own IDE or build tool and see your errors directly. When you use other autobuild tools they have the errors so you can't access them.
2. You get an immediate restart after your build succeeds, rather than waiting for an external tool to detect you made changes, run the build, and then restart your browser.
3. You avoid the "auto reload deadlock" where you run a build, the server is shutdown, but you hit refresh too soon, so now your browser waits for an error, then you have to refresh again. Way easier to just not use any of that.
4. No more silent failures hiding inside your terminal that you can't see. Just manually restart it.

For more, checkout the Ozai Project.

Tour of Directories

The directories are organized in a way that separates "App Stuff" from "Content Stuff".

• admin App The code for the built-in database admin tool.
• bin App Binaries that run your app like webapp.
• features App Where you put your features.
• commonApp Common helper functions.
• config App The configuration for your site, which is config.json loaded into Go structs.
• data App Your database stuff. You put your SQL models in here, and then add them to the bottom Map to make them magically show up in the /admin/table/ tool.
• migrations App When you use goose it puts the migrations in here.
• tests App Your tests go in here. I use chromedp but the chromedp API is too low level (and weird) to be useful so look in tests/tools.go for what I use.
• tools App Various admin automation tools go here.
• views Content This is where you put the App contents, not the static content. Sadly, this should be in features/ but Fiber makes that too difficult.
• emails Content This where you put templates for emails you'll send.
• pages Content These are templates that are generated once and then served statically.
• static Static This is static content like .js, images, and .css.

The following directories are considered "build junk" and you should not commit them to your git.

• public Junk This is generated by ssgod and then served by your webapp. Don't edit anything in here, and instead edit in pages or static.
• tmp Junk This is made during the build/testing process.

Dev Workflow

The first thing I do is run each of these commands in different terminals, either with tmux or just using my fingers to run multiple tabs in PowerShell:

make dev

1. Create a Feature

The first step is to create a feature, which is nothing more than some files in a directory that you edit to make your pages/ actually work. You can create a feature like this:

./bin/fgen -name myfeatue

This will create the files and directories you need to make a new feature:

features/myfeature/

This contains all of the go code.

views/myfeature/

This is where your HTML goes.

tests/myfeature/

This is where your tests go.

Next you add it to features/init.go to enable it:

@@ -8,2 +8,3 @@ import (
   "MY/webapp/features/fakepay"
+  "MY/webapp/features/myfeature"
 )
@@ -15,2 +16,3 @@ func Setup(app *fiber.App) {
   features_fakepay.Setup(app)
+  features_myfeature.Setup(app)
 }

NOTE: I'm going to automate this in the future, so expect this file to be generated.

After that run make to build everything again and you should have a new feature at http://127.0.0.1:7001/myfeature/ with your browser.

2. Create Initial Views

The design of this system is that you do not have to edit 16 files just get an idea going. You can have an idea and create a single .html or .md file in views/myfeature to see it happen. This lets you work out all of the URLs, page flows, layouts, and basic design before writing any more code.

This is generally a better way to work than creating the backend models first as it constrains what you create to only exactly what you need for the UI you'll implement.

What I do is make a new file in myfeature/views/, cook up a quick 2D layout that's mostly what I want, then make it link to other myfeature/views/ to figure out the feature. Once I have a rough UI for the feature worked out I move on to the next step.

Using the Tailwind Starter

I've included tailwind in this setup, since it's decent at getting things working quickly right in the HTML. The only thing I've fully rejected is the idea that "@apply is bad m'kay." No, @apply makes it so I don't rage at all the idiots who think 92 repeated CSS classes on every div is "good practice."

NOTE You can view a sample page with all of these in /examples/sample.html and you can look at the other files in examples to get an idea of what you can do. Some of these just use tailwind, others use my starter kit.

In my setup there's a static/input_style.css that is used to generate a static/style.css and it comes preconfigured with many of the things you'll need to get a page up and running. In theory you can prototype a first page with just HTML and a bit of tailwind as needed. It also uses many of the stock HTML tags like aside and mark so you can keep things nice.

I've added 4 additional layout tags for you to use:

• grid -- For stuff in a grid. Add class="grid-cols-2" from tailwind to change its layout.
• block -- A simple pre-configured vertical flex box. Put it around vertical stuff.
• bar -- A simple pre-configured horizontal flex box. Pit it around horizontal stuff.
• stack -- A simple way to stack stuff on top of other stuff. You know, like in every single 2d compositional system since the 1500s.
• shape -- A simple 2d box to use for placeholders where you'll put images. It's a lot quicker to drop a little shape than to go find an image. I think it also makes it easier to focus on the design.

3. Create Fake Data

Create a fake set of data that comes from your page prototype. For example, if I have a User Profile page then I might add this:

<script type="module">
  const data = {
    "Username": "Frank",
    "Email": "zed@zed.com",
  }
</script>

This will give you something to work with while you sort out how the interactive parts of your page will work. If you don't have any then you can skip this part.

4. Refine the UI Interactions

Using the fake data determine the user interactions for the view. Remember that this system is designed to minimize the amount of work you need on the front-end, so all you're doing here is simple things like validating a form, getting data, disabling/enabling fields, and anything that stays on one page. If you find yourself trying to cram multiple pages into one then you've gone too far.

With my simple use-profile example I could create the function that renders the table on page load:

$boot(async() => {
  const tmpl = $id('table-row');
  const target = $id('table-items');

  for(let key in data) {
    $append(target,
      $render(tmpl, {key, value: data[key]}));
  }
});

This comes from my jzed.js file that's a minimalist old-school DOM helper. I also include lit.js if you prefer to create little components for things.

5. Move the Fake Data to api.go

You should be spending a lot of time working out the data and interactions in the nice convenience of the page you're working on, but eventually that data needs to come from the server. To continue the User Profile example we can create a handler in api.go that returns this same fake data:

package features_myfeature

import (
  "github.com/gofiber/fiber/v2"
)

type UserProfile struct {
  UserName string
  Email string
}

func GetApiUserProfile(c *fiber.Ctx) error {
  profile := UserProfile{
    UserName: "Zed",
    Email: "zed@zed.com",
  }

  return c.JSON(profile)
}

func SetupApi(app *fiber.App) {
  app.Get("/api/myfeature/user_profile", GetApiUserProfile)
}

This is just enough code to get a simple JSON API going that we can load from the index.html you've been editing. To grab the data just do this:

    const data = await GetJson('/api/myfeature/user_profile');

In place of your prototype data. Doing that will get the data from the api.go code you just wrote.

6. Store the Data in db.go

Once you've sorted out how the data is going to come from your API you can start to store it in the database. The first step for that is to move your data into db.go:

package features_myfeature

type UserProfile struct {
  UserName string
  Email string
}

func GetUserProfile(id int) (UserProfile, error) {
  profile := UserProfile{
    UserName: "John",
    Email: "zed@zed.com",
  }

  return profile, nil
}

The UserProfile is removed from api.go and placed in db.go. You then call your GetUserProfile function to get it:

import (
  "github.com/gofiber/fiber/v2"
  . "MY/webapp/common"
)

func GetApiUserProfile(c *fiber.Ctx) error {
  profile, err := GetUserProfile(1)
  if err != nil { return IfErrNil(err, c) }

  return c.JSON(profile)
}

NOTE: This is intended as a simplified way to grow a feature from idea to working backend. Eventually you would want to create a separate module for your data models, create your tests for them, and other important design considerations.

7. Move Hard Stuff to views.go

There's some things that are too hard to do in JavaScript, but are very easy to do in views.go usign the basic templates available. An example of this is defining the structure of a <table> tag. Doing this in JavaScript is a massive pain, but doing it with a template in views.go is easy. Another example would be setting the id for some data object the page should display. It's far easier to do that right in the template.

I won't get into how this is done, but look at any of the examples in features/ and admin/ to see me do this.

8. Put it In The Database

Once you have your data API worked out you can then use squirrel, goose, and sqlx to make it store everything in the database.

First you create a migration for your data:

goose sqlite3 db.sqlite3 create create_user_profile sql -dir migrations

You then edit the _create_user_profile.sql file to create the new table:

-- +goose Up
-- +goose StatementBegin
CREATE TABLE user_profile (
  username TEXT
  email TEXT
);
-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin
DROP TABLE user_profile;
-- +goose StatementEnd

You use goose to migrate the database up:

goose sqlite3 db.sqlite3 -dir migrations up

If you make a mistake, you can migrate "down" to undo your last migration:

goose sqlite3 db.sqlite3 -dir migrations down

I also have convenient Makefile actions for this:

make migrate_up
make migrate_down

9. Rewrite db.go to Store It

The next step needs refinement, so I'll only briefly describe it now:

1. You have your data, so annotate it with db: "" struct tags.
2. Use the squirrel and sqlx APIs to store it.
3. Add your usual CRUD operations.

I'm going to improve this so most of this can be automated, so stay tuned. For an example of doing this look in admin/table/db.go.

10. Refine, Refine, Refine

Finally, nothing in this process is meant to be a lock-step one and done operation. This is simply a mechanism to help people go from abstract idea to concrete working code with all the stages necessary. Once you have everything "working" you'll need to refine it. The key to refinement is to think about how this feature works with other features:

1. Are you repeating data that some other feature has? Just use that feature's data operations then.
2. Are you repeating components that another feature has? Try making a lit.js component you both use.
3. Should you move the data used by many features into a single place? Look at data/model.go.

You should also go over the API and page interactions in your feature, and when it's good move onto the next feature. The only way to learn this step is to do it a lot, but in general, less is more. If you've got 10k lines of code in a feature then rethink it.

Note on The Process for Pros

This process is mostly for people who are learning how to build a web application. Once you've done it this way for a few web apps then you can do it however works best for your idea.

Some people's ideas are about the data, so starting with the UI like this may not work. If your idea is about the data, start with the api.go and db.go, and then work the tests.

Generally though, I've found that people who start with the backend are mostly only starting there because that's what they know. They also tend to create glorious crystal palaces that are completely unnecessary because they aren't grounded in the reality of a UI actual people need to use.

When you start with the UI can you can prototype all of the interactions, figure out the data, and aim for only what you need. Keep doing this and you'll slowly build a nice data model with less cruft for the application you're creating.

Additional Features

Sending Emails

WARNING: The email sending feature is currently very simple. Don't think you can send 200 million spam messages with it.

The email system uses a Redis queue to schedule messages, and an email router to actually deliver them. The idea is that you don't want to wait for email crafting and delivery in your web application most of the time. Ideally what you want to do is:

1. In an api.go you receive a form or action from the user that needs an email. Say a password reset.
2. You use the common/email module to craft a simple message and send it to the queue. Your message at this stage is usually a very simple struct with the information your email router needs to build the email.
3. In your api.go file immediately return rather than wait for email delivery.
4. Your email router then sits there and processes the email requests. When a new message comes in it takes the data, queries your database, configures the templates, and finally sends it to the email server.

Since email is asynchronous there's no point in your web application waiting around for all of #4 to be done. Just toss out a message on a queue and move on.

Running the mailer

There's a simple router implemented in tools/cmd/mailer that you can run to get going. You can use this to send simple emails but you should probably write a new one that handles your more sophisticated emails like password resets and receipts.

You can run it from the command line with:

./bin/mailer

It will require that you install redis for it to work.

Testing Your Email Locally

Running an email server locally is kind of a pain, and setting up everything so you can view sent emails is even harder. That's way the awesome and fabulous MailHog exists.

You can install it easily (it's written in Go) and then run it:

MailHog

By default it opens an SMTP port on 1025 and a web server at 127.0.0.1:8025. If you point your browser at 127.0.0.:8025 you'll get a fake email inbox. This inbox will show you both the HTML and TEXT versions of your email, plus other useful things.

How To Send

Here's an example of sending an email from a form submission in features/email/api.go.

package features_email

import (
  email "MY/webapp/common/email"
  "github.com/gofiber/fiber/v2"
)

func PostApiEmailSend(c *fiber.Ctx) error {
  go email.OneShotSend(email.EmailMessage{
    To: c.FormValue("To"),
    From: c.FormValue("From"),
    Subject: c.FormValue("Subject"),
    Template: c.FormValue("Template"),
  })

  return c.Redirect("/email/")
}

func SetupApi(app *fiber.App) {
  app.Post("/api/email/send", PostApiEmailSend)
}

Email Templates

The templates use the same default text/template engine that the rest of the system uses, but you should use Markdown to write them. This automatically gives you both a nice text format (Markdown already is like an email) and an HTML output. Look at emails/signup.md for a simple example.

Receiving Payments

Coming soon...

FAQ

Here's some questions I get when people see this.

What if My Email Templates Need HTML?

You should be able to work with the code in common/email to make it do what you want. I don't think it's specifically looking for any extension, but I haven't tested it. Try just dropping the .html you want into the emails/ directory and sending it.

You should also know that really complex HTML usually get filtered, so it's best to keep it simple HTML and Markdown will help you stick to that.

Isn't Fiber Banned?!

First off, nobody has the right to "ban" a project. If you're hanging out in the Go discord and you've seen Fiber on some ban list then whoever did that is a fucking asshole. Just ignore that bullshit and use the best tech you can.

Second, if you're only supposed to use the Go standard library then why is there a whole fucking document on the official go website explaining how to use Gin to make an API?. It seems like whoever is claiming that you are only allowed to use the Go standard library is in disagreement with the actual fucking Go project.

Finally, you don't have to do what you're told, especially when the person telling you what to do is some random asshole hiding in a chat room being a random asshole. Use what you want, and Fiber is in that sweet spot of being a nice API that isn't slow dogshit like Gin.

Why Not Gin?

Gin is a piece of crap, that's why. Look at the Techempower Benchmarks to get an idea of how bad Gin is. It's at the bottom of the list, performing even worse that some of the slowest Ruby web frameworks out there. Make a compiled language like Go perform that bad takes a special kind of incompetence.

Why Not Only the Go Standard Library?

Because I like nice things, and while the Go Standard Library has everything you need, it frequently is missing things you want. This project is about adding the things you'll probably want without getting to abstract and far away from learning the core web technologies.

Are Your Tags "Accessible"?! What about SEO?!

Yes, I actually tested all of this with NV Access and the only thing NV Access seems to have a problem with is the pre and code tags. That's because apparently those are deemed as "images" so they get ignored...which is about the dumbest fucking shit I've ever heard. I'm working on a way to fix that.

Anyone telling you that my grid, bar, block, or stack tags impact accessibility most likely has never ran NV Access one time so they're just wrong. Additionally, it's better to have these be ignored by screen readers because they're just 2D layout junk, so they're kind of irrelevant to a linear 1D screen reader.

Finally, SEO isn't impacted because Google has literally said HTML quality doesn't matter for SEO ranking. Think about it, if Google went around dictating exact HTML they would get in so much Monopoly trouble. Instead they tend to focus more on content and organization, and I believe the official quote is, "Making everyone use the same HTML would make the web boring."

However, take this with a grain of salt because the same Google people who've said this also said that backlinks didn't impact performance and many SEO experts say this is totally not true. Do your own testing, and if it impacts your SEO than just change them after you get your initial design up.