go-web-starter-kit/README.md

# Go Web Starter Kit

This is a fairly complete web development starter kit in Go. It tries to be as simple as possible
without leaving out modern features. The goal is to create a setup that a normal person can learn
and study core web technologies like the DOM and web servers.  I would classify it as, "Old school
with modern things."

## The Stack

Currently I'm using the following components in a tastefully crafted stack similar to a Crepe Cake:

* [goose](https://github.com/pressly/goose) -- This manages the database migrations.
* [sqlx](https://github.com/jmoiron/sqlx) -- This is the database driver.
* [squirrel](https://github.com/Masterminds/squirrel) -- This is the SQL query generator.
* [Fiber](https://gofiber.io/) -- This is the main web API and does most everything you need.
* [tailwind](https://tailwindcss.com/) -- This makes your sites perty and is easy to use, but I
  reject the `@apply` slander and use `@apply` to give you nice things.
* [ssgod](https://lcthw.dev/go/ssgod) -- A static site generator I wrote that _only_ does static site generation and nothing else...unlike Hugo.
* [ozai](https://lcthw.dev/go/ozai) -- A process manager that replaced Air because Air went crazy like Hugo.
* [chromedp](https://github.com/chromedp/chromedp) -- This does your automated testing for you.

I then add a few little "sweetener" APIs on top of this for simple things like, making the tests
nicer or making it easier to quickly return a page from a view.

## Getting Started

First, install a couple tools that'll be used everywhere, and that have huge dependencies you don't
actually need in your project:

```shell
go install golang.org/x/pkgsite/cmd/pkgsite@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
```

To start your own project do this:

```shell
git clone https://lcthw.dev/go/go-web-starter-kit.git my-project
cd my-project
```

Then on Linux/OSX you want to delete the `.git` with this:

```shell
rm -rf .git
mv LICENSE LICENSE.old # it's MIT
```

And on Windows you use the ever more clear and totally easier:

```shell
rm -recurse -force .git
mv LICENSE LICENSE.old # it's MIT
```

Once you have that you can make it your own:

```shell
git init .
go mod tidy
cp config_example.json config.json
make migrate_up
sqlite3 db.sqlite3 ".read tools/pragmas.sql"
make
make dev
```

This gets the site running and ready for development.  You only need _to do this once._.  The next
day that you want to work, cd to your project and only run `make dev`:

```shell
make dev
```

## Linux Bug?

I found that on Ubuntu the `tailwindcss` command "wouldn't run." Turns out when `tailwindcss` forks
it does the right thing and closes all possible open file descriptors, _but_ Ubuntu has set the hard
limit to 1 billion:

```shell
$ ulimit -Hn
1073741816
```

This is a _lot_ of files to try to close, so `tailwindcss` is actually stuck doing that.  You can
fix this with:

```shell
ulimit -n 65536
```

You can pick any reasonable number, and then `tailwindcss` works like expected.

## Configuration

There's 3 basic moving parts to the whole system:  `webapp`, `ssgod`, `tailwind`, and `ozai`.

You can configure the `webapp` using the `config.json` which you copied over from `config_example.json`.  The options are fairly self-explanatory.

You can configure [ssgod](https://lcthw.dev/go/ssgod) using the
`ssgod.json` file.  You'll notice that there's _two_ layouts with this setup, one for `webapp`
and another for `ssgod`.  You can just point `ssgod` at your `views/layouts/main.html` file if you
want, but it seems people want two different layouts for their static site vs. "the app."

You can configure `tailwind` using the `static/input_style.css` and editing the `Makefile` to have
it run how you want.  I use the `@apply` heavily because I'm a programmer and no programmer would
ever tell someone they should sprinkle repetitive bullshit all over their HTML. Resist the
authority. Use `@apply`.

Finally, you can configure other processes to run with [Ozai](https://lcthw.dev/go/ozai) in the
`.ozai.json` file. Why so many different formats?  I'm still learning which is best to use and while
I like TOML I think I may go with `.json` files to reduce dependencies. You tell me though.

## Ozai Autorestarts

Ozai is a simpler tool that doesn't do auto restart.  Instead it will create a little webserver at
`http://127.0.0.1:9999` with endpoints you can hit with curl to force a restart.  Look in the
.ozai.json file to see what's configured.  This makes it easy to simple add a `curl` call in your
build process and kick everything over.  Look in the `Makefile` for how I do this.

This works better than the way `Air` does it for a few reasons:

1. You can run your build like normal in your own IDE or build tool and see your errors directly.
   When you use other autobuild tools they have the errors so you can't access them.
2. You get an immediate restart after your build succeeds, rather than waiting for an external tool
   to detect you made changes, run the build, and then restart your browser.
3. You avoid the "auto reload deadlock" where you run a build, the server is shutdown, but you hit
   refresh too soon, so now your browser waits for an error, then you have to refresh again.  Way
easier to just not use any of that.
4. No more silent failures hiding inside your terminal that you can't see.  Just manually restart
   it.

For more, checkout the [Ozai Project](https://lcthw.dev/go/ozai").

## Tour of Directories

The directories are organized in a way that separates "App Stuff" from "Content Stuff".

* `admin` <b>App</b> The code for the built-in database admin tool.
* `bin` <b>App</b> Binaries that run your app like `webapp`.
* `features` <b>App</b> Where you put your features.
* `common`<b>App</b> Common helper functions.
* `config` <b>App</b> The configuration for your site, which is `config.json` loaded into Go structs.
* `data` <b>App</b> Your database stuff.  You put your SQL models in here, and then add them to the bottom `Map` to
make them magically show up in the `/admin/table/` tool.
* `migrations` <b>App</b> When you use [goose](https://github.com/pressly/goose) it puts the migrations in here.
* `tests` <b>App</b> Your tests go in here. I use [chromedp](https://github.com/chromedp/chromedp) but the `chromedp`
API is too low level (and weird) to be useful so look in `tests/tools.go` for what I use.
* `tools` <b>App</b> Various admin automation tools go here.
* `views` <b>Content</b> This is where you put the _App_ contents, not the static content.  Sadly, this
  should be in `features/` but Fiber makes that too difficult.
* `emails` <b>Content</b> This where you put templates for emails you'll send.
* `pages` <b>Content</b> These are templates that are generated once and then served statically.
* `static` <b>Static</b> This is static content like `.js`, images, and `.css`.

The following directories are considered "build junk" and you _should not_ commit them to your git.

* `public` <b>Junk</b> This is generated by `ssgod` and then served by your `webapp`.  Don't
  edit anything in here, and instead edit in `pages` or `static`.
* `tmp` <b>Junk</b> This is made during the build/testing process.

## Dev Workflow

The first thing I do is run each of these commands in different terminals, either with tmux or just
using my fingers to run multiple tabs in PowerShell:

```shell
make dev
```

### 1. Create a Feature

The first step is to create a `feature`, which is nothing more than some files in a directory that
you edit to make your `pages/` actually work.  You can create a feature like this:

```shell
./bin/fgen -name myfeatue
```

This will create the files and directories you need to make a new feature:

`features/myfeature/`
: This contains all of the go code.

`views/myfeature/`
: This is where your HTML goes.

`tests/myfeature/`
: This is where your tests go.

Next you add it to `features/init.go` to enable it:

```diff
@@ -8,2 +8,3 @@ import (
   "MY/webapp/features/fakepay"
+  "MY/webapp/features/myfeature"
 )
@@ -15,2 +16,3 @@ func Setup(app *fiber.App) {
   features_fakepay.Setup(app)
+  features_myfeature.Setup(app)
 }
```

> __NOTE__: I'm going to automate this in the future, so expect this file to be generated.

After that run `make` to build everything again and you should have a new feature at
`http://127.0.0.1:7001/myfeature/`
with your browser.

### 2. Create Initial Views

The design of this system is that you do _not_ have to edit 16 files just get an idea going.  You
can have an idea and create a single `.html` or `.md` file in `views/myfeature` to see it happen.  This lets
you work out all of the URLs, page flows, layouts, and basic design before writing any more code.

This is generally a better way to work than creating the backend models first as it constrains what
you create to only exactly what you need for the UI you'll implement.

What I do is make a new file in `myfeature/views/`, cook up a quick 2D layout that's mostly what I want, then
make it link to other `myfeature/views/` to figure out the feature.  Once I have a rough UI for the feature
worked out I move on to the next step.

### Using the Tailwind Starter

I've included [tailwind](https://tailwindcss.com/) in this setup, since it's decent at getting
things working quickly right in the HTML.  The only thing I've fully rejected is the idea that
"@apply is bad m'kay."  No, `@apply` makes it so I don't rage at all the idiots who think 92
repeated CSS classes on every `div` is "good practice."

> __NOTE__ You can view a sample page with all of these in [/examples/sample.html](/examples/sample.html) and you can look at the other files in `examples` to get an idea of what you can do. Some of these just use tailwind, others use my starter kit.

In my setup there's a `static/input_style.css` that is used to generate a `static/style.css` and it
comes preconfigured with many of the things you'll need to get a page up and running.   In theory
you can prototype a first page with just HTML and a bit of tailwind as needed.  It also uses many of
the stock HTML tags like `aside` and `mark` so you can keep things nice.

I've added 4 additional layout tags for you to use:

* `grid` -- For stuff in a grid.  Add `class="grid-cols-2"` from tailwind to change its layout.
* `block` -- A simple pre-configured vertical flex box.  Put it around vertical stuff.
* `bar` -- A simple pre-configured horizontal flex box. Pit it around horizontal stuff.
* `stack` -- A simple way to stack stuff on top of other stuff.  You know, like in every single 2d compositional system since the 1500s.
* `shape` -- A simple 2d box to use for placeholders where you'll put images. It's a lot quicker to drop a little `shape` than to go find an image. I think it also makes it easier to focus on the design.


### 3. Create Fake Data

Create a fake set of data that comes from your page prototype.  For example, if I have a User
Profile page then I might add this:

```html
<script type="module">
  const data = {
    "Username": "Frank",
    "Email": "zed@zed.com",
  }
</script>
```

This will give you something to work with while you sort out how the interactive parts of your page
will work.  If you don't have any then you can skip this part.

### 4. Refine the UI Interactions

Using the fake data determine the user interactions for the view.  Remember that this system is
designed to minimize the amount of work you need on the front-end, so all you're doing here is
simple things like validating a form, getting data, disabling/enabling fields, and anything that
stays _on one page_.  If you find yourself trying to cram multiple pages into one then you've gone
too far.

With my simple use-profile example I could create the function that renders the table on page load:

```javascript
$boot(async() => {
  const tmpl = $id('table-row');
  const target = $id('table-items');

  for(let key in data) {
    $append(target,
      $render(tmpl, {key, value: data[key]}));
  }
});
```

This comes from my `jzed.js` file that's a minimalist old-school DOM helper. I also include
[lit.js](https://lit.dev/) if you prefer to create little components for things.

### 5. Move the Fake Data to `api.go`

You should be spending a lot of time working out the data and interactions in the nice convenience
of the page you're working on, but eventually that data needs to come from the server.  To continue
the _User Profile_ example we can create a handler in `api.go` that returns this same fake data:

```go
package features_myfeature

import (
  "github.com/gofiber/fiber/v2"
)

type UserProfile struct {
  UserName string
  Email string
}

func GetApiUserProfile(c *fiber.Ctx) error {
  profile := UserProfile{
    UserName: "Zed",
    Email: "zed@zed.com",
  }

  return c.JSON(profile)
}

func SetupApi(app *fiber.App) {
  app.Get("/api/myfeature/user_profile", GetApiUserProfile)
}
```

This is just enough code to get a simple JSON API going that we can load from the `index.html`
you've been editing.  To grab the data just do this:

```javascript
    const data = await GetJson('/api/myfeature/user_profile');
```

In place of your prototype data.  Doing that will get the data from the `api.go` code you just
wrote.

### 6. Store the Data in `db.go`

Once you've sorted out how the data is going to come from your API you can start to store it in the
database.  The first step for that is to move your data into `db.go`:

```go
package features_myfeature

type UserProfile struct {
  UserName string
  Email string
}

func GetUserProfile(id int) (UserProfile, error) {
  profile := UserProfile{
    UserName: "John",
    Email: "zed@zed.com",
  }

  return profile, nil
}
```

The `UserProfile` is removed from `api.go` and placed in `db.go`.  You then call your
`GetUserProfile` function to get it:

```go
import (
  "github.com/gofiber/fiber/v2"
  . "MY/webapp/common"
)

func GetApiUserProfile(c *fiber.Ctx) error {
  profile, err := GetUserProfile(1)
  if err != nil { return IfErrNil(err, c) }

  return c.JSON(profile)
}
```

> __NOTE__: This is intended as a simplified way to _grow_ a feature from idea to working backend.
> Eventually you would want to create a separate module for your data models, create your tests for
> them, and other important design considerations.

### 7. Move Hard Stuff to `views.go`

There's some things that are too hard to do in JavaScript, but are very easy to do in `views.go`
usign the basic templates available.  An example of this is defining the structure of a `<table>`
tag.  Doing this in JavaScript is a massive pain, but doing it with a template in `views.go` is
easy.  Another example would be setting the `id` for some data object the page should display. It's
far easier to do that right in the template.

I won't get into how this is done, but look at any of the examples in `features/` and `admin/` to
see me do this.

### 8. Put it In The Database

Once you have your data API worked out you can then use
[squirrel](https://github.com/Masterminds/squirrel), [goose](https://github.com/pressly/goose), and
[sqlx](https://github.com/jmoiron/sqlx) to make it store everything in the database.

First you create a migration for your data:

```shell
goose sqlite3 db.sqlite3 create create_user_profile sql -dir migrations
```

You then edit the `_create_user_profile.sql` file to create the new table:

```sql
-- +goose Up
-- +goose StatementBegin
CREATE TABLE user_profile (
  username TEXT
  email TEXT
);
-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin
DROP TABLE user_profile;
-- +goose StatementEnd
```

You use `goose` to migrate the database up:

```shell
goose sqlite3 db.sqlite3 -dir migrations up
```

If you make a mistake, you can migrate "down" to undo your last migration:

```shell
goose sqlite3 db.sqlite3 -dir migrations down
```

I also have convenient `Makefile` actions for this:

```shell
make migrate_up
make migrate_down
```

### 9. Rewrite `db.go` to Store It

The next step needs refinement, so I'll only briefly describe it now:

1. You have your data, so annotate it with `db: ""` struct tags.
2. Use the `squirrel` and `sqlx` APIs to store it.
3. Add your usual CRUD operations.

I'm going to improve this so most of this can be automated, so stay tuned. For an example of doing
this look in `admin/table/db.go`.

### 10. Refine, Refine, Refine

Finally, nothing in this process is meant to be a lock-step one and done operation. This is simply a
mechanism to help people go from abstract idea to concrete working code with all the stages
necessary.  Once you have everything "working" you'll need to refine it. The key to refinement is to
think about how this feature works with other features:

1. Are you repeating data that some other feature has? Just use that feature's data operations then.
2. Are you repeating components that another feature has? Try making a `lit.js` component you both use.
3. Should you move the data used by many features into a single place?  Look at `data/model.go`.

You should also go over the API and page interactions in your feature, and when it's good move onto
the next feature.  The only way to learn this step is to do it a lot, but in general, less is more.
If you've got 10k lines of code in a feature then rethink it.

## Note on The Process for Pros

This process is mostly for people who are learning how to build a web application.  Once you've done
it this way for a few web apps then you can do it however works best for your idea.

Some people's ideas are about the data, so starting with the UI like this may not work.  If your
idea is about the data, start with the `api.go` and `db.go`, and then work the tests.

Generally though, I've found that people who start with the backend are mostly only starting there
because that's what they know.  They also tend to create glorious crystal palaces that are
completely unnecessary because they aren't grounded in the reality of a UI actual people need to
use.

When you start with the UI can you can prototype all of the interactions, figure out the data, and
aim for only what you need. Keep doing this and you'll slowly build a nice data model with less
cruft for the application you're creating.

## Additional Features

## Sending Emails

> __WARNING__: The email sending feature is currently very simple. Don't think you can send 200
> million spam messages with it.

The email system uses a [Redis]() queue to schedule messages, and an email router to actually
deliver them.  The idea is that you don't want to wait for email crafting and delivery in your web
application most of the time.  Ideally what you want to do is:

1. In an `api.go` you receive a form or action from the user that needs an email. Say a password
   reset.
2. You use the `common/email` module to craft a simple message and send it to the queue. Your
   message at this stage is usually a very simple `struct` with the information your email router
   needs to build the email.
3. In your `api.go` file _immediately return rather than wait for email delivery._
4. Your email router then sits there and processes the email requests. When a new message comes in
   it takes the data, queries your database, configures the templates, and finally sends it to the
   email server.

Since email is asynchronous there's no point in your web application waiting around for all of #4 to
be done.  Just toss out a message on a queue and move on.

### Running the `mailer`

There's a simple router implemented in `tools/cmd/mailer` that you can run to get going.  You can
use this to send simple emails but you should probably write a new one that handles your more
sophisticated emails like password resets and receipts.

You can run it from the command line with:

```shell
./bin/mailer
```

It will require that you install `redis` for it to work.

### Testing Your Email Locally

Running an email server locally is kind of a pain, and setting up everything so you can view sent
emails is even harder. That's way the awesome and fabulous [MailHog](https://github.com/mailhog/MailHog) exists.

You can install it easily (it's written in Go) and then run it:

```shell
MailHog
```

By default it opens an SMTP port on `1025` and a web server at 127.0.0.1:8025. If you point your
browser at 127.0.0.:8025 you'll get a fake email inbox. This inbox will show you both the HTML and
TEXT versions of your email, plus other useful things.

### How To Send

Here's an example of sending an email from a form submission in `features/email/api.go`.

```go
package features_email

import (
  email "MY/webapp/common/email"
  "github.com/gofiber/fiber/v2"
)

func PostApiEmailSend(c *fiber.Ctx) error {
  go email.OneShotSend(email.EmailMessage{
    To: c.FormValue("To"),
    From: c.FormValue("From"),
    Subject: c.FormValue("Subject"),
    Template: c.FormValue("Template"),
  })

  return c.Redirect("/email/")
}

func SetupApi(app *fiber.App) {
  app.Post("/api/email/send", PostApiEmailSend)
}
```

### Email Templates

The templates use the same default `text/template` engine that the rest of the system uses, but you
should use Markdown to write them. This automatically gives you both a nice text format (Markdown
already is like an email) and an HTML output.  Look at `emails/signup.md` for a simple example.

## Receiving Payments

Coming soon...

## FAQ

Here's some questions I get when people see this.

### What if My Email Templates Need HTML?

You should be able to work with the code in `common/email` to make it do what you want. I don't
think it's specifically looking for any extension, but I haven't tested it.  Try just dropping the
`.html` you want into the `emails/` directory and sending it.

You should also know that really complex HTML usually get filtered, so it's best to keep it simple
HTML and Markdown will help you stick to that.

### Isn't Fiber Banned?!

First off, nobody has the right to "ban" a project.  If you're hanging out in the Go discord and
you've seen Fiber on some ban list then whoever did that is a fucking asshole.  Just ignore that
bullshit and use the best tech you can.

Second, if you're only supposed to use the Go standard library then why [is there a whole fucking
document on the official go website explaining how to use Gin to make an API?](https://go.dev/doc/tutorial/web-service-gin). It seems like whoever is claiming that you are only allowed to use the Go standard library is in disagreement with the _actual fucking Go project_.

Finally, you don't have to do what you're told, especially when the person telling you what to do is
some random asshole hiding in a chat room being a random asshole.  Use what you want, and Fiber is
in that sweet spot of being a nice API that isn't slow dogshit like [Gin](https://www.techempower.com/benchmarks/).

### Why Not Gin?

Gin is a piece of crap, that's why.  Look at the [Techempower Benchmarks](https://www.techempower.com/benchmarks/) to get an idea of how bad Gin is.  It's at the _bottom_ of the list, performing even _worse_ that some of the slowest _Ruby_ web frameworks out there.  Make a _compiled language_ like Go perform that bad takes a special kind of incompetence.

### Why Not Only the Go Standard Library?

Because I like nice things, and while the Go Standard Library has everything you _need_, it
frequently is missing things you _want_.  This project is about adding the things you'll probably
want without getting to abstract and far away from learning the core web technologies.

### Are Your Tags "Accessible"?! What about SEO?!

Yes, I actually tested all of this with [NV Access](https://www.nvaccess.org/) and the only thing NV
Access seems to have a problem with is the `pre` and `code` tags. That's because apparently those
are deemed as "images" so they get ignored...which is about the dumbest fucking shit I've ever
heard.  I'm working on a way to fix that.

Anyone telling you that my `grid`, `bar`, `block`, or `stack` tags impact accessibility most likely
has never ran NV Access one time so they're just wrong.  Additionally, it's better to have these be
_ignored_ by screen readers because they're just 2D layout junk, so they're kind of irrelevant to a
linear 1D screen reader.

Finally, SEO isn't impacted because Google has literally said HTML quality doesn't matter for SEO
ranking.  Think about it, if Google went around dictating exact HTML they would get in so much
Monopoly trouble.  Instead they tend to focus more on content and organization, and I believe the
official quote is, "Making everyone use the same HTML would make the web boring."

However, take this with a grain of salt because the same Google people who've said this also said
that backlinks didn't impact performance and many SEO experts say this is totally not true.  Do your
own testing, and if it impacts your SEO than just change them after you get your initial design up.