# Go Web Starter Kit This is a fairly complete web development starter kit in Go. It tries to be as simple as possible without leaving out modern features. The goal is to create a setup that a normal person can learn and study core web technologies like the DOM and web servers. I would classify it as, "Old school with modern things." ## The Stack Currently I'm using the following components in a tastefully crafted stack similar to a Crepe Cake: * [goose](https://github.com/pressly/goose) -- This manages the database migrations. * [sqlx](https://github.com/jmoiron/sqlx) -- This is the database driver. * [squirrel](https://github.com/Masterminds/squirrel) -- This is the SQL query generator. * [Fiber](https://gofiber.io/) -- This is the main web API and does most everything you need. * [tailwind](https://tailwindcss.com/) -- This makes your sites perty and is easy to use, but I reject the `@apply` slander and use `@apply` to give you nice things. * [ssgod](https://lcthw.dev/go/ssgod) -- A static site generator I wrote that _only_ does static site generation and nothing else...unlike Hugo. * [ozai](https://lcthw.dev/go/ozai) -- A process manager that replaced Air because Air went crazy like Hugo. * [chromedp](https://github.com/chromedp/chromedp) -- This does your automated testing for you. I then add a few little "sweetener" APIs on top of this for simple things like, making the tests nicer or making it easier to quickly return a page from a view. ## Getting Started First, install a couple tools that'll be used everywhere, and that have huge dependencies you don't actually need in your project: ```shell go install golang.org/x/pkgsite/cmd/pkgsite@latest go install github.com/pressly/goose/v3/cmd/goose@latest ``` To start your own project do this: ```shell git clone https://lcthw.dev/go/go-web-starter-kit.git my-project cd my-project ``` Then on Linux/OSX you want to delete the `.git` with this: ```shell rm -rf .git mv LICENSE LICENSE.old # it's MIT ``` And on Windows you use the ever more clear and totally easier: ```shell rm -recurse -force .git mv LICENSE LICENSE.old # it's MIT ``` Once you have that you can make it your own: ```shell git init . go mod tidy cp config_example.json config.json make migrate_up sqlite3 db.sqlite3 ".read tools/pragmas.sql" make make dev ``` This gets the site running and ready for development. You only need _to do this once._. The next day that you want to work, cd to your project and only run `make dev`: ```shell make dev ``` ## Linux Bug? I found that on Ubuntu the `tailwindcss` command "wouldn't run." Turns out when `tailwindcss` forks it does the right thing and closes all possible open file descriptors, _but_ Ubuntu has set the hard limit to 1 billion: ```shell $ ulimit -Hn 1073741816 ``` This is a _lot_ of files to try to close, so `tailwindcss` is actually stuck doing that. You can fix this with: ```shell ulimit -n 65536 ``` You can pick any reasonable number, and then `tailwindcss` works like expected. ## Configuration There's 3 basic moving parts to the whole system: `webapp`, `ssgod`, `tailwind`, and `ozai`. You can configure the `webapp` using the `config.json` which you copied over from `config_example.json`. The options are fairly self-explanatory. You can configure [ssgod](https://lcthw.dev/go/ssgod) using the `ssgod.json` file. You'll notice that there's _two_ layouts with this setup, one for `webapp` and another for `ssgod`. You can just point `ssgod` at your `views/layouts/main.html` file if you want, but it seems people want two different layouts for their static site vs. "the app." You can configure `tailwind` using the `static/input_style.css` and editing the `Makefile` to have it run how you want. I use the `@apply` heavily because I'm a programmer and no programmer would ever tell someone they should sprinkle repetitive bullshit all over their HTML. Resist the authority. Use `@apply`. Finally, you can configure other processes to run with [Ozai](https://lcthw.dev/go/ozai) in the `.ozai.json` file. Why so many different formats? I'm still learning which is best to use and while I like TOML I think I may go with `.json` files to reduce dependencies. You tell me though. ## Ozai Autorestarts Ozai is a simpler tool that doesn't do auto restart. Instead it will create a little webserver at `http://127.0.0.1:9999` with endpoints you can hit with curl to force a restart. Look in the .ozai.json file to see what's configured. This makes it easy to simple add a `curl` call in your build process and kick everything over. Look in the `Makefile` for how I do this. This works better than the way `Air` does it for a few reasons: 1. You can run your build like normal in your own IDE or build tool and see your errors directly. When you use other autobuild tools they have the errors so you can't access them. 2. You get an immediate restart after your build succeeds, rather than waiting for an external tool to detect you made changes, run the build, and then restart your browser. 3. You avoid the "auto reload deadlock" where you run a build, the server is shutdown, but you hit refresh too soon, so now your browser waits for an error, then you have to refresh again. Way easier to just not use any of that. 4. No more silent failures hiding inside your terminal that you can't see. Just manually restart it. For more, checkout the [Ozai Project](https://lcthw.dev/go/ozai"). ## Tour of Directories The directories are organized in a way that separates "App Stuff" from "Content Stuff". * `admin` App The code for the built-in database admin tool. * `bin` App Binaries that run your app like `webapp`. * `features` App Where you put your features. * `common`App Common helper functions. * `config` App The configuration for your site, which is `config.json` loaded into Go structs. * `data` App Your database stuff. You put your SQL models in here, and then add them to the bottom `Map` to make them magically show up in the `/admin/table/` tool. * `migrations` App When you use [goose](https://github.com/pressly/goose) it puts the migrations in here. * `tests` App Your tests go in here. I use [chromedp](https://github.com/chromedp/chromedp) but the `chromedp` API is too low level (and weird) to be useful so look in `tests/tools.go` for what I use. * `tools` App Various admin automation tools go here. * `views` Content This is where you put the _App_ contents, not the static content. Sadly, this should be in `features/` but Fiber makes that too difficult. * `emails` Content This where you put templates for emails you'll send. * `pages` Content These are templates that are generated once and then served statically. * `static` Static This is static content like `.js`, images, and `.css`. The following directories are considered "build junk" and you _should not_ commit them to your git. * `public` Junk This is generated by `ssgod` and then served by your `webapp`. Don't edit anything in here, and instead edit in `pages` or `static`. * `tmp` Junk This is made during the build/testing process. ## Dev Workflow The first thing I do is run each of these commands in different terminals, either with tmux or just using my fingers to run multiple tabs in PowerShell: ```shell make dev ``` ### 1. Create a Feature The first step is to create a `feature`, which is nothing more than some files in a directory that you edit to make your `pages/` actually work. You can create a feature like this: ```shell ./bin/fgen -name myfeatue ``` This will create the files and directories you need to make a new feature: `features/myfeature/` : This contains all of the go code. `views/myfeature/` : This is where your HTML goes. `tests/myfeature/` : This is where your tests go. Next you add it to `features/init.go` to enable it: ```diff @@ -8,2 +8,3 @@ import ( "MY/webapp/features/fakepay" + "MY/webapp/features/myfeature" ) @@ -15,2 +16,3 @@ func Setup(app *fiber.App) { features_fakepay.Setup(app) + features_myfeature.Setup(app) } ``` > __NOTE__: I'm going to automate this in the future, so expect this file to be generated. After that run `make` to build everything again and you should have a new feature at `http://127.0.0.1:7001/myfeature/` with your browser. ### 2. Create Initial Views The design of this system is that you do _not_ have to edit 16 files just get an idea going. You can have an idea and create a single `.html` or `.md` file in `views/myfeature` to see it happen. This lets you work out all of the URLs, page flows, layouts, and basic design before writing any more code. This is generally a better way to work than creating the backend models first as it constrains what you create to only exactly what you need for the UI you'll implement. What I do is make a new file in `myfeature/views/`, cook up a quick 2D layout that's mostly what I want, then make it link to other `myfeature/views/` to figure out the feature. Once I have a rough UI for the feature worked out I move on to the next step. ### Using the Tailwind Starter I've included [tailwind](https://tailwindcss.com/) in this setup, since it's decent at getting things working quickly right in the HTML. The only thing I've fully rejected is the idea that "@apply is bad m'kay." No, `@apply` makes it so I don't rage at all the idiots who think 92 repeated CSS classes on every `div` is "good practice." > __NOTE__ You can view a sample page with all of these in [/examples/sample.html](/examples/sample.html) and you can look at the other files in `examples` to get an idea of what you can do. Some of these just use tailwind, others use my starter kit. In my setup there's a `static/input_style.css` that is used to generate a `static/style.css` and it comes preconfigured with many of the things you'll need to get a page up and running. In theory you can prototype a first page with just HTML and a bit of tailwind as needed. It also uses many of the stock HTML tags like `aside` and `mark` so you can keep things nice. I've added 4 additional layout tags for you to use: * `grid` -- For stuff in a grid. Add `class="grid-cols-2"` from tailwind to change its layout. * `block` -- A simple pre-configured vertical flex box. Put it around vertical stuff. * `bar` -- A simple pre-configured horizontal flex box. Pit it around horizontal stuff. * `stack` -- A simple way to stack stuff on top of other stuff. You know, like in every single 2d compositional system since the 1500s. * `shape` -- A simple 2d box to use for placeholders where you'll put images. It's a lot quicker to drop a little `shape` than to go find an image. I think it also makes it easier to focus on the design. ### 3. Create Fake Data Create a fake set of data that comes from your page prototype. For example, if I have a User Profile page then I might add this: ```html ``` This will give you something to work with while you sort out how the interactive parts of your page will work. If you don't have any then you can skip this part. ### 4. Refine the UI Interactions Using the fake data determine the user interactions for the view. Remember that this system is designed to minimize the amount of work you need on the front-end, so all you're doing here is simple things like validating a form, getting data, disabling/enabling fields, and anything that stays _on one page_. If you find yourself trying to cram multiple pages into one then you've gone too far. With my simple use-profile example I could create the function that renders the table on page load: ```javascript $boot(async() => { const tmpl = $id('table-row'); const target = $id('table-items'); for(let key in data) { $append(target, $render(tmpl, {key, value: data[key]})); } }); ``` This comes from my `jzed.js` file that's a minimalist old-school DOM helper. I also include [lit.js](https://lit.dev/) if you prefer to create little components for things. ### 5. Move the Fake Data to `api.go` You should be spending a lot of time working out the data and interactions in the nice convenience of the page you're working on, but eventually that data needs to come from the server. To continue the _User Profile_ example we can create a handler in `api.go` that returns this same fake data: ```go package features_myfeature import ( "github.com/gofiber/fiber/v2" ) type UserProfile struct { UserName string Email string } func GetApiUserProfile(c *fiber.Ctx) error { profile := UserProfile{ UserName: "Zed", Email: "zed@zed.com", } return c.JSON(profile) } func SetupApi(app *fiber.App) { app.Get("/api/myfeature/user_profile", GetApiUserProfile) } ``` This is just enough code to get a simple JSON API going that we can load from the `index.html` you've been editing. To grab the data just do this: ```javascript const data = await GetJson('/api/myfeature/user_profile'); ``` In place of your prototype data. Doing that will get the data from the `api.go` code you just wrote. ### 6. Store the Data in `db.go` Once you've sorted out how the data is going to come from your API you can start to store it in the database. The first step for that is to move your data into `db.go`: ```go package features_myfeature type UserProfile struct { UserName string Email string } func GetUserProfile(id int) (UserProfile, error) { profile := UserProfile{ UserName: "John", Email: "zed@zed.com", } return profile, nil } ``` The `UserProfile` is removed from `api.go` and placed in `db.go`. You then call your `GetUserProfile` function to get it: ```go import ( "github.com/gofiber/fiber/v2" . "MY/webapp/common" ) func GetApiUserProfile(c *fiber.Ctx) error { profile, err := GetUserProfile(1) if err != nil { return IfErrNil(err, c) } return c.JSON(profile) } ``` > __NOTE__: This is intended as a simplified way to _grow_ a feature from idea to working backend. > Eventually you would want to create a separate module for your data models, create your tests for > them, and other important design considerations. ### 7. Move Hard Stuff to `views.go` There's some things that are too hard to do in JavaScript, but are very easy to do in `views.go` usign the basic templates available. An example of this is defining the structure of a `` tag. Doing this in JavaScript is a massive pain, but doing it with a template in `views.go` is easy. Another example would be setting the `id` for some data object the page should display. It's far easier to do that right in the template. I won't get into how this is done, but look at any of the examples in `features/` and `admin/` to see me do this. ### 8. Put it In The Database Once you have your data API worked out you can then use [squirrel](https://github.com/Masterminds/squirrel), [goose](https://github.com/pressly/goose), and [sqlx](https://github.com/jmoiron/sqlx) to make it store everything in the database. First you create a migration for your data: ```shell goose sqlite3 db.sqlite3 create create_user_profile sql -dir migrations ``` You then edit the `_create_user_profile.sql` file to create the new table: ```sql -- +goose Up -- +goose StatementBegin CREATE TABLE user_profile ( username TEXT email TEXT ); -- +goose StatementEnd -- +goose Down -- +goose StatementBegin DROP TABLE user_profile; -- +goose StatementEnd ``` You use `goose` to migrate the database up: ```shell goose sqlite3 db.sqlite3 -dir migrations up ``` If you make a mistake, you can migrate "down" to undo your last migration: ```shell goose sqlite3 db.sqlite3 -dir migrations down ``` I also have convenient `Makefile` actions for this: ```shell make migrate_up make migrate_down ``` ### 9. Rewrite `db.go` to Store It The next step needs refinement, so I'll only briefly describe it now: 1. You have your data, so annotate it with `db: ""` struct tags. 2. Use the `squirrel` and `sqlx` APIs to store it. 3. Add your usual CRUD operations. I'm going to improve this so most of this can be automated, so stay tuned. For an example of doing this look in `admin/table/db.go`. ### 10. Refine, Refine, Refine Finally, nothing in this process is meant to be a lock-step one and done operation. This is simply a mechanism to help people go from abstract idea to concrete working code with all the stages necessary. Once you have everything "working" you'll need to refine it. The key to refinement is to think about how this feature works with other features: 1. Are you repeating data that some other feature has? Just use that feature's data operations then. 2. Are you repeating components that another feature has? Try making a `lit.js` component you both use. 3. Should you move the data used by many features into a single place? Look at `data/model.go`. You should also go over the API and page interactions in your feature, and when it's good move onto the next feature. The only way to learn this step is to do it a lot, but in general, less is more. If you've got 10k lines of code in a feature then rethink it. ## Note on The Process for Pros This process is mostly for people who are learning how to build a web application. Once you've done it this way for a few web apps then you can do it however works best for your idea. Some people's ideas are about the data, so starting with the UI like this may not work. If your idea is about the data, start with the `api.go` and `db.go`, and then work the tests. Generally though, I've found that people who start with the backend are mostly only starting there because that's what they know. They also tend to create glorious crystal palaces that are completely unnecessary because they aren't grounded in the reality of a UI actual people need to use. When you start with the UI can you can prototype all of the interactions, figure out the data, and aim for only what you need. Keep doing this and you'll slowly build a nice data model with less cruft for the application you're creating. ## Additional Features ## Sending Emails > __WARNING__: The email sending feature is currently very simple. Don't think you can send 200 > million spam messages with it. The email system uses a [Redis]() queue to schedule messages, and an email router to actually deliver them. The idea is that you don't want to wait for email crafting and delivery in your web application most of the time. Ideally what you want to do is: 1. In an `api.go` you receive a form or action from the user that needs an email. Say a password reset. 2. You use the `common/email` module to craft a simple message and send it to the queue. Your message at this stage is usually a very simple `struct` with the information your email router needs to build the email. 3. In your `api.go` file _immediately return rather than wait for email delivery._ 4. Your email router then sits there and processes the email requests. When a new message comes in it takes the data, queries your database, configures the templates, and finally sends it to the email server. Since email is asynchronous there's no point in your web application waiting around for all of #4 to be done. Just toss out a message on a queue and move on. ### Running the `mailer` There's a simple router implemented in `tools/cmd/mailer` that you can run to get going. You can use this to send simple emails but you should probably write a new one that handles your more sophisticated emails like password resets and receipts. You can run it from the command line with: ```shell ./bin/mailer ``` It will require that you install `redis` for it to work. ### Testing Your Email Locally Running an email server locally is kind of a pain, and setting up everything so you can view sent emails is even harder. That's way the awesome and fabulous [MailHog](https://github.com/mailhog/MailHog) exists. You can install it easily (it's written in Go) and then run it: ```shell MailHog ``` By default it opens an SMTP port on `1025` and a web server at 127.0.0.1:8025. If you point your browser at 127.0.0.:8025 you'll get a fake email inbox. This inbox will show you both the HTML and TEXT versions of your email, plus other useful things. ### How To Send Here's an example of sending an email from a form submission in `features/email/api.go`. ```go package features_email import ( email "MY/webapp/common/email" "github.com/gofiber/fiber/v2" ) func PostApiEmailSend(c *fiber.Ctx) error { go email.OneShotSend(email.EmailMessage{ To: c.FormValue("To"), From: c.FormValue("From"), Subject: c.FormValue("Subject"), Template: c.FormValue("Template"), }) return c.Redirect("/email/") } func SetupApi(app *fiber.App) { app.Post("/api/email/send", PostApiEmailSend) } ``` ### Email Templates The templates use the same default `text/template` engine that the rest of the system uses, but you should use Markdown to write them. This automatically gives you both a nice text format (Markdown already is like an email) and an HTML output. Look at `emails/signup.md` for a simple example. ## Receiving Payments Coming soon... ## FAQ Here's some questions I get when people see this. ### What if My Email Templates Need HTML? You should be able to work with the code in `common/email` to make it do what you want. I don't think it's specifically looking for any extension, but I haven't tested it. Try just dropping the `.html` you want into the `emails/` directory and sending it. You should also know that really complex HTML usually get filtered, so it's best to keep it simple HTML and Markdown will help you stick to that. ### Isn't Fiber Banned?! First off, nobody has the right to "ban" a project. If you're hanging out in the Go discord and you've seen Fiber on some ban list then whoever did that is a fucking asshole. Just ignore that bullshit and use the best tech you can. Second, if you're only supposed to use the Go standard library then why [is there a whole fucking document on the official go website explaining how to use Gin to make an API?](https://go.dev/doc/tutorial/web-service-gin). It seems like whoever is claiming that you are only allowed to use the Go standard library is in disagreement with the _actual fucking Go project_. Finally, you don't have to do what you're told, especially when the person telling you what to do is some random asshole hiding in a chat room being a random asshole. Use what you want, and Fiber is in that sweet spot of being a nice API that isn't slow dogshit like [Gin](https://www.techempower.com/benchmarks/). ### Why Not Gin? Gin is a piece of crap, that's why. Look at the [Techempower Benchmarks](https://www.techempower.com/benchmarks/) to get an idea of how bad Gin is. It's at the _bottom_ of the list, performing even _worse_ that some of the slowest _Ruby_ web frameworks out there. Make a _compiled language_ like Go perform that bad takes a special kind of incompetence. ### Why Not Only the Go Standard Library? Because I like nice things, and while the Go Standard Library has everything you _need_, it frequently is missing things you _want_. This project is about adding the things you'll probably want without getting to abstract and far away from learning the core web technologies. ### Are Your Tags "Accessible"?! What about SEO?! Yes, I actually tested all of this with [NV Access](https://www.nvaccess.org/) and the only thing NV Access seems to have a problem with is the `pre` and `code` tags. That's because apparently those are deemed as "images" so they get ignored...which is about the dumbest fucking shit I've ever heard. I'm working on a way to fix that. Anyone telling you that my `grid`, `bar`, `block`, or `stack` tags impact accessibility most likely has never ran NV Access one time so they're just wrong. Additionally, it's better to have these be _ignored_ by screen readers because they're just 2D layout junk, so they're kind of irrelevant to a linear 1D screen reader. Finally, SEO isn't impacted because Google has literally said HTML quality doesn't matter for SEO ranking. Think about it, if Google went around dictating exact HTML they would get in so much Monopoly trouble. Instead they tend to focus more on content and organization, and I believe the official quote is, "Making everyone use the same HTML would make the web boring." However, take this with a grain of salt because the same Google people who've said this also said that backlinks didn't impact performance and many SEO experts say this is totally not true. Do your own testing, and if it impacts your SEO than just change them after you get your initial design up.