diff --git a/README.md b/README.md index 55fa180..1a3192e 100644 --- a/README.md +++ b/README.md @@ -1,627 +1,5 @@ -# Go Web Starter Kit +# Simple Survey Site -This is a fairly complete web development starter kit in Go. It tries to be as simple as possible -without leaving out modern features. The goal is to create a setup that a normal person can learn -and study core web technologies like the DOM and web servers. I would classify it as, "Old school -with modern things." - -## The Stack - -Currently I'm using the following components in a tastefully crafted stack similar to a Crepe Cake: - -* [goose](https://github.com/pressly/goose) -- This manages the database migrations. -* [sqlx](https://github.com/jmoiron/sqlx) -- This is the database driver. -* [squirrel](https://github.com/Masterminds/squirrel) -- This is the SQL query generator. -* [Fiber](https://gofiber.io/) -- This is the main web API and does most everything you need. -* [tailwind](https://tailwindcss.com/) -- This makes your sites perty and is easy to use, but I - reject the `@apply` slander and use `@apply` to give you nice things. -* [ssgod](https://lcthw.dev/go/ssgod) -- A static site generator I wrote that _only_ does static site generation and nothing else...unlike Hugo. -* [ozai](https://lcthw.dev/go/ozai) -- A process manager that replaced Air because Air went crazy like Hugo. -* [chromedp](https://github.com/chromedp/chromedp) -- This does your automated testing for you. - -I then add a few little "sweetener" APIs on top of this for simple things like, making the tests -nicer or making it easier to quickly return a page from a view. - -## Getting Started - -First, install a couple tools that'll be used everywhere, and that have huge dependencies you don't -actually need in your project: - -```shell -go install golang.org/x/pkgsite/cmd/pkgsite@latest -go install github.com/pressly/goose/v3/cmd/goose@latest -``` - -To start your own project do this: - -```shell -git clone https://lcthw.dev/go/go-web-starter-kit.git my-project -cd my-project -``` - -Then on Linux/OSX you want to delete the `.git` with this: - -```shell -rm -rf .git -mv LICENSE LICENSE.old # it's MIT -``` - -And on Windows you use the ever more clear and totally easier: - -```shell -rm -recurse -force .git -mv LICENSE LICENSE.old # it's MIT -``` - -Once you have that you can make it your own: - -```shell -git init . -go mod tidy -cp config_example.json config.json -make migrate_up -sqlite3 db.sqlite3 ".read tools/pragmas.sql" -make -make dev -``` - -This gets the site running and ready for development. You only need _to do this once._. The next -day that you want to work, cd to your project and only run `make dev`: - -```shell -make dev -``` - -## Linux Bug? - -I found that on Ubuntu the `tailwindcss` command "wouldn't run." Turns out when `tailwindcss` forks -it does the right thing and closes all possible open file descriptors, _but_ Ubuntu has set the hard -limit to 1 billion: - -```shell -$ ulimit -Hn -1073741816 -``` - -This is a _lot_ of files to try to close, so `tailwindcss` is actually stuck doing that. You can -fix this with: - -```shell -ulimit -n 65536 -``` - -You can pick any reasonable number, and then `tailwindcss` works like expected. - -## Configuration - -There's 3 basic moving parts to the whole system: `webapp`, `ssgod`, `tailwind`, and `ozai`. - -You can configure the `webapp` using the `config.json` which you copied over from `config_example.json`. The options are fairly self-explanatory. - -You can configure [ssgod](https://lcthw.dev/go/ssgod) using the -`ssgod.json` file. You'll notice that there's _two_ layouts with this setup, one for `webapp` -and another for `ssgod`. You can just point `ssgod` at your `views/layouts/main.html` file if you -want, but it seems people want two different layouts for their static site vs. "the app." - -You can configure `tailwind` using the `static/input_style.css` and editing the `Makefile` to have -it run how you want. I use the `@apply` heavily because I'm a programmer and no programmer would -ever tell someone they should sprinkle repetitive bullshit all over their HTML. Resist the -authority. Use `@apply`. - -Finally, you can configure other processes to run with [Ozai](https://lcthw.dev/go/ozai) in the -`.ozai.json` file. Why so many different formats? I'm still learning which is best to use and while -I like TOML I think I may go with `.json` files to reduce dependencies. You tell me though. - -## Ozai Autorestarts - -Ozai is a simpler tool that doesn't do auto restart. Instead it will create a little webserver at -`http://127.0.0.1:9999` with endpoints you can hit with curl to force a restart. Look in the -.ozai.json file to see what's configured. This makes it easy to simple add a `curl` call in your -build process and kick everything over. Look in the `Makefile` for how I do this. - -This works better than the way `Air` does it for a few reasons: - -1. You can run your build like normal in your own IDE or build tool and see your errors directly. - When you use other autobuild tools they have the errors so you can't access them. -2. You get an immediate restart after your build succeeds, rather than waiting for an external tool - to detect you made changes, run the build, and then restart your browser. -3. You avoid the "auto reload deadlock" where you run a build, the server is shutdown, but you hit - refresh too soon, so now your browser waits for an error, then you have to refresh again. Way -easier to just not use any of that. -4. No more silent failures hiding inside your terminal that you can't see. Just manually restart - it. - -For more, checkout the [Ozai Project](https://lcthw.dev/go/ozai"). - -## Tour of Directories - -The directories are organized in a way that separates "App Stuff" from "Content Stuff". - -* `admin` App The code for the built-in database admin tool. -* `bin` App Binaries that run your app like `webapp`. -* `features` App Where you put your features. -* `common`App Common helper functions. -* `config` App The configuration for your site, which is `config.json` loaded into Go structs. -* `data` App Your database stuff. You put your SQL models in here, and then add them to the bottom `Map` to -make them magically show up in the `/admin/table/` tool. -* `migrations` App When you use [goose](https://github.com/pressly/goose) it puts the migrations in here. -* `tests` App Your tests go in here. I use [chromedp](https://github.com/chromedp/chromedp) but the `chromedp` -API is too low level (and weird) to be useful so look in `tests/tools.go` for what I use. -* `tools` App Various admin automation tools go here. -* `views` Content This is where you put the _App_ contents, not the static content. Sadly, this - should be in `features/` but Fiber makes that too difficult. -* `emails` Content This where you put templates for emails you'll send. -* `pages` Content These are templates that are generated once and then served statically. -* `static` Static This is static content like `.js`, images, and `.css`. - -The following directories are considered "build junk" and you _should not_ commit them to your git. - -* `public` Junk This is generated by `ssgod` and then served by your `webapp`. Don't - edit anything in here, and instead edit in `pages` or `static`. -* `tmp` Junk This is made during the build/testing process. - -## Dev Workflow - -The first thing I do is run each of these commands in different terminals, either with tmux or just -using my fingers to run multiple tabs in PowerShell: - -```shell -make dev -``` - -### 1. Create a Feature - -The first step is to create a `feature`, which is nothing more than some files in a directory that -you edit to make your `pages/` actually work. You can create a feature like this: - -```shell -./bin/fgen -name myfeatue -``` - -This will create the files and directories you need to make a new feature: - -`features/myfeature/` -: This contains all of the go code. - -`views/myfeature/` -: This is where your HTML goes. - -`tests/myfeature/` -: This is where your tests go. - -Next you add it to `features/init.go` to enable it: - -```diff -@@ -8,2 +8,3 @@ import ( - "MY/webapp/features/fakepay" -+ "MY/webapp/features/myfeature" - ) -@@ -15,2 +16,3 @@ func Setup(app *fiber.App) { - features_fakepay.Setup(app) -+ features_myfeature.Setup(app) - } -``` - -> __NOTE__: I'm going to automate this in the future, so expect this file to be generated. - -After that run `make` to build everything again and you should have a new feature at -`http://127.0.0.1:7001/myfeature/` -with your browser. - -### 2. Create Initial Views - -The design of this system is that you do _not_ have to edit 16 files just get an idea going. You -can have an idea and create a single `.html` or `.md` file in `views/myfeature` to see it happen. This lets -you work out all of the URLs, page flows, layouts, and basic design before writing any more code. - -This is generally a better way to work than creating the backend models first as it constrains what -you create to only exactly what you need for the UI you'll implement. - -What I do is make a new file in `myfeature/views/`, cook up a quick 2D layout that's mostly what I want, then -make it link to other `myfeature/views/` to figure out the feature. Once I have a rough UI for the feature -worked out I move on to the next step. - -### Using the Tailwind Starter - -I've included [tailwind](https://tailwindcss.com/) in this setup, since it's decent at getting -things working quickly right in the HTML. The only thing I've fully rejected is the idea that -"@apply is bad m'kay." No, `@apply` makes it so I don't rage at all the idiots who think 92 -repeated CSS classes on every `div` is "good practice." - -> __NOTE__ You can view a sample page with all of these in [/examples/sample.html](/examples/sample.html) and you can look at the other files in `examples` to get an idea of what you can do. Some of these just use tailwind, others use my starter kit. - -In my setup there's a `static/input_style.css` that is used to generate a `static/style.css` and it -comes preconfigured with many of the things you'll need to get a page up and running. In theory -you can prototype a first page with just HTML and a bit of tailwind as needed. It also uses many of -the stock HTML tags like `aside` and `mark` so you can keep things nice. - -I've added 4 additional layout tags for you to use: - -* `grid` -- For stuff in a grid. Add `class="grid-cols-2"` from tailwind to change its layout. -* `block` -- A simple pre-configured vertical flex box. Put it around vertical stuff. -* `bar` -- A simple pre-configured horizontal flex box. Pit it around horizontal stuff. -* `stack` -- A simple way to stack stuff on top of other stuff. You know, like in every single 2d compositional system since the 1500s. -* `shape` -- A simple 2d box to use for placeholders where you'll put images. It's a lot quicker to drop a little `shape` than to go find an image. I think it also makes it easier to focus on the design. - - -### 3. Create Fake Data - -Create a fake set of data that comes from your page prototype. For example, if I have a User -Profile page then I might add this: - -```html - -``` - -This will give you something to work with while you sort out how the interactive parts of your page -will work. If you don't have any then you can skip this part. - -### 4. Refine the UI Interactions - -Using the fake data determine the user interactions for the view. Remember that this system is -designed to minimize the amount of work you need on the front-end, so all you're doing here is -simple things like validating a form, getting data, disabling/enabling fields, and anything that -stays _on one page_. If you find yourself trying to cram multiple pages into one then you've gone -too far. - -With my simple use-profile example I could create the function that renders the table on page load: - -```javascript -$boot(async() => { - const tmpl = $id('table-row'); - const target = $id('table-items'); - - for(let key in data) { - $append(target, - $render(tmpl, {key, value: data[key]})); - } -}); -``` - -This comes from my `jzed.js` file that's a minimalist old-school DOM helper. I also include -[lit.js](https://lit.dev/) if you prefer to create little components for things. - -### 5. Move the Fake Data to `api.go` - -You should be spending a lot of time working out the data and interactions in the nice convenience -of the page you're working on, but eventually that data needs to come from the server. To continue -the _User Profile_ example we can create a handler in `api.go` that returns this same fake data: - -```go -package features_myfeature - -import ( - "github.com/gofiber/fiber/v2" -) - -type UserProfile struct { - UserName string - Email string -} - -func GetApiUserProfile(c *fiber.Ctx) error { - profile := UserProfile{ - UserName: "Zed", - Email: "zed@zed.com", - } - - return c.JSON(profile) -} - -func SetupApi(app *fiber.App) { - app.Get("/api/myfeature/user_profile", GetApiUserProfile) -} -``` - -This is just enough code to get a simple JSON API going that we can load from the `index.html` -you've been editing. To grab the data just do this: - -```javascript - const [data, status] = await $get_json('/api/myfeature/user_profile'); - if($no_error(data, status)) { - // use it - } -``` - -In place of your prototype data. Doing that will get the data from the `api.go` code you just -wrote. - -### 6. Store the Data in `db.go` - -Once you've sorted out how the data is going to come from your API you can start to store it in the -database. The first step for that is to move your data into `db.go`: - -```go -package features_myfeature - -type UserProfile struct { - UserName string - Email string -} - -func GetUserProfile(id int) (UserProfile, error) { - profile := UserProfile{ - UserName: "John", - Email: "zed@zed.com", - } - - return profile, nil -} -``` - -The `UserProfile` is removed from `api.go` and placed in `db.go`. You then call your -`GetUserProfile` function to get it: - -```go -import ( - "github.com/gofiber/fiber/v2" - . "MY/webapp/common" -) - -func GetApiUserProfile(c *fiber.Ctx) error { - profile, err := GetUserProfile(1) - if err != nil { return IfErrNil(err, c) } - - return c.JSON(profile) -} -``` - -> __NOTE__: This is intended as a simplified way to _grow_ a feature from idea to working backend. -> Eventually you would want to create a separate module for your data models, create your tests for -> them, and other important design considerations. - -### 7. Move Hard Stuff to `views.go` - -There's some things that are too hard to do in JavaScript, but are very easy to do in `views.go` -usign the basic templates available. An example of this is defining the structure of a `` -tag. Doing this in JavaScript is a massive pain, but doing it with a template in `views.go` is -easy. Another example would be setting the `id` for some data object the page should display. It's -far easier to do that right in the template. - -I won't get into how this is done, but look at any of the examples in `features/` and `admin/` to -see me do this. - -### 8. Put it In The Database - -Once you have your data API worked out you can then use -[squirrel](https://github.com/Masterminds/squirrel), [goose](https://github.com/pressly/goose), and -[sqlx](https://github.com/jmoiron/sqlx) to make it store everything in the database. - -First you create a migration for your data: - -```shell -goose sqlite3 db.sqlite3 create create_user_profile sql -dir migrations -``` - -You then edit the `_create_user_profile.sql` file to create the new table: - -```sql --- +goose Up --- +goose StatementBegin -CREATE TABLE user_profile ( - username TEXT - email TEXT -); --- +goose StatementEnd - --- +goose Down --- +goose StatementBegin -DROP TABLE user_profile; --- +goose StatementEnd -``` - -You use `goose` to migrate the database up: - -```shell -goose sqlite3 db.sqlite3 -dir migrations up -``` - -If you make a mistake, you can migrate "down" to undo your last migration: - -```shell -goose sqlite3 db.sqlite3 -dir migrations down -``` - -I also have convenient `Makefile` actions for this: - -```shell -make migrate_up -make migrate_down -``` - -### 9. Rewrite `db.go` to Store It - -The next step needs refinement, so I'll only briefly describe it now: - -1. You have your data, so annotate it with `db: ""` struct tags. -2. Use the `squirrel` and `sqlx` APIs to store it. -3. Add your usual CRUD operations. - -I'm going to improve this so most of this can be automated, so stay tuned. For an example of doing -this look in `admin/table/db.go`. - -### 10. Refine, Refine, Refine - -Finally, nothing in this process is meant to be a lock-step one and done operation. This is simply a -mechanism to help people go from abstract idea to concrete working code with all the stages -necessary. Once you have everything "working" you'll need to refine it. The key to refinement is to -think about how this feature works with other features: - -1. Are you repeating data that some other feature has? Just use that feature's data operations then. -2. Are you repeating components that another feature has? Try making a `lit.js` component you both use. -3. Should you move the data used by many features into a single place? Look at `data/model.go`. - -You should also go over the API and page interactions in your feature, and when it's good move onto -the next feature. The only way to learn this step is to do it a lot, but in general, less is more. -If you've got 10k lines of code in a feature then rethink it. - -## Note on The Process for Pros - -This process is mostly for people who are learning how to build a web application. Once you've done -it this way for a few web apps then you can do it however works best for your idea. - -Some people's ideas are about the data, so starting with the UI like this may not work. If your -idea is about the data, start with the `api.go` and `db.go`, and then work the tests. - -Generally though, I've found that people who start with the backend are mostly only starting there -because that's what they know. They also tend to create glorious crystal palaces that are -completely unnecessary because they aren't grounded in the reality of a UI actual people need to -use. - -When you start with the UI can you can prototype all of the interactions, figure out the data, and -aim for only what you need. Keep doing this and you'll slowly build a nice data model with less -cruft for the application you're creating. - -## Additional Features - -## Sending Emails - -> __WARNING__: The email sending feature is currently very simple. Don't think you can send 200 -> million spam messages with it. - -The email system uses a [Redis]() queue to schedule messages, and an email router to actually -deliver them. The idea is that you don't want to wait for email crafting and delivery in your web -application most of the time. Ideally what you want to do is: - -1. In an `api.go` you receive a form or action from the user that needs an email. Say a password - reset. -2. You use the `common/email` module to craft a simple message and send it to the queue. Your - message at this stage is usually a very simple `struct` with the information your email router - needs to build the email. -3. In your `api.go` file _immediately return rather than wait for email delivery._ -4. Your email router then sits there and processes the email requests. When a new message comes in - it takes the data, queries your database, configures the templates, and finally sends it to the - email server. - -Since email is asynchronous there's no point in your web application waiting around for all of #4 to -be done. Just toss out a message on a queue and move on. - -### Running the `mailer` - -There's a simple router implemented in `tools/cmd/mailer` that you can run to get going. You can -use this to send simple emails but you should probably write a new one that handles your more -sophisticated emails like password resets and receipts. - -You can run it from the command line with: - -```shell -./bin/mailer -``` - -It will require that you install `redis` for it to work. - -### Testing Your Email Locally - -Running an email server locally is kind of a pain, and setting up everything so you can view sent -emails is even harder. That's way the awesome and fabulous [MailHog](https://github.com/mailhog/MailHog) exists. - -You can install it easily (it's written in Go) and then run it: - -```shell -MailHog -``` - -By default it opens an SMTP port on `1025` and a web server at 127.0.0.1:8025. If you point your -browser at 127.0.0.:8025 you'll get a fake email inbox. This inbox will show you both the HTML and -TEXT versions of your email, plus other useful things. - -### How To Send - -Here's an example of sending an email from a form submission in `features/email/api.go`. - -```go -package features_email - -import ( - email "MY/webapp/common/email" - "github.com/gofiber/fiber/v2" -) - -func PostApiEmailSend(c *fiber.Ctx) error { - go email.OneShotSend(email.EmailMessage{ - To: c.FormValue("To"), - From: c.FormValue("From"), - Subject: c.FormValue("Subject"), - Template: c.FormValue("Template"), - }) - - return c.Redirect("/email/") -} - -func SetupApi(app *fiber.App) { - app.Post("/api/email/send", PostApiEmailSend) -} -``` - -### Email Templates - -The templates use the same default `text/template` engine that the rest of the system uses, but you -should use Markdown to write them. This automatically gives you both a nice text format (Markdown -already is like an email) and an HTML output. Look at `emails/signup.md` for a simple example. - -## Receiving Payments - -Coming soon... - -## FAQ - -Here's some questions I get when people see this. - -### What if My Email Templates Need HTML? - -You should be able to work with the code in `common/email` to make it do what you want. I don't -think it's specifically looking for any extension, but I haven't tested it. Try just dropping the -`.html` you want into the `emails/` directory and sending it. - -You should also know that really complex HTML usually get filtered, so it's best to keep it simple -HTML and Markdown will help you stick to that. - -### Isn't Fiber Banned?! - -First off, nobody has the right to "ban" a project. If you're hanging out in the Go discord and -you've seen Fiber on some ban list then whoever did that is a fucking asshole. Just ignore that -bullshit and use the best tech you can. - -Second, if you're only supposed to use the Go standard library then why [is there a whole fucking -document on the official go website explaining how to use Gin to make an API?](https://go.dev/doc/tutorial/web-service-gin). It seems like whoever is claiming that you are only allowed to use the Go standard library is in disagreement with the _actual fucking Go project_. - -Finally, you don't have to do what you're told, especially when the person telling you what to do is -some random asshole hiding in a chat room being a random asshole. Use what you want, and Fiber is -in that sweet spot of being a nice API that isn't slow dogshit like [Gin](https://www.techempower.com/benchmarks/). - -### Why Not Gin? - -Gin is a piece of crap, that's why. Look at the [Techempower Benchmarks](https://www.techempower.com/benchmarks/) to get an idea of how bad Gin is. It's at the _bottom_ of the list, performing even _worse_ that some of the slowest _Ruby_ web frameworks out there. Make a _compiled language_ like Go perform that bad takes a special kind of incompetence. - -### Why Not Only the Go Standard Library? - -Because I like nice things, and while the Go Standard Library has everything you _need_, it -frequently is missing things you _want_. This project is about adding the things you'll probably -want without getting to abstract and far away from learning the core web technologies. - -### Are Your Tags "Accessible"?! What about SEO?! - -Yes, I actually tested all of this with [NV Access](https://www.nvaccess.org/) and the only thing NV -Access seems to have a problem with is the `pre` and `code` tags. That's because apparently those -are deemed as "images" so they get ignored...which is about the dumbest fucking shit I've ever -heard. I'm working on a way to fix that. - -Anyone telling you that my `grid`, `bar`, `block`, or `stack` tags impact accessibility most likely -has never ran NV Access one time so they're just wrong. Additionally, it's better to have these be -_ignored_ by screen readers because they're just 2D layout junk, so they're kind of irrelevant to a -linear 1D screen reader. - -Finally, SEO isn't impacted because Google has literally said HTML quality doesn't matter for SEO -ranking. Think about it, if Google went around dictating exact HTML they would get in so much -Monopoly trouble. Instead they tend to focus more on content and organization, and I believe the -official quote is, "Making everyone use the same HTML would make the web boring." - -However, take this with a grain of salt because the same Google people who've said this also said -that backlinks didn't impact performance and many SEO experts say this is totally not true. Do your -own testing, and if it impacts your SEO than just change them after you get your initial design up. +Just a simple little site for publishing surveys for people to take. Nothing fancy. It's just +something I needed.