Skip to content

Oudwins/zog

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ZOG - A Zod & Yup like Schema Parser & Validator for GO

Go Reference License: MIT Go Report Card Coverage Status

Zog is a schema builder for runtime value parsing and validation. Define a schema, transform a value to match, assert the shape of an existing value, or both. Zog schemas are extremely expressive and allow modeling complex, interdependent validations, or value transformations.

Killer Features:

  • Concise yet expressive schema interface, equipped to model simple to complex data models
  • Zod-like API, use method chaining to build schemas in a typesafe manner
  • Extensible: add your own validators, schemas and data providers
  • Rich error details, make debugging a breeze
  • Almost no reflection when using primitive types
  • Built-in coercion support for most types
  • Zero dependencies!
  • Three Helper Packages
    • zenv: parse environment variables
    • zhttp: parse http forms & query params
    • i18n: Opinionated solution to good i18n zog errors

API Stability:

  • I will consider the API stable when we reach v1.0.0
  • However, I believe very little API changes will happen from the current implementation. The APIs are are most likely to change are the data providers (please don't make your own if possible use the helpers whose APIs will not change meaningfully) and the ParseCtx most other APIs should remain the same
  • Zog will not respect semver until v1.0.0 is released. Expect breaking changes (mainly in non basic apis) until then.

Introduction

1 Install

go get github.com/Oudwins/zog

2 Create a schema & a struct

import (
  z "github.com/Oudwins/zog"
   )

var nameSchema = z.Struct(z.Schema{
  // its very important that schema keys like "name" match the struct field name NOT the input data
  "name": z.String().Min(3, z.Message("Override default message")).Max(10),
})

var ageSchema = z.Struct(z.Schema{
  "age": z.Int().GT(18).Required(z.Message("is required")),
})

// Merge the schemas creating a new schema
var schema = nameSchema.Merge(ageSchema)

type User struct {
  Name string `zog:"firstname"` // tag is optional. If not set zog will check for "name" field in the input data
  Age int
}

3 Parse the struct

func main() {
  u := User{}
  m := map[string]string{
    "firstname": "", // won't return an error because fields are optional by default
    "age": "30", // will get casted to int
  }
  errsMap := schema.Parse(m, &u)
  if errsMap != nil {
    // handle errors -> see Errors section
  }
  u.Name // ""
  u.Age // 30
}

4 You can also parse individual fields

var t = time.Time
errsList := Time().Required().Parse("2020-01-01T00:00:00Z", &t)

5 And do stuff before and after parsing

var dest []string
Slice(String().Email().Required()).PreTransform(func(data any, ctx z.ParseCtx) (any, error) {
  s := val.(string)
  return strings.Split(s, ","), nil
}).PostTransform(func(destPtr any, ctx z.ParseCtx) error {
  s := val.(*[]string)
  for i, v := range s {
    s[i] = strings.TrimSpace(v)
  }
  return nil
}).Parse("foo@bar.com,bar@foo.com", &dest) // dest = [foo@bar.com bar@foo.com]

6 Use the zhttp package to parse JSON, Forms or Query Params

err := userSchema.Parse(zhttp.Request(r), &user)

Core Design Decisions

  • All fields optinal by default. Same as graphql
  • When parsing into structs, private fields are ignored (same as stdlib json.Unmarshal)
  • The struct parser expects a DataProvider (although if you pass something else to the data field it will try to coerce it into a DataProvider), which is an interface that wraps around an input like a map. This is less efficient than doing it directly but allows us to reuse the same code for all kinds of data sources (i.e json, query params, forms, etc).
  • Errors returned by you can be the ZogError interface or an error. If you return an error, it will be wrapped in a ZogError. ZogError is just a struct that wraps around an error and adds a message field which is is text that can be shown to the user.
  • You should not depend on test execution order. They might run in parallel in the future

A WORD OF CAUTION. ZOG & PANICS Zog will never panic due to invalid input but will always panic if invalid destination is passed to the Parse function (i.e if the destination does not match the schema).

var schema = z.Struct(z.Schema{
  "name": z.String().Required(),
})
// This struct is a valid destionation for the schema
type User struct {
  Name string
  Age int // age will be ignored since it is not a field in the schema
}
// this struct is not a valid destination for the schema. It is missing the name field. This will cause a panic even if the input data is map[string]any{"name": "zog"}
type User2 struct {
  Email string,
  Age int
}

Changes from zod:

  • Zog is Zod inspired, we adhere to the Zod API whenever possible but there are significant differences because:
    1. Go is statically typed and does not allow optional function params
    2. I have chosen to make Zog prioritize idiomatic Golang over the Zod API. Meaning some of the schemas & tests (validation rules) have changed names, z.Array() is z.Slice(), z.String().StartsWith() is z.String().HasPrefix (to follow the std lib). Etc.
    3. When I felt like a Zod method name would be confusing for Golang devs I changed it
  • Some other changes:
    • The refine method for providing a custom validation function is renamed to schema.Test()
    • schemas are optional by default (in zod they are required)
    • The z.Enum() type from zod is removed in favor of z.String().OneOf() and is only supported for strings and numbers
    • string().regex is renamed to z.String().Match() as that is in line with the regexp methods from the standard library (i.e regexp.Match and regexp.MatchString())

Limitations

Most of these things are issues we would like to address in future versions.

  • Structs do not support pointers at the moment
  • slices do not support pointers
  • maps are not a supported schema type
  • structs and slices don't support catch, and structs don't suppoort default values
  • Validations and parsing cannot be run separately
  • It is not recommended to use very deeply nested schemas since that requires a lot of reflection and can have a negative impact on performance

Helper Packages

For convenience zog provides three helper packages:

zhttp: helps parse http requests

import (
  z "github.com/Oudwins/zog"
  "github.com/Oudwins/zog/zhttp"
)
var userSchema = z.Struct(z.Schema{
  "name": z.String().Required(),
  "age": z.Int().Required().GT(18),
})

func handlePostRequest(w http.ResponseWriter, r *http.Request) {
  var user := struct {
    Name string
    Age int
  }
  // if using json (i.e json Content-Type header):
  errs := userSchema.Parse(zhttp.Request(r), &user)
  // if using form data (i.e Content-Type header = application/x-www-form-urlencoded)
  errs := userSchema.Parse(zhttp.Request(r), &user)
  // if using query params (i.e no http Content-Type header)
  errs := userSchema.Parse(zhttp.Request(r), &user)
  if errs != nil {
  }
  user.Name // defined
  user.Age // defined
}

zenv: helps validate environment variables

import (
  z "github.com/Oudwins/zog"
  "github.com/Oudwins/zog/zenv"
)

var envSchema = z.Struct(z.Schema{
	"PORT": z.Int().GT(1000).LT(65535).Default(3000),
	"Db": z.Struct(z.Schema{
		"Host": z.String().Default("localhost"),
		"User": z.String().Default("root"),
		"Pass": z.String().Default("root"),
	}),
})
var Env = struct {
	PORT int // zog will automatically coerce the PORT env to an int
	Db   struct {
		Host string `zog:"DB_HOST"` // we specify the zog tag to tell zog to parse the field from the DB_HOST environment variable
		User string `zog:"DB_USER"`
		Pass string `zog:"DB_PASS"`
	}
}{}

// Init our typesafe env vars, panic if any envs are missing
func Init() {
  errs := envSchema.Parse(zenv.NewDataProvider(), &Env)
  if errs != nil {
    log.Fatal(errs)
  }
}

// if you want to always panic on error
var Env = parse()
func Parse() env {
	var e env
	errs := envSchema.Parse(zenv.NewDataProvider(), &e)
	if errs != nil {
		fmt.Println("FAILURE TO PARSE ENV VARIABLES")
		log.Fatal(z.Errors.SanitizeMap(errs))
	}
	return e
}

zi18n: helps with having error messages in multiple languages There are two use cases for zi18n:

  1. You just want to change the language of your error messages
  2. You want to support having errors in multiple languages because your app is in 2+ languages

USE CASE 1:

// import one of our supported languages
import (
  "github.com/Oudwins/zog/i18n/es"
  "github.com/Oudwins/zog/i18n/en"
  "github.com/Oudwins/zog/conf" // import the zog configuration packag
)
// override the default error map
conf.DefaultErrMsgMap = es.Map // now all errors will be in spanish!

USE CASE 2:

// Somewhere when you start your app
import (
  "github.com/Oudwins/zog/i18n" // import the i18n library
  "github.com/Oudwins/zog/i18n/es" // import any of the supported language maps or build your own
  "github.com/Oudwins/zog/i18n/en"
)
i18n.SetLanguagesErrsMap(map[string]i18n.LangMap{
  "es": es.Map,
  "en": en.Map,
},
"es", // default language
i18n.WithLangKey("langKey"), // (optional) default lang key is "lang"
)


// Now when we parse
schema.Parse(data, &dest, z.WithCtxValue("langKey", "es")) // get spanish errors
schema.Parse(data, &dest, z.WithCtxValue("langKey", "en")) // get english errors
schema.Parse(data, &dest) // get default lang errors (spanish in this case)

Parsing Context

Zog uses a ParseCtx to pass around information related to a specific schema.Parse() call. Currently use of the parse context is quite limited but it will be expanded upon in the future.

uses

  1. Pass custom data to custom functions

Here is an example with a pretransform

nameSchema := z.String().Min(3).PreTransform(func(data any, ctx z.ParseCtx) (any, error) {
  char := ctx.Get("split_by")
  return strings.Split(data.(string), char), nil
})
nameSchema.Parse("Michael Jackson", &dest, z.WithCtxValue("split_by", " "))
  1. Change the error formatter
nameSchema := z.String().Min(3)
nameSchema.Parse(data, &dest, z.WithErrFormatter(MyCustomErrorMessageFormatter))

Errors

Zog creates its own error interface called ZogError which also implements the error interface.

type ZogError interface {
	Code() ZogErrCode // unique error code
	Value() any    // the value that caused the error
	Dtype() ZogType // type of the destionation
	Params() map[string]any // params for the error as defined by
	Message() string
	SetMessage(string)
	Error() string // returns the string representation of the ZogError. The same as String(). Printf will not print correctly otherwise.
  String() string // returns the string representation of the ZogError. Example: "ZogError{Code: '', ...}"
	Unwrap() error // returns the wrapped error.
}

This is what will be returned by the Parse function. To be precise:

  • Primitive types will return a list of ZogError instances.
  • Complex types will return a map of ZogError instances. Which uses the field path as the key & the list of errors as the value.

For example:

// ! WARNING EXAMPLES HERE ARE SHOWN AS IF ZogError IS A STRUCT FOR EASIER READING BUT IT IS AN INTERFACE
errList := z.String().Min(5).Parse("foo", &dest) // can return []z.ZogError{z.ZogError{Message: "min length is 5"}} or nil
errMap := z.Struct(z.Schema{"name": z.String().Min(5)}).Parse(data, &dest) // can return map[string][]z.ZogError{"name": []z.ZogError{{Message: "min length is 5"}}} or nil

// Slice of 2 strings with min length of 5
errsMap2 := z.Slice(z.String().Min(5)).Len(2).Parse(data, &dest) // can return map[string][]z.ZogError{"$root": []z.ZogError{{Message: "slice length is not 2"}, "[0]": []z.ZogError{{Message: "min length is 5"}}}} or nil

Additionally, z.ZogErrMap will use the field path as the key. Meaning

// ! WARNING EXAMPLES HERE ARE SHOWN AS IF ZogError IS A STRUCT FOR EASIER READING BUT IT IS AN INTERFACE
errsMap := z.Struct(z.Schema{"inner": z.Struct(z.Schema{"name": z.String().Min(5)}), "slice": z.Slice(z.String().Min(5))}).Parse(data, &dest)
errsMap["inner.name"] // will return []z.ZogError{{Message: "min length is 5"}}
errsMap["slice[0]"] // will return []z.ZogError{{Message: "min length is 5"}}

$root & $first are reserved keys for both Struct & Slice validation, they are used for root level errors and for the first error found in a schema, for xample:

errsMap := z.Slice(z.String()).Min(2).Parse(data, &dest)
errsMap["$root"] // will return []z.ZogError{{Message: "slice length is not 2"}}
errsMap["$first"] || errsMap.First() // (equivalent) will return the same in this case []z.ZogError{{Message: "slice length is not 2"}}

Sanitizing Errors

If you want to return errors to the user without the possibility of exposing internal errors, you can use the Zog sanitizer functions z.Errors.SanitizeMap(errsMap) or z.Errors.SanitizeSlice(errsSlice). These functions will return a map or slice of strings of the error messages (stripping out the internal error).

Error Formatting & i18n

Please also look at the section on the i18n package above.

You have many different options for handling the formatting of error messages. Here are a few examples:

OPTION 1: Use z.Message() or z.MessageFunc() to customize the error messages

userSchema := z.Struct(z.Schema{
  "name": z.String().Min(3, z.Message("min length is 3")),
  "age": z.Int().GT(18, z.MessageFunc(func(e z.ZogError, p z.ParseCtx) {
    e.SetMessage("age must be greater than 18")
  })),
})

This is the simplest option but it has some limitations, for example you cannot customize the error messages for coercion errors.

OPTION 2: Iterate over the returned errors and create custom messages

errs := userSchema.Parse(data, &user)
msgs := formatZogErrors(errs)

func FormatZogErrors(errs z.ZogErrMap) map[string][]string {
  // iterate over errors and create custom messages based on the error code, the params and destination type
}

OPTION 3: Override the default error formatter Under the conf package you will find ErrorFormatter which is a function that is in charge of formatting the error messages. You can override this function to customize the error messages for all zog schemas.

You can also override the DefaultErrMsgMap which is a map[zogType][ErrCode] or just specific keys to customize the error messages for specific zog types or codes.

For more information on this see the overriding defaults section

OPTION 4: Override the error formatter for a specific parsing execution

userSchema.Parse(data, &user, z.WithErrFormatter(conf.ErrorFormatter)) // set the error formatter for this parse to be the default formatter. This doesn't make a lot of sense but you can pass a custom one

What do I recommend for i18n? If you are serius about i18n and want to translate all your messages and return meaningful error messages I recommend you choose option 3 and do something like this:

zconf.ErrorFormatter = func(e p.ZogError, p z.ParseCtx) {
  lang := p.Get("lang").(string)
  // generate a different error message based on the language
}

// then when you parse a schema you can pass the language
userSchema.Parse(data, &user, z.WithCtxValue("lang", "en"))

Example ways of delivering errors to users

Using HTML templates

(In this example I'll use go templ, but you can use any template engine)

Example use case: simplified Signup form validation Imagine our handler looks like this:

type SignupFormData struct {
  Email string
  Password string
}

schema := z.Struct(z.Schema{
  "email": z.String().Email().Required(),
  "password": z.String().Min(8).Required(),
})

func handleSignup(w http.ResponseWriter, r *http.Request) {
  var signupFormData = SignupFormData{}
  errs := schema.Parse(zhttp.NewRequestDataProvider(r), &signupFormData)

  if errs != nil {
    www.Render(signupFormTempl(&signupFormData, errs))
  }
  // handle successful signup
}

templ signupFormTempl(data *SignupFormData, errs z.ZogErrMap) {
  <input type="text" name="email" value={data.Email}>
  // display only the first error
  if e, ok := errs["email"]; ok {
    <p class="error">{e[0].Message()}</p>
  }
  <input type="text" name="password" value={data.Password}>
  // display only the first error
  if e, ok := errs["password"]; ok {
    <p class="error">{e[0].Message()}</p>
  }
}

PS: If you are using go html templates & tailwindcss you might be interesting in my port of tailwind-merge to go.

REST API Responses

Zog providers a helper function called z.Errors.SanitizeMap(errsMap) that will return a map of strings of the error messages (stripping out the internal error). So, if you do not mind sending errors to your users in the same form zog returns them, you can do something like this:

errs := schema.Parse(zhttp.Request(r), &userFormData)

if errs != nil {
  sanitized := z.Errors.SanitizeMap(errs)
  // sanitize will be map[string][]string
  // for example:
  // {"name": []string{"min length is 5", "max length is 10"}, "email": []string{"is not a valid email"}}

  // ... marshal sanitized to json and send to the user
}

Creating Custom Tests

Zog supports two main ways of creating custom tests a simple function and a struct.

OPTION 1: Full Test Definition

nameSchema := z.String().Test(z.Test{
  ErrCode: "custom_error_code", // this is the error code for all errors constructed from this test
  Params: map[string]any{"foo": "bar"}, // these will be passed to the error formatter via the constructed error
  ErrFmt: func(e z.ZogError, p z.ParseCtx) // This is what z.Message() sets, its a function to format the error message
  ValidateFunc: func(data any, ctx z.ParseCtx) bool{...}, // this is the validate func
})

Beware that if using the default error formatter you will get the default error message for any error code that is not defined see error formatting. You can define the ErrFmt function or use z.Message() like so:

nameSchema := z.String().Test(MyCustomTest(), z.Message("hello world") ) // MyCustomTest() returns the z.Test so it can be reused

OPTION 2: use z.TestFunc Helper

// here I will set a custom message so I don't have to add my message to the error messages map
nameSchema := z.String().Test(z.TestFunc("error_code", func (data any, ctx z.ParseCtx)bool{...}), z.Message("custom msg"))

Reference

Generic Schema Methods

These are methods that can be used on most types of schemas

// marks the schema as required. Remember fields are optional by default
schema.Required(z.Message("message or function"))
schema.Optional() // marks the schema as optional (default)
// optional & required are mutually exclusive
schema.Required().Optional() // marks the schema as optional

schema.Default(val) // sets the default value. See Zog execution flow
schema.Catch(val) // sets the catch value. A value to use if the validation fails. See Zog execution flow

schema.PreTransform(func(data any, ctx z.ParseCtx) (any, error) {}) // transforms the value before validation. returned value will override the input value. See Zog execution flow. errors returned from this will be wrapped in a ZogError under the "custom" code (You may also return a ZogError from this function)

schema.PostTransform(func(destPtr any, ctx z.ParseCtx) error {}) // transforms the value after validation. Receives a pointer to the destination value. errors returned from this will be wrapped in a ZogError under the "custom" code (you may also return a ZogError from this function)

Types

// Primtives. Calling .Parse() on these will return []ZogError
String()
Int()
Float()
Bool()
Time()

// Complex Types. Calling .Parse() on these will return map[string][]ZogError. Where the key is the field path ("user.email") & $root is the list of complex type level errors not the specific field errors
Struct(Schema{
  "name": String(),
})
Slice(String())

Strings

// Validations
String().Min(5)
String().Max(10)
String().Len(5)
String().Email()
String().URL()
String().UUID()
String().Match()
String().Contains(string)
String().ContainsUpper()
String().ContainsDigit()
String().ContainsSpecial()
String().HasPrefix(string)
String().HasSuffix(string)
String().OneOf([]string{"a", "b", "c"})

Numbers

// Validators
Int().GT(5)
Float().GTE(5)
Int().LT(5)
Float().LTE(5)
Int().EQ(5)
Float().OneOf([]float64{1.0, 2.0, 3.0})

Booleans

Bool().True()
Bool().False()

Times & Dates

Use Time to validate time.Time instances

Time().After(time.Now())
Time().Before(time.Now())
Time().Is(time.Now())

Structs

s := z.Struct(z.Schema{
  "name": String().Required(),
  "age": Int().Required(),
})
user := struct {
  Name string `zog:"firstname"` // name will be parsed from the firstname field
  Age int // since zog tag is not set, age will be parsed from the age field
}
s.Parse(map[string]any{"firstname": "hello", "age": 10}, &user)

Slices

s := Slice(String())

Slice(Int()).Min(5)
Slice(Float()).Max(5)
Slice(Bool()).Length(5)
Slice(String()).Contains("foo")

Overriding Defaults

Zog uses internal functions to handle many aspects of validation & parsing. We aim to provide a simple way for you to customize the default behaviour of Zog through simple declarative code inside your project. You can find the options you can tweak & override in the conf package (github.com/Oudwins/zog/conf).

Overriding the default coercer functions

Lets go through an example of overriding the float64 coercer function, because we want to support floats that use a comma as the decimal separator.

import (
  // import the conf package
	"github.com/Oudwins/zog/conf"
)

// we override the coercer function for float64
conf.Coercers.Float64 = func(data any) (any, error) {
  str, ok := data.(string)
  // identify the case we want to override
  if !ok && strings.Contains(str, ",") {
    return MyCustomFloatCoercer(str)
  }
  // fallback to the original function
  return conf.DefaultCoercers.Float64(data)
}

Overriding the default error formatting

There are two different things you can override from the errors configuration:

  1. The error formatter function
  2. The error messages map. Which is a map[zogType][ErrCode] or just specific keys to customize the error messages for specific zog types or codes.
// override the error formatter function
conf.ErrorFormatter = func(e p.ZogError, p z.ParseCtx) {
  // do something with the error
  ...
  // fallback to the default error formatter
  conf.DefaultErrorFormatter(e, p)
}

// override specific error messages
// For this I recommend you import `zod/zconst` which contains zog constants
conf.DefaultErrMsgMap[zconst.TypeString]["my_custom_error_code"] = "my custom error message"
conf.DefaultErrMsgMap[zconst.TypeString][zconst.ErrCodeRequired] = "Now all required errors will get this message"

Zog Schema Parsign Execution Structure

Zog Schema Parsign Execution Structure

  1. Pretransforms
    • On error all parsing and validation stops and error is returned.
    • Can be caught by catch
  2. Default Check -> Assigns default value if the value is nil value
  3. Optional Check -> Stops validation if the value is nil value
  4. Casting -> Attempts to cast the value to the correct type
    • On error all parsing and validation stops and error is returned
    • Can be caught by catch
  5. Required check ->
    • On error: aborts if the value is its nil value and returns required error.
    • Can be caught by catch
  6. Tests -> Run all tests on the value (including required)
    • On error: validation errors are added to the errors. All validation functions are run even if one of them fails.
    • Can be caught by catch
  7. PostTransforms -> Run all postTransforms on the value.
    • On error you return: aborts and adds your error to the list of errors
    • Only run on valid values. Won't run if an error was created before the postTransforms

Roadmap

These are the things I want to add to zog before v1.0.0

  • For structs & slices: support pointers
  • Support for schema.Clone()
  • support for catch & default for structs & slices
  • Add additional tests
  • Better docs

Acknowledgments

  • Big thank you to @AlexanderArvidsson for being there to talk about architecture and design decisions. It helped a lot to have someone to bounce ideas off of
  • Credit for all the inspiration goes to /colinhacks/zod & /jquense/yup
  • Credit for the initial idea goes to anthony -> /anthdm/superkit he made a hacky version of this idea that I used as a starting point, I was never happy with it so I inspired me to rewrite it from scratch
  • Credit for the zod logo goes to /colinhacks/zod
  • To @anthonyGG for the idea to do a Zog like validation library for golang. I wouldn't have started working on it if it weren't for him

License

This project is licensed under the MIT License - see the LICENSE file for details.