🐛 [Sessions] pq: invalid byte sequence for encoding "UTF8": 0xff

[hugmouse]

Fiber version

Fiber v2.7.1

Postgres version
PostgreSQL 13.2

Issue description

For some reason, when I try to save data to middleware, there is an error related to UTF8.

Code snippet

package main

import (
	"fmt"
	"github.com/gofiber/fiber/v2"
	"github.com/gofiber/fiber/v2/middleware/session"
	"github.com/gofiber/storage/postgres"
)

func main() {
	store := session.New(session.Config{
		Storage: postgres.New(postgres.Config{
			Host:     "localhost",
			Username: "postgres",
			Password: "password",
			Database: "postgres",
		}),
	})
	app := fiber.New()

	app.Get("/set", func(c *fiber.Ctx) error {
		sess, err := store.Get(c)
		if err != nil {
			panic(err)
		}
		sess.Set("key", "val")
		if err := sess.Save(); err != nil {
			panic(err)
		}
		return c.Status(fiber.StatusOK).SendString("ok")
	})

	app.Listen(":3000")
}

[ReneWerner87]

Your report is very unusual, usually you create an issie first when you are sure and can give others the possibility to work on the bug.

Please add a reproducible example and/or information about the bug, otherwise this offers no possibility to do anything here and we have to close it.

[ReneWerner87]

As I mentioned, we are closing the ticket, if you have a reproducible example and an error message, feel free to open it again

[hugmouse]

Okay, the server started up after I deleted the table with the sessions, which is created by Fiber middleware package that uses Postgres.

But now a new error: pq: invalid byte sequence for encoding "UTF8": 0xff. haha. I'm going to keep digging, so far I haven't been able to make a repeatable error.

[hugmouse]

Okay I was finally able to make a somewhat reproducible example!

How to
Use this code snippet and replace postgres.Config with your postgres database details.
Then go run this snippet and go to http://localhost:3000/set

Code snippet:

package main

import (
	"fmt"
	"github.com/gofiber/fiber/v2"
	"github.com/gofiber/fiber/v2/middleware/session"
	"github.com/gofiber/storage/postgres"
)

func main() {
	store := session.New(session.Config{
		Storage: postgres.New(postgres.Config{
			Host:     "localhost",
			Username: "postgres",
			Password: "password",
			Database: "postgres",
		}),
	})
	app := fiber.New()

	app.Get("/set", func(c *fiber.Ctx) error {
		sess, err := store.Get(c)
		if err != nil {
			panic(err)
		}
		sess.Set("key", "val")
		if err := sess.Save(); err != nil {
			panic(err)
		}
		return c.Status(fiber.StatusOK).SendString("ok")
	})

	app.Listen(":3000")
}

Log:

panic: pq: invalid byte sequence for encoding "UTF8": 0xff

goroutine 22 [running]:
main.main.func1(0xc0000ce2c0, 0xc00009c4a0, 0x4)
        /home/mysh/go/src/fiberPostgres/main.go:28 +0x186
github.com/gofiber/fiber/v2.(*App).next(0xc000138500, 0xc0000ce2c0, 0xc0000ce2c0, 0xc00fafb10c, 0xfafb10c00000230)
        /home/mysh/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:126 +0x1ab
github.com/gofiber/fiber/v2.(*App).handler(0xc000138500, 0xc0000d0000)
        /home/mysh/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:154 +0x11a
github.com/valyala/fasthttp.(*Server).serveConn(0xc000121680, 0x8b68f8, 0xc000096028, 0x0, 0x0)
        /home/mysh/go/pkg/mod/github.com/valyala/[email protected]/server.go:2168 +0x11cc
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc00010efa0, 0xc0000b87e0)
        /home/mysh/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:223 +0xba
github.com/valyala/fasthttp.(*workerPool).getCh.func1(0xc00010efa0, 0xc0000b87e0, 0x7b12a0, 0xc0000b87e0)
        /home/mysh/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x35
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /home/mysh/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:194 +0x11f
exit status 2

go.mod:

github.com/gofiber/fiber/v2 v2.7.1
github.com/gofiber/storage/postgres v0.0.0-20210331140731-4d84709369ab

[hugmouse]

btw can't reopen the Issue myself, @ReneWerner87

[hi019]

I've ran your code with my database credentials and I got the ok response. What if you create a new database instead of using the default one, postgres?

[hugmouse]

I've ran your code with my database credentials and I got the ok response. What if you create a new database instead of using the default one, postgres?

Made a new db, ran the same code and I got the same error. Also tried another machine -- Windows + Postgres, I got the same result.

I wonder what might be different between our configurations that we get different results 🤔

I'm creating a new db like this:

CREATE DATABASE fiber_utf8
    WITH 
    OWNER = postgres
    ENCODING = 'UTF8'
    CONNECTION LIMIT = -1;

Running go version go1.16.2 linux/amd64

[hugmouse]

Tried renting a VPS. Took Debian 10.9.0, installed Postgres, then Golang and tried to use my code. Got the same error.

Go: go version go1.15.9 linux/amd64
Postgres: psql (PostgreSQL) 11.11 (Debian 11.11-0+deb10u1)

[hi019]

Thanks for the info, will use a VPS to try to solve this issue tomorrow.

[puuuuh]

Thats because you try to store some blob data in TEXT column. Use BLOB column type and remove cast to string in
https://github.com/gofiber/storage/blob/main/postgres/postgres.go

[ReneWerner87]

oh, you are right

https://stackoverflow.com/questions/1942586/comparison-of-database-column-types-in-mysql-postgresql-and-sqlite-cross-map

https://stackoverflow.com/questions/53251572/golang-postgres-storing-gob-data

hmm, we should use a blob type in our storages

func (s *Storage) Set(key string, val []byte, exp time.Duration) error {

since we take bytes everywhere and don't know exactly if that includes utf8 we should try to use a binary/blob/byte type everywhere

@hi019 could you create a PR for this change

[hi019]

Nice find @puuuuh . I'll make a PR.