Updating docker compose documentation

[Inventhrice]

This is me documenting my steps as I setup a dev environment for a Windows Machine.

1. time="2024-10-22T16:20:51-04:00" level=warning msg="\\hoarder-dev\\hoarder\\docker\\docker-compose.dev.yml: version is obsolete" env file \hoarder-dev\hoarder\docker\.env not found: CreateFile \hoarder-dev\hoarder\docker\.env: The system cannot find the file specified.

The instructions on the website says to Start by copying the template by cp .env.sample .env. If we want this to be true, then we should change the docker-compose.dev.yaml to look one directory above.

Alternatively, we can add a specific instruction for docker set ups to copy the .env file into the docker directory.

2. So far I am seeing a lot of Progress: resolved 0, reused 0, downloaded 442, added 456. Will keep updated on if this is expected.

If I have understood this correctly, apparently this step is Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.0.0-alpha.8.tgz. It looks like it is installing 3121 packages. Just noting here that this is giving my laptop a RUN for it's money, fans are spinning lol.

i was right 😄

3. I am waiting for some confirmation that I can start developing.
   This came in the form of

web-1          |   ▲ Next.js 14.2.13
web-1          |   - Local:        http://localhost:3000
web-1          |
web-1          |  ✓ Starting...
web-1          | > [PWA] PWA support is disabled
web-1          | > [PWA] PWA support is disabled
web-1          |  ✓ Ready in 81.8s

Continued below

Device Details

Windows 11

Exact Hoarder Version

v0.18.0

[Inventhrice]

4. Upon navigation to localhost:3000, i got Error: Cannot open database because the directory does not exist

From /packages/db/drizzle.config.ts, we find:

const databaseURL = serverConfig.dataDir
  ? `${serverConfig.dataDir}/db.db`
  : "./db.db";

serverConfig.dataDir comes from ..\shared\config.ts. dataDir is defined at line 106 dataDir: val.DATA_DIR,, which comes from line 45 DATA_DIR: z.string().default(""), which comes from line 1 import { z } from "zod";.

what the heck is "zod".

I have DATA_DIR=../hoarder-data/ in .env, and that does not seem to have worked.

Like Mohammad said, I will have to make sure that .env gets symlink'd in each app directory (e.g. /apps/web, /apps/workers) and also /packages/db.

[MohamedBassem]

did you symlink the '.env' file to the apps/web directory? You need to do that and then run the db migrations.

[MohamedBassem]

oh, you're using the dev docker compose. Those instructions are for non-docker dev.

In the docker dev setup. The data dir should point to the dir inside the container. You can mount a volume at the same place if you want persistence.

[Inventhrice]

👀 First off, thank you for peeking in! I'm looking to update the instructions for the docker dev environment, because I was looking to contribute to issue 556, and thought I might as well document this set up while we are at it.

volumes:
  meilisearch: ./meilisearch
  data: ./data

Is this what you mean when you say "The data dir should point to the dir inside the container. You can mount a volume at the same place if you want persistence."? @MohamedBassem

UPDATE: For context, I am not immediately familiar with using volumes as a separate thing. I am looking up the documentation right now, however.

UDPATE 2: Ah I see. Per the docker compose documentation, "The db-data volume is mounted at the /var/lib/backup/data and /etc/data container paths for backup and backend respectively.". Which means that it will be mounted at data, which is... a place? I can find it in the Docker Desktop application, but I'd rather it be in a place I can locate.

UPDATE 3: I've created /docker/data and /docker/meili_search directories, and changed the docker compose dev yaml around.

UPDATE 4: I haven't seen if the above changes will work, but if it doesn't, I may forego that and instead make changes to something that works (of which I will be documenting)

UPDATE 5:

I am tackling both point 1 and point 4 by moving the .env into /docker, and including DATA_DIR=/docker/ as the data directory. I should also add a gitignore for this, just to be safe.

[Inventhrice]

Note for Disabling Sign In's as per Issue 556
There seems to be variables called

  DEMO_MODE: stringBool("false"),
  DEMO_MODE_EMAIL: z.string().optional(),
  DEMO_MODE_PASSWORD: z.string().optional(),

which could be repurposed to solve #556 ? I haven't looked into it though

[Inventhrice]

Okay, I am at a complete loss. I am getting a TypeError: Cannot open database because the directory does not exist error, which is not particularly helpful, as I do not know what it thinks the directory is. Touching base with the discord now.

Copy pasting my message in the discord

Hey gang, this is following up from #580. I have no idea how to begin debugging this error. I have an .env file in /packages/db, /app/workers, and /app/web.

I keep on getting the "Cannot open database because the directory does not exist" error. I've tried what feels like everything.
Tried adding the DATA_DIR variable with the absolute path (eg C:\Users etc) and /data, hasnt worked
Tried not including the DATA_DIR variable in the .env, no dice.

I've also tried changing the default value in /packages/shared/config.ts but also no dice 😭

I would like to know what data the program is getting from process.env in line 113 in /packages/shared/config.ts so that maybe I can also get an idea of what it is or isn't seeing. If not that, then perhaps some way to check what the database url is looking like in line 10 of /packages/db/drizzle.ts?

Help would be greatly appriciated!!

I can confirm that docker is not the issue as there exists a db.db in the persisted /data directory. (also when I execute some commands in the meilisearch container, i can see the persistant directory as well)

[Inventhrice]

Trying docker-compose -f docker/docker-compose.dev.yml up --build as suggested from the discord

THAT WORKED. OKAY THEN!! Massive shoutout to for coming in clutch there!

[Dnugs]

fresh install (this is my first try at hoarder) added to my docker-compose.yml and the web service won't start with a similar error, I'm not using the docker-compose.dev.yml file (I'm following the instructions here: https://docs.hoarder.app/Installation/docker/)

"Running db migration script
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
/db_migrations/index.js:811
throw new TypeError('Cannot open database because the directory does not exist');
^

TypeError: Cannot open database because the directory does not exist
at new Database (/db_migrations/index.js:811:9)
at /db_migrations/index.js:11751:16
at /db_migrations/index.js:11825:3
at Object. (/db_migrations/index.js:11828:12)
at Module._compile (node:internal/modules/cjs/loader:1546:14)
at Object..js (node:internal/modules/cjs/loader:1689:10)
at Module.load (node:internal/modules/cjs/loader:1318:32)
at Function._load (node:internal/modules/cjs/loader:1128:12)
at TracingChannel.traceSync (node:diagnostics_channel:315:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:218:24)

Node.js v22.11.0
s6-rc: warning: unable to start service init-db-migration: command exited 1"

[imaron85]

fresh install (this is my first try at hoarder) added to my docker-compose.yml and the web service won't start with a similar error, I'm not using the docker-compose.dev.yml file (I'm following the instructions here: https://docs.hoarder.app/Installation/docker/)

"Running db migration script s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started /db_migrations/index.js:811 throw new TypeError('Cannot open database because the directory does not exist'); ^

TypeError: Cannot open database because the directory does not exist at new Database (/db_migrations/index.js:811:9) at /db_migrations/index.js:11751:16 at /db_migrations/index.js:11825:3 at Object. (/db_migrations/index.js:11828:12) at Module._compile (node:internal/modules/cjs/loader:1546:14) at Object..js (node:internal/modules/cjs/loader:1689:10) at Module.load (node:internal/modules/cjs/loader:1318:32) at Function._load (node:internal/modules/cjs/loader:1128:12) at TracingChannel.traceSync (node:diagnostics_channel:315:14) at wrapModuleLoad (node:internal/modules/cjs/loader:218:24)

Node.js v22.11.0 s6-rc: warning: unable to start service init-db-migration: command exited 1"

I can second this. I tried to deploy a new instance to coolify via the services tab. Create the service, change .env variable and am getting the same error.

[NeoMod]

I'm leaving this here for anyone reaching this issue via google:

The "cannot open database because the directory does not exist" issue derives from editing the "DATA_DIR" env.

The default docker-compose.yml example could benefit from a comment (something like #DO NOT EDIT near the line) or maybe it's just me assuming that an ENV could be edited. Anyway, do not edit the DATA_DIR: /data declaration when deploying.

EDIT: There is actually a discussion (#917) where the issue is acknowledged, never mind my suggestion about editing documentation above.