Handle multiple environments of the same product within one PostHog project

[Twixes]

Is your feature request related to a problem?

Currently if you're running your software in separate environments, e.g. the classic: development/staging/production, and are instrumenting analytics within it, you have two options of using PostHog:

1. Keep all data within one project and only label events with an environment (or similar in idea) property.
   This however has the downside of forcing you to add the appropriate event property everywhere, and may mess up person data.
2. Create separate projects for each environment. This provides total separation of data, but creates multiple places to set up analytics (e.g. actions, dashboards, feature flags) for the same product, which is a lot of work with flimsy results.

Describe the solution you'd like

Switching between environments within a project, keeping the setup the same, only with completely separate data.

Or… something else? Ideas welcome!

Additional context

Multiple users have been asking as about this exact same use case and so far our solutions have always been underwhelming due to the above issues.

Thank you for your feature request – we love each and every one!

[yakkomajuri]

+1 for this from a user

[Twixes]

The reason we haven't found a need for this while dogfooding is because we deploy to production continuously without staging, and in development we just run the local instance with SELF_CAPTURE. This is clearly a pain point for a number of our users though.

Probably the best solution would be an intraproject environment switch. Sentry has solved this quite well:

[MrTin]

Pitching in on our use case and hoping it can be helpful, but I don't have a strong opinion on how to go about other than just being able to separate the environments when necessary.

For analytics services -- I tend to setup an environment for dev+staging and one separate for production. The reason behind this is that in my experience, we typically use the dev+staging environment to manually make sure events are pushed and received with the correct name and the properties expected. We might play around with a funnel and set that up to verify it works but we wouldn't really make any use of the data or setup actions other than for testing.

Production is where we make it important to keep data clean from testing, where either developers or analysts go in and setup actions, dashboards, funnels etc. that we actually use every day.

In contrast, for exception tracking (Sentry/AppSignal etc.), I would setup a development, staging and production separately, keeping notifications disabled for dev and enabled for staging + prod.

[buwilliams]

Do you know of a way to achieve this without touching every query?

[EDsCODE]

I would suggest that @Twixes's (2) option is actually sufficient. Wrapping a separate project as a "test" environment would be fine because there would be some unduplicated entities such as actions, feature flags, dashboards but you'll be able to clearly recognize you're on your test environment as @paolodamico has previously mentioned. You'll also be able to clear test data freely.

If needed, I think we should add a "sync" option in the test instance that gives the option to sync the test environment with the production environment entities after the fact. This way you can clear your test instance and reload as needed

[buwilliams]

Since we already support multiple projects, I'm thinking we could extend this feature by:

• Create multiple environments for the user with a button click.
• Sync settings across environments when we create it and with button click.
• Provide a sync feature that allows a user to copy settings from another environment.

[paolodamico]

100% aligned with @EDsCODE's approach (it's what we've been discussing offline). Basically this would only be a frontend change, where we treat a special project as a test project and instead of showing the project in the regular project list, we have a test/live switch that makes it more clear. Let me know if you want some input on the design for this.

Also, to avoid confusion with internal users filters and such I would call this very explictly something like "Sandbox" mode.

In addition a few nice things that could make the experience better:

• +1 on @EDsCODE/@Twixes's sync option
• Have the API key for the test project start with "test_" or "sandbox_" to avoid any confusion there.

[paolodamico]

To your last comment @buwilliams, at first glance I think it would be better to be explicit about a test environment. I think having multiple environments could potentially cause more confusion or dilute the impact of this, but happy to discuss further.

[buwilliams]

Should we create a path around multiple projects for self-hosted being a contact sales feature?

[paolodamico]

I think we already have that, we take them to the pricing page, but if not, makes sense! Though having a test env/project I would not consider a premium feature. I'd still allow users to have the demo project, the test project and the default project.

[buwilliams]

Right exactly if I'm understanding you. We should allow people to have access to a default project AND a test project so that they get multiple environment support for free? I don't yet understand how the paywall works.

[buwilliams]

You mentioned that this is a frontend change but it seems more like a backend change. Should we automatically create the internal_[name] environment when we create the default project? Then on the frontend if we detect a internal_[name] project, we adjust the UI to show it. I'm unsure how to copy the settings between projects, that seems like a backend change as well. I think internal_ is a better name.

[buwilliams]

After a chat with @paolodamico where we uncovered some interesting approaches we landed on two:

1. Prepend api key with test_[apiKey] to automatically mark events with property test_environment=True
2. Use prop test_environment=True to track it yourself.

This means:

• Instead of a on/off toggle in the UI, we'd have a state toggle prod/test
• We are still thinking over how this will impact Filter out internal and test users toggle.
• Saves us from having to create test_[project] for each project a user has.
• The other implication is that it will only work for all new events, not retroactively.

[kpthatsme]

2c based on the demo from today–

I feel like we're giving the test toggle a lot of prominence, but in practice it doesn't strike me as something that would need to be toggled a whole lot. How much do we expect users to be using this and how often do we think teams would need to navigate between test data and production data– When this ships can we do a follow up on how often it's being used?

One thing to realize too is that when most things are in production, test environment data is usually filtered out as a side effect of law of large numbers.

Generally you will have orders of magnitude less people using a feature in test environments – most teams don't need such a hard distinction and I wonder, for those who do if this would better belong as a separate project than magic functionality on top of current projects. To me it feels a lot cleaner and easier to reason about a complete test environment, rather than a filter being applied ontop of your prod dataset.

I realize that not all teams have access to that, but IMO not out of line with our product positioning and could be a good upgrade path.

Also just semantically feels like this will get confusing with the test filter. Maybe a way to combine these for the time being would be to just add your prop test_environment=True as a rule to the currently exists 'Filter our test users' setting – nixing the global toggle.

Though, I think this too would cause issues since a lot of users are probably testing with accounts that have the exact filters that would be filtering them out of this.

[buwilliams]

@kpthatsme if you haven't taken a look at the PR some of your concerns were address there. I agree about prominence and large numbers. Using projects for environments is messy to implement and changes the meaning of project.

[Twixes]

IMO this feature should just definitely be clearly delineated from the "Test accounts" filter, by marking it as separation of Environments. Take a look at Sentry:

This is kind of the goal if we want to service large customers in the future. For now a boolean switch will be largely enough.

[paolodamico]

Yes, this should replace the current test filters UI in every insights page as it would be quite confusing. What we also need to consider is if we want to keep the existing filtering functionality based on user-defined filters.

[buwilliams]

@paolodamico @Twixes doesn't sound like you guys are saying the same thing. I agree with @Twixes that it should be separate. Remember guys, this is all behind a feature flag and a work in-progress. Let's improve the experience on the next round. And... if we all decide we'd rather remove this approach in favor of another that's possible too.

[Twixes]

@paolodamico Hm, well, I still see a point in filtering out internal users, the way I think Buddy was talking about yesterday. We wouldn't really dogfood this feature because of reasons mentioned in my original post up top, but it's been repeatedly brought up by users who do run separate dev/staging/prod environments.
As for the internal users filter, we do dogfood that pretty well: no need to slap the "Real users" cohort on everything when there's the toggle.
The use cases are very distinct, I think it's just a matter of making the distinction clear UX-wise.

[paolodamico]

Hmmm yes I see your point, I think it's worth discussing in a separate issue (let me give it some more thought too). Agreed with the approach @buwilliams !

[buwilliams]

Merged.