[v10] Enable building with the new UI location (#21226)

* Changes necessary to get v10 building after the webapps merge.

* move tsconfig.json to root dir, remove web makefile and dockerfile.

* Remove web/packages/build from ignored files (#20400)

* Run webassets build in Docker as current OS user (#20426)

Set the current user when running webassets builds. In this way, the build artifacts will have correct permissions.

* Add .prettierrc to teleport root (#20448)

* Add nodejs to the fips dockerfile. (#20487)

* Conditionally build the UI if there are changes. (#20489)

* Conditionally build the UI if there are changes.

`ensure-webassets` and `ensure-webassets-e` will now build the UI if the
hash-of-hashes has changed since the last time the webassets were built.

* Update Makefile

Co-authored-by: Jeff Pihach <[email protected]>

* Check for the existence of the make command.

* Choose a shasum exec from a preconfigured list or use an override.

* Shellcheck fix.

* Remove yarn.lock from sha calculation.

* Recalc SHA on successful build.

* Fix linting issues.

* Use Python instead of shasum.

* Remove unnecessary comments from sh file.

* Unconditionally rebuild if Python executable is not found.

* Revert "Unconditionally rebuild if Python executable is not found."

This reverts commit 9d2b0e3.

* Revert "Remove unnecessary comments from sh file."

This reverts commit 600e028.

* Revert use of Python, use simplified shell scripting.

* Linting fixes.

* Apply suggestions from code review

Co-authored-by: Edoardo Spadolini <[email protected]>

* Remove reference to realpath as apparently the MacOS builder doesn't have it.

* calculate_sha must return 0.

* Use echo in calculate_sha or else set -o pipefail causes failures.

* Suppress SC2005.

---------

Co-authored-by: Jeff Pihach <[email protected]>
Co-authored-by: Edoardo Spadolini <[email protected]>

* Add BUILDARCH arg to fips dockerfile. (#20495)

* Resolve issues to enable passing CI tests. (#20582)

* Move lint commands to root package.json

* Correctly resolve e-teleport paths.

* Fix prettier issues.

* move eslint configuration files to root.

* resolve lint errors.

* Update xterm-addon-fit (#20696)

This resolves an issue when starting an SSH session from the web UI,
and also fixes the out-of-date lock file.

* Move jest, eslint & prettier to the root (#20698)

* Move jest, eslint & prettier to the root

* Update e ref

* Move storybook to root package.json and fix config so e components are rendered. (#20723)

* update yarn lock.

---------

Co-authored-by: Rafał Cieślak <[email protected]>
Co-authored-by: Jakub Nyckowski <[email protected]>
Co-authored-by: Michael <[email protected]>
Co-authored-by: Michael Wilson <[email protected]>
Co-authored-by: Edoardo Spadolini <[email protected]>
Co-authored-by: Zac Bergquist <[email protected]>
Co-authored-by: Ryan Clark <[email protected]>

Author: 8 people

.drone.yml

@@ -0,0 +1,2 @@
+const eslint = require('./web/packages/build/.eslintrc');
+module.exports = eslint;

web/.eslintignore .eslintignore

@@ -20,6 +20,7 @@ darwin
 # Compiled binaries, Object files, Static and Dynamic libs (Shared Objects)
 out
 build
+!/web/packages/build
 *.o
 *.a
 *.so
@@ -82,3 +83,7 @@ go.work.sum
 
 # Buf side-effects
 /github.com
+node_modules
+
+.air.toml
+webassets/

.eslintrc.js

@@ -1,6 +1,3 @@
 [submodule "e"]
 	path = e
 	url = [email protected]:gravitational/teleport.e.git
-[submodule "webassets"]
-	path = webassets
-	url = https://github.com/gravitational/webassets.git

.gitignore

@@ -1,6 +1,6 @@
 # Ignore directories:
-packages/design/src/assets/**
-packages/**/dist/**
+*/packages/design/src/assets/**
+*/packages/**/dist/**
 
 # Ignore protobuf files:
 **/*_pb.*

.gitmodules

@@ -17,6 +17,9 @@ DOCKER_IMAGE ?= teleport
 
 GOPATH ?= $(shell go env GOPATH)
 
+# This directory will be the real path of the directory of the first Makefile in the list.
+MAKE_DIR := $(dir $(realpath $(firstword $(MAKEFILE_LIST))))
+
 # These are standard autotools variables, don't change them please
 ifneq ("$(wildcard /bin/bash)","")
 SHELL := /bin/bash -o pipefail
@@ -221,10 +224,10 @@ CLANG_FORMAT_STYLE = '{ColumnLimit: 100, IndentWidth: 4, Language: Proto}'
 #
 # 'make all' builds all 3 executables and places them in the current directory.
 #
-# IMPORTANT: the binaries will not contain the web UI assets and `teleport`
-#            won't start without setting the environment variable DEBUG=1
-#            This is the default build target for convenience of working on
-#            a web UI.
+# IMPORTANT:
+# Unless called with the `WEBASSETS_TAG` env variable set to "webassets_embed"
+# the binaries will not contain the web UI assets and `teleport` won't start
+# without setting the environment variable DEBUG=1.
 .PHONY: all
 all: version
 	@echo "---> Building OSS binaries."
@@ -324,7 +327,7 @@ endif
 # make full-ent - Builds Teleport enterprise binaries
 #
 .PHONY:full-ent
-full-ent:
+full-ent: ensure-webassets-e
 ifneq ("$(OS)", "windows")
 	@if [ -f e/Makefile ]; then \
 	rm $(ASSETS_BUILDDIR)/webassets; \
@@ -335,7 +338,7 @@ endif
 # make clean - Removes all build artifacts.
 #
 .PHONY: clean
-clean:
+clean: clean-ui
 	@echo "---> Cleaning up OSS build artifacts."
 	rm -rf $(BUILDDIR)
 	rm -rf $(ER_BPF_BUILDDIR)
@@ -348,13 +351,18 @@ clean:
 	rm -f gitref.go
 	rm -rf build.assets/tooling/bin
 
+.PHONY: clean-ui
+clean-ui:
+	rm -rf webassets/*
+	find . -type d -name node_modules -prune -exec rm -rf {} \;
+
 #
 # make release - Produces a binary release tarball.
 #
 .PHONY:
 export
 release:
-	@echo "---> $(RELEASE_MESSAGE)"
+	@echo "---> OSS $(RELEASE_MESSAGE)"
 ifeq ("$(OS)", "windows")
 	$(MAKE) --no-print-directory release-windows
 else ifeq ("$(OS)", "darwin")
@@ -1137,39 +1145,17 @@ test-compat:
 
 .PHONY: ensure-webassets
 ensure-webassets:
-	@if [ ! -d $(shell pwd)/webassets/teleport/ ]; then \
-		$(MAKE) init-webapps-submodules; \
-	fi;
+	@MAKE="$(MAKE)" "$(MAKE_DIR)/build.assets/build-webassets-if-changed.sh" OSS webassets/oss-sha build-ui web
 
 .PHONY: ensure-webassets-e
 ensure-webassets-e:
-	@if [ ! -d $(shell pwd)/webassets/e/teleport ]; then \
-		$(MAKE) init-webapps-submodules-e; \
-	fi;
-
-.PHONY: init-webapps-submodules
-init-webapps-submodules:
-	echo "init webassets submodule"
-	git submodule update --init webassets
-
-.PHONY: init-webapps-submodules-e
-init-webapps-submodules-e:
-	echo "init webassets oss and enterprise submodules"
-	git submodule update --init --recursive webassets
+	@MAKE="$(MAKE)" "$(MAKE_DIR)/build.assets/build-webassets-if-changed.sh" Enterprise webassets/e/e-sha build-ui-e web e/web
 
 .PHONY: init-submodules-e
-init-submodules-e: init-webapps-submodules-e
+init-submodules-e:
 	git submodule init e
 	git submodule update
 
-# update-webassets updates the minified code in the webassets repo using the latest webapps
-# repo and creates a PR in the teleport repo to update webassets submodule.
-.PHONY: update-webassets
-update-webassets: WEBAPPS_BRANCH ?= 'master'
-update-webassets: TELEPORT_BRANCH ?= 'master'
-update-webassets:
-	build.assets/webapps/update-teleport-webassets.sh -w $(WEBAPPS_BRANCH) -t $(TELEPORT_BRANCH)
-
 # dronegen generates .drone.yml config
 #
 #    Usage:
@@ -1187,3 +1173,19 @@ dronegen:
 .PHONY: backport
 backport:
 	(cd ./assets/backport && go run main.go -pr=$(PR) -to=$(TO))
+
+.PHONY: ensure-js-deps
+ensure-js-deps:
+	yarn install --ignore-scripts
+
+.PHONY: build-ui
+build-ui: ensure-js-deps
+	yarn build-ui-oss
+
+.PHONY: build-ui-e
+build-ui-e: ensure-js-deps
+	yarn build-ui-e
+
+.PHONY: docker-ui
+docker-ui:
+	$(MAKE) -C build.assets ui

web/.prettierignore .prettierignore

@@ -42,21 +42,21 @@ SSH, HTTPS, RDP, Kubernetes API, MySQL, MongoDB and PostgreSQL wire protocols.
 On the server-side, Teleport is a single binary which enables convenient secure
 access to behind-NAT resources such as:
 
-* [SSH nodes](https://goteleport.com/docs/getting-started/) - SSH works in browsers too!
-* [Kubernetes clusters](https://goteleport.com/docs/kubernetes-access/introduction/)
-* [PostgreSQL, MongoDB, CockroachDB and MySQL databases](https://goteleport.com/docs/database-access/introduction/)
-* [Internal Web apps](https://goteleport.com/docs/application-access/introduction/)
-* [Windows Hosts](https://goteleport.com/docs/desktop-access/introduction/)
-* [Networked servers](https://goteleport.com/docs/server-access/introduction/)
+- [SSH nodes](https://goteleport.com/docs/getting-started/) - SSH works in browsers too!
+- [Kubernetes clusters](https://goteleport.com/docs/kubernetes-access/introduction/)
+- [PostgreSQL, MongoDB, CockroachDB and MySQL databases](https://goteleport.com/docs/database-access/introduction/)
+- [Internal Web apps](https://goteleport.com/docs/application-access/introduction/)
+- [Windows Hosts](https://goteleport.com/docs/desktop-access/introduction/)
+- [Networked servers](https://goteleport.com/docs/server-access/introduction/)
 
 Teleport is trivial to set up as a Linux daemon or in a Kubernetes pod. It's rapidly
 replacing legacy `sshd`-based setups at organizations who need:
 
-* Developer convenience of having instant secure access to everything they need
+- Developer convenience of having instant secure access to everything they need
   across many environments and cloud providers.
-* Audit log with session recording/replay for multiple protocols
-* Easily manage trust between teams, organizations and data centers.
-* Role-based access control (RBAC) and flexible access workflows (one-time [access requests](https://goteleport.com/features/access-requests/))
+- Audit log with session recording/replay for multiple protocols
+- Easily manage trust between teams, organizations and data centers.
+- Role-based access control (RBAC) and flexible access workflows (one-time [access requests](https://goteleport.com/features/access-requests/))
 
 In addition to its hallmark features, Teleport is interesting for smaller teams
 because it facilitates easy adoption of the best infrastructure security
@@ -72,16 +72,15 @@ Teleport is built upon the high-quality [Golang SSH](https://godoc.org/golang.or
 implementation. It is _fully compatible with OpenSSH_,
 `sshd` servers, and `ssh` clients.
 
-|Project Links| Description
-|---|----
-| [Teleport Website](https://goteleport.com/) | The official website of the project. |
-| [Documentation](https://goteleport.com/docs/) | Admin guide, user manual and more. |
-| [Demo Video](https://www.youtube.com/watch?v=b1WHFW0NIoM) | 3-minute video overview of Teleport. |
-| [Blog](https://goteleport.com/blog/) | Our blog where we publish Teleport news. |
-| [Forum](https://github.com/gravitational/teleport/discussions) | Ask us a setup question, post your tutorial, feedback, or idea on our forum. |
-| [Slack](https://goteleport.com/slack) | Need help with your setup? Ping us in our Slack channel. |
-| [Cloud-hosted](https://goteleport.com/pricing) | We offer Enterprise with a Cloud-hosted option. For teams that require easy and secure access to their computing environments. |
-
+| Project Links                                                  | Description                                                                                                                    |
+| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| [Teleport Website](https://goteleport.com/)                    | The official website of the project.                                                                                           |
+| [Documentation](https://goteleport.com/docs/)                  | Admin guide, user manual and more.                                                                                             |
+| [Demo Video](https://www.youtube.com/watch?v=b1WHFW0NIoM)      | 3-minute video overview of Teleport.                                                                                           |
+| [Blog](https://goteleport.com/blog/)                           | Our blog where we publish Teleport news.                                                                                       |
+| [Forum](https://github.com/gravitational/teleport/discussions) | Ask us a setup question, post your tutorial, feedback, or idea on our forum.                                                   |
+| [Slack](https://goteleport.com/slack)                          | Need help with your setup? Ping us in our Slack channel.                                                                       |
+| [Cloud-hosted](https://goteleport.com/pricing)                 | We offer Enterprise with a Cloud-hosted option. For teams that require easy and secure access to their computing environments. |
 
 ## Installing and Running
 
@@ -101,7 +100,7 @@ In a production environment, Teleport must run as `root`. For testing or non-pro
 
 `chown $USER /var/lib/teleport`
 
-* In this case, you will not be able to log in as another user.
+- In this case, you will not be able to log in as another user.
 
 ## Docker
 
@@ -110,10 +109,12 @@ In a production environment, Teleport must run as `root`. For testing or non-pro
 ### Deploy Teleport
 
 If you wish to deploy Teleport inside a Docker container:
+
 ```
 # This command will pull the Teleport container image for version 8
 docker pull public.ecr.aws/gravitational/teleport:8
 ```
+
 View latest tags on [Amazon ECR Public | gravitational/teleport](https://gallery.ecr.aws/gravitational/teleport)
 
 ### For Local Testing and Development
@@ -127,8 +128,8 @@ and a web UI written in Javascript (a git submodule located in the `webassets/`
 directory).
 
 If your intention is to build and deploy for use in a production infrastructure
-a released tag should be used.  The default branch, `master`, is the current
-development branch for an upcoming major version.  Get the latest release tags
+a released tag should be used. The default branch, `master`, is the current
+development branch for an upcoming major version. Get the latest release tags
 listed at https://goteleport.com/download/ and then use that tag in the `git clone`.
 For example `git clone https://github.com/gravitational/teleport.git -b v9.1.2` gets release v9.1.2.
 
@@ -164,32 +165,29 @@ If the build succeeds, the installer will place the binaries in the `build` dire
 
 **Important:**
 
-* The Go compiler is somewhat sensitive to the amount of memory: you will need
+- The Go compiler is somewhat sensitive to the amount of memory: you will need
   **at least** 1GB of virtual memory to compile Teleport. A 512MB instance
   without swap will **not** work.
-* This will build the latest version of Teleport, **regardless** of whether it
+- This will build the latest version of Teleport, **regardless** of whether it
   is stable. If you want to build the latest stable release, run `git checkout`
   to the corresponding tag (for example, run `git checkout v8.0.0`) **before**
   running `make full`.
 
 ### Web UI
 
-The Teleport Web UI resides in the [Gravitational Webapps](https://github.com/gravitational/webapps) repo.
+The Teleport Web UI resides in the [web](web) directory.
 
 #### Rebuilding Web UI for development
 
-To clone this repository and rebuild the Teleport UI package, run the following commands:
+To rebuild the Teleport UI package, run the following command:
 
 ```bash
-$ git clone [email protected]:gravitational/webapps.git
-$ cd webapps
-$ make build-teleport
+make docker-ui
 ```
 
 Then you can replace Teleport Web UI files with the files from the newly-generated `/dist` folder.
 
-To enable speedy iterations on the Web UI, you can run a
-[local web-dev server](https://github.com/gravitational/webapps/tree/master/packages/teleport).
+To enable speedy iterations on the Web UI, you can run a [local web-dev server](web#web-ui)
 
 You can also tell Teleport to load the Web UI assets from the source directory.
 To enable this behavior, set the environment variable `DEBUG=1` and rebuild with the default target:
@@ -200,7 +198,7 @@ $ DEBUG=1 ./build/teleport start -d
 ```
 
 Keep the server running in this mode, and make your UI changes in `/dist` directory.
-For instructions about how to update the Web UI, read [the `webapps` README](https://github.com/gravitational/webapps/blob/master/README.md.) file.
+For instructions about how to update the Web UI, read [the `web` README](web#readme).
 
 #### Updating Web UI assets
 
@@ -228,7 +226,6 @@ go get github.com/new/dependency
 
 and update the source to use this dependency.
 
-
 To get a specific version, use `go get github.com/new/dependency@version` instead.
 
 #### Set dependency to a specific version
@@ -271,18 +268,18 @@ We had a choice, either start a security consulting business or build a solution
 
 ## More Information
 
-* [Teleport Getting Started](https://goteleport.com/docs/getting-started/)
-* [Teleport Architecture](https://goteleport.com/teleport/docs/architecture)
-* [Reference](https://goteleport.com/docs/setup/reference/)
-* [FAQ](https://goteleport.com/docs/faq)
+- [Teleport Getting Started](https://goteleport.com/docs/getting-started/)
+- [Teleport Architecture](https://goteleport.com/teleport/docs/architecture)
+- [Reference](https://goteleport.com/docs/setup/reference/)
+- [FAQ](https://goteleport.com/docs/faq)
 
 ## Support and Contributing
 
 We offer a few different options for support. First of all, we try to provide clear and comprehensive documentation. The docs are also in Github, so feel free to create a PR or file an issue if you have ideas for improvements. If you still have questions after reviewing our docs, you can also:
 
-* Join [Teleport Discussions](https://github.com/gravitational/teleport/discussions) to ask questions. Our engineers are available there to help you.
-* If you want to contribute to Teleport or file a bug report/issue, you can create an issue here in Github.
-* If you are interested in Teleport Enterprise or more responsive support during a POC, we can also create a dedicated Slack channel for you during your POC. You can [reach out to us through our website](https://goteleport.com/pricing/) to arrange for a POC.
+- Join [Teleport Discussions](https://github.com/gravitational/teleport/discussions) to ask questions. Our engineers are available there to help you.
+- If you want to contribute to Teleport or file a bug report/issue, you can create an issue here in Github.
+- If you are interested in Teleport Enterprise or more responsive support during a POC, we can also create a dedicated Slack channel for you during your POC. You can [reach out to us through our website](https://goteleport.com/pricing/) to arrange for a POC.
 
 ## Is Teleport Secure and Production Ready?
 

web/.prettierrc .prettierrc

@@ -1,5 +1,5 @@
 const baseCfg = require('@gravitational/build/.babelrc');
-module.exports = function(api) {
+module.exports = function (api) {
   api.cache(true);
   return baseCfg;
 };

Makefile

@@ -67,6 +67,20 @@ ENV GOEXPERIMENT=boringcrypto \
     GOROOT="/opt/go" \
     PATH="$PATH:/opt/go/bin:/go/bin:/go/src/github.com/gravitational/teleport/build"
 
+ARG BUILDARCH
+
+# Install Nodejs
+ARG NODE_VERSION
+ENV NODE_URL="https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${BUILDARCH}.tar.xz"
+ENV NODE_PATH="/usr/local/lib/nodejs-linux"
+ENV PATH="$PATH:${NODE_PATH}/bin"
+RUN export NODE_ARCH=$(if [ "$BUILDARCH" = "amd64" ]; then echo "x64"; else echo "arm64"; fi) && \
+     export NODE_URL="https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz" && \
+     mkdir -p ${NODE_PATH} && \
+     curl -o /tmp/nodejs.tar.xz -L ${NODE_URL} && \
+     tar -xJf /tmp/nodejs.tar.xz -C /usr/local/lib/nodejs-linux --strip-components=1
+RUN corepack enable yarn
+
 # Install libbpf
 ARG LIBBPF_VERSION
 RUN mkdir -p /opt && cd /opt && curl -L https://github.com/gravitational/libbpf/archive/refs/tags/v${LIBBPF_VERSION}.tar.gz | tar xz && \