222 lines
8.3 KiB
Markdown
222 lines
8.3 KiB
Markdown
# Deployment
|
|
|
|
This service runs active-active on both webserver VMs. Both installations use
|
|
the same MariaDB Galera database, Gitea application, package registry, and
|
|
secrets. The containers use Linux host networking so `127.0.0.1:3306` reaches
|
|
each webserver's local HAProxy listener.
|
|
|
|
## 1. DNS and TLS
|
|
|
|
Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
|
|
frontend. Install the certificate using the existing certificate automation.
|
|
Set the broker to listen on all addresses on both webservers:
|
|
|
|
```dotenv
|
|
TAPM_LISTEN_ADDR=0.0.0.0:8080
|
|
TAPM_DISPLAY_TIME_ZONE=America/Chicago
|
|
```
|
|
|
|
The wildcard bind accepts connections through each node's physical address and
|
|
through the Keepalived address `10.10.1.120` on whichever node currently owns
|
|
it. The container health check independently uses `127.0.0.1:8080`. Restrict
|
|
port 8080 to the load-balancer and trusted management addresses. If Nginx
|
|
terminates TLS locally and neither direct nor Keepalived access is required,
|
|
the broker can instead bind to `127.0.0.1:8080`.
|
|
|
|
An example virtual host is in
|
|
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
|
|
Adjust only the certificate paths if the local convention differs.
|
|
|
|
## 2. MariaDB Galera
|
|
|
|
Create a dedicated database and application users limited to the two webserver
|
|
addresses. Run this once against the cluster as a database administrator,
|
|
replacing the password with the same locally generated value in both accounts:
|
|
|
|
```sql
|
|
CREATE DATABASE tapm_broker
|
|
CHARACTER SET utf8mb4
|
|
COLLATE utf8mb4_unicode_ci;
|
|
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
|
|
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
|
|
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
|
|
ON tapm_broker.* TO 'tapm'@'10.10.1.121';
|
|
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
|
|
ON tapm_broker.* TO 'tapm'@'10.10.1.122';
|
|
FLUSH PRIVILEGES;
|
|
```
|
|
|
|
The Keepalived address `10.10.1.120` is not included because outbound HAProxy
|
|
connections normally originate from the physical webserver address. Add a
|
|
third account for `10.10.1.120` only if a connection test or the HAProxy
|
|
configuration proves that it explicitly uses the virtual address as its source.
|
|
|
|
## 3. Gitea accounts and OAuth
|
|
|
|
Create these separately:
|
|
|
|
1. The `taiadmin` technician account used to sign in to the portal.
|
|
2. A `tapm-packages` service account with read access only to the private
|
|
`TAI/files` package owner. Create a personal access token for this account
|
|
and store it only in the broker environment file.
|
|
3. A `tapm-publisher` service account with `write:package` access. Its token is
|
|
used only by authenticated portal uploads.
|
|
4. An OAuth2 application with callback URL
|
|
`https://tapm.scity.us/auth/callback`.
|
|
|
|
The OAuth client secret authenticates the portal to Gitea. The read token is
|
|
used only while proxying an authorized file, and the write token is used only
|
|
for authenticated uploads. None are returned to a technician or Proxmox host.
|
|
The broker requests only Gitea's `read:user` OAuth scope, which it uses to
|
|
confirm the signed-in username against `TAPM_ALLOWED_GITEA_USERS`.
|
|
|
|
## 4. Broker configuration
|
|
|
|
On each webserver:
|
|
|
|
```sh
|
|
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
|
|
cd TA-Deployment-Broker
|
|
cp .env.example .env
|
|
chmod 600 .env
|
|
```
|
|
|
|
Edit `.env` and use the same secrets on both webservers. The listen address is
|
|
the one node-specific value. Generate the cookie secret with:
|
|
|
|
```sh
|
|
openssl rand -base64 48
|
|
```
|
|
|
|
The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
|
|
encoded UTC `time_zone` setting from `.env.example`. Its address remains
|
|
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
|
|
single Galera member.
|
|
|
|
## 5. Build and migrate
|
|
|
|
Build the image on each webserver:
|
|
|
|
```sh
|
|
docker compose build
|
|
```
|
|
|
|
Apply migrations from only one webserver:
|
|
|
|
```sh
|
|
docker compose run --rm migrate
|
|
```
|
|
|
|
Migrations use a MariaDB advisory lock, so an accidental simultaneous run
|
|
cannot apply the same migration twice.
|
|
|
|
## 6. Start both instances
|
|
|
|
On both webservers:
|
|
|
|
```sh
|
|
docker compose up -d broker
|
|
curl --fail "http://127.0.0.1:8080/health/ready"
|
|
```
|
|
|
|
There is no local application state and no session affinity requirement.
|
|
|
|
For later updates, run the repository update helper from either webserver:
|
|
|
|
```sh
|
|
/opt/idssys/ta-deployment-broker/update.sh
|
|
```
|
|
|
|
The helper updates and health-checks the local broker first, then connects as
|
|
`TAPM_UPDATE_SSH_USER` to each comma-separated host in `TAPM_UPDATE_PEERS` and
|
|
performs the same local-only check there. This provides a sequential rolling
|
|
update from any node. An unreachable peer is skipped with a warning; a reachable
|
|
peer whose update fails causes the command to fail. The SSH user defaults to
|
|
`root` when omitted.
|
|
|
|
Configure each webserver's `.env` with its other broker:
|
|
|
|
```dotenv
|
|
# Webserver-Node1
|
|
TAPM_UPDATE_PEERS=10.10.1.122
|
|
TAPM_UPDATE_SSH_USER=root
|
|
|
|
# Webserver-Node2
|
|
TAPM_UPDATE_PEERS=10.10.1.121
|
|
TAPM_UPDATE_SSH_USER=root
|
|
```
|
|
|
|
Additional peers can be listed with commas. The updater reads only these named
|
|
settings and does not source `.env` as executable shell code.
|
|
|
|
Each node fetches and compares its current tracked branch with its upstream. If
|
|
both commits match, that node's Docker services are untouched. When an update
|
|
exists, the helper fast-forwards the branch, rebuilds the image, applies pending
|
|
migrations, recreates the broker container, and waits up to 150 seconds for its
|
|
Docker health check. It refuses tracked local changes and detached, untracked,
|
|
locally-ahead, or diverged branches.
|
|
|
|
## 7. HAProxy health check
|
|
|
|
The load balancer should consider an origin healthy only when
|
|
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
|
|
`/health/ready` also verifies Galera connectivity.
|
|
|
|
Both healthy webservers can receive traffic. `leastconn` is useful because
|
|
package uploads and downloads stay open longer than portal requests.
|
|
|
|
## 8. Publish a protected package
|
|
|
|
The normal workflow is **Protected packages → Update a package** in the portal.
|
|
The package ID is the stable name used by the broker, ProxMenu, and Gitea; a
|
|
separate registry package name is not required. The broker streams the
|
|
replacement directly to Gitea, calculates SHA-256, updates the catalog, removes
|
|
the prior Gitea version, and writes audit events without buffering the installer
|
|
on local disk.
|
|
|
|
An authorization grants access to the stable package ID, not a particular
|
|
version. Existing active codes and download sessions resolve the package's
|
|
current registry version and checksum, so they automatically use a replacement
|
|
after it is registered.
|
|
|
|
The load balancer or Nginx must allow request bodies up to
|
|
`TAPM_MAX_UPLOAD_BYTES` and use a sufficiently long request timeout. The
|
|
default is 1 GiB and 30 minutes.
|
|
|
|
For recovery, a package can still be published directly:
|
|
|
|
```sh
|
|
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
|
|
--upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
|
|
'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'
|
|
```
|
|
|
|
Calculate its checksum:
|
|
|
|
```sh
|
|
sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb
|
|
```
|
|
|
|
Sign in at `https://tapm.scity.us`, expand **Register package metadata
|
|
manually**, enter the matching values, and enable it. The broker never uses the
|
|
technician's Gitea credentials for package operations.
|
|
|
|
## 9. Backup and rotation
|
|
|
|
- Include `tapm_broker` in the existing Galera backup policy.
|
|
- Audit records are retained indefinitely unless an administrator establishes a
|
|
database retention policy. The Codes view previews the newest 15 records; the
|
|
Audit view can filter retained history by event, customer/deployment label,
|
|
user, host, package, LAN/WAN IP, or details and displays up to the newest 250
|
|
matching events. Audit timestamps are stored in UTC and rendered in
|
|
`TAPM_DISPLAY_TIME_ZONE`. Deployment exchanges record both the device LAN
|
|
address reported by TAPM and the WAN/source address received through the
|
|
trusted proxy chain.
|
|
- Keep `.env` outside Git and readable only by the service administrator.
|
|
- Rotate both Gitea package tokens and the OAuth secret if either webserver is
|
|
compromised.
|
|
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
|
|
active deployment codes.
|
|
- Revoke active deployment authorizations from the portal when a deployment
|
|
ends early.
|