Metadata-Version: 2.4
Name: rosbackup-ng
Version: 0.10.0
Summary: Automated, resilient RouterOS backup utility (binary + plaintext, parallel, tmpfs-staged)
Author: rosbackup-ng contributors
License: MIT License
        
        Copyright (c) 2025 JD Bungart <me@jdneer.com>
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://git.jdneer.com/jd/rosbackup-ng
Project-URL: Repository, https://git.jdneer.com/jd/rosbackup-ng
Keywords: routeros,mikrotik,backup,ssh,network-automation
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Archiving :: Backup
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: paramiko>=4.0.0
Requires-Dist: PyYAML>=6.0.3
Requires-Dist: rich>=13.7.0
Requires-Dist: scp>=0.15.0
Provides-Extra: tzdata
Requires-Dist: tzdata>=2024.1; extra == "tzdata"
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == "s3"
Provides-Extra: web
Requires-Dist: fastapi>=0.115; extra == "web"
Requires-Dist: uvicorn>=0.30; extra == "web"
Requires-Dist: jinja2>=3.1; extra == "web"
Requires-Dist: python-multipart>=0.0.9; extra == "web"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12; extra == "dev"
Requires-Dist: ruff<0.16,>=0.15.18; extra == "dev"
Requires-Dist: mypy<2.0,>=1.10; extra == "dev"
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Dynamic: license-file

<p align="center"><img src="img/logo.png" alt="rosbackup-ng" width="104"></p>
<h1 align="center">rosbackup-ng</h1>
<p align="center"><em>Next Generation RouterOS Device Fleet Management &amp; Backup</em></p>

<p align="center">
  <a href="https://git.jdneer.com/jd/rosbackup-ng/actions?workflow=ci.yml"><img src="https://git.jdneer.com/jd/rosbackup-ng/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
  <a href="LICENSE"><img src="img/badges/license-mit.svg" alt="License: MIT"></a>
  <img src="img/badges/python.svg" alt="Python 3.9+">
  <img src="img/badges/ruff.svg" alt="Code style: ruff">
  <img src="img/badges/mypy.svg" alt="Types: mypy">
</p>

<p align="center">
⚠️ <strong>Under active development — feature-incomplete until v1.0.</strong><br>
rosbackup-ng is usable today, but behavior, configuration, and APIs may change between releases until <strong>v1.0</strong> arrives.
</p>

---

Automated, resilient backup utility for MikroTik RouterOS devices. It creates, downloads,
and stores both binary (`.backup`) and plaintext (`.rsc`) backups — in parallel, with
encryption, tmpfs staging, tag/group targeting, pluggable storage destinations, and
connectivity built to survive flaky overlay links (e.g. ZeroTier). An optional self-hosted
web platform adds a dashboard, HTTP API, schedules, and push-restore.

It runs as a lean CLI (`rosbackup`) or as a long-running web service (`rosbackup-web`). The web
platform deploys as a Docker stack (see [Deploy with Docker](#deploy-with-docker)); pip, a single-file
zipapp, and from-source installs live in [`doc/INSTALLATION-METHODS.md`](doc/INSTALLATION-METHODS.md).

![rosbackup-ng web dashboard](img/web/web-dashboard.png)

*The optional self-hosted web platform — fleet dashboard with reachability, backup health,
latency, and recent runs. See [Web platform](#web-platform).*

## At a glance

`rosbackup --compose-style` renders a live, scale-aware view — a per-row list for small fleets, or an
aggregate dashboard (progress bar, success/failure/running counts, throughput ETA, failures tail) at
thousands of targets. See [Compose-style output](#compose-style-output).

## Highlights

- **Both backup formats** — binary (`.backup`) + plaintext (`.rsc`), in parallel, with
  encryption and tmpfs staging to spare router flash.
- **Resilient connectivity** — retry-within-a-window, TCP keepalives, and operation-level
  reconnect, built to survive flaky overlay links (e.g. ZeroTier).
- **Fleet at scale** — tags/groups, scoped runs (`--group`/`--tag`), and a compose-style
  live view that scales from 4 routers to 5,000.
- **Pluggable storage** — mirror to `local` / `sftp` / `nextcloud` (and `s3` via an extra),
  with per-destination retention and group/tag routing.
- **Optional web platform** — self-hosted UI + `/api/v1`: dashboard, schedules, push-restore,
  firmware upgrades, an encrypted **Vault** for keys/passwords, API tokens, and an audit log.
- **Secure by default** — SSH key auth, self-contained host-key verification, encrypted
  backups, `0600` files, and log redaction.

See **[`doc/FEATURES.md`](doc/FEATURES.md)** for the full (non-exhaustive) feature list.

## How it fits together

One backup engine, two front-ends. The `rosbackup` CLI and the `rosbackup-web` service both drive the
**same** console-free core (`orchestrator.run_backup`); the web adds the encrypted **Vault**, the
scheduler, the UI, and the `/api/v1` HTTP API on top.

```
        config dir · global.yaml · targets.yaml · web.db (vault, web only)
                                  │
                 ┌────────────────┴─────────────────┐
                 ▼                                   ▼
        rosbackup (CLI)                     rosbackup-web (service)
        standalone · no server              UI · /api/v1 · scheduler · Vault
        filesystem SSH key                  ◀── browser / API client (Bearer token)
                 │                                   │
                 │                                   │  inject_vault_keys():
                 │                                   │  key_secret → in-memory key
                 └────────────────┐   ┌───────────────┘
                                  ▼   ▼
        core engine · orchestrator.run_backup(config, targets) → BackupRunResult
        parallel SSH · tmpfs staging · retention
                                  │
                 ┌────────────────┴─────────────────┐
                 ▼                                   ▼
        RouterOS fleet (SSH / SCP)          storage: local · sftp · nextcloud · s3
```

- **Shared core** — both front-ends load the config and call the same `run_backup(config, targets)`; the
  web is a thin adapter, not a second implementation.
- **Vault is web-only** — the web decrypts a target's `ssh.key_secret` into an in-memory key just before a
  run (`inject_vault_keys`); the standalone CLI authenticates with filesystem keys.
- **HTTP API** — `POST /api/v1/runs` triggers a run and `GET /api/v1/runs/{id}` polls results (Bearer-token
  auth), so backups can be driven programmatically.

Full internals are in **[`doc/DESIGN_REFERENCE.md`](doc/DESIGN_REFERENCE.md)**.

## Documentation

- [`doc/FEATURES.md`](doc/FEATURES.md) — Full (non-exhaustive) feature list
- [`doc/BOOTSTRAP.md`](doc/BOOTSTRAP.md) — Preparing RouterOS devices for automated backups with the bootstrap utility
- [`doc/COMMAND_REFERENCE.md`](doc/COMMAND_REFERENCE.md) — Complete reference of all command-line options
- [`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md) — Detailed reference of all `global.yaml` / `targets.yaml` parameters
- [`doc/STORAGE.md`](doc/STORAGE.md) — Storage destinations: inheritance model, routing, and monitoring
- [`doc/FILESYSTEM_STRUCTURE.md`](doc/FILESYSTEM_STRUCTURE.md) — Project directory structure and organization
- [`doc/BACKUP_STRUCTURE.md`](doc/BACKUP_STRUCTURE.md) — Backup file formats, naming conventions, and info files
- [`doc/BACKUP-AND-RESTORE.md`](doc/BACKUP-AND-RESTORE.md) — What backups contain and how to restore a device (binary-only, with safety steps)
- [`doc/TMPFS_FEATURES.md`](doc/TMPFS_FEATURES.md) — tmpfs staging feature documentation
- [`doc/WEB_PLATFORM.md`](doc/WEB_PLATFORM.md) — Self-hosted web UI / API (`rosbackup-web`), reverse-proxy TLS, and the HTTP API
- [`doc/NETWORKING.md`](doc/NETWORKING.md) — Container network profiles (bridge/NAT66 vs macvlan native address) with diagrams
- [`doc/DEPLOY.md`](doc/DEPLOY.md) — Docker Compose deploy (registry pull + local build), TLS, profiles
- [`doc/INSTALLATION-METHODS.md`](doc/INSTALLATION-METHODS.md) — Non-Docker installs: pip (registry), single-file zipapp, from source
- [`doc/WEB_STYLE_GUIDE.md`](doc/WEB_STYLE_GUIDE.md) — Web UI design system (tokens, components; repo-only, not served)
- [`doc/DESIGN_REFERENCE.md`](doc/DESIGN_REFERENCE.md) — Architecture and developer documentation
- [`doc/TECH_STACK.md`](doc/TECH_STACK.md) — Tech choices and rationale for every layer of the stack
- [`testlab/README.md`](testlab/README.md) — Self-contained libvirt RouterOS test lab (dual-stack virtual routers)

## Prerequisites

- Docker Engine + Compose v2 — or Python 3.9+ for the [other install methods](doc/INSTALLATION-METHODS.md)
- RouterOS devices with SSH access enabled
- An SSH key pair for the backup user (see [BOOTSTRAP](doc/BOOTSTRAP.md))
- RouterOS v7.7 or later for tmpfs staging (falls back to flash on older versions)

## Deploy with Docker

The supported way to run rosbackup-ng is the Docker Compose stack under [`deploy/`](deploy). It pulls a
pre-built image from the registry (a local build is the fallback), runs `rosbackup-web` behind a Caddy
TLS front door, and keeps your config + backups on the host. **Prerequisite:** Docker Engine + Compose v2.

### Quickstart

```bash
git clone https://git.jdneer.com/jd/rosbackup-ng /opt/rosbackup-ng && cd /opt/rosbackup-ng/deploy
sudo ./deploy.sh up                 # first run scaffolds .env + ./config and pulls the image
$EDITOR config/global.yaml config/targets.yaml   # add your routers + SSH key (see CONFIG_REFERENCE)
sudo ./deploy.sh set-password       # set the web admin password (once)
sudo ./deploy.sh up                 # apply — idempotent, safe to re-run
```

Open **`https://<host>/`** (self-signed cert by default) and log in. That's the whole setup.

### TLS (optional)

```bash
sudo ./deploy.sh set-fqdn backup.example.com        # public hostname
sudo ./deploy.sh set-tls-desec <deSEC-token>        # automatic Let's Encrypt via DNS-01
# or bring your own cert:
sudo ./deploy.sh set-cert ./fullchain.pem ./key.pem
```

### Day-to-day

```bash
sudo ./deploy.sh status     # health + container state
sudo ./deploy.sh logs       # follow logs
sudo ./deploy.sh update     # pull the latest image + restart
sudo ./deploy.sh down       # stop the stack
```

Network profiles (bridge/NAT66 vs macvlan native addressing), manual-cert import, and the full
`deploy.sh` reference are in **[`doc/DEPLOY.md`](doc/DEPLOY.md)** and **[`doc/NETWORKING.md`](doc/NETWORKING.md)**.

> **Prefer not to use Docker?** pip (from the package registry), a single-file zipapp, and from-source
> installs are in **[`doc/INSTALLATION-METHODS.md`](doc/INSTALLATION-METHODS.md)**.

## Configuration

The CLI and the web platform read the same config dir (`global.yaml` + `targets.yaml`). A minimal
`global.yaml`:

```yaml
backup_path_parent: backups          # where backups are stored (required)
backup_password: change-me           # used when a target sets `encrypted: true` (optional)

ssh:
  user: rosbackup                    # default SSH username
  connect_retry:                     # resilient connect for flaky/overlay links (ZeroTier), on by default
    total_timeout: 180               # retry window, seconds

tmpfs:
  enabled: true                      # stage binary backups in RAM to spare router flash
```

A minimal `targets.yaml`:

```yaml
targets:
  - name: ROUTER-1
    host: 192.168.88.1
    tags: [site-hq, core]                          # labels for groups / --tag (optional)
    ssh:
      private_key: ./ssh-keys/private/id_rosbackup
```

See **[`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md)** for the full reference — retention,
encryption, per-target overrides, notifications, groups, storage routing, and web capability toggles.

## Usage

```bash
# Back up all enabled targets
rosbackup

# A specific target only
rosbackup --target ROUTER-1

# A named group or a tag (selectors are mutually exclusive with --target)
rosbackup --group critical
rosbackup --tag site-hq

# Discover configured groups/tags with device counts
rosbackup --list

# Scale-aware compose-style output
rosbackup --compose-style

# Validate connectivity/access without writing anything
rosbackup --dry-run
```

### Compose-style output

`--compose-style` (`-x`) renders a live view that **scales from a handful of routers to
thousands**. Small fleets get a per-row view:

```
[+] Executing backup for 4 targets ...

    ✔ ROUTER1                  Finished            3.8s
    ✔ ROUTER2                  Finished            3.8s
    ⠹ ROUTER3                  Backing Up          2.1s
    ⠹ ROUTER4                  Connecting          0.6s

Summary:
    Total time: 3.9s
    Total size: 107.0KB
    Success: 4 | Failed: 0 | Total: 4
```

Large fleets automatically switch to an aggregate dashboard — a live progress bar,
success/failure/running counts, throughput-based ETA, the currently-running window, and a
failures tail:

```
[+] Backing up 5000 targets
    ██████████░░░░░░░░░░░░░░░░░░░░ 1640/5000  33%   ✔ 1628  ✘ 12  ⠹ 48 running   2m14s   eta 4m31s

Active:
    ⠹ router-1641             Backing Up          1.2s
    ⠼ router-1642             Getting Info        0.4s
    … and 46 more running

Recent failures:
    router-0044, router-0210
```

State updates are O(1) and rendering is O(visible rows), so the view stays responsive
regardless of fleet size.

### Other examples

```bash
rosbackup --no-parallel              # sequential
rosbackup --max-parallel 10          # cap concurrency
rosbackup --log-level DEBUG          # verbose
rosbackup --no-tmpfs                 # disable tmpfs staging for this run
rosbackup --tmpfs-size-mb 25         # override tmpfs size for this run
rosbackup --log-file backups.log     # also write a 0600 log file
```

## Bootstrap (provisioning a router)

Create the backup user and install its key on a device (see [`doc/BOOTSTRAP.md`](doc/BOOTSTRAP.md)):

```bash
# Generate a key pair for the backup user
ssh-keygen -t ed25519 -f ssh-keys/private/id_rosbackup

# Provision a router (prompts for the existing admin password)
rosbackup-bootstrap -H 192.168.88.1 -k ssh-keys/private/id_rosbackup.pub

# See exactly what it would do, without changing anything
rosbackup-bootstrap -H 192.168.88.1 -k ssh-keys/private/id_rosbackup.pub --dry-run
```

## Web platform

`rosbackup-web` is a self-hosted UI + HTTP API that reads the same config directory as the CLI
(dashboard pictured near the top). Deploy it with the Docker stack
([`doc/DEPLOY.md`](doc/DEPLOY.md)); to run it without Docker, see
[`doc/INSTALLATION-METHODS.md`](doc/INSTALLATION-METHODS.md). Once it's installed:

```bash
rosbackup-web -c /etc/rosbackup --set-password   # set the admin password once
rosbackup-web -c /etc/rosbackup                  # serve on 127.0.0.1:8474
```

It gives you a dashboard, targets inventory (table + cards), run-now with live progress,
schedules, a backup browser with download + push-restore, firmware upgrades, an encrypted
**Vault** for SSH keys and passwords, API tokens, and an audit log. Bind it to localhost and
terminate TLS at a reverse proxy (sample Caddyfile + systemd unit under
[`doc/examples/`](doc/examples)).

Full details — auth, the Vault, schedules, push-restore, capability toggles, the API, and the
single-file build — are in [`doc/WEB_PLATFORM.md`](doc/WEB_PLATFORM.md).

## Command completion

Enable bash completion for the current shell session:

```bash
source scripts/rosbackup-ng-completion.bash
```

It provides option, file-path, and target-name completion (from `targets.yaml`) for the
`rosbackup` and `rosbackup-bootstrap` commands.

## Logging

Console logging is colored via `rich`. With `--log-file`, logs are also written to a file
(created `0600`) including backup status, connection details, retries, and errors. Format:
`MM-DD-YYYY HH:MM:SS [LEVEL] [target] message`.

## Development

```bash
pip install -e ".[dev,web]"
ruff check src tests && ruff format --check src tests
mypy src
bandit -q -r src
pytest -q
```

CI runs this same battery on every push via self-hosted [Forgejo
Actions](https://git.jdneer.com/jd/rosbackup-ng/actions) (`.forgejo/workflows/ci.yml`),
deliberately using **no external/marketplace actions** — every step is hand-written shell.

**Before pushing, run the whole gate in one shot** with a venv that matches CI's resolved
dependencies — this catches the failures a `pytest`-only check misses (ruff/format/bandit, and
runner-environment flakes):

```bash
scripts/ci-local.sh            # ruff -> format -> mypy -> bandit -> pytest -> shiv build + smoke
scripts/ci-local.sh --fresh    # rebuild the .civenv from scratch first
```

It runs every gate (not just the first failure) and exits non-zero if any fails. The `.civenv`
it creates is gitignored and reused across runs.

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Run the checks above (they must all pass)
4. Commit and open a Pull Request

## License

This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

## Support

Open an issue or send a PR.
