Metadata-Version: 2.4
Name: rosbackup-ng
Version: 0.8.0.dev2
Summary: Automated, resilient RouterOS backup utility (binary + plaintext, parallel, tmpfs-staged)
Author: rosbackup-ng contributors
License: MIT License
        
        Copyright (c) 2025 JD Bungart <me@jdneer.com>
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://git.jdneer.com/jd/rosbackup-ng
Project-URL: Repository, https://git.jdneer.com/jd/rosbackup-ng
Keywords: routeros,mikrotik,backup,ssh,network-automation
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Archiving :: Backup
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: paramiko>=4.0.0
Requires-Dist: PyYAML>=6.0.3
Requires-Dist: rich>=13.7.0
Requires-Dist: scp>=0.15.0
Provides-Extra: tzdata
Requires-Dist: tzdata>=2024.1; extra == "tzdata"
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == "s3"
Provides-Extra: web
Requires-Dist: fastapi>=0.115; extra == "web"
Requires-Dist: uvicorn>=0.30; extra == "web"
Requires-Dist: jinja2>=3.1; extra == "web"
Requires-Dist: python-multipart>=0.0.9; extra == "web"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12; extra == "dev"
Requires-Dist: ruff<0.16,>=0.15.18; extra == "dev"
Requires-Dist: mypy<2.0,>=1.10; extra == "dev"
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Dynamic: license-file

# RouterOS Backup NG

[![CI](https://git.jdneer.com/jd/rosbackup-ng/actions/workflows/ci.yml/badge.svg?branch=refactor)](https://git.jdneer.com/jd/rosbackup-ng/actions?workflow=ci.yml)
[![License: MIT](img/badges/license-mit.svg)](LICENSE)
![Python 3.9+](img/badges/python.svg)
![Code style: ruff](img/badges/ruff.svg)
![Types: mypy](img/badges/mypy.svg)

Automated, resilient backup utility for MikroTik RouterOS devices. It creates, downloads,
and stores both binary (`.backup`) and plaintext (`.rsc`) backups — in parallel, with
encryption, tmpfs staging, tag/group targeting, pluggable storage destinations, and
connectivity built to survive flaky overlay links (e.g. ZeroTier). An optional self-hosted
web platform adds a dashboard, HTTP API, schedules, and push-restore.

It runs as a lean CLI (`rosbackup`) or as a long-running web service (`rosbackup-web`),
and ships as a regular pip package or a single self-contained zipapp.

![rosbackup-ng web dashboard](img/web/web-dashboard.png)

*The optional self-hosted web platform — fleet dashboard with reachability, backup health,
latency, and recent runs. See [Web platform](#web-platform).*

## At a glance

`rosbackup --compose-style` renders a live, scale-aware view. At fleet scale it switches
from per-row output to an aggregate dashboard — a progress bar, success/failure/running
counts, throughput-based ETA, the active window, and a failures tail:

```
[+] Backing up 5000 targets
    ██████████░░░░░░░░░░░░░░░░░░░░ 1640/5000  33%   ✔ 1628  ✘ 12  ⠹ 48 running   2m14s   eta 4m31s

Active:
    ⠹ router-1641             Backing Up          1.2s
    ⠼ router-1642             Getting Info        0.4s
    … and 46 more running

Recent failures:
    router-0044, router-0210
```

(See [Compose-style output](#compose-style-output) for the per-row view used by smaller fleets.)

## Highlights

- **Both backup formats** — binary (`.backup`) + plaintext (`.rsc`), in parallel, with
  encryption and tmpfs staging to spare router flash.
- **Resilient connectivity** — retry-within-a-window, TCP keepalives, and operation-level
  reconnect, built to survive flaky overlay links (e.g. ZeroTier).
- **Fleet at scale** — tags/groups, scoped runs (`--group`/`--tag`), and a compose-style
  live view that scales from 4 routers to 5,000.
- **Pluggable storage** — mirror to `local` / `sftp` / `nextcloud` (and `s3` via an extra),
  with per-destination retention and group/tag routing.
- **Optional web platform** — self-hosted UI + `/api/v1`: dashboard, schedules, push-restore,
  firmware upgrades, an encrypted **Vault** for keys/passwords, API tokens, and an audit log.
- **Secure by default** — SSH key auth, self-contained host-key verification, encrypted
  backups, `0600` files, and log redaction.

See **[`doc/FEATURES.md`](doc/FEATURES.md)** for the full (non-exhaustive) feature list.

## Documentation

- [`doc/FEATURES.md`](doc/FEATURES.md) — Full (non-exhaustive) feature list
- [`doc/BOOTSTRAP.md`](doc/BOOTSTRAP.md) — Preparing RouterOS devices for automated backups with the bootstrap utility
- [`doc/COMMAND_REFERENCE.md`](doc/COMMAND_REFERENCE.md) — Complete reference of all command-line options
- [`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md) — Detailed reference of all `global.yaml` / `targets.yaml` parameters
- [`doc/FILESYSTEM_STRUCTURE.md`](doc/FILESYSTEM_STRUCTURE.md) — Project directory structure and organization
- [`doc/BACKUP_STRUCTURE.md`](doc/BACKUP_STRUCTURE.md) — Backup file formats, naming conventions, and info files
- [`doc/BACKUP-AND-RESTORE.md`](doc/BACKUP-AND-RESTORE.md) — What backups contain and how to restore a device (binary-only, with safety steps)
- [`doc/TMPFS_FEATURES.md`](doc/TMPFS_FEATURES.md) — tmpfs staging feature documentation
- [`doc/WEB_PLATFORM.md`](doc/WEB_PLATFORM.md) — Self-hosted web UI / API (`rosbackup-web`), reverse-proxy TLS, and the HTTP API
- [`doc/NETWORKING.md`](doc/NETWORKING.md) — Container network profiles (bridge/NAT66 vs macvlan native address) with diagrams
- [`doc/DEPLOY.md`](doc/DEPLOY.md) — Docker Compose deploy (registry pull + local build), TLS, profiles
- [`doc/WEB_STYLE_GUIDE.md`](doc/WEB_STYLE_GUIDE.md) — Web UI design system (tokens, components; repo-only, not served)
- [`doc/DESIGN_REFERENCE.md`](doc/DESIGN_REFERENCE.md) — Architecture and developer documentation
- [`testlab/README.md`](testlab/README.md) — Self-contained libvirt RouterOS test lab (dual-stack virtual routers)

## Prerequisites

- Python 3.9 or higher
- RouterOS devices with SSH access enabled
- An SSH key pair for the backup user (see [BOOTSTRAP](doc/BOOTSTRAP.md))
- RouterOS v7.7 or later for tmpfs staging (falls back to flash on older versions)

## Installation

```bash
git clone https://git.jdneer.com/jd/rosbackup-ng.git
cd rosbackup-ng
python3 -m venv venv && source venv/bin/activate
pip install .                 # provides `rosbackup` and `rosbackup-bootstrap`
```

Optional extras:

```bash
pip install ".[web]"          # self-hosted web platform (`rosbackup-web`)
pip install ".[s3]"           # S3 storage backend (boto3)
pip install -e ".[dev]"       # editable install + linters/tests (ruff, mypy, pytest, bandit)
```

The `local`, `sftp`, and `nextcloud` storage backends need no extras. From a checkout,
`pip install -e .` provides the `rosbackup`, `rosbackup-bootstrap`, and `rosbackup-web`
commands.

> **Docker:** see [doc/DEPLOY.md](doc/DEPLOY.md) for the Compose stack (pull-from-registry).

### Single-file build (no install)

`scripts/build-pyz.sh` bundles the web platform + all `[web]` dependencies + vendored
assets into one executable zipapp you can copy to any host with Python ≥ 3.9:

```bash
scripts/build-pyz.sh          # -> dist/rosbackup-web.pyz
./dist/rosbackup-web.pyz --help
```

See [`doc/WEB_PLATFORM.md`](doc/WEB_PLATFORM.md) for details.

### Configuration files

```bash
cp config/global.yaml.sample config/global.yaml
cp config/targets.yaml.sample config/targets.yaml
```

## Configuration

See [`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md) for the full reference. A minimal
`global.yaml`:

```yaml
# Parent directory for storing backups (required)
backup_path_parent: backups

# Days to keep backups; omit or null to disable pruning (optional)
#backup_retention_days: 90

# Global backup password (optional; used when a target sets `encrypted: true`)
backup_password: your-global-backup-password

ssh:
  user: rosbackup                 # default SSH username
  #known_hosts_file: null         # default: <config-dir>/known_hosts
  #add_target_host_key: true      # true = trust-on-first-use; false = strict

  # Resilient connect for flaky / overlay links (e.g. ZeroTier). On by default.
  connect_retry:
    total_timeout: 180            # retry window in seconds (0 = fail fast)
    #initial_delay: 5
    #max_delay: 30

  args:
    keepalive_interval: 60        # keep long backups alive across NAT/overlay timeouts

tmpfs:
  enabled: true                   # stage binary backups in RAM to reduce flash wear
  fallback_enabled: true          # fall back to flash if tmpfs can't be used

# Email notifications (nested schema). Additional channels plug in under notifications.*
#notifications:
#  enabled: true
#  notify_on_failed_backups: true
#  smtp:
#    enabled: true
#    host: smtp.example.com
#    port: 587
#    use_tls: true
#    from_email: backups@example.com
#    to_emails: [ops@example.com]

# Named groups for scoped runs (-g/--group) and storage routing
#groups:
#  - name: critical
#    match_tags: [critical]        # dynamic: any target tagged "critical"
#  - name: hq
#    members: [ROUTER-1]           # explicit member list

# Mirror backups to additional destinations (local kept by default)
#storage:
#  - name: dr-sftp
#    type: sftp
#    host: sftp.example.com
#    user: rosbackup
#    private_key: ./ssh-keys/private/id_offsite
#    remote_path: /srv/backups/routeros
#    groups: [critical]            # route by group name or tag

# Web platform capability toggles (only affect rosbackup-web; both default true)
#web:
#  allow_download: true
#  allow_restore: true
```

A minimal `targets.yaml`:

```yaml
targets:
  - name: ROUTER-1
    enabled: true
    host: 192.168.88.1
    description: "HQ core router, rack A03"   # free-text metadata (optional)
    tags: [site-hq, core, critical]           # labels for groups/--tag (optional)
    ssh:
      private_key: ./ssh-keys/private/id_rosbackup
```

See [`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md) for per-target overrides
(encryption, retention, tmpfs, per-target `connect_retry`, keeping backups on the router,
storage routing, web capability overrides, etc.).

## Usage

```bash
# Back up all enabled targets
rosbackup

# A specific target only
rosbackup --target ROUTER-1

# A named group or a tag (selectors are mutually exclusive with --target)
rosbackup --group critical
rosbackup --tag site-hq

# Discover configured groups/tags with device counts
rosbackup --list

# Scale-aware compose-style output
rosbackup --compose-style

# Validate connectivity/access without writing anything
rosbackup --dry-run
```

### Compose-style output

`--compose-style` (`-x`) renders a live view that **scales from a handful of routers to
thousands**. Small fleets get a per-row view:

```
[+] Executing backup for 4 targets ...

    ✔ ROUTER1                  Finished            3.8s
    ✔ ROUTER2                  Finished            3.8s
    ⠹ ROUTER3                  Backing Up          2.1s
    ⠹ ROUTER4                  Connecting          0.6s

Summary:
    Total time: 3.9s
    Total size: 107.0KB
    Success: 4 | Failed: 0 | Total: 4
```

Large fleets automatically switch to an aggregate dashboard — a live progress bar,
success/failure/running counts, throughput-based ETA, the currently-running window, and a
failures tail:

```
[+] Backing up 5000 targets
    ██████████░░░░░░░░░░░░░░░░░░░░ 1640/5000  33%   ✔ 1628  ✘ 12  ⠹ 48 running   2m14s   eta 4m31s

Active:
    ⠹ router-1641             Backing Up          1.2s
    ⠼ router-1642             Getting Info        0.4s
    … and 46 more running

Recent failures:
    router-0044, router-0210
```

State updates are O(1) and rendering is O(visible rows), so the view stays responsive
regardless of fleet size.

### Other examples

```bash
rosbackup --no-parallel              # sequential
rosbackup --max-parallel 10          # cap concurrency
rosbackup --log-level DEBUG          # verbose
rosbackup --no-tmpfs                 # disable tmpfs staging for this run
rosbackup --tmpfs-size-mb 25         # override tmpfs size for this run
rosbackup --log-file backups.log     # also write a 0600 log file
```

## Bootstrap (provisioning a router)

Create the backup user and install its key on a device (see [`doc/BOOTSTRAP.md`](doc/BOOTSTRAP.md)):

```bash
# Generate a key pair for the backup user
ssh-keygen -t ed25519 -f ssh-keys/private/id_rosbackup

# Provision a router (prompts for the existing admin password)
rosbackup-bootstrap -H 192.168.88.1 -k ssh-keys/private/id_rosbackup.pub

# See exactly what it would do, without changing anything
rosbackup-bootstrap -H 192.168.88.1 -k ssh-keys/private/id_rosbackup.pub --dry-run
```

## Web platform

The optional `rosbackup-ng[web]` extra adds `rosbackup-web`, a self-hosted UI + HTTP API
that reads the same config directory as the CLI (dashboard pictured [above](#routeros-backup-ng)):

```bash
pip install ".[web]"
rosbackup-web -c /etc/rosbackup --set-password   # set the admin password once
rosbackup-web -c /etc/rosbackup                  # serve on 127.0.0.1:8474
```

It gives you a dashboard, targets inventory (table + cards), run-now with live progress,
schedules, a backup browser with download + push-restore, firmware upgrades, an encrypted
**Vault** for SSH keys and passwords, API tokens, and an audit log. Bind it to localhost and
terminate TLS at a reverse proxy (sample Caddyfile + systemd unit under
[`doc/examples/`](doc/examples)).

Full details — auth, the Vault, schedules, push-restore, capability toggles, the API, and the
single-file build — are in [`doc/WEB_PLATFORM.md`](doc/WEB_PLATFORM.md).

## Command completion

Enable bash completion for the current shell session:

```bash
source scripts/rosbackup-ng-completion.bash
```

It provides option, file-path, and target-name completion (from `targets.yaml`) for the
`rosbackup` and `rosbackup-bootstrap` commands.

## Logging

Console logging is colored via `rich`. With `--log-file`, logs are also written to a file
(created `0600`) including backup status, connection details, retries, and errors. Format:
`MM-DD-YYYY HH:MM:SS [LEVEL] [target] message`.

## Development

```bash
pip install -e ".[dev,web]"
ruff check src tests && ruff format --check src tests
mypy src
bandit -q -r src
pytest -q
```

CI runs this same battery on every push via self-hosted [Forgejo
Actions](https://git.jdneer.com/jd/rosbackup-ng/actions) (`.forgejo/workflows/ci.yml`),
deliberately using **no external/marketplace actions** — every step is hand-written shell.

**Before pushing, run the whole gate in one shot** with a venv that matches CI's resolved
dependencies — this catches the failures a `pytest`-only check misses (ruff/format/bandit, and
runner-environment flakes):

```bash
scripts/ci-local.sh            # ruff -> format -> mypy -> bandit -> pytest -> shiv build + smoke
scripts/ci-local.sh --fresh    # rebuild the .civenv from scratch first
```

It runs every gate (not just the first failure) and exits non-zero if any fails. The `.civenv`
it creates is gitignored and reused across runs.

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Run the checks above (they must all pass)
4. Commit and open a Pull Request

## License

This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

## Support

Open an issue or send a PR.
