Appendix 

Encryption and key management 

This section follows "Encryption" in the "Overview and Concepts" section.

The user's password is used to derive two 192-bit keys (the "L" and "R" keys) via PBKDF2-SHA512, with hard-coded parameters for repeatable output.

  • The L-key is used to log in to the Auth Role server in place of the real password; the server stores only a bcrypt(sha512) hash of this L-key.
  • The R-key never leaves the client, and is used to encrypt secret keys stored within the user's profile on the server.

This means that one password can be used for all client-side account operations, while preventing servers from uncovering client-only secrets.

When Comet sets up a Storage Vault for the first time, it generates two high-entropy random keys (the 256-bit "A" and 128-bit "E" keys). All user data in the Storage Vault is stored encrypted with the A-key using AES-256 in CTR mode, and authenticated using Poly1305 in AEAD (encrypt-then-MAC) mode.

The permanent A-key is stored inside the Storage Vault, encrypted with the E-key. The E-key is then encrypted with the R-key and stored in the user's profile on the Auth Role server. When a backup is performed, the client uses its password to derive the private R-key, to decrypt the E-key from the vault, to decrypt the A-key for data storage. This extra level of indirection enables some key rotation scenarios, as a new E-key can be generated without needing to re-encrypt all the data in the Storage Vault.

If the Storage Vault is for a Storage Role bucket, a high-entropy random 128-bit PSK is used to gate access to the bucket. The Storage Role server stores only a bcrypt(sha512) hash of this PSK. The client encrypts this PSK with the R-key and stores it in the user's profile on the Auth Role server.

Compatibility events 

This section follows "Backward Compatibility" in the "Overview and Concepts" section.

The following compatibility events have occurred:

Version Details Upgrade Compatible Downgrade Compatible
17.9.3 A metadata format was enforced. Yes, but pre-existing Storage Vaults will not take advantage of the new features No. Storage Vaults created with newer versions of Comet cannot be used in old versions of Comet
17.6.1 A compression format was added. Yes Partial. If the new compression format was used, old versions of Comet will not be able to restore data
2.8.2 Beta The encryption format changed. Yes. Accounts are automatically upgraded upon login No. Old versions of Comet will not understand secrets in the new format
2.3.0 Beta The compression format changed. Yes. New versions of Comet can still read old backed-up data No. Old versions of Comet cannot read newly backed-up data
2.2.0 Beta The license server address changed. Yes No. Old versions of Comet Server can no longer be used
2.0.0 Beta The Storage Vault format changed. Partial. Storage Vaults must be recreated No. Old versions of Comet are unable to use new Storage Vaults
1.7.0 Beta The Storage Vault format changed. Yes. Comet will automatically upgrade Local Copy and Comet Server types, but S3 and SFTP types must be upgraded manually No. Old versions of Comet are unable to use new Storage Vaults
1.0.0 Beta Initial compatibility milestone Yes No
0.?.? Beta No compatibility guarantees made in either direction. n/a n/a

Migrating user data 

It is possible to migrate user data to balance your storage requirements.

Migrating user data to a different volume 

If multiple volumes are configured in the same Storage Role Comet Server, then you can move files freely. Comet Server will instantly recognize the changes, because it looks in all attached volumes when looking for a chunk.

Migrating user data to a different Comet Server 

  1. Create a new bucket on a different Comet Server

    • You can either manually create the bucket, or Request a bucket on a new target server
  2. Copy the file content to the new server

  3. Edit the user's profile in the Auth Role Comet Server to change the address, bucket, and bucket-key that it points to

    • You should first ensure that this user is not running any backup jobs to the original server.
    • You must take care to preserve the Encryption Key settings. The key absolutely must not change.
  4. Remove the file content from the original server
    • You should first ensure that this user is not running any restore jobs from the original server.

Migrating user data between Storage Vault types 

All Storage Vault types (e.g. Comet Server, Local Copy, SFTP, Amazon S3 etc) use the same on-disk layout. It is possible to follow the above steps for "Migrating user data to a different Comet Server" even when either the old- or new- targets are not a Comet Server.

For more information, please see the "Comet Backup Usage" > "Seed Load" section.

Multiple Comet Server instances 

It is possible to run multiple Comet Server instances on the same machine or IP, by using a load balancer or a frontend proxy software. All Comet Server communication is performed over HTTP / HTTPS / Websockets, so applications such as nginx, Apache, HAProxy, Traefik or Caddy are all suitable for this purpose.

If you choose to do this, take care that the frontend proxy does not introduce additional buffering or timeouts that could interrupt the connection between Comet Backup and Comet Server.

For instance with nginx, the following configuration could be used:

proxy_connect_timeout 3000;
proxy_send_timeout    3000;
proxy_read_timeout    3000;
client_body_timeout   3000;
proxy_buffering off;

Using Comet Backup behind a network proxy 

Comet Backup can be used behind an HTTP or SOCKS proxy.

In Comet 17.11.x, proxy settings are controlled by an environment variable named HTTP_PROXY.

Multiple programs use this configuration method, so the environment variable may already be present for other software on the machine.

On Windows, the environment variable should be set in the "System variables" section to ensure that any settings also apply to background services.

On Linux, environment variables can be set system-wide (e.g. in /etc/environment or /etc/profile.d/my-custom-proxy.sh), or for the root user running Comet Backup (e.g. in /root/.profile), or in your startup script for Comet Backup (e.g. in /etc/rc.local).

The HTTP_PROXY environment variable should be set to a string of the form https://username:password@my.proxy.host.com/ or http://my.proxy.host.com/ or socks5://username:password@my.proxy.host.com/.

A future version of Comet will provide GUI settings to configure the network proxy.

Troubleshooting 

Error "local error: tls: record overflow" 

This message means the connection was corrupted over the network, and Comet aborted the connection.

This can happen because of random network conditions. Retrying the operation should fix the issue.

If the issue keeps happening repeatedly, this message indicates that something is interfering with packets in your network.

  • Failing NIC
  • Bad NIC driver or driver configuration
  • Failing RAM, on either the endpoint machine or any of the intermediate routers
  • Outdated firewall or proxy, performing incorrect SSL interception

For more information, please see the record_overflow section in IETF RFC 5246.

"Paused" state on Windows service 

Comet Server on Windows consists of two parts: cometd.exe and cometd-service.exe. The latter binary is registered as a service with Windows, and is responsible for ensuring that the former binary stays running.

If Comet Server encounters an error and closes, the service binary will restart it. If the Comet Server is repeatedly unable to start - if it closes immediately when launched, several hundred times consecutively - then the service manager will assume the error is permanent and abandon restarting the process. This condition is displayed as the "Paused" state.

You can resolve this issue by resolving the underlying issue with the service. In Comet Server Service Manager, use the Service menu > Diagnostic > "Run in popup" option to determine the underlying issue. Alternatively, you can read Comet Server's log files from the disk.

Forgotten administrator password 

If you are locked out of the Comet Server web interface, you can change your administrator password by editing the cometd.cfg file.

  1. Stop the server, and edit the cometd.cfg file
  2. Find the AdminUsers section for the administrator user in question
  3. Set "PasswordFormat" to 0
  4. Set "Password" to e.g. "admin"
  5. Save the file and restart the server.

You should now be able to log in with the reset password. The password will be hashed and/or encrypted after first use.

Storage server "127.0.0.1" in use by accounts but not managed by Constellation 

A user account has a bucket on a Storage Role Comet Server at the address 127.0.0.1, but this address was not selected for management by Constellation. Aside from the standard warnings about managing Constellation, the 127.0.0.1 is a special case.

If Comet Server is configured to listen on all network interfaces, the server will be accessible on both 127.0.0.1 as well as LAN, WAN, or DNS addresses. If you log in to either the Comet Backup application or the Comet Server web interface at 127.0.0.1, and request a new Storage Vault from a Comet Server configured as "Local" (or $self$ in cometd.cfg), the resulting Storage Vault will be configured using 127.0.0.1 as the remote network address.

However, 127.0.0.1 has a different meaning depending on where it is found. The connection will fail if the Comet Backup client is not running on the same machine. There may also be unintended consequences if the account is replicated to another server.

To avoid this problem, either

  • always use the external DNS name for your Comet Server when requesting new Storage Vaults, or
  • change the Request destination to use "Remote" with an external DNS name instead of "Local" (or $self$ in cometd.cfg).

A similar caveat applies to the software downloads, which can include embedded server address details.

Error "Can't supply TLS certificate for domain XYZ: acme/autocert: host not configured" 

When Let's Encrypt is used, the Comet Server associates your domain name with the valid SSL certificate.

However, if someone tries to connect to the server using https:// {direct IP} or https:// {wrong domain name} there is a problem as Comet does not have a valid certificate for the request. In the past, Comet would serve the certificate, but the client would see a mismatched-certificate error and fail. As of Comet 17.3.12, Comet Server will instead just drop the connection since there is an obvious misconfiguration.

If you see this error message including a wrong domain name:

  • There is a misconfiguration. Either a client is unintentionally connecting using the wrong domain name, or, you have misconfigured Let's Encrypt to not mention a domain name that should be included.
  • A researcher may be scanning the entire IPv4 internet using the Reverse DNS (RDNS) name of your server. It's possible that the RDNS name is out of date compared to the expected domain name for accessing this server.

If you see this error message including an IP address:

  • It's possible that someone tried to connect to your server on https:// {direct IP}. At the time of writing, it is not possible to receive a Let's Encrypt certificate for a bare IPv4 address.
  • A researcher may be scanning the entire IPv4 internet, and connecting to the open :443 port using the IP address of your server.

Microsoft SQL Server backup encountered a VDI error 

You should ensure that the necessary VDI .dll files are registered and are the correct version for your SQL Server installation. You can use Microsoft SQL Server Backup Simulator to check the status of the VDI .dll files.

"Access is denied" backing up files and folders on Windows 

Please ensure that the "Take filesystem snapshot" checkbox option is enabled in the Protected Item settings.

By default, Comet Backup runs as your local Windows user, and has the same restrictions on accessing files. When this checkbox option is enabled, Comet runs the backup job as Administrator under a VSS snapshot.

Error "language file uses a codepage that is not supported on this system, using ACP" 

This error message can occur when generating branded software downloads from Comet Server for Linux that does not have all the system locales installed. The resulting Comet Backup software installer might not be usable in multiple languages.

You can resolve this issue by installing missing system locales.

System locales are an ISO C and POSIX standard that is implemented by glibc on most GNU/Linux distributions.

For Debian, and for Ubuntu 16.04 and later:

  • install the locales-all package

For versions of Ubuntu before 16.04:

  • install the locales package
  • copy /usr/share/i18n/SUPPORTED to /var/lib/locales/supported.d/all to mark all locales for installation
  • run locales-gen

For Arch Linux:

For other Linux distributions:

  • install system locales by a distribution-specific method

Antivirus detects Comet Backup as a virus or malware 

Comet Backup is a safe application. Any such detection is a "false positive".

When Comet Backup is rebranded, it might seem like a new, unknown program. An unknown program that installs system services, accesses files on the disk and uploads them to the network, might be considered to be malware if it was installed without consent. Unfortunately it's understandable for an Antivirus product to detect this.

In this situation, there are some actions you can take:

  • Please ensure your Antivirus product is fully up-to-date.
  • Please contact Comet Support with a screenshot of the error message. In some situations, it may be possible for our developers to resolve the issue.
  • Choose to "allow" or "white-list" the file in the Antivirus software. This may send a signal to the Antivirus software vendor that the software is safe (e.g. ESET LiveGrid, Windows Defender Automatic Sample Submission, Kaspersky KSN, etc).
  • Enable Authenticode signing on Windows. This may provide additional "reputation" to the software installer.