For similar capabilities to Amazon Timestream for LiveAnalytics, consider Amazon Timestream for InfluxDB. It offers simplified data ingestion and single-digit millisecond query response times for real-time analytics. Learn more [here](https://docs.aws.amazon.com//timestream/latest/developerguide/timestream-for-influxdb.html). # Migrating data from self-managed InfluxDB to Timestream for InfluxDB The [Influx migration script](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influx-migration) is a Python script that migrates data between InfluxDB OSS instances, whether those instances are managed by AWS or not. InfluxDB is a time series database. InfluxDB contains *points*, which contain a number of key-value pairs and a timestamp. When points are grouped by key-value pairs, they form a series. A series is grouped by a string identifier called a *measurement*. InfluxDB is often used for operations monitoring, IOT data, and analytics. A *bucket* is a kind of container within InfluxDB to store data. AWS-managed InfluxDB is InfluxDB within the AWS ecosystem. InfluxDB provides the InfluxDB v2 API for accessing data and making changes to the database. The InfluxDB v2 API is what the Influx migration script uses to migrate data. + The Influx migration script can migrate buckets and their metadata, migrate all buckets from all organizations, or do a full migration, which replaces all data on the destination instance. + The script backups data from the source instance locally, on whatever system executes the script, then restores the data to the destination instance. The data is kept in code>influxdb-backup- directories, one for each migration. + The script provides a number of options and configurations including mounting S3 buckets to limit local storage usage during migration and choosing which organizations to use during migration. **Topics** + [Preparation](timestream-for-influx-getting-started-migrating-data-prepare.md) + [How to use scripts](timestream-for-influx-getting-started-migrating-data-using-script.md) + [Migration Overview](timestream-for-influx-getting-started-migrating-data-overview.md) # Preparation Data migration for InfluxDB is accomplished with a Python script that utilizes InfluxDB CLI features and the InfluxDB v2 API. Execution of the migration script requires the following environment configuration: + **Supported Versions:** A minimum version of 2.3 of InfluxDB and Influx CLI is supported. + **Token Environment Variables** + Create the environment variable `INFLUX_SRC_TOKEN` containing the token for your source InfluxDB instance. + Create the environment variable `INFLUX_DEST_TOKEN` containing the token for your destination InfluxDB instance. + **Python 3** + Check installation: `python3 --version`. + If not installed, install from the Python website. Minimum version 3.7 required. On Windows the default Python 3 alias is simply python. + The Python module requests is required. Install it with: `shell python3 -m pip install requests` + TThe Python module influxdb\$1client is required. Install it with: `shell python3 -m pip install influxdb_client` + **InfluxDB CLI** + Confirm installation: `influx version`. + If not installed, follow the installation guide in the [InfluxDB documentation](https://docs.influxdata.com/influxdb/cloud/tools/influx-cli/#install-the-influx-cli). Add influx to your \$1PATH. + **S3 Mounting Tools (Optional)** When S3 mounting is used, all backup files are stored in a user-defined S3 bucket. S3 mounting can be useful to save space on the executing machine or when backup files need to be shared. If S3 mounting isn't used, by omitting the `--s3-bucket` option, then a local `influxdb-backup-` directory will be created to store backup files in the same directory that the script was run. For Linux: [mountpoint-s3](https://github.com/awslabs/mountpoint-s3). For Windows: [rclone](https://rclone.org/) (Prior rclone configuration is needed). + **Disk Space** + The migration process automatically creates unique directories to store sets of backup files and retains these backup directories in either S3 or on the local filesystem, depending on the program arguments provided. + Ensure there is enough disk space for database backup, ideally double the size of the existing InfluxDB database if you choose to omit the `--s3-bucket` option and use local storage for backup and restoration. + Check space with `df -h (UNIX/Linux)` or by checking drive properties on Windows. + **Direct Connection** Ensure a direct network connection exists between the system running the migration script and the source and destination systems. `influx ping --host ` is one way to verify a direct connection. # How to use scripts A simple example of running the script is the command: ``` python3 influx_migration.py --src-host --src-bucket --dest-host ``` Which migrates a single bucket. All options can be viewed by running: ``` python3 influx_migration.py -h ``` **Usage** ``` shell influx_migration.py [-h] [--src-bucket SRC_BUCKET] [--dest-bucket DEST_BUCKET] [--src-host SRC_HOST] --dest-host DEST_HOST [--full] [--confirm-full] [--src-org SRC_ORG] [--dest-org DEST_ORG] [--csv] [--retry-restore-dir RETRY_RESTORE_DIR] [--dir-name DIR_NAME] [--log-level LOG_LEVEL] [--skip-verify] [--s3-bucket S3_BUCKET] ``` **Options** + **-confirm-full** (optional): Using `--full` without `--csv` will replace all tokens, users, buckets, dashboards, and any other key-value data in the destination database with the tokens, users, buckets, dashboards, and any other key-value data in the source database. `--full` with `--csv` only migrates all bucket and bucket metadata, including bucket organizations. This option (`--confirm-full`) will confirm a full migration and proceed without user input. If this option is not provided, and `--full` has been provided and `--csv` not provided, then the script will pause for execution and wait for user confirmation. This is a critical action, proceed with caution. Defaults to false. + **-csv** (optional): Whether to use csv files for backing up and restoring. If `--full` is passed as well then all user-defined buckets in all organizations will be migrated, not system buckets, users, tokens, or dashboards. If a singular organization is desired for all buckets in the destination server instead of their already-existing source organizations, use `--dest-org`. + **-dest-bucket DEST\$1BUCKET** (optional): The name of the InfluxDB bucket in the destination server, must not be an already existing bucket. Defaults to value of `--src-bucket` or `None` if `--src-bucket` not provided. + **-dest-host DEST\$1HOST**: The host for the destination server. Example: http://localhost:8086. + **-dest-org DEST\$1ORG** (optional): The name of the organization to restore buckets to in the destination server. If this is omitted, then all migrated buckets from the source server will retain their original organization and migrated buckets may not be visible in the destination server without creating and switching organizations. This value will be used in all forms of restoration whether a single bucket, a full migration, or any migration using csv files for backup and restoration. + **-dir-name DIR\$1NAME** (optional): The name of the backup directory to create. Defaults to `influxdb-backup-`. Must not already exist. + **-full** (optional): Whether to perform a full restore, replacing all data on destination server with all data from source server from all organizations, including all key-value data such as tokens, dashboards, users, etc. Overrides `--src-bucket` and `--dest-bucket`. If used with `--csv`, only migrates data and metadata of buckets. Defaults to false. + **h, --help**: Shows help message and exits. + **-log-level LOG\$1LEVEL**(optional): The log level to be used during execution. Options are debug, error, and info. Defaults to info. + **-retry-restore-dir RETRY\$1RESTORE\$1DIR**(optional): Directory to use for restoration when a previous restore failed, will skip backup and directory creation, will fail if the directory doesn't exist, can be a directory within an S3 bucket. If a restoration fails, the backup directory path that can be used for restoration will be indicated relative to the current directory. S3 buckets will be in the form `influxdb-backups//`. The default backup directory name is `influxdb-backup-`. + **-s3-bucket S3\$1BUCKET**(optional): The name of the S3 bucket to use to store backup files. On Linux this is simply the name of the S3 bucket, such as `amzn-s3-demo-bucket1`, given `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables have been set or `${HOME}/.aws/credentials` exists. On Windows, this is the `rclone` configured remote and bucket name, such as `my-remote:amzn-s3-demo-bucket1`. All backup files will be left in the S3 bucket after migration in a created `influxdb-backups-` directory. A temporary mount directory named `influx-backups` will be created in the directory from where this script is ran. If not provided, then all backup files will be stored locally in a created `influxdb-backups-` directory from where this script is run. + **-skip-verify**(optional): Skip TLS certificate verification. + **-src-bucket SRC\$1BUCKET**(optional): The name of the InfluxDB bucket in the source server. If not provided, then `--full` must be provided. + **-src-host SRC\$1HOST**(optional): The host for the source server. Defaults to http://localhost:8086. As noted previously, `mountpoint-s3` and `rclone` are needed if `--s3-bucket` is to be used, but can be ignored if the user doesn't provide a value for `--s3-bucket`, in which case backup files will be stored in a unique directory locally. # Migration Overview After meeting the prerequisites: 1. Run Migration Script: Using a terminal app of your choice, run the Python script to transfer data from the source InfluxDB instance to the destination InfluxDB instance. 1. Provide Credentials: Provide host addresses and ports as CLI options. 1. Verify Data: Ensure the data is correctly transferred by: 1. Using the InfluxDB UI and inspecting buckets. 1. Listing buckets with `influx bucket list -t --host --skip-verify`. 1. Using `influx v1 shell -t --host --skip-verify` and running `SELECT * FROM .. LIMIT 100 to view contents of a bucket or SELECT COUNT(*) FROM ..` to verify the correct number of records have been migrated. **Example run** 1. Open a terminal app of your choice and make sure the required prerequisites are properly installed: ![\[Script prerequisites.\]](http://docs.aws.amazon.com/timestream/latest/developerguide/images/script-pre-reqs.png) 1. Navigate to the migration script: ![\[Script location\]](http://docs.aws.amazon.com/timestream/latest/developerguide/images/script-navigate.png) 1. Prepare the following information: 1. Name of the source bucket to be migrated. 1. (Optional) Choose a new bucket name for the migrated bucket in the destination server. 1. Root token for source and destination influx instances. 1. Host address of source and destination influx instances. 1. (Optional) S3 bucket name and credentials; AWS Command Line Interface credentials should be set in the OS environment variables. ``` # AWS credentials (for timestream testing) export AWS_ACCESS_KEY_ID="xxx" export AWS_SECRET_ACCESS_KEY="xxx" ``` 1. Construct the command as: ``` python3 influx_migration.py --src-bucket [amzn-s3-demo-source-bucket] --dest-bucket [amzn-s3-demo-destination-bucket] --src-host [source host] --dest-host [dest host] --s3-bucket [amzn-s3-demo-bucket2](optional) --log-level debug ``` 1. Execute the script: ![\[Script execution\]](http://docs.aws.amazon.com/timestream/latest/developerguide/images/script-execution.png) 1. Wait for the script to finish executing. 1. Check the newly migrated bucket for data integrity, `performance.txt`. This file, located under the same directory where the script was run, contains some basic information on how long each step took. ## Migration scenarios **Example 1: Simple Migration Using Local Storage** You want to migrate a single bucket, amzn-s3-demo-primary-bucket, from the source server `(http://localhost:8086)` to a destination server `(http://dest-server-address:8086)`. After ensuring you have TCP access (for HTTP access) to both machines hosting the InfluxDB instances on port 8086 and you have both source and destination tokens and have stored them as the environment variables `INFLUX_SRC_TOKEN` and `INFLUX_DEST_TOKEN`, respectively, for added security: ``` python3 influx_migration.py --src-bucket amzn-s3-demo-primary-bucket --src-host http://localhost:8086 --dest-host http://dest-server-address:8086 ``` The output should look similar to the following: ``` INFO: influx_migration.py: Backing up bucket data and metadata using the InfluxDB CLI 2023/10/26 10:47:15 INFO: Downloading metadata snapshot 2023/10/26 10:47:15 INFO: Backing up TSM for shard 1 2023/10/26 10:47:15 INFO: Backing up TSM for shard 8245 2023/10/26 10:47:15 INFO: Backing up TSM for shard 8263 [More shard backups . . .] 2023/10/26 10:47:20 INFO: Backing up TSM for shard 8240 2023/10/26 10:47:20 INFO: Backing up TSM for shard 8268 2023/10/26 10:47:20 INFO: Backing up TSM for shard 2 INFO: influx_migration.py: Restoring bucket data and metadata using the InfluxDB CLI 2023/10/26 10:47:20 INFO: Restoring bucket "96c11c8876b3c016" as "amzn-s3-demo-primary-bucket" 2023/10/26 10:47:21 INFO: Restoring TSM snapshot for shard 12772 2023/10/26 10:47:22 INFO: Restoring TSM snapshot for shard 12773 [More shard restores . . .] 2023/10/26 10:47:28 INFO: Restoring TSM snapshot for shard 12825 2023/10/26 10:47:28 INFO: Restoring TSM snapshot for shard 12826 INFO: influx_migration.py: Migration complete ``` The directory `influxdb-backup-` will be created and stored in the directory from where the script was run, containing backup files. **Example 2: Full Migration Using Local Storage and Debug Logging** Same as above except you want to migrate all buckets, tokens, users, and dashboards, deleting the buckets in the destination server, and proceeding without user confirmation of a complete database migration by using the `--confirm-full` option. You also want to see what the performance measurements are so you enable debug logging. ``` python3 influx_migration.py --full --confirm-full --src-host http://localhost:8086 --dest-host http://dest-server-address:8086 --log-level debug ``` The output should look similar to the following: ``` INFO: influx_migration.py: Backing up bucket data and metadata using the InfluxDB CLI 2023/10/26 10:55:27 INFO: Downloading metadata snapshot 2023/10/26 10:55:27 INFO: Backing up TSM for shard 6952 2023/10/26 10:55:27 INFO: Backing up TSM for shard 6953 [More shard backups . . .] 2023/10/26 10:55:36 INFO: Backing up TSM for shard 8268 2023/10/26 10:55:36 INFO: Backing up TSM for shard 2 DEBUG: influx_migration.py: backup started at 2023-10-26 10:55:27 and took 9.41 seconds to run. INFO: influx_migration.py: Restoring bucket data and metadata using the InfluxDB CLI 2023/10/26 10:55:36 INFO: Restoring KV snapshot 2023/10/26 10:55:38 WARN: Restoring KV snapshot overwrote the operator token, ensure following commands use the correct token 2023/10/26 10:55:38 INFO: Restoring SQL snapshot 2023/10/26 10:55:39 INFO: Restoring TSM snapshot for shard 6952 2023/10/26 10:55:39 INFO: Restoring TSM snapshot for shard 6953 [More shard restores . . .] 2023/10/26 10:55:49 INFO: Restoring TSM snapshot for shard 8268 2023/10/26 10:55:49 INFO: Restoring TSM snapshot for shard 2 DEBUG: influx_migration.py: restore started at 2023-10-26 10:55:36 and took 13.51 seconds to run. INFO: influx_migration.py: Migration complete ``` **Example 3: Full Migration Using CSV, Destination Organization, and S3 Bucket** Same as the previous example but using Linux or Mac and storing the files in the S3 bucket, `amzn-s3-demo-bucket`. This avoids backup files overloading the local storage capacity. ``` python3 influx_migration.py --full --src-host http://localhost:8086 --dest-host http://dest-server-address:8086 --csv --dest-org MyOrg --s3-bucket amzn-s3-demo-bucket ``` The output should look similar to the following: ``` INFO: influx_migration.py: Creating directory influxdb-backups INFO: influx_migration.py: Mounting amzn-s3-demo-influxdb-migration-bucket INFO: influx_migration.py: Creating directory influxdb-backups/amzn-s3-demo-bucket/influxdb-backup-1698352128323 INFO: influx_migration.py: Backing up bucket data and metadata using the InfluxDB v2 API INFO: influx_migration.py: Restoring bucket data and metadata from csv INFO: influx_migration.py: Restoring bucket amzn-s3-demo-some-bucket INFO: influx_migration.py: Restoring bucket amzn-s3-demo-another-bucket INFO: influx_migration.py: Restoring bucket amzn-s3-demo-primary-bucket INFO: influx_migration.py: Migration complete INFO: influx_migration.py: Unmounting influxdb-backups INFO: influx_migration.py: Removing temporary mount directory ```