New migration (#17)

* Begin support for CentOS > Ubuntu migration.

* Don't rely on the script being on the source host.  Copy it there each
  time.
* Remove the prep and unprep options.  OpenSSH must be enabled on the
  remote host.
* Allow for moving from 2.2.2 to 2.3.0.
* Fail if all labs cannot be stopped.

* WIP: Switch to rsync for migration

Support broader migration targets and handle the case where libvirt
might not be an overlay (in 2.3).

TODO: Need to support non-overlay for offline migration and need to call
the actual migration script once the restore/migration is complete.

* Format and fix some function use.

* Mount the overlay on the right machine.

* Begin to add offline support.

TODO: Determine the correct place to extract refplat images.

* Map LIBVIRT_IMAGES to LIBVIRT_IMAGES.

* Don't assume the script is in /usr/local/bin.

* Don't wait for labs to stop when we don't stop them.

* Fix a number of bugs in migration.

* Fix a bug in the use of `cut`.

While here, make rsync show progress and ensure we use rsync as root on
the remote machine.

* Refactor when the data migration is done.

* Make sure we copy the right local directories.

Also, fix rsync command line to enable compression.

* Fix the conversion of libvirt from CentOS to Ubuntu.

* Make sure migration mode is properly passed.

Also, handle image copying correctly.

* Check to see if we're migrating from the local host.

* Make sure we check an IP.

* Check that hostname resolution is working.

* Allow alembic to handle the libvirt domain changes.

* Fix permissions of node disks.

Also, don't mask errors.

* Add a comment.

* Flesh out support for offline migration.

* Make the backup file very explicit.

* Don't try and mount the overlay if we don't need it.

Also add support for running without confirmation.

* Fix some issues spotted by shellcheck.

* Update docs for version migration.

* Fix logic error to allow for migration.

* Expose tar output so people can see progress.

* Add progress for tar via checkpoints.

Use checksum verification to decide what to rsync.

* Provide a better means to show progress.

Submitted by:	Justin Guagliata

* Remove errant '-'.

* Remove the 'z'.

* This needs to be exported.

* Add a single occurrence for each tar extract command.

* Some fixes.

* Move back to straight tar to fix estimates
* Fix double extract on restore
* Uniform progress indicators for backup and restore

Submitted by:	Justin Guagliata

* Add some signal traps.

* Add some echoes to clean up the output.

* Only remove the backup file on error.

* Use grep on a string properly.

* Give more verbosity to rsync.

* Some fixes from Tomas.

* Exclude *.orig and *.rej files that may be found in node defs.
* Make sure we only migrate mapped dirs when doing migration.

* Add support for migrating libvirt networks.

* Added Palo Alto

* ignore vscode workspace

* added readme and renamed directory

* removed vscode workspace file

* update

* rename directory palo-alto

* edited while loop to account for deleted devices

this will max out at 100 nodes added. 
to add more than 100 nodes, edit line 26

* updated while loop to account for deleted devices

this will now check all nodes from n0 to n99. 
I found that deleted nodes breaks the previous script because after 1 node was not found, it ended. 
to check for nodes higher than n99, edit line 24

* Update main.py

* updated to account for deleted nodes

* Add Cumulus VX image

* Updated contributer list

* Add Charlie Fenwick's cool use case around Jupyter.

This use case demonstrates how a Jupyter notebook can be used to
provision a new lab from zero (with a nod toward Ansible as the way to
configure it).

* fixed readability / adjusted for login.yaml

* Create login.yaml

* added support for more default devices

added support for:
unmanaged switches
asav
ftdv
server
alpine
coreos
desktop
ubuntu

* Make user configurable.

Also, do not migrate the default network.

* Add support for migrating libvirt ifaces.

* Add the --get-ifaces command.

* Update docs with --user option.

* Support iface-list differences.

* Fix numerous issues.

* Add a missing sudo.

* Make sure we account for migration when checking for space.

* Fix typo.

Reported by:	Tomas Mikuska

* Fix some markdown issues and clarify bridge migration.

* More markdown consistency fixes.

* Add the SD-WAN as code use case.

* This is done as part of cleanup.

* Add version info and a --version flag for displaying it.

* Add the ability to update the script itself from GitHub.

* Use an equals expression.

* Fix version comparison.

* Fix --version comflict.

* Bump version.

* Document the --update flag.

* Make the URL a bit more modular.

Co-authored-by: Alexander Deca <alexander@deca-consulting.be>
Co-authored-by: M35a2 <49795550+M35a2@users.noreply.github.com>

scripts/migration-tool/README.md

@@ -1,16 +1,18 @@
-# Migrating Data Between CML Servers
+# Copying Data Between CML Servers
 
-The `virl2-migrate-data.sh` script allows you to migrate CML configuration, lab, node, and image data between two CML servers.  Migration can be done "online" whereby the data is copied directly from one system to another; or "offline" where the data is first archived, and then the archive can be transferred to the new system and then extracted.
+The `virl2-migrate-data.sh` script allows you to copy CML configuration, lab, node, and image data between two CML servers.  Copying can be done "online" whereby the data is copied directly from one system to another; or "offline" where the data is first archived, and then the archive can be transferred to the new system and then extracted.
+
+You can also migration between CML 2.2 and CML 2.3.  This is a special case as the underlying OS has changed from CentOS to Ubuntu.  Both online and offline migrations are supported.
 
 The script requires the following:
 
--   Both CML servers must be running the _exact same version of CML_.
--   The target CML server must have sufficient disk space to copy all of the data from the source CML server (in online operation).
--   The source server must have sufficient disk space to hold an archive of all of its data (in offline operation).
+- Both CML servers must be running the *exact same version of CML* (unless doing a version migration).
+- The target CML server must have sufficient disk space to copy all of the data from the source CML server (in online operation).
+- The source server must have sufficient disk space to hold an archive of all of its data (in offline operation).
 
 ## Installation
 
-To perform any migration, the `virl2-migrate-data.sh` script needs to be copied to both servers.  The best way to transfer the script to a CML server is to place it on an external SFTP or SCP server, and then execute the following from Cockpit's Terminal as root (i.e., after using `sudo -E -s`):
+To perform any data copy or version migration, the `virl2-migrate-data.sh` script needs to be copied to the target (i.e., *new* server).  The best way to transfer the script to a CML server is to place it on an external SFTP or SCP server, and then execute the following from Cockpit's Terminal as root (i.e., after using `sudo -E -s`):
 
 SFTP:
 
@@ -34,69 +36,93 @@ In both cases, then run the following to make the script executable:
 # chmod +x virl2-migrate-data.sh
 ```
 
-## What Is Migrated?
+> If you are doing an offline copy or version migration, you must copy the script to both the source and target CML servers.
 
-The script migrates the following data:
+## What Is Copied?
 
--   Labs and the non-volatile and data storage for nodes within the labs
--   Custom node definitions
--   Custom image definitions
--   CML server config including users and groups
+The script copies the following data:
 
-Notably, what is **not** migrated are:
+- Labs and the non-volatile and data storage for nodes within the labs
+- Custom node definitions
+- Custom image definitions
+- CML server config including users and groups
 
--   Licensing (the target server must already have its own license)
--   Stock node and image definitions (make sure you either have the same refplat ISO mounted on or contents copied to the target server)
--   Custom bridge interfaces on the Linux side
+Notably, items **not** copied are:
+
+- Licensing (the target server must already have its own license)
+- Stock node and image definitions (make sure you either have the same refplat ISO mounted on or contents copied to the target server)
+  - **Note:** Version migration *does* copy stock node and image definitions
+- Custom bridge interfaces on the Linux side
+  - **Note**: The bridge interfaces are copied from a virtualization standpoint, but the underlying bridge-to-physical interface mappings are not
 
 ## General Operations
 
-The script itself will handle the shutting down and restarting of CML services on both the source and target servers.  However, it is **strongly** recommended that you shutdown all running labs/nodes on both servers prior to beginning migration.
+The script itself will handle the shutting down and restarting of CML services on both the source and target servers.  However, it is **strongly** recommended that you shutdown all running labs/nodes on both servers prior to beginning the data copy or version migration.
 
-## Online Migration
+## Online Copy or Version Migration
 
-To perform an online, server-to-server transfer, on the source server, run the following command to enable the OpenSSH service:
+To perform an online, server-to-server copy or version migration, on the source server, run the following commands to enable the OpenSSH service:
 
 ```bash
-$ sudo /usr/local/bin/virl2-migrate-data.sh --prep
+sudo systemctl enable sshd
+sudo systemctl start sshd
 ```
 
 Back on the target server, run the following command to initiate the transfer:
 
 ```bash
-$ sudo /usr/local/bin/virl2-migrate-data.sh --host <SOURCE CML SERVER IP>
+sudo /usr/local/bin/virl2-migrate-data.sh --host <SOURCE CML SERVER IP>
 ```
 
 Here, `<SOURCE CML SERVER IP>` is the IP address or hostname of the source CML server.  This will prompt you to confirm a few things and do some checks to ensure the migration is most likely to succeed.
 
-When migration completes, you can disable the OpenSSH server on the source server to maintain tight security.  Run the following command to do so:
+When migration completes, you can disable the OpenSSH server on the source server to maintain tight security.  Run the following commands to do so:
 
 ```bash
-$ sudo /usr/local/bin/virl2-migrate-data.sh --unprep
+sudo systemctl stop sshd
+sudo systemctl disable sshd
 ```
 
 > **Note:** This is an optional step... If the OpenSSH service was already enabled prior to running the migration tool then it is up to the system administrator whether they want to leave it enabled or disable it at this point.
 
-## Offline Migration
+## Offline Copy or Version Migration
 
-To perform an offline, archive file migration, run the following command on the source server to create an archive of all configuration, lab, node, and image data:
+To perform an offline, archive file copy, run the following command on the source server to create an archive of all configuration, lab, node, and image data:
 
 ```bash
-$ sudo /usr/local/bin/virl2-migrate-data.sh --backup --file /path/to/archive.tar
+sudo /usr/local/bin/virl2-migrate-data.sh --backup --file /path/to/archive.tar
+```
+
+If you will be doing a version migration (i.e., from 2.2 to 2.3), run the following command instead:
+
+```bash
+sudo /usr/local/bin/virl2-migrate-data.sh --backup --migration --file /path/to/archive.tar
 ```
 
 You can specify a file path for the archive where ever you have the requisite disk space.  The backup command will perform some source checks and then create this archive tar file.
 
 Once the backup portion completes, transfer this archive file to the target CML server.  How you do this transfer is up to you.  Using an intermediate SFTP or SCP server might be the easiest way.
 
-When the archive file is on the target server, run the following command to restore it:
+When the archive file is on the target server, run the following command to restore it (this is the same command for both data copy and version migration):
 
 ```bash
-$ sudo /usr/local/bin/virl2-migrate-data.sh --restore --file /path/to/archive.tar
+sudo /usr/local/bin/virl2-migrate-data.sh --restore --file /path/to/archive.tar
 ```
 
 The restore command will perform target checks and then prompt you to confirm you want to restore the source data onto this server.
 
+## Updating the Script
+
+If you run the script with the `--update` flag, it will check GitHub to see if a new release is available.  If so, it will download the new script (overwriting itself) and prepare itself for execution.  If there are no updates available, the script will not be touched.  For example:
+
+```bash
+/usr/local/bin/virl2-migrate-data.sh --update
+```
+
+This step isn't typically required, but if you are running into problems it can be something to try before reporting the problem upstream.
+
 ## Caveats
 
-This script is distributed as-is without formal support.  While it has been tested to work, it cannot anticipate all conditions of the source and target CML servers and may not properly migrate all data in all cases.  Prior to running the script, you should have a backup of the source CML server (and/or VMware snapshot) just in case something goes wrong.  You should also wait to delete the source server after migration until you have thoroughly tested the target and confirm all functionality is properly working.
+This script is distributed as-is without formal support.  While it has been tested to work, it cannot anticipate all conditions of the source and target CML servers and may not properly copy all data in all cases.  Prior to running the script, you should have a backup of the source CML server (and/or VMware snapshot) just in case something goes wrong.  You should also wait to delete the source server after migration until you have thoroughly tested the target and confirm all functionality is properly working.
+
+If you have renamed the *sysadmin* user, you must pass the `--user` parameter to the migration command when doing online migration/copying.  For example: `--user cmladmin`.