🚧 images, logging

.pre-commit-config.yaml

@@ -0,0 +1,25 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    -   id: check-added-large-files
+-   repo: https://github.com/psf/black
+    rev: 23.10.0
+    hooks:
+    -   id: black
+        args: ["--line-length=79"]
+-   repo: https://github.com/PyCQA/isort
+    rev: 5.12.0
+    hooks:
+    - id: isort
+      args: ["--profile", "black"]
+-   repo: https://github.com/PyCQA/flake8
+    rev: 6.1.0
+    hooks:
+    - id: flake8
+      exclude: migrations/

README.md

@@ -1,6 +1,4 @@
-A quick and dirty script to help set up a RHOSP AIO
+A set of quick and dirty scripts to help set up a RHOSP AIO
-
-This has grown a bit. At some point I will make this a proper readme, but for now a few notes:
 
 The goal is to build a VM (although you could use baremetal) and deploy a RHOSP AIO to it. Then to add a project, flavors, images, networks to support testing os-migrate.
 
@@ -8,20 +6,38 @@ After that create a VM to migrate, a conversion host to do the migration.
 
 This may well be useful for other things, but this is the current aim.
 
+There are/will be example scripts in the examples directory
+
+Everything is a bit of a mess but here's a brief overview:
+
+## create_aio_vm.py
+
+Uses virt-install to build a VM to deploy a RHOSP 16 AIO on.
+
 I am primarily targetting RHEL8.4 and RHOSP16.2. As a result I'm tied to virt-install v3.2.0 which makes using cloud-init for configuring network on the AIO VM a little annoying. Later versions allow you to pass in a network config, but 3.2.0 means I have to use nmcli and bootcmd to get the network configure.
 
 There is a vestigial network-config.tpl in the virt-install directory but it is not actually used yet.
 
-EVerything is a bit of a mess but here's a brief overview:
+## os_migrate_setup.py
 
-main.py - original script. Generates the yaml files and a dploy script to deploy a RHOSP AIO. Assumes a VM to run on. Don't run this on your laptop. You'll be sorry if you do.
+build the OSP infrastructure to allow testing os-migrate.
 
-os_migrate_setup.py - build the OSP infrastructure to allow testing os-migrate.
+## os_migrate_teardown.py
 
-os_migrate_teardown.py - remove that infrastructure (to allow for refinement/iteration)
+remove that infrastructure (to allow for refinement/iteration)
 
 virt-install/create_aio_vm.py - use virt-install to create and configure a VM to run the other scripts on.
 
+## prepare-deployment
+
+Original script. Generates the yaml files and a dploy script to deploy a RHOSP AIO. Assumes a VM to run on. Don't run this on your laptop. You'll be sorry if you do.
+
+## os_migrate_validate.py
+
+Check that the necessary infrastructure has been deployed.
+
+# Remember to mask and disable cloud-init if installing on a VM
+
 ```
 usage: main.py [-h] -u USERNAME -p PASSWORD -a ADDRESS [-i INTERFACE]
                [-m NETMASK] -d DNS [DNS ...] [-D DOMAIN]

TODO.md

@@ -0,0 +1,21 @@
+# Tests
+
+There aren't any which. At least basic unit tests would be good.
+
+# Documentation
+
+There's a README, but it should be expanded.
+
+# Error handling.
+
+There's very litle and this is a complex process involving talking to other systems over the net.
+
+# Convert to Ansible
+
+Maybe this should all have been an ansible role.  Maybe that will happen yet.
+
+# Ideas for specific scripts:
+
+## create_aio_vm
+
+Quite a few compromises to do with the versions of virt_install / libvirt available to me.

dev_requirements.txt

@@ -4,3 +4,4 @@ flake8==5.0.4
 pyflakes==2.5.0
 pylint==2.13.9
 PyYAML==6.0.1
+pre-commit

docs/create_aio_vm.md

@@ -0,0 +1,5 @@
+# Documentation for create_aio_vm
+
+## Networking
+
+Networking has been a real pain. The RHOSP AIO deployment seems to force the vr-ctlplane interface to use dhcp. Originally I was trying to use static IPs configured with cloud-init (using bootcmd because of the version) but the deploy would die. To fix this I now speciiy MACs for the interfaces and then use an external DHCP server. Maybe this is configurable in the RHOSP standalone deploy. If so it would be better to use that. It would also be nice to do some feature switches so that if the version of virt-install allows I could use network-config instead of bootcmd

docs/notes.md

@@ -0,0 +1,84 @@
+# Installing os-migrate
+
+I now have two RHOSP-16.2 AIOs running on my homelab so it's time to install os-migrate.
+
+The intallation instructions are at: https://os-migrate.github.io/os-migrate/user/install-from-galaxy.html
+
+I setup a new Fedora 38 VM to run os-migrate from:
+
+```bash
+virt-install -n os-migrate \
+  --osinfo fedora37 \
+  --memory 2048 \
+  --vcpus 2 \
+  --import \
+  --disk /data/os-migrate.qcow2 \
+  --graphics vnc \
+  --cloud-init disable=on,clouduser-ssh-key=ncox.pub,root-password-file=rpw  \
+  --network bridge=br0
+  ```
+
+  This VM is running at 172.23.0.53, on test-04(173.23.0.25)
+
+  RHEL9 doesn't know about fedora38
+
+  Lots of stuffing around to get the bridge working
+
+  ## Install and run os-miograte
+
+`dnf install ansible `
+
+`vi os-migrate-vars.yml`
+
+Largely copied from the ~/.config/openstack/clouds.yaml files
+
+
+Hints from os-migrate doco:
+
+
+```bash
+export OSM_DIR=/root/.ansible/collections/ansible_collections/os_migrate/os_migrate
+export OSM_CMD="ansible-playbook -v -i $OSM_DIR/localhost_inventory.yml -e @os-migrate-vars.yml"
+```
+
+Despite what the os-migrate docs say, do not install python3-openstacksdk because on f38 the version is too recent.
+
+```bash
+dnf install iputils python3-openstackclient
+```
+
+Let's try networks:
+
+```$OSM_CMD $OSM_DIR/playbooks/export_networks.yml```
+
+Hmm, problems with versions.
+
+Going to need devel tools for openstacksdk
+
+```
+dnf install python3-devel
+dnf group install "C Development Tools and Libraries"
+```
+
+We're going to need ansible-core in the venv or it will pick up system versions of things.
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install --upgrade pip
+python3 -m pip install --upgrade 'openstacksdk>=0.36,<0.99'
+python3 -m pip install --upgrade 'ansible-core'
+```
+
+
+Try again:
+
+```
+OSM_CMD $OSM_DIR/playbooks/export_networks.yml
+```
+
+```bash
+   60  less os-migrate-data-test-1/clouds.yaml
+   61  less os-migrate-data-test-1/networks.yml
+   67  $OSM_CMD $OSM_DIR/playbooks/export_subnets.yml
+   68  less os-migrate-data-test-1/subnets.yml
+```

examples/create_aio.bash

@@ -0,0 +1,20 @@
+AIO_NAME=test5
+
+
+create_aio_vm \
+  --password secrete123 \
+  --public-key ~/.ssh/id_rsa.pub \
+  --output-image /data/${AIO_NAME}.qcow2 \
+  --input-image ~/rhel-guest-image-8.4-1269.x86_64.qcow2  \
+  --image-size "800G" \
+  --os-variant rhel8.4 \
+  --name ${AIO_NAME} \
+  --local-hostname ${AIO_NAME} \
+  --instance-id ${AIO_NAME} \
+  --gateway 172.23.0.1 \
+  --dns 172.23.0.14 \
+  --mac-1 52:54:00:a5:48:03 \
+  --mac-2 52:54:00:a5:48:04 \
+  --search-domain evatt.ingenious.com.au \
+  --rhn-user $AIO_RHN_USER \
+  --rhn-password $AIO_RHN_PASSWORD

examples/install_remote_venv.bash

@@ -0,0 +1,19 @@
+REMOTE=172.23.0.34
+AIOH_ENV="/home/stack/tripleo-aio-helpers"
+AIO_SRC=$HOME/Projects
+VENV=$AIOH_ENV/venv
+rsync -avz --exclude venv $AIO_SRC/tripleo-aio-helpers stack@$REMOTE:
+ssh stack@$REMOTE  "sudo dnf install -y python3-tripleoclient"
+ssh stack@$REMOTE "cd $AIOH_ENV && python3 -m venv venv"
+ssh stack@$REMOTE "$VENV/bin/pip install --upgrade pip"
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && pip install -r requirements.txt"
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && pip install --editable ."
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && prepare_deployment -u $AIO_RHN_USER --password $AIO_RHN_PASSWORD --address 172.23.0.34 --interface eth1 --dns 172.23.0.14 --gateway 172.23.0.1"
+ssh stack@$REMOTE "sudo systemctl stop cloud-init"
+ssh stack@$REMOTE "sudo systemctl disable cloud-init"
+# ssh stack@$REMOTE "cp containers-prepare-parameters.yaml.gen containers-prepare-parameters.yaml"
+# ssh stack@$REMOTE "cp standalone_parameters.yaml.gen standalone_parameters.yaml"
+# ssh stack@$REMOTE "cp deploy.sh.gen deploy.sh"
+ssh stack@$REMOTE "sudo dnf install -y tmux"
+ssh stack@$REMOTE "echo \"tmux new -d \; setw remain-on-exit on \; respawnw -k bash deploy.sh\""
+# ssh stack@$REMOTE "OS_CLOUD=standalone openstack endpoint list"

examples/prepare_deployment.bash

@@ -0,0 +1,7 @@
+prepare_deployment \
+    -u $AIO_RHN_USER \
+    --password $AIO_RHN_PASSWORD \
+    --address 172.23.0.35/24 \
+    --interface eth1 \
+    --dns 172.23.0.14 \
+    --gateway 172.23.0.1

examples/setup.bash

@@ -0,0 +1,16 @@
+FIP_START=117
+FIP_END=129
+AIO_HOST=test-5
+
+echo "Setting up $AIO_HOST"
+os_migrate_setup \
+  --gateway 172.23.0.1 \
+  --public-network-cidr=172.23.0.1/24 \
+  --public-net-start 172.23.0.$FIP_START \
+  --public-net-end 172.23.0.$FIP_END \
+  --dns-server 172.23.0.14 \
+  --ssh stack@$AIO_HOST \
+  --cirros-url https://isos.evatt.ingenious.com.au/cirros-0.6.2-x86_64-disk.img \
+  --rhel-url https://isos.evatt.ingenious.com.au/rhel-guest-image-8.4-1269.x86_64.qcow2 \
+  --ssh-key ~/.ssh/id_rsa.pub \
+  --debug

examples/teardown.bash

@@ -0,0 +1,10 @@
+AIO_HOST=test-5
+
+echo "Tearing down $AIO_HOST"
+
+os_migrate_teardown \
+    --project-name test-project \
+    --username test-user \
+    --cloud standalone \
+    --ssh stack@$AIO_HOST \
+    --delete-all

pyproject.toml

@@ -25,5 +25,6 @@ classifiers = [
 create_aio_vm = "tripleo_aio_helpers.create_aio_vm:main"
 os_migrate_setup = "tripleo_aio_helpers.os_migrate_setup:main"
 os_migrate_teardown = "tripleo_aio_helpers.os_migrate_teardown:main"
-os_migrate_validate = "tripleo_aio_helpers.os_migrate_validate:main"
+os_migrate_validate_dst = "tripleo_aio_helpers.os_migrate_validate_dst:main"
+os_migrate_validate_src = "tripleo_aio_helpers.os_migrate_validate_src:main"
 prepare_deployment = "tripleo_aio_helpers.prepare_deployment:main"

requirements.txt

@@ -4,3 +4,4 @@
 # pyflakes==2.5.0
 # pylint==2.13.9
 PyYAML==6.0.1
+rich

src/tripleo_aio_helpers/create_aio_vm.py

@@ -14,6 +14,8 @@ ND_PATH = "./network-config"
 
 def parse_args():
     """Parse the command line arguments"""
+    # pylint: disable=too-many-statements
+
     template_path = pathlib.Path(__file__).parent.parent / "virt-install"
     parser = argparse.ArgumentParser()
     parser.add_argument("--password", required=True)
@@ -21,12 +23,10 @@ def parse_args():
     parser.add_argument("--local-hostname", required=True)
     parser.add_argument("--user-data", default=template_path / "user-data.tpl")
     parser.add_argument("--meta-data", default=template_path / "meta-data.tpl")
-    parser.add_argument(
+    parser.add_argument("--network-data", default=template_path / "network-config.tpl")
-        "--network-data", default=template_path / "network-config.tpl"
-    )
     parser.add_argument("--instance-id", required=True, help="Hostname for the new VM")
     parser.add_argument("--output-image", required=True)
-    parser.add_argument("--image-size", default="100G")
+    parser.add_argument("--image-size", default="800G")
     parser.add_argument("--input-image", required=True)
     parser.add_argument("--os-variant", required=True)
     parser.add_argument("--name", required=True)
@@ -35,9 +35,11 @@ def parse_args():
     parser.add_argument("--gateway")
     parser.add_argument("--cidr-1")
     parser.add_argument("--cidr-2")
-    parser.add_argument("--mac-1",default="RANDOM")
+    parser.add_argument("--mac-1", default="RANDOM")
-    parser.add_argument("--mac-2",default="RANDOM")
+    parser.add_argument("--mac-2", default="RANDOM")
-    parser.add_argument("--dns",)
+    parser.add_argument(
+        "--dns",
+    )
     parser.add_argument("--search-domain")
     parser.add_argument("-v", "--verbose", action="store_true")
     parser.add_argument("--rhn-user", required=True)
@@ -61,22 +63,18 @@ def parse_args():
         print("You must specify a DNS server if you specify any addresses")
         sys.exit(1)
 
-    if not (args.cidr_1 or args.cidr_2) and (args.dns or args.gateway or args.search_domain):
-        print("There's no point specifying DNS, gateway or search_domain if yoou don't specify addresses")
-        sys.exit(1)
-
     args.public_key = args.public_key.read()
 
-    with open(args.user_data) as user_data:
+    with open(args.user_data, encoding="utf-8") as user_data:
         args.user_data = user_data.read()
     generate_boot_cmd(args)
     args.user_data = args.user_data.format(data=args)
 
-    with open(args.meta_data) as meta_data:
+    with open(args.meta_data, encoding="utf-8") as meta_data:
         args.meta_data = meta_data.read()
     args.meta_data = args.meta_data.format(data=args)
 
-    with open(args.network_data) as network_data:
+    with open(args.network_data, encoding="utf-8") as network_data:
         args.network_data = network_data.read()
     args.network_data = args.network_data.format(data=args)
 
@@ -92,17 +90,24 @@ def parse_args():
 
     return args
 
+
 def generate_boot_cmd(data):
-    data.bootcmd=""
+    """Generate the bootcmd section of the user-data"""
+    data.bootcmd = ""
     if data.cidr_1:
         data.bootcmd = (
             "bootcmd:\n"
-            f'  - "nmcli con modify \'System eth0\' ipv4.address {data.cidr_1} ipv4.method static ipv4.gateway {data.gateway} ipv4.dns {data.dns}"\n'
+            f"  - \"nmcli con modify 'System eth0' ipv4.address {data.cidr_1}"
-            f'  - "nmcli con modify \'Wired connection 1\' ipv4.address {data.cidr_2} ipv4.method static ipv4.gateway {data.gateway} ipv4.dns {data.dns}"\n'
+            f" ipv4.method static ipv4.gateway {data.gateway} "
+            f'ipv4.dns {data.dns}"\n'
+            "  - \"nmcli con modify 'Wired connection 1' ipv4.address "
+            f"{data.cidr_2} ipv4.method static ipv4.gateway {data.gateway} "
+            f'ipv4.dns {data.dns}"\n'
             '  - "nmcli networking off"\n'
             '  - "nmcli networking on"\n'
         )
 
+
 def write_user_data(data):
     """Write out a temporary user data file"""
 
@@ -139,7 +144,7 @@ def create_image(args):
         print(result)
 
     print("Resizing image")
-    cmd = f"virt-resize --expand /dev/sda3 {args.input_image} {args.output_image}"
+    cmd = f"virt-resize --expand /dev/sda3 {args.input_image} " f"{args.output_image}"
 
     result = subprocess.check_output(cmd, shell=True, universal_newlines=True)
 
@@ -148,6 +153,7 @@ def create_image(args):
 
     print("Image resized")
 
+
 def virt_install_cmd(args):
     """Build and execute the virt-install command"""
     cmd = f"""
@@ -189,17 +195,12 @@ def delete_user_data():
 
 def install_packages(args):
     """Install the packages needed for virt-install to work"""
-    cmd = (
+    cmd = "sudo dnf install -y virt-install virt-viewer qemu-img " "libguestfs.x86_64"
-        "sudo dnf install -y virt-install virt-viewer qemu-img "
-        "libguestfs.x86_64"
-        )
 
     print("installing needed packages...")
 
     if args.verbose:
-        print(
+        print(subprocess.check_output(cmd, shell=True, universal_newlines=True))
-            subprocess.check_output(cmd, shell=True, universal_newlines=True)
-        )
 
 
 def main():

src/tripleo_aio_helpers/os_migrate_setup.py

@@ -3,8 +3,22 @@ Quick and dirty script to help setup project, flavors, networks, images
 """
 import argparse
 import json
+import subprocess
+import tempfile
+
+from .utils import get_from_env, openstack_cmd, test_user_openstack_cmd
+
+
+def execute_cmd(cmd):
+    """Execute a command"""
+    return subprocess.check_output(cmd, shell=True, universal_newlines=True)
+
+
+def execute_ssh_cmd(cmd, args):
+    """Execute a command on a remote host using ssh"""
+    cmd = f'ssh {args.ssh} "{cmd}"'
+    return subprocess.check_output(cmd, shell=True, universal_newlines=True)
 
-from .utils import get_from_env,openstack_cmd
 
 def parse_args():
     """Parse the command line arguments"""
@@ -20,12 +34,29 @@ def parse_args():
     parser.add_argument("-c", "--cloud", default="standalone")
     parser.add_argument("-g", "--gateway")
     parser.add_argument("-C", "--public-network-cidr")
+    parser.add_argument(
+        "--ssh-key",
+        help="File containing a public key to inject into the instances",
+    )
     parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
     parser.add_argument("--public-net-start")
     parser.add_argument("--public-net-end")
     parser.add_argument("--dns-server")
     parser.add_argument("--dry-run", action="store_true")
-    parser.add_argument("--ssh", help="Connection string to run commands on a remote host.")
+    parser.add_argument(
+        "--ssh", help="Connection string to run commands on a remote host."
+    )
+    parser.add_argument(
+        "--cirros-url",
+        help="a URL that a cirros image can be downloaded from. Required. "
+        "Can be set in AIO_CIRROS_URL",
+    )
+    parser.add_argument(
+        "--rhel-url",
+        help="a URL that a RHEL image can be downloaded from. Required. Can "
+        "be set in AIO_RHEL_URL",
+    )
+    parser.add_argument("--debug", action="store_true")
 
     # export OS_CLOUD=standalone
     # export STANDALONE_HOST=10.76.23.39
@@ -36,10 +67,14 @@ def parse_args():
         args.gateway = get_from_env("--gateway", "AIO_GATEWAY")
 
     if not args.public_network_cidr:
-        args.public_network_cidr = get_from_env("--public-network-cidr", "AIO_PUBLIC_CIDR")
+        args.public_network_cidr = get_from_env(
+            "--public-network-cidr", "AIO_PUBLIC_CIDR"
+        )
 
     if not args.public_net_start:
-        args.public_net_start = get_from_env("--public-net-start", "AIO_PUBLIC_NET_START")
+        args.public_net_start = get_from_env(
+            "--public-net-start", "AIO_PUBLIC_NET_START"
+        )
 
     if not args.public_net_end:
         args.public_net_end = get_from_env("--public-net-end", "AIO_PUBLIC_NET_END")
@@ -47,15 +82,24 @@ def parse_args():
     if not args.dns_server:
         args.dns_server = get_from_env("--dns-server", "AIO_DNS_SERVER")
 
+    if not args.cirros_url:
+        args.dns_server = get_from_env("--dns-server", "AIO_CIRROS_URL")
+
+    if not args.rhel_url:
+        args.dns_server = get_from_env("--dns-server", "AIO_RHEL_URL")
+
     if not args.ssh:
         args.ssh = get_from_env("--ssh", "AIO_SSH", required=False)
 
+    if not args.ssh_key:
+        args.ssh_key = get_from_env("--ssh-key", "AIO_SSH_KEY")
+
     return args
 
+
 def create_project(args):
     """Create the project if it doesn't already exist"""
 
-    
     cmd = "project list -f json"
     project_exists = [
         x
@@ -129,66 +173,370 @@ def assign_member_role(args):
 
 
 def create_public_network(args):
-    """Coming soon - create the public network"""
+    """Create the public network"""
-    # pylint: disable=unused-argument,unused-variable
+
-    print("creating public network - NYI")
+    network_exists = json.loads(
+        openstack_cmd("network list -f json --name public", args)
+    )
+
+    if network_exists:
+        print("Public network already exists - skipping")
+        args.public_network_id = network_exists[0]["ID"]
+    else:
         cmd = (
-        "network create --external --provider-physical-network datacentre "
+            "network create -f json  --external "
+            "--provider-physical-network datacentre "
             "--provider-network-type flat public"
         )
-    cmd = (
+        try:
-        f"subnet create public-net --subnet-range {args.public_network_cidr} "
+            args.public_network_id = json.loads(openstack_cmd(cmd, args))["id"]
-        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+        except json.decoder.JSONDecodeError:
-        f"start={args.public_net_start},end={args.public_net_end} "
+            print(cmd)
-        "--network public"
+            raise
+        print("Public network created.")
+
+    subnet_exists = json.loads(
+        openstack_cmd("subnet list -f json --name public-net", args)
     )
+    if subnet_exists:
+        print("Public subnet exists - skipping")
+    else:
+        cmd = (
+            "subnet create public-net -f json "
+            f"--subnet-range {args.public_network_cidr} "
+            f"--gateway {args.gateway} "
+            f"--allocation-pool start={args.public_net_start},"
+            f"end={args.public_net_end} "
+            "--network public "
+            f"--host-route destination=0.0.0.0/0,gateway={args.gateway} "
+            f"--dns-nameserver {args.dns_server}"
+        )
+        args.public_subnet_id = json.loads(openstack_cmd(cmd, args))
+        print("Public subnet created.")
 
 
 def create_private_network(args):
-    """Coming soon - create the private network"""
+    """Create the private network and subnet"""
-    # pylint: disable=unused-argument,unused-variable
+
-    cmd = "openstack network create --internal private"
+    network_exists = json.loads(
-    cmd = (
+        openstack_cmd(
-        "openstack subnet create private-net "
+            f"network list -f json --project {args.project_id} --name private",
-        f"--subnet-range {args.private_network_cidr} --network private"
+            args,
         )
-    print("creating private network - NYI")
+    )
+
+    if network_exists:
+        print("Private network already exists - skipping")
+        args.private_network_id = network_exists[0]["ID"]
+    else:
+        cmd = (
+            f"network create -f json --internal private " f"--project {args.project_id}"
+        )
+        args.private_network_id = json.loads(openstack_cmd(cmd, args))["id"]
+        print("Private network created.")
+
+    subnet_exists = json.loads(
+        openstack_cmd(
+            f"subnet list -f json --project {args.project_id} " f"--name private-net",
+            args,
+        )
+    )
+    if subnet_exists:
+        print("Private subnet exists - skipping")
+        args.private_subnet_id = subnet_exists[0]["ID"]
+    else:
+        cmd = (
+            f"subnet create private-net -f json --project {args.project_id} "
+            f"--subnet-range {args.private_network_cidr} "
+            f"--network {args.private_network_id}"
+        )
+        args.private_subnet_id = json.loads(openstack_cmd(cmd, args))
+        print("Private subnet created.")
+
+
+def create_flavor(args, flavor_name, memory, disk, cpus):
+    """Create a flavor - DRY"""
+    cmd = "flavor list -f json --all"
+    # Note we are going to assume that there is only one cirros flavor. This
+    # will be more of a problem come deletion time.
+    result = json.loads(openstack_cmd(cmd, args))
+    flavor_exists = [x for x in result if x["Name"] == flavor_name]
+    if flavor_exists:
+        print(f"{flavor_name} flavor already exists. Skipping creation")
+        args.__dict__[flavor_name + "_flavor"] = flavor_exists[0]["ID"]
+        return
+
+    print("creating cirros flavor")
+    # Note can't add a description in RHOSP16, but perhaps should add for 17
+    cmd = f"""
+        flavor create -f json
+            --ram {memory}
+            --disk {disk}
+            --vcpus {cpus}
+            --private
+            --project {args.project_id}
+            {flavor_name}
+        """.replace(
+        "\n", " "
+    )
+
+    result = openstack_cmd(cmd, args, as_json=True)
+    args.__dict__[flavor_name] = result["id"]
+    print(result)
 
 
 def create_cirros_flavor(args):
-    """Coming soon - create the cirros flavor"""
+    """create the cirros flavor"""
-    # pylint: disable=unused-argument
+    create_flavor(args, "cirros", 256, 20, 1)
-    print("creating cirros flavor - NYI")
 
 
 def create_rhel_flavor(args):
-    """Coming soon - create the rhel flavor"""
+    """create the rhel flavor"""
-    # pylint: disable=unused-argument
+    create_flavor(args, "rhel", 1536, 120, 2)
-    print("creating rhel flavor - NYI")
+
+
+def create_image(image_name, image_url, disk_size, memory, args):
+    """Create an image - DRY"""
+
+    cmd = "image list -f json"
+    image_exists = [
+        image
+        for image in json.loads(openstack_cmd(cmd, args))
+        if image["Name"] == image_name
+    ]
+
+    if image_exists:
+        print(f"{image_name} already exists - not creating.")
+        args.__dict__[image_name] = image_exists[0]["ID"]
+        return
+
+    with tempfile.TemporaryDirectory() as tmp_dir:
+        # download image to tmpdir
+        fname = f"{tmp_dir}/{image_name}.img"
+        cmd = f"wget -O {fname} {image_url}"
+        _ = execute_cmd(cmd)
+
+        if args.ssh:
+            # Copy img to remote host
+            cmd = f"scp {fname} {args.ssh}:{image_name}.img"
+            execute_cmd(cmd)
+
+        # create image - note this will only work on a remote ssh host at
+        # the moment
+        cmd = f"""
+            image create -f json
+                --container-format bare
+                --disk-format qcow2
+                --min-disk {disk_size} --min-ram {memory}
+                --file {image_name}.img
+                --private
+                --project {args.project_id}
+                {image_name}
+            """
+
+        result = openstack_cmd(cmd, args, as_json=True)
+        args.__dict__[image_name] = result["id"]
+
+        if args.ssh:
+            # Delete image from remote host
+            cmd = f"rm {image_name}.img"
+            execute_ssh_cmd(cmd, args)
 
 
 def create_cirros_image(args):
-    """Coming soon - create the cirros image"""
+    """create the cirros image"""
-    # pylint: disable=unused-argument
+
-    print("creating cirros image - NYI")
+    create_image("cirros_image", args.cirros_url, 20, 256, args)
 
 
 def create_rhel_image(args):
-    """Coming soon - create the rhel image"""
+    """create the rhel image"""
-    # pylint: disable=unused-argument
+    create_image("rhel_image", args.rhel_url, 120, 1536, args)
-    print("creating rhel image - NYI")
+
+
+def create_instance(args, name, flavor, image, security_group, boot_size):
+    """Create an instance"""
+    # pylint:disable=too-many-arguments
+    instance_exists = [
+        instance
+        for instance in test_user_openstack_cmd(
+            "server list -f json", args, as_json=True
+        )
+        if instance["Name"] == name
+    ]
+
+    if instance_exists:
+        print(f"{name} instance exists - skipping")
+    else:
+        cmd = (
+            f"server create --flavor {flavor} "
+            f"--image {image} "
+            f"--key-name test_keypair "
+            f"--security-group {security_group} "
+            "--network private "
+            f"--boot-from-volume {boot_size} "
+            "-f json "
+            f"{name}"
+        )
+
+        server = test_user_openstack_cmd(cmd, args, as_json=True)
+
+        print(f"{name} instance created")
+        # assign floating IP
+        fip = test_user_openstack_cmd(
+            "floating ip create -f json public", args, as_json=True
+        )
+        _ = test_user_openstack_cmd(
+            f"server add floating ip {server['id']} " f"{fip['floating_ip_address']}",
+            args,
+        )
 
 
 def create_cirros_instance(args):
-    """Coming soon - create the cirros instance"""
+    """Create the cirros instance"""
-    # pylint: disable=unused-argument
+
-    print("creating cirros instance - NYI")
+    create_instance(
+        args,
+        "cirros-test-instance",
+        args.cirros_flavor,
+        args.cirros_image,
+        args.sg_id,
+        20,
+    )
 
 
 def create_rhel_instance(args):
     """Coming soon - create the rhel instance"""
-    # pylint: disable=unused-argument
+    create_instance(
-    print("creating rhel instance - NYI")
+        args,
+        "rhel-test-instance",
+        args.rhel_flavor,
+        args.rhel_image,
+        args.sg_id,
+        120,
+    )
+
+
+def create_conversion_instance(args):
+    """Coming soon - create the rhel instance"""
+    create_instance(
+        args,
+        "rhel-conversion",
+        args.rhel_flavor,
+        args.rhel_image,
+        args.sg_id,
+        120,
+    )
+
+
+def create_keypair(args):
+    """Create a keypair to allow ssh later"""
+
+    key_exists = [
+        kp
+        for kp in json.loads(test_user_openstack_cmd("keypair list -f json ", args))
+        if kp["Name"] == "test_keypair"
+    ]
+
+    if key_exists:
+        args.keypair_name = key_exists[0]["Name"]
+    else:
+        if args.ssh:
+            fname = "temp_pub_key"
+            cmd = f"scp {args.ssh_key} {args.ssh}:{fname}"
+
+            execute_cmd(cmd)
+
+            args.keypair_name = json.loads(
+                test_user_openstack_cmd(
+                    f"keypair create -f json " f"--public-key {fname} test_keypair",
+                    args,
+                )
+            )[
+                "name"
+            ]  # a little inconsistent here...
+
+            _ = execute_ssh_cmd(f"rm {fname}", args)
+        else:
+            raise NotImplementedError("Only works over ssh")
+
+
+def create_router(args):
+    """Create a router"""
+    router_id = None
+
+    try:
+        router_id = [
+            router
+            for router in openstack_cmd("router list -f json", args, as_json=True)
+            if router["Name"] == "test-router"
+        ][0]["ID"]
+    except IndexError:
+        pass
+
+    if not router_id:
+        router_id = test_user_openstack_cmd(
+            "router create test-router -f json", args, as_json=True
+        )["id"]
+        print("router created")
+
+    router_info = openstack_cmd(f"router show -f json {router_id}", args, as_json=True)
+
+    if router_info["external_gateway_info"]:
+        print("router gateway already set")
+    else:
+        openstack_cmd(
+            f"router set {router_id} " f"--external-gateway {args.public_network_id}",
+            args,
+        )
+        print("router gateway added")
+
+    if [
+        interface
+        for interface in router_info["interfaces_info"]
+        if interface["subnet_id"] == args.private_subnet_id
+    ]:
+        print("router already connected to private subnet")
+    else:
+        cmd = f"router add subnet {router_id} {args.private_subnet_id}"
+        print(cmd)
+        openstack_cmd(cmd, args)
+        print("router subnet added")
+
+    args.router_id = router_id
+
+
+def create_security_group(args):
+    """Create a security group that allows ssh and icmp"""
+
+    sg_exists = [
+        sg
+        for sg in json.loads(
+            test_user_openstack_cmd("security group list -f json", args)
+        )
+        if sg["Name"] == "test-sg"
+    ]
+
+    if sg_exists:
+        args.sg_id = sg_exists[0]["ID"]
+    else:
+        sec_grp = json.loads(
+            test_user_openstack_cmd("security group create test-sg -f json", args)
+        )
+        args.sg_id = sec_grp["id"]
+
+        _ = test_user_openstack_cmd(
+            (
+                "security group rule create --dst-port 22 "
+                f"--protocol tcp {args.sg_id}"
+            ),
+            args,
+        )
+
+        _ = test_user_openstack_cmd(
+            ("security group rule create --protocol icmp " f"{args.sg_id}"),
+            args,
+        )
 
 
 def main():
@@ -202,6 +550,7 @@ def main():
 
     create_public_network(args)
     create_private_network(args)
+    create_router(args)
 
     create_cirros_flavor(args)
     create_rhel_flavor(args)
@@ -209,9 +558,14 @@ def main():
     create_cirros_image(args)
     create_rhel_image(args)
 
+    create_keypair(args)
+    create_security_group(args)
+
     create_cirros_instance(args)
     create_rhel_instance(args)
 
+    # create_os_migrate host
+
 
 if __name__ == "__main__":
     main()

src/tripleo_aio_helpers/os_migrate_teardown.py

@@ -3,22 +3,9 @@ Quick and dirty script to help setup project, flavors, networks, images
 """
 import argparse
 import json
+import sys
 
-# import os
+from .utils import openstack_cmd, test_user_openstack_cmd
-import subprocess
-
-
-def openstack_cmd(cmd, args):
-    """Utility function to run an openstack command agains the standalone cloud"""
-    cmd = "OS_CLOUD=standalone openstack " + cmd
-
-    if args.dry_run:
-        print("dry-run specified. Not executing cmd. cmd was:")
-        print(cmd)
-        return
-
-    result = subprocess.check_output(cmd, shell=True, universal_newlines=True)
-    return result
 
 
 def parse_args():
@@ -27,23 +14,24 @@ def parse_args():
     # home = os.environ.get('HOME')
 
     parser = argparse.ArgumentParser()
-    parser.add_argument("-d", "--project-domain", default="default")
     parser.add_argument("-n", "--project-name", default="test-project")
-    parser.add_argument("-D", "--project-description", default="Test project")
     parser.add_argument("-u", "--username", default="test-user")
     parser.add_argument("-p", "--password", default="secrete123")
-    parser.add_argument("-c", "--cloud", default="standalone")
+    parser.add_argument("-c", "--cloud", default="dst_admin")
-    parser.add_argument("-g", "--gateway", default="10.76.23.254")
+    parser.add_argument("--delete-all", action="store_true")
-    parser.add_argument(
+    parser.add_argument("--delete-images", action="store_true")
-        "-C", "--public-network-cider", default="10.76.23.0/24"
+    parser.add_argument("--delete-flavors", action="store_true")
-    )
+    parser.add_argument("--delete-private-network", action="store_true")
-    parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
+    parser.add_argument("--delete-public-network", action="store_true")
-    parser.add_argument("--publice-net-start", default="10.76.23.50")
+    parser.add_argument("--delete-instances", action="store_true")
-    parser.add_argument("--publice-net-end", default="10.76.23.52")
+    parser.add_argument("--delete-user", action="store_true")
-    parser.add_argument("--dns-server", default="10.76.23.245")
+    parser.add_argument("--delete-project", action="store_true")
 
-    # export OS_CLOUD=standalone
+    parser.add_argument("--dry-run", action="store_true")
-    # export STANDALONE_HOST=10.76.23.39
+    parser.add_argument(
+        "--ssh", help="Connection string to run commands on a remote host."
+    )
+    parser.add_argument("--debug", action="store_true")
 
     args = parser.parse_args()
 
@@ -63,13 +51,11 @@ def destroy_project(args):
         print(f"Project {args.project_name} exists. Will delete")
         args.project_id = project_exists[0]["ID"]
 
-        cmd = (
+        cmd = f"project delete {args.project_id}"
-            f"project delete -f json --domain {args.project_domain} "
+        _ = openstack_cmd(cmd, args)
-            f"{args.project_id}"
-        )
         print(f"Project {args.project_name} deleted")
     else:
-        print("Project {args.project_name} not found.")
+        print(f"Project {args.project_name} not found.")
 
 
 def destroy_user(args):
@@ -80,120 +66,184 @@ def destroy_user(args):
     ]
 
     if user_exists:
-        print(f"User {args.username} already exists. Skipping creation")
+        user_id = user_exists[0]["ID"]
-        args.user_id = user_exists[0]["ID"]
-        return
 
-    cmd = (
+        openstack_cmd(f"user delete {user_id}", args)
-        f"user create  -f json --project {args.project_id} "
+        print(f"User deleted - id: {user_id}")
-        f"--password {args.password} {args.username}"
+    else:
-    )
+        print(f"User {args.username} not found.")
-
-    args.user_id = json.loads(openstack_cmd(cmd, args))["id"]
-
-    print(f"User created - id: {args.user_id}")
-
-
-def assign_member_role(args):
-    """
-    Assign the member role to the user.
-
-    Note: it doesn't matter if the role is assigned multiple times, so not
-    bothering to check.
-    """
-
-    cmd = f"role add --user {args.username} --project {args.project_id} member"
-
-    result = openstack_cmd(cmd, args)
-
-    cmd = f"role assignment list --user {args.user_id} --role member -f json"
-    result = json.loads(openstack_cmd(cmd, args))
-
-    if result:
-        print("User has member role")
 
 
 def destroy_public_network(args):
-    """Coming soon - create the public network"""
+    """Delete the public network"""
-    # pylint: disable=unused-argument,unused-variable
+    print("deleting public network")
-    print("creating public network - NYI")
+
-    cmd = (
+    public_network_exists = [
-        "network create --external --provider-physical-network datacentre "
+        network
-        "--provider-network-type flat public"
+        for network in openstack_cmd("network list -f json", args, as_json=True)
+        if network["Name"] == "public"
+    ]
+
+    if public_network_exists:
+        routers = test_user_openstack_cmd("router list -f json", args, as_json=True)
+
+        for router in routers:
+            router_details = test_user_openstack_cmd(
+                f"router show -f json {router['ID']}", args, as_json=True
             )
-    cmd = (
+
-        f"subnet create public-net --subnet-range {args.publice_network_cidr} "
+            for interface in router_details["interfaces_info"]:
-        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+                test_user_openstack_cmd(
-        f"start={args.public_net_start},end={args.public_net_end} "
+                    f"router remove port {router['ID']} " f"{interface['port_id']}",
-        "--network public"
+                    args,
                 )
 
+            test_user_openstack_cmd(f"router delete {router['ID']}", args)
+
+        openstack_cmd("network delete public", args)
+
 
 def destroy_private_network(args):
-    """Coming soon - create the private network"""
+    """Delete the private network"""
-    # pylint: disable=unused-argument,unused-variable
+    private_network_exists = [
-    cmd = "openstack network create --internal private"
+        network
-    cmd = (
+        for network in test_user_openstack_cmd("network list -f json", args, as_json=True)
-        "openstack subnet create private-net "
+        if network["Name"] == "private"
-        f"--subnet-range {args.private_network_cidr} --network private"
+    ]
-    )
+
-    print("creating private network - NYI")
+    if private_network_exists:
+        print("deleting private network")
+        test_user_openstack_cmd(f"network delete private", args)
+
+
+def delete_flavor(args, name):
+    flavor_exists = [
+        flavor
+        for flavor in openstack_cmd("flavor list -f json --all", args, as_json=True)
+        if flavor["Name"] == name
+    ]
+    if flavor_exists:
+        flavor_id = flavor_exists[0]["ID"]
+        print(f"deleting {name} flavor")
+        openstack_cmd(f"flavor delete {flavor_id}", args)
+    else:
+        print(f"{name} flavour not found")
 
 
 def destroy_cirros_flavor(args):
-    """Coming soon - create the cirros flavor"""
+    """Delete the cirros flavor"""
-    # pylint: disable=unused-argument
+    delete_flavor(args, "cirros")
-    print("creating cirros flavor - NYI")
 
 
 def destroy_rhel_flavor(args):
-    """Coming soon - create the rhel flavor"""
+    """Delete the rhel flavor"""
-    # pylint: disable=unused-argument
+    delete_flavor(args, "rhel")
-    print("creating rhel flavor - NYI")
+
+
+def delete_image(args, name):
+    image_exists = [
+        image
+        for image in openstack_cmd("image list -f json", args, as_json=True)
+        if image["Name"] == name
+    ]
+
+    if image_exists:
+        print(f"deleting {name} image")
+        openstack_cmd(f"image delete {name}", args)
+    else:
+        print(f"{name} image not found")
 
 
 def destroy_cirros_image(args):
-    """Coming soon - create the cirros image"""
+    """Delete the cirros image"""
-    # pylint: disable=unused-argument
+    delete_image(args, "cirros_image")
-    print("creating cirros image - NYI")
 
 
 def destroy_rhel_image(args):
-    """Coming soon - create the rhel image"""
+    """Delete the rhel image"""
-    # pylint: disable=unused-argument
+    delete_image(args, "rhel_image")
-    print("creating rhel image - NYI")
 
 
 def destroy_cirros_instance(args):
-    """Coming soon - create the cirros instance"""
+    """Delete the cirros instance"""
-    # pylint: disable=unused-argument
+    try:
-    print("creating cirros instance - NYI")
+        cirros_instance = [
+            instance
+            for instance in test_user_openstack_cmd(
+                "server list -f json", args, as_json=True
+            )
+            if instance["Name"] == "cirros-test-instance"
+        ][0]["ID"]
+
+        test_user_openstack_cmd(f"server delete {cirros_instance}", args)
+    except IndexError:
+        print("No cirros instance found")
 
 
 def destroy_rhel_instance(args):
-    """Coming soon - create the rhel instance"""
+    """Delete the cirros instance"""
-    # pylint: disable=unused-argument
+    try:
-    print("creating rhel instance - NYI")
+        rhel_instance = [
+            instance
+            for instance in test_user_openstack_cmd(
+                "server list -f json", args, as_json=True
+            )
+            if instance["Name"] == "rhel-test-instance"
+        ][0]["ID"]
+
+        test_user_openstack_cmd(f"server delete {rhel_instance}", args)
+    except IndexError:
+        print("No RHEL instance found")
+
+
+def get_project_id(args):
+    """Get the id of the test project"""
+    result = openstack_cmd("project list -f json", args, as_json=True)
+
+    try:
+        args.project_id = [r for r in result if r["Name"] == args.project_name][0]["ID"]
+    except IndexError:
+        args.project_id = None
 
 
 def main():
     """main function"""
     args = parse_args()
 
-    # destroy_cirros_instance(args)
+    get_project_id(args)
-    # destroy_rhel_instance(args)
 
-    # destroy_cirros_image(args)
+    if args.delete_instances or args.delete_all:
-    # destroy_rhel_image(args)
+        if args.project_id:
+            destroy_cirros_instance(args)
+            destroy_rhel_instance(args)
+        else:
+            print("Project not found, no instances to delete")
 
-    # destroy_cirros_flavor(args)
+    if args.delete_images or args.delete_all:
-    # destroy_rhel_flavor(args)
+        destroy_cirros_image(args)
+        destroy_rhel_image(args)
 
-    # destroy_public_network(args)
+    if args.delete_flavors or args.delete_all:
-    # destroy_private_network(args)
+        destroy_cirros_flavor(args)
+        destroy_rhel_flavor(args)
 
+    if args.delete_public_network:
+        destroy_public_network(args)
+
+    if args.delete_private_network or args.delete_all:
+        if args.project_id:
+            destroy_private_network(args)
+        else:
+            print("Project not found, no private network to delete")
+
+    if args.delete_user or args.delete_all:
         destroy_user(args)
+
+    if args.delete_project or args.delete_all:
+        if args.project_id:
             destroy_project(args)
+        else:
+            print("Project not found. Can't delete.")
 
 
 if __name__ == "__main__":

src/tripleo_aio_helpers/os_migrate_validate_dst.py

@@ -0,0 +1,318 @@
+"""
+Quick and dirty script to validate that the necessary project, user,roles,
+flavors, networks, images have not been created
+"""
+import argparse
+import json
+import logging
+
+from rich import print
+
+from .utils import openstack_cmd
+
+
+def failed(msg, args):
+    args.failed = True
+    print(f"[red]{msg} :x:[/red]")
+
+
+def passed(msg):
+    print(f"[green]{msg} :white_heavy_check_mark:[/green]")
+
+
+def parse_args():
+    """Parse the command line arguments"""
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-n", "--project-name", default="test-project")
+    parser.add_argument("-u", "--username", default="test-user")
+    parser.add_argument("-c", "--cloud", default="dst_admin")
+    parser.add_argument("--private-network-name", default="private")
+    parser.add_argument("--cirros-flavor-name", default="cirros")
+    parser.add_argument("--rhel-flavor-name", default="rhel")
+    parser.add_argument("--cirros-image-name", default="cirros_image")
+    parser.add_argument("--rhel-image-name", default="rhel_image")
+    parser.add_argument("--cirros-instance-name", default="cirros-test-instance")
+    parser.add_argument("--rhel-instance-name", default="rhel-test-instance")
+    parser.add_argument("--router-name", default="test-router")
+    parser.add_argument("--group-name", default="os-migrate")
+    parser.add_argument("-l", "--log-level", default="WARN")
+
+    # parser.add_argument(
+    #    "--ssh", help="Connection string to run commands on a remote host."
+    # )
+
+    args = parser.parse_args()
+
+    return args
+
+
+def validate_project(args):
+    """Validate that the project exists"""
+    cmd = "project list -f json"
+    project_exists = [
+        x
+        for x in json.loads(openstack_cmd(cmd, args))
+        if x["Name"] == args.project_name
+    ]
+
+    if project_exists:
+        passed(f"project {args.project_name} exists with id {project_exists[0]['ID']}")
+    else:
+        failed("Project {args.project_name} not found.", args)
+
+
+def validate_user(args):
+    """Validate that the user exists"""
+    cmd = "user list -f json"
+    user_exists = [
+        x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
+    ]
+
+    if user_exists:
+        passed(f"user {args.username} exists with ID: {user_exists[0]['ID']}")
+    else:
+        failed("User not found", args)
+
+
+def validate_member_role(args):
+    """
+    Validate that the member role has been assigned to the user.
+    """
+    cmd = f"role assignment list --user {args.username} --names --project {args.project_name} --role member -f json"
+    logging.debug(cmd)
+
+    if len(openstack_cmd(cmd, args, as_json=True)) > 0:
+        passed(f"User has member role in {args.project_name}")
+    else:
+        failed(f"User does not have  member role in {args.project_name}")
+
+
+def validate_group(args):
+    check(
+        args,
+        "group list -f json",
+        "group",
+        f"group {args.group_name} exists",
+        "Name",
+        "os-migrate",
+        fail_if="not found"
+    )
+
+def validate_groups_has_member_role(args):
+    cmd = f"role assignment list --group {args.group_name} --names --project {args.project_name} --role member -f json"
+
+    if len(openstack_cmd(cmd, args, as_json=True)) > 0:
+        passed(f"Group {args.group_name} has member role in {args.project_name}")
+    else:
+        failed(f"Group {args.group_name} does not have  member role in {args.project_name}")
+
+
+def validate_admin_in_group(args):
+    cmd = f"group contains user {args.group_name} admin"
+    result = openstack_cmd(cmd, args, as_json=False)
+
+    if "admin in group" in result:
+        passed(result[:-1])
+    else:
+        failed(result[:-1], args)
+
+def validate_public_network(args):
+    """Coming soon - validate the public network"""
+    # pylint: disable=unused-argument,unused-variable
+
+    cmd = "network list -f json"
+
+    try:
+        network_exists = [
+            n for n in openstack_cmd(cmd, args, as_json=True) if n["Name"] == "public"
+        ][0]
+        passed(f"public network exists with id: {network_exists['ID']}")
+    except IndexError:
+        failed("public network not found", args)
+
+    cmd = f"subnet list -f json"
+    try:
+        subnet = [
+            s
+            for s in openstack_cmd(cmd, args, as_json=True)
+            if s["Name"] == "public-net"
+        ][0]
+        subnet_details = openstack_cmd(
+            f"subnet show -f json {subnet['ID']}", args, as_json=True
+        )
+        passed(
+            f"public subnet found allocation pool: {subnet_details['allocation_pools'][0]}"
+        )
+    except IndexError:
+        failed("public subnet not found", args)
+
+
+def validate_private_network(args):
+    """Coming soon - validate the private network"""
+    # pylint: disable=unused-argument,unused-variable
+    cmd = "network list -f json"
+    try:
+        network_exists = [
+            n
+            for n in openstack_cmd(cmd, args, as_json=True)
+            if n["Name"] == args.private_network_name
+        ][0]
+        failed(
+            f"private ({args.private_network_name}) network exists with id: {network_exists['ID']}",
+            args,
+        )
+    except IndexError:
+        passed("private network not found")
+
+
+def validate_cirros_flavor(args):
+    """Validate that the cirros flavor doesn't exist"""
+
+    cmd = "flavor list -f json --all"
+
+    try:
+        flavor = [
+            f
+            for f in openstack_cmd(cmd, args, as_json=True)
+            if f["Name"] == args.cirros_flavor_name
+        ][0]
+        failed(f"cirros flavor found (ID:{flavor['ID']})", args)
+    except IndexError:
+        passed("Cirros flavor not found.")
+
+
+def validate_rhel_flavor(args):
+    """Validate that the rhel flavor doesn't exist"""
+
+    cmd = "flavor list -f json --all"
+
+    try:
+        flavor = [
+            f
+            for f in openstack_cmd(cmd, args, as_json=True)
+            if f["Name"] == args.rhel_flavor_name
+        ][0]
+        failed(f"rhel flavor found (ID:{flavor['ID']})", args)
+    except IndexError:
+        passed("RHEL flavor not found.")
+
+
+def validate_cirros_image(args):
+    """Validate that the cirros image does not exist"""
+
+    cmd = "image list -f json --all"
+    images = openstack_cmd(cmd, args, as_json=True)
+
+    logging.debug(images)
+
+    try:
+        image = [
+            i
+            for i in images
+            if i["Name"] == args.cirros_image_name
+        ][0]
+        failed(f"Cirros image ({args.cirros_image_name}) found (ID:{image['ID']})", args)
+    except IndexError:
+        passed(f"Cirros image ({args.cirros_image_name}) not found.")
+
+
+def validate_rhel_image(args):
+    """Validate that the rhel image does not exist"""
+    # pylint: disable=unused-argument
+    cmd = "image list -f json --all"
+
+    try:
+        image = [
+            f
+            for f in openstack_cmd(cmd, args, as_json=True)
+            if f["Name"] == args.rhel_image_name
+        ][0]
+        failed(f"RHEL image ({args.rhel_image_name}) found (ID:{image['ID']})", args)
+    except IndexError:
+        passed(f"RHEL image ({args.rhel_image_name}) not found.")
+
+
+def validate_cirros_instance(args):
+    """Validate that the cirros instance does not exist"""
+    cmd = "server list -f json --all"
+
+    try:
+        instance = [
+            f
+            for f in openstack_cmd(cmd, args, as_json=True)
+            if f["Name"] == args.cirros_instance_name
+        ][0]
+        failed(f"Cirros instance found (ID:{instance['ID']})", args)
+    except IndexError:
+        passed("Cirros instance not found.")
+
+
+def validate_rhel_instance(args):
+    """Validate that the RHEL instance does not exist"""
+    cmd = "server list -f json --all"
+
+    try:
+        instance = [
+            f
+            for f in openstack_cmd(cmd, args, as_json=True)
+            if f["Name"] == args.rhel_instance_name
+        ][0]
+        failed(f"RHEL instance found (ID:{instance['ID']})", args)
+    except IndexError:
+        passed("RHEL instance not found.")
+
+
+def check(args, cmd, thing, msg, key, value, id_key="ID", fail_if="found"):
+    if fail_if not in ["found", "not found"]:
+        raise ValueError("fail_if must be eitther 'found' or 'not found'")
+
+    try:
+        found = [i for i in openstack_cmd(cmd, args, as_json=True) if i[key] == value][0]
+        if fail_if == "found":
+            failed(msg.format(key=id_key, value=found[id_key]), args)
+        else:
+            passed(f"{thing} not found")
+    except IndexError:
+        if fail_if == "found":
+            passed(f"{thing} not found")
+        else:
+            failed(msg.format(key=id_key, value=found[id_key]), args)
+
+
+def validate_router(args):
+    check(
+        args,
+        f"router list --project {args.project_name} -f json",
+        "router",
+        "router found ",
+        "Name",
+        args.router_name,
+    )
+
+
+def main():
+    """main function"""
+    args = parse_args()
+
+    logging.basicConfig(level=args.log_level)
+
+    validate_user(args)
+    validate_project(args)
+    validate_member_role(args)
+    validate_group(args)
+    validate_groups_has_member_role(args)
+    validate_admin_in_group(args)
+    validate_public_network(args)
+    validate_private_network(args)
+    validate_cirros_flavor(args)
+    validate_rhel_flavor(args)
+    validate_cirros_image(args)
+    validate_rhel_image(args)
+    validate_cirros_instance(args)
+    validate_rhel_instance(args)
+    validate_router(args)
+
+
+if __name__ == "__main__":
+    main()

src/tripleo_aio_helpers/os_migrate_validate_src.py

@@ -0,0 +1,169 @@
+"""
+Quick and dirty script to validate that the necessary project, user,roles,
+flavors, networks, images have been created
+"""
+import argparse
+import json
+
+from .utils import openstack_cmd, passed, failed
+
+
+def parse_args():
+    """Parse the command line arguments"""
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-n", "--project-name", default="test-project")
+    parser.add_argument("-u", "--username", default="test-user")
+    parser.add_argument("-c", "--cloud", default="src_admin")
+    parser.add_argument("--private-network-name", default="private")
+    parser.add_argument("--cirros-flavor-name", default="cirros")
+    parser.add_argument("--rhel-flavor-name", default="rhel")
+
+    args = parser.parse_args()
+
+    return args
+
+
+def validate_project(args):
+    """Validate that the project exists"""
+    cmd = "project list -f json"
+    project_exists = [
+        x
+        for x in json.loads(openstack_cmd(cmd, args))
+        if x["Name"] == args.project_name
+    ]
+
+    if project_exists:
+        passed(project_exists)
+    else:
+        failed("Project {args.project_name} not found.", args)
+
+
+def validate_user(args):
+    """Validate that the user exists"""
+    cmd = "user list -f json"
+    user_exists = [
+        x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
+    ]
+
+    if user_exists:
+        passed(user_exists)
+    else:
+        failed("User not found", args)
+
+
+def validate_group_exists(args):
+    failed("NYI", args)
+
+
+def validate_admin_in_group(args):
+    failed("NYI", args)
+
+
+def validate_group_has_member_role(args):
+    failed("NYI", args)
+
+
+def validate_member_role(args):
+    """
+    Validate that the member role has been assigned to the user.
+    """
+    print(args)
+    # cmd = (
+    #       f"role add --user {args.username} "
+    #       f"--project {args.project_id} member"
+    #       )
+
+    # result = openstack_cmd(cmd, args)
+
+    # cmd = f"role assignment list --user {args.user_id} --role member -f json"
+    # result = json.loads(openstack_cmd(cmd, args))
+
+    # if result:
+    #     print("User has member role")
+
+
+def validate_public_network(args):
+    """Coming soon - validate the public network"""
+    # pylint: disable=unused-argument,unused-variable
+    failed("Validate public network - NYI", args)
+    cmd = (
+        "network create --external --provider-physical-network datacentre "
+        "--provider-network-type flat public"
+    )
+    cmd = (
+        f"subnet create public-net --subnet-range {args.publice_network_cidr} "
+        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+        f"start={args.public_net_start},end={args.public_net_end} "
+        "--network public"
+    )
+    print(cmd)
+
+
+def validate_private_network(args):
+    """Coming soon - validate the private network"""
+    # pylint: disable=unused-argument,unused-variable
+    _ = "openstack network create --internal private"
+    _ = (
+        "openstack subnet create private-net "
+        f"--subnet-range {args.private_network_cidr} --network private"
+    )
+    failed("validate private network - NYI", args)
+
+
+def validate_cirros_flavor(args):
+    """Coming soon - create the cirros flavor"""
+    # pylint: disable=unused-argument
+    failed("validate cirros flavor - NYI", args)
+
+
+def validate_rhel_flavor(args):
+    """Coming soon - validate the rhel flavor"""
+    # pylint: disable=unused-argument
+    failed("validate rhel flavor - NYI", args)
+
+
+def validate_cirros_image(args):
+    """Coming soon - validate the cirros image"""
+    # pylint: disable=unused-argument
+    failed("validate cirros image - NYI", args)
+
+
+def validate_rhel_image(args):
+    """Coming soon - validate the rhel image"""
+    # pylint: disable=unused-argument
+    failed("validate rhel image - NYI", args)
+
+
+def validate_cirros_instance(args):
+    """Coming soon - validate the cirros instance"""
+    # pylint: disable=unused-argument
+    failed("validate cirros instance - NYI", args)
+
+
+def validate_rhel_instance(args):
+    """Coming soon - create the rhel instance"""
+    # pylint: disable=unused-argument
+    failed("validate rhel instance - NYI", args)
+
+
+def main():
+    """main function"""
+    args = parse_args()
+
+    validate_user(args)
+    validate_group_exists(args)
+    validate_admin_in_group(args)
+    validate_group_has_member_role(args)
+    validate_public_network(args)
+    validate_private_network(args)
+    validate_cirros_flavor(args)
+    validate_rhel_flavor(args)
+    validate_cirros_image(args)
+    validate_rhel_image(args)
+    validate_cirros_instance(args)
+    validate_rhel_instance(args)
+
+
+if __name__ == "__main__":
+    main()

src/tripleo_aio_helpers/prepare_deployment.py

@@ -4,6 +4,7 @@ Quick and dirty script to help setup config filers for a RHOSP AIO install
 import argparse
 import os
 import subprocess
+
 import yaml
 
 
@@ -18,9 +19,7 @@ def parse_args():
     parser.add_argument("-a", "--address", required=True)
     parser.add_argument("-i", "--interface", required=True)
     parser.add_argument("-m", "--netmask", default=24)
-    parser.add_argument(
+    parser.add_argument("-d", "--dns", nargs="+", action="append", required=True)
-        "-d", "--dns", nargs="+", action="append", required=True
-    )
     parser.add_argument("-D", "--domain", default="aio")
     parser.add_argument("--using-multiple-nics", action="store_true")
     parser.add_argument("-U", "--deployment-user")
@@ -37,6 +36,11 @@ def parse_args():
     parser.add_argument(
         "--deploy-script-out", default=f"{home}/deploy.sh.gen", dest="deploy"
     )
+    parser.add_argument(
+        "--cinder-pool-size",
+        default="500280",
+        help="Size of the loopback device allocated to cinder. " "Defaults to 500280",
+    )
 
     args = parser.parse_args()
 
@@ -69,6 +73,7 @@ def get_standalone_parameters(args):
             "StandaloneEnableRoutedNetworks": False,
             "StandaloneHomeDir": args.deployment_dir,
             "StandaloneLocalMtu": 1500,
+            "CinderLVMLoopDeviceSize": args.cinder_pool_size,
         }
     }
 
@@ -128,22 +133,17 @@ def set_hostname(args):
     print("set_hostname is not yet implemented")
 
 
-
 def main():
     """main function"""
     args = parse_args()
 
     containers_yaml = build_containers_yaml(args)
-    with open(
+    with open(args.containers_yaml_out, "w", encoding="utf-8") as containers_out:
-        args.containers_yaml_out, "w", encoding="utf-8"
-    ) as containers_out:
         containers_out.write(yaml.dump(containers_yaml))
     print(f"containers yaml written to {args.containers_yaml_out}")
 
     standalone_parameters = get_standalone_parameters(args)
-    with open(
+    with open(args.standalone_yaml_out, "w", encoding="utf-8") as parameters_out:
-        args.standalone_yaml_out, "w", encoding="utf-8"
-    ) as parameters_out:
         parameters_out.write(yaml.dump(standalone_parameters))
     print(f"standalone parameters yaml written to {args.standalone_yaml_out}")
 
@@ -151,7 +151,10 @@ def main():
         deploy.write(deploy_sh(args))
     print(f"deploy script written to {args.deploy}")
 
-    print("If you are running on a cloud image remenber to disable cloud-init before running the deploy.")
+    print(
+        "If you are running on a cloud image remenber to disable cloud-init "
+        "before running the deploy."
+    )
     print("sudo systemctl stop cloud-init")
     print("sudo systemctl disable cloud-init")
     print("Make sure you have specifed the correct interface to use!")

src/tripleo_aio_helpers/utils.py

@@ -4,6 +4,8 @@ import os
 import subprocess
 import sys
 
+from rich import print
+
 
 def get_from_env(name, envvar, required=True):
     """Get the value for a parameter from an environment variable"""
@@ -20,15 +22,17 @@ def get_from_env(name, envvar, required=True):
 
 
 def openstack_cmd(cmd, args, as_json=False):
-    """Utility function to run an openstack command agains the standalone cloud"""
+    """
+    Utility function to run an openstack command agains the standalone cloud
+    """
 
     cmd = cmd.replace("\n", " ")
-    cmd = "OS_CLOUD=standalone openstack " + cmd
+    cmd = f"OS_CLOUD={args.cloud} openstack " + cmd
 
-    if args.ssh:
+    if "ssh" in args and args.ssh:
         cmd = f'ssh {args.ssh} "{cmd}"'
 
-    if args.dry_run:
+    if "dry_run" in args and args.dry_run:
         print("dry-run specified. Not executing cmd. cmd was:")
         print(cmd)
         return None
@@ -60,3 +64,12 @@ def test_user_openstack_cmd(cmd, args, as_json=False):
         print(cmd)
 
     return openstack_cmd(cmd, args, as_json=as_json)
+
+
+def failed(msg, args):
+    args.failed = True
+    print(f"[red]{msg} :x:[/red]")
+
+
+def passed(msg):
+    print(f"[green]{msg} :white_heavy_check_mark:[/green]")

src/virt-install/user-data.tpl

@@ -25,4 +25,3 @@ runcmd:
   - "echo '***                                                                     ***'"
   - "echo '***************************************************************************'"
   - "reboot"
-