✨ add os_migrate_validate.py

README.md

@@ -1,6 +1,4 @@
-A quick and dirty script to help set up a RHOSP AIO
+A set of quick and dirty scripts to help set up a RHOSP AIO
-
-This has grown a bit. At some point I will make this a proper readme, but for now a few notes:
 
 The goal is to build a VM (although you could use baremetal) and deploy a RHOSP AIO to it. Then to add a project, flavors, images, networks to support testing os-migrate.
 
@@ -8,20 +6,38 @@ After that create a VM to migrate, a conversion host to do the migration.
 
 This may well be useful for other things, but this is the current aim.
 
+There are/will be example scripts in the examples directory
+
+Everything is a bit of a mess but here's a brief overview:
+
+## create_aio_vm.py
+
+Uses virt-install to build a VM to deploy a RHOSP 16 AIO on.
+
 I am primarily targetting RHEL8.4 and RHOSP16.2. As a result I'm tied to virt-install v3.2.0 which makes using cloud-init for configuring network on the AIO VM a little annoying. Later versions allow you to pass in a network config, but 3.2.0 means I have to use nmcli and bootcmd to get the network configure.
 
 There is a vestigial network-config.tpl in the virt-install directory but it is not actually used yet.
 
-EVerything is a bit of a mess but here's a brief overview:
+## os_migrate_setup.py 
 
-main.py - original script. Generates the yaml files and a dploy script to deploy a RHOSP AIO. Assumes a VM to run on. Don't run this on your laptop. You'll be sorry if you do.
+build the OSP infrastructure to allow testing os-migrate.
 
-os_migrate_setup.py - build the OSP infrastructure to allow testing os-migrate.
+## os_migrate_teardown.py 
 
-os_migrate_teardown.py - remove that infrastructure (to allow for refinement/iteration)
+remove that infrastructure (to allow for refinement/iteration)
 
 virt-install/create_aio_vm.py - use virt-install to create and configure a VM to run the other scripts on.
 
+## prepare-deployment
+
+Original script. Generates the yaml files and a dploy script to deploy a RHOSP AIO. Assumes a VM to run on. Don't run this on your laptop. You'll be sorry if you do.
+
+## os_migrate_validate.py 
+
+Check that the necessary infrastructure has been deployed.
+
+# Remember to mask and disable cloud-init if installing on a VM 
+
 ```
 usage: main.py [-h] -u USERNAME -p PASSWORD -a ADDRESS [-i INTERFACE]
                [-m NETMASK] -d DNS [DNS ...] [-D DOMAIN]

TODO.md

@@ -0,0 +1,21 @@
+# Tests
+
+There aren't any which. At least basic unit tests would be good.
+
+# Documentation
+
+There's a README, but it should be expanded.
+
+# Error handling.
+
+There's very litle and this is a complex process involving talking to other systems over the net.
+
+# Convert to Ansible
+
+Maybe this should all have been an ansible role.  Maybe that will happen yet.
+
+# Ideas for specific scripts:
+
+## create_aio_vm
+
+Quite a few compromises to do with the versions of virt_install / libvirt available to me.

docs/create_aio_vm.md

@@ -0,0 +1,5 @@
+# Documentation for create_aio_vm
+
+## Networking
+
+Networking has been a real pain. The RHOSP AIO deployment seems to force the vr-ctlplane interface to use dhcp. Originally I was trying to use static IPs configured with cloud-init (using bootcmd because of the version) but the deploy would die. To fix this I now speciiy MACs for the interfaces and then use an external DHCP server. Maybe this is configurable in the RHOSP standalone deploy. If so it would be better to use that. It would also be nice to do some feature switches so that if the version of virt-install allows I could use network-config instead of bootcmd

docs/notes.md

@@ -0,0 +1,84 @@
+# Installing os-migrate
+
+I now have two RHOSP-16.2 AIOs running on my homelab so it's time to install os-migrate.
+
+The intallation instructions are at: https://os-migrate.github.io/os-migrate/user/install-from-galaxy.html
+
+I setup a new Fedora 38 VM to run os-migrate from:
+
+```bash
+virt-install -n os-migrate \
+  --osinfo fedora37 \
+  --memory 2048 \
+  --vcpus 2 \
+  --import \
+  --disk /data/os-migrate.qcow2 \
+  --graphics vnc \
+  --cloud-init disable=on,clouduser-ssh-key=ncox.pub,root-password-file=rpw  \
+  --network bridge=br0
+  ```
+
+  This VM is running at 172.23.0.53, on test-04(173.23.0.25)
+
+  RHEL9 doesn't know about fedora38
+
+  Lots of stuffing around to get the bridge working
+
+  ## Install and run os-miograte
+
+`dnf install ansible `
+
+`vi os-migrate-vars.yml`
+
+Largely copied from the ~/.config/openstack/clouds.yaml files
+
+
+Hints from os-migrate doco:
+
+
+```bash
+export OSM_DIR=/root/.ansible/collections/ansible_collections/os_migrate/os_migrate
+export OSM_CMD="ansible-playbook -v -i $OSM_DIR/localhost_inventory.yml -e @os-migrate-vars.yml"
+```
+
+Despite what the os-migrate docs say, do not install python3-openstacksdk because on f38 the version is too recent.
+
+```bash
+dnf install iputils python3-openstackclient
+```
+
+Let's try networks:
+
+```$OSM_CMD $OSM_DIR/playbooks/export_networks.yml```
+
+Hmm, problems with versions.
+
+Going to need devel tools for openstacksdk
+
+```
+dnf install python3-devel
+dnf group install "C Development Tools and Libraries"
+```
+
+We're going to need ansible-core in the venv or it will pick up system versions of things.
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install --upgrade pip
+python3 -m pip install --upgrade 'openstacksdk>=0.36,<0.99'
+python3 -m pip install --upgrade 'ansible-core'
+```
+
+
+Try again:
+
+```
+OSM_CMD $OSM_DIR/playbooks/export_networks.yml
+```
+
+```bash
+   60  less os-migrate-data-test-1/clouds.yaml 
+   61  less os-migrate-data-test-1/networks.yml 
+   67  $OSM_CMD $OSM_DIR/playbooks/export_subnets.yml 
+   68  less os-migrate-data-test-1/subnets.yml 
+```

examples/create_aio.bash

@@ -0,0 +1,21 @@
+AIO_NAME=test5
+
+
+create_aio_vm \
+  --password secrete123 \
+  --public-key ~/.ssh/id_rsa.pub \
+  --output-image /data/${AIO_NAME}.qcow2 \
+  --input-image ~/rhel-guest-image-8.4-1269.x86_64.qcow2  \
+  --image-size "800G" \
+  --os-variant rhel8.4 \
+  --name ${AIO_NAME} \
+  --local-hostname ${AIO_NAME} \
+  --instance-id ${AIO_NAME} \
+  --gateway 172.23.0.1 \
+  --dns 172.23.0.14 \
+  --mac-1 52:54:00:a5:48:03 \
+  --mac-2 52:54:00:a5:48:04 \
+  --search-domain evatt.ingenious.com.au \
+  --rhn-user $AIO_RHN_USER \
+  --rhn-password $AIO_RHN_PASSWORD
+  

examples/install_remote_venv.bash

@@ -0,0 +1,19 @@
+REMOTE=172.23.0.34
+AIOH_ENV="/home/stack/tripleo-aio-helpers"
+AIO_SRC=$HOME/Projects
+VENV=$AIOH_ENV/venv
+rsync -avz --exclude venv $AIO_SRC/tripleo-aio-helpers stack@$REMOTE:
+ssh stack@$REMOTE  "sudo dnf install -y python3-tripleoclient"
+ssh stack@$REMOTE "cd $AIOH_ENV && python3 -m venv venv"
+ssh stack@$REMOTE "$VENV/bin/pip install --upgrade pip"
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && pip install -r requirements.txt"
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && pip install --editable ."
+ssh stack@$REMOTE "cd $AIOH_ENV && source venv/bin/activate && prepare_deployment -u $AIO_RHN_USER --password $AIO_RHN_PASSWORD --address 172.23.0.34 --interface eth1 --dns 172.23.0.14 --gateway 172.23.0.1"
+ssh stack@$REMOTE "sudo systemctl stop cloud-init"
+ssh stack@$REMOTE "sudo systemctl disable cloud-init"
+# ssh stack@$REMOTE "cp containers-prepare-parameters.yaml.gen containers-prepare-parameters.yaml"
+# ssh stack@$REMOTE "cp standalone_parameters.yaml.gen standalone_parameters.yaml"
+# ssh stack@$REMOTE "cp deploy.sh.gen deploy.sh"
+ssh stack@$REMOTE "sudo dnf install -y tmux"
+ssh stack@$REMOTE "echo \"tmux new -d \; setw remain-on-exit on \; respawnw -k bash deploy.sh\""
+# ssh stack@$REMOTE "OS_CLOUD=standalone openstack endpoint list"

examples/prepare_deployment.bash

@@ -0,0 +1,7 @@
+prepare_deployment \
+    -u $AIO_RHN_USER \
+    --password $AIO_RHN_PASSWORD \
+    --address 172.23.0.35/24 \
+    --interface eth1 \
+    --dns 172.23.0.14 \
+    --gateway 172.23.0.1

examples/setup.bash

@@ -0,0 +1,16 @@
+FIP_START=117
+FIP_END=129
+AIO_HOST=test-5
+
+echo "Setting up $AIO_HOST"
+os_migrate_setup \
+  --gateway 172.23.0.1 \
+  --public-network-cidr=172.23.0.1/24 \
+  --public-net-start 172.23.0.$FIP_START \
+  --public-net-end 172.23.0.$FIP_END \
+  --dns-server 172.23.0.14 \
+  --ssh stack@$AIO_HOST \
+  --cirros-url https://isos.evatt.ingenious.com.au/cirros-0.6.2-x86_64-disk.img \
+  --rhel-url https://isos.evatt.ingenious.com.au/rhel-guest-image-8.4-1269.x86_64.qcow2 \
+  --ssh-key ~/.ssh/id_rsa.pub \
+  --debug

examples/teardown.bash

@@ -0,0 +1,11 @@
+AIO_HOST=test-5
+
+echo "Tearing down $AIO_HOST"
+
+os_migrate_teardown \
+    --project-name test-project \
+    --username test-user \
+    --cloud standalone \
+    --ssh stack@$AIO_HOST \
+    --delete-instances \
+    --delete-networks

src/tripleo_aio_helpers/os_migrate_setup.py

@@ -3,8 +3,22 @@ Quick and dirty script to help setup project, flavors, networks, images
 """
 import argparse
 import json
+import subprocess
+import tempfile
+
+from .utils import get_from_env, openstack_cmd, test_user_openstack_cmd
+
+
+def execute_cmd(cmd):
+    """Execute a command"""
+    return subprocess.check_output(cmd, shell=True, universal_newlines=True)
+
+
+def execute_ssh_cmd(cmd, args):
+    """Execute a command on a remote host using ssh"""
+    cmd = f'ssh {args.ssh} "{cmd}"'
+    return subprocess.check_output(cmd, shell=True, universal_newlines=True)
 
-from .utils import get_from_env,openstack_cmd
 
 def parse_args():
     """Parse the command line arguments"""
@@ -20,12 +34,28 @@ def parse_args():
     parser.add_argument("-c", "--cloud", default="standalone")
     parser.add_argument("-g", "--gateway")
     parser.add_argument("-C", "--public-network-cidr")
+    parser.add_argument(
+        "--ssh-key", help="File containing a public key to inject into the instances"
+    )
     parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
     parser.add_argument("--public-net-start")
     parser.add_argument("--public-net-end")
     parser.add_argument("--dns-server")
     parser.add_argument("--dry-run", action="store_true")
-    parser.add_argument("--ssh", help="Connection string to run commands on a remote host.")
+    parser.add_argument(
+        "--ssh", help="Connection string to run commands on a remote host."
+    )
+    parser.add_argument(
+        "--cirros-url",
+        help="a URL that a cirros image can be downloaded from. Required. "
+        "Can be set in AIO_CIRROS_URL",
+    )
+    parser.add_argument(
+        "--rhel-url",
+        help="a URL that a RHEL image can be downloaded from. Required. Can "
+        "be set in AIO_RHEL_URL",
+    )
+    parser.add_argument("--debug", action="store_true")
 
     # export OS_CLOUD=standalone
     # export STANDALONE_HOST=10.76.23.39
@@ -36,10 +66,14 @@ def parse_args():
         args.gateway = get_from_env("--gateway", "AIO_GATEWAY")
 
     if not args.public_network_cidr:
-        args.public_network_cidr = get_from_env("--public-network-cidr", "AIO_PUBLIC_CIDR")
+        args.public_network_cidr = get_from_env(
+            "--public-network-cidr", "AIO_PUBLIC_CIDR"
+        )
 
     if not args.public_net_start:
-        args.public_net_start = get_from_env("--public-net-start", "AIO_PUBLIC_NET_START")
+        args.public_net_start = get_from_env(
+            "--public-net-start", "AIO_PUBLIC_NET_START"
+        )
 
     if not args.public_net_end:
         args.public_net_end = get_from_env("--public-net-end", "AIO_PUBLIC_NET_END")
@@ -47,15 +81,24 @@ def parse_args():
     if not args.dns_server:
         args.dns_server = get_from_env("--dns-server", "AIO_DNS_SERVER")
 
+    if not args.cirros_url:
+        args.dns_server = get_from_env("--dns-server", "AIO_CIRROS_URL")
+
+    if not args.rhel_url:
+        args.dns_server = get_from_env("--dns-server", "AIO_RHEL_URL")
+
     if not args.ssh:
         args.ssh = get_from_env("--ssh", "AIO_SSH", required=False)
 
+    if not args.ssh_key:
+        args.ssh_key = get_from_env("--ssh-key", "AIO_SSH_KEY")
+
     return args
 
+
 def create_project(args):
     """Create the project if it doesn't already exist"""
 
-    
     cmd = "project list -f json"
     project_exists = [
         x
@@ -129,66 +172,362 @@ def assign_member_role(args):
 
 
 def create_public_network(args):
-    """Coming soon - create the public network"""
+    """Create the public network"""
-    # pylint: disable=unused-argument,unused-variable
+
-    print("creating public network - NYI")
+    network_exists = json.loads(
+        openstack_cmd("network list -f json --name public", args)
+    )
+
+    if network_exists:
+        print("Public network already exists - skipping")
+        args.public_network_id = network_exists[0]["ID"]
+    else:
         cmd = (
-        "network create --external --provider-physical-network datacentre "
+            "network create -f json  --external --provider-physical-network datacentre "
             "--provider-network-type flat public"
         )
-    cmd = (
+        try:
-        f"subnet create public-net --subnet-range {args.public_network_cidr} "
+            args.public_network_id = json.loads(openstack_cmd(cmd, args))["id"]
-        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+        except json.decoder.JSONDecodeError:
-        f"start={args.public_net_start},end={args.public_net_end} "
+            print(cmd)
-        "--network public"
+            raise
+        print("Public network created.")
+
+    subnet_exists = json.loads(
+        openstack_cmd("subnet list -f json --name public-net", args)
     )
+    if subnet_exists:
+        print("Public subnet exists - skipping")
+    else:
+        cmd = (
+            "subnet create public-net -f json "
+            f"--subnet-range {args.public_network_cidr} "
+            f"--gateway {args.gateway} "
+            f"--allocation-pool start={args.public_net_start},end={args.public_net_end} "
+            "--network public "
+            f"--host-route destination=0.0.0.0/0,gateway={args.gateway} "
+            f"--dns-nameserver {args.dns_server}"
+        )
+        args.public_subnet_id = json.loads(openstack_cmd(cmd, args))
+        print("Public subnet created.")
 
 
 def create_private_network(args):
-    """Coming soon - create the private network"""
+    """Create the private network and subnet"""
-    # pylint: disable=unused-argument,unused-variable
+
-    cmd = "openstack network create --internal private"
+    network_exists = json.loads(
-    cmd = (
+        openstack_cmd(
-        "openstack subnet create private-net "
+            f"network list -f json --project {args.project_id} --name private", args
-        f"--subnet-range {args.private_network_cidr} --network private"
         )
-    print("creating private network - NYI")
+    )
+
+    if network_exists:
+        print("Private network already exists - skipping")
+        args.private_network_id = network_exists[0]["ID"]
+    else:
+        cmd = f"network create -f json --internal private --project {args.project_id}"
+        args.private_network_id = json.loads(openstack_cmd(cmd, args))["id"]
+        print("Private network created.")
+
+    subnet_exists = json.loads(
+        openstack_cmd(
+            f"subnet list -f json --project {args.project_id} --name private-net", args
+        )
+    )
+    if subnet_exists:
+        print("Private subnet exists - skipping")
+        args.private_subnet_id = subnet_exists[0]["ID"]
+    else:
+        cmd = (
+            f"subnet create private-net -f json --project {args.project_id} "
+            f"--subnet-range {args.private_network_cidr} --network {args.private_network_id}"
+        )
+        args.private_subnet_id = json.loads(openstack_cmd(cmd, args))
+        print("Private subnet created.")
+
+
+def create_flavor(args, flavor_name, memory, disk, cpus):
+    """Create a flavor - DRY"""
+    cmd = "flavor list -f json --all"
+    # Note we are going to assume that there is only one cirros flavor. This
+    # will be more of a problem come deletion time.
+    result = json.loads(openstack_cmd(cmd, args))
+    flavor_exists = [x for x in result if x["Name"] == flavor_name]
+    if flavor_exists:
+        print(f"{flavor_name} flavor already exists. Skipping creation")
+        args.__dict__[flavor_name + "_flavor"] = flavor_exists[0]["ID"]
+        return
+
+    print("creating cirros flavor")
+    # Note can't add a description in RHOSP16, but perhaps should add for 17
+    cmd = f"""
+        flavor create -f json 
+            --ram {memory}
+            --disk {disk}
+            --vcpus {cpus} 
+            --private 
+            --project {args.project_id} 
+            {flavor_name}
+        """.replace(
+        "\n", " "
+    )
+
+    result = openstack_cmd(cmd, args, as_json=True)
+    args.__dict__[flavor_name] = result["id"]
+    print(result)
 
 
 def create_cirros_flavor(args):
-    """Coming soon - create the cirros flavor"""
+    """create the cirros flavor"""
-    # pylint: disable=unused-argument
+    create_flavor(args, "cirros", 256, 20, 1)
-    print("creating cirros flavor - NYI")
 
 
 def create_rhel_flavor(args):
-    """Coming soon - create the rhel flavor"""
+    """create the rhel flavor"""
-    # pylint: disable=unused-argument
+    create_flavor(args, "rhel", 1536, 120, 2)
-    print("creating rhel flavor - NYI")
+
+
+def create_image(image_name, image_url, disk_size, memory, args):
+    """Create an image - DRY"""
+
+    cmd = "image list -f json"
+    image_exists = [
+        image
+        for image in json.loads(openstack_cmd(cmd, args))
+        if image["Name"] == image_name
+    ]
+
+    if image_exists:
+        print(f"{image_name} already exists - not creating.")
+        args.__dict__[image_name] = image_exists[0]["ID"]
+        return
+
+    with tempfile.TemporaryDirectory() as tmp_dir:
+
+        # download image to tmpdir
+        fname = f"{tmp_dir}/{image_name}.img"
+        cmd = f"wget -O {fname} {image_url}"
+        _ = execute_cmd(cmd)
+
+        if args.ssh:
+            # Copy img to remote host
+            cmd = f"scp {fname} {args.ssh}:{image_name}.img"
+            execute_cmd(cmd)
+
+        # create image - note this will only work on a remote ssh host at the moment
+        cmd = f"""
+            image create -f json
+                --container-format bare
+                --disk-format qcow2
+                --min-disk {disk_size} --min-ram {memory}
+                --file {image_name}.img
+                --private
+                --project {args.project_id}
+                {image_name}
+            """
+
+        result = openstack_cmd(cmd, args, as_json=True)
+        args.__dict__[image_name] = result["id"]
+
+        if args.ssh:
+            # Delete image from remote host
+            cmd = f"rm {image_name}.img"
+            execute_ssh_cmd(cmd, args)
 
 
 def create_cirros_image(args):
-    """Coming soon - create the cirros image"""
+    """create the cirros image"""
-    # pylint: disable=unused-argument
+
-    print("creating cirros image - NYI")
+    create_image("cirros_image", args.cirros_url, 20, 256, args)
 
 
 def create_rhel_image(args):
-    """Coming soon - create the rhel image"""
+    """create the rhel image"""
-    # pylint: disable=unused-argument
+    create_image("rhel_image", args.rhel_url, 120, 1536, args)
-    print("creating rhel image - NYI")
+
+
+def create_instance(args, name, flavor, image, security_group, boot_size):
+    """Create an instance"""
+    # pylint:disable=too-many-arguments
+    instance_exists = [
+        instance
+        for instance in test_user_openstack_cmd(
+            "server list -f json", args, as_json=True
+        )
+        if instance["Name"] == name
+    ]
+
+    if instance_exists:
+        print(f"{name} instance exists - skipping")
+    else:
+        cmd = (
+            f"server create --flavor {flavor} "
+            f"--image {image} "
+            f"--key-name test_keypair "
+            f"--security-group {security_group} "
+            "--network private "
+            f"--boot-from-volume {boot_size} "
+            "-f json "
+            f"{name}"
+        )
+
+        server = test_user_openstack_cmd(cmd, args, as_json=True)
+
+        print(f"{name} instance created")
+        # assign floating IP
+        fip = test_user_openstack_cmd(
+            "floating ip create -f json public", args, as_json=True
+        )
+        _ = test_user_openstack_cmd(
+            f"server add floating ip {server['id']} {fip['floating_ip_address']}", args
+        )
 
 
 def create_cirros_instance(args):
-    """Coming soon - create the cirros instance"""
+    """Create the cirros instance"""
-    # pylint: disable=unused-argument
+
-    print("creating cirros instance - NYI")
+    create_instance(
+        args,
+        "cirros-test-instance",
+        args.cirros_flavor,
+        args.cirros_image,
+        args.sg_id,
+        20,
+    )
 
 
 def create_rhel_instance(args):
     """Coming soon - create the rhel instance"""
-    # pylint: disable=unused-argument
+    create_instance(
-    print("creating rhel instance - NYI")
+        args,
+        "rhel-test-instance",
+        args.rhel_flavor,
+        args.rhel_image,
+        args.sg_id,
+        120
+    )
+
+
+def create_conversion_instance(args):
+    """Coming soon - create the rhel instance"""
+    create_instance(
+        args,
+        "rhel-conversion",
+        args.rhel_flavor,
+        args.rhel_image,
+        args.sg_id,
+        120
+    )
+
+
+def create_keypair(args):
+    """Create a keypair to allow ssh later"""
+
+    key_exists = [
+        kp
+        for kp in json.loads(test_user_openstack_cmd("keypair list -f json ", args))
+        if kp["Name"] == "test_keypair"
+    ]
+
+    if key_exists:
+        args.keypair_name = key_exists[0]["Name"]
+    else:
+        if args.ssh:
+            fname = "temp_pub_key"
+            cmd = f"scp {args.ssh_key} {args.ssh}:{fname}"
+
+            execute_cmd(cmd)
+
+            args.keypair_name = json.loads(
+                test_user_openstack_cmd(
+                    f"keypair create -f json --public-key {fname} test_keypair", args
+                )
+            )[
+                "name"
+            ]  # a little inconsistent here...
+
+            _ = execute_ssh_cmd(f"rm {fname}", args)
+        else:
+            raise NotImplementedError("Only works over ssh")
+
+
+def create_router(args):
+    """Create a router"""
+    router_id = None
+
+    try:
+        router_id = [
+            router
+            for router in openstack_cmd("router list -f json", args, as_json=True)
+            if router["Name"] == "test-router"
+        ][0]["ID"]
+    except IndexError:
+        pass
+
+    if not router_id:
+        router_id = test_user_openstack_cmd(
+            "router create test-router -f json", args, as_json=True
+        )["id"]
+        print("router created")
+
+    router_info = openstack_cmd(f"router show -f json {router_id}", args, as_json=True)
+
+    if router_info["external_gateway_info"]:
+        print("router gateway already set")
+    else:
+        openstack_cmd(
+            f"router set {router_id} --external-gateway {args.public_network_id}", args
+        )
+        print("router gateway added")
+
+    if [
+        interface
+        for interface in router_info["interfaces_info"]
+        if interface["subnet_id"] == args.private_subnet_id
+    ]:
+        print("router already connected to private subnet")
+    else:
+        cmd = f"router add subnet {router_id} {args.private_subnet_id}"
+        print(cmd)
+        openstack_cmd(cmd, args)
+        print("router subnet added")
+
+    args.router_id = router_id
+
+
+def create_security_group(args):
+    """Create a security group that allows ssh and icmp"""
+
+    sg_exists = [
+        sg
+        for sg in json.loads(
+            test_user_openstack_cmd("security group list -f json", args)
+        )
+        if sg["Name"] == "test-sg"
+    ]
+
+    if sg_exists:
+        args.sg_id = sg_exists[0]["ID"]
+    else:
+        sec_grp = json.loads(
+            test_user_openstack_cmd(
+                "security group create test-sg -f json",
+                args
+                )
+        )
+        args.sg_id = sec_grp["id"]
+
+        _ = test_user_openstack_cmd(
+            (
+                "security group rule create --dst-port 22 "
+                f"--protocol tcp {args.sg_id}"
+            ),
+            args,
+        )
+
+        _ = test_user_openstack_cmd(
+            ("security group rule create --protocol icmp " f"{args.sg_id}"), args
+        )
 
 
 def main():
@@ -202,6 +541,7 @@ def main():
 
     create_public_network(args)
     create_private_network(args)
+    create_router(args)
 
     create_cirros_flavor(args)
     create_rhel_flavor(args)
@@ -209,9 +549,14 @@ def main():
     create_cirros_image(args)
     create_rhel_image(args)
 
+    create_keypair(args)
+    create_security_group(args)
+
     create_cirros_instance(args)
     create_rhel_instance(args)
 
+    ## create_os_migrate host
+
 
 if __name__ == "__main__":
     main()

src/tripleo_aio_helpers/os_migrate_teardown.py

@@ -4,21 +4,7 @@ Quick and dirty script to help setup project, flavors, networks, images
 import argparse
 import json
 
-# import os
+from .utils import openstack_cmd, test_user_openstack_cmd
-import subprocess
-
-
-def openstack_cmd(cmd, args):
-    """Utility function to run an openstack command agains the standalone cloud"""
-    cmd = "OS_CLOUD=standalone openstack " + cmd
-
-    if args.dry_run:
-        print("dry-run specified. Not executing cmd. cmd was:")
-        print(cmd)
-        return
-
-    result = subprocess.check_output(cmd, shell=True, universal_newlines=True)
-    return result
 
 
 def parse_args():
@@ -27,23 +13,23 @@ def parse_args():
     # home = os.environ.get('HOME')
 
     parser = argparse.ArgumentParser()
-    parser.add_argument("-d", "--project-domain", default="default")
     parser.add_argument("-n", "--project-name", default="test-project")
-    parser.add_argument("-D", "--project-description", default="Test project")
     parser.add_argument("-u", "--username", default="test-user")
     parser.add_argument("-p", "--password", default="secrete123")
     parser.add_argument("-c", "--cloud", default="standalone")
-    parser.add_argument("-g", "--gateway", default="10.76.23.254")
+    parser.add_argument("--delete-all", action="store_true")
-    parser.add_argument(
+    parser.add_argument("--delete-images", action="store_true")
-        "-C", "--public-network-cider", default="10.76.23.0/24"
+    parser.add_argument("--delete-flavors", action="store_true")
-    )
+    parser.add_argument("--delete-networks", action="store_true")
-    parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
+    parser.add_argument("--delete-instances", action="store_true")
-    parser.add_argument("--publice-net-start", default="10.76.23.50")
+    parser.add_argument("--delete-user", action="store_true")
-    parser.add_argument("--publice-net-end", default="10.76.23.52")
+    parser.add_argument("--delete-project", action="store_true")
-    parser.add_argument("--dns-server", default="10.76.23.245")
 
-    # export OS_CLOUD=standalone
+    parser.add_argument("--dry-run", action="store_true")
-    # export STANDALONE_HOST=10.76.23.39
+    parser.add_argument(
+        "--ssh", help="Connection string to run commands on a remote host."
+    )
+    parser.add_argument("--debug", action="store_true")
 
     args = parser.parse_args()
 
@@ -63,13 +49,11 @@ def destroy_project(args):
         print(f"Project {args.project_name} exists. Will delete")
         args.project_id = project_exists[0]["ID"]
 
-        cmd = (
+        cmd = f"project delete {args.project_id}"
-            f"project delete -f json --domain {args.project_domain} "
+        _ = openstack_cmd(cmd, args)
-            f"{args.project_id}"
-        )
         print(f"Project {args.project_name} deleted")
     else:
-        print("Project {args.project_name} not found.")
+        print(f"Project {args.project_name} not found.")
 
 
 def destroy_user(args):
@@ -80,64 +64,57 @@ def destroy_user(args):
     ]
 
     if user_exists:
-        print(f"User {args.username} already exists. Skipping creation")
+        user_id = user_exists[0]["ID"]
-        args.user_id = user_exists[0]["ID"]
-        return
 
-    cmd = (
+        openstack_cmd(f"user delete {user_id}", args)
-        f"user create  -f json --project {args.project_id} "
+        print(f"User deleted - id: {user_id}")
-        f"--password {args.password} {args.username}"
+    else:
-    )
+        print(f"User {args.username} not found.")
-
-    args.user_id = json.loads(openstack_cmd(cmd, args))["id"]
-
-    print(f"User created - id: {args.user_id}")
-
-
-def assign_member_role(args):
-    """
-    Assign the member role to the user.
-
-    Note: it doesn't matter if the role is assigned multiple times, so not
-    bothering to check.
-    """
-
-    cmd = f"role add --user {args.username} --project {args.project_id} member"
-
-    result = openstack_cmd(cmd, args)
-
-    cmd = f"role assignment list --user {args.user_id} --role member -f json"
-    result = json.loads(openstack_cmd(cmd, args))
-
-    if result:
-        print("User has member role")
 
 
 def destroy_public_network(args):
-    """Coming soon - create the public network"""
+    """Delete the public network"""
-    # pylint: disable=unused-argument,unused-variable
+    print("deleting public network")
-    print("creating public network - NYI")
+
-    cmd = (
+    public_network_exists = [
-        "network create --external --provider-physical-network datacentre "
+        network
-        "--provider-network-type flat public"
+        for network in openstack_cmd("network list -f json", args, as_json=True)
+        if network["Name"] == "public"
+    ]
+
+    if public_network_exists:
+        routers = test_user_openstack_cmd(
+                "router list -f json", args, as_json=True
             )
-    cmd = (
+
-        f"subnet create public-net --subnet-range {args.publice_network_cidr} "
+        for router in routers:
-        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+
-        f"start={args.public_net_start},end={args.public_net_end} "
+            router_details = test_user_openstack_cmd(
-        "--network public"
+                f"router show -f json {router['ID']}", args, as_json=True
             )
 
+            for interface in router_details["interfaces_info"]:
+                test_user_openstack_cmd(
+                    f"router remove port {router['ID']} {interface['port_id']}",
+                    args,
+                )
+
+            test_user_openstack_cmd(f"router delete {router['ID']}", args)
+
+        openstack_cmd("network delete public", args)
+
 
 def destroy_private_network(args):
-    """Coming soon - create the private network"""
+    """Delete the private network"""
-    # pylint: disable=unused-argument,unused-variable
+    private_network_exists = [
-    cmd = "openstack network create --internal private"
+        network
-    cmd = (
+        for network in openstack_cmd("network list -f json", args, as_json=True)
-        "openstack subnet create private-net "
+        if network["Name"] == "private"
-        f"--subnet-range {args.private_network_cidr} --network private"
+    ]
-    )
+
-    print("creating private network - NYI")
+    if private_network_exists:
+        print("deleting private network")
+        test_user_openstack_cmd("network delete private", args)
 
 
 def destroy_cirros_flavor(args):
@@ -153,9 +130,9 @@ def destroy_rhel_flavor(args):
 
 
 def destroy_cirros_image(args):
-    """Coming soon - create the cirros image"""
+    """Coming soon - destroy the cirros image"""
     # pylint: disable=unused-argument
-    print("creating cirros image - NYI")
+    print("destroying cirros image - NYI")
 
 
 def destroy_rhel_image(args):
@@ -165,34 +142,69 @@ def destroy_rhel_image(args):
 
 
 def destroy_cirros_instance(args):
-    """Coming soon - create the cirros instance"""
+    """Delete the cirros instance"""
-    # pylint: disable=unused-argument
+    try:
-    print("creating cirros instance - NYI")
+        cirros_instance = [
+            instance
+            for instance in test_user_openstack_cmd(
+                "server list -f json", args, as_json=True
+            )
+            if instance["Name"] == "cirros-test-instance"
+        ][0]["ID"]
+
+        test_user_openstack_cmd(f"server delete {cirros_instance}", args)
+    except IndexError:
+        print("No cirros instance found")
 
 
 def destroy_rhel_instance(args):
-    """Coming soon - create the rhel instance"""
+    """Delete the cirros instance"""
-    # pylint: disable=unused-argument
+    try:
-    print("creating rhel instance - NYI")
+        rhel_instance = [
+            instance
+            for instance in test_user_openstack_cmd(
+                "server list -f json", args, as_json=True
+            )
+            if instance["Name"] == "rhel-test-instance"
+        ][0]["ID"]
+
+        test_user_openstack_cmd(f"server delete {rhel_instance}", args)
+    except IndexError:
+        print("No RHEL instance found")
+
+
+def get_project_id(args):
+    """Get the id of the test project"""
+    result = openstack_cmd("project list -f json", args, as_json=True)
+    args.project_id = [r for r in result if r["Name"] == args.project_name][0]["ID"]
 
 
 def main():
     """main function"""
     args = parse_args()
 
-    # destroy_cirros_instance(args)
+    get_project_id(args)
-    # destroy_rhel_instance(args)
 
-    # destroy_cirros_image(args)
+    if args.delete_instances or args.delete_all:
-    # destroy_rhel_image(args)
+        destroy_cirros_instance(args)
+        destroy_rhel_instance(args)
 
-    # destroy_cirros_flavor(args)
+    if args.delete_images or args.delete_all:
-    # destroy_rhel_flavor(args)
+        destroy_cirros_image(args)
+        destroy_rhel_image(args)
 
-    # destroy_public_network(args)
+    if args.delete_flavors or args.delete_all:
-    # destroy_private_network(args)
+        destroy_cirros_flavor(args)
+        destroy_rhel_flavor(args)
 
+    if args.delete_networks or args.delete_all:
+        destroy_public_network(args)
+        destroy_private_network(args)
+
+    if args.delete_user or args.delete_all:
         destroy_user(args)
+
+    if args.delete_project or args.delete_all:
         destroy_project(args)
 
 

src/tripleo_aio_helpers/os_migrate_validate.py

@@ -0,0 +1,170 @@
+"""
+Quick and dirty script to validate that the necessary project, user,roles, 
+flavors, networks, images have been created
+"""
+import argparse
+import json
+
+from .utils import openstack_cmd
+
+
+def parse_args():
+    """Parse the command line arguments"""
+
+    # home = os.environ.get('HOME')
+
+    parser = argparse.ArgumentParser()
+    # parser.add_argument("-d", "--project-domain", default="default")
+    parser.add_argument("-n", "--project-name", default="test-project")
+    # parser.add_argument("-D", "--project-description", default="Test project")
+    # parser.add_argument("-u", "--username", default="test-user")
+    # parser.add_argument("-p", "--password", default="secrete123")
+    # parser.add_argument("-c", "--cloud", default="standalone")
+    # parser.add_argument("-g", "--gateway", default="10.76.23.254")
+    # parser.add_argument(
+    #     "-C", "--public-network-cider", default="10.76.23.0/24"
+    # )
+    # parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
+    # parser.add_argument("--publice-net-start", default="10.76.23.50")
+    # parser.add_argument("--publice-net-end", default="10.76.23.52")
+    # parser.add_argument("--dns-server", default="10.76.23.245")
+
+    # parser.add_argument("--dry-run", action="store_true")
+    parser.add_argument(
+        "--ssh", help="Connection string to run commands on a remote host."
+    )
+
+    # export OS_CLOUD=standalone
+    # export STANDALONE_HOST=10.76.23.39
+
+    args = parser.parse_args()
+
+    return args
+
+
+def validate_project(args):
+    """Validate that the project exists"""
+    cmd = "project list -f json"
+    project_exists = [
+        x
+        for x in json.loads(openstack_cmd(cmd, args))
+        if x["Name"] == args.project_name
+    ]
+
+    if project_exists:
+        print(project_exists)
+    else:
+        print("Project {args.project_name} not found.")
+
+
+def validate_user(args):
+    """Validate that the user exists"""
+    cmd = "user list -f json"
+    user_exists = [
+        x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
+    ]
+
+    if user_exists:
+        print(user_exists)
+    else:
+        print("User not found")
+
+
+def validate_member_role(args):
+    """
+    Validate that the member role has been assigned to the user.
+    """
+    print(args)
+    # cmd = f"role add --user {args.username} --project {args.project_id} member"
+
+    # result = openstack_cmd(cmd, args)
+
+    # cmd = f"role assignment list --user {args.user_id} --role member -f json"
+    # result = json.loads(openstack_cmd(cmd, args))
+
+    # if result:
+    #     print("User has member role")
+
+
+def validate_public_network(args):
+    """Coming soon - validate the public network"""
+    # pylint: disable=unused-argument,unused-variable
+    print("Validate public network - NYI")
+    cmd = (
+        "network create --external --provider-physical-network datacentre "
+        "--provider-network-type flat public"
+    )
+    cmd = (
+        f"subnet create public-net --subnet-range {args.publice_network_cidr} "
+        f"--no-dhcp --gateway {args.gateway} --allocation-pool "
+        f"start={args.public_net_start},end={args.public_net_end} "
+        "--network public"
+    )
+
+
+def validate_private_network(args):
+    """Coming soon - validate the private network"""
+    # pylint: disable=unused-argument,unused-variable
+    cmd = "openstack network create --internal private"
+    cmd = (
+        "openstack subnet create private-net "
+        f"--subnet-range {args.private_network_cidr} --network private"
+    )
+    print("validate private network - NYI")
+
+
+def validate_cirros_flavor(args):
+    """Coming soon - create the cirros flavor"""
+    # pylint: disable=unused-argument
+    print("validate cirros flavor - NYI")
+
+
+def validate_rhel_flavor(args):
+    """Coming soon - validate the rhel flavor"""
+    # pylint: disable=unused-argument
+    print("validate rhel flavor - NYI")
+
+
+def validate_cirros_image(args):
+    """Coming soon - validate the cirros image"""
+    # pylint: disable=unused-argument
+    print("validate cirros image - NYI")
+
+
+def validate_rhel_image(args):
+    """Coming soon - validate the rhel image"""
+    # pylint: disable=unused-argument
+    print("validate rhel image - NYI")
+
+
+def validate_cirros_instance(args):
+    """Coming soon - validate the cirros instance"""
+    # pylint: disable=unused-argument
+    print("validate cirros instance - NYI")
+
+
+def validate_rhel_instance(args):
+    """Coming soon - create the rhel instance"""
+    # pylint: disable=unused-argument
+    print("validate rhel instance - NYI")
+
+
+def main():
+    """main function"""
+    args = parse_args()
+
+    validate_user(args)
+    validate_project(args)
+    validate_member_role(args)
+    validate_public_network(args)
+    validate_private_network(args)
+    validate_cirros_flavor(args)
+    validate_rhel_flavor(args)
+    validate_cirros_image(args)
+    validate_rhel_image(args)
+    validate_cirros_instance(args)
+    validate_rhel_instance(args)
+
+
+if __name__ == "__main__":
+    main()

src/tripleo_aio_helpers/prepare_deployment.py

@@ -18,9 +18,7 @@ def parse_args():
     parser.add_argument("-a", "--address", required=True)
     parser.add_argument("-i", "--interface", required=True)
     parser.add_argument("-m", "--netmask", default=24)
-    parser.add_argument(
+    parser.add_argument("-d", "--dns", nargs="+", action="append", required=True)
-        "-d", "--dns", nargs="+", action="append", required=True
-    )
     parser.add_argument("-D", "--domain", default="aio")
     parser.add_argument("--using-multiple-nics", action="store_true")
     parser.add_argument("-U", "--deployment-user")
@@ -37,6 +35,11 @@ def parse_args():
     parser.add_argument(
         "--deploy-script-out", default=f"{home}/deploy.sh.gen", dest="deploy"
     )
+    parser.add_argument(
+        "--cinder-pool-size",
+        default="500280",
+        help="Size of the loopback device allocated to cinder. " "Defaults to 500280",
+    )
 
     args = parser.parse_args()
 
@@ -69,6 +72,7 @@ def get_standalone_parameters(args):
             "StandaloneEnableRoutedNetworks": False,
             "StandaloneHomeDir": args.deployment_dir,
             "StandaloneLocalMtu": 1500,
+            "CinderLVMLoopDeviceSize": args.cinder_pool_size,
         }
     }
 
@@ -128,22 +132,17 @@ def set_hostname(args):
     print("set_hostname is not yet implemented")
 
 
-
 def main():
     """main function"""
     args = parse_args()
 
     containers_yaml = build_containers_yaml(args)
-    with open(
+    with open(args.containers_yaml_out, "w", encoding="utf-8") as containers_out:
-        args.containers_yaml_out, "w", encoding="utf-8"
-    ) as containers_out:
         containers_out.write(yaml.dump(containers_yaml))
     print(f"containers yaml written to {args.containers_yaml_out}")
 
     standalone_parameters = get_standalone_parameters(args)
-    with open(
+    with open(args.standalone_yaml_out, "w", encoding="utf-8") as parameters_out:
-        args.standalone_yaml_out, "w", encoding="utf-8"
-    ) as parameters_out:
         parameters_out.write(yaml.dump(standalone_parameters))
     print(f"standalone parameters yaml written to {args.standalone_yaml_out}")
 
@@ -151,7 +150,10 @@ def main():
         deploy.write(deploy_sh(args))
     print(f"deploy script written to {args.deploy}")
 
-    print("If you are running on a cloud image remenber to disable cloud-init before running the deploy.")
+    print(
+        "If you are running on a cloud image remenber to disable cloud-init "
+        "before running the deploy."
+    )
     print("sudo systemctl stop cloud-init")
     print("sudo systemctl disable cloud-init")
     print("Make sure you have specifed the correct interface to use!")