neillc/tripleo-aio-helpers
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

🚨 Add pre-commit

Browse source
This commit is contained in:
Neill Cox 2023-10-19 17:57:12 +11:00
parent f85e69e08f
commit e62f806e86
17 changed files with 158 additions and 80 deletions
Download patch file Download diff file Expand all files Collapse all files

25
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,25 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/psf/black
rev: 23.10.0
hooks:
- id: black
args: ["--line-length=79"]
- repo: https://github.com/PyCQA/isort
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/PyCQA/flake8
rev: 6.1.0
hooks:
- id: flake8
exclude: migrations/

10
README.md
View file

@ -18,11 +18,11 @@ I am primarily targetting RHEL8.4 and RHOSP16.2. As a result I'm tied to virt-in
There is a vestigial network-config.tpl in the virt-install directory but it is not actually used yet.
## os_migrate_setup.py
## os_migrate_setup.py
build the OSP infrastructure to allow testing os-migrate.
## os_migrate_teardown.py
## os_migrate_teardown.py
remove that infrastructure (to allow for refinement/iteration)
@ -32,11 +32,11 @@ virt-install/create_aio_vm.py - use virt-install to create and configure a VM to
Original script. Generates the yaml files and a dploy script to deploy a RHOSP AIO. Assumes a VM to run on. Don't run this on your laptop. You'll be sorry if you do.
## os_migrate_validate.py
## os_migrate_validate.py
Check that the necessary infrastructure has been deployed.
# Remember to mask and disable cloud-init if installing on a VM
# Remember to mask and disable cloud-init if installing on a VM
```
usage: main.py [-h] -u USERNAME -p PASSWORD -a ADDRESS [-i INTERFACE]
@ -69,4 +69,4 @@ By default the files will be placed in the home directory of the user running th
When happy with the contents rename them to remove the .gen suffix and run the deploy script.
The doco for the AIO deply says to run the deploy using sudo - I don't think this is correct. If you use sudo you will get a warning about files ending up /root.
The doco for the AIO deply says to run the deploy using sudo - I don't think this is correct. If you use sudo you will get a warning about files ending up /root.

2
TODO.md
View file

@ -18,4 +18,4 @@ Maybe this should all have been an ansible role. Maybe that will happen yet.
## create_aio_vm
Quite a few compromises to do with the versions of virt_install / libvirt available to me.
Quite a few compromises to do with the versions of virt_install / libvirt available to me.

1
dev_requirements.txt
View file

@ -4,3 +4,4 @@ flake8==5.0.4
pyflakes==2.5.0
pylint==2.13.9
PyYAML==6.0.1
pre-commit

2
docs/create_aio_vm.md
View file

@ -2,4 +2,4 @@
## Networking
Networking has been a real pain. The RHOSP AIO deployment seems to force the vr-ctlplane interface to use dhcp. Originally I was trying to use static IPs configured with cloud-init (using bootcmd because of the version) but the deploy would die. To fix this I now speciiy MACs for the interfaces and then use an external DHCP server. Maybe this is configurable in the RHOSP standalone deploy. If so it would be better to use that. It would also be nice to do some feature switches so that if the version of virt-install allows I could use network-config instead of bootcmd
Networking has been a real pain. The RHOSP AIO deployment seems to force the vr-ctlplane interface to use dhcp. Originally I was trying to use static IPs configured with cloud-init (using bootcmd because of the version) but the deploy would die. To fix this I now speciiy MACs for the interfaces and then use an external DHCP server. Maybe this is configurable in the RHOSP standalone deploy. If so it would be better to use that. It would also be nice to do some feature switches so that if the version of virt-install allows I could use network-config instead of bootcmd

10
docs/notes.md
View file

@ -77,8 +77,8 @@ OSM_CMD $OSM_DIR/playbooks/export_networks.yml
```
```bash
60 less os-migrate-data-test-1/clouds.yaml
61 less os-migrate-data-test-1/networks.yml
67 $OSM_CMD $OSM_DIR/playbooks/export_subnets.yml
68 less os-migrate-data-test-1/subnets.yml
```
60 less os-migrate-data-test-1/clouds.yaml
61 less os-migrate-data-test-1/networks.yml
67 $OSM_CMD $OSM_DIR/playbooks/export_subnets.yml
68 less os-migrate-data-test-1/subnets.yml
```

1
examples/create_aio.bash
View file

@ -18,4 +18,3 @@ create_aio_vm \
--search-domain evatt.ingenious.com.au \
--rhn-user $AIO_RHN_USER \
--rhn-password $AIO_RHN_PASSWORD

2
examples/setup.bash
View file

@ -13,4 +13,4 @@ os_migrate_setup \
--cirros-url https://isos.evatt.ingenious.com.au/cirros-0.6.2-x86_64-disk.img \
--rhel-url https://isos.evatt.ingenious.com.au/rhel-guest-image-8.4-1269.x86_64.qcow2 \
--ssh-key ~/.ssh/id_rsa.pub \
--debug
--debug

2
pyproject.toml
View file

@ -26,4 +26,4 @@ create_aio_vm = "tripleo_aio_helpers.create_aio_vm:main"
os_migrate_setup = "tripleo_aio_helpers.os_migrate_setup:main"
os_migrate_teardown = "tripleo_aio_helpers.os_migrate_teardown:main"
os_migrate_validate = "tripleo_aio_helpers.os_migrate_validate:main"
prepare_deployment = "tripleo_aio_helpers.prepare_deployment:main"
prepare_deployment = "tripleo_aio_helpers.prepare_deployment:main"

30
src/tripleo_aio_helpers/create_aio_vm.py
View file

@ -23,8 +23,12 @@ def parse_args():
parser.add_argument("--local-hostname", required=True)
parser.add_argument("--user-data", default=template_path / "user-data.tpl")
parser.add_argument("--meta-data", default=template_path / "meta-data.tpl")
parser.add_argument("--network-data", default=template_path / "network-config.tpl")
parser.add_argument("--instance-id", required=True, help="Hostname for the new VM")
parser.add_argument(
"--network-data", default=template_path / "network-config.tpl"
)
parser.add_argument(
"--instance-id", required=True, help="Hostname for the new VM"
)
parser.add_argument("--output-image", required=True)
parser.add_argument("--image-size", default="800G")
parser.add_argument("--input-image", required=True)
@ -63,15 +67,6 @@ def parse_args():
print("You must specify a DNS server if you specify any addresses")
sys.exit(1)
# if not (args.cidr_1 or args.cidr_2) and (
# args.dns or args.gateway or args.search_domain
# ):
# print(
# "There's no point specifying DNS, gateway or search_domain if you"
# " don't specify addresses"
# )
# sys.exit(1)
args.public_key = args.public_key.read()
with open(args.user_data, encoding="utf-8") as user_data:
@ -153,7 +148,10 @@ def create_image(args):
print(result)
print("Resizing image")
cmd = f"virt-resize --expand /dev/sda3 {args.input_image} {args.output_image}"
cmd = (
f"virt-resize --expand /dev/sda3 {args.input_image} "
f"{args.output_image}"
)
result = subprocess.check_output(cmd, shell=True, universal_newlines=True)
@ -205,14 +203,16 @@ def delete_user_data():
def install_packages(args):
"""Install the packages needed for virt-install to work"""
cmd = (
"sudo dnf install -y virt-install virt-viewer qemu-img "
"sudo dnf install -y virt-install virt-viewer qemu-img "
"libguestfs.x86_64"
)
)
print("installing needed packages...")
if args.verbose:
print(subprocess.check_output(cmd, shell=True, universal_newlines=True))
print(
subprocess.check_output(cmd, shell=True, universal_newlines=True)
)
def main():

82
src/tripleo_aio_helpers/os_migrate_setup.py
View file

@ -35,7 +35,8 @@ def parse_args():
parser.add_argument("-g", "--gateway")
parser.add_argument("-C", "--public-network-cidr")
parser.add_argument(
"--ssh-key", help="File containing a public key to inject into the instances"
"--ssh-key",
help="File containing a public key to inject into the instances",
)
parser.add_argument("--private-network-cidr", default="192.168.100.0/24")
parser.add_argument("--public-net-start")
@ -76,7 +77,9 @@ def parse_args():
)
if not args.public_net_end:
args.public_net_end = get_from_env("--public-net-end", "AIO_PUBLIC_NET_END")
args.public_net_end = get_from_env(
"--public-net-end", "AIO_PUBLIC_NET_END"
)
if not args.dns_server:
args.dns_server = get_from_env("--dns-server", "AIO_DNS_SERVER")
@ -130,7 +133,9 @@ def create_user(args):
cmd = "user list -f json"
user_exists = [
x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
x
for x in json.loads(openstack_cmd(cmd, args))
if x["Name"] == args.username
]
if user_exists:
@ -183,7 +188,8 @@ def create_public_network(args):
args.public_network_id = network_exists[0]["ID"]
else:
cmd = (
"network create -f json --external --provider-physical-network datacentre "
"network create -f json --external "
"--provider-physical-network datacentre "
"--provider-network-type flat public"
)
try:
@ -203,7 +209,8 @@ def create_public_network(args):
"subnet create public-net -f json "
f"--subnet-range {args.public_network_cidr} "
f"--gateway {args.gateway} "
f"--allocation-pool start={args.public_net_start},end={args.public_net_end} "
f"--allocation-pool start={args.public_net_start},"
f"end={args.public_net_end} "
"--network public "
f"--host-route destination=0.0.0.0/0,gateway={args.gateway} "
f"--dns-nameserver {args.dns_server}"
@ -217,7 +224,8 @@ def create_private_network(args):
network_exists = json.loads(
openstack_cmd(
f"network list -f json --project {args.project_id} --name private", args
f"network list -f json --project {args.project_id} --name private",
args,
)
)
@ -225,13 +233,18 @@ def create_private_network(args):
print("Private network already exists - skipping")
args.private_network_id = network_exists[0]["ID"]
else:
cmd = f"network create -f json --internal private --project {args.project_id}"
cmd = (
f"network create -f json --internal private "
f"--project {args.project_id}"
)
args.private_network_id = json.loads(openstack_cmd(cmd, args))["id"]
print("Private network created.")
subnet_exists = json.loads(
openstack_cmd(
f"subnet list -f json --project {args.project_id} --name private-net", args
f"subnet list -f json --project {args.project_id} "
f"--name private-net",
args,
)
)
if subnet_exists:
@ -240,7 +253,8 @@ def create_private_network(args):
else:
cmd = (
f"subnet create private-net -f json --project {args.project_id} "
f"--subnet-range {args.private_network_cidr} --network {args.private_network_id}"
f"--subnet-range {args.private_network_cidr} "
f"--network {args.private_network_id}"
)
args.private_subnet_id = json.loads(openstack_cmd(cmd, args))
print("Private subnet created.")
@ -261,12 +275,12 @@ def create_flavor(args, flavor_name, memory, disk, cpus):
print("creating cirros flavor")
# Note can't add a description in RHOSP16, but perhaps should add for 17
cmd = f"""
flavor create -f json
flavor create -f json
--ram {memory}
--disk {disk}
--vcpus {cpus}
--private
--project {args.project_id}
--vcpus {cpus}
--private
--project {args.project_id}
{flavor_name}
""".replace(
"\n", " "
@ -303,7 +317,6 @@ def create_image(image_name, image_url, disk_size, memory, args):
return
with tempfile.TemporaryDirectory() as tmp_dir:
# download image to tmpdir
fname = f"{tmp_dir}/{image_name}.img"
cmd = f"wget -O {fname} {image_url}"
@ -314,7 +327,8 @@ def create_image(image_name, image_url, disk_size, memory, args):
cmd = f"scp {fname} {args.ssh}:{image_name}.img"
execute_cmd(cmd)
# create image - note this will only work on a remote ssh host at the moment
# create image - note this will only work on a remote ssh host at
# the moment
cmd = f"""
image create -f json
--container-format bare
@ -379,7 +393,9 @@ def create_instance(args, name, flavor, image, security_group, boot_size):
"floating ip create -f json public", args, as_json=True
)
_ = test_user_openstack_cmd(
f"server add floating ip {server['id']} {fip['floating_ip_address']}", args
f"server add floating ip {server['id']} "
f"{fip['floating_ip_address']}",
args,
)
@ -404,7 +420,7 @@ def create_rhel_instance(args):
args.rhel_flavor,
args.rhel_image,
args.sg_id,
120
120,
)
@ -416,7 +432,7 @@ def create_conversion_instance(args):
args.rhel_flavor,
args.rhel_image,
args.sg_id,
120
120,
)
@ -425,7 +441,9 @@ def create_keypair(args):
key_exists = [
kp
for kp in json.loads(test_user_openstack_cmd("keypair list -f json ", args))
for kp in json.loads(
test_user_openstack_cmd("keypair list -f json ", args)
)
if kp["Name"] == "test_keypair"
]
@ -440,7 +458,9 @@ def create_keypair(args):
args.keypair_name = json.loads(
test_user_openstack_cmd(
f"keypair create -f json --public-key {fname} test_keypair", args
f"keypair create -f json "
f"--public-key {fname} test_keypair",
args,
)
)[
"name"
@ -458,7 +478,9 @@ def create_router(args):
try:
router_id = [
router
for router in openstack_cmd("router list -f json", args, as_json=True)
for router in openstack_cmd(
"router list -f json", args, as_json=True
)
if router["Name"] == "test-router"
][0]["ID"]
except IndexError:
@ -470,13 +492,17 @@ def create_router(args):
)["id"]
print("router created")
router_info = openstack_cmd(f"router show -f json {router_id}", args, as_json=True)
router_info = openstack_cmd(
f"router show -f json {router_id}", args, as_json=True
)
if router_info["external_gateway_info"]:
print("router gateway already set")
else:
openstack_cmd(
f"router set {router_id} --external-gateway {args.public_network_id}", args
f"router set {router_id} "
f"--external-gateway {args.public_network_id}",
args,
)
print("router gateway added")
@ -511,9 +537,8 @@ def create_security_group(args):
else:
sec_grp = json.loads(
test_user_openstack_cmd(
"security group create test-sg -f json",
args
)
"security group create test-sg -f json", args
)
)
args.sg_id = sec_grp["id"]
@ -526,7 +551,8 @@ def create_security_group(args):
)
_ = test_user_openstack_cmd(
("security group rule create --protocol icmp " f"{args.sg_id}"), args
("security group rule create --protocol icmp " f"{args.sg_id}"),
args,
)
@ -555,7 +581,7 @@ def main():
create_cirros_instance(args)
create_rhel_instance(args)
## create_os_migrate host
# create_os_migrate host
if __name__ == "__main__":

24
src/tripleo_aio_helpers/os_migrate_teardown.py
View file

@ -60,7 +60,9 @@ def destroy_user(args):
"""Delete the user if it exists"""
cmd = "user list -f json"
user_exists = [
x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
x
for x in json.loads(openstack_cmd(cmd, args))
if x["Name"] == args.username
]
if user_exists:
@ -78,24 +80,26 @@ def destroy_public_network(args):
public_network_exists = [
network
for network in openstack_cmd("network list -f json", args, as_json=True)
for network in openstack_cmd(
"network list -f json", args, as_json=True
)
if network["Name"] == "public"
]
if public_network_exists:
routers = test_user_openstack_cmd(
"router list -f json", args, as_json=True
)
"router list -f json", args, as_json=True
)
for router in routers:
router_details = test_user_openstack_cmd(
f"router show -f json {router['ID']}", args, as_json=True
)
for interface in router_details["interfaces_info"]:
test_user_openstack_cmd(
f"router remove port {router['ID']} {interface['port_id']}",
f"router remove port {router['ID']} "
f"{interface['port_id']}",
args,
)
@ -108,7 +112,9 @@ def destroy_private_network(args):
"""Delete the private network"""
private_network_exists = [
network
for network in openstack_cmd("network list -f json", args, as_json=True)
for network in openstack_cmd(
"network list -f json", args, as_json=True
)
if network["Name"] == "private"
]
@ -176,7 +182,9 @@ def destroy_rhel_instance(args):
def get_project_id(args):
"""Get the id of the test project"""
result = openstack_cmd("project list -f json", args, as_json=True)
args.project_id = [r for r in result if r["Name"] == args.project_name][0]["ID"]
args.project_id = [r for r in result if r["Name"] == args.project_name][0][
"ID"
]
def main():

20
src/tripleo_aio_helpers/os_migrate_validate.py
View file

@ -1,5 +1,5 @@
"""
Quick and dirty script to validate that the necessary project, user,roles,
Quick and dirty script to validate that the necessary project, user,roles,
flavors, networks, images have been created
"""
import argparse
@ -16,7 +16,9 @@ def parse_args():
parser = argparse.ArgumentParser()
# parser.add_argument("-d", "--project-domain", default="default")
parser.add_argument("-n", "--project-name", default="test-project")
# parser.add_argument("-D", "--project-description", default="Test project")
# parser.add_argument(
# "-D", "--project-description", default="Test project"
# )
# parser.add_argument("-u", "--username", default="test-user")
# parser.add_argument("-p", "--password", default="secrete123")
# parser.add_argument("-c", "--cloud", default="standalone")
@ -61,7 +63,9 @@ def validate_user(args):
"""Validate that the user exists"""
cmd = "user list -f json"
user_exists = [
x for x in json.loads(openstack_cmd(cmd, args)) if x["Name"] == args.username
x
for x in json.loads(openstack_cmd(cmd, args))
if x["Name"] == args.username
]
if user_exists:
@ -75,7 +79,10 @@ def validate_member_role(args):
Validate that the member role has been assigned to the user.
"""
print(args)
# cmd = f"role add --user {args.username} --project {args.project_id} member"
# cmd = (
# f"role add --user {args.username} "
# f"--project {args.project_id} member"
# )
# result = openstack_cmd(cmd, args)
@ -100,13 +107,14 @@ def validate_public_network(args):
f"start={args.public_net_start},end={args.public_net_end} "
"--network public"
)
print(cmd)
def validate_private_network(args):
"""Coming soon - validate the private network"""
# pylint: disable=unused-argument,unused-variable
cmd = "openstack network create --internal private"
cmd = (
_ = "openstack network create --internal private"
_ = (
"openstack subnet create private-net "
f"--subnet-range {args.private_network_cidr} --network private"
)

16
src/tripleo_aio_helpers/prepare_deployment.py
View file

@ -4,6 +4,7 @@ Quick and dirty script to help setup config filers for a RHOSP AIO install
import argparse
import os
import subprocess
import yaml
@ -18,7 +19,9 @@ def parse_args():
parser.add_argument("-a", "--address", required=True)
parser.add_argument("-i", "--interface", required=True)
parser.add_argument("-m", "--netmask", default=24)
parser.add_argument("-d", "--dns", nargs="+", action="append", required=True)
parser.add_argument(
"-d", "--dns", nargs="+", action="append", required=True
)
parser.add_argument("-D", "--domain", default="aio")
parser.add_argument("--using-multiple-nics", action="store_true")
parser.add_argument("-U", "--deployment-user")
@ -38,7 +41,8 @@ def parse_args():
parser.add_argument(
"--cinder-pool-size",
default="500280",
help="Size of the loopback device allocated to cinder. " "Defaults to 500280",
help="Size of the loopback device allocated to cinder. "
"Defaults to 500280",
)
args = parser.parse_args()
@ -137,12 +141,16 @@ def main():
args = parse_args()
containers_yaml = build_containers_yaml(args)
with open(args.containers_yaml_out, "w", encoding="utf-8") as containers_out:
with open(
args.containers_yaml_out, "w", encoding="utf-8"
) as containers_out:
containers_out.write(yaml.dump(containers_yaml))
print(f"containers yaml written to {args.containers_yaml_out}")
standalone_parameters = get_standalone_parameters(args)
with open(args.standalone_yaml_out, "w", encoding="utf-8") as parameters_out:
with open(
args.standalone_yaml_out, "w", encoding="utf-8"
) as parameters_out:
parameters_out.write(yaml.dump(standalone_parameters))
print(f"standalone parameters yaml written to {args.standalone_yaml_out}")

8
src/tripleo_aio_helpers/utils.py
View file

@ -20,7 +20,9 @@ def get_from_env(name, envvar, required=True):
def openstack_cmd(cmd, args, as_json=False):
"""Utility function to run an openstack command agains the standalone cloud"""
"""
Utility function to run an openstack command agains the standalone cloud
"""
cmd = cmd.replace("\n", " ")
cmd = "OS_CLOUD=standalone openstack " + cmd
@ -34,7 +36,9 @@ def openstack_cmd(cmd, args, as_json=False):
return None
try:
result = subprocess.check_output(cmd, shell=True, universal_newlines=True)
result = subprocess.check_output(
cmd, shell=True, universal_newlines=True
)
if as_json:
result = json.loads(result)

2
src/virt-install/network-config.tpl
View file

@ -17,4 +17,4 @@ network:
address:
- {data.dns}
search:
- {data.search_domain}
- {data.search_domain}

1
src/virt-install/user-data.tpl
View file

@ -25,4 +25,3 @@ runcmd:
- "echo '*** ***'"
- "echo '***************************************************************************'"
- "reboot"