This document describes how one can implement infrastructure maintenance in interaction with VNFM by utilizing the OPNFV Doctor project framework and to meet the set requirements. Document concentrates to OpenStack and VMs while the concept designed is generic for any payload or even different VIM. Admin tool should be also for controller and other cloud hardware, but that is not the main focus in OPNFV Doctor and should be defined better in the upstream implementation. Same goes for any more detailed work to be done.
Telco application need to know when infrastructure maintenance is going to happen in order to guarantee zero down time in its operation. It needs to be possible to make own actions to have application running on not affected resource or give guidance to admin actions like migration. More details are defined in requirement documentation: use cases, architecture and implementation.
Concepts used:
Infrastructure admin needs to make notification with two different event types. One is meant for admin and one for project. Notification payload can be consumed by application and admin by subscribing to corresponding event alarm trough alarming service like OpenStack AODH.
Before maintenance starts application needs to be able to make switch over for his ACT-STBY service affected, do operation to move service to not effected part of infrastructure or give a hint for admin operation like migration that can be automatically issued by admin tool according to agreed policy.
There should be at least one empty host compatible to host under maintenance in order to have a smooth rolling maintenance done. For this to be possible also down scaling the application instances should be possible.
Infrastructure admin should have a tool that is responsible for hosting a maintenance work flow session with needed APIs for admin and for applications. The Group of hosts in single maintenance session should always have the same physical capabilities, so the rolling maintenance can be guaranteed.
Flow diagram is meant to be as high level as possible. It currently does not try to be perfect, but to show the most important interfaces needed between VNFM and infrastructure admin. This can be seen e.g. as missing error handling that can be defined later on.
Flow diagram:
Flow diagram step by step:
Infrastructure admin makes a maintenance session to maintain and upgrade certain group of hardware. At least compute hardware in single session should be having same capabilities like the amount number of VCPUs to ensure the maintenance can be done node by node in rolling fashion. Maintenance session need to have a session_id that is a unique ID to be carried throughout all events and can be used in APIs needed when interacting with the session. Maintenance session needs to have knowledge about when maintenance will start and what capabilities the possible upgrade to infrastructure will bring to application payload on top of it. It will be matter of the implementation to define in more detail whether some more data is needed when creating a session or if it is defined in the admin tool configuration.
There can be several parallel maintenance sessions and a single session can include multiple projects payload. Typically maintenance session should include similar type of compute hardware, so you can guarantee moving of instances on top of them can work between the compute hosts.
State MAINTENANCE project event and reply ACK_MAINTENANCE. Immediately after a maintenance session is created, infrastructure admin tool will send a project specific ‘notification’ which application manager can consume by subscribing to AODH alarm for this event. As explained already earlier all `project event`s will only be sent in case the project subscribes to alarm and otherwise the interaction with application will simply not be done and operations could be forced.
The state MAINTENANCE event should at least include:
- session_id to reference correct maintenance session.
- state as MAINTENANCE to identify event action needed.
- instance_ids to tell project which of his instances will be affected by the maintenance. This might be a link to admin tool project specific API as AODH variables are limited to string of 255 character.
- reply_url for application to call admin tool project specific API to answer ACK_MAINTENANCE including the session_id.
- project_id to identify project.
- actions_at time stamp to indicate when maintenance work flow will start. ACK_MAINTENANCE reply is needed before that time.
- metadata to include key values pairs of a capabilities coming over the maintenance operation like ‘openstack_version’: ‘Queens’
Optional state DOWN_SCALE project event and reply ACK_DOWN_SCALE. When it is time to start the maintenance work flow as the time reaches the actions_at defined in previous state event, admin tool needs to check if there is already an empty compute host needed by the rolling maintenance. In case there is no empty host, admin tool can ask application to down scale by sending project specific DOWN_SCALE state event.
The state DOWN_SCALE event should at least include:
- session_id to reference correct maintenance session.
- state as DOWN_SCALE to identify event action needed.
- reply_url for application to call admin tool project specific API to answer ACK_DOWN_SCALE including the session_id.
- project_id to identify project.
- actions_at time stamp to indicate when is the last moment to send ACK_DOWN_SCALE. This means application can have time to finish some ongoing transactions before down scaling his instances. This guarantees a zero downtime for his service.
Optional state PREPARE_MAINTENANCE project event and reply ACK_PREPARE_MAINTENANCE. In case still after down scaling the applications there is still no empty compute host, admin tools needs to analyze the situation on compute host under maintenance. It needs to choose compute node that is now almost empty or has otherwise least critical instances running if possible, like looking if there is floating IPs. When compute host is chosen, a PREPARE_MAINTENANCE state event can be sent to projects having instances running on this host to migrate them to other compute hosts. It might also be possible to have another round of DOWN_SCALE state event if necessary, but this is not proposed here.
The state PREPARE_MAINTENANCE event should at least include:
- session_id to reference correct maintenance session.
- state as PREPARE_MAINTENANCE to identify event action needed.
- instance_ids to tell project which of his instances will be affected by the state event. This might be a link to admin tool project specific API as AODH variables are limited to string of 255 character.
- reply_url for application to call admin tool project specific API to answer ACK_PREPARE_MAINTENANCE including the session_id and instance_ids with list of key value pairs with key as instance_id and chosen action from allowed actions given via allowed_actions as value.
- project_id to identify project.
- actions_at time stamp to indicate when is the last moment to send ACK_PREPARE_MAINTENANCE. This means application can have time to finish some ongoing transactions within his instances and make possible switch over. This guarantees a zero downtime for his service.
- allowed_actions to tell what admin tool supports as action to move instances to another compute host. Typically a list like: [‘MIGRATE’, ‘LIVE_MIGRATE’]
Optional state INSTANCE_ACTION_DONE project event. In case admin tool needed to make action to move instance like migrating it to another compute host, this state event will be sent to tell the operation is complete.
The state INSTANCE_ACTION_DONE event should at least include:
- session_id to reference correct maintenance session.
- instance_ids to tell project which of his instance had the admin action done.
- project_id to identify project.
At this state it is guaranteed there is an empty compute host. It would be maintained first trough IN_MAINTENANCE and MAINTENANCE_COMPLETE steps, but following the flow chart PLANNED_MAINTENANCE will be explained next.
Optional state PLANNED_MAINTENANCE project event and reply ACK_PLANNED_MAINTENANCE. In case compute host to be maintained has instances, projects owning those should have this state event. When project receives this state event it knows instances moved to other compute host as resulting actions will now go to host that is already maintained. This means it might have new capabilities that project can take into use. This gives the project the possibility to upgrade his instances also to support new capabilities over the action chosen to move instances.
The state PLANNED_MAINTENANCE event should at least include:
- session_id to reference correct maintenance session.
- state as PLANNED_MAINTENANCE to identify event action needed.
- instance_ids to tell project which of his instances will be affected by the event. This might be a link to admin tool project specific API as AODH variables are limited to string of 255 character.
- reply_url for application to call admin tool project specific API to answer ACK_PLANNED_MAINTENANCE including the session_id and instance_ids with list of key value pairs with key as instance_id and chosen action from allowed actions given via allowed_actions as value.
- project_id to identify project.
- actions_at time stamp to indicate when is the last moment to send ACK_PLANNED_MAINTENANCE. This means application can have time to finish some ongoing transactions within his instances and make possible switch over. This guarantees a zero downtime for his service.
- allowed_actions to tell what admin tool supports as action to move instances to another compute host. Typically a list like: [‘MIGRATE’, ‘LIVE_MIGRATE’, ‘OWN_ACTION’] OWN_ACTION means that application may want to re-instantiate his instance perhaps to take into use the new capability coming over the infrastructure maintenance. Re-instantiated instance will go to already maintained host having the new capability.
- metadata to include key values pairs of a capabilities coming over the maintenance operation like ‘openstack_version’: ‘Queens’
State IN_MAINTENANCE and MAINTENANCE_COMPLETE admin event`s. Just before host goes to maintenance the IN_MAINTENANCE state event will be send to indicate host is entering to maintenance. Host is then taken out of production and can be powered off, replaced, or rebooted during the operation. During the maintenance and upgrade host might be moved to admin’s own host aggregate, so it can be tested to work before putting back to production. After maintenance is complete MAINTENANCE_COMPLETE state event will be sent to know host is back in use. Adding or removing of a host is yet not included in this concept, but can be addressed later.
The state IN_MAINTENANCE and MAINTENANCE_COMPLETE event should at least include:
- session_id to reference correct maintenance session.
- state as IN_MAINTENANCE or MAINTENANCE_COMPLETE to indicate host state.
- project_id to identify admin project needed by AODH alarm.
- host to indicate the host name.
State MAINTENANCE_COMPLETE project event and reply MAINTENANCE_COMPLETE_ACK. After all compute nodes in the maintenance session have gone trough maintenance operation this state event can be send to all projects that had instances running on any of those nodes. If there was a down scale done, now the application could up scale back to full operation.
- session_id to reference correct maintenance session.
- state as MAINTENANCE_COMPLETE to identify event action needed.
- instance_ids to tell project which of his instances are currently running on hosts maintained in this maintenance session. This might be a link to admin tool project specific API as AODH variables are limited to string of 255 character.
- reply_url for application to call admin tool project specific API to answer ACK_MAINTENANCE including the session_id.
- project_id to identify project.
- actions_at time stamp to indicate when maintenance work flow will start.
- metadata to include key values pairs of a capabilities coming over the maintenance operation like ‘openstack_version’: ‘Queens’
At the end admin tool maintenance session can enter to MAINTENANCE_COMPLETE state and session can be removed.
There was a Maintenance POC demo ‘How to gain VNF zero down-time during Infrastructure Maintenance and Upgrade’ in the OCP and ONS summit March 2018. Similar concept is also being made as OPNFV Doctor project new test case scenario.