Skip to end of metadata
Go to start of metadata
 
  • Replace the gray cursive text with the actual content.
  • Remove non-applicable sections or insert N/A.

High Level Functional Overview and Design Specifications

Problem Statement

  • CoprHD has an integration with OpenStack Cinder, calling it a OpenStack southbound integration to manage all third party storage systems supported and managed by OpenStack Cinder.
  • This solution is not fully tested for various customer use cases, for e.g Add host to cluster, remove host from cluster were not real part of the test plan and were never tested.
  • VPLEX+Cinder support is introduced from Darth, however the qualification and test effort carried out was targeted to a single customer. There is a need to cover all cases and scenarios supported by VPLEX.
  • VPLEX + Cinder + RP support is claimed but the testing is not done to cover various RP configurations.
  • There are quite number of jiras which are in unresolved state, since the priority was to fix it if there exists a customer hence they had taken a backseat. 

Functional Requirements

    • Specific requirements.
    • Include JIRA IDs where applicable:
JIRA ID1-Line DescriptionComments, notes, etc.

COP-31668 - Getting issue details... STATUS

Openstack SB Quality Hardening

COP-31666 - Getting issue details... STATUS

Reread the code written for Cinder southbound integration

COP-31669 - Getting issue details... STATUS

Inject Failures/Errors in SB Cinder Integration code and validate results

Design Approach / High Level Implementation Details

Approach

The quality hardening would be approached by doing multi step analysis in order to find issues and fix 

  1. Identify major use use-cases to be tested
  2. Analysis of unresolved and open jiras in all three areas - EE, CTRL and COP
  3. Derive various topology options to be tested, first priority is to cover currently deployed topology in current customer environments and latter priority would be to cover anything which seems critical/major not covered by customer deployments
  4. Thoroughly read the existing OpenStack southbound implementation ( code ) to figure out if there are issues
  5. Perform failure injection to ensure rollback cleans up everything that is intended to be cleaned

Analysis Results

Major Use Cases to be considered

Sr.NoUse CaseDescriptionComments
1Bulk volume create

Using service catalog perform bulk volume creation request, all volumes name starting with similar prefix.

Try in terms of 5 numbers, 10 numbers, 15 numbers, 20 numbers etc


2Bulk volume create

Using catalog service perform bulk volume creation request, all volumes name starting with different prefixes.

Try in terms of 5 numbers, 10 numbers, 15 numbers, 20 numbers etc

For e.g set of 5 with prefix as "FirstSet", next set of volumes with prefix as "SecondSet" and so on


3Bulk volume create and export

Using service catalog perform bulk volume creation request, all volumes name starting with similar prefix.

Try in terms of 5 numbers, 10 numbers, 15 numbers, 20 numbers etc


4Bulk volume create and export

Using catalog service perform bulk volume creation request, all volumes name starting with different prefixes.

Try in terms of 5 numbers, 10 numbers, 15 numbers, 20 numbers etc

For e.g set of 5 with prefix as "FirstSet", next set of volumes with prefix as "SecondSet" and so on


5Bulk volume deleteDelete multiple volumes in a single request
6Bulk volume un-export and deleteUn-export and delete multiple volumes in a single request
7Create and Delete in parallel

Volume "vol1" should have been created prior performing this case

Now, trigger the deletion of volume "vol1" and creation of volume "vol2" in parallel


8Bulk volume createPerform bulk volume creation with each volume having different size specified
9Create/Delete snapshot of multiple volumes in parallelPerform snapshot creation/deletion for multiple volumes in parallel
10Create/Delete full copy of multiple volumes in parallelPerform full-copy creation/deletion for multiple volumes in parallel
11Add initiator to host ( for standalone host and as well as for the host present in the cluster )

Perform volume export to a host

Add a new initiator and ensure that a new initiator can see already provisioned volume


12Remove initiator from host ( for standalone host and as well as for the host present in the cluster )

Have a host having at least 2 initiators

Perform volume export

Remove initiator and ensure that the initiator gets removed from the export group, export mask


13Add host to cluster

Have a cluster having at least 2 hosts

Perform volume export

Now add a new host to cluster

Ensure that newly added host sees already provisioned volumes to cluster


14Remove host from cluster

Have a clustering having at least 2 hosts

Perform volume export

Now remove one host from the cluster

Ensure that provisioned volumes are not visible to the removed host



JIRA Analysis

We have analysed COP – 10 , CTRL ~30 and EE jiras ~20. Major finding are as below 

Issues have fallen into following buckets, majorly

1)    Cinder.conf related – user misconfigured settings and engineering have to intervene to solve

2)    User attempted to perform unsupported use cases

3)    Few in the areas of rollback did not clean completely what was created by the operation 

None of the issues were DU/DL. 

Many of the customers tried POC and never went beyond that, unaware of the reasons for it. We may need to talk to folks who have worked with customers directly for POC implementations. 

Here are the queries used to find issues in EE,  COP and CTRL

CTRL - component = Controllers-Cinder AND status = Open AND resolution = Unresolved ORDER BY updatedDate DESC

https://asdjira.isus.emc.com:8443/browse/CTRL-8168?filter=-1&jql=project%20%3D%20CTRL%20AND%20component%20%3D%20Controllers-Cinder%20AND%20status%20%3D%20Open%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updatedDate%20DESC

COP - component = Cinder AND status = Open AND resolution = Unresolved ORDER BY updatedDate DESC 

Getting issues...  ( The query resulted both southbound and Enterprise cinder issues, picked up the subset related to southbound only )

https://asdjira.isus.emc.com:8443/issues/?filter=-1&jql=project%20%3D%20EE%20AND%20text%20~%20%22cinder%22%20ORDER%20BY%20updatedDate%20DESC

Exclusions and Limitations

      • Which requirements and/or functionality are deliberately excluded from this project?

Future Work and Related Projects

      • Is this change setting ground for a future project? In other words, are there projects that depend on tis change?
      • Does this change depend on other projects? 

Implementation Strategy

Implementation Timeframe

      • What are the milestones (code complete, code ready for review, code ready to commit)?
      • How much time will be needed to test this change and address all the issues after the initial commit?
      • Is this a multi-phase project?

Virtual Team

      • Who will work on this change?
      • Does the virtual team span multiple physical teams? If so, how are you going to share responsibilities and ensure consistency? 

Testing Strategy

      • How are you going to ensure good quality?
      • Does this change require any specific performance and scale testing?
      • Does this change require security scans?

Documentation Strategy

      • What are the necessary changes to CoprHD documentation?

Impact Evaluation

Public APIs and public functionality (end-user impact)

      • Is this change backward compatible? Could incompatibilities be avoided? If not, what changes will have customer and CoprHD QE apply to their automation software developed against the old APIs?

Other components and component interaction

      • If the change spans multiple components and/or changes interaction between components, can it create any new problems?
      • Does this change affect dependencies between services?
      • Does it change startup or shutdown order?
      • Does it require any new management interfaces?

Persistence model


      • Are schema changes need in dbsvc or coordinatorsvc?
      • If so, what is the upgrade procedure?Is the schema migration (conversion) going to be reliable and scalable?
      • Does this change affect disk usage?

Upgrades

      • Are there any other special considerations regarding upgrades? Think twice - this is rarely obvious (wink)

      • Will we need any new tools to debug or repair system if the upgrade code specific to this change fails?

Performance

      •  Can this change adversely affect performance? If so, how are you going to test it? Is there a way to test this early, before the entire implementation is ready?

Scalability and resource consumption

      • Will it scale? How long will essential operations take at the scale of O(10,000,000)? How are you going to test it?
      • Will specific performance at scale testing be required?
      • Does this change have impact on memory and CPU usage? How does memory and CPU usage scale with the number of objects?

Security

      • Are there any implications for CoprHD security?
      • Will new security scans be required?

Deployment and serviceability

      • Will any error messages associated with this feature include "friendly" names for the elements involved, rather than just opaque URIs?

Developer impact, dependencies and conditional inclusion

      • What is the impact on build time?
      • Is there any impact on integration with IDEs (Eclipse and IntelliJ)?
      • Are there any new third party dependencies? If so, please list these dependencies here:
Package NamePackage VersionLicenseURLDescription of the package and what it is used for















      • Shall this feature by included with a special build-time flag?
      • Shall this feature be enabled only under certain run-time condition?

Reviewed and Approved By

NameDateVoteComments, remarks, etc.

































    • The members of the CoprHD Technical Steering Committee (TSC) have been automatically added to your reviewer list
      • A majority of them must approve this document before the feature can be merged to a project integration branch (e.g. integration-*, release-*, master)
      • Please un-check the boxes next to their names when your document is ready for their review
    • Please refer to Design Approval Guidance for information on additional reviewers who should be added and the approval process
  • No labels