In ICN, we have ONAP4k8s as the service orchestration and Kubernetes as the resource orchestration. In the edge deployment, there will be multiple end-users sharing the same edge compute resource. The challenges are to isolate the end-users deployment and allocate the resource as per their demand and quota. This proposal addresses these challenges by creating a "Logical cloud" for the set of users, and provide logical isolation and resource quota.
Focusing on the solution within service orchestration
Working in Kubernetes core or API is clearly out of the scope of these documents. There are the solutions available to provide a separate control plane to each tenant in a cluster. But the creation of tenant within a cluster does not address the shared clusters and tenant creation should be at service orchestration instead of resource orchestration
Outline
In this section, we define Logical cloud in general for the Service Orchestration engine. A Logical cloud can be defined as a group of resources bounded and isolated amount of compute, storage, networking and control plane in a kubernetes cluster. A Logical cloud can also be defined as a group of users slicing a bounded resource allocated for them. These resources can be as follows:
Objectives:
DCM Flow:
Logical cloud creation(With default resource quota & users)
The following steps explain how to run the tenant controller in kubernetes
URL: /v2/projects/<project-name>/logical-clouds POST BODY: { "name": "logical-cloud-1", //unique name for the new logical cloud "description": "logical cloud for walmart finance department", //description for the logical cloud "cluster-labels": "abc,xyz", "resources": { "cpu": "400", "memory": "1000Gi", "pods": "500", "dummy/dummyResource": 100, }, "user" : [{ "name" : "user-1", //name of user for this cloud "type" : "certificate", //type of authentication credentials used by user (certificate, APIKey, UNPW) "certificate" : "/path/to/user1/logical cloud-1-user1.csr" , //Path to user certificate "permissions" : { "apiGroups" : ["stable.example.com"] "resources" : ["secrets", "pods"] "verbs" : ["get", "watch", "list", "create"] }, "quota" : { "cpu": "100", "memory": "500Gi", "pods": "100", "dummy/dummyResource": 20 }] } } curl -d @create_logical_cloud-1.json http://onap4k8s:<multicloud-k8s_NODE_PORT>/v2/projects/<project-name>/logical-clouds \ --key ./logical cloud-t1-admin-key.pem \ --cert ./logical cloud-t1-admin.pem \ Return Status: 201 Return Body: { "name" : "logical-cloud-1" "user" : "user-1" "Message" : "logical cloud and associated user successfully created" } |
Logical cloud admin key and certificate should be created by Logical cloud admin(the one who create the curl command). Authentication is required for the curl command. DCM should have the Admin logical cloud information to authenticate the curl command. Unauthorized users can't create the logical cloud.
This information should be created before creating logical cloud and inserted in the logical cloud creation
User controller does the following steps to bind the user certificate with the cluster using the cluster-labels : abc and xyz. Itohan Ukponmwan Please get the GET URL from HPA controller to get the cluster list with cluster labels
DCM queries HPA controller the list of cluster having cluster-labels abc and gets the cluster list c1 and c2
Each cluster(C1, C2) has the Kubernetes cluster certificate cluster (CA – c1-ca.crt & c1-ca.key), generate the final certificate logical cloud-1-user1.crt by using logical cloud-1-user1.csr (do the same for the cluster c2). user controller does the following steps once the logical cloud curl command is post through grpc with goclient API
$ openssl x509 -req -in logical cloud-1-user1.csr -CA CA_LOCATION/c1-ca.crt -CAkey CA_LOCATION/c1-ca.key -CAcreateserial -out logical cloud-1-user1-c1.crt -days 500
$ kubectl –kubeconfig=/path/to/c1/kubeconfig config set-credentials user-1 --client-certificate=./ logical cloud-1-user1-c1.crt --client-key=./logical cloud-1-user-1.key
The following steps explain how to run the tenant controller in kubernetes
GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/users RETURN STATUS: 200 RETURN BODY: { users" : [{ "name" : "user-1", //name of user for this cloud "type" : "certificate", //type of authentication credentials used by user (certificate, APIKey, UNPW) "certificate" : "/path/to/user1/logical cloud-1-user1.csr" , //Path to user certificate "permissions" : { "apiGroups" : ["stable.example.com"] "resources" : ["secrets", "pods"] "verbs" : ["get", "watch", "list", "create"] }, "quota" : { "cpu": "100", "memory": "500Gi", "pods": "100", "dummy/dummyResource": 20 } }, { "name" : "user-2", //name of user for this cloud "type" : "certificate", //type of authentication credentials used by user (certificate, APIKey, UNPW) "certificate" : "/path/to/user2/logical cloud-1-user1.csr" , //Path to user certificate "permissions" : { "apiGroups" : ["stable.example.com"] "resources" : ["secrets", "pods"] "verbs" : ["get", "watch", "list", "create"] }, "quota" : { "cpu": "100", "memory": "500Gi", "pods": "100", "dummy/dummyResource": 20 } } ] DELETE URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/users URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/user/<user-name> RETURN STATUS: 204 |
DCM queries the namespace controller through grpc to create namespace "logical cloud-1-ns" for the cluster with cluster labels abc and xyz. Namespace controller does the the following steps, to create the namespace and set the user with namespace through grpc with goclient API
$ kubectl create namespace logical cloud-1-ns --kubeconfig=/path/to/c1/kubeconfig
$ kubectl config set-context logical-cloud-1-user-1-context --cluster=c1 --namespace= logical cloud-1-ns --user=user1 --kubeconfig=/path/to/c1/kubeconfig
The following steps explain how to run the tenant controller in kubernetes
GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/namespaces RETURN STATUS: 200 RETURN BODY: { "clusters": {c1, c2} namespaces" : { "name" : "logical cloud-1-ns", //name of namespace for the logical cloud } } DELETE URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/namespaces RETURN STATUS: 204 |
DCM Database is based on Mongo DB.
URL: /v2/projects/<project-name>/logical-clouds/control-plane POST BODY: { "name": "logical-cloud-1", //unique name for the new logical cloud "namespace": "Logical-cloud-1-istio-system", "ca-cert": "/path/to/ca-cert.pem", "ca-key": "/path/to/ca-key.pem", "root-cert": "/path/to/root-cert.pem", "cert-chain" "/path/to/cert-chain.pem" } curl -d @create_logical_cloud-1-user-2.json http://onap4k8s:<multicloud-k8s_NODE_PORT>/v2/projects/<project-name>/logical-clouds/control-plane \ --key ./logical cloud-t1-admin-key.pem \ --cert ./logical cloud-t1-admin.pem \ Return Status: 201 Return Body: { "name" : "logical-cloud-1" "Message" : "logical cloud 1 control plane is successfully created" } GET URL: /v2/projects/<project-name>/logical-clouds/<Logical-cloud-name>/control-planes RETURN STATUS: 200 RETURN BODY: { "name" : "logical-cloud-1", //name of namespace for the logical cloud "gateways" : "istio-egressgateway", "dns": "istiocoredns", "clusters": {c1, c2} } DELETE URL: /v2/projects/<project-name>/logical-clouds/<Logical-cloud-name>/control-planes RETURN STATUS: 204 |
Adding new users in existing Logical cloud 1
URL: /v2/projects/<project-name>/logical-clouds<logical-cloud-name>/users POST BODY: { "user" : { "name" : "user-2", //name of user for this cloud "type" : "certificate", //type of authentication credentials used by user (certificate, APIKey, UNPW) "certificate" : "/path/to/user2/logical cloud-1-user2.csr" , //Path to user certificate "permissions" : { "apiGroups" : ["stable.example.com"] "resources" : ["secrets", "pods"] "verbs" : ["get", "watch", "list", "create"] }, "quota" : { "cpu": "200", "memory": "300Gi", "pods": "200", "dummy/dummyResource": 30, } } } curl -d @create_logical_cloud-1-user-2.json http://onap4k8s:<multicloud-k8s_NODE_PORT>/v2/projects/<project-name>/logical-clouds \ --key ./logical cloud-t1-admin-key.pem \ --cert ./logical cloud-t1-admin.pem \ Return Status: 201 Return Body: { "name" : "logical-cloud-1" "user" : "user-2" "Message" : "logical cloud and associated user successfully created" } |
This feature allows the logical cloud to tune their resources.
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/quotas POST BODY: { "cluster-labels": "abc, xyz", "resources": { "cpu": "400", "memory": "1000Gi", "pods": "500", "dummy/dummyResource": 100, } } curl -d @create_logical_cloud-1.json http://onap4k8s:<multicloud-k8s_NODE_PORT>/v2/projects/<project-name>/logical-clouds \ --key ./logical cloud-t1-admin-key.pem \ --cert ./logical cloud-t1-admin.pem \ Return Status: 201 Return Body: { "name" : "logical-cloud-1" "Message" : "logical cloud 1 is successfully tuned" } GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/quotas RETURN STATUS: 200 RETURN BODY: { "resources": { "cpu": "400", "memory": "1000Gi", "pods": "500", "dummy/dummyResource": 100, } } DELETE URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/quotas RETURN STATUS: 204 |
The following steps explain to get the cluster labels
GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/cluster-labels RETURN STATUS: 200 RETURN BODY: [{ "cluster": c1 "labels" : {abc,xyz,ijk,dfg} }, { "cluster": c2 "labels" : {abc,xyz,irk,iop} } }] |
DCM merge the kube config of each cluster list c1 and c2.
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kubeconfig GET Return Status: 201 Return Body : { apiVersion: v1 clusters: - cluster: certificate-authority: path/to/my/cafile server: http://2.2.2.2:6443 name: cluster-abc - cluster: certificate-authority: path/to/my/cafile server: https://1.1.1.1:6443 name: cluster-xyz contexts: - context: cluster: kubernetes namespace: ns-1 user: user-1 name: logical-cloud-1 current-context: logical-cloud-1 kind: Config preferences: {} users: - name: user-1 user: client-certificate: path/to/my/client/cert client-key: path/to/my/client/key } |
Kubernetes Multi-Tenancy Draft Proposal
Tenant Concept in Kubernetes
Kubernetes Tenant CRD
K8s Multi-tenancy WG Plan