RESTful API Binary Provisioning Agent (BPA)

This is a Golang application providing a RESTful API to interact with and upload image objects.

The API application source files are in the icn/cmd/bpa-restapi-agent directory.

While the database back-end is extensible, this initial release requires mongodb.

Motivation:

The ICN RESTful API intends to provide access to resources inside the system using a uniform interface through the provision of one logical URI. As a constriant, "Servers and clients may also be replaced and developed independently, as long as the interface between them is not altered."

Requirements:

This RESTful API service will expose the following resources to a user:

Binary Images
Container Images
OS Images

Design Proposal:

Use nouns to describe resources
In the top URL, identify version, bare-metal cluster name, and resource
GET - This will be used to list image resources
POST - This will be used to create image resources
PUT - This will be used to upload image resources
DELETE - This will be used to delete image resources

Implementation:

Sample GET Implementation:

The sample API above was created using the swagger.yaml below.

swagger.yml

swagger: "2.0"
  info:
  description: "Addresses deployment of workloads in the edge"
  version: "1.0.0"
  title: "Integrated Cloud Native RESTful API"
schemes:
- "http"
consumes:
- "application/json"
produces:
- "application/json"
paths:
  /v1/baremetalcluster/{clustername}/images/:
     get:
         tags:
         - "Deployment of Images"
        summary: "List all Images."
        description: "Endpoint to list all Images."
        produces:
        - "application/json"
        parameters:
        - name: "clustername"
           in: "path"
          description: "Name of the cluster used to query"
          required: true
          type: "string"
       responses:
          200:
              description: "successful operation"
              schema:
                 $ref: "#/definitions/GETResponse"
         default:
             description: generic error response
             schema:
                $ref: "#/definitions/error"
/v1/baremetalcluster/{clustername}/images/{name}:
    get:
     tags:
     - "Deployment of Images"
     summary: "Get details of an image."
     description: "Endpoint to get details of ICN available images."
     produces:
     - "application/json"
     parameters:
     - name: "clustername"
        in: "path"
        description: "Name of the cluster used to query"
        required: true
        type: "string"
     - name: "name"
        in: "path"
        description: "Name used to query"
        required: true
        type: "string"
     responses:
        200:
            description: "successful operation"
            schema:
               $ref: "#/definitions/GETResponse"
        default:
            description: generic error response
            schema:
               $ref: "#/definitions/error"
definitions:
    GETResponse:
         type: "object"
         properties:
             ID:
                 type: "string"
             image_id:
                 type: "string"
             repo:
                type: "string"
             tag:
                type: "string"
             description:
                 type: "string"
                 minLength: 1

    error:
       type: "object"
       required:
       - "message"
       properties:
           code:
               type: "integer"
               format: "int64"
           message:
           type: "string"

Usage:

Proposed sample command to GET all binary images

# Get all binary images
curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/binary_images/

# Get one binary image
curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/binary_images/{name}

Proposed sample command to GET all container images

# Get all container images curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/container_images/ # Get one container image curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/container_images/{name}

Proposed sample command to GET all OS images

# Get all container images curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/os_images/ # Get one container image curl -i http://localhost:9015/v1/baremetalcluster/{clustername}/os_images/{name}

How To Use The BPA RESTful API Service

Install and start mongodb. For instructions: https://docs.mongodb.com/manual/installation/

git clone "https://gerrit.akraino.org/r/icn"
cd icn/cmd/bpa-restapi-agent

Run the application
go run main.go

Sample output without a config file:

2019/08/22 14:08:41 Error loading config file. Using defaults
2019/08/22 14:08:41 Starting Integrated Cloud Native API

# Cloud Storage with MinIO

Start MinIO server daemon with docker command before run REST API agent, default settings in config/config.go.
AccessKeyID: ICN-ACCESSKEYID
SecretAccessKey: ICN-SECRETACCESSKEY
MinIO Port: 9000

```
$ docker run -p 9000:9000 --name minio1 \
-v /mnt/data:/data \
-v /mnt/config:/root/.minio \
minio/minio server /data
```

MinIO Client will automatic initialize in main.go, and create 3 buckets: binary, container, operatingsystem.
The Upload image will "PUT" to corresponding buckets by HTTP PATCH request url.
You can also check by open browser: http://127.0.0.1:9000/

RESTful API usage examples

Sample POST request format

curl -i -F "metadata=<jsonfile;type=application/json" -F file=@/home/<username>/<dir>/jsonfile -X POST http://NODE_IP:9015//baremetalcluster/{owner}/{clustername}/<image_type>

#image type can be binary_image, container_image, or os_image

Example requests and responses:

Create image - POST

#Using a json file called sample.json
#image_length in sample.json can be determined with the command
ls -al <image_file>

Request

curl -i -F "metadata=<sample.json;type=application/json" -F file=@/home/enyi/workspace/icn/cmd/bpa-restapi-agent/sample.json -X POST http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images

Response

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Content-Type: application/json
Date: Thu, 22 Aug 2019 22:56:16 GMT
Content-Length: 239

{"owner":"alpha","cluster_name":"beta","type":"container","image_name":"asdf246","image_offset":0,"image_length":29718177,"upload_complete":false,"description":{"image_records":[{"image_record_name":"iuysdi1234","repo":"icn","tag":"1"}]}}

#this creates a database entry for the image, and an empty file in the file system

List image - GET

curl -i -X GET http://localhost:9015/v1/baremetalcluster/{owner}/{clustername}/<image_type>/{imgname}

example:
#continuing with our container image from above

Request

curl -i -X GET http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246

Response

HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 22 Aug 2019 22:57:10 GMT
Content-Length: 239

{"owner":"alpha","cluster_name":"beta","type":"container","image_name":"asdf246","image_offset":0,"image_length":29718177,"upload_complete":false,"description":{"image_records":[{"image_record_name":"iuysdi1234","repo":"icn","tag":"1"}]}}

Upload container image - PATCH

Request

curl --request PATCH --data-binary "@/home/enyi/workspace/icn/cmd/bpa-restapi-agent/sample_image" http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246 --header "Upload-Offset: 0" --header "Expect:" -i

Response

HTTP/1.1 204 No Content
Upload-Offset: 29718177
Date: Thu, 22 Aug 2019 23:19:44 GMT

Check uploaded image - GET

Request

curl -i -X GET http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246

Response

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 23 Aug 2019 17:12:07 GMT
Content-Length: 245

{"owner":"alpha","cluster_name":"beta","type":"container","image_name":"asdf246","image_offset":29718177,"image_length":29718177,"upload_complete":true,"description":{"image_records":[{"image_record_name":"iuysdi1234","repo":"icn","tag":"1"}]}}

#after the upload, the image_offset is now the same as image_length and upload_complete changed to true
#if upload was incomplete

Resumable upload instructions

Resumable upload -PATCH

#this is the current resumable upload mechanism

Request

curl --request PATCH --data-binary "@/home/enyi/workspace/icn/cmd/bpa-restapi-agent/sample_image" http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246 --header "Upload-Offset: 0" --header "Expect:" -i --limit-rate 200K

#the above request limits transfer for testing purposes
#'ctl c' out after a few seconds, to stop file transfer
#check image_offset with a GET

Check upload - GET

Request

curl -i -X GET http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246

Response

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 24 Aug 2019 00:30:00 GMT
Content-Length: 245

{"owner":"alpha","cluster_name":"beta","type":"container","image_name":"asdf246","image_offset":4079616,"image_length":29718177,"upload_complete":false,"description":{"image_records":[{"image_record_name":"iuysdi1234","repo":"icn","tag":"2"}]}}

#from our response you can see that image_offset is still less than image_length and #upload_complete is still false
#next we use the dd command (no limiting this time)

Request

dd if=/home/enyi/workspace/icn/cmd/bpa-restapi-agent/sample_image skip=4079616 bs=1 | curl --request PATCH --data-binary @- http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246 --header "Upload-Offset: 4079616" --header "Expect:" -i

#the request skips already uploaded 4079616 bytes of data

Response

25638561+0 records in
25638561+0 records out
25638561 bytes (26 MB, 24 MiB) copied, 207.954 s, 123 kB/s
HTTP/1.1 204 No Content
Upload-Offset: 29718177
Date: Sat, 24 Aug 2019 00:43:18 GMT

Update image description - PUT

# let's change the tag in description from 1 to latest
# once the change is made in sample.json (or your json file)

Request

curl -i -F "metadata=<sample.json;type=application/json" -F file=@/home/enyi/workspace/icn/cmd/bpa-restapi-agent/sample.json -X PUT http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246

Response

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Content-Type: application/json
Date: Fri, 23 Aug 2019 17:21:01 GMT
Content-Length: 239

{"owner":"alpha","cluster_name":"beta","type":"container","image_name":"asdf246","image_offset":0,"image_length":29718177,"upload_complete":false,"description":{"image_records":[{"image_record_name":"iuysdi1234","repo":"icn","tag":"2"}]}}

Delete an image - DELETE

Request

curl -i -X DELETE http://localhost:9015/v1/baremetalcluster/alpha/beta/container_images/asdf246

How To Write ICN Unit Tests in Go using Testify

Go version used is go1.12.9

To install Go, follow the instructions on the official site, (https://golang.org/doc/install).

Testify documentation can be found on Github, https://github.com/stretchr/testify

The first question is, "what to test?" In this case, I'll test a method called GetDirPath.

GetDirPath uses the Go core packages user and path to return a file path.

func (v *ImageClient) GetDirPath(imageName string) (string, string, error) {

	home, err := v.GetHomeDir()
	dirPath := path.Join(home, "images", v.storeName)
	filePath := path.Join(dirPath, imageName)

	return filePath, dirPath, err
}

To Do - Explain

To test, I'll create the test file image_test.go. A test for the GetDirPath could look like this:

To Do - Explain

References:

https://restfulapi.net/rest-architectural-constraints/

Space shortcuts

Page tree

Motivation:

Requirements:

Design Proposal:

Implementation:

How To Use The BPA RESTful API Service

RESTful API usage examples

Sample POST request format

Create image - POST

Request

Response

List image - GET

Request

Response

Upload container image - PATCH

Request

Response

Check uploaded image - GET

Request

Response

Resumable upload instructions

Resumable upload -PATCH

Request

Check upload - GET

Request

Response

Request

Response

Update image description - PUT

Request

Response

Delete an image - DELETE

Request

How To Write ICN Unit Tests in Go using Testify