Introduction

FATE-Flow is the job scheduling system of the federated learning framework FATE, which realizes the complete management of the federated learning job life cycle, including data input, training job scheduling, indicator tracking, model center and other functions. It provides CLI and HTTP API for users to manage data and federated jobs.

FATE-Flow REST API

• HTTP Method: POST

• Content-Type: application/json

DataAccess

/v1/data/upload

• request structure

  • namespace: Required,String: upload data table namespace

  • table_name: Required,String: upload data table name

  • work_mode: Required,Integer: eggroll’s working mode

  • file: Required, String: upload file location

  • head: Required,Integer: determine if there is a data header

  • partition: Required,Integer: set the number of partitions to save data

  • module: Optional,String: If you need to use the data of the machine where the FATE-Flow server is located, this value is not empty.

  • use_local_data: Optional,String: If you need to use the data of the machine where the FATE-Flow server is located, this value is 0.

  • drop: Optional, Integer: When the cluster deployment uses the same table to upload data, it is necessary to carry the drop parameter,0 represents overwriting upload, 1 represents deleting the previous data and re-uploading

• response structure

  • job_id: upload job id,String

  • data: return data for submitting job ,Object

/v1/data/download

• request structure

  • namespace: Required,String: download data table namespace

  • table_name: Required,String: download data table name

  • output_path: Required, String: download file location

  • work_mode: Required,Integer:working mode

  • delimitor: Optional,String: download data delimitor

• response structure

  • job_id: download job id,String

  • data: return data for submitting job ,Object

/v1/data/upload/history

• request structure

  • job_id: Optional,String:download job id

  • limit: Optional, Integer:Limit quantity

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: return data for submitting job ,Object

Job

/v1/job/submit

• request structure

  • job_runtime_conf: Required,Object: configuration information for the currently submitted job

  • job_dsl: Required,Object: dsl of the currently submitted job

• response structure

  • job_id: job id of the currently submitted job,String

  • data: return data for submitting job ,Object

/v1/job/stop

• request structure

  • job_id: Required, String: job id

• response structure

  • job_id: job id of the currently submitted job,String

  • retmsg: return code description,String

/v1/job/query

• request structure

  • job_id: Optional,String: job id

  • name: Optional,String: job name

  • description: Optional,String: job description

  • tag: Optional,String:Optional,String: job tag

  • role: Optional,String: job role

  • party_id: Optional,String: job party id

  • roles: Optional,String: job roles

  • initiator_party_id: Optional,String: initiator’s party id

  • is_initiator: Optional,Integer: mark if it is the initiator

  • dsl: Optional,String: job dsl

  • runtime_conf : Optional,String: configuration information for the job

  • run_ip: Optional,String: job run ip

  • status: Optional,String: job status

  • current_steps: Optional,String:record component id in DSL

  • current_tasks: Optional,String: record task id

  • progress: Optional,Integer: job progress

  • create_time: Optional,Integer: job create time

  • update_time: Optional,Integer:job update time

  • start_time: Optional,Integer: job start time

  • end_time: Optional,Integer: job end time

  • elapsed: Optional,Integer: job elapsed time

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: job data, Array

/v1/job/update

• request structure

  • job_id: Required,String: job id

  • role: Required,String: job role

  • party_id: Required,String: job party id

  • notes: Required, String: remark Information

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

/v1/job/config

• request structure

  • job_id: Optional,String: job id

  • name: Optional,String: job name

  • description: Optional,String: job description

  • tag: Optional,String:Optional,String: job tag

  • role: Optional,String: job role

  • party_id: Optional,String: job party id

  • roles: Optional,String: job roles

  • initiator_party_id: Optional,String: initiator’s party id

  • is_initiator: Optional,Integer: mark if it is the initiator

  • dsl: Optional,String: job dsl

  • runtime_conf : Optional,String: configuration information for the job

  • run_ip: Optional,String: job run ip

  • status: Optional,String: job status

  • current_steps: Optional,String:record component id in DSL

  • current_tasks: Optional,String: record task id

  • progress: Optional,Integer: job progress

  • create_time: Optional,Integer: job create time

  • update_time: Optional,Integer:job update time

  • start_time: Optional,Integer: job start time

  • end_time: Optional,Integer: job end time

  • elapsed: Optional,Integer: job elapsed time

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: config data, Object

/v1/job/task/query

• request structure

  • job_id: Optional,String: job id

  • name: Optional,String: job name

  • description: Optional,String: job description

  • tag: Optional,String:Optional,String: job tag

  • role: Optional,String: job role

  • party_id: Optional,String: job party id

  • roles: Optional,String: job roles

  • initiator_party_id: Optional,String: initiator’s party id

  • is_initiator: Optional,Integer: mark if it is the initiator

  • dsl: Optional,String: job dsl

  • runtime_conf : Optional,String: configuration information for the job

  • run_ip: Optional,String: job run ip

  • status: Optional,String: job status

  • current_steps: Optional,String:record component id in DSL

  • current_tasks: Optional,String: record task id

  • progress: Optional,Integer: job progress

  • create_time: Optional,Integer: job create time

  • update_time: Optional,Integer:job update time

  • start_time: Optional,Integer: job start time

  • end_time: Optional,Integer: job end time

  • elapsed: Optional,Integer: job elapsed time

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: tasks data, Array

Tracking

/v1/tracking/job/data_view

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: job view data,Object

/v1/tracking/component/metric/all

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer

  • component_name: Required,String: conponent name

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: all metric data,Object

/v1/tracking/component/metrics

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer

  • component_name: Required,String: component name

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: metrics data,Object

/v1/tracking/component/metric_data

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

  • component_name: Required,String: component name

  • meric_name: Required,String: metric name

  • metric_namespace: Required,String: metric namespace

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: metric data, Array

  • meta: metric meta, Object

/v1/tracking/component/parameters

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

  • component_name: Required,String: component name

• response structure

  • retcode:return code,Integer

  • retmsg: return code description,String

  • data: output parameters, Object

/v1/tracking/component/output/model

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

  • component_name: Required,String: component name

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: output model, Object

  • meta: component model meta,Object

/v1/tracking/component/output/data

• request structure

  • job_id: Required,String: job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

  • component_name: Required,String: component name

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: output data, Array

  • meta: schema header information, Object

Pipeline

/v1/pipeline/dag/dependency

• request structure

  • job_id: Required,String:job id

  • role: Required,String: role information

  • party_id: Required,Integer: party id

• response structure

  • retcode: return code,Integer

  • retmsg: return code description,String

  • data: pipeline dag dependency data,Object

Model

/v1/model/load

• request structure

  • initiator: Required,Object: job initiator information, including party_id and role

  • job_parameters: Required,Object: job parameters information, including work_mode, model_id and model_version

  • role: Required,Object: role information of the parties

  • servings: Optional,Array: fate serving address and port

• response structure

  • job_id:job id, String

  • retcode: return code, Integer

  • retmsg: return code description, String

  • data: status info, Object

/v1/model/bind

• request structure

  • service_id: Required,String: service id

  • initiator: Required,Object: job initiator information, including party_id and role

  • job_parameters: Required,Object: job parameters information, including work_mode, model_id and model_version

  • role: Required,Object: role information of the parties

  • servings: Optional,Array: fate serving address and port

• response structure

  • retcode: return code, Integer

/v1/model/transfer

• request structure

  • name: Requied,String: model version

  • namespace: Requied,String: model id

• response structure

  • retcode: return code, Integer

  • retmsg: return code description, String

  • data: model data, Object

Table

/v1/table/table_info

• request structure

  • create: Optional, Boolean: whether to create

  • namespace: Optional,String: download data table namespace, need to be used with table_name

  • table_name: Optional,String: download data table name, need to be used with namespace

  • local: Optional,Object: local configuration

  • role: Optional,Object: role information

  • data_type: Optional,String: download file data type

  • gen_table_info: Optional,Boolean: tag table information

• response structure

  • retcode: return code, Integer

  • retmsg: return code description, String

  • data: table information

/v1/table/delete

• request structure

  • namespace: Optional,String: download data table namespace, need to be used with table_name

  • table_name: Optional,String: download data table name, need to be used with namespace

• response structure

  • retcode: return code, Integer

  • retmsg: return code description, String

  • data: table information

FATE-Flow Client Command Line Interface

Usage

python fate_flow_client.py -f $command

JOB_OPERATE

submit_job

• description: submit a pipeline job

• parameter:

  • -c -config: runtime conf path, Required

  • -d -dsl: dsl path, Required

  python fate_flow_client.py -f submit_job -c examples/test_hetero_lr_job_conf.json -d examples/test_hetero_lr_job_dsl.json
  

stop_job

• description: cancel job or stop job

• parameter:

  • -j -job_id: job id, Required

  python fate_flow_client.py -f stop_job -j $job_id
  

query_job

• description: query job information by filters

• parameter:

  • -j -job_id: filter by job id, Optional

  • -r -role : filter by role, Optional

  • -p -party_id: filter by party id, Optional

  • -s -status: filter by status, Optional

  python fate_flow_client.py -f query_job -j $job_id
  

clean_job

• description: clean processor,data table and metric data

• parameter:

  • -j -job_id: filter by job id, Optional

  • -r -role : filter by role, Optional

  • -p -party_id: filter by party id, Optional

  • -cpn -component_name: component name, Optional

  python fate_flow_client.py -f clean_job -j $job_id
  

data_view_query

• description: query data view information by filters

• parameter:j -job_id: filter by job id, Optionalr -role : filter by role, Optionalp -party_id: filter by party id, Optionals -status: filter by status, Optional

python fate_flow_client.py -f data_view_query -j $job_id

JOB

job_config

• description: download the configuration of this job

• parameter:

  • -j -job_id: job id, Required

  • -r -role : role, Required

  • -p -party_id: party id, Required

  • -o -output_path: config output directory path, Required

  python fate_flow_client.py -f job_config -j $job_id -r $role -p $party_id -o $output_path
  

job_log

• description: download the log of this job

• parameter:

  • -j -job_id: job id, Required

  • -o -output_path: config output directory path, Required

  python fate_flow_client.py -f job_log -j $job_id -o $output_path
  

TASK_OPERATE

query_task

• description: query task information by filters

• parameter:

  • -j -job_id: filter by job id, Optional

  • -cpn -component_name: filter by component name, Optional

  • -r -role : filter by role, Optional

  • -p -party_id: filter by party id, Optional

  • -s -status: filter by status, Optional

  python fate_flow_client.py -f query_task -j $job_id
  

TRACKING

component_parameters

• description: query the parameters of this component

• parameter:

  • -j -job_id: job id, Required

  • -cpn -component_name: component name, Required

  • -r -role: role, Required

  • -p -party_id: party id, Required

  python fate_flow_client.py -f component_parameters -j $job_id -r $role -p $party_id -cpn $component_name
  

component_metric_all

• description: query all metric data

• parameter:

  • -j –job_id: job id, Required

  • -cpn –component_name: component name, Required

  • -r –role: role, Required

  • -p –party_id: party id, Required

  python fate_flow_client.py -f component_metric_all -j $job_id -r $role -p $party_id -cpn $component_name
  

component_metrics

• description: query the list of metrics

• parameter:

  • -j –job_id: job id, Required

  • -cpn –component_name: component name, Required

  • -r –role: role, Required

  • -p –party_id: party id, Required

  python fate_flow_client.py -f component_metrics -j $job_id -r $role -p $party_id -cpn $component_name
  

component_output_model

• description: query this component model

• parameter:

  • -j –job_id: job id, Required

  • -cpn –component_name: component name, Required

  • -r –role: role, Required

  • -p –party_id: party id, Required

  python fate_flow_client.py -f component_output_model -j $job_id -r $role -p $party_id -cpn $component_name
  

component_output_data

• description: download the output data of this component

• parameter:

  • -j –job_id: job id, Required

  • -cpn –component_name: component name, Required

  • -r –role: role, Required

  • -p –party_id: party id, Required

  • -o –output_path: config output path, Required

  • -limit –limit: Limit quantity, Optional

  python fate_flow_client.py -f component_output_data -j $job_id -r $role -p $party_id -cpn $component_name -o $output_path
  

component_output_data_table

• description: view table name and namespace

• parameter:

  • -j –ob_id: job id, Required

  • -cpn –component_name: component name, Required

  • -r –role: role, Required

  • -p –party_id: party id, Required

  python fate_flow_client.py -f component_output_data_table -j $job_id -r $role -p $party_id -cpn $component_name
  

DATA

download

• description: download table

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f download -c examples/download_guest.json
  

upload

• description: upload table

• parameter:

  • -c –config: config path, Required

  • -drop –drop: Operations before file upload, Optional

  python fate_flow_client.py -f upload -c examples/upload_guest.json
  python fate_flow_client.py -f upload -c examples/upload_guest.json -drop 0(or1)
  

upload_history

• description: upload table history

• parameter:

  • -j –job_id: job id, Optional

  • -limit –limit: Limit quantity, Optional

  python fate_flow_client.py -f upload_history -j $job_id
  python fate_flow_client.py -f upload_history -limit 5
  

Table

table_info

• description: query table information

• parameter:

  • -n –namespace: namespace, Required

  • -t –table_name: table name, Required

  python fate_flow_client.py -f table_info -n $namespace -t $table_name
  

table_delete

• description: delete table

• parameter:

  • -n –namespace: namespace, Optional

  • -t –table_name: table name, Optional

  • -j –job_id: job id, Optional

  • -cpn –component_name: component name, Optional

  • -r –role: role, Optional

  • -p –party_id: party id, Optional

  python fate_flow_client.py -f table_delete -n $namespace -t $table_name
  python fate_flow_client.py -f table_delete -j $job_id
  

Model

load

• description: load model.

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f load -c $conf_path
  

• response example:

  {
      "data": {
          "detail": {
              "guest": {
                  "9999": {
                      "retcode": 0,
                      "retmsg": "xxx"
                  }
              },
              "host": {
                  "10000": {
                      "retcode": 0,
                      "retmsg": "xxx"
                  }
              }
          },
          "guest": {
              "9999": 0
          },
          "host": {
              "10000": 0
          }
      },
      "jobId": "xxxxxxxxxxxxxxxx",
      "retcode": 0,
      "retmsg": "success"
  }
  

bind

• description: bind model.

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f bind -c $conf_path
  

• response example:

  {
      "retcode": 0,
      "retmsg": "service id is xxx"
  }
  

store

• description: store model

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f store -c $conf_path
  

restore

• description: restore mode

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f restore -c $conf_path
  

export

• description: export model

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f export -c $conf_path
  

import

• description: import model

• parameter:

  • -c –config: config path, Required

  python fate_flow_client.py -f import -c $conf_path

Next  Previous