Introduction

All test results and logs have been pushed to Akraino NEXUS:

https://nexus.akraino.org/content/sites/logs/fate/job/Fate_test/15/

Running Unit tests

A script to run all the unittests has been provided in ./python/federatedml/test folder.

Once FATE is installed, tests can be run using:

sh ./python/federatedml/test/run_test.sh

All the unit tests shall pass if FATE is installed properly.


Pipeline Examples


Introduction

We provide some example scripts of running FATE jobs with FATE-Pipeline.

Please refer to the document linked above for details on FATE-Pipeline and FATE-Flow CLI v2. DSL version of provided Pipeline examples can be found here.


Quick Start

Here is a general guide to quick start a FATE job.

  1. (optional) create virtual env

    python -m venv venv
source venv/bin/activate
pip install -U pip

  2. install fate_client

    # this step installs FATE-Pipeline, FATE-Flow CLI v2, and FATE-Flow SDK
pip install fate_client
pipeline init --help

  3. configure server information

    # configure by conf file
pipeline init -c pipeline/config.yaml
# alternatively, input real ip address and port info to initialize pipeline
# optionally, set log directory for Pipeline
pipeline init --ip 127.0.0.1 --port 9380 --log-directory ./logs

  4. upload data with FATE-Pipeline

    #  upload demo data to FATE data storage, optionally provide path to where deployed examples/data locates

python demo/pipeline-upload.py --base /data/projects/fate

    If upload job is invoked correctly, job id will be printed to terminal and an upload bar is shown. If FATE-Board is available, job progress can be monitored on Board as well.

    UPLOADING:||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.00%
2020-11-02 15:37:01.030 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215370091210977
Job is still waiting, time elapse: 0:00:01
Running component upload_0, time elapse: 0:00:09
2020-11-02 15:37:13.410 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215370091210977

  1. run a FATE-Pipeline fit job

    python demo/pipeline-quick-demo.py

    This quick demo shows how to build to a heterogeneous SecureBoost job. Progress of job execution will be printed as modules run. A message indicating final status ("success") will be printed when job finishes. The script queries final model information when model training completes.

    2020-11-02 10:45:29.875 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110210452959882932
Job is still waiting, time elapse: 0:00:01
Running component reader_0, time elapse: 0:00:07
Running component dataio_0, time elapse: 0:00:10
Running component intersection_0, time elapse: 0:00:14
Running component hetero_secureboost_0, time elapse: 0:00:46
Running component evaluation_0, time elapse: 0:00:50
2020-11-02 10:46:21.889 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110210452959882932
2020-11-02 10:46:21.890 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:52

  2. (another example) run FATE-Pipeline fit and predict jobs

    python demo/pipeline-mini-demo.py

    This script trains a heterogeneous logistic regression model and then runs prediction with the trained model.

    2020-11-02 15:40:43.907 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215404362914679
Job is still waiting, time elapse: 0:00:01
Running component reader_0, time elapse: 0:00:08
Running component dataio_0, time elapse: 0:00:10
Running component intersection_0, time elapse: 0:00:15
Running component hetero_lr_0, time elapse: 0:00:42
2020-11-02 15:41:27.622 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215404362914679
2020-11-02 15:41:27.622 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:43

    Once fit job completes, demo script will print coefficients and training information of model.

    After having completed the fit job, script will invoke a predict job with the trained model. Note that Evaluation component is added to the prediction workflow. For more information on using FATE-Pipeline, please refer to this guide.

    2020-11-02 15:41:28.255 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215412764443280
Job is still waiting, time elapse: 0:00:02
Running component reader_1, time elapse: 0:00:08
Running component dataio_0, time elapse: 0:00:11
Running component intersection_0, time elapse: 0:00:15
Running component hetero_lr_0, time elapse: 0:00:20
Running component evaluation_0, time elapse: 0:00:25
2020-11-02 15:41:54.605 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215412764443280
2020-11-02 15:41:54.605 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:26
  • No labels