When using external build nodes or CI labs, the logs still need to be push to a public, centralized server. The following are the steps needed to get access to push logs to nexus.akraino.org.
Step-by-step guide
- Create a helpdesk ticket (currently, go to Jira Support Desk; always-up-to-date list here: https://docs.releng.linuxfoundation.org/en/latest/helpdesk.html). The request should indicate that you are requesting "external log pushing permissions". You must provide the following information:
- A name for the log directory/silo (we have generally just been using company name, e.g. "att" or "ericsson").
- An LFID to be granted appropriate permissions on the Nexus server. Please create an ID specifically for this purpose at identity.linuxfoundation.org, rather than using a personal LFID.
Pushing the logs from within your internal jobs can be handled however you like, but below is a suggested script (adapted from one provided by Andrew Wilkinson). Please note that we run a cleanup script that check the age of directories at a depth of 4 within the logs, so it is important to use the path of "${SILO}/job/${JOB_NAME}/${BUILD_NUMBER}".
push_logs.sh# Deploying logs to LF Nexus log server ## # BUILD_NUMBER and JOB_NAME should be set by Jenkins NEXUS_URL=https://nexus.akraino.org SILO=<silo name provided above> JENKINS_HOSTNAME=<hostname/IP of your Jenkins host> BUILD_URL="${JENKINS_HOSTNAME}/job/${JOB_NAME}/${BUILD_NUMBER}/" NEXUS_PATH="${SILO}/job/${JOB_NAME}/${BUILD_NUMBER}" lftools deploy logs $NEXUS_URL $NEXUS_PATH $BUILD_URL echo "Logs uploaded to $NEXUS_URL/content/sites/logs/$NEXUS_PATH"
Another user-suggested way to structure this script is to push all the log files within a given folder to Nexus, with the below suggested script (please note that this version has not been tested by the LF):
# Archive files to LF Nexus log server ## NEXUS_URL=https://nexus.akraino.org SILO=<silo name provided above> NEXUS_PATH=<the destination folder on Nexus, such as $SILO/job> WORKSPACE=<archives directory provided by the user, with the -p option below it specifies the file pattern as .log files> lftools deploy archives -p '**/*.log' $NEXUS_URL $NEXUS_PATH $WORKSPACE echo "Logs uploaded to $NEXUS_URL/content/sites/logs/$NEXUS_PATH"
This method requires installing lftools via pip: https://pypi.org/project/lftools/. The credentials for your service account's LFID should be stored in $HOME/.netrc
Related articles
7 Comments
Mika Rautakumpu
Hi Andrew Wilkinson,
what is content inside .netrc file? What are needed parameters?
machine nexus.akraino.org
login <it this LFID or mail address>
password
Br,
Mika Rautakumpu
Robert Eby
See here:
https://docs.releng.linuxfoundation.org/projects/lftools/en/latest/commands/deploy.html
xin qiu
would suggest installing Nginx prior to upload the log.
Cedric Ollivier
I would rather suggest to use the S3 protocol without any third-party tools (lftools).
It's supported by LFN common test frameworks such as Xtesting without any effort (3 env variables).
It's already in place in OPNFV (centralized artifact repository - artifacts.opnfv.org) and CNTT RC cookbook (local artifact repository - Minio)
CC: Tapio Tallgren Juha Kosonen
jin peng
Hi Andrew Wilkinson,
I have a problem when I upload logs. error log is "ERROR: Failed to upload to Nexus with status code: 401. " can you help me reslove this probelm.
haihui wang
Please refer https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-22616
hanyu ding
Any one knows if our BP logs are stored in local host, could we need deploy lftools first to push log to nexus.akraino.org? ( we use private lab and argoCD not Jenkins),thanks !