h1. CN Buildout Testing and Release

Cutting a stable software release from development¶

Tools¶

The description of the tools below are for reference in case the day comes that they are updated to conform to new practices.
In order to make branching and tagging easier, I have committed some perl scripts.
If you checkout https://repository.dataone.org/software/tools/trunk/tag_release/

Determine the components and products to release¶

You will need to communicate with other developers, look at redmine tickets and review subversion logs to determine the the extent of changes that will need to be included in a production release. These changes should be noted in a redmine ticket. Ideally every file that has been modified/added/deleted will be noted in redmine tickets that relate to the release.
Always create a release story ticket to which all activities of the current release are related. All software components that will have a new version number will be added as a task for the release story. Each story or bug that affects the components of the release should be added as 'related issue' to the release story ticket.
see https://redmine.dataone.org/issues/5462

Development Process¶

Check Out Trunk¶

Ensure, that the development trunk of dataone should be checked out to a user's filesystem
svn checkout https://repository.dataone.org/software/cicore/trunk trunk

Beta Release Process¶

Check Out Subversion Branches¶

If you do not have have the subversion branches checked out, then do so:

XXX todo update this for beta branching code. each new CN minor release should have a branch that contains links to the beta branches under development for a CN release level.... (?)

svn co --depth immediates https://repository.dataone.org/software/cicore/branches branches

The subversion branches will meet the criteria for Major.Minor releases. All patches between minor/major releases will occcur within the branches checked out for that release. For instance, when development has ended upon the 1.1 release of DataONE, branches for every component will need to be created to support maintanence releases. The format of the branch names are such:
_v.

Configure Beta Build Control file¶

The beta build is controlled through a configuration file.
If you have not done so, then checkout the d1_beta_build_control directory:

svn checkout https://repository.dataone.org/software/cicore/trunk/d1_beta_build_control d1_beta_build_control
The property that builds the current beta environment is named current_beta_control.properties.
You are then ready to modify the current_release_control.properties file.
The Beta build control file only needs to be updated upon a new Major or Minor release of the products. For maintenance or bugfix releases, modifications are performed in the same branch and thus the beta control file may point to the same branch.

Compare the Trunk to the Branches¶

In order to ensure that all changes have been merged, a visual inspection of the differences should be made of all the modifications made in trunk to the files in the branch.
on Apple Macintosh OSX, additional tools will need to be installed.
(confirm that the script runs with the Beta Channel instead of the Staging Channel)
Add in the merge command details. https://repository.dataone.org/software/tools/trunk/control_release/mergeDiffProjects.pl
After ensuring that all the files in the projects have been merged, increment the version number in the pom.xml file for java projects, or the control file for debian package builds.
Make certain to test building the newly changed java components. Ensure any dependencies in the pom have been updated to point to the appropriate version of the dependency.

Run Beta Jenkins Build Process & Upgrade¶

Login to Jenkins at the default 'Jenkins' screen as admin (trying from any other screen will cause a failure).
Click on the tab 'Beta Builds'
In the project dataone-cn-metacat-deb-beta, edit one place:

    under Build -> Execute -> Command edit the string 'JARS=( Metacat_stable/workspace/METACAT_X_X_X/dist/knb.war );' (substitute release numbers for X_X_X )

In the project Metacat_Stage, edit four places:

    Source Code Management -> Subversion -> Modules -> Repository URL edit the string to be https://code.ecoinformatics.org/code/metacat/tags/METACAT_X_X_X (substitute release numbers for X_X_X )

    Execute shell -> Command edit the string 'cd "$WORKSPACE/METACAT_X_X_X/docs/user/metacat"' (substitute release numbers for X_X_X )

    Invoke Ant -> Targets -> Press Advanced... Button -> edit the string 'metacat.dir=/home/jenkins/work/jobs/Metacat_stable/workspace/METACAT_X_X_X' (substitute release numbers for X_X_X )

    Javadoc directory -> edit the string 'METACAT_X_X_X/build/docs'  (substitute release numbers for X_X_X )

Once these edits have been made, then click on the job 'Build_Beta' in the 'Staging Beta' tab. OIn the left side bar of the page, click 'Build Now' to begin the build process.
The build process is complete when 'Build_Beta_Level_8' had completed running its last job, currently 'dataone-mercury-deb-stagedataone-mercury-deb-stage'. Login to the staging machines, and execute the 'sudo apt-get update' and 'sudo apt-get upgrade' commands.
note: as of 09/07/2012 if a beta package needs to be rebuilt and redeployed, the package will have to be removed 'sudo apt-get remove ' and then re-installed. Due to package dependencies, this may a cause all of the packages to be removed and reinstalled. Please take note of any packages that are removed, so as to reinstall them later.

Upgrade Beta CN environment¶

Test Beta CN Environment¶

Configure Stable Build Control file¶

The stable build is controlled through a configuration file.
If you have not done so, then checkout the d1_stable_build_control directory:

svn checkout https://repository.dataone.org/software/cicore/trunk/d1_stable_build_control d1_stable_build_control
The property file that builds the current release is named current_release_control.properties. Each historical release and release candidate should have its own tagged d1_stable_build_control properties file.
svn info current_release_control.properties
Before modifying the current_release_control.properties file for a new build, confirm that the previous version was tagged, and if not then tag it.
For example, is the previous release was 1.0.4 for the CN ant the next release is 1.1.0, then the following command would be appropriate to execute:
svn copy -m"Preparing for 1.1.0 tagging by backing up current stable build properties" https://repository.dataone.org/software/cicore/trunk/d1_stable_build_control https://repository.dataone.org/software/cicore/tags/D1_STABLE_BUILD_CONTROL_v1.0.4
Now, you are ready to modify the current_release_control.properties file.
Confirm for each component that will be released in the CCI release that the current_release_control.properties has the component equivalent to the tag location.
For instance, d1_common_java is a component that will be released in CCI verion 1.2.6. The version number of d1_common_java will be 1.2.0 for CCI version 1.2.6. The entry in the current_release_control.properties should look like:
D1_COMMON_JAVA=tags/D1_COMMON_JAVA_v1.2.0
This nomenclature indicates to Jenkins a relative path in the subversion repository from which to pull the current code for compilation and packaging.

Commit Control File¶

Once all the components have been tagged and branched for a release, then you will need to commit the current_release_control.properties into svn.

Tag a Component¶

When the current_release_control.properties file is edited and committed, each branch should be copied into a new tag.
If you perform this on the commandline, then it is tedious. It is best to use a program to ensure consistency. Check out the control_release project under tools if you have not already done so. It contains two programs. Once manages merging changes from the trunk into a branch. The other manages creating new branches as well as creating tags from branches. AT this phase, the program controlStageAndRelease.pl is the perl file to execute.
svn checkout https://repository.dataone.org/software/tools/trunk/control_release control_release
change directory into control_release and run
./controlStageAndRelease.pl --tag --all
using the --all tag will force each component as listed in the current_release_control.properties to be compared against the subversion repository to determine if it does in fact exist. If it does not exist, then a new tag will be created by copying the branch as listed in the current_beta_control.properties to the new tag.
If you only need to check on a single tag, then you can specify the component.
./controlStageAndRelease.pl --tag --component componentName

Update Trunk to new Snapshot level¶

1) Maven Projects
Lastly, edit the pom.xml in the trunk of the project and change the revision numbers. You will modify the version element of the project to increase the revision number and you will modify the d1 component property elements to point to the lastest SNAPSHOTs of those dependencies, such that if dataone_common_java had just been updated from RC7 to RC8, along with all of its dependencies from RC3 to RC4, then the new trunk pom.xml would look like:
d1_common_java
jar
1.0.1-RC9-SNAPSHOT
DataONE_Common_Java
http://dataone.org
DataONE Common Code with Service Interface Definitions

1.2.3
1.0.0-RC5-SNAPSHOT
1.0.0-RC5-SNAPSHOT

while the pom.xml in the tag would look like:
d1_common_java
jar
1.0.1-RC8
DataONE_Common_Java
http://dataone.org
DataONE Common Code with Service Interface Definitions

1.2.3
1.0.0-RC4
1.0.0-RC4

commit the pom to the latest SNAPSHOT.
2) Debian Packages
Edit the control file and increase the RC level and then commit.

Update Branch to next Release level¶

Jenkins Stable Build Processes¶

The build process on Jenkins is as follows:
Login to Jenkins at the default 'Jenkins' screen as admin (trying from any other screen will cause a failure).
Click on the tab 'Stable - tags'
If dataone-cn-metadata-deb-stable project is being released, then follow the next steps:

In the project dataone-cn-metacat-deb-stable, edit two places: 

under String -> Parameter edit 'DATAONE_CN_METACAT_DEB' to be  represented by the default value tags/DATAONE-CN-METACAT_vx.x.x (where  x.x.x is the tag for dataone-cn-metadata-deb-stable being released)

under Build -> Execute -> Command edit the string 'JARS=(  Metacat_stable/workspace/METACAT_X_X_X/dist/knb.war );' (substitute Metacat project release numbers for X_X_X )

If the Metacat project is being released, then follow the next steps:

In the project Metacat_Stable, edit four places:

Source Code Management -> Subversion -> Modules -> Repository URL edit the string to be https://code.ecoinformatics.org/code/metacat/tags/METACAT_X_X_X (substitute Metacat project release numbers for X_X_X )

Execute shell -> Command edit the string 'cd  "$WORKSPACE/METACAT_X_X_X/docs/user/metacat"' (substitute Metacat project release numbers for X_X_X )

Invoke Ant -> Targets -> Press Advanced... Button -> edit the  string  'metacat.dir=/home/hudson/work/jobs/Metacat_stable/workspace/METACAT_X_X_X'  (substitute Metacat project release numbers for X_X_X )

Javadoc directory -> edit the string 'METACAT_X_X_X/build/docs'  (substitute Metacat project release numbers for X_X_X )

To build out the new production release, click on the Job entitled 'Build_Stable' and then click on the sidebar link 'Build Now'

Addendum on Jenkins¶

Notes about Jenkins builds:
There are three different apt repositories built by hudson:
http://dev-testing.dataone.org/ubuntu-unstable/
and
http://dev-testing.dataone.org/ubuntu-stage/
and
http://dev-testing.dataone.org/ubuntu-stable/
The unstable packages are built from the subversion trunk. Jenkins will take the existing version number in the package and append RYYYYMMDD.nnn to the version number, where YYYY=year, MM=month, DD=day, nnn = build number for the day. So for example, the package "dataone-cn-portal" currently has Version: "1.0.0-RC2~unstable" in the DEBIAN/control file. The resulting file name and package version as generated by Jenkins are:
dataone-cn-portal_1.0.0R20120303.002-RC2~unstable_all.deb
for the file name and the version information contained in Packages.gz is:
1.0.0R20120303.002-RC2~unstable
The auto-generated version number means that any build of the package is equivalent to a snapshot, and apt will always install the new version during upgrade or install operations.
Add discussion of Beta testing and releases.
The stable packages, build by the Stable_Build job on Jenkins, use the version information exactly as recorded in the DEBIAN/control file. These packages are built from subversion tags, and so will not change between compilations.
The result is that the ubuntu-unstable repository always contains only unstable (trunk) builds, and they always have a modified version tag.
The ubuntu-stable repository always contains only the stable builds from tags, and these have the version as recored in the DEBIAN/control file.
As a result, there should never by a mix-up between unstable and stable packages.
However, it of course critical that the version numbers in the DEBIAN/control files are updated when a new release is being made. Apt uses that information to determine if it needs to install the package over an existing installation.

Project

General

Profile

Infrastructure

Wiki