Child pages
  • 3. Common Tasks
Skip to end of metadata
Go to start of metadata

Workflow

Before working on any fix or feature, rebase you local 'oi/hipster' branch and create a new branch:

$ git checkout oi/hipster
$ git pull --rebase origin oi/hipster
$ git checkout -b feature_XXX

Updating an existing component

Version bump

Let us consider that component 'library/foo' should be updated to latest version.

These instructions provide a generic overview of the process for updating a component: oi-userland best practices and tasks should be reviewed as well for a description of current tasks and framework modifications.

Preparation

The component directory should be cleaned up prior to any modification to the files:

$ cd components/library/foo
$ gmake clobber

Before building an existing component, its dependencies should be installed on the system:

$ gmake env-prep

Packages listed in the Makefile variable REQUIRED_PACKAGES will be installed or updated.

Please note that the automatic procedure for generating REQUIRED_PACKAGES only detects runtime dependencies: build dependencies like system headers may be added manually.

If you find a missing dependency, add it to REQUIRED_PACKAGES and commit the change.

Version update

Update the following variables in the component Makefile:

  1. COMPONENT_VERSION: version number of the new package.
  2. COMPONENT_REVISION: remove the line if present in the Makefile.
  3. COMPONENT_ARCHIVE_HASH: checksum hash if provided on the project download site.

If the checksum hash is not provided, use the output of the checksum verification stage in 'gmake prep'.

Download and patch the package:

$ gmake prep

Existing patches may not apply correctly against the new version and need be updated, in which case 'gmake prep' will fail to complete.

$ gmake sample-manifest

The component will be built, installed, and a new manifest copied to 'manifest/sample-manifest.p5m': review the difference and update the component manifest accordingly.

$ vimdiff manifest/sample-manifest.p5m foo.p5m

To check if the publication of the component can complete without actually publishing the package to the local repository, execute:

$ gmake pre-publish

If 'REQUIRED_PACKAGES' is not provided at the end of the component Makefile, the list of dependencies should be generated:

gmake REQUIRED_PACKAGES

Testing

If a test suite is provided by the package, run:

$ gmake test

If test results have changed or simply testing was not set up for the component, follow the instructions for testing a component.

Verification

Anytime a component is updated, it is recommended to review existing security issues and opened issues which may affect illumos.

When the component publishes correctly, verify that the process completes as expected:

$ gmake clobber; gmake publish; gmake publish; gmake publish;

The component may re-publish the second time but should not the third time.

Patches and security fixes

Adding patches to components for fixing bugs or security issues *without* updating to a newer version consists of a few simple steps:

  1. Patches should be added to the 'patches' directory.
  2. The Makefile variable COMPONENT_REVISION should be added and set to '1', or incremented if already present.
  3. If the modification affects dependent components, a list of dependencies should be provided in the pull-request.

Example: list dependencies

$ pkg search -r -o pkg.name 'depend:required:library/foo'

Creating a new component

Setup the initial component layout

After choosing where to place the new component based on its category, create the component directoy and copy a sample Makefiles from the template directory.

The template to copy depends on the build system used by the software to be packaged.

Before any further work on the component, fill-in all the fields required for the package metadata.

Patch, configure, buid, and install

TBD

Create the package manifests

A default manifest is generated using the target sample-manifest and created in manifest/sample-manifest.p5m

Copy the sample manifest in the component directory as $(COMPONENT_NAME).p5m, or if necessary split it in different manifests.

This manifest can be modified in several ways, for example:

  1. Commenting out lines to avoid delivering files.
  2. Apply transforms to change the path, the permissions, or set facet to files.
  3. Adding IPS actions triggered at package installation: user, groups, or even directory to create empty directories, usually placeholders of log and other service files, ...
  4. Adding IPS link actions to create symbolic links or hardlinks.
  5. Setting modifiers to the file actions: for example to modify the default permissions, or ability to retain local modifications on package update.

Create the list of dependencies

The list of dependencies is automatically generated using the target REQUIRED_PACKAGES.

After executing:

$ gmake REQUIRED_PACKAGES

A list of detected runtime dependencies will be appended to the Makefile.

Example: Minimum dependency for a library.

# Auto-generated dependencies
REQUIRED_PACKAGES += system/library

The procedure does not detect build or test dependencies, nor may find dependencies that are only text files.

They should be added manually in a separate list before the runtime dependencies.

Example: Test dependencies were added to ensure reproducibility of the test results

# Test dependencies:
REQUIRED_PACKAGES += developer/versioning/cvs
REQUIRED_PACKAGES += developer/versioning/git

# Auto-generated dependencies
REQUIRED_PACKAGES += system/library
REQUIRED_PACKAGES += system/library/math

In some particular cases dependency resolution may fails for some files contained in the manifest: it is then possible to bypass or override the dependency mechanism.

Refer to the IPS documentation for more details.

Optional: components part of a software metapackage

If the component is to be delivered as part of a software group, it should be added to the corresponding metapackage.

Metapackage components are listed under the meta-packages directory.

Add the newly created component to the manifest, for example in xorg-video.p5m:

depend fmri=pkg:/x11/server/xorg/driver/xorg-video-vboxvideo type=require

xorg-video-vboxvideo was added to the xorg-video metapackage with a "require" dependency.

Finally, increment the component revision in the Makefile since the package content was modified.

Deprecating a component

If a package has no consumer and may cause security issues or unjustified maintenance burden it should be made obsolete in oi-userland.

Such decision should be motivated and discussed amongst developers.

Verify that there is no consumer in the main repository

$ pkg search -r -o pkg.name "depend:require:full/package/name"

List known package FMRIs

First list all the known package FMRIs provided by the component:

  1. FMRI of latest version of all packages provided by the component (typically one per *.p5m file) found in the repository,
  2. FMRI of all the legacy names found in the component's history file if the component was renamed.

The following rules for deprecation apply:

TypeRule
Latest FMRIIncrement the COMPONENT_REVISION
Renamed FMRISet the branch version to the current BRANCH_ID

Nomenclature: FMRI versioning

package/name@$(COMPONENT_VERSION)$(PKG_SOLARIS_VERSION)-$(BRANCHID)

with

BRANCHID ?= $(RELEASE_MAJOR).$(RELEASE_MINOR).$(UPDATENUM).$(COMPONENT_REVISION)

The base BRANCHID is set with COMPONENT_REVISION = 0, then increment as the component is republished.

Add the component FMRIs to the file components/meta-packages/history/history, listed in alphabetical order.

 

Example: clang-34 deprecation

1. The last component FMRI found in the repository was:

developer/clang-34@3.4.2,5.11-2017.0.0.5

The FMRI to add to the history file show have an incremented COMPONENT_REVISION from 5 to 6.

2. The component contains the history file 'components/developer/clang-34/history':

developer/clang-3.4@3.4.2,5.11-2016.0.0.4 developer/clang-34

The current BRANCH_ID is 2017.0.0.0 since the latest published package is developer/clang-34@3.4.2,5.11-2017.0.0.5.

The list to add to the history file is:

developer/clang-3.4@3.4.2,5.11-2017.0.0.0
developer/clang-34@3.4.2,5.11-2017.0.0.6

Delete the component

Remove the directory containing the component:

$ git rm -r component_directory

 

Renaming a component

Whenever one (or more) package FMRI changes the component needs to take into account of the change.

To be able to resolve version constraints across renames the package management system needs to generate rename actions: this way the package can be tracked as the same even if the name changed.

The rename actions are generated via a 'history' file which contains mappings from the old name to the new name.

In the same commit as the package FMRI rename, add a file name 'history' or modify it if it exists,

For each package devliered by the component and affected by the change, add a line containing the full FMRI of the package (including the version string) and the new package FMRI (without version), separated by a space.

Example: When updating the component jq to version 1.5 a typo was discovered. The following line maps what the full FMRI would have been without renaming to the new package FMRI.

test/jq@1.5-2018.0.0.0 text/jq

Reproducible builds and test suites

Test suites can be easily run for any component supporting them, and the test results compared with reference output.

Even when results are not reproducible, it is always a good idea to add support for this target in components so that regressions can be caught.

Test run

After the component is built and installed test suites can be run using 'gmake test' if the 'test' target is declared in the component Makefile.

Example: tests for 32-bit and 64-bit versions

test target
test:	$(TEST_32_and_64)

The environment of the test suite can be changed with the variable 'COMPONENT_TEST_ENV'.

Comparison

If a master result file exists in the component test directory, the output produced by the test run is compared against master test results.

The default directory for master test results is 'test' as defined in the variable 'COMPONENT_TEST_RESULTS_DIR'.

Default names for the master files are defined in the variable 'COMPONENT_TEST_MASTER' as:

  1. results-32.master,
  2. results-64.master.

and correspond to 32-bit and 64-bit builds respectively.

The first time a master result file is added to the component, an empty file should be created prior to running 'gmake test' to trigger the comparison stage.

Example: initial creation of a master results file for a 32-bit build

$ mkdir test
$ touch test/results-32.master
$ gmake test

The comparison will fail as the master results file is empty: the actual test output should be copied to the test result directory (do not forget to commit the newly created file).

$ cp build/test/i86/results-32.snapshot test/results-32.master

If the results are architecture independent, the variable may be set to:

COMPONENT_TEST_MASTER =		$(COMPONENT_TEST_RESULTS_DIR)/results-all.master

Test results may be processed to retain relevant machine-independent output only.

Transforms of the output are controlled by two variables:

  1. the tool used for the transform is set by 'COMPONENT_TEST_TRANSFORMER' (default: GNU sed),
  2. the rules defined in 'COMPONENT_TEST_TRANSFORMS'.

Example:

Autotools

Only test results are retained:

COMPONENT_TEST_TRANSFORMS += \
        '-n ' \
        '-e "/TOTAL:/p" ' \
        '-e "/SKIP:/p" ' \
        '-e "/PASS:/p" ' \
        '-e "/FAIL:/p" ' \
        '-e "/ERROR:/p" '
CTest

Timestamps should be removed:

COMPONENT_TEST_TRANSFORMS+= '-e "s/[0-9. ]*sec//g"'

 

 

Testing components in a new Boot Environment

 

The components updated or modified may affect the system in a critical way, possibly putting services in a non-functioning state.

In order to avoid damaging your current installation, the recommended practice is to create a new Boot Environment (BE) then install the packages to be tested in this new environment.

The following commands should be run with administrative rights (using sudo) or by a user with "Software Installation" profile (using pfexec).

Creation of a new BE

Choose a name for the new BE which corresponds to the nature of the testing, say "testX".

# beadm create testX
# beadm mount testX

Note the mounting point of the BE, to substitute to /path/to/be in the following commands.

Most pkg commands can be applied to a mounted BE using the option -R and the path to the BE mountpoint:

pkg -R /path/to/be ...

Installation of the test packages to the BE

If the package to be to tested is constrained by an incorporation, the version lock should be removed, or the incorporation can simply be uninstalled in the created BE.

For example this step is required for packages listed in the userland-incorporation:

# pkg -R /path/to/be change-facet facet.require.consolidation/userland/userland-incorporation=false
# pkg -R /path/to/be uninstall userland-incorporation

If only a few packages are version locked, the constraint can be removed on an individual basis.

For example, removing the version lock for xeyes:

# pkg -R /path/to/be change-facet facet.version-lock.x11/xeyes=false

Two common cases of package testing are described below.

Updating from a remote repository:

If the packages are delivered from a repository as a bulk update, the test repository should be added as preferred and the main repository set as non-sticky to allow cross-publisher updates.

For example, installing new bits from the xorg-testing repository requires:

# pkg -R /path/to/be set-publisher -P -O http://pkg.openindiana.org/xorg-testing/ userland
# pkg -R /path/to/be set-publisher --non-sticky openindiana.org
# pkg -R /path/to/be update -v

Where:

  1. the first line adds the userland publisher from the xorg-testing repository as preferred (-P) so that packages from this repository get priority,
  2. then the main publisher openindiana.org is set as non-sticky to allow update of packages from openindiana.org to userland.
Installing specific packages from a local repository:

If the packages to be tested are published to your local repository, they can be installed to the new BE by providing the full path to the local repository and the list of full packages FMRIs including version and timestamp.

For example:

# pkg -R /path/to/be install -g file:///path/to/oi-userland/i386/repo pkg://userland/x11/server/xorg@1.19.6-2018.0.0.1:20180307T130609Z

Activation of a new BE

When the packages are installed, the test BE can be activated to make it default at the next reboot:

# beadm activate testX

Submitting a component for review

As soon as the component you added or modified is ready:

  1. squash your local commits into one commit,
  2. push your branch to GitHub,
  3.  and open a pull-request at https://github.com/openindiana/oi-userland.

Optionally you can select one or two reviewers familiar with the component or the software category to speed-up the review process.

Using pkg(5) for packaging tasks


By default all the commands work with the local list of installed packages: to query information from remote publishers the option '-r' shoud be specified.


Listing the publishers configured on the system:

Example: list of publisher on a system with both official OpenIndiana repositories and local repositories used by the developer.

$ pkg publisher 
PUBLISHER                   TYPE     STATUS P LOCATION
openindiana.org (non-sticky) origin   online F http://pkg.openindiana.org/hipster/
userland       (non-sticky) origin   online F file:///scratch/oi-userland/i386/repo/
hipster-encumbered (non-sticky) origin   online F http://pkg.openindiana.org/hipster-encumbered/
userland-encumbered (non-sticky) origin   online F file:///scratch/oi-userland/i386/encumbered-repo/

The official repositories are placed in higher priority and all repositories are set non-sticky so that:

  1. any locally compiled packages could be installed locally for testing,
  2. any package update in the official repositories would be applied over locally compiled packages.

If a publisher is set as sticky updating any package from this publisher to another package provided by a different publisher is forbidden: in short, no cross-publisher update is allowed.

 

Print information about a given package:

Example: print information on libjpeg-turbo (the full FMRI need not be passed if there is not ambiguity with the package name)

 

$ pkg info libjpeg-turbo
          Name: image/library/libjpeg-turbo
       Summary: libjpeg -JPEG Turbo libraries
      Category: System/Multimedia Libraries
         State: Installed
     Publisher: userland
       Version: 1.5.1
        Branch: 2017.0.0.0
Packaging Date: Wed Jun  7 19:36:05 2017
          Size: 2.79 kB
          FMRI: pkg://userland/image/library/libjpeg-turbo@1.5.1-2017.0.0.0:20170607T193605Z
   Project URL: http://www.libjpeg-turbo.org/
    Source URL: http://sourceforge.net/projects/libjpeg-turbo/files/1.5.1/libjpeg-turbo-1.5.1.tar.gz/download

 

 

 

List packages depending on the current package:

Example: packages depending on libjpeg-turbo (the full FMRI is needed).

$ pkg search -r -o pkg.name "depend:require:image/library/libjpeg-turbo"

 

Best practice

A list of recommendation is updated at Best Practices for oi-userland

 

 

 

  • No labels