.. _ome-files-maint:

Maintenance
===========

This section describes various tasks which are performed on a periodic
basis to keep the codebase up to date with various third-party
packages.  This includes the versions used for the superbuild, and
ensuring that the code will compile against various upstream releases,
from the latest stable release to the versions in common use over a
range of platforms.  It also includes keeping the codebase usable as
the toolchains on the supported platforms are updated.


Super-build packages
--------------------

General
^^^^^^^

Keep up to date with all new upstream releases of third-party
packages, in particular for security updates.  ``zlib`` and ``png``
are notable here, but this applies to all packages.  Check all
packages are the current stable version before making a new release.
In the case of a security vulnerability, it is possible to release and
provide builds with just the vulnerable package updated.

In the simple case of a package which requires no special additional
patching, for example ``png`` or ``zlib``, updating is as simple as
editing :file:`packages/<package>/superbuild.cmake`:

- update the source URL
- update the source hash

For a point release, this is typically sufficient.  For a major
release, also:

- check for any changes in the package prerequisites
- add or update missing prerequisites if required
- update the prerequisites if required

boost
^^^^^

Boost releases are approximately every six months.  In addition to the
general steps:

- update `FindBoost.cmake
  <https://github.com/Kitware/CMake/blob/master/Modules/FindBoost.cmake>`__;
  this includes the supported versions and the dependency information;
  instructions are in the comments in :file:`FindBoost.cmake`
- open a PR against `cmake.git <https://github.com/Kitware/CMake>`__
- update the embedded copy of :file:`FindBoost.cmake` in all OME
  components so that they can detect and use the new Boost version

Updating the upstream version is needed to keep our code in sync with
the canonical version.  It is also a necessary requirement for
end-users to be able to use our exported dependency information in
their projects.  It may be the case that the Boost project will
provide CMake configuration files in the future, containing the
dependency information; at this point the embedded
:file:`FindBoost.cmake` may be dropped entirely.

bzip2
^^^^^

Not released very frequently.  In addition to the general steps, we
use a custom CMake build under :file:`packages/bzip2/files` which
requires updating:

- update the release version and shared library version in
  :file:`CMakeLists.txt`
- check that the source file list in :file:`CMakeLists.txt` is
  up to date (a diff of the old and new bzip2 sources will show the
  changes in the upstream build files)

icu
^^^

Releases are approximately six-monthly to annually.  In addition to
the general steps, the Windows solution and project files require
creating for the supported versions of Visual Studio.

- delete all files and directories under :file:`packages/icu/files`

For each Visual Studio version:

- start the Visual Studio application
- open :file:`source/allinone/allinone.sln`, and allow Visual Studio
  to upgrade the projects to the current version
- save the solution (you must Save As to overwrite the existing
  solution; Save is insufficient)
- copy the solution and project files under
  :file:`packages/icu/files/VC<ver>`

xalan
^^^^^

Releases are infrequent.  In addition to the general steps, the
Windows solution and project files require creating for the supported
versions of Visual Studio.  Also, the project files at the time of
writing have broken ICU support and require manual fixing; they try to
build ICU support even when ICU support is not configured (see the
`JIRA bug <https://issues.apache.org/jira/browse/XALANC-767>`__).

- delete all files and directories under :file:`packages/xalan/files`
- apply this change to xalan :file:`c/Makefile.incl.in` (prevents
  overriding of system libraries causing breakage due to link
  failures)::

    --- a/c/Makefile.incl.in
    +++ b/c/Makefile.incl.in
       MAKE_SHARED = ${CXX} $(CXXFLAGS) -D${PLATFORM} -dynamiclib -prebind -seg1addr 0x38000000  -compatibility_version 1 -current_version $(LIB_MAJOR_DOT_VER) -install_name ${LINK_NAME} ${LDFLAGS}
       LINK = ${CXX} $(CXXFLAGS) ${PLATFORM_COMPILE_OPTIONS} ${LDFLAGS}
    
    -  export DYLD_LIBRARY_PATH := $(XERCESCROOT)/lib:$(ICULIB_LOC):$(DYLD_LIBRARY_PATH)
    +  export DYLD_FALLBACK_LIBRARY_PATH := $(XERCESCROOT)/lib:$(ICULIB_LOC):$(DYLD_FALLBACK_LIBRARY_PATH)
       LOC_LIB = ${LOC_LIBNAME}.${LIB_MAJOR_VER}.${LIB_MINOR_VER}$(SHLIBSUFFIX)
       LOC_SONAME = ${LOC_LIBNAME}.${LIB_MAJOR_VER}${SHLIBSUFFIX}
       MAKE_SHARED_LOC= ${CXX} $(CXXFLAGS) -D${PLATFORM} -dynamiclib -prebind -seg1addr 0x40000000  -compatibility_version 1 -current_version $(LIB_MAJOR_DOT_VER) -install_name ${LOC_LIBNAME}${SHLIBSUFFIX} ${LDFLAGS}

- copy xalan :file:`c/Makefile.incl.in` to :file:`packages/xalan/files/c`

For each Visual Studio version:

- copy xalan :file:`c/Projects/Win32/VC10` (the latest at the time of
  writing) to :file:`c/Projects/Win32/VC<ver>`
- start the Visual Studio application
- open xalan :file:`c/Projects/Win32/VC<ver>/Xalan.sln`, and allow Visual
  Studio to upgrade the projects to the current version
- save the solution (you must Save As to overwrite the existing
  solution; Save is insufficient)
- apply this change to :file:`XalanExe.vcxproj` (shown here for
  VC12)::

    --- a/VC12/XalanExe/XalanExe.vcxproj
    +++ b/VC12/XalanExe/XalanExe.vcxproj
    @@ -546,11 +546,11 @@
         <ClCompile Include="..\..\..\..\src\xalanc\XalanExe\XalanExe.cpp" />
       </ItemGroup>
       <ItemGroup>
    -    <ProjectReference Include="..\AllInOne\AllInOne.vcxproj">
    +    <ProjectReference Include="..\AllInOne\AllInOne.vcxproj" Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
           <Project>{e1d6306e-4ff8-474a-be7f-45dcba4888b6}</Project>
           <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
         </ProjectReference>
    -    <ProjectReference Include="..\AllInOne\AllInOneWithICU.vcxproj">
    +    <ProjectReference Include="..\AllInOne\AllInOneWithICU.vcxproj" Condition="'$(Configuration)'=='Debug with ICU' Or '$(Configuration)'=='Release with ICU'">
           <Project>{755ad11c-80b9-4e33-9d3d-9a68884a3ec8}</Project>
           <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
         </ProjectReference>
- copy the xalan solution and project files under
  :file:`packages/xalan/c/files/projects/Win32/VC<ver>`

xerces
^^^^^^

Releases are infrequent.  In addition to the general steps, the
Windows solution and project files require creating for the supported
versions of Visual Studio.  Also, the project files at the time of
writing have broken ICU support and require manual fixing; the ICU
libraries are either missing entirely or the debug and release
variants are mixed up.

- delete all files and directories under :file:`packages/xerces/files`

For each Visual Studio version:

- apply the following patch, shown here for VC11 but applies to all
  versions; if your version does not exist, make this change to the
  VC12 version::

    --- a/projects/Win32/VC11/xerces-all/XercesLib/XercesLib.vcxproj
    +++ b/projects/Win32/VC11/xerces-all/XercesLib/XercesLib.vcxproj
    @@ -538,7 +538,7 @@
         </ResourceCompile>
         <Link>
           <AdditionalOptions>%(AdditionalOptions)</AdditionalOptions>
    -      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuuc.lib;%(AdditionalDependencies)</AdditionalDependencies>
    +      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuucd.lib;%(AdditionalDependencies)</AdditionalDependencies>
           <OutputFile>$(TargetPath)</OutputFile>
           <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
           <GenerateDebugInformation>true</GenerateDebugInformation>
    @@ -578,7 +578,7 @@
           <Culture>0x0409</Culture>
         </ResourceCompile>
         <Link>
    -      <AdditionalDependencies>ws2_32.lib;advapi32.lib;%(AdditionalDependencies)</AdditionalDependencies>
    +      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuucd.lib;%(AdditionalDependencies)</AdditionalDependencies>
           <OutputFile>$(TargetPath)</OutputFile>
           <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
           <GenerateDebugInformation>true</GenerateDebugInformation>
    @@ -669,7 +669,7 @@
           <Culture>0x0409</Culture>
         </ResourceCompile>
         <Link>
    -      <AdditionalDependencies>ws2_32.lib;advapi32.lib;%(AdditionalDependencies)</AdditionalDependencies>
    +      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuuc.lib;%(AdditionalDependencies)</AdditionalDependencies>
           <OutputFile>$(TargetPath)</OutputFile>
           <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
           <GenerateDebugInformation>true</GenerateDebugInformation>

- check for a supported set of project files under
  :file:`projects/Win32/VC<ver>`; if a suitable version does not
  exist:

  - copy :file:`projects/Win32/VC12` (the latest at the time of
    writing) to :file:`projects/Win32/VC<ver>`; make sure you applied
    the patch in the previous step beforehand
  - start the Visual Studio application
  - open :file:`projects/Win32/VC<ver>/xerces-all/xerces-all.sln`, and
    allow Visual Studio to upgrade the projects to the current version
  - save the solution (you must Save As to overwrite the existing
    solution; Save is insufficient)
    
- copy the solution and project files under
  :file:`packages/xerces/files/projects/Win32/VC<ver>`


Toolchain support and standards conformance
-------------------------------------------

Currently the code is built and tested on several platforms,
including:

- FreeBSD with clang++ 3.4
- Linux with GCC 4.8 and 4.9
- MacOS X with clang++ 3.4 and 3.6 (non-standard Mac versions)
- Windows with VC12

Testing on a range of version combinations of compilers and platforms
is helpful in picking up bugs which would otherwise go undiagnosed
until encountered by an end-user.  For example, the FreeBSD builds
pick up problems which are not noticed on MacOS X by default, though
they could still occur in practice (e.g. since MacOS X has a
non-standard clang++ and libc++ which have a number of odd quirks).
There are also differences between compiler versions e.g. GCC on Linux
and clang++ on MacOS, making testing multiple versions useful to pick
up portability issues as early as practical.  We cannot test every
compiler and OS version, along with all the different versions of
third-party libraries we depend upon, so this is a best effort to test
as wide a range of what is in common use as possible.  Inevitably,
some issues will remain undiscovered if they are only seen with an
untested set of combinations.

Over time, OS releases will reach their end of life, and new
replacements will appear.  The test matrix will require adjusting to
add new platforms and drop old ones.  This should be straightforward,
but if the platform is new or significantly changed then it may
require code changes to correct any exposed portability issue or
latent bug which might break the CI builds.  It may also require
special-casing support for the platform; there are portability headers
in ome-common for this purpose, as well as CMake platform checks,
which may be updated as required to add support.


Feature testing and compatibility code
--------------------------------------

Currently, we support and test two separate sets of third-party
library versions for each supported platform:

- the libraries provided by the distribution's package management
  system, where applicable (includes homebrew on MacOS X)
- the libraries provided by the superbuild

The first ensures compatibility of our libraries and headers with the
system as a whole; this is necessary to allow use of our libraries
with all the libraries provided by the system.  The second ensures
that the current version of each library is buildable and usable
across all the supported platforms, and this allows for the use of
current libraries on older systems such as enterprise Linux
distributions.  This also means we test up to several versions of each
library, increasing our test coverage for portability and correctness
purposes.

Over time, portability workarounds we have put in place may be
dropped.  Examples include:

- dropping support for older versions of libraries, e.g. we have
  extensive workarounds to support older libtiff releases which can be
  removed entirely once all the Linux distribution versions we support
  are 4.0.0 or later, or 4.0.4 or later (there are two sets of
  workarounds); this will simplify the code, and also allow
  assumptions to be made regarding the availability of BigTIFF support
- dropping functionality checks and workarounds for functionality and
  behaviour differences, e.g. missing filesystem functions and
  geometry and endian libraries in older Boost releases
- using the standard implementation of various functions,
  e.g. :cpp:class:`std::shared_ptr` and :cpp:class:`std::regex` in
  C++11, :cpp:func:`std::make_unique` in C++14, ``filesystem``
  functions in C++17, where the Boost equivalents are currently used;
  this will reduce our dependency upon Boost over time, once the set
  of platforms we support all support the functionality we require;
  note that having the functionality does not make it usable, e.g. GCC
  has :cpp:class:`std::regex` but it is broken until 5.1 meaning that
  we need to use :cpp:class:`boost::regex` for the time being; use
  CMake feature tests to test that each feature is functional as well
  as present

Such changes should be safe to make with the existing test coverage we
have in place.
