Navigation

OME-Files

Main Downloads
C++ Downloads
Licensing

Page Contents

Previous topic

ome-files view

This Page

Maintenance

This section describes various tasks which are performed on a periodic basis to keep the codebase up to date with various third-party packages. This includes the versions used for the superbuild, and ensuring that the code will compile against various upstream releases, from the latest stable release to the versions in common use over a range of platforms. It also includes keeping the codebase usable as the toolchains on the supported platforms are updated.

Super-build packages

General

Keep up to date with all new upstream releases of third-party packages, in particular for security updates. zlib and png are notable here, but this applies to all packages. Check all packages are the current stable version before making a new release. In the case of a security vulnerability, it is possible to release and provide builds with just the vulnerable package updated.

In the simple case of a package which requires no special additional patching, for example png or zlib, updating is as simple as editing packages/<package>/superbuild.cmake:

  • update the source URL
  • update the source hash

For a point release, this is typically sufficient. For a major release, also:

  • check for any changes in the package prerequisites
  • add or update missing prerequisites if required
  • update the prerequisites if required

boost

Boost releases are approximately every six months. In addition to the general steps:

  • update FindBoost.cmake; this includes the supported versions and the dependency information; instructions are in the comments in FindBoost.cmake
  • open a PR against cmake.git
  • update the embedded copy of FindBoost.cmake in all OME components so that they can detect and use the new Boost version

Updating the upstream version is needed to keep our code in sync with the canonical version. It is also a necessary requirement for end-users to be able to use our exported dependency information in their projects. It may be the case that the Boost project will provide CMake configuration files in the future, containing the dependency information; at this point the embedded FindBoost.cmake may be dropped entirely.

bzip2

Not released very frequently. In addition to the general steps, we use a custom CMake build under packages/bzip2/files which requires updating:

  • update the release version and shared library version in CMakeLists.txt
  • check that the source file list in CMakeLists.txt is up to date (a diff of the old and new bzip2 sources will show the changes in the upstream build files)

icu

Releases are approximately six-monthly to annually. In addition to the general steps, the Windows solution and project files require creating for the supported versions of Visual Studio.

  • delete all files and directories under packages/icu/files

For each Visual Studio version:

  • start the Visual Studio application
  • open source/allinone/allinone.sln, and allow Visual Studio to upgrade the projects to the current version
  • save the solution (you must Save As to overwrite the existing solution; Save is insufficient)
  • copy the solution and project files under packages/icu/files/VC<ver>

xalan

Releases are infrequent. In addition to the general steps, the Windows solution and project files require creating for the supported versions of Visual Studio. Also, the project files at the time of writing have broken ICU support and require manual fixing; they try to build ICU support even when ICU support is not configured (see the JIRA bug).

  • delete all files and directories under packages/xalan/files

  • apply this change to xalan c/Makefile.incl.in (prevents overriding of system libraries causing breakage due to link failures):

    --- a/c/Makefile.incl.in
+++ b/c/Makefile.incl.in
   MAKE_SHARED = ${CXX} $(CXXFLAGS) -D${PLATFORM} -dynamiclib -prebind -seg1addr 0x38000000  -compatibility_version 1 -current_version $(LIB_MAJOR_DOT_VER) -install_name ${LINK_NAME} ${LDFLAGS}
   LINK = ${CXX} $(CXXFLAGS) ${PLATFORM_COMPILE_OPTIONS} ${LDFLAGS}

-  export DYLD_LIBRARY_PATH := $(XERCESCROOT)/lib:$(ICULIB_LOC):$(DYLD_LIBRARY_PATH)
+  export DYLD_FALLBACK_LIBRARY_PATH := $(XERCESCROOT)/lib:$(ICULIB_LOC):$(DYLD_FALLBACK_LIBRARY_PATH)
   LOC_LIB = ${LOC_LIBNAME}.${LIB_MAJOR_VER}.${LIB_MINOR_VER}$(SHLIBSUFFIX)
   LOC_SONAME = ${LOC_LIBNAME}.${LIB_MAJOR_VER}${SHLIBSUFFIX}
   MAKE_SHARED_LOC= ${CXX} $(CXXFLAGS) -D${PLATFORM} -dynamiclib -prebind -seg1addr 0x40000000  -compatibility_version 1 -current_version $(LIB_MAJOR_DOT_VER) -install_name ${LOC_LIBNAME}${SHLIBSUFFIX} ${LDFLAGS}

  • copy xalan c/Makefile.incl.in to packages/xalan/files/c

For each Visual Studio version:

  • copy xalan c/Projects/Win32/VC10 (the latest at the time of writing) to c/Projects/Win32/VC<ver>

  • start the Visual Studio application

  • open xalan c/Projects/Win32/VC<ver>/Xalan.sln, and allow Visual Studio to upgrade the projects to the current version

  • save the solution (you must Save As to overwrite the existing solution; Save is insufficient)

  • apply this change to XalanExe.vcxproj (shown here for VC12):

    --- a/VC12/XalanExe/XalanExe.vcxproj
+++ b/VC12/XalanExe/XalanExe.vcxproj
@@ -546,11 +546,11 @@
     <ClCompile Include="..\..\..\..\src\xalanc\XalanExe\XalanExe.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\AllInOne\AllInOne.vcxproj">
+    <ProjectReference Include="..\AllInOne\AllInOne.vcxproj" Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
       <Project>{e1d6306e-4ff8-474a-be7f-45dcba4888b6}</Project>
       <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
     </ProjectReference>
-    <ProjectReference Include="..\AllInOne\AllInOneWithICU.vcxproj">
+    <ProjectReference Include="..\AllInOne\AllInOneWithICU.vcxproj" Condition="'$(Configuration)'=='Debug with ICU' Or '$(Configuration)'=='Release with ICU'">
       <Project>{755ad11c-80b9-4e33-9d3d-9a68884a3ec8}</Project>
       <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
     </ProjectReference>

  • copy the xalan solution and project files under packages/xalan/c/files/projects/Win32/VC<ver>

xerces

Releases are infrequent. In addition to the general steps, the Windows solution and project files require creating for the supported versions of Visual Studio. Also, the project files at the time of writing have broken ICU support and require manual fixing; the ICU libraries are either missing entirely or the debug and release variants are mixed up.

  • delete all files and directories under packages/xerces/files

For each Visual Studio version:

  • apply the following patch, shown here for VC11 but applies to all versions; if your version does not exist, make this change to the VC12 version:

    --- a/projects/Win32/VC11/xerces-all/XercesLib/XercesLib.vcxproj
+++ b/projects/Win32/VC11/xerces-all/XercesLib/XercesLib.vcxproj
@@ -538,7 +538,7 @@
     </ResourceCompile>
     <Link>
       <AdditionalOptions>%(AdditionalOptions)</AdditionalOptions>
-      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuuc.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuucd.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <OutputFile>$(TargetPath)</OutputFile>
       <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
       <GenerateDebugInformation>true</GenerateDebugInformation>
@@ -578,7 +578,7 @@
       <Culture>0x0409</Culture>
     </ResourceCompile>
     <Link>
-      <AdditionalDependencies>ws2_32.lib;advapi32.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuucd.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <OutputFile>$(TargetPath)</OutputFile>
       <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
       <GenerateDebugInformation>true</GenerateDebugInformation>
@@ -669,7 +669,7 @@
       <Culture>0x0409</Culture>
     </ResourceCompile>
     <Link>
-      <AdditionalDependencies>ws2_32.lib;advapi32.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalDependencies>ws2_32.lib;advapi32.lib;icuuc.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <OutputFile>$(TargetPath)</OutputFile>
       <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
       <GenerateDebugInformation>true</GenerateDebugInformation>

  • check for a supported set of project files under projects/Win32/VC<ver>; if a suitable version does not exist:

    • copy projects/Win32/VC12 (the latest at the time of writing) to projects/Win32/VC<ver>; make sure you applied the patch in the previous step beforehand
    • start the Visual Studio application
    • open projects/Win32/VC<ver>/xerces-all/xerces-all.sln, and allow Visual Studio to upgrade the projects to the current version
    • save the solution (you must Save As to overwrite the existing solution; Save is insufficient)

  • copy the solution and project files under packages/xerces/files/projects/Win32/VC<ver>

Toolchain support and standards conformance

Currently the code is built and tested on several platforms, including:

  • FreeBSD with clang++ 3.4
  • Linux with GCC 4.8 and 4.9
  • MacOS X with clang++ 3.4 and 3.6 (non-standard Mac versions)
  • Windows with VC12

Testing on a range of version combinations of compilers and platforms is helpful in picking up bugs which would otherwise go undiagnosed until encountered by an end-user. For example, the FreeBSD builds pick up problems which are not noticed on MacOS X by default, though they could still occur in practice (e.g. since MacOS X has a non-standard clang++ and libc++ which have a number of odd quirks). There are also differences between compiler versions e.g. GCC on Linux and clang++ on MacOS, making testing multiple versions useful to pick up portability issues as early as practical. We cannot test every compiler and OS version, along with all the different versions of third-party libraries we depend upon, so this is a best effort to test as wide a range of what is in common use as possible. Inevitably, some issues will remain undiscovered if they are only seen with an untested set of combinations.

Over time, OS releases will reach their end of life, and new replacements will appear. The test matrix will require adjusting to add new platforms and drop old ones. This should be straightforward, but if the platform is new or significantly changed then it may require code changes to correct any exposed portability issue or latent bug which might break the CI builds. It may also require special-casing support for the platform; there are portability headers in ome-common for this purpose, as well as CMake platform checks, which may be updated as required to add support.

Feature testing and compatibility code

Currently, we support and test two separate sets of third-party library versions for each supported platform:

  • the libraries provided by the distribution’s package management system, where applicable (includes homebrew on MacOS X)
  • the libraries provided by the superbuild

The first ensures compatibility of our libraries and headers with the system as a whole; this is necessary to allow use of our libraries with all the libraries provided by the system. The second ensures that the current version of each library is buildable and usable across all the supported platforms, and this allows for the use of current libraries on older systems such as enterprise Linux distributions. This also means we test up to several versions of each library, increasing our test coverage for portability and correctness purposes.

Over time, portability workarounds we have put in place may be dropped. Examples include:

  • dropping support for older versions of libraries, e.g. we have extensive workarounds to support older libtiff releases which can be removed entirely once all the Linux distribution versions we support are 4.0.0 or later, or 4.0.4 or later (there are two sets of workarounds); this will simplify the code, and also allow assumptions to be made regarding the availability of BigTIFF support
  • dropping functionality checks and workarounds for functionality and behaviour differences, e.g. missing filesystem functions and geometry and endian libraries in older Boost releases
  • using the standard implementation of various functions, e.g. std::shared_ptr and std::regex in C++11, std::make_unique() in C++14, filesystem functions in C++17, where the Boost equivalents are currently used; this will reduce our dependency upon Boost over time, once the set of platforms we support all support the functionality we require; note that having the functionality does not make it usable, e.g. GCC has std::regex but it is broken until 5.1 meaning that we need to use boost::regex for the time being; use CMake feature tests to test that each feature is functional as well as present

Such changes should be safe to make with the existing test coverage we have in place.

Navigation

© Copyright 2000-2016, The Open Microscopy Environment . Last updated on Apr 25, 2016. Created using Sphinx 1.2.3.