Introduction

bazelizing, bazelization, to bazelize, bazelized: The act of converting some existing software artifact to be easily consumed by the Bazel build tool

OpenEXR is a library to store high-dynamic-range (HDR) image files. Usually, the extension .exr is chosen for such files. For instance, an HRD image can store 32-Bits per color channel instead of only 8-Bits. For viewing HDR images you need either an HDR screen or proper tone mapping on an LDR screen.

There are some free HDR Viewers that can be used to display HDR data on an LDR monitor, e.g.:

Luminance HDR has support for various tone mapping operators, including Reinhard 02 tone mapping.

This post will cover how I bazelized OpenEXR.

As a starting point for my bazelization I was looking for other Bazel related projects that make use of OpenEXR. I found seurat. Unfortunately, the way this project handles OpenEXR became in the meantime outdated. For instance, the half does not exist anymore in newer OpenEXR versions, Imath has been shifted to an own repository and OpenEXR developers decided to add pre-generated files for lookup tables. In the end, it is still possible to run seurant with an up to date version of OpenEXR when applying a few minor modifications, but it does not reflect the current OpenEXR project structure.

Besides seurat I did not find any other open-source efforts for an OpenEXR bazelization. If you know any other open source project that can be shared here, drop me a mail or post a comment.

My first attempt

Around 28. September 2020 I forked OpenEXR master and did some modifications to seurat to get it working with Bazel based on the Bazel build files provided by seurat.

One of the most important changes was in the file openexr/OpenEXR/IlmImf/dwaLookups.cpp. I had to change this:

void
generateLutHeader()
{
    // ...
        for (size_t i=0; i<workers.size(); ++i) {
            runners[i]->join(); // without this I experience virtual method called...
            delete runners[i];
        }
    // ...

Otherwise, there seem to be some thread problems and the code generation works sometimes and sometimes not.

Revisitng OpenEXR master again

At the time of writing this posting, I revisited the OpenEXR master.

I took a look at the OpenEXR dependencies and figured out the following dependencies/project structure:

OpenEXR dependencies

I translated this to Bazel build files and it finally worked on Windows 10 x64 and Ubuntu 20.04, at least for my use case. My OpenEXR build files can be found here.

I also used a Bazel query to plot a dependency graph (zoom to be able to read it):

bazel query --noimplicit_deps 'deps(//:OpenEXR)' --output graph > graph.in
dot -Tpng < graph.in > graph.png

OpenEXR Bazel Dependencies

Future

I had the idea of merging my change to the official OpenEXR master. The main build system of OpenEXR is currently CMake. Previously gnu/autotools have been used to build OpenEXR. Supporting two build systems (CMake and Bazel) is of course more effort to maintain but helps to spread OpenEXR. In the worst case, the Bazel build does simply not work. In the best case, someone who uses Bazel can benefit from it.

I got this feedback from the Technical Steering Committee of OpenEXR:

we’re concerned about the ongoing maintenance and support burden. Whenever we make changes to the build setup in the future (add/move/rename a file, etc), someone will need to change, and test, the Bazel build.

First of all, I was very happy that it was even considered and I really appreciate that I got the review comments so fast.

I was thinking about it. Yes, it is true, every time the CMake build configuration changes someone has to check if the Bazel build files need to be adapted. Since I want to keep track of the newest master of OpenEXR, I came up with the following question to myself. Can I automate the conversion from CMake to Bazel? This seems to be a recurring problem.

I decided on the following strategy:

'git clone https://github.com/AcademySoftwareFoundation/openexr
mkdir build_openexr
cmake -G"Visual Studio 16 2019" -A"x64" `
-Hopenexr `
-Bbuild_openexr
bazel run //:Edelweiss
# python3 main.py
buildifier edelweiss/BUILD.bazel

The above script fetches OpenEXR from GitHub, and then generates build files for Visual Studio 2019. Afterward, there is a magic tool called Edelweiss which reads the .vcproj files and converts them to a Bazel build file:

Here is a sketch of the idea:

import os
import pathlib
from xml.dom import minidom

def get_vcxproj_files(vcxproj_filename, strip_suffix):
    # Check what files Iex has
    Iex_srcs = []
    Iex_hdrs = []

    mydoc = minidom.parse(vcxproj_filename)
    items = mydoc.getElementsByTagName('ClInclude')
    for item in items:
        strIncludePath = str(item.attributes['Include'].value)
        strIncludePath = strIncludePath.replace(strip_suffix, "")
        strIncludePath = strIncludePath.replace("\\", "/")
        Iex_hdrs.append(strIncludePath)

    items = mydoc.getElementsByTagName('ClCompile')
    for item in items:
        if item.hasAttribute("Include"):
            strIncludePath = str(item.attributes['Include'].value)
            strIncludePath = strIncludePath.replace(strip_suffix, "")
            strIncludePath = strIncludePath.replace("\\", "/")
            Iex_srcs.append(strIncludePath)

    return Iex_srcs, Iex_hdrs

def extendBuildFile(build_file, target_name, srcs, hdrs):
    build_file.write("cc_library(name=\"" + target_name + "\",")
    build_file.write("srcs = " + str(srcs) + ",")
    build_file.write("hdrs = " + str(hdrs) + ",")
    build_file.write("includes = [\"src/lib/Iex\"],")
    build_file.write("deps = [\":ilm_base_config\"],")
    build_file.write(")\n")


def check():
    # Check if Iex dependency exists

    # there should be a "Iex.vcxproj" file
    if not os.path.exists("build_openexr/src/lib/Iex/Iex.vcxproj"):
        return False

    # Check what files Iex has
    Iex_srcs, Iex_hdrs = get_vcxproj_files("build_openexr/src/lib/Iex/Iex.vcxproj",
                    "G:\\dev\\Piper\\Edelweiss\\openexr\\")

    f = open("/home/q448004/dev/Piper/Edelweiss/edelweiss/BUILD.bazel", "w") 

    extendBuildFile(f, "Iex", Iex_srcs, Iex_hdrs)

    # Check if IlmThread dependency exists
    if not os.path.exists("build_openexr/src/lib/IlmThread/IlmThread.vcxproj"):
        return False

    IlmThread_srcs, IlmThread_hdrs = get_vcxproj_files("build_openexr/src/lib/IlmThread/IlmThread.vcxproj",
                    "G:\\dev\\Piper\\Edelweiss\\openexr\\")

    extendBuildFile(f, "IlmThread", IlmThread_srcs, IlmThread_hdrs)

    # Check if OpenEXR dependency exists
    OpenEXR_srcs, OpenEXR_hdrs = get_vcxproj_files("build_openexr/src/lib/OpenEXR/OpenEXR.vcxproj",
                    "G:\\dev\\Piper\\Edelweiss\\openexr\\")

    extendBuildFile(f, "OpenEXR", IlmThread_srcs, IlmThread_hdrs)

    f.close()

    return True

if check():
    print("Everything seems to be ok")
else:
    print("Differences found")

This is currently not completely correct and fully working but at least it shows me if dependencies disappeared, if files were added or removed and I think it is a good starting point to semi-automate the conversion from OpenEXR CMake files to Bazel.

Of course, to protect this also a CI check can be added. OpenEXR used GitHub actions. Here is a check if the Bazel build still works.

name: CI

on:
push: {}

jobs:
build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1

    - name: Mount bazel cache
    uses: actions/cache@v1
    with:
        path: "/home/runner/.cache/bazel"
        key: bazel

    - name: Install bazelisk
    run: |
        curl -LO "https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64"
        mkdir -p "${GITHUB_WORKSPACE}/bin/"
        mv bazelisk-linux-amd64 "${GITHUB_WORKSPACE}/bin/bazel"
        chmod +x "${GITHUB_WORKSPACE}/bin/bazel"
    - name: Build
    run: |
        "${GITHUB_WORKSPACE}/bin/bazel" build //...

UPDATE

Meanwhile, my pull request to OpenEXR was accepted.

Here is an example what do to to use OpenEXR with Bazel:

Create a WORKSPACE.bazel file:

# SPDX-License-Identifier: BSD-3-Clause
# Copyright Contributors to the OpenEXR Project.

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "openexr",
    # branch = "master",
    commit = "28fa44d5c24dbed3f05c38b81d75ae08d72c0cf9",
    remote = "https://github.com/AcademySoftwareFoundation/openexr",
    shallow_since = "1614357727 -0800",
)

load("@openexr//:bazel/third_party/openexr.bzl", "openexr_deps")

openexr_deps()

Create a BUILD.bazel file:

# SPDX-License-Identifier: BSD-3-Clause
# Copyright Contributors to the OpenEXR Project.

cc_binary(
    name = "Demo",
    srcs = ["main.cpp"],
    deps = ["@openexr//:OpenEXR"],
)

Create a main.cpp file:

#include <ImfRgba.h>
#include <ImfRgbaFile.h>

#include <string>

// Function copied (with minior modifications) from pbrt-v3 (https://github.com/mmp/pbrt-v3) which is under BSD-2-Clause License
static void WriteImageEXR(const std::string &name, const float *pixels,
                        int xRes, int yRes, int totalXRes, int totalYRes,
                        int xOffset, int yOffset) {
    using namespace Imf;
    using namespace Imath;

    Rgba *hrgba = new Rgba[xRes * yRes];
    for (int i = 0; i < xRes * yRes; ++i)
        hrgba[i] = Rgba(pixels[3 * i], pixels[3 * i + 1], pixels[3 * i + 2]);

    // OpenEXR uses inclusive pixel bounds.
    Box2i displayWindow(V2i(0, 0), V2i(totalXRes - 1, totalYRes - 1));
    Box2i dataWindow(V2i(xOffset, yOffset),
                    V2i(xOffset + xRes - 1, yOffset + yRes - 1));

    try {
        RgbaOutputFile file(name.c_str(), displayWindow, dataWindow,
                            WRITE_RGB);
        file.setFrameBuffer(hrgba - xOffset - yOffset * xRes, 1, xRes);
        file.writePixels(yRes);
    } catch (const std::exception &exc) {
        throw std::runtime_error("Error writing");
    }

    delete[] hrgba;
}

int main() {
    float data [3*100*100];

    for(int y = 0; y < 100; ++y)
        for(int x = 0; x < 100; ++x) {
            data[(y*100+x)*3+0] = 0.f; 
            data[(y*100+x)*3+1] = 1.f; 
            data[(y*100+x)*3+2] = 0.f; 
        }

    WriteImageEXR("test.exr", data, 100, 100, 100, 100, 0, 0);

    return 0;
}

Execute bazel run //:Demo to generate the test.exr OpenEXR image file.

You can find the source code of this demo (and maybe an updated version of it) here.