Render time KPI

A good key performance indicator (KPI) for a renderer is render time. Everyone who waited for a few hours for rendering to get finished knows that render time is important. As a user and even as a developer you always hope for short render times. The worst thing that can happen is that more and more features are added to a renderer software that makes the whole rendering process slow and slower over time. This hurts a lot when the render engine does not even make use of any of the newly introduced features or only a non-functional refactoring is causing longer render times.

To avoid such situations it is a good idea to measure the time it takes to render different predefined scenes after each change in your codebase so one can easily track the influence of code changes on render time.

Here is a plot of the render time per code change for the rendering of the Ajax scene using ambient occlusion. The last recent commit on this graph is leftmost and the oldest commit correspondingly on the right-hand side:

Ajax

Of course, when doing such comparisons you have also to care about image quality assessment. The generated images of the same scene should have comparable image quality, otherwise, render time comparison does not make sense.

When looking at the above plot it can easily depict that the render time for the Ajax went from about 20 seconds to 37 seconds, which is bad. The big question is why did this happen? The suspicious commit has the commit message “Add a bunch of BUILD files”. My render system is using Bazel as a build system. What I did was splitting my single static cc_library into three static libraries. The performance drop was not expected by me.

Just for the records: Along with the render time additional information is stored such as the used operating system or the used CPU:

- git_commit:
    hash: f597d404641d6a68051f8f434c28eb7d6abc4a18
    message: "Add a bunch of BUILD files"
    benchmark_runs:
        - scene: Okapi/scenes/ajax/ajax.ao.okapi.xml
          command: "bazel run --config=gcc9 --compilation_mode=opt //okapi.cli:okapi -- ${HOME}/dev/Piper/devertexwahn/  scenes/okapi/ajax/ajax.ao.okapi.xml"
          datetime: 2021-08-21 11:19:45
          render_time_in_seconds: 37.0412
          cpu_model_name: AMD Ryzen 9 3900X 12-Core Processor
          os: Linux (5.11.0-27-generic) 20.04.3 LTS (Focal Fossa)
- git_commit:
    hash: ee15d48eb72d03953e6eb1189dd58710ece38d0f
    message: "Remove note needed includes"
    benchmark_runs:
        - scene: Okapi/scenes/ajax/ajax.ao.okapi.xml
          command: "bazel run --config=gcc9 --compilation_mode=opt //okapi.cli:okapi -- ${HOME}/dev/Piper/devertexwahn/  scenes/okapi/ajax/ajax.ao.okapi.xml"
          datetime: 2021-08-21 11:20:32
          render_time_in_seconds: 21.0938
          cpu_model_name: AMD Ryzen 9 3900X 12-Core Processor
          os: Linux (5.11.0-27-generic) 20.04.3 LTS (Focal Fossa)

Another question that comes into my mind: What is about other scenes? Did the performance drop also for other scenes? I took the performance measurement also for another scene:

BMW_315_DA2

Surprisingly the render time did improve for the BMW DA2 scene, by more than 15 minutes.

What did change?

The intention of the change was a simple refactoring. Before the change my renderer called Okapi was inside a (static) library called okapi. I decided to split this library into four libraries:

core: Logging
math: Vectors, Matrices
imaging: IO for misc image file formates (e.g. OpenEXR)
okapi: Everything else

I generated a patch set of the change using:

git format-patch -1 f597d404641d6a68051f8f434c28eb7d6abc4a18

Here is the content of the patch set:

From f597d404641d6a68051f8f434c28eb7d6abc4a18 Mon Sep 17
From: Julian Amann <julian.amann@tum.de>
Subject: [PATCH] Add a bunch of BUILD files

---
Okapi/BUILD.bazel                             | 46 ++++---------------
Okapi/okapi/core/BUILD.bazel                  | 22 +++++++++
Okapi/okapi/imaging/BUILD.bazel               | 27 +++++++++++
Okapi/okapi/material.h                        |  2 +-
Okapi/okapi/math/BUILD.bazel                  | 33 +++++++++++++
Okapi/okapi/{core => }/object_factory.cpp     |  2 +-
Okapi/okapi/{core => }/object_factory.h       |  2 +-
Okapi/okapi/{core => }/property_set.cpp       |  2 +-
Okapi/okapi/{core => }/property_set.h         |  2 +-
.../okapi/{core => }/property_set_formatter.h |  0
Okapi/okapi/sampler.h                         |  2 +-
Okapi/okapi/scene/load_scene.cpp              |  4 +-
Okapi/okapi/scene/shape/disk.h                |  2 +-
Okapi/okapi/sensor/reconstruction_filter.h    |  2 +-
Okapi/okapi/sensor/sensor.h                   |  2 +-
.../tests/okapi/core/test_object_factory.cpp  |  2 +-
Okapi/tests/okapi/core/test_property_set.cpp  |  2 +-
.../core/test_property_set_formatter.cpp      |  2 +-
Okapi/tests/okapi/scene/test_scene.cpp        |  2 +-
19 files changed, 105 insertions(+), 53 deletions(-)
create mode 100644 Okapi/okapi/core/BUILD.bazel
create mode 100644 Okapi/okapi/imaging/BUILD.bazel
create mode 100644 Okapi/okapi/math/BUILD.bazel
rename Okapi/okapi/{core => }/object_factory.cpp (95%)
rename Okapi/okapi/{core => }/object_factory.h (97%)
rename Okapi/okapi/{core => }/property_set.cpp (87%)
rename Okapi/okapi/{core => }/property_set.h (98%)
rename Okapi/okapi/{core => }/property_set_formatter.h (100%)

diff --git a/devertexwahn/BUILD.bazel b/devertexwahn/BUILD.bazel
index 84fbfdcf..279e1fbe 100644
--- a/devertexwahn/BUILD.bazel
+++ b/devertexwahn/BUILD.bazel
@@ -8,19 +8,10 @@ exports_files(["LICENSE.md"])
cc_library(
    name = "okapi",
    srcs = [
-        "okapi/core/object.cpp",
-        "okapi/core/object_factory.cpp",
-        "okapi/core/property_set.cpp",
-        "okapi/core/timer.cpp",
-        "okapi/imaging/color.cpp",
-        "okapi/imaging/image.cpp",
-        "okapi/imaging/image_quality_assessment.cpp",
-        "okapi/imaging/io.cpp",
        "okapi/integrator/path_specular_transmission.cpp",
        "okapi/material.cpp",
-        "okapi/math/bounds.cpp",
-        "okapi/math/geometry.inc",
-        "okapi/math/refraction.cpp",
+        "okapi/object_factory.cpp",
+        "okapi/property_set.cpp",
        "okapi/rendering.cpp",
        "okapi/scene/load_scene.cpp",
        "okapi/scene/shape/shape.cpp",
@@ -33,19 +24,6 @@ cc_library(
        "okapi.h",
        "okapi/canvas/label.h",
        "okapi/canvas/svg_canvas.h",
-        "okapi/core/logging.h",
-        "okapi/core/namespace.h",
-        "okapi/core/non_copyable.h",
-        "okapi/core/object.h",
-        "okapi/core/object_factory.h",
-        "okapi/core/property_set.h",
-        "okapi/core/property_set_formatter.h",
-        "okapi/core/reference_counted.h",
-        "okapi/core/timer.h",
-        "okapi/imaging/color.h",
-        "okapi/imaging/image.h",
-        "okapi/imaging/image_quality_assessment.h",
-        "okapi/imaging/io.h",
        "okapi/integrator/ambient_occlusion.h",
        "okapi/integrator/hit_integrator.h",
        "okapi/integrator/integrator.h",
@@ -56,19 +34,9 @@ cc_library(
        "okapi/intersector/embree_intersector.h",
        "okapi/intersector/intersector.h",
        "okapi/material.h",
-        "okapi/math/bounds.h",
-        "okapi/math/frame.h",
-        "okapi/math/geometry.h",
-        "okapi/math/intersection.h",
-        "okapi/math/matrix.h",
-        "okapi/math/normal.h",
-        "okapi/math/point.h",
-        "okapi/math/ray.h",
-        "okapi/math/refraction.h",
-        "okapi/math/transform.h",
-        "okapi/math/util.h",
-        "okapi/math/vector.h",
-        "okapi/math/warping.h",
+        "okapi/object_factory.h",
+        "okapi/property_set.h",
+        "okapi/property_set_formatter.h",
        "okapi/rendering.h",
        "okapi/sampler.h",
        "okapi/scene/load_scene.h",
@@ -87,11 +55,13 @@ cc_library(
    ],
    visibility = ["//visibility:public"],
    deps = [
+        "//okapi/core",
+        "//okapi/imaging",
+        "//okapi/math",
        "@boost//:algorithm",
        "@eigen",
        "@embree",
        "@fmt",
-        "@glog",
        "@libpng",
        "@oneTBB//:tbb",
        "@com_openexr//:OpenEXR",
diff --git a/devertexwahn/okapi/core/BUILD.bazel b/devertexwahn/okapi/core/BUILD.bazel
new file mode 100644
index 00000000..b415d61d
--- /dev/null
+++ b/devertexwahn/okapi/core/BUILD.bazel
@@ -0,0 +1,22 @@
+"""
+    SPDX-FileCopyrightText: 2021 Julian Amann <dev@vertexwahn.de>
+    SPDX-License-Identifier: Apache-2.0
+"""
+
+cc_library(
+    name = "core",
+    srcs = [
+        "object.cpp",
+        "timer.cpp",
+    ],
+    hdrs = [
+        "logging.h",
+        "namespace.h",
+        "non_copyable.h",
+        "object.h",
+        "reference_counted.h",
+        "timer.h",
+    ],
+    visibility = ["//visibility:public"],
+    deps = ["@glog"],
+)
diff --git a/devertexwahn/okapi/imaging/BUILD.bazel b/devertexwahn/okapi/imaging/BUILD.bazel
new file mode 100644
index 00000000..1b7b587c
--- /dev/null
+++ b/devertexwahn/okapi/imaging/BUILD.bazel
@@ -0,0 +1,27 @@
+"""
+    SPDX-FileCopyrightText: 2021 Julian Amann <dev@vertexwahn.de>
+    SPDX-License-Identifier: Apache-2.0
+"""
+
+cc_library(
+    name = "imaging",
+    srcs = [
+        "color.cpp",
+        "image.cpp",
+        "image_quality_assessment.cpp",
+        "io.cpp",
+    ],
+    hdrs = [
+        "color.h",
+        "image.h",
+        "image_quality_assessment.h",
+        "io.h",
+    ],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//okapi/core",
+        "//okapi/math",
+        "@libpng",
+        "@com_openexr//:OpenEXR",
+    ],
+)
diff --git a/devertexwahn/okapi/material.h b/devertexwahn/okapi/material.h
index 3ae6d8b8..0ff8a0f2 100644
--- a/devertexwahn/okapi/material.h
+++ b/devertexwahn/okapi/material.h
@@ -8,7 +8,7 @@
#define Okapi_Material_11d72dcc_a416_4530_b706_b1e69711c5c5_h

#include "okapi/core/namespace.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/core/reference_counted.h"
#include "okapi/imaging/color.h"

diff --git a/devertexwahn/okapi/math/BUILD.bazel b/devertexwahn/okapi/math/BUILD.bazel
new file mode 100644
index 00000000..afacafd5
--- /dev/null
+++ b/devertexwahn/okapi/math/BUILD.bazel
@@ -0,0 +1,33 @@
+"""
+    SPDX-FileCopyrightText: 2021 Julian Amann <dev@vertexwahn.de>
+    SPDX-License-Identifier: Apache-2.0
+"""
+
+cc_library(
+    name = "math",
+    srcs = [
+        "bounds.cpp",
+        "geometry.inc",
+        "refraction.cpp",
+    ],
+    hdrs = [
+        "bounds.h",
+        "frame.h",
+        "geometry.h",
+        "intersection.h",
+        "matrix.h",
+        "normal.h",
+        "point.h",
+        "ray.h",
+        "refraction.h",
+        "transform.h",
+        "util.h",
+        "vector.h",
+        "warping.h",
+    ],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//okapi/core",
+        "@eigen",
+    ],
+)
diff --git a/devertexwahn/okapi/core/object_factory.cpp b/devertexwahn/okapi/object_factory.cpp
similarity index 95%
rename from Okapi/okapi/core/object_factory.cpp
rename to Okapi/okapi/object_factory.cpp
index e71f9bcd..028de81f 100644
--- a/devertexwahn/okapi/core/object_factory.cpp
+++ b/devertexwahn/okapi/object_factory.cpp
@@ -3,7 +3,7 @@
*  SPDX-License-Identifier: Apache-2.0
*/

-#include "okapi/core/object_factory.h"
+#include "okapi/object_factory.h"

OKAPI_BEGIN_NAMESPACE

diff --git a/devertexwahn/okapi/core/object_factory.h b/devertexwahn/okapi/object_factory.h
similarity index 97%
rename from Okapi/okapi/core/object_factory.h
rename to Okapi/okapi/object_factory.h
index c96d01f6..1dee825c 100644
--- a/devertexwahn/okapi/core/object_factory.h
+++ b/devertexwahn/okapi/object_factory.h
@@ -7,7 +7,7 @@
#ifndef Okapi_ShapeFactory_b184faf0_dc66_451e_a52f_615a4caa7c35_h
#define Okapi_ShapeFactory_b184faf0_dc66_451e_a52f_615a4caa7c35_h

-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/core/reference_counted.h"
#include "okapi/core/namespace.h"

diff --git a/devertexwahn/okapi/core/property_set.cpp b/devertexwahn/okapi/property_set.cpp
similarity index 87%
rename from Okapi/okapi/core/property_set.cpp
rename to Okapi/okapi/property_set.cpp
index 7ed8f1e4..e1bf0857 100644
--- a/devertexwahn/okapi/core/property_set.cpp
+++ b/devertexwahn/okapi/property_set.cpp
@@ -3,7 +3,7 @@
*  SPDX-License-Identifier: Apache-2.0
*/

-#include "property_set.h"
+#include "okapi/property_set.h"

OKAPI_BEGIN_NAMESPACE

diff --git a/devertexwahn/okapi/core/property_set.h b/devertexwahn/okapi/property_set.h
similarity index 98%
rename from Okapi/okapi/core/property_set.h
rename to Okapi/okapi/property_set.h
index 91a43e07..17291fe1 100644
--- a/devertexwahn/okapi/core/property_set.h
+++ b/devertexwahn/okapi/property_set.h
@@ -9,7 +9,7 @@

#include "okapi/core/namespace.h"
#include "okapi/core/object.h"
-#include "okapi/core/property_set_formatter.h"
+#include "okapi/property_set_formatter.h"
#include "okapi/core/reference_counted.h"

#include "fmt/core.h"
diff --git a/devertexwahn/okapi/core/property_set_formatter.h b/devertexwahn/okapi/property_set_formatter.h
similarity index 100%
rename from Okapi/okapi/core/property_set_formatter.h
rename to Okapi/okapi/property_set_formatter.h
diff --git a/devertexwahn/okapi/sampler.h b/devertexwahn/okapi/sampler.h
index 65d617c8..53949398 100644
--- a/devertexwahn/okapi/sampler.h
+++ b/devertexwahn/okapi/sampler.h
@@ -9,7 +9,7 @@

#include "okapi/core/namespace.h"
#include "okapi/core/object.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/math/point.h"

#include "external/pcg/_virtual_includes/pcg/pcg_random.hpp"
diff --git a/devertexwahn/okapi/scene/load_scene.cpp b/devertexwahn/okapi/scene/load_scene.cpp
index aef74157..28e32f31 100644
--- a/devertexwahn/okapi/scene/load_scene.cpp
+++ b/devertexwahn/okapi/scene/load_scene.cpp
@@ -6,8 +6,8 @@
#include "okapi/scene/load_scene.h"

#include "okapi/core/logging.h"
-#include "okapi/core/object_factory.h"
-#include "okapi/core/property_set.h"
+#include "okapi/object_factory.h"
+#include "okapi/property_set.h"
#include "okapi/integrator/ambient_occlusion.h"
#include "okapi/integrator/hit_integrator.h"
#include "okapi/integrator/normal_integrator.h"
diff --git a/devertexwahn/okapi/scene/shape/disk.h b/devertexwahn/okapi/scene/shape/disk.h
index cb97952d..39a0a26d 100644
--- a/devertexwahn/okapi/scene/shape/disk.h
+++ b/devertexwahn/okapi/scene/shape/disk.h
@@ -8,7 +8,7 @@
#define Okapi_Disk2f_9deef3ad_50b0_4b94_b279_21c462b565e7_h

#include "okapi/core/namespace.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/math/intersection.h"
#include "okapi/scene/shape/shape.h"

diff --git a/devertexwahn/okapi/sensor/reconstruction_filter.h b/devertexwahn/okapi/sensor/reconstruction_filter.h
index dd42f3cf..1d14df6c 100644
--- a/devertexwahn/okapi/sensor/reconstruction_filter.h
+++ b/devertexwahn/okapi/sensor/reconstruction_filter.h
@@ -9,7 +9,7 @@

#include "okapi/core/namespace.h"
#include "okapi/core/object.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"

OKAPI_BEGIN_NAMESPACE

diff --git a/devertexwahn/okapi/sensor/sensor.h b/devertexwahn/okapi/sensor/sensor.h
index 25829a4a..914aa72c 100644
--- a/devertexwahn/okapi/sensor/sensor.h
+++ b/devertexwahn/okapi/sensor/sensor.h
@@ -8,7 +8,7 @@
#define Okapi_Sensor_14b27f8d_8dca_49d6_8087_3d67c31d29ec_h

#include "okapi/core/namespace.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/math/transform.h"
#include "okapi/sensor/film.h"

diff --git a/devertexwahn/tests/okapi/core/test_object_factory.cpp b/devertexwahn/tests/okapi/core/test_object_factory.cpp
index 251e9457..12841cb1 100644
--- a/devertexwahn/tests/okapi/core/test_object_factory.cpp
+++ b/devertexwahn/tests/okapi/core/test_object_factory.cpp
@@ -3,7 +3,7 @@
*  SPDX-License-Identifier: Apache-2.0
*/

-#include "okapi/core/object_factory.h"
+#include "okapi/object_factory.h"
#include "okapi/integrator/integrator.h"
#include "okapi/scene/shape/disk.h"

diff --git a/devertexwahn/tests/okapi/core/test_property_set.cpp b/devertexwahn/tests/okapi/core/test_property_set.cpp
index 91b309d5..622fdc1d 100644
--- a/devertexwahn/tests/okapi/core/test_property_set.cpp
+++ b/devertexwahn/tests/okapi/core/test_property_set.cpp
@@ -3,7 +3,7 @@
*  SPDX-License-Identifier: Apache-2.0
*/

-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/sensor/reconstruction_filter.h"

#include "gmock/gmock.h"
diff --git a/devertexwahn/tests/okapi/core/test_property_set_formatter.cpp b/devertexwahn/tests/okapi/core/test_property_set_formatter.cpp
index 88aea5a6..33f98c21 100644
--- a/devertexwahn/tests/okapi/core/test_property_set_formatter.cpp
+++ b/devertexwahn/tests/okapi/core/test_property_set_formatter.cpp
@@ -3,7 +3,7 @@
*  SPDX-License-Identifier: Apache-2.0
*/

-#include "okapi/core/property_set_formatter.h"
+#include "okapi/property_set_formatter.h"

#include "gmock/gmock.h"

diff --git a/devertexwahn/tests/okapi/scene/test_scene.cpp b/devertexwahn/tests/okapi/scene/test_scene.cpp
index 97a656e9..4eabe485 100644
--- a/devertexwahn/tests/okapi/scene/test_scene.cpp
+++ b/devertexwahn/tests/okapi/scene/test_scene.cpp
@@ -4,7 +4,7 @@
*/

#include "okapi/scene/load_scene.h"
-#include "okapi/core/property_set.h"
+#include "okapi/property_set.h"
#include "okapi/core/reference_counted.h"
#include "okapi/math/util.h"
#include "okapi/scene/shape/disk.h"
-- 
2.25.1

Conclusion so far

To figure out things like this it makes sense to track render time. Currently, I do not understand where these tremendous changes in render time come from. As a compiler GCC 9.3.0 was used here. It would be interesting to see how Clang 12 or Visual Studio 2019 behave here. I am just surprised by the impacts that rearranging libraries can have.

Vertexwahn

Render time KPI reveals unexpected negative effect of Bazel BUILD file refactoring

Render time KPI

What did change?

Conclusion so far