How to use Conan packages in your Waf project

Oct 23, 2023

For a short quickstart guide, check out this article instead

A Conan package is an archive of a compiled library or tool for a single architecture/platform. For example, a zlib package may have header files, a .so file, and some configuration info. If you want to use that package in your project, you need to change your compiler/linker flags to e.g. find those headers and library files.

Conan's solution to this is called "generators" (not to be confused with Waf's "task generators"). A Conan generator's job is to generate the config/data/options that your build system needs in order to properly use a package. This is usually seamless, or even invisible to an end user.

Since there was no official Conan generator for waf, I wrote one here. That will give you access to the over 1,600 libraries/tools on Conan center in your waf projects.

Getting Started
Sample Project

Getting Started

Since this requires using my custom generator, you need to do some setup to install it.

There are [two ways] (https://docs.conan.io/2/reference/extensions/custom_generators.html) to use a custom generator, but here I'll show the "python_requires" method (check this article for an alternate method)

git clone https://github.com/AlexRamallo/waf-conan-generator
cd waf-conan-generator
conan export .

The conan export command will copy the generator to your local Conan cache.

Sample Project

As a basic example, I'll show how to use the spdlog library to build a program with fancy logging features. Here's the full source code for the app we're about to build:

//main.cpp
#include <spdlog/spdlog.h>

int main(int argc, char *argv[]){
    spdlog::info("Hello, waf! {}", argv[0]);
    return 0;
}

A good first step to using a library is to search Conan Center and see if there's a recipe for it. Since there is a recipe for spdlog, we now have to choose the version that we want to use. For our purposes, the latest version spdlog/1.12.0 will work.

Terminology note: A recipe is a script that Conan uses to create usable packages for a library.

Creating the conanfile

The conanfile is how we tell Conan which dependencies we want. There are two formats, but for this demo, we need to use conanfile.py due to the way we plan to use the custom generator.

Note: It is possible to use conanfile.txt instead for this if you install the custom generator globally

#conanfile.py
from conan import ConanFile

class MyProject(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    python_requires = ["wafgenerator/0.1"]
    requires = ["spdlog/1.12.0"]

    def generate(self):
        Waf = self.python_requires["wafgenerator"].module.Waf
        wgen = Waf(self)
        wgen.generate()

I won't do a deep dive into Conanfiles here, just explain how we use the custom Waf generator we installed earlier.

The python_requires attribute tells Conan that our conanfile depends on a package, and it should be built/downloaded before processing anything else. The package we're depending on is obviously the waf generator. This makes the module accessible to us in the conanfile, which we need to use in the generate () method (duh).

If you've used Conan before, then this should look similar to the modern CMakeDeps and CMakeToolchain generators. The only difference is that we need to load the Python modules from the self.python_requires attribute, whereas with CMake you typically do a global import (since those are built-in to Conan).

That's all we need to do from the Conan side. When we call conan install for our project, Conan will use the Waf generator to generate some Waf-specific file that we can use from our wscript.

Loading package info from Waf

This generator works by generating a waf tool which, when loaded, will populate your Waf environment with the correct information needed to compile/link with dependencies in your Conan graph.

The generated waf tool has everything you need, so there's no need to ship anything alongside the standard waf script.

#wscript
def options(opt):
    opt.load('compiler_cxx')

def configure(conf):
    conf.load('conan_deps', tooldir='build')
    conf.load('compiler_cxx')

In the configure method, we load the tool (called "conan_deps") from the build folder, which is where we expect conan to write its output to when we call conan install later. You don't need to have conan generate files in the build folder, but it's convenient since we can clean it with a simple waf distclean, and don't need to add anything extra to our .gitignore besides the usual.

This tool will do a couple of things to ensure we can access the libraries/tools in our Conan dependency graph. The main thing it will do is populate the current context's environment with the variables that we need to compile a C++ project with the spdlog package. These variables are namespaced as described in section 10.3.3 of the Waf book.

In other words, include flags like -I/conan/spdlog/include will not be added to conf.env.INCLUDES, but instead will be added to conf.env.INCLUDES_spdlog. So if we want to use spdlog to compile some code, we need to add the usename "spdlog" to the use attribute of the task generator.

Additionally, it will modify sys.path, os.environ, and conf.environ to include environment variables from the Conan build environment. This is important in case we want to use tools as Conan packages (including waf tools/python scripts).

For example, flatbuffers is a serialization library that includes a custom schema language and a compiler for that language. Thanks to the environment manipulations performed by the tool, we can expect a call to conf.find_program("flatc") to locate the flatc binary inside of the Conan package, even if it's not installed on our system.

Using packages

Using a package in your Waf generators is very easy, just add the package you want to the use attribute and include "conan" in your features attribute:

def build(bld):
    bld(
        features = "cxx cxxprogram conan",
        source = "main.cpp",
        use = "spdlog",
        target = "app"
    )

The "conan" feature is necessary in order to expand the use attribute with transient dependencies. For example, spdlog actually depends on a library called fmt. So if we only use the spdlog package, we'll get compile/link time errors.

While you could manually add "fmt" to the use attribute, this is a bad idea since it makes our wscript reliant on a transient dependency, and makes us responsible for correctly representing the dependency graph... which is pointless because Conan did all of that already.

So the "conan" feature method is there to handle this case. As a rule of thumb: if a package is NOT listed in your conanfile's requirements, then it should NEVER be listed in a use attribute. If you're getting compile/link errors, then it's either a bug with my generator, or some other problem.

Build and run it

Let's review the structure of our sample:

─ sample_project
  ├── conanfile.py
  ├── main.cpp
  └── wscript

The first thing we need to do is execute conan install (make sure to run these inside the sample_project folder):

conan install . -of=build --build=missing
# "-of" stands for "output folder"
# "--build=missing" tells Conan to locally compile any packages that are not available as downloads

Note: for the guide, you should ensure the output folder is the same as waf's 'out' folder

This will cause Conan to do the following things:

build or download a package for the waf generator (the one we listed as a python_requires)
build or download all packages needed to satisfy our requirements (that's spdlog and its dependency fmt)
Execute the waf generator, producing the conan_deps.py waf tool in the build folder

Once that process completes, we can simply configure/build our waf app as usual:

Configure

$ waf configure
Setting top to                           : /sample_project 
Setting out to                           : /sample_project/build 
Conan usename                            : spdlog_libspdlog 
Conan usename                            : spdlog 
Conan usename                            : fmt__fmt 
Conan usename                            : fmt 
Checking for 'gxx' (C++ compiler)        : /usr/bin/g++ 
'configure' finished successfully (0.057s)

During configure, you'll see a bunch of "Conan usename" lines being logged. These are the names that are available to you when adding to the use attribute of your C++ task generators. Above, you can see that there's also an entry for fmt, which is a dependency of spdlog.

You'll also notice that there are some weird ones, like spdlog_libspdlog and fmt__fmt. These are actually Components.

In this example, the spdlog package defines a component called "spdlog::libspdlog", and the fmt package defines one called "fmt::_fmt", and my waf generator mangles those a bit for compatibility reasons. The linked article about components explains the what and why of Components, but generally, you can usually get away with just using the root package rather than having to worry about individual components (although it depends on the package). That's why we use "spdlog" in our example rather than "spdlog_libspdlog".

Build

$ waf build -v
Waf: Entering directory `/sample_project/build'
[1/2] Compiling main.cpp
13:44:40 runner ['/usr/bin/g++', '-D_GLIBCXX_USE_CXX11_ABI=1', '--std', 'c++17', '-O3', '-DNDEBUG', '-I/home/aramallo/.conan2/p/b/spdlob874f902f94c3/p/include', '-I/home/aramallo/.conan2/p/b/fmta627fb0eeee1d/p/include', '-DSPDLOG_FMT_EXTERNAL', '-DSPDLOG_COMPILED_LIB', '../main.cpp', '-c', '-o/sample_project/build/main.cpp.1.o']
[2/2] Linking build/app
13:44:40 runner ['/usr/bin/g++', 'main.cpp.1.o', '-o/sample_project/build/app', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/home/aramallo/.conan2/p/b/spdlob874f902f94c3/p/lib', '-L/home/aramallo/.conan2/p/b/spdlob874f902f94c3/p/lib', '-L/home/aramallo/.conan2/p/b/fmta627fb0eeee1d/p/lib', '-L/home/aramallo/.conan2/p/b/fmta627fb0eeee1d/p/lib', '-lspdlog', '-lpthread', '-lfmt', '-lm']
Waf: Leaving directory `/sample_project/build'
'build' finished successfully (0.491s)

Looking at the verbose output above, we can clearly see the extra flags added by the waf generator. Besides the obvious -I and -L flags, there are also these:

--std c++17
-O3
-DNDEBUG
-D_GLIBCXX_USE_CXX11_ABI=1

Those flags were added due to the Conan profile that I used to build it. Yours may be slightly different, but here's mine for reference:

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

Addtionally, the following definitions were added when compiling:

-DSPDLOG_FMT_EXTERNAL
-DSPDLOG_COMPILED_LIB

Those are just some config defs needed by spdlog, which are declared in its package.

Run

Our sample app statically links with spdlog and fmt because that's the default behavior for both of those packages. That means we can run the app by just executing it as you'd expect:

$ ./build/app
[2023-10-23 13:54:35.972] [info] Hello, waf! ./build/app

However, if we'd have dynamically linked with those dependencies, then we would need to either install the shared libraries to our operating system, or add them to the loader's search path.

Note: if you want to try this, add this attribute to your conanfile (e.g. under requires = ...) and then repeat all the steps above: default_options = { "*/*:shared": True }

Luckily, we don't have to worry about that because Conan install also generated some helpful scripts when we called conan install. These are in the build folder:

conanbuild.sh (or .bat on windows)
deactivate_conanbuild.sh
conanrun.sh
deactivate_conanrun.sh

These will activate and deactivate either the build or run environment. Among other things, it ensures that the correct dynamic libraries that live in our Conan cache will be found when executing our application.

An interesting usecase for this feature is to use Conan to obtain build tools. For example, you could build a CMake project even if you don't have CMake installed by using the Conan Center recipe for it (which can build CMake from source if needed):

$ conan install --tool-requires=cmake/3.27.7 --build=missing

$ source conanbuild.sh

#cmake is now in our PATH

$ cmake --version
cmake version 3.27.7

CMake suite maintained and supported by Kitware (kitware.com/cmake).

#...and it lives in our Conan cache

$ command -v cmake
/home/aramallo/.conan2/p/cmake857e6d21f32d2/p/bin/cmake

$ source deactivate_conanbuild.sh 
Restoring environment

#our PATH is restored, so cmake is gone!

$ cmake --version
zsh: command not found: cmake

And that's it! You can see the full source for this sample project here.

The generator repo also has some other sample projects in the tests folder which use some more advanced features of Conan, like using build tools in your wscripts and cross-compilation.

Conan and Waf Quickstart Guide (Index)