Skip to main content

Binary vs Executable

stackoverflow:

  • An executable file is one which can be executed; you would run it on the commandline by writing the name of the file itself as the command.
  • A binary file is simply one in a binary (i.e. non-text) format.
  • Binary files are not necessarily executable, for example a library compiled to .dll or .so form is a binary but not an executable.
  • Executable files are not necessarily binary, for example a Python script in text form can be made executable on Unix systems by writing a shebang line #!/usr/bin/python3 and setting the file's executable flag.

gcc, g++

Compiling Multiple Files​

First, if you have put your function definitions in foo.C and the corresponding function declarations in foo.h, then you have to add

#include "foo.h"

to bar.C when the functions in foo.h are used in bar.C.

Then, you can compile all files like so:

courses.cs.washington.edu

Most likely, you will be compiling separate modules and linking them into a single executable. Here's the basic idea: compile each .C file into a .o file, then link the .o files (along with any libraries) into an executable. Of course, one of these .C files has to define the main() or else the linker will complain. Suppose we have main.C, foo.C and bar.C and want to create an executable fubar, and suppose further that we need the math library:

g++ -c -o foo.o foo.C
g++ -c -o main.o main.C
g++ -c -o bar.o bar.C
g++ -o fubar foo.o main.o bar.o -lm

The first three commands generate foo.o, main.o and bar.o respectively. The last line links them together along with the math library, libm.a.

Standards​

Default Standard​

stackoverflow:

Which C++ standard is the default when compiling with g++?

If your version of g++ is later than 4.7 I think you can find the default version of C++ standard supported like so:

g++ -dM -E -x c++  /dev/null | grep -F __cplusplus

Available Standards​

stackoverflow:

gcc -v --help 2> /dev/null | sed -n '/^ *-std=\([^<][^ ]\+\).*/ {s//\1/p}'

Flags​

in official doc

also see Basic Overview

Order of Arguments​

stackoverflow

Generally the order of arguments doesn't matter, but there are of course exceptions. For example if you provide multiple -O flags it will be the last one that is used, the same for other flags.

Libraries are a little different though, because for them the order is significant. If object file or library A depends on library B, then A must come before B on the command line. This is because of how the linker scans for symbols: When you use a library the linker will check if there are any symbols that could be resolved. Once this scan is over the library is discarded and will not be searched again.

This means when you have -lorocos-kdl -lkdl_parser test.cpp the linker will scan the libraries orocos-kdl and kdl_parser first, notice that there aren't dependencies on these library, no symbols from the libraries are needed, and continue with the object file generated by the source file.

When you change the order to test.cpp -lorocos-kdl -lkdl_parser the linker will be able to resolve the undefined symbols referenced by test.cpp when it comes to the libraries.

-c​

source: man gcc

"Compile or assemble the source files, but do not link. The linking stage simply is not done. The ultimate output is in the form of an object file for each source file. By default, the object file name for a source file is made by replacing the suffix .c, .i, .s, etc., with .o. Unrecognized input files, not requiring compilation or assembly, are ignored."

-l​

Linking with a shared library (e.g. -lfoo links against libfoo.so).

GCC first searches for libraries in /usr/local/lib, then in /usr/lib. Following that, it searches for libraries in the directories specified by the -L parameter, in the order specified on the command line. source

  • -pthread vs -lpthread, stackoverflow
    • -pthread tells the compiler to link in the pthread library as well as configure the compilation for threads.
    • Using the -lpthread option only causes the pthread library to be linked - the pre-defined macros don't get defined.
    • Bottom line: you should use the -pthread option.

-L​

Tell GCC where to find a shared library (e.g. libfoo.so).

-include​

Rarely used flag. Only relevant for some rare use cases: stackoverflow.

In official doc

-I​

In official doc

Show default include directories:

# for C
gcc -xc -E -v -

# for C++
gcc -xc++ -E -v -

see How to tell g++ compiler where to search for include files?

ld

source: man ld

ld - The GNU linker

ld [options] objfile ...

ld combines a number of object and archive files, relocates their data and ties up symbol references. Usually the last step in compiling a program is to run ld.

source: stackoverflow

The name is actually short for "load", and "loader" was what linkers were originally called

ldconfig

source: man ldconfig

ldconfig - configure dynamic linker run-time bindings

ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib (on some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries).

The cache is used by the run-time linker, ld.so or ld-linux.so. ldconfig checks the header and filenames of the libraries it encounters when determining which versions should have their links updated. ldconfig should normally be run by the superuser as it may require write permission on some root owned directories and files.

ldconfig will look only at files that are named lib*.so* (for regular shared objects) or ld-*.so* (for the dynamic loader itself). Other files will be ignored.

ldd

man ldd: ldd prints the shared objects (shared libraries) required by each program or shared object specified on the command line.

Use cases:

  • ldd test shows which dependencies were not found.
  • ldd test | grep foo to check which instance of the shared library libfoo.so is used by the program test.

configure script

  • purpose of configure scripts
    • "Using configure scripts is an automated method of generating makefiles before compilation to tailor the software to the system on which the executable is to be compiled and run."
  • check "failed" warnings
    • The "failed" configure tests are not typically critical, they only try to find different alternatives, or to enable/disable certain futures. If a required test fails it will be very evident by a large error printout at the end of ./configure execution.

configure workflow​

mkdir build/
cd build/

Then

  • run ../configure,
  • check in the output which dependencies are missing,
  • if libXYZ is missing, install sudo apt install libXYZ libXYZ-dev (libXYZ-dev is necessary for the headers)
  • run ../configure again
  • check in the output which dependencies are missing now,
  • ... etc., until ../configure finishes without errors
    • usually there will be some message like Now type 'make' to build at the end

Options​

Often ../configure can be run with some flags.

Run ./configure --help to show all options stackoverflow.

Example:

../configure --disable-check
  • if the check package is a dependency (Can be disabled using the --disable-check configure flag)
    • check is used for build-time tests and does not affect functionality.

Example Output​

When all dependencies are met:

bra-ket@braket-pc:~/git/rofi/build$ ../configure --disable-check 
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for flex... flex
checking lex output file root... lex.yy
checking lex library... -lfl
checking whether yytext is a pointer... yes
checking for bison... bison -y
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking if bison is the parser generator... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for gawk... no
checking for mawk... mawk
checking for flex version... 2.6.4
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking whether make sets $(MAKE)... yes
checking whether make supports the include directive... yes (GNU style)
checking whether make supports nested variables... yes
checking dependency style of gcc... gcc3
checking whether make supports nested variables... (cached) yes
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking for gcc option to accept ISO C99... none needed
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
...
configure: creating ./config.status
config.status: creating Makefile
config.status: creating doc/rofi.doxy
config.status: creating pkgconfig/rofi.pc
config.status: creating config.h
config.status: executing depfiles commands

-------------------------------------
Desktop File drun            Enabled
Window switcher mode         Enabled
Asan address sanitize        Disabled
Code Coverage                Disabled
Check based tests            Disabled
-------------------------------------
Now type 'make' to build

make

Using less cores:

N_Cores=$(( $(nproc --all) / 2 ))
make -j $N_Cores

Makefile​

Source: doc

Rule​

  • A rule appears in the makefile and says
    • when and how to remake certain files, called the rule's targets (most often only one per rule).
    • It lists the other files that are the prerequisites of the target,
    • and the recipe to use to create or update the target.
  • the all target (aka "default goal")
    • The order of rules is not significant, except for determining the default goal: the target for make to consider, if you do not otherwise specify one. The default goal is the first target of the first rule in the first makefile.
      • (...) We usually write the makefile so that the first rule is the one for compiling the entire program or all the programs described by the makefile (often with a target called all).

Recipe​

The recipe of a rule consists of one or more shell command lines to be executed, one at a time, in the order they appear. Typically, the result of executing these commands is that the target of the rule is brought up to date.

Users use many different shell programs, but recipes in makefiles are always interpreted by /bin/sh unless the makefile specifies otherwise.

Variables​

VAR ?= VALUE

  • set to a value only if VAR not already set

Pre-defined Variables​

gnu.org: 10.3 Variables Used by Implicit Rules

Examples:

  • CC - Program for compiling C programs; default cc.
  • CXX - Program for compiling C++ programs; default g++.

Phony Targets​

Common phony targets: all, install, clean, distclean, TAGS, info, check

see simple explanation

  • in comments:
    • @eSKay: 'why is it called 'phony'?' -- because it's not a real target. That is, the target name isn't a file that is produced by the commands of that target. - Bernard
    • "phony" means fraudulent; fake; having a misleading appearance

CMake

Note: You must always link shared libraries (.so files) and include header files. best stackoverflow explanation

  • i.e. just linking an .so library does not automatically make all the header files of this library available! include_directories must be specified for the header files.

Basic Workflow​

mkdir build
cd build/
cmake ..
make   # or to get more information: `make VERBOSE=1`
sudo make install
sudo make uninstall   # only possible if you specified an uninstall target in CMakeLists.txt

ccmake​

Install:

sudo apt install cmake-curses-gui

Statically Linked Libraries / Archives (.a files) vs Shared Libraries / Shared Objects (.so files)​

see stackoverflow

Understanding Shared Libraries (.so files)​

Best Tutorial

Three files needed:

File 1: foo.h:

#ifndef foo_h__
#define foo_h__
 
extern void foo(void);
 
#endif  // foo_h__

File 2: foo.c:

#include <stdio.h>
 
 
void foo(void)
{
    puts("Hello, I am a shared library");
}

File 3: main.c:

#include <stdio.h>
#include "foo.h"
 
int main(void)
{
    puts("This is a shared library test...");
    foo();
    return 0;
}

Basic steps:

$ gcc -c -Wall -Werror -fpic foo.c   # fpic: position independent code (see below)
$ gcc -shared -o libfoo.so foo.o
$ gcc -Wall -o test main.c -lfoo
/usr/bin/ld: cannot find -lfoo
collect2: ld returned 1 exit status
$ gcc -L/home/username/foo -Wall -o test main.c -lfoo
$ ./test
./test: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory
$ echo $LD_LIBRARY_PATH
$ LD_LIBRARY_PATH=/home/username/foo:$LD_LIBRARY_PATH
$ ./test
./test: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory
$ export LD_LIBRARY_PATH=/home/username/foo:$LD_LIBRARY_PATH
$ ./test
This is a shared library test...
Hello, I am a shared library

Position independent code: PIC is code that works no matter where in memory it is placed. Because several different programs can all use one instance of your shared library, the library cannot store things at fixed addresses, since the location of that library in memory will vary from program to program.

If we want to install our library so everybody on the system can use it:

$ cp /home/username/foo/libfoo.so /usr/lib
$ chmod 0755 /usr/lib/libfoo.so
$ ldconfig
$ ldconfig -p | grep foo
libfoo.so (libc6) => /usr/lib/libfoo.so
$ unset LD_LIBRARY_PATH
$ gcc -Wall -o test main.c -lfoo
$ ldd test | grep foo
libfoo.so => /usr/lib/libfoo.so (0x00a42000)
$ ./test
This is a shared library test...
Hello, I am a shared library

Think in modules! (Best CMakeLists.txt Intro!)​

Start simple and then complicate the project further!

See this stackoverflow post!

Simple example project: stackoverflow

Quick Uninstall​

from stackoverflow:

install_manifest.txt file is created when you run make install.

So just run:

xargs rm < install_manifest.txt

Add an Uninstall Target, uninstall.cmake​

Add an uninstall target to the CMAKE generated Makefile:

see github gist

Show Variables​

see stackoverflow

To show the value of CMakeLists.txt variables, add a line like so:

message(STATUS "Value of VAR = ${VAR}")

The message should be visible in the terminal after running cmake .. (look carefully!).

Or: Add set(CMAKE_VERBOSE_MAKEFILE ON) at the top of your CMakeLists.txt.

In events.log (colcon):

If you are using colcon, you can also find the message in the events.log. Run

cat log/latest_build/events.log

and search through all the StdoutLines to see the value of VAR.

Check if a library is installed​

From stackoverflow:

ldconfig -p | grep libjpeg
ldconfig -p | grep libnvinfer_plugin.so
pkg-config --list-all | grep jpeg
dpkg -l | grep libjpeg

pkg-config​

see Overview

commanddescription
pkg-config --list-allShow all the packages that have a .pc extension on your system. Contains all packages listed in the PKG_CONFIG_PATH variable path
pkg-config opencv4 --libsdisplay the link flags associated with a given package
pkg-config opencv4 --cflagsprints the compile flags and the associated pre-processor required to compile a package plus the flags for its dependencies
pkg-config opencv4 --modversioncheck the version of a library
pkg-config mylib --cflags-only-Ionly yields the include paths in Cflags

pkg-config in g++​

g++ myprogram.cpp `pkg-config --libs opencv` -o myprogram

pkg-config file in CMake​

From stackoverflow:

You may also export a pkg-config file. This file allows a third-party application to easily import your library:

  • with Makefile, see pkg-config
  • with Autotools, see PKG_CHECK_MODULES
  • with cmake, see pkg_check_modules

Example: OpenCV: /usr/local/lib/pkgconfig/opencv.pc (see stackoverflow)

cmake config file​

doc

Example: OpenCV

  • defines variables such as OpenCV_LIBS and OpenCV_INCLUDE_DIRS
  • see stackoverflow
  • makes OpenCV available for external projects by using:
#    find_package(OpenCV REQUIRED)
#    include_directories(${OpenCV_INCLUDE_DIRS}) # Not needed for CMake >= 2.8.11
#    target_link_libraries(MY_TARGET_NAME ${OpenCV_LIBS})

include_directories vs. target_include_directories​

stackoverflow

E.g. if you have a CMakeLists.txt that builds two ROS nodes, then it will have two targets. If both targets use the include directories use include_directories. If only one of them uses the include directories use target_include_directories.

set_target_properties​

Specifies which public header files should be installed when running sudo make install. The install location is specified by

install(TARGETS galaxis_lib
    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
    PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

Do not forget the quotation signs around "${HEADERS}"! Otherwise only the first header file in the ${HEADERS} list will be installed.

set_target_properties(galaxis_lib PROPERTIES
    PUBLIC_HEADER "${HEADERS}")

find_library​

From stackoverflow:

This is a good idea, even if you know the path to your library. CMake will error out if the library vanished or got a new name. This helps to spot error early and to make it clear to the user (may yourself) what causes a problem.

To find a library foo and store the path in FOO_LIB use

find_library(FOO_LIB foo)

also see stackoverflow

Troubleshooting​

/tmp/ccsulwjG.o: In function cv::String::~String()':
  tmp.cpp:(.text._ZN2cv6StringD2Ev[_ZN2cv6StringD5Ev]+0x14): undefined reference tocv::String::deallocate()' /tmp/ccsulwjG.o: In function cv::String::operator=(cv::String const&)':
  tmp.cpp:(.text._ZN2cv6StringaSERKS0_[_ZN2cv6StringaSERKS0_]+0x28): undefined reference tocv::String::deallocate()' collect2: error: ld returned 1 exit status
  • A linker error.
  • Set dependencies, like -lopencv_core -lopencv_highgui -lopencv_imgproc.

ament_cmake (ROS 2)

doc

  • ament_cmake is the build system for CMake based packages in ROS 2
  • It is a set of scripts enhancing CMake and adding convenience functionality for package authors.

A basic CMake outline can be produced using ros2 pkg create <package_name> on the command line. The basic build information is then gathered in two files: the package.xml and the CMakeLists.txt.

package.xml in ament_cmake​

The package.xml must contain all dependencies and a bit of metadata

  • to allow colcon to find the correct build order for your packages,
  • to install the required dependencies in CI as well as
  • provide the information for a release with bloom.
    • note: Bloom is a release automation tool.

CMakeLists.txt in ament_cmake​

The CMakeLists.txt contains the commands to build package executables and libraries.

Autotools

Wikipedia:

  • GNU Autotools (aka GNU Build System)
  • a suite of programming tools designed to assist in making source code packages portable to many Unix-like systems.
  • Autotools consists of the GNU utility programs
    • Autoconf,
    • Automake, and
    • Libtool.

Overview​

Wikipedia:

Autoconf

  • Autoconf generates a configure script based on the contents of a configure.ac file, which characterizes a particular body of source code.
  • The configure script, when run, scans the build environment and generates a subordinate config.status script
  • which, in turn, converts other input files and most commonly Makefile.in into output files (Makefile), which are appropriate for that build environment.
  • Finally, the make program uses Makefile to generate executable programs from source code.

Automake

  • Automake helps to create portable Makefiles, which are in turn processed with the make utility.
  • It takes its input as Makefile.am, and turns it into Makefile.in, which is used by the configure script to generate the file Makefile output.
  • It also performs automatic dependency tracking;
    • every time a source file is compiled, the list of dependencies (e.g., C header files) is recorded.
    • Later, any time make is run and a dependency appears to have changed, the dependent files will be rebuilt.

Libtool

  • Libtool helps manage the creation of static and dynamic libraries on various Unix-like operating systems.
  • Libtool accomplishes this by abstracting the library-creation process, hiding differences between various systems (e.g. Linux systems vs. Solaris).

Usage​

Before running ../configure run

autoreconf -i

Bear

Bear is a tool that generates a compilation database (compile_commands.json) for clang tooling.

sudo apt install bear
bear make   # generates the compile_commands.json file

Other Tools to Generate a Compilation Database​

Bear is just one tool that can generate a compilation database. How to generate a compilation database in CMake, Clang, Ninja.

clang

  • a C++ compiler

clangd​

clangd is a language server.

  • clangd is based on the Clang C++ compiler

.clangd file​

  • a project configuration file
  • yaml format

Use cases:

  • e.g. to suppress the warning In included file: main file cannot be included recursively when building a preamble add
Diagnostics:
  Suppress: 'pp_including_mainfile_in_preamble'

to .clangd

Troubleshooting

General​

  • was not declared, is not a member of ... and other header related errors
    • Check if ROS2 packages that have the same name as packages in your ROS2 workspace have been installed via apt install (e.g. ros-eloquent-some-package). The headers of apt packages may be included instead of the headers of ROS2 packages in your ROS2 workspace, so that there will be some include conflicts.