Improving Makefile and general c++ project structure

Question

I am a single developer on a numerics project. As it is growing, I would like to improve

• the Makefile
• the general organisation of the project
• structuring of the files
• workflow

As it stands now I have a few classes (where a few depend on each other) and a main.cc. In total maybe 20 files. From that the executable is build via make and ./projectName creates data files, which are later processed with gnuplot script files. This whole process can also be compressed in a shell script run.sh.

EDIT: I am working under Linux using gnu make and g++ compiler.

What bothers me is:

1. As I add new classes make takes more and more time.
2. Everything is build from scratch each time make runs, even if only one class file is changed.
3. Related to that I have seen others using a build/ directory where object files files are stored. I Havent used .obj files so far. Does that solve the problem of (2.) and if yes how do I create them using make?
4. A debug and release build would be nice too, but i have seen questions on that on StackOverflow, so i'll get to that.

What I have is the following:

Dir Tree

ProjectName
|+data/     <-- data files generated by program
|+scripts/  <-- scripts that handle data processing
|~includes/
| |-headerClass1.h
| |-headerClass2.h
| |-...
| |-main.h
| |-parameters.h
|~src/
| |-main.cc
| |-class1.cc
| |-class2.cc
| |...
|-Makefile
|-tags
|-projectname <--executable

Makefile

workdir:=$(shell pwd)

SRCS =  $(workdir)/src/main.cc   \
          $(workdir)/src/class1.cc \
          $(workdir)/src/class2.cc \
                ...

INCLUDE = -I$(workdir)/include/\
                    -I$(workdir)/src/

CTALL = $(workdir)/include/*\
                $(workdir)/src/*

CTAGS = ctags

DEL = /bin/rm -f


RESULT = projectname
CC = g++
LIBS= -lm -lboost_timer -lboost_filesystem -lboost_system
OPT = -O3 -Wall  -ggdb -D__STDC_FORMAT_MACROS -std=c++0x -fipa-matrix-reorg

all: clean $(RESULT) ctags

$(RESULT): $(SRCS)
    $(CC) $(LIBS) $(OPT) $(INCLUDE) $(SRCS) -o $@

# Rules for generating the tags.
# #-------------------------------------
ctags: $(CTALL)
        $(CTAGS) $(CTALL)

clean:
    @echo "cleaning ..."
    $(DEL) $(RESULT) 

General remarks on project organization, improvement of the Makefile, etc. would by highly appreciated.

EDIT3:

Ok my makefile grew a bit and it is running now, working with release and debug mode and it is creating the corresponding directories obj_debug or obj_release. It also rebuilds only the .o files needed and creates the dependency files on the fly while compiling the source code and puts them into the obj_$(BUILD) dir.

Here the revised Makefile in all its glory:

# default value
BUILD ?=*

# directories# {{{
SRCS_DIR:=src
INCLUDE_DIR:=include
OBJS_DIR:=obj_$(BUILD)
# }}}

# source files# {{{
SRCS_FILES +=  main.cc
SRCS_FILES +=  domain.cc
SRCS_FILES +=  bulk.cc
SRCS_FILES +=  wellelectron.cc
SRCS_FILES +=  bbscatt.cc
SRCS_FILES +=  bwscatt.cc
SRCS   = $(SRCS_FILES:%.cc=$(SRCS_DIR)/%.cc)

# include files
INCLUDE += -I$(INCLUDE_DIR)/
#}}}

# Shell commands# {{{
#SHELL  = /bin/bash
CP    := /bin/cp
RM    := /bin/rm -f
MKDIR := /bin/mkdir -p
ECHO  := @/bin/echo
# }}}

# object files
OBJS   = $(SRCS_FILES:%.cc=$(OBJS_DIR)/%.o)

# dependency files
DEPS   = $(OBJS:.o=.d)

# target
TARGET = numScattRates_$(BUILD)

# compiler
CXX = g++

# Libraries# {{{
# add these lines to .bash_rc
#export CPP_INCLUDE_PATH=$HOME/bin/boost_1_51_0/include
#export LD_LIBRARY_PATH=$HOME/bin/boost_1_51_0/lib:$LDLIBRARY_PATH

#
# for warnings comping out of boost libs: 
# Use -isystem instead of -I to add Boost headers to include path.
# This option means to treat headers found there as system headers,
# and suppress warnings originating there.

INCLUDE_BOOST += -isystem$(HOME)/bin/boost_1_51_0/include
LDLIBS_BOOST  += -L$(HOME)/bin/boost_1_51_0/lib

LDLIBS += -lm
LDLIBS += -lboost_filesystem
LDLIBS += -lboost_timer
LDLIBS += -lboost_system
# }}}

# compiler options# {{{
# show all warnings # {{{
CXXFLAGS_WARNING +=  -W
CXXFLAGS_WARNING +=  -Wall
CXXFLAGS_WARNING +=  -Wextra
CXXFLAGS_WARNING +=  -Wcast-align
CXXFLAGS_WARNING +=  -Wcast-qual
CXXFLAGS_WARNING +=  -Wconversion
CXXFLAGS_WARNING +=  -Wdisabled-optimization
CXXFLAGS_WARNING +=  -Wfloat-equal
CXXFLAGS_WARNING +=  -Wformat-nonliteral
CXXFLAGS_WARNING +=  -Wformat-security
CXXFLAGS_WARNING +=  -Wformat-y2k
CXXFLAGS_WARNING +=  -Winit-self
CXXFLAGS_WARNING +=  -Winline
CXXFLAGS_WARNING +=  -Winvalid-pch
CXXFLAGS_WARNING +=  -Wlogical-op
CXXFLAGS_WARNING +=  -Wunsafe-loop-optimizations
CXXFLAGS_WARNING +=  -Wmissing-declarations
CXXFLAGS_WARNING +=  -Wmissing-include-dirs
CXXFLAGS_WARNING +=  -Wmissing-noreturn
CXXFLAGS_WARNING +=  -Wnon-virtual-dtor
CXXFLAGS_WARNING +=  -Wpacked
CXXFLAGS_WARNING +=  -Wpadded
CXXFLAGS_WARNING +=  -Wparentheses
CXXFLAGS_WARNING +=  -pedantic-errors
CXXFLAGS_WARNING +=  -Wpointer-arith
CXXFLAGS_WARNING +=  -Wredundant-decls
CXXFLAGS_WARNING +=  -Wswitch-default
CXXFLAGS_WARNING +=  -Wswitch-enum
#CXXFLAGS_WARNING +=  -Wsystem-headers
CXXFLAGS_WARNING +=  -Wunreachable-code
CXXFLAGS_WARNING +=  -Wunsafe-loop-optimizations
CXXFLAGS_WARNING +=  -Wunused
CXXFLAGS_WARNING +=  -Wvariadic-macros
CXXFLAGS_WARNING +=  -Wvolatile-register-var
CXXFLAGS_WARNING +=  -Wwrite-strings

# do not use that option together with BOOST as this options makes objects with internal pointers a nightmare.
#OPT_WARN += -Weffc++

# treat warnings as errors
CXXFLAGS_ERROR += -Werror
# }}}

# use for debuggin# {{{
CXXFLAGS_debug +=  -g3
CXXFLAGS_debug +=  -O0
CXXFLAGS_debug +=  -ggdb3
CXXFLAGS_debug +=  -gdwarf-2
CXXFLAGS_debug +=  -feliminate-dwarf2-dups

#OPT_DBUG +=  -ffloat-store
#http://stackoverflow.com/questions/7517588/is-this-an-g-optimization-bug
# }}}

# use for optimization, NEVER together with debug mode# {{{
# http://gcc.gnu.org/onlinedocs/gcc-3.4.4/gcc/Optimize-Options.html

# Optimize for speed# {{{
OPT_SPEED +=  -O2 # no inlining
#OPT_OPT +=  -O3 # inlining of function
OPT_SPEED += -DNDEBUG
OPT_SPEED += -fsignaling-nans
OPT_SPEED += -finline-functions
OPT_SPEED += -fipa-matrix-reorg
OPT_SPEED += -msse2
OPT_SPEED += -mfpmath=sse
OPT_SPEED += -fopenmp
#looses precisicion on float operations. http://stackoverflow.com/questions/7420665/what-does-gccs-ffast-math-actually-do
OPT_SPEED += -ffast-math 
OPT_SPEED += -fassociative-math
# use vectorizer always together with fmath options from above.
OPT_SPEED += -ftree-vectorize 
OPT_SPEED += -ftree-vectorizer-verbose=2
#===============================================================================================================
# compiler that places a  random canary between any stack allocated character buffers and the return
# pointer [5]. It then validates that the canary has not been dirtied by an
# overflowed buffer before the function returns. ProPolice can also reorder local
# variables to protect local pointers from being overwritten in a buffer overflow.
#===============================================================================================================
#OPT_SPEED += -fstack-protector-all -Wstack-protector \
# }}}

# Optimize for size# {{{
OPT_OPTSIZE += -Os
#std::vector still has code that throws exceptions and may use rtti. To suppress that use:
OPT_OPTSIZE += -fno-rtti 
OPT_OPTSIZE += -fno-exceptions 
# see also http://stackoverflow.com/questions/1512972/what-is-the-optimization-level-g-you-use-while-compairing-two-different-al
OPT_OPTSIZE += -fomit-frame-pointer
OPT_OPTSIZE += -Os
# }}}

CXXFLAGS_release += $(OPT_SPEED)
#CXXFLAGS_release += $(OPT_SIZE)
#}}}

# use as default# {{{
CXXFLAGS_DEFAULT += -std=c++0x
CXXFLAGS_DEFAULT += -march=native # Optimize for this architecture. If you want the application to run quickly on any architecture (our condor cluster), don't specify that option.
CXXFLAGS_DEFAULT += -mtune=native
CXXFLAGS_DEFAULT += -fshort-enums # Allocate to an enum type only as many bytes as it needs for the declared range of possible values. Specifically, the enum type will be equivalent to the smallest integer type which has enough room. 
# }}}

# }}}

# compiler flags from options# {{{
CXXFLAGS += $(INCLUDE)
CXXFLAGS += $(INCLUDE_BOOST)

CXXFLAGS += $(CXXFLAGS_DEFAULT)
CXXFLAGS += $(CXXFLAGS_ERROR)
CXXFLAGS += $(CXXFLAGS_WARNING)
CXXFLAGS += $(CXXFLAGS_$(BUILD))

# flags to create dependency files
DEPFLAGS += -MMD
DEPFLAGS += -MF
# }}}

# These are only Makefile targets and do not refer to files
# with the same name
.PHONY : clean veryclean all debug release target ctags

###########################################
# RULES
###########################################
# {{{
all: debug

debug:
    $(MAKE) BUILD=debug target

release:
    $(MAKE) BUILD=release target

-include $(DEPS)

target: $(TARGET)

$(TARGET): $(OBJS)
    $(ECHO) Linking $^ ...
    $(CXX)  $(LDFLAGS) -o $@  $^ $(LDLIBS_BOOST) $(LDLIBS)

# compile and create dependency files
$(OBJS_DIR)/%.o: $(SRCS_DIR)/%.cc
    $(MKDIR) $(OBJS_DIR)
    $(ECHO) Compiling $< and create dependency for $(patsubst %.o,%.d,$@)...
    $(CXX) $(CXXFLAGS) -o $@ -c $< $(DEPFLAGS)  $(patsubst %.o,%.d,$@)

# Rules for generating the tags.
ctags: $(CTALL)
    $(ECHO) Creating ctags...
    $(CTAGS) $(CTALL)

clean:
    $(ECHO) "Cleaning ..."
    $(RM) $(TARGET)

veryclean:
    $(ECHO) "Cleaning all ..."
    $(RM) $(TARGET) $(OBJS) $(DEPS)

show:
    $(ECHO) target.......$(TARGET)
    $(ECHO) source files.$(SRCS)
    $(ECHO) source dir...$(SRCS_DIR)
    $(ECHO) obj files....$(OBJS)
    $(ECHO) obj dir......$(OBJS_DIR)
    $(ECHO) dep files....$(DEPS)
# }}}

Answer 1

Common Conventions

1. The C++ compiler is named CXX
2. What you call RESULT is usually named TARGET
3. In addition to SRC you probably need OBJ
4. Commands are usually replaced by uppercase version varaiable names that name themselves.
   • ie. RM not DEL

Building issues

It rebuilds everything every time for two reasons.

The all: rule performs a clean: before it starts. So obviously it has to rebuild everything:

all: clean $(RESULT) ctags
   # ^^^^^^ Get rid of this

RESULT is dependent on SRC and rebuilds the executable from scratch each time. What you need to do is depend on OBJ and then you only need to link the objects together to build the executable. Each object will have its own dependency and only be re-built if the source file changes.

Note: Any common commands should follow the conventions set by the implicit rules:

The implicit rule for linking is:

$(CC) $(LDFLAGS) n.o $(LOADLIBES) $(LDLIBS)

You have a slightly more complex case but I would thus make my link command:

$(RESULT): $(OBJ)
    $(CXX)  $(LDFLAGS) -o $@  $(OBJ) $(LOADLIBES) $(LDLIBS) 

In your case you have a set of extra libs I would then add them as follows

LDLIBS  += -lm -lboost_timer -lboost_filesystem -lboost_system
  #    ^^^^  Notice the +=
  #          This makes it easy to add future enhancements to the makefile

The default rule for building *.o files from C++ files is:

$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c

Your header files inclusion is part of the pre-processor stage so should be added to the CPPFLAGS macro.

CPPFLAGS += $(INCLUDE)

Now your object files will build as you expect.

But normally you don't want to build your object files into the same directory as your source files. This is because you can have different build versions. eg you can have a debug build and a release build. You should never mix and match object from different builds. In fact no compiler manufacturer makes any guarantees about object files built with different compiler flags. So any differences in compiler flags can potentially make the object files incompatible. As a result build your object files into different sub directories based on the type of build.

Personally I build release into a directory called release and debug into a directory called debug. I know very boring. I also suffix any targets with their type.

Say I am building the lib plop.

libplop.<version>.so        // release version of plop
libplopD.<version>.so       // debug version of plop
libplopS.<version>.so       // Single threaded version (only build this if there is a specific difference for a single threaded version of the library that can be exploited (rare))
libplopSD.<version>.so      // Single threaded debug version

This way I can install both normal and debug version of a library and explicitly link other code against them.

Anyway back to your make file. To achieve this I normally do

OBJ     = $(patsubst %.cc,$(BUILD)/%.o, $(SRC))
BUILD  ?= .
TARGET  = projectname$(BUILD_EXT)
   # BUIlD_EXT is the suffix I place on executable to tell if it is debug/relese and version information
   # BUILD defines the build type. If this is not specified an unadulterated
   # build is done into the current directory (rather than debug/release)
   # You can do this by directory building target `make target`

## You can add build specific flags like this
## Set BUILD_EXT using the same technique.
CXX_BUILD_FLAGS_debug   = -g
CXX_BUILD_FLAGS_release = -O3
CXXFLAGS       += $(CXX_BUILD_FLAGS_$(BUILD))

all:  debug ctags
  #  by default build debug version
  #  if you want release you must explicitly ask for it

debug:
    $(MAKE) BUILD=debug target
release:
    $(MAKE) BUILD=release target

target: $(TARGET)

$(TARGET): $(OBJ)
    $(CXX)  $(LDFLAGS) -o $@  $(OBJ) $(LOADLIBES) $(LDLIBS) 

$(BUILD)/%.o: %.cc
    $(CXX) $^ -c -o $@ $(CPPFLAGS) $(CXXFLAGS)

Organization of files.

I normally put header and source files into the same directory. And I normally break these into directories based on libraries or executable.

But I add an install target into to my makefile that uploads the binary and required header files to a build directory for other projects so they can use it.

# Note this is not real makefile (pseudo make)
# It is designed to show intention not real make instructions.
install:
   $(generate_source_control_tag)
   $(CP) $(TARGET)  $(BUILD)/bin    # used for executables
   $(CP) $(TARGET)  $(BUILD)/lib    # used for libs
   $(CP) $(TARGET_INCLUDES) $(BUILD)/include/$(TARGET_BASE)/

This way if I am working on a project I can leave it (potentially in mid modification) and work on another library without the two being affected. The two will only ever interact after they have been installed into the build directory.

This means I also add the following too my makefile

 CPPFLAGS += -I$(BUILD)/include
 LDFLAGS  += -L$(BUILD)/lib

Now if project B wants to use the library from project A. Then the convention becomes.

 #include "A/A_HeaderFile_1.h"

 LDLIBS += -lA

Answer 2

Note that some of the commands below may be specific to GNU Make - you haven't said what tools you're using (although I guess you're on Windows since you mention .obj files).

Rules and object files

Currently, you have $(RESULT): $(SRCS), so it has to re-run your whole (single) compile line every time anything in $(SRCS) changes. Cacheing the compiler output for each individual file can indeed save time here:

$(OBJS) = $(patsubst %.cc, %.obj, $(SRCS))

$(RESULT): $(OBJS)
    $(CC) $(LIBS) $(OPT) $(INCLUDE) -o $@ $^

would do this (it just links all the .obj files to form your executable).

Now you need to tell make how to build those .obj files, which you can either leave up to the implicit rule (setting CXXFLAGS appropriately) or explicitly with:

%.obj : %.cc
    $(CC) $(OPT) $(INCLUDE) -c -o $@ $^

---

Directory structure

Note that leaving your build objects and output data inside your project tree isn't ideal - specifically it can add noise to whatever SCM tool you use, risks adding .obj files to source code control, which you almost certainly don't want, and makes it generally harder than necessary to clean your source tree.

You can use the patsubst command to change the path as well as the suffix of your object files, eg.

$(OBJS) = $(patsubst src/%.cc, build/%.obj, $(SRCS))

---

Dependencies

One major thing you're missing is header file dependencies: if you change a .h file, your makefile should be able to figure out which .o files need re-building. I don't know your toolchain, but for example gcc can automate this.