Book Image

Professional Plone 4 Development

Book Image

Professional Plone 4 Development

Overview of this book

A years of experience in Plone development and project management are combined with an approachable writing style to create an engaging and highly-informative guide to working with Plone. Professional Plone 4 Development stands as an excellent resource for developers of all levels. - Eric Steele, Plone Release Manager Plone is a web content management system that features among the top 2% of open source projects and is used by more than 300 solution providers in 57 countries. Its powerful workflow system, outstanding security track record, friendly user interface, elegant development model and vibrant community makes Plone a popular choice for building content-centric applications. By customising and extending the base platform, integrators can build unique solutions tailored to specific projects quickly and easily.If you want to create your own web applications and advanced websites using Plone 4, Professional Plone 4 Development is the book you need.The https://www.packtpub.com/Professional-Plone-web-applications-CMS/book first edition of this book remains one of the most widely read and recommended Plone books. This second edition is completely revised and up-to-date for Plone 4.1, covering new topics such as Dexterity, Diazo, jQuery and z3c.form, as well as improved ways of working with existing technologies such as Buildout, SQLAlchemy and the Pluggable Authentication Service. It retains the writing style and comprehensive approach that made the first edition so popular. Built around a realistic case study, Professional Plone 4 Development will take you from an understanding of Plone’s central concepts, through basic customization, theming, and custom development, to deployment and optimization. The book is divided into four sections: First, you will be introduced to Plone and the case study, and learn how to set up a development environment. The second section covers basic customization, including theming a Plone site using Diazo. The third section focuses on custom development – building new content types and user interfaces, customizing security and integrating with external databases. The final chapters cover deployment and performance optimization.
Table of Contents (6 chapters)

Chapter 3. The Development Environment

Before we can start building our application, we must set up a suitable development environment. This should mirror the final 'live' server configuration as closely as possible, so that we can test our software locally before deploying it. The environment should also provide appropriate tools to support us during development.

In this chapter we will learn about:

  • The elements of Zope's software stack
  • How these can be configured for development on various platforms
  • Supporting technologies, such as distributions, Setup tools/Distribute, and Buildout
  • Some useful development tools

Installers

Understanding your development environment is an important prerequisite to becoming a productive developer. If you need to get up and running quickly, however, you can start with one of the prepackaged Plone installers.

Note

The Plone community generally discourages the use of third-party operating system packages such as RPMs or DEBs for installing Zope and Plone. These are often out of date, and sometimes make non-standard assumptions that make them difficult to support. Furthermore, as a developer you will need write access to the build configuration and source code of your own packages. As such, you should run Zope from within your home directory. We will cover production server deployment in the final part of this book.

Since Plone 3.2, the various platform-specific installers have been based on Buildout, Plone's preferred build tool. You can download the latest Plone 4 installer for your platform from http://plone.org/downloads.

The rest of this chapter will describe the building blocks of a development environment and illustrate how to set up such an environment from scratch. If you use an installer, the result may be slightly different, but once you understand how Buildout works, you will be able to customize the installer-configured environment as necessary.

Note

Note that on Mac OS X, the Unified Installer is the best choice for developers, as it is more Buildout-friendly.

Glossary of terms

The table below summarizes the various terms and technologies which we will encounter in this chapter. It pays to be familiar with these, as you will find them again not only throughout this book, but also in other Plone documentation.

Term

Definition

Module

A single .py file of Python code.

Package

A bundle of Python modules and supporting files. At the most basic level, a package is a directory with an __init__.py file and zero or more other Python modules. Packages may be nested. See the Python documentation for details about how to create and import packages.

Setuptools / Distribute

Setuptools is a Python library that extends Python's built-in distutils library to support extended distribution metadata, including dependency management (see below).

Distribute is a fork of Setuptools that provides some additional features and bug fixes. At the time of writing, it is the more actively maintained of the two, and the default choice for Plone 4.

Namespace package

Nested packages can be used to build namespaces. For example, most Plone software lives in a top-level package called plone, for example plone.browserlayer and plone.memoize. This is useful to avoid clashes with other projects and to signal ownership over a particular package.

Namespace packages, a feature of Setuptools/Distribute, allow such namespaces to be declared explicitly. This in turn makes it possible to release packages independently: plone.browserlayer and plone.memoize are independent distributions on PyPI (see below), which can be installed independently and/or concurrently.

Distribution

A means of packaging and distributing Python code. A distribution consists of a directory with a setup.py file, Python code, and potentially other resources. The metadata in setup.py may include a version, installation dependencies and license information.

Tools based on Setuptools/Distribute, including Buildout, can use the metadata to manage concurrent installations of different versions of a given distribution, automatically fetch dependencies, and more.

It is common to create a distribution containing, and named after, a single package. For example, the distribution plone.memoize contains the package plone.memoize (which may of course contain sub-packages). However, distribution names do not have to follow this pattern.

Egg

A binary distribution format used by Setuptools/Distribute. Each distribution archive is specific to a particular platform and Python version. Therefore, it is normally safer to use a source format when distributing packages, which is effectively just a compressed archive of the code and metadata. (Setuptools/Distribute will turn a source distribution into an egg specific to the local development environment upon installation.)

The downside of using source distributions is that if a package has binary dependencies (for example Python extensions written in C), the necessary compiler and libraries need to be available wherever the distribution is installed. It is common to assume Linux and Mac OS X users have these tools available, but make binary eggs available for Windows.

Develop egg

Normally, we let Setuptools/Distribute (via Buildout) manage the downloading and installation of distributions. For packages we are developing, however, we normally want to control the source code ourselves, and have the abilityto have changes reflected immediately without having to rebuild and reinstall a distribution. This can be achieved with develop eggs, which use a placeholder file to inform Setuptools/Distribute about the location of the in-development source code.

Product

Filesystem-based Python extensions for Zope 2 were originally called Products. Placed in a special directory (Products/), they would automatically be discovered by Zope at startup, and implicitly appear under the Products.* namespace. Although this method of deployment is still supported for legacy software, virtually all products have now been converted to regular packages.

Products may contain an initialize() function, called when Zope starts up. For legacy reasons, this function is called automatically for packages in the Products.* namespace. Packages in other namespaces can opt into this behavior using the <five:registerPackage /> directive in their configure.zcml file. This is mainly useful for registering new content types. We will see an example of this in Chapter 10, Custom Content Types.

The term "product" is also sometimes used more generally to refer to add-ons that extend Plone, even if they are not actually packaged as old-style Zope 2 Products.

Python path

The Python path lists directories and distributions containing Python packages that are available for import by a running Python program. The current path is stored in the variable sys.path. We will use Buildout to ensure that this path contains the packages we need to run Plone and our custom software, and – ideally – no more.

The Python Package Index (PyPI)

Previously known as the "Cheese Shop", this is an online repository of Python distributions. Anyone can upload a distribution here. Setuptools/Distribute-based tools such as Buildout and easy_install can use PyPI to locate and install distributions and their dependencies.

easy_install

A command-line tool installed with Setuptools/Distribute, which searches PyPI for a given distribution and downloads and installs it.

By default, easy_install puts packages in the global site-packages folder for the Python interpreter which was used to install easy_install itself. This, coupled with limited dependency version management features, makes easy_install unsuitable for installing Plone and Zope.

Pip

A more advanced alternative to easy_install, which can better manage dependency versions and supports distribution uninstallation. This is rarely used in Plone development (and unsuitable for installing Plone itself), but useful for installing tools written in Python.

Buildout

A tool for managing a self-contained environment through a single configuration file (buildout.cfg). Buildout itself is installed by the zc.buildout distribution. We also use the noun 'buildout' (lowercase 'b') to describe a particular environment managed by Buildout (capital 'B').

Most Plone projects use Buildout to download, install, and configure Zope, Plone, and other dependencies. Buildouts are 'repeatable', meaning that they can be used to replicate a particular setup across multiple servers or developers' machines.

Known Good Set (KGS)

Zope and Plone consist of hundreds of distributions, all available on PyPI. These evolve semi-independently—new versions may be released at any time.

By default, Buildout, easy_install, and Pip will download the most recent distributions of a package and its dependencies. This may be fine for simple projects, but for complex systems such as Zope or Plone, there is no guarantee that the latest versions of all dependencies will work together. Furthermore, a new release of a dependency could make a 'stable' build difficult to replicate in the future.

To avoid such problems, the Zope and Plone developers publish 'known good sets' that 'pin' every distribution to a tested version. Custom projects usually start from a standard KGS, and then override or amend the version pins as necessary to accommodate additional software.

Virtualenv

Just as stray versions can destabilize an installation, so can conflicting packages installed globally into the Python interpreter's site-packages directory, e.g. using a tool like easy_install or Pip. Many operating systems also ship with Python installations that contain operating-system specific packages, some of which have been known to conflict with Zope and Plone.

Virtualenv is a script that can be used to create a clean, isolated Python installation. It also allows distributions to be installed in a way that is local to this environment, so that they do not affect other installations.

ZopeSkel

A collection of Paste Script templates for Zope and Plone development, accessible through the zopeskel command. We will use this to create new package distributions.

Prerequisites

Before we can get into the business of setting up a Plone development environment, we must install a few prerequisites:

  • Python interpreter: This is suitable for running Plone. As of Plone 4.0, that means Python 2.6. If your operating system does not have Python 2.6 installed, you can get the latest version in the 2.6 series from http://python.org/. On Mac OS X, you can also use MacPorts (http://www.macports.org/) to install it. Some members of the Plone community have also created a cross-platform Buildout that can be used to build Python itself. This is particularly popular for installing Python 2.4 (used in Plone 3) on Mac OS X Snow Leopard. See http://svn.plone.org/svn/collective/buildout/python.
  • Python Windows Extensions (On Windows only): It is an installer suitable for your Python version and architecture can be downloaded from http://sourceforge.net/projects/pywin32/files/. Presuming you installed Python to C:\Python26, you should add C:\Python2.6;C:\Python2.6\Scripts to your system PATH.
  • Python Imaging Library (PIL): This should be installed for this Python interpreter. Download it from http://www.pythonware.com/products/pil. This may have other dependencies, such as the JPEG, PNG, and GIF libraries. On Mac OS X, you can use MacPorts to install PIL, if you also used MacPorts to install Python 2.6. On Windows, there is a PIL binary installer. On other operating systems, you may want to install the development versions of libjpeg, libpng, and libgif using operating system packages, and then install PIL manually from source. Simply unpack it and run python setup.py install from inside the newly unpacked directory, where python is the binary you intend to use to run Plone.
  • C compiler (On Unix-like operating systems): You will need this to compile Zope and its dependencies. The default choice is GCC. On Mac OS X, this means installing XCode with command line tools from the Mac OS X DVD. On Linux, use your operating system's package manager to install the base GCC package. On Windows, there should be binary distributions available for all dependencies, making a compiler unnecessary.
  • An internet connection: Buildout needs to download and install Plone. If your connection requires a proxy server, Python should respect the operating system proxy settings. If it does not, you may need to set the HTTP_PROXY environment variable to the proxy server's URL.
  • A programmer's text editor: The editor should be preferably one with Python, XML, HTML, and CSS syntax highlighting. You should set up your editor, so that a tab or indent is output as four (4) spaces. This makes Python development a lot more predictable. Popular choices include TextMate on Mac OS X, vi or Emacs on Linux, and SciTE on Windows.
  • A shell: Most examples in this book will show a Bash interpreter shell, though we will cover Windows syntax when it differs significantly. Remember, that path separators on Windows are backslashes (\), while other environments use forward slashes (/). Also, environment variables on Windows are referred to as %NAME%, whilst in most Unix shells, including Bash, variables are dereferenced with $NAME.

Note

This book assumes that you are comfortable using the shell to move around the filesystem, run commands, and manipulate files; and that you know how the PATH environment variable works and how to change it.

Creating an isolated Python environment

If you installed or compiled a version of Python specifically for the purposes of Plone development, and you did not install anything else globally, you can skip this step.

Note

This is likely to be the case if you are on Windows (see the next section), which is just as well, as it is tricky to install PIL into a Virtualenv on Windows.

If, however, you want to use an operating system-provided Python installation, or you intend to install anything into the global Python environment, you should create an isolated Python environment for Plone development using Virtualenv by executing the following steps:

  1. First, make sure easy_install is available. You can try this on a command line with:
    $ easy_install -h
    
  2. If this prints a help message, all is good. Otherwise, download distribute_setup.py from http://python-distribute.org/ and run it with:
    $ python distribute_setup.py
    

    Note

    If you get a permission denied error, you may need to run this command as root or under sudo.

  3. You should then be able to install that latest version of Virtualenv using easy_install:
    $ easy_install -U virtualenv
    

    Note

    Again, you may need to do this as root or under sudo. If the command is not found, look at the output of the distribute_setup.py command to find out where easy_install was configured, and add this to your PATH environment variable.

  4. At this point, you should have a virtualenv command available. Use this to create a new environment, such as in your home directory. For example:
    $ virtualenv --no-site-packages plone-python
    

    Note

    This will use the same Python interpreter as the one used by easy_install. If you want to use a different Python binary (for example: if easy_install does not run under Python 2.6), you can specify the full path to the python binary with the -p command line option.

This will install a 'clean' Python binary in plone-python/bin/python. The --no-site-packages ensures no global site packages 'bleed' into the environment.

  1. This unfortunately also hides PIL, so you will need to install the Python components of this again. Download the latest version from http://pythonware.com/products/pil/. Unpack it to reveal the directory Imaging-1.1.7 (assuming 1.1.7 is the version you downloaded). Then do:
    $ cd /path/to/Imaging-1.1.7
    $ /path/to/plone-python/bin/python setup.py install
    

    Note

    Please note that at the time of writing this book, PIL is not compatible with setuptools and thus cannot be installed with easy_install or Buildout.

Creating an isolated Python environment

If you installed or compiled a version of Python specifically for the purposes of Plone development, and you did not install anything else globally, you can skip this step.

Note

This is likely to be the case if you are on Windows (see the next section), which is just as well, as it is tricky to install PIL into a Virtualenv on Windows.

If, however, you want to use an operating system-provided Python installation, or you intend to install anything into the global Python environment, you should create an isolated Python environment for Plone development using Virtualenv by executing the following steps:

  1. First, make sure easy_install is available. You can try this on a command line with:
    $ easy_install -h
    
  2. If this prints a help message, all is good. Otherwise, download distribute_setup.py from http://python-distribute.org/ and run it with:
    $ python distribute_setup.py
    

    Note

    If you get a permission denied error, you may need to run this command as root or under sudo.

  3. You should then be able to install that latest version of Virtualenv using easy_install:
    $ easy_install -U virtualenv
    

    Note

    Again, you may need to do this as root or under sudo. If the command is not found, look at the output of the distribute_setup.py command to find out where easy_install was configured, and add this to your PATH environment variable.

  4. At this point, you should have a virtualenv command available. Use this to create a new environment, such as in your home directory. For example:
    $ virtualenv --no-site-packages plone-python
    

    Note

    This will use the same Python interpreter as the one used by easy_install. If you want to use a different Python binary (for example: if easy_install does not run under Python 2.6), you can specify the full path to the python binary with the -p command line option.

This will install a 'clean' Python binary in plone-python/bin/python. The --no-site-packages ensures no global site packages 'bleed' into the environment.

  1. This unfortunately also hides PIL, so you will need to install the Python components of this again. Download the latest version from http://pythonware.com/products/pil/. Unpack it to reveal the directory Imaging-1.1.7 (assuming 1.1.7 is the version you downloaded). Then do:
    $ cd /path/to/Imaging-1.1.7
    $ /path/to/plone-python/bin/python setup.py install
    

    Note

    Please note that at the time of writing this book, PIL is not compatible with setuptools and thus cannot be installed with easy_install or Buildout.

A minimal buildout

With the prerequisites out of the way, we can now install Zope and Plone using Buildout. Later in this chapter, we will create a flexible development buildout, which we will use and extend throughout the book. First, however, we will illustrate a minimal Plone buildout and show how to stop and start Zope in debug mode.

  1. First, we create a new directory to contain the buildout somewhere we have write access as a non-administrative (root) user, such as our home directory. It is preferable to use a short directory path, without spaces, as some older tools do not deal well with spaces in file paths.
  2. Next, we download bootstrap.py, which is used to install Buildout itself. It is best to download the version used by the Plone developers: http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.1/bootstrap.py (adjust the version as necessary for Plone 4.2 and onward).
  3. We place this file in our newly created directory. In the same directory, we use a text editor to create a buildout.cfg file with the following contents:
    [buildout]
    extends =
        http://dist.plone.org/release/4.1/versions.cfg
    parts = instance
    
    [instance]
    recipe = plone.recipe.zope2instance
    user = admin:admin
    eggs = Plone

    Note

    The extends line is used to include a 'known good set' of distributions corresponding to a particular Plone release. You should adjust the version number (4.1 in this case, which is the latest release at the time of writing) as necessary.

  4. To initialize the buildout, open a shell and move to the directory containing the two files. Then we run the following command line which will download Buildout and create a few directories and files.
    $ python bootstrap.py --distribute
    

    Note

    If you want to use a different Python binary to run Plone, you can invoke one by absolute path. For example, using the Virtualenv example (shown earlier), you could run /path/to/plone-python/bin/python bootstrap.py --distribute.

  5. We can then run:
    $ bin/buildout

    This will download, compile, and install Zope and Plone. The first time we do this, it can take a long time.

    Note

    You may also see some warnings about 'syntax errors' fly past when Setuptools/Distribute tries to pre-compile certain Python scripts. You can safely ignore these. They are issued because, these scripts are not normal Python modules, but rather scripts intended to be executed in Zope's untrusted scripting environment.

  6. Once the buildout has finished, we can start Zope in debug mode with:
    $ bin/instance fg

When we see the line Zope is ready to handle requests, we can open a web browser and go to http://localhost:8080. We should see a welcome page, inviting you to create a new Plone site. Using the administrative username and password (admin/admin) specified in the buildout.cfg file, we can do so. For now, we will leave all options at their default settings.

Presuming we kept Plone as the Plone site id, we should now be able to access our Plone site at http://localhost:8080/Plone.

When we are done, we can shut down Zope by pressing Ctrl+C in the terminal window.

Buildout basics

The simple buildout (mentioned in the previous section) illustrates most of Buildout's core features. Let us look at these in a bit more detail.

The bootstrap.py script installs zc.buildout itself, and gives us the bin/buildout command. This looks for build instructions in a file called buildout.cfg in the current directory. An alternative file may be specified with the -c command line option:

$ bin/buildout -c somefile.cfg

Note

The bin/buildout command must be re-run for any changes to the buildout configuration file to take effect.

To run a buildout in offline mode, we can use:

$ bin/buildout -o

This is useful if we are not connected to the internet, or simply to speed up buildout execution.

Note

An offline build will fail unless all required distributions and files are available on the local machine. This usually means that you need to run the buildout at least once already.

See the output of bin/buildout --help for other options.

The buildout configuration file consists of sections, in square brackets, containing options, given as name = value pairs. Options that accept multiple values use whitespace as a delimiter. Such values may also be given on multiple, indented lines:

[instance]
eggs =
    Plone
    Products.PloneFormGen

A value can be referenced from another value as ${section:option}. For example:

[hosts]
devserver = localhost

[ports]
devserver = 8080

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = Plone
http-address = ${hosts:devserver}:${ports:devserver}

This kind of variable substitution becomes important once we start to define sections that are shared by multiple buildouts.

It is also possible to add to or remove from a multi-line option using += and -=:

[instance]
eggs =
    Plone
    Products.PloneFormGen

# later, possibly in another file:

[instance]
eggs +=
    collective.googleanalytics

Lines beginning with a # are taken as comments and ignored.

In a buildout configuration file, the [buildout] section, which controls global options, usually comes first:

[buildout]
extends =
    http://dist.plone.org/release/4.1/versions.cfg
parts = instance

The extends line is used to include other configuration files, which can be specified by relative path or remote URL. All included files are merged into a single logical buildout. If, for a particular named section, an option in an extended file is also specified in the extending file, the latter's value is used.

The extends line references a known good versions set for Plone. If you open it in a web browser, you will see something like this (truncated for brevity):

[buildout]
extends =
        http://download.zope.org/zopetoolkit/index/1.0.3/zopeapp-versions.cfg
        http://download.zope.org/Zope2/index/2.13.8/versions.cfg

[versions]
Plone                                 = 4.1
Products.ATContentTypes   = 2.1.3

This file extends two other files, which in turn contain statements like (again truncated):

[buildout]
versions = versions

[versions]
Zope2 = 2.13.8
...

When these files are merged, Buildout will look for version pins in the [versions] section (as indicated by the ${buildout:versions} option). The Zope 2 has known good set pins numerous distributions in its [versions] block, to which the Plone known good set adds its own. We could also add or override some version pins using a [versions] section in our own top level buildout.cfg.

The parts option in the [buildout] section is used to list, in order, the steps that Buildout should follow when executing the build. Each part is defined by a section containing a recipe option.

A recipe is a named distribution which exposes the logic to perform a particular build task. It will be downloaded from PyPI as required. Most recipes accept and/or require various options, read from the relevant part section. It is possible to use a given recipe multiple times (in different parts) in a single buildout.

Note

Writing a new recipe is beyond the scope of this book, but it is not particularly difficult. See http://buildout.org for details. With numerous recipes available on PyPI, however, custom recipes are rarely required. You can look for buildout recipes on PyPI: http://pypi.python.org/pypi?:action=browse&c=512

In the example above, we specified a single part, instance, which was defined in the following section:

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = Plone

This particular recipe is used to configure a Zope 2 instance, into which we install the Plone distribution, and configure an initial user called admin, with the password admin. See http://pypi.python.org/pypi/plone.recipe.zope2instance for details about other options supported by this recipe.

Several recipes, including plone.recipe.zope2instance, use the eggs option to define a working set of eggs that should be provided for any console scripts generated. When Buildout generates a script, it will embed the working set in the sys.path variable. For example, bin/instance will look something like this (truncated for brevity):

#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = [
  '/path/to/buildout/eggs/Plone-4.1-py2.6.egg',
  '/path/to/buildout/eggs/bpython-0.9.7.1-py2.6.egg',
  '/path/to/buildout/eggs/plone.reload-1.5-py2.6.egg',
  ...
  ]

import plone.recipe.zope2instance.ctl

if __name__ == '__main__':
    plone.recipe.zope2instance.ctl.main(
        ["-C", '/path/to/buildout/parts/instance/etc/zope.conf']
        + sys.argv[1:])

This 'path mangling' ensures that the relevant packages, at the required versions, are available at runtime, and provides for isolation between scripts and buildouts.

The working set is calculated from all distributions listed in the eggs option, in addition to their dependencies. When the relevant part is executed, Buildout will download and install distributions from PyPI as necessary. Downloaded source distributions are kept in the dist directory in the downloads cache, if one is configured. Platform-specific egg installations are kept in the eggs/ directory, either locally in buildout root or in a shared eggs cache.

Alternatively, we can specify one or more develop eggs. If no version pin says otherwise, a develop egg normally takes precedence over downloaded distributions. During development, we will usually manage all our custom code as develop eggs in the src/ directory inside the buildout. Develop eggs must be explicitly listed in the develop option in the 'buildout' section. For example:

[buildout]
develop =
    src/my.package
    src/my.otherpackage

It is also possible to use a wildcard match:

[buildout]
develop =
    src/*

We will demonstrate a more comprehensive buildout. For more details about Buildout, see http://www.buildout.org/ and http://pypi.python.org/pypi/zc.buildout.

The buildout directory

Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.

We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.

Inside a Plone development buildout, we may find the following files and directories:

File or directory

Version control

Purpose

bootstrap.py

Yes

Installs zc.buildout in a freshly created buildout.

*.cfg

Yes

Buildout configuration files, including the default buildout.cfg.

.installed.cfg

No

Describes the currently installed buildout configuration. This allows buildout to run already-installed recipes in 'update', rather than 'install' mode.

.mr.developer.cfg

No

Describes currently installed develop eggs when using the mr.developer extension.

src/

Yes

The default location for custom distributions. You should version control the src/ directory itself, but if you use the mr.developer extension to manage your develop eggs, you should ignore src/* (all files inside the src/ directory) to avoid accidentally checking them into the build itself. See further for more details.

bin/

No

Contains installed scripts, such as bin/buildout and bin/instance.

eggs/

No

Contains all eggs ever installed from binary or source distributions. (The currently active set of eggs for a given runtime is embedded in the scripts in the bin/ directory.) It is possible to share an eggs directory among multiple buildouts.

develop-eggs/

No

Contains the placeholder files for currently active develop eggs.

parts/

No

Used by recipes to store internal files. Buildout is liable to delete or overwrite anything inside this directory at any time.

var/

No

Contains runtime configuration such as logs (var/logs/) and the Zope database files (var/filestorage/ and var/blobstorage/).

coverage/

No

Default output location for HTML test coverage reports created with z3c.coverage. We will describe this tool in more detail shortly.

*.pyc, *.pyo

No

These are byte-compiled Python files, which should never be under version control. They may appear in distributions inside the src/ directory, for example.

Tip

If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='[email protected]',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

The buildout directory

Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.

We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.

Inside a Plone development buildout, we may find the following files and directories:

File or directory

Version control

Purpose

bootstrap.py

Yes

Installs zc.buildout in a freshly created buildout.

*.cfg

Yes

Buildout configuration files, including the default buildout.cfg.

.installed.cfg

No

Describes the currently installed buildout configuration. This allows buildout to run already-installed recipes in 'update', rather than 'install' mode.

.mr.developer.cfg

No

Describes currently installed develop eggs when using the mr.developer extension.

src/

Yes

The default location for custom distributions. You should version control the src/ directory itself, but if you use the mr.developer extension to manage your develop eggs, you should ignore src/* (all files inside the src/ directory) to avoid accidentally checking them into the build itself. See further for more details.

bin/

No

Contains installed scripts, such as bin/buildout and bin/instance.

eggs/

No

Contains all eggs ever installed from binary or source distributions. (The currently active set of eggs for a given runtime is embedded in the scripts in the bin/ directory.) It is possible to share an eggs directory among multiple buildouts.

develop-eggs/

No

Contains the placeholder files for currently active develop eggs.

parts/

No

Used by recipes to store internal files. Buildout is liable to delete or overwrite anything inside this directory at any time.

var/

No

Contains runtime configuration such as logs (var/logs/) and the Zope database files (var/filestorage/ and var/blobstorage/).

coverage/

No

Default output location for HTML test coverage reports created with z3c.coverage. We will describe this tool in more detail shortly.

*.pyc, *.pyo

No

These are byte-compiled Python files, which should never be under version control. They may appear in distributions inside the src/ directory, for example.

Tip

If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='[email protected]',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='[email protected]',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='[email protected]',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

The development buildout

We will now create the development buildout that will be used throughout this book. You can find this in the book's accompanying source code.

We currently have three buildout configuration files , all found at the root of the buildout:

File

Purpose

versions.cfg

Contains our own known good set of distributions, allowing us to pin down custom dependencies and override any version pins from other known good sets we extend.

packages.cfg

Extends versions.cfg and other known good version sets, including the known good set for Plone, sets some global options, and defines various working sets. This file allows us to manage the distributions that make up our project in a way that can be reused by multiple top-level buildout configuration files.

buildout.cfg

This is the top-level buildout file for our development build, containing the parts and options we need to set up a development environment. Later in the book, we will add additional top-level buildout files for production deployment.

The first of these files, versions.cfg, is straightforward:

# Project-specific version pins
# =============================

[versions]
# Buildout
mr.developer = 1.17
collective.recipe.omelette = 0.10

# Development tools
bpython = 0.9.7.1
pygments = 1.4
Products.DocFinderTab = 1.0.4
Products.PDBDebugMode = 1.3.1
Products.PrintingMailHost = 0.7
z3c.coverage = 1.2.0
jarn.mkrelease = 3.0.9
setuptools-git = 0.4.2
setuptools-hg = 0.2

# ZopeSkel
ZopeSkel = 2.19
Cheetah = 2.2.1
Paste = 1.7.5.1
PasteScript = 1.7.3
PasteDeploy = 1.3.4

Note

The version pins shown here were the most appropriate versions at the time of writing. You may want to update some of these versions. See the description of z3c.checkversions mentioned in the following sections for details.

Next up, packages.cfg is a little more interesting:

# Information about packages and known good version sets
# ======================================================

[buildout]
extensions = mr.developer buildout.dumppickedversions
extends = 
# Known good sets of eggs we may be using
    http://dist.plone.org/release/4.1/versions.cfg
    versions.cfg

versions = versions
unzip = true

# Egg sets
[eggs]
main =
    Plone
test = 
devtools =
    bpython
    plone.reload
    Products.PDBDebugMode
    Products.PrintingMailHost
    Products.DocFinderTab

# Checkout locations
[sources]

This first installs two buildout extensions: mr.developer, which is used to manage develop eggs, and buildout.dumppickedversions, which will print the picked versions for any unpinned distributions at the end of the buildout run. We will describe these in more detail in the next section.

Next, we extend two known good sets—the one for our chosen Plone version, and our own local versions.cfg, which should come last so that it can override any versions from the external sets.

We also set the unzip option to true, which means that buildout will always unzip eggs in eggs/ directory. This makes debugging easier, avoids potential problems with Zope code that does not properly set the zip_safe flag to False, and is a prerequisite for using collective.recipe.omelette. Two recipe-less sections are then defined: [sources] is used by mr.developer, and will be described in a moment. [eggs] is used to define working sets that are referenced in other buildout sections:

Working set

Purpose

main

The minimal set of distributions required to run our application. Until we create some distributions of our own, we use the Plone egg here. In Chapter 5, Developing a Site Strategy we will replace this with our own 'policy package', which in turn will depend on all the software we need, including Plone.

test

This option lists all distributions for which we want to run automated tests. Usually, that means every custom distribution we create. If distributions have a [test] extra for test-only dependencies, this should be used here. Until we create our own distributions, this is empty.

devtools

Additional distributions that we want to install in the development environment, but not on a production server.

Finally, buildout.cfg contains the actual build instructions for our development environment:

# Development environment buildout
# ================================

[buildout]
parts =
    instance
    test
    coverage-report
    omelette
    zopepy
    zopeskel
    checkversions
    mkrelease

extends =
    packages.cfg

# Packages to check out/update when buildout is run
auto-checkout =
    
# Make sure buildout always attempts to update packages
always-checkout = force

# Development Zope instance. Installs the ``bin/instance`` script
[instance]
recipe = plone.recipe.zope2instance
http-address = 8080
user = admin:admin
verbose-security = on
eggs =
    ${eggs:main}
    ${eggs:devtools}
# Test runner. Run: ``bin/test`` to execute all tests
[test]
recipe = zc.recipe.testrunner
eggs = ${eggs:test}
defaults = ['--auto-color', '--auto-progress']

# Coverage report generator.
# Run: ``bin/test --coverage=coverage``
# and then: ``bin/coveragereport``
[coverage-report]
recipe = zc.recipe.egg
eggs = z3c.coverage
scripts = coveragereport
arguments = ('parts/test/coverage', 'coverage')

# Installs links to all installed packages to ``parts/omelette``.
# On Windows, you need to install junction.exe first
[omelette]
recipe = collective.recipe.omelette
eggs = 
    ${eggs:main}
    ${eggs:devtools}

# Installs the ``bin/zopepy`` interpreter.
[zopepy]
recipe = zc.recipe.egg
eggs = 
    ${eggs:main}
    ${eggs:devtools}
interpreter = zopepy

# Installs ZopeSkel, which can be used to create new packages
# Run: ``bin/zopeskel``
[zopeskel]
recipe = zc.recipe.egg
eggs = ZopeSkel

# Tool to help check for new versions.
# Run: ``bin/checkversions versions.cfg``
[checkversions]
recipe = zc.recipe.egg
eggs = z3c.checkversions [buildout]

# Tool to make releases
# Run: ``bin/mkrelease --help``
[mkrelease]
recipe = zc.recipe.egg
eggs = jarn.mkrelease

First, we list the parts to execute, and extend packages.cfg to gain access to our working- and known good version sets. The last two options in the [buildout] section, auto-checkout and always-checkout, are used by mr.developer and described below.

Next, we define our development Zope instance. This runs on port 8080, has an administrative Zope user called admin with password admin, provides additional debug information for security violations, and is installed with a working set combining ${eggs:main} and ${eggs:devtools}.

The subsequent parts install various development tools, which are described in more detail in the subsequent sections.

Note

You may have noticed that several parts use zc.recipe.egg. This recipe is used to install one or more distributions. If a distribution declares one or more console scripts (through an entry point), these are installed to the bin/ directory refer http://pypi.python.org/pypi/zc.recipe.egg for details.

Development tools

Let us now examine the various development tools that we will use throughout the book.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Development tools

Let us now examine the various development tools that we will use throughout the book.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Learning to help yourself

During development, there will probably be times when you are stumped. Plone is fairly well-documented, but the documentation is certainly not perfect. The mailing lists and chat room are great resources if you need help, but it is also very important to learn how to help yourself.

Find the documentation

Documentation for Plone code can usually be found in one of the following places:

  • High level and tutorial-style documentation is normally found at http://plone.org/documentation.
  • Many distributions include documentation in their PyPI listings. For example, plone.testing's documentation can be found at http://pypi.python.org/pypi/plone.testing.
  • Distributions often include documentation in their README file. This is often also the source of the documentation published to PyPI.
  • Internal documentation in the form of comments or tests is often very valuable. Most packages use Zope interfaces (more on those in Chapter 9, Nine Core Concepts of Zope Programming) to describe their key APIs. Look for a file called interfaces.py.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Find the documentation

Documentation for Plone code can usually be found in one of the following places:

  • High level and tutorial-style documentation is normally found at http://plone.org/documentation.
  • Many distributions include documentation in their PyPI listings. For example, plone.testing's documentation can be found at http://pypi.python.org/pypi/plone.testing.
  • Distributions often include documentation in their README file. This is often also the source of the documentation published to PyPI.
  • Internal documentation in the form of comments or tests is often very valuable. Most packages use Zope interfaces (more on those in Chapter 9, Nine Core Concepts of Zope Programming) to describe their key APIs. Look for a file called interfaces.py.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Summary

In this chapter, we have seen:

  • How to prepare a development environment for Zope and Plone development
  • How to set up a minimal Zope buildout and install a Plone site into it
  • How Buildout and Setuptools/Distribute work
  • How to set up a more sophisticated and flexible development buildout. This buildout will be used throughout the book
  • Some important development debugging tools
  • A few tips on how you can more effectively help yourself by learning to look at source code, using PDB effectively, prototyping things with zopepy, and looking at log files

This concludes Part 1 of the book. In Part 2, we will learn more about customizing Plone for our specific needs.