Look Ma! No Resource Registries

Over the past years I have spent a lot of time trying to make our add-ons compatible with Plone 4.3 and Plone 5. This has been difficult for many reasons including differences in the API of default content types frameworks and registration of static resources, just to mention some.

On the latest topic I've been very critic of the approach taken and even tried to propose a different one to avoid some well known problems on PLIP 1896. Unfortunately this was not understood at the time and I just gave up because we didn't had use cases to justify the migration to Plone 5.

This has started to change in latest months and we can't continue kicking the can down the road any longer: Python 2.7 will be EOL in 2020, Zope 4 has been released, and there has been a lot of work lately to make Plone compatible with Python 3.

IMO, Plone 5 resource registries are complex and must be avoided for many reasons as stated in the PLIP aforementioned:

• the revolutionary approach chosen (rewrite everything) has proven buggy and difficult to maintain, as shown by the many issues reported over the last years (the latest opened just 2 days ago)
• some of the JavaScript tools used at the time (Bower, Grunt, RequireJS…) are less attractive nowadays than more modern options like npm, Yarn and webpack
• resource bundling was a workaround for a limitation of the HTTP/1.1 protocol and not a CMS feature; HTTP/2 made this unnecessary and even undesirable
• LESS is now way less popular than SASS

I still think that every add-on author must be able to use the tools of their choice, and I still think Plone should not have to worry about resource bundling.

In early 2016 we selected webpack for our projects and in early 2017 we developed a Buildout recipe (called sc.recipe.staticresources) to make the inclusion of it into the Plone ecosystem easier. Over the past 2 years we have been developing, testing, and enhancing this approach in many add-ons and projects. But the resource registries were still there…

Not anymore.

I'm pleased to announce the release of collective.lazysizes 4.1.1.1, the first add-on that uses a new and very opinionated approach on how to handle static resources in Plone: we just deprecated resource registries in favor of a viewlet registered in plone.htmlhead. This simplifies maintenance among multiple Plone versions and avoids bundling of unrelated resources.

How does it work?

We use webpack to generate all static resources (in our case, a JavaScript file, a page template and an icon; check the webpack.config.js file). The page template is used for the viewlet and includes  the following code on it:

<script async="" src="https://www.example.com/++resource++collective.lazysizes/lazysizes-43c36fc.js"></script>

As you can see, the JavaScript id already includes a hash (lazysizes-43c36fc.js), so you don't have to worry about cooking resources, neither caching the wrong file. The hash will only change if the related code changes.

(To understand better the work done, check the following pull requests: Deprecate resource registries and Create JS resource with unique ID.)

What are the benefits of this for developers?

• you can work on your project using latest state-of-the-art JavaScript technologies like ES2015 and Vue.js
• you don't have to worry about resource registries anymore as this approach works out of the box in all Plone versions: no more duplicated registrations, cleaner and simpler ZCML and XML files
• you don't have to worry about upgrade steps anymore (at least the ones related with resource registry cooking/bundling)
• you end up with better static resources: all CSS and JS files minimized, all images optimized
• your code will be faster, as rendering a viewlet is less expensive than rendering a resource in the registry (at least, in Plone 4.3)

 What are the benefits of this for system administrators?

• less steps to upgrade your sites
• less unneeded cache invalidations
• faster sites as resources can be loaded in parallel

See how easy is to add a new feature to our resources: Add lazysizes print plugin; and how easy is to update the version of the JavaScript library we're using: Upgrade lazysizes to 4.1.1.

Now you can concentrate in the things that matter and stop worrying about the ugly parts; isn't that fun?

Share and enjoy!

Configuring better load balancing and health checks for Plone with HAProxy

In the previous post of this blog I enumerated some problems we face when using normal load balancer health checks on Plone instances and I described a possible solution using five.z2monitor. In this post I'm going to show you how to configure HAProxy and other components of the stack in order to get the most of it.

As mentioned previously, we can do load balancing using nginx, Varnish or HAProxy. Both nginx and Varnish provide more features on their commercial versions, but I'm going to focus only on the community versions. In my opinion, if you ever need more features in your load balancer you should try HAProxy before buying one of those subscriptions.

On nginx the load balancer is handled by the ngx_http_upstream_module, it's very easy to enable and configure and it has some nice features (like backup servers and the least connections method), but also many limitations (like basic health checks and no queue control). This works very well on small sites.

On Varnish the load balancer is handled by the directors module, and I have to admit that I'm not impressed by its features as it lacks support for least connections which is the chief method with Plone instances. On the other side, backend definitions and health checks are more configurable, but support HTTP requests only. You may consider Varnish as a load balancer if you already use it on high traffic sites and is going to do a good and honest work.

(Previously, I was arguing in favor of the hash method, but later tests showed that the standard round robin is less prone to instance overloads.)

As you can see both nginx and Varnish can only handle HTTP requests for health checks and we have very limited control of request queues, something that in my experience has prove to be an issue.

(I never care about the ngnix queue, but in Varnish I always configure a max_connections directive to avoid flooding instances with requests that could make them unreachable for a long time.)

Let's go back on track: HAProxy is a very complex piece of software; it's so complex and its documentation so huge that I preferred just to removed it from the equation when I first had to take care of the infrastructure in our company some years ago.

That choice proved to be right at the moment, but I always like to investigate and act as my own devil's advocate from time to time. So, when one of our main sites started receiving some very nasty DoS attacks I started playing with it again.

It took me a whole week to prepare, test and review the configuration I'm going to share with you, but first I'm going to talk a little bit about our infrastructure and rationale.

The sites I'm talking about use (at least) 2 servers with a full stack: nginx, Varnish, HAProxy and Zope/Plone; both servers are configured using ZODB Replicated Storage (ZRS), one acting as the master ZEO server and the other as the slave. All Plone instances point to both ZEO servers. When the master ZEO server fails, all instances connect automatically to the slave in read-only mode. When all instances in one server fail, Varnish connects to the HAProxy in the other server as a backup. When Varnish fails, nginx tries to use the Varnish on the other server. That's more or less how it works.

This is our working haproxy.cfg and I going to explain some choices I made:


First the standard listen section: we use option tcp-check (layer 4) to make the health checks as is way faster that doing a HTTP (layer 7) check: we ask Zope on the alternate binded port for the ok command and expect the OK string as a result.

In my tests layer 4 checks took just a couple of milliseconds to finish, 2 orders of magnitude less than layer 7 checks. Faster, non-blocking responses mean you can check more frequently and you're going to detect sooner any failure on the instance.

Health checks are done every 2 seconds and, in case of failure, the slowstart parameter avoids flooding an instance during it's defined warming up period of one minute.

We inform the status of the backend upstream to Varnish using the monitor-uri directive: if we have no backends alive, HAProxy's monitor will also fail.

In Varnish we have have configured the backend like this:


Varnish will detect any backend failure in 2 seconds, at worst; Varnish will then try to use the HAProxy backend on the other server as a backup.

Back into HAProxy's configuration, we set maxconn 4 on backend servers to avoid sending more that 4 request to any instance; similarly, we set timeout queue 1m to keep the queue size under control.

Finally, the global section: you'll see we're using maxconn 256; in my opinion there is no reason to use a bigger value: if you have more that 256 requests on queue, you obviously have a problem and you need more instances to serve the traffic.

The typical stats screen of HAProxy with this configuration looks like this:

As you can see, over a period of almost 25 days we have no backend failures in this server, even with a large number of individual instance failures.

Last, but not least: if you want to cut the start up time of your instances you have to include the following directives on your buildout configuration (more about that on this Plone forum thread):

[instance]
…
# play with this value depending on the number of objects in your ZODB
zeo-client-cache-size = 256MB
zeo-client-client = zeoclient

Share and enjoy!

We have been doing health checks wrong in Plone

In the previous post of this blog I was arguing on how to increase the performance of high-traffic Plone sites. As mentioned there, we have different ways to do so: increasing the number of server threads, increasing the number of instances, and increasing both.

Increasing the number of threads is easier and consumes less memory but, as mentioned, is less resilient and can be affected by the Python GIL on multicore servers. Increasing the number of instances, on the other side, will increase the complexity of our stack as we will need to install a load balancer, a piece of hardware or software that distributes the incoming traffic across all the backend instances.

Over the years we have tried different software load balancers depending on the traffic of a site: we use nginx alone as web server and load balancer on smaller sites and we add Varnish as web accelerator and load balancer when the load increases; lately we started using HAProxy again to solve extreme problems on sites being continuously attacked (I'll write about on a different post).

When you use a load balancer you have to do health checking, as the load balancer needs to know when one of the backend instances has become unavailable because is quite busy to answer further requests, has been restarted, is out for maintenance, or is simply dead. And, in my opinion, we have been doing it wrong.

The typical configuration for health checking is to send requests to the same port used by the Zope HTTP server and this has some fundamental problems: First, the Zope HTTP server is slow to answer requests: it can take hundreds of milliseconds to answer even the most simple HEAD request.

To make things worst, the requests that are answered by the Zope HTTP server are the slower ones (content not in ZODB cache, search results, you name it…), as the most common requests are already being served by the intermediate caches. In the tests I made, I found that even a well configured server running a mature code base can take as many as 10 seconds to answer this kind of requests. This is a huge problem as health check requests start queuing and timing out taking perfectly functional instances out of the pool, and making things just worst.

To avoid this problem we normally configure health checks with long intervals (typically 10 seconds) and windows of 3 failures for every 5 checks. And this, of course, creates another problem as the load balancer takes, in our case, up to 30 seconds to discover that an instance has been restarted when using things like Supervisor and its memmon plugin, leading to a lot of 503 Service Unavailable errors in the mean time.

So, it's a complete mess no mater how you analyze it and I needed to find a way to solve it: enters five.z2monitor.

five.z2monitor plugs zc.monitor and zc.z3monitor into Zope 2, enabling another thread and port to handle monitoring. To install it you just need to add something like this into your buildout configuration:

[buildout]
eggs =
    …
    five.z2monitor
zcml =
    …
    five.z2monitor

[instance]
…
zope-conf-additional =
    <product-config five.z2monitor>
        bind 127.0.0.1:8881
    </product-config>


After running buildout and restarting your instance you can communicate with your Zope server over the new port using different commands, called probes:

$ bin/instance monitor help
Supported commands:
  dbinfo -- Get database statistics
  help -- Get help about server commands
  interactive -- Turn on monitor's interactive mode
  monitor -- Get general process info
  ok -- Return the string 'OK'.
  quit -- Quit the monitor
  zeocache -- Get ZEO client cache statistics
  zeostatus -- Get ZEO client status information

Suppose you want to get the ZEO client cache statistics for this instance; all you have to do is use the following command:

$ bin/instance monitor zeocache main
417554 895451465 435095 900622900 35429160

You can also use the Netcat utility to get the same information:

$ echo 'zeocache main' | nc -i 1 127.0.0.1 8881
417753 896710955 435422 901905068 35467686

It's easy to extend the list of supported commands by writing you own probes; in the list above I have added to the default command set one that I create to know if the server is running or not; it's called "ok", and here is its source code:


We have now a dedicate port and thread that can be used for health checking:

$ echo 'ok' | nc -i 1 127.0.0.1 8881
OK

With this we solved most of the problems I mentioned above: we have faster response time and no queuing; we can decrease the health check interval to a couple of seconds and we are almost sure that a failure is a failure and not just a timeout.

Note we can't use this with nginx, nor Varnish, as their health checks are limited and expect the same port used for HTTP requests; only HAProxy supports this configuration.

So, in the next post I'll show you how to configure HAProxy health checks to use this probe and how to reduce the latency to a couple of milliseconds.

Plone performance: threads or instances?

Recently we had a discussion on the Plone community forum on how to increase performance of Plone-based sites.

I was arguing in favor of instances, because some time ago I read a blog post by davisagli taking about the impact of the Python GIL on performance of multicore servers. Others, like jensens and djay, were skeptical on this argument and told me not to overestimate that.

So, I decide to test this using the production servers of one of our customers.

The site is currently running on 2 different DigitalOcean servers with 4 processors and 8GB RAM; we are using Cloudflare in front of it, and round-robin, DNS-based load balancing.

Prior to my changes, both servers were running with the same configuration:

• nginx, doing caching of static files and proxy rewrites
• Varnish, doing caching and URL-based load balancing
• 4 Plone instances running on ZEO client mode, with 1 thread and 100.000 objects in cache

Both servers where running also a ZEO server on ZRS configuration, one as a master and the other as a slave, doing blob storage replication.

First, here we have some information from this morning, before I made the changes. Here are some graphics I obtained using New Relic on the master:

Here is the same information from the slave:

As you can see, everything is running smoothly: CPU consumption is low and memory consumption is high and… yes, we have an issue with some PhamtomJS processes left behind.

This is what I did later:

• on the master server, I restarted the 4 instances
• on the slave server, I changed the configuration of instance1 to use 4 threads and restarted it; I stopped the other 3 instances

I also stopped the memmon Supervisor plugin (just because I had no idea on how much memory the slave server instance will be consuming after the changes), and killed all PhamtomJS processes.

The servers have been running for a couple of hours now and I can share the results. This is the master server:

And this is now the slave:

The only obvious change here is in memory consumption: wow! the sole instance on the slave server is consuming 1GB less than the 4 instances in the master server!

Let's do a little bit more research now. Here we have some information on database activity on the master server (just one instance for the sake of simplicity):

Now here is some similar information for the slave server:

I can say that I was expecting this: there's a lot more activity and the caching is not very well utilized on the slave server (see, smcmahon, that's the beauty of the URL-based load balancing on Varnish demonstrated).

Let's try a different look, now using the vmstat command:

Not many differences here: the CPU is idle most of the time and the interrupts and context switching are almost the same.

Now let's see how much our instances are being used, with the varnishstat command:

Here you can see why there's not too much difference: in fact Varnish is taking care of nearly 90% of the requests and we have only around 3 requests/second hitting the servers.

Let's make another test to see how quickly we are responding the requests using the varnishhist command:

Again, there's almost no difference here.

Conclusion: for our particular case, using threads seems not to affect too much the performance and has a positive impact on memory consumption.

What I'm going to do now is to change the configuration used in production to have 2 instances and 2 threads… why? because restarting a single instance on a server for maintenance purposes would let us without backends during the process if we were using only one instance.

Share and enjoy!

Using pep257, a Python docstring style checker, with Buildout

PEP 257 documents the semantics and conventions associated with Python docstrings. According to it:

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.

This is specially useful, for instance, when you are introspecting code and you want to know what a specific method or function does.

pep257 is a Python docstring style checker.

I already filled a feature request on plone.recipe.codeanalysis to add support for it; meanwhile, if you want to use it in your Buildout-based projects, you can add the following to your buildout configuration:

After running bin/buildout you will find a bin/pep257 script.

A typical output will look like this:

(python-2.7)# hvelarde@nanovac (master * u+1) ~/collective/polls 
# bin/pep257 src/
src/collective/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:8 in public class `LegendOthersTranslation`:
        D101: Docstring missing
src/collective/polls/js_i18n_helper.py:14 in public method `render`:
        D102: Docstring missing
src/collective/polls/testing.py:1 at module level:
        D100: Docstring missing
src/collective/polls/testing.py:10 in public class `Fixture`:
        D101: Docstring missing
src/collective/polls/testing.py:14 in public method `setUpZope`:
        D102: Docstring missing
src/collective/polls/testing.py:19 in public method `setUpPloneSite`:
        D102: Docstring missing
src/collective/polls/config.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:21 in public class `IPolls`:
        D101: Docstring missing
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:52 in public class `Polls`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:52 in public class `Polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:52 in public class `Polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:60 in public method `ct`:
        D102: Docstring missing
src/collective/polls/polls.py:64 in public method `mt`:
        D102: Docstring missing
src/collective/polls/polls.py:68 in public method `wt`:
        D102: Docstring missing
src/collective/polls/polls.py:72 in public method `member`:
        D102: Docstring missing
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D400: First line should end with '.', not '?'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D400: First line should end with '.', not 'w'
src/collective/polls/polls.py:181 in public method `render_portlet`:
        D202: No blank lines allowed *after* method docstring, found 1
src/collective/polls/polls.py:251 in public method `render`:
        D400: First line should end with '.', not 'e'
src/collective/polls/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D400: First line should end with '.', not 'f'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D400: First line should end with '.', not 't'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/setuphandlers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/setuphandlers.py:12 in public class `HiddenProfiles`:
        D101: Docstring missing
src/collective/polls/setuphandlers.py:18 in public method `getNonInstallableProfiles`:
        D102: Docstring missing
src/collective/polls/setuphandlers.py:23 in public function `updateWorkflowDefinitions`:
        D103: Docstring missing
src/collective/polls/setuphandlers.py:29 in public function `setupVarious`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:1 at module level:
        D100: Docstring missing
src/collective/polls/portlet/voteportlet.py:24 in public function `PossiblePolls`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D400: First line should end with '.', not 't'
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:87 in public class `Assignment`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:102 in public method `__init__`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D400: First line should end with '.', not 'e'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:118 in public class `Renderer`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:129 in public method `utility`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:135 in public method `portlet_manager_name`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:149 in public method `poll`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:170 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:176 in public method `getVotingResults`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:184 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:193 in public method `available`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:202 in public method `is_closed`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:217 in public method `create`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:7 in public function `uninstall`:
        D103: Docstring missing
src/collective/polls/Extensions/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/tests/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:34 in public class `InsuficientOptions`:
        D101: Docstring missing
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:80 in public method `validate_options`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D400: First line should end with '.', not 'e'
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:101 in public method `annotations`:
        D102: Docstring missing
src/collective/polls/content/poll.py:105 in public method `utility`:
        D102: Docstring missing
src/collective/polls/content/poll.py:109 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:115 in private method `_getVotes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:133 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:144 in private method `_validateVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:155 in private method `_setVoter`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:180 in public method `voters`:
        D102: Docstring missing
src/collective/polls/content/poll.py:186 in public method `total_votes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:192 in public method `setVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:223 in public method `create`:
        D102: Docstring missing
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:241 in public method `updateWidgets`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:258 in public method `applyChanges`:
        D102: Docstring missing
src/collective/polls/content/poll.py:270 in public class `View`:
        D101: Docstring missing
src/collective/polls/content/poll.py:277 in public method `update`:
        D102: Docstring missing
src/collective/polls/content/poll.py:346 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/content/poll.py:357 in public method `can_edit`:
        D102: Docstring missing
src/collective/polls/content/poll.py:362 in public method `has_voted`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:371 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:377 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:382 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines

How to get statistics about your contributions on a GitHub organization

GitHub, the web-based hosting service for software development projects, lets you check statistics on your contributions on specific projects easily.

You can check, for instance, the statistics on collective.cover just by visiting the following page:

https://github.com/collective/collective.cover/graphs/contributors

GitHub gives you a count of commits and a visual representation of them among time.

If you want to check only the number of commits by author, you could use the git-shortlog command:

The -s option gives you a summary output and the sort command shows the information sorted in reverse order (-nr).

You could improve this listing by using the .mailmap feature to add commits belonging to the same author using two different email addresses.

Image now that you want to get a list of all your contributions on a specific organization, lets say, the Plone collective.

Enter the GitHub API and github3.py, a Python wrapper for it.

Until today, the Plone collective has more that 1.1k repositories so we need to use the authenticated access to the API to avoid depletion of requests (rate limit allow us to make up to 60 requests per hour for unauthenticated requests and 5,000 for authenticated requests). In this specific example we used 1,171 requests to the API.


Using github3.py we iterate over all of the organization repositories and get information of each one only if my user name is listed among the contributors. We get the results as a list of tuples (repo, total).

After that we sort the list in reverse order using the total commits as the key and make the sum of all commits in general.

As you can see I have contributed to 74 repositories on the Plone collective making 2,975 commits in total.

Not bad, isn't it? :-)

Integrating Travis CI with your Plone add-ons hosted on GitHub

Update 15/9/2012: Kudos for Asko Soukka who has developed an alternative method of installing Plone using the old good universal installer that reduces the amount of time needed by half. So go and read his post instead of loosing your time with mine.

I took me a little bit but, with the help of Mikko and Martin, I've got a couple of add-ons running tests with Travis CI.

Before setting up Travis CI, you have to make some changes to the Setup Script of your package.

In my case, my add-on package only works for Plone versions 4.1 and later, so I have added Products.CMFPlone as a dependency:

    …
    install_requires=[
        'setuptools',
        'Products.CMFPlone>=4.1',
        ],
    extras_require={
        'test': ['plone.app.testing'],
        },
    …

Products.CMFPlone contains a cut down feature set: just the things I need to run my tests.

Setting up Travis CI is pretty easy: just sign in and activate your GitHub Service Hook

Now, you need to configure your Travis CI build with a .travis.yml file in the root of your repo:

In my case I'm running tests for Plone's latest stable release (4.2 at I write this post) on top of Python 2.7.

Let's take a look at the travis.cfg buildout configuration file:

The main issue I experimented on my first tests was timeouts, so I have a couple of tricks here for you: first, we are extending the standard Plone testing buildout configuration that includes most declarations for us and takes care of always running the latest stable version; we are only going to use the test part. You need to add the package-extras just if your add-on is using plone.app.testing on the test option in extras_require of your package declaration as mentioned above.

zope.globalrequest is needed to run the tests and it was not included on Products.CMFPlone (this is already fixed on Plone's branches for versions 4.2 and 4.3). You may also need to include Pillow in test-eggs; just uncomment it the line.

We also need to add a socket-timeout of 3 seconds (only available on zc.buildout >= 1.5.0) and a list of allow-hosts to download the dependencies. This is pretty important and will avoid timeouts as Travis CI has hard time limits and timeouts are between 10 and 15 minutes for test suite runs (1).

Last, we have to replace the eggs option on the test part; we need to do this because we don't want to include neither Plone or plone.app.upgrade on the tests.

Finally, you can add a Status Image with a link back to the result of your last build on your README.txt file:

.. image:: https://secure.travis-ci.org/collective/your.package.png
    :target: http://travis-ci.org/collective/your.package

To run the tests you only need to make a push to your GitHub repo. Easy, isn't it?

For a live example of all I mentioned above, take a look at the collective.prettydate package.

Travis CI is really easy to set up and fun to use; I strongly recommend it and, if you like it, please show your love donating.