This talk is going to be about old code patterns, both real and mythical!

Because the standard patterns in Python has changed throughout time, as Python gained more features.

I'll start off with a patterns about dictionaries, and this one is a fairly common pattern.

When you have many values per key, so that every value in your dictionary should be a list or a set or a tuple or another dictionary, or in fact anything mutable, you use a defaultdict.

Essentially, if you access a key that doesn't exist, it creates the key!

Well, you did your check manually.

And it isn't. Except on Jython.

Using a defaultdict is 1.6 times faster on CPython, 1.2 times on PyPy, and for some reason less three times as slow on Jython!

I guess the Jython defaultdict implementation is very unoptimized.

Using defaultdict is less code = less bugs and it's faster as well!

Using setdefault is shorter, and clearer, right? And less code and more clarity means less bugs?

Well...

Not only is setdefault misnamed. It sounds like it sets a default. In fact what it does is get a value of a key, and if that key does not exist, then it sets it to a default. Hence most people simply don't know what it does.

Secondly, as you see in the example above, it needs to create an empty set for each call, even if it doesn't actually use it.

So setdefault makes more sense in cases where the default value isn't mutable.

OK, enough about defaultdict. Now let's talk about the standard dictionary!

It has also changed quite a lot throughout time. The most common changed you are likely to encounter are 'keys()' and 'has_key()'.

And now you say that keys() is not outdated, but in fact it mostly is. Up until Python 2.1, you had to use the keys() method to get a list if the keys, if you wanted to loop over them for example.

Today you would simply do for x in mydict instead. This avoids creating a separate list object and using up memory for that. Sure, you can also use iterkeys(), but there is no point, and iterkeys() is gone in Python 3. So the code shown here is the best way to do it.

The same goes for making a list of the keys. The old way is to call keys(). But in Python 3 keys() doesn't return a list anymore.

So the best way is to not use keys() or iterkeys(), and just pass the dictionary directly to the list constructor.

And the same goes for the old has_key().

OK, enough about dictionaries, now let's talk about sets!

Sets are useful, the values in a set must be unique, lookup in sets are fast, although they aren't ordered.

Sets first appeared as a standard library module in Python 2.3, and as a built in type in Python 2.4.

So what did you do before? What else do we have that has Unique values, fast lookup and no ordering?

Yes! Dictionary keys! So in fact I lied, this pattern isn't about sets, it's also about dictionaries!

This code example makes a list unique by putting it into a dictionary as keys with a value of None, and then getting a list of keys back.

I could not, to my dissapointment find any examples of this in Django. :-)

Another usage of dictionary keys like this is when you wanted to do very fast lookups. Checking if a value exists in a dictionary is way faster than checking if it exists in a list.

This is simply looking if a value exists in a dictionary vs a list. Data is random integers.

And as you see, dictionaries are way faster than lists. So it used to be a pattern that if you needed to do that a lot, you used a dictionary.

However, sets are a little bit faster than dictionaries.

OK, enough with dictionaries for real now. Now lets talk about sorting. This code is also from Django 1.5.1.

Now of course, the first mistake in this code is to use a list in the first place. That's not a prehistoric pattern, I think it's just a mistake in the code in this case, likely the list(set()) call was added later than the main loop.

Sure, updating lists are faster than updating sets, but first creating a long list and then making it a set is not faster than using a set from the start.

But the point here is this change. Instead of creating a list and then sorting it, you can now use sorted().

The next old sorting pattern is all about speed. And this is nothing you will find in Django 1.5, because this doesn't even work under Python 3.

So this example is from Plone, and in fact an old version of Plone, Plone 4.0.

This is easier to read, but it has the same end-result.

Reference: Jarret Hardie in Python Magazine

But also since Python 2.4 we can sort with a key function instead.

Yeah, you get exactly one call per item, always. With the earlier code, we get in average 680,000 calls to the modified() method when sorting 40.000 items.

Now we get 40,000 calls. That's 1/17th the amount of calls. Which essentially means that sorting 40,000 items takes just a tenth of the time.

This looks like a logic expression, but it isn't. It's a sneaky conditonal! If means that if include_blank is True, then first_choice gets set to blank_choice other wise it's an empty list.

But blank_choice is a parameter. What if it is something that evaluates to false, like a None or an empty set?

Yes: first_choice will be an empty list, not what you pass in as blank_choice.

In this example from Django, this is not an important issue, because a blank blank_choice makes no sense. But a blank blank_choice should really result in an error because explicit is better than implicit.

This is the new syntax for one line conditionals. When I say "New" I mean since Python 2.5.

This is a pattern that was suggested to me that I should bring up. And I wasn't going to do it until I started benchmarking it.

Here we see something simple, calculating a constant outside the loop. That should speed up the loop, right because you don't have to calculate the constant, right?

Well, kinda. It used to be much faster, but since Python 2.5 it isn't. CPython will find that multiplication and calculate only once. In Jython it's still marginally faster to calculate it outside.

PyPy of course is ridicolously fast with this code, it does this some 30-40 times faster than Python 2.7.

So if you have a division in the calculation, the Python 2.7 gets slow again!

Python 3.3 and PyPy are still fine, though.

But of course, my example is stupid. 5 * 3.5 is actually 17.5, so when you have constants, you can simply change the code to the constant! Problem solved!

Here the constant is "semi-constant" and we multiply with each item in the iterable. This makes more sense.

Now the optimization dissappeared. Calculating the constant outside of the loop is now faster again.

Except on PyPy which still succeeds in optimizing this.

Unless you use a power in the calculation of the constant, where PyPy's optimization also dissapears!

On PyPy it's now 33 times faster to calculate this constant outside the loop! But still twice as fast as Python 2.7.

So this pattern turns out not to be prehistoric at all!

You should calculate constants outside of the loop.

And now, the prehistoric pattern that was the catalyst for this talk.

You'll hear many people claiming that concatenating strings with + is slow, and that doing a join is faster. But, since CPython 2.5 there are optimizations in string concatenation, so now it is fast.

But of course, not on PyPy. At least according to the PyPy people. Unless you have a compile time parameter, apparently.

So let's look at the benchmarks.

These benchmarks have been a big problem. It's been very hard to get something sensible, simple, that measures actual concatention, and doesn't get completely optimized away by PyPy.

And this is the best I can do. It adds strings between 0 and 999 characters long. There is overhead in the tests, but I believe that it's not enough to make a significant difference to the numbers.

And you see that using addition to concatenate is faster. Even on Python 2.4!

So where does this claim that join is faster come from? I think this is a big misunderstandning.

This only copies each of the strings once.

This also copies the strings ony once, but it goes via creating a list. And creating that list also takes time.

So if adding strings are fast when you are adding two strings, and joining is fast if you have many strings, where is the breakpoint?

Well, it depends. It depends on how long your strings are and how many you have. With typical cases it seems join() is faster on CPython at somewhere around 4-5 strings.

With PyPy up to ten strings are still as fast to use addition as to use join, and I stopped testing there because it was getting silly.

I like alliteration. Can you tell?

The conclusion is that you should do what feels natural. If the easiest way to concatenate a bunch of strings is by using +, then do that. If the strings you have are in a list or generated in a loop, then use join.

And it's the same with calculating constants outside of the loop. It feels like it should be faster, and it often is. Python is such a fantastic language partly because what intuitively feels like the right thing to do, tends to in fact be the right thing to do.