Using Threads in Twisted

Introduction

Before you start using threads, make sure you do at the start of your program:

from twisted.python import threadable
threadable.init()

This will make certain parts of Twisted thread-safe so you can use them safely. However, note that most parts of Twisted are not thread-safe.

Running code in a thread-safe manner

Most code in Twisted is not thread-safe. For example, writing data to a transport from a protocol is not thread-safe. Therefore, we want a way to schedule methods to be run in the main event loop. This can be done using the function twisted.internet.interfaces.IReactorCore.callFromThread:

from twisted.internet import reactor
from twisted.python import threadable
threadable.init(1)

def notThreadSafe(x):
     """do something that isn't thread-safe"""
     # ...

def threadSafeScheduler():
    """Run in thread-safe manner."""
    reactor.callFromThread(notThreadSafe, 3) # will run 'notThreadSafe(3)' in the event loop

Running code in threads

Sometimes we may want to run methods in threads - for example, in order to access blocking APIs. Twisted provides methods for doing so using the IReactorThreads API (twisted.internet.interfaces.IReactorThreads). Additional utility functions are provided in twisted.internet.threads. Basically, these methods allow you to queue methods to be run by a thread pool.

For example, to run a method in a thread we can do:

from twisted.internet import reactor

def aSillyBlockingMethod(x):
    import time
    time.sleep(2)
    print x

# run method in thread
reactor.callInThread(aSillyBlockingMethod, "2 seconds have passed")

For functions whose results we wish to get, we can have the result returned as a Deferred:

from twisted.internet import threads

def doLongCalculation():
    # .... do long calculation here ...
    return 3

def printResult(x):
    print x

# run method in thread and get result as defer.Deferred
d = threads.deferToThread(doLongCalculation)
d.addCallback(printResult)
d.arm()