Input:
What's the difference between list and tuples? What's the difference?
What are the advantages / disadvantages of tuples / lists?

Output:
Apart from tuples being immutable there is also a semantic distinction that should guide their usage. Tuples are heterogeneous data structures (i.e., their entries have different meanings), while lists are homogeneous sequences. Tuples have structure, lists have order.
Using this distinction makes code more explicit and understandable.
One example would be pairs of page and line number to reference locations in a book, e.g.:
my_location = (42, 11)  # page number, line number

You can then use this as a key in a dictionary to store notes on locations. A list on the other hand could be used to store multiple locations. Naturally one might want to add or remove locations from the list, so it makes sense that lists are mutable. On the other hand it doesn't make sense to add or remove items from an existing location - hence tuples are immutable.
There might be situations where you want to change items within an existing location tuple, for example when iterating through the lines of a page. But tuple immutability forces you to create a new location tuple for each new value. This seems inconvenient on the face of it, but using immutable data like this is a cornerstone of value types and functional programming techniques, which can have substantial advantages.
There are some interesting articles on this issue, e.g. "Python Tuples are Not Just Constant Lists" or "Understanding tuples vs. lists in Python". The official Python documentation also mentions this ("Tuples are immutable, and usually contain an heterogeneous sequence ...").
In a statically typed language like Haskell the values in a tuple generally have different types and the length of the tuple must be fixed. In a list the values all have the same type and the length is not fixed. So the difference is very obvious.
Finally there is the namedtuple in Python, which makes sense because a tuple is already supposed to have structure. This underlines the idea that tuples are a light-weight alternative to classes and instances.


Input:
Import a module from a relative path How do I import a python module given its relative path?
For example, if dirFoo contains Foo.py and dirBar, and dirBar contains Bar.py, how do I import Bar.py into Foo.py?
Here's a visual representation:
dirFoo\
    Foo.py
    dirBar\
        Bar.py

Foo wishes to include Bar, but restructuring the folder hierarchy is not an option.

Output: Be sure that dirBar has the __init__.py file -- this makes a directory into a Python package.


Input: ASCII value of a character in Python How do I get the ASCII value of a character as an int in Python?

Output:
From here:

function ord() would get the int value
  of the char. And in case you want to
  convert back after playing with the
  number, function chr() does the trick.

>>> ord('a')
97
>>> chr(97)
'a'
>>> chr(ord('a') + 3)
'd'
>>>

There is also the unichr function, returning the Unicode character whose ordinal is the unichr argument:
>>> unichr(97)
u'a'
>>> unichr(1234)
u'\u04d2'


Input:
What is a mixin, and why are they useful? In "Programming Python", Mark Lutz mentions "mixins". I'm from a C/C++/C# background and I have not heard the term before. What is a mixin?
Reading between the lines of this example (which I've linked to because it's quite long), I'm presuming it's a case of using multiple inheritance to extend a class as opposed to 'proper' subclassing. Is this right?
Why would I want to do that rather than put the new functionality into a subclass? For that matter, why would a mixin/multiple inheritance approach be better than using composition?
What separates a mixin from multiple inheritance? Is it just a matter of semantics?

Output:
A mixin is a special kind of multiple inheritance.  There are two main situations where mixins are used:

You want to provide a lot of optional features for a class.
You want to use one particular feature in a lot of different classes.

For an example of number one, consider werkzeug's request and response system.  I can make a plain old request object by saying:
from werkzeug import BaseRequest

class Request(BaseRequest):
    pass

If I want to add accept header support, I would make that
from werkzeug import BaseRequest, AcceptMixin

class Request(BaseRequest, AcceptMixin):
    pass

If I wanted to make a request object that supports accept headers, etags, authentication, and user agent support, I could do this:
from werkzeug import BaseRequest, AcceptMixin, ETagRequestMixin, UserAgentMixin, AuthorizationMixin

class Request(BaseRequest, AcceptMixin, ETagRequestMixin, UserAgentMixin, AuthorizationMixin):
    pass

The difference is subtle, but in the above examples, the mixin classes weren't made to stand on their own.  In more traditional multiple inheritance, The AuthenticationMixin (for example) would probably be something more like Authenticator.  That is, the class would probably be designed to stand on its own.


Input:
Convert bytes to a Python string I'm using this code to get standard output from an external program:
>>> from subprocess import *
>>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0]

The communicate() method returns an array of bytes:
>>> command_stdout
b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n'

However, I'd like to work with the output as a normal Python string. So that I could print it like this:
>>> print(command_stdout)
-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1
-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2

I thought that's what the binascii.b2a_qp() method is for, but when I tried it, I got the same byte array again:
>>> binascii.b2a_qp(command_stdout)
b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n'

Does anybody know how to convert the bytes value back to string? I mean, using the "batteries" instead of doing it manually. And I'd like it to be ok with Python 3.

Output:
You need to decode the bytes object to produce a string:
>>> b"abcde"
b'abcde'

# utf-8 is used here because it is a very common encoding, but you
# need to use the encoding your data is actually in.
>>> b"abcde".decode("utf-8")
'abcde'


Input:
Most elegant way to check if the string is empty in Python? Does Python have something like an empty string variable where you can do?:
if myString == string.empty:

Regardless what's the most elegant way to check for empty string values? I find hard coding "" every time for checking an empty string not as good.

Output:
Empty strings are "falsy" which means they are considered false in a Boolean context, so you can just do this:
if not myString:

This is the preferred way if you know that your variable is a string.  If your variable could also be some other type then you should use myString == "".  See the documentation on Truth Value Testing for other values that are false in Boolean contexts.


Input:
How can I do a line break (line continuation) in Python? I have a long line of code that I want to break up among multiple lines. What do I use and what is the syntax?
For example, adding a bunch of strings,
e = 'a' + 'b' + 'c' + 'd'

and have it like this:
e = 'a' + 'b' +
    'c' + 'd'

Output:
What is the line?  You can just have arguments on the next line without any problems:
a = dostuff(blahblah1, blahblah2, blahblah3, blahblah4, blahblah5,
            blahblah6, blahblah7)

Otherwise you can do something like this:
if a == True and \
   b == False

Check the style guide for more information.
From your example line:
a = '1' + '2' + '3' + \
    '4' + '5'

Or:
a = ('1' + '2' + '3' +
    '4' + '5')

Note that the style guide says that using the implicit continuation with parentheses is preferred, but in this particular case just adding parentheses around your expression is probably the wrong way to go.


Input:
How can I get a list of locally installed Python modules? I would like to get a list of Python modules, which are in my Python installation (UNIX server).
How can you get a list of Python modules installed in your computer?

Output:
help('modules')

in a Python shell/prompt.


Input:
How to use threading in Python? I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them.
How do you clearly show tasks being divided for multi-threading?

Output:
Here's a simple example: you need to try a few alternative URLs and return the contents of the first one to respond.
import Queue
import threading
import urllib2

# called by each thread
def get_url(q, url):
    q.put(urllib2.urlopen(url).read())

theurls = ["http://google.com", "http://yahoo.com"]

q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target=get_url, args = (q,u))
    t.daemon = True
    t.start()

s = q.get()
print s

This is a case where threading is used as a simple optimization: each subthread is waiting for a URL to resolve and respond, in order to put its contents on the queue; each thread is a daemon (won't keep the process up if main thread ends -- that's more common than not); the main thread starts all subthreads, does a get on the queue to wait until one of them has done a put, then emits the results and terminates (which takes down any subthreads that might still be running, since they're daemon threads).
Proper use of threads in Python is invariably connected to I/O operations (since CPython doesn't use multiple cores to run CPU-bound tasks anyway, the only reason for threading is not blocking the process while there's a wait for some I/O).  Queues are almost invariably the best way to farm out work to threads and/or collect the work's results, by the way, and they're intrinsically threadsafe so they save you from worrying about locks, conditions, events, semaphores, and other inter-thread coordination/communication concepts.


Input:
How to remove a key from a python dictionary? When trying to delete a key from a dictionary, I write:
if 'key' in myDict:
    del myDict['key']

Is there a one line way of doing this?

Output:
Use dict.pop():
my_dict.pop('key', None)

This will return my_dict[key] if key exists in the dictionary, and None otherwise. If the second parameter is not specified (ie. my_dict.pop('key')) and key does not exist, a KeyError is raised.


Input: How do I check what version of Python is running my script? How can I check what version of the Python Interpreter is interpreting my script?

Output:
This information is available in the sys.version string in the sys module:
>>> import sys

Human readable:
>>> print (sys.version) #parentheses necessary in python 3.
2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)]

For further processing:
>>> sys.version_info
(2, 5, 2, 'final', 0)
# or
>>> sys.hexversion
34014192

To ensure a script runs with a minimal version requirement of the Python interpreter add this to your code:
assert sys.version_info >= (2,5)

This compares major and minor version information. Add micro (=0, 1, etc) and even releaselevel (='alpha','final', etc) to the tuple as you like. Note however, that it is almost always better to "duck" check if a certain feature is there, and if not, workaround (or bail out). Sometimes features go away in newer releases, being replaced by others.


Input:
How do I parse XML in Python? I have many rows in a database that contains xml and I'm trying to write a Python script that will go through those rows and count how many instances of a particular node attribute show up. For instance, my tree looks like:
<foo>
   <bar>
      <type foobar="1"/>
      <type foobar="2"/>
   </bar>
</foo>

How can I access the attributes 1 and 2 in the XML using Python?

Output:
I suggest ElementTree.  There are other compatible implementations of the same API, such as lxml, and cElementTree in the Python standard library itself; but, in this context, what they chiefly add is even more speed -- the ease of programming part depends on the API, which ElementTree defines.
After building an Element instance e from the XML, e.g. with the XML function, or by parsing a file with something like
import xml.etree.ElementTree
e = xml.etree.ElementTree.parse('thefile.xml').getroot()

or any of the many other ways shown at ElementTree, you just do something like:
for atype in e.findall('type'):
    print(atype.get('foobar'))

and similar, usually pretty simple, code patterns.


Input: How do you change the size of figures drawn with matplotlib? How do you change the size of figure drawn with matplotlib?

Output:
figure tells you the call signature:
figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')

So figure(figsize=(1,1)) creates an inch-by-inch image, which will be 80-by-80 pixels unless you also give a different dpi argument.


Input:
Parsing values from a JSON file in Python I have this JSON in a file:
{
    "maps": [
        {
            "id": "blabla",
            "iscategorical": "0"
        },
        {
            "id": "blabla",
            "iscategorical": "0"
        }
    ],
    "masks": [
        "id": "valore"
    ],
    "om_points": "value",
    "parameters": [
        "id": "valore"
    ]
}

I wrote this script which prints all of the json text:
json_data=open(file_directory).read()

data = json.loads(json_data)
pprint(data)

How can I parse the file and extract single values?

Output:
I think what Ignacio is saying is that your JSON file is incorrect. You have [] when you should have {}. [] are for lists, {} are for dictionaries.
Here's how your JSON file should look, your JSON file wouldn't even load for me:
{
    "maps": [
        {
            "id": "blabla",
            "iscategorical": "0"
        },
        {
            "id": "blabla",
            "iscategorical": "0"
        }
    ],
    "masks": {
        "id": "valore"
    },
    "om_points": "value",
    "parameters": {
        "id": "valore"
    }
}

Then you can use your code:
import json
from pprint import pprint

with open('data.json') as data_file:
    data = json.load(data_file)

pprint(data)

With data, you can now also find values like so:
data["maps"][0]["id"]
data["masks"]["id"]
data["om_points"]

Try those out and see if it starts to make sense.


Input:
How can I print a literal "{}" characters in python string and also use .format on it? x = " \{ Hello \} {0} "
print x.format(42)

gives me : Key Error: Hello\\
I want to print the output: {Hello} 42

Output:
You need to double the {{ and }}:
>>> x = " {{ Hello }} {0} "
>>> print x.format(42)
' { Hello } 42 '

Here's the relevant part of the Python documentation for format string syntax:

Format strings contain âreplacement fieldsâ surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: {{ and }}.


Input: If Python is interpreted, what are .pyc files? I've been given to understand that Python is an interpreted language... However, when I look at my Python source code I see .pyc files, which Windows identifies as "Compiled Python Files". Where do these come in?

Output:
I've been given to understand that
  Python is an interpreted language...

This popular meme is incorrect, or, rather, constructed upon a misunderstanding of (natural) language levels: a similar mistake would be to say "the Bible is a hardcover book".  Let me explain that simile...
"The Bible" is "a book" in the sense of being a class of (actual, physical objects identified as) books; the books identified as "copies of the Bible" are supposed to have something fundamental in common (the contents, although even those can be in different languages, with different acceptable translations, levels of footnotes and other annotations) -- however, those books are perfectly well allowed to differ in a myriad of aspects that are not considered fundamental -- kind of binding, color of binding, font(s) used in the printing, illustrations if any, wide writable margins or not, numbers and kinds of builtin bookmarks, and so on, and so forth.
It's quite possible that a typical printing of the Bible would indeed be in hardcover binding -- after all, it's a book that's typically meant to be read over and over, bookmarked at several places, thumbed through looking for given chapter-and-verse pointers, etc, etc, and a good hardcover binding can make a given copy last longer under such use.  However, these are mundane (practical) issues that cannot be used to determine whether a given actual book object is a copy of the Bible or not: paperback printings are perfectly possible!
Similarly, Python is "a language" in the sense of defining a class of language implementations which must all be similar in some fundamental respects (syntax, most semantics except those parts of those where they're explicitly allowed to differ) but are fully allowed to differ in just about every "implementation" detail -- including how they deal with the source files they're given, whether they compile the sources to some lower level forms (and, if so, which form -- and whether they save such compiled forms, to disk or elsewhere), how they execute said forms, and so forth.
The classical implementation, CPython, is often called just "Python" for short -- but it's just one of several production-quality implementations, side by side with Microsoft's IronPython (which compiles to CLR codes, i.e., ".NET"), Jython (which compiles to JVM codes), PyPy (which is written in Python itself and can compile to a huge variety of "back-end" forms including "just-in-time" generated machine language).  They're all Python (=="implementations of the Python language") just like many superficially different book objects can all be Bibles (=="copies of The Bible").
If you're interested in CPython specifically: it compiles the source files into a Python-specific lower-level form (known as "bytecode"), does so automatically when needed (when there is no bytecode file corresponding to a source file, or the bytecode file is older than the source or compiled by a different Python version), usually saves the bytecode files to disk (to avoid recompiling them in the future).  OTOH IronPython will typically compile to CLR codes (saving them to disk or not, depending) and Jython to JVM codes (saving them to disk or not -- it will use the .class extension if it does save them).
These lower level forms are then executed by appropriate "virtual machines" also known as "interpreters" -- the CPython VM, the .Net runtime, the Java VM (aka JVM), as appropriate.
So, in this sense (what do typical implementations do), Python is an "interpreted language" if and only if C# and Java are: all of them have a typical implementation strategy of producing bytecode first, then executing it via a VM/interpreter.
More likely the focus is on how "heavy", slow, and high-ceremony the compilation process is.  CPython is designed to compile as fast as possible, as lightweight as possible, with as little ceremony as feasible -- the compiler does very little error checking and optimization, so it can run fast and in small amounts of memory, which in turns lets it be run automatically and transparently whenever needed, without the user even needing to be aware that there is a compilation going on, most of the time.  Java and C# typically accept more work during compilation (and therefore don't perform automatic compilation) in order to check errors more thoroughly and perform more optimizations.  It's a continuum of gray scales, not a black or white situation, and it would be utterly arbitrary to put a threshold at some given level and say that only above that level you call it "compilation"!-)


Input:
Use different Python version with virtualenv I have a Debian system currently running with python 2.5.4. I got virtualenv properly installed, everything is working fine. Is there a possibility that I can use a virtualenv with a different version of Python?
I compiled Python 2.6.2 and would like to use it with some virtualenv. Is it enough to overwrite the binary file? Or do I have to change something in respect to the libraries?

Output:
Just use the -p flag when creating your virtualenv instance to specify the Python executable you want to use, e.g.:
virtualenv -p /usr/bin/python2.6 <path/to/new/virtualenv/>


Input: How to import a module given the full path? How can I load a Python module given its full path? Note that the file can be anywhere in the filesystem, as it is a configuration option.

Output:
For Python 3.5+ use:
import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()

For Python 3.3 and 3.4 use:
from importlib.machinery import SourceFileLoader

foo = SourceFileLoader("module.name", "/path/to/file.py").load_module()
foo.MyClass()

(Although this has been deprecated in Python 3.4.)
Python 2 use:
import imp

foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()

There are equivalent convenience functions for compiled Python files and DLLs.
See also. http://bugs.python.org/issue21436.


Input:
Python @property versus getters and setters Here is a pure Python-specific design question:
class MyClass(object):
    ...
    def get_my_attr(self):
        ...

    def set_my_attr(self, value):
        ...

and
class MyClass(object):
    ...
    @property
    def my_attr(self):
        ...

    @my_attr.setter
    def my_attr(self, value):
        ...

Python lets us to do it either way. If you would design a Python program, which approach would you use and why?

Output:
Prefer properties. It's what they're there for.
The reason is that all attributes are public in Python. Starting names with an underscore or two is just a warning that the given attribute is an implementation detail that may not stay the same in future versions of the code. It doesn't prevent you from actually getting or setting that attribute. Therefore, standard attribute access is the normal, Pythonic way of, well, accessing attributes.
The advantage of properties is that they are syntactically identical to attribute access, so you can change from one to another without any changes to client code. You could even have one version of a class that uses properties (say, for code-by-contract or debugging) and one that doesn't for production, without changing the code that uses it. At the same time, you don't have to write getters and setters for everything just in case you might need to better control access later.


Input:
How do I download a file over HTTP using Python? I have a small utility that I use to download a MP3 from a website on a schedule and then builds/updates a podcast XML file which I've obviously added to iTunes.
The text processing that creates/updates the XML file is written in Python. I use wget inside a Windows .bat file to download the actual MP3 however. I would prefer to have the entire utility written in Python though.
I struggled though to find a way to actually down load the file in Python, thus why I resorted to wget.
So, how do I download the file using Python?

Output:
One more, using urlretrieve:
import urllib
urllib.urlretrieve ("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

(for Python 3+ use 'import urllib.request' and urllib.request.urlretrieve)
Yet another one, with a "progressbar"
import urllib2

url = "http://download.thinkbroadband.com/10MB.zip"

file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
meta = u.info()
file_size = int(meta.getheaders("Content-Length")[0])
print "Downloading: %s Bytes: %s" % (file_name, file_size)

file_size_dl = 0
block_sz = 8192
while True:
    buffer = u.read(block_sz)
    if not buffer:
        break

    file_size_dl += len(buffer)
    f.write(buffer)
    status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
    status = status + chr(8)*(len(status)+1)
    print status,

f.close()


Input:
Way to create multiline comments? I have recently started studying Python, but I couldn't find how to implement multi-line comments. Most languages have block comment symbols like
/*

*/

I tried this in Python, but it throws an error, so this probably is not the correct way. Does Python actually have a multiline comment feature?

Output:
You can use triple-quoted strings. When they're not a docstring (first thing in a class/function/module), they are ignored.
'''
This is a multiline
comment.
'''

Guido van Rossum (creator of Python) tweeted this as a "pro tip".
However, Python's style guide, PEP8, favors using consecutive single-line comments, and this is also what you'll find in many projects. Editors usually have a shortcut to do this easily.


Input:
python setup.py uninstall I have installed a python package with python setup.py install.
How do I uninstall it?

Output:
You need to remove all files manually, and also undo any other stuff that installation did manually.
If you don't know the list of all files, you can reinstall it with the --record option, and take a look at the list this produces.
To record list of installed files, you can use:
python setup.py install --record files.txt

Once you want to uninstall you can use xargs to do the removal:
cat files.txt | xargs rm -rf


Input:
How do I remove/delete a folder that is not empty with Python? I am getting an 'access is denied' error when I attempt to delete a folder that is not empty. I used the following command in my attempt: os.remove("/folder_name").
What is the most effective way of removing/deleting a folder/directory that is not empty?

Output:
import shutil

shutil.rmtree('/folder_name')

Standard Library Reference: shutil.rmtree.


Input:
Why shouldn't I use PyPy over CPython if PyPy is 6.3 times faster? I've been hearing a lot about the PyPy project. They claim it is 6.3 times faster than the CPython interpreter on their site.
Whenever we talk about dynamic languages like Python, speed is one of the top issues. To solve this, they say PyPy is 6.3 times faster.
The second issue is parallelism, the infamous Global Interpreter Lock (GIL). For this, PyPy says it can give GIL-less Python.
If PyPy can solve these great challenges, what are its weaknesses that are preventing wider adoption? That is to say, what's preventing someone like me, a typical Python developer, from switching to PyPy right now?

Output:
PyPy, as others have been quick to mention, has tenuous support for C extensions. It has support, but typically at slower-than-Python speeds and it's iffy at best. Hence a lot of modules simply require CPython. Cython and Numpy are awesome for numerics, and most people who actually need speed in Python are using those (+ Pandas, SciPy, etc.) heavily. Since they're either non-existent or tenuously supported and slow the people who need a fast Python often are better off with CPython both for speed and ease-of-use.
Secondly, Python 3 support is experimental at the moment. has just reached stable! As of 20th June 2014, PyPy3 2.3.1 - Fulcrum is out!
Thirdly, PyPy sometimes isn't actually faster for "scripts", which a lot of people use Python for. These are the short-running programs that do something simple and small. Because PyPy is a JIT compiler its main advantages come from long run times and simple types (such as numbers). Frankly, PyPy's pre-JIT speeds are pretty bad compared to CPython.
Fourthly, inertia. Moving to PyPy often requires retooling, which for some people and organizations is simply too much work.
Those are the main reasons that affect me, I'd say.


Input:
Correct way to write line to file in Python I'm used to doing print >>f, "hi there"
However, it seems that print >> is getting deprecated. What is the recommended way to do the line above?
Update:
Regarding all those answers with "\n"...is this universal or Unix-specific? IE, should I be doing "\r\n" on Windows?

Output:
You should use the new print() statement, available with Python 2.6+
from __future__ import print_function
print("hi there", file=f)

The alternative would be to use:
f = open('myfile','w')
f.write('hi there\n') # python will convert \n to os.linesep
f.close() # you can omit in most cases as the destructor will call it

Quoting from Python documentation regarding newlines:

On output, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.


Input:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128) I'm having problems dealing with unicode characters from text fetched from different web pages (on different sites). I am using BeautifulSoup.
The problem is that the error is not always reproducible; it sometimes works with some pages, and sometimes, it barfs by throwing a UnicodeEncodeError. I have tried just about everything I can think of, and yet I have not found anything that works consistently without throwing some kind of Unicode-related error.
One of the sections of code that is causing problems is shown below:
agent_telno = agent.find('div', 'agent_contact_number')
agent_telno = '' if agent_telno is None else agent_telno.contents[0]
p.agent_info = str(agent_contact + ' ' + agent_telno).strip()

Here is a stack trace produced on SOME strings when the snippet above is run:
Traceback (most recent call last):
  File "foobar.py", line 792, in <module>
    p.agent_info = str(agent_contact + ' ' + agent_telno).strip()
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

I suspect that this is because some pages (or more specifically, pages from some of the sites) may be encoded, whilst others may be unencoded. All the sites are based in the UK and provide data meant for UK consumption - so there are no issues relating to internalization or dealing with text written in anything other than English.
Does anyone have any ideas as to how to solve this so that I can CONSISTENTLY fix this problem?

Output:
You need to read the Python Unicode HOWTO. This error is the very first example.
Basically, stop using str to convert from unicode to encoded text / bytes.
Instead, properly use .encode() to encode the string:
p.agent_info = u' '.join((agent_contact, agent_telno)).encode('utf-8').strip()

or work entirely in unicode.


Input:
How to get file creation & modification date/times in Python? I have a script that needs to do some stuff based on file creation & modification dates but has to run on Linux & Windows.
What's the best cross-platform way to get file creation & modification date/times in Python?

Output:
You have a couple of choices. For one, you can use the os.path.getmtime and os.path.getctime functions:
import os.path, time
print "last modified: %s" % time.ctime(os.path.getmtime(file))
print "created: %s" % time.ctime(os.path.getctime(file))

Your other option is to use os.stat:
import os, time
(mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(file)
print "last modified: %s" % time.ctime(mtime)

Note: ctime() does not refer to creation time on *nix systems, but rather the last time the inode data changed. (thanks to kojiro for making that fact more clear in the comments by providing a link to an interesting blog post)


Input:
Installing specific package versions with pip I'm trying to install version 1.2.2 of the MySQL_python adaptor. The current version shown in PyPi is 1.2.3. Is there a way to install the older version? I found an article stating that this should do it:
pip install MySQL_python==1.2.2

When installed, however, it still shows MySQL_python-1.2.3-py2.6.egg-info in the site packages. Is this a problem specific to this package, or am I doing something wrong?

Output:
First, I see two issues with what you're trying to do. Since you already have an installed version, you should either uninstall the current existing driver or use pip install -I MySQL_python==1.2.2
However, you'll soon find out that this doesn't work. If you look at pip's installation log, or if you do a pip install -Iv MySQL_python==1.2.2 you'll find that the PyPI URL link does not work for MySQL_python v1.2.2. You can verify this here: http://pypi.python.org/pypi/MySQL-python/1.2.2
The download link 404s and the fallback URL links are re-directing infinitely due to sourceforge.net's recent upgrade and PyPI's stale URL.
So to properly install the driver, you can follow these steps:
pip uninstall MySQL_python
pip install -Iv http://sourceforge.net/projects/mysql-python/files/mysql-python/1.2.2/MySQL-python-1.2.2.tar.gz/download


Input:
What are "named tuples" in Python? Reading the changes in Python 3.1, I found something... unexpected:

The sys.version_info tuple is now a named tuple:

I never heard about named tuples before, and I thought elements could either be indexed by numbers (like in tuples and lists) or by keys (like in dicts). I never expected they could be indexed both ways.
Thus, my questions are:

What are named tuples?
How to use them?
Why/when should I use named tuples instead of normal tuples?
Why/when should I use normal tuples instead of named tuples?
Is there any kind of "named list" (a mutable version of the named tuple)?

Output:
Named tuples are basically easy to create, lightweight object types.  Named tuple instances can be referenced using object like variable deferencing or the standard tuple syntax.  They can be used similarly to struct or other common record types, except that they are immutable.  They were added in Python 2.6 and Python 3.0, although there is a recipe for implementation in Python 2.4.
For example, it is common to represent a point, for example as a tuple (x, y).  This leads to code like the following:
pt1 = (1.0, 5.0)
pt2 = (2.5, 1.5)

from math import sqrt
line_length = sqrt((pt1[0]-pt2[0])**2 + (pt1[1]-pt2[1])**2)

Using a named tuple it becomes more readable:
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

from math import sqrt
line_length = sqrt((pt1.x-pt2.x)**2 + (pt1.y-pt2.y)**2)

However, named tuples are still backwards compatible with normal tuples, so the following will still work:
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

from math import sqrt
# use index referencing
line_length = sqrt((pt1[0]-pt2[0])**2 + (pt1[1]-pt2[1])**2)
 # use tuple unpacking
x1, y1 = pt1

Thus, you should use named tuples instead of tuples anywhere you think object notation will make your code more pythonic and more easily readable.  I personally have started using them to represent very simple value types, particularly when passing them as parameters to functions.  It makes the functions more readable, without seeing the context of the tuple packing.
Furthermore, you can also replace ordinary immutable classes that have no functions, only fields with them.  You can even use your named tuple types as base classes:
class Point(namedtuple('Point', 'x y')):
    [...]

However, as with tuples, attributes in named tuples are immutable:
>>> Point = namedtuple('Point', 'x y')
>>> pt1 = Point(1.0, 5.0)
>>> pt1.x = 2.0
AttributeError: can't set attribute

If you want to be able change the values, you need another type.  There is a handy recipe for mutable recordtypes which allow you to set new values to attributes.
>>> from rcdtype import *
>>> Point = recordtype('Point', 'x y')
>>> pt1 = Point(1.0, 5.0)
>>> pt1 = Point(1.0, 5.0)
>>> pt1.x = 2.0
>>> print(pt1[0])
    2.0

I am not aware of any form of "named list" that lets you add new fields, however.  You may just want to use a dictionary in this situation. Named tuples can be converted to dictionaries using pt1._asdict() which returns {'x': 1.0, 'y': 5.0} and can be operated upon with all the usual dictionary functions.
As already noted, you should check the documentation for more information from which these examples were constructed.


Input:
Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell I have taken Problem #12 from Project Euler as a programming exercise and to compare my (surely not optimal) implementations in C, Python, Erlang and Haskell. In order to get some higher execution times, I search for the first triangle number with more than 1000 divisors instead of 500 as stated in the original problem.
The result is the following:
C:
lorenzo@enzo:~/erlang$ gcc -lm -o euler12.bin euler12.c
lorenzo@enzo:~/erlang$ time ./euler12.bin
842161320

real    0m11.074s
user    0m11.070s
sys 0m0.000s

python:
lorenzo@enzo:~/erlang$ time ./euler12.py
842161320

real    1m16.632s
user    1m16.370s
sys 0m0.250s

python with pypy:
lorenzo@enzo:~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin$ time ./pypy /home/lorenzo/erlang/euler12.py
842161320

real    0m13.082s
user    0m13.050s
sys 0m0.020s

erlang:
lorenzo@enzo:~/erlang$ erlc euler12.erl
lorenzo@enzo:~/erlang$ time erl -s euler12 solve
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> 842161320

real    0m48.259s
user    0m48.070s
sys 0m0.020s

haskell:
lorenzo@enzo:~/erlang$ ghc euler12.hs -o euler12.hsx
[1 of 1] Compiling Main             ( euler12.hs, euler12.o )
Linking euler12.hsx ...
lorenzo@enzo:~/erlang$ time ./euler12.hsx
842161320

real    2m37.326s
user    2m37.240s
sys 0m0.080s

Summary:

C: 100%
python: 692% (118% with pypy)
erlang: 436% (135% thanks to RichardC)
haskell: 1421%

I suppose that C has a big advantage as it uses long for the calculations and not arbitrary length integers as the other three. Also it doesn't need to load a runtime first (Do the others?).
Question 1:
Do Erlang, Python and Haskell lose speed due to using arbitrary length integers or don't they as long as the values are less than MAXINT?
Question 2:
Why is Haskell so slow? Is there a compiler flag that turns off the brakes or is it my implementation? (The latter is quite probable as Haskell is a book with seven seals to me.)
Question 3:
Can you offer me some hints how to optimize these implementations without changing the way I determine the factors? Optimization in any way: nicer, faster, more "native" to the language.
EDIT:
Question 4:
Do my functional implementations permit LCO (last call optimization, a.k.a tail recursion elimination) and hence avoid adding unnecessary frames onto the call stack?
I really tried to implement the same algorithm as similar as possible in the four languages, although I have to admit that my Haskell and Erlang knowledge is very limited.

Source codes used:
#include <stdio.h>
#include <math.h>

int factorCount (long n)
{
    double square = sqrt (n);
    int isquare = (int) square;
    int count = isquare == square ? -1 : 0;
    long candidate;
    for (candidate = 1; candidate <= isquare; candidate ++)
        if (0 == n % candidate) count += 2;
    return count;
}

int main ()
{
    long triangle = 1;
    int index = 1;
    while (factorCount (triangle) < 1001)
    {
        index ++;
        triangle += index;
    }
    printf ("%ld\n", triangle);
}

#! /usr/bin/env python3.2

import math

def factorCount (n):
    square = math.sqrt (n)
    isquare = int (square)
    count = -1 if isquare == square else 0
    for candidate in range (1, isquare + 1):
        if not n % candidate: count += 2
    return count

triangle = 1
index = 1
while factorCount (triangle) < 1001:
    index += 1
    triangle += index

print (triangle)

-module (euler12).
-compile (export_all).

factorCount (Number) -> factorCount (Number, math:sqrt (Number), 1, 0).

factorCount (_, Sqrt, Candidate, Count) when Candidate > Sqrt -> Count;

factorCount (_, Sqrt, Candidate, Count) when Candidate == Sqrt -> Count + 1;

factorCount (Number, Sqrt, Candidate, Count) ->
    case Number rem Candidate of
        0 -> factorCount (Number, Sqrt, Candidate + 1, Count + 2);
        _ -> factorCount (Number, Sqrt, Candidate + 1, Count)
    end.

nextTriangle (Index, Triangle) ->
    Count = factorCount (Triangle),
    if
        Count > 1000 -> Triangle;
        true -> nextTriangle (Index + 1, Triangle + Index + 1)
    end.

solve () ->
    io:format ("~p~n", [nextTriangle (1, 1) ] ),
    halt (0).

factorCount number = factorCount' number isquare 1 0 - (fromEnum $ square == fromIntegral isquare)
    where square = sqrt $ fromIntegral number
          isquare = floor square

factorCount' number sqrt candidate count
    | fromIntegral candidate > sqrt = count
    | number `mod` candidate == 0 = factorCount' number sqrt (candidate + 1) (count + 2)
    | otherwise = factorCount' number sqrt (candidate + 1) count

nextTriangle index triangle
    | factorCount triangle > 1000 = triangle
    | otherwise = nextTriangle (index + 1) (triangle + index + 1)

main = print $ nextTriangle 1 1

Output:
Using GHC 7.0.3, gcc 4.4.6, Linux 2.6.29 on an x86_64 Core2 Duo (2.5GHz) machine, compiling using ghc -O2 -fllvm -fforce-recomp for Haskell and gcc -O3 -lm for C.

Your C routine runs in 8.4 seconds (faster than your run probably because of -O3)
The Haskell solution runs in 36 seconds (due to the -O2 flag)
Your factorCount' code isn't explicitly typed and defaulting to Integer (thanks to Daniel for correcting my misdiagnosis here!).  Giving an explicit type signature (which is standard practice anyway) using Int and the time changes to 11.1 seconds
in factorCount' you have needlessly called fromIntegral. A fix results in no change though (the compiler is smart, lucky for you).
You used mod where rem is faster and sufficient. This changes the time to 8.5 seconds.
factorCount' is constantly applying two extra arguments that never change (number, sqrt).  A worker/wrapper transformation gives us:

 $ time ./so
 842161320

 real    0m7.954s
 user    0m7.944s
 sys     0m0.004s

That's right, 7.95 seconds.  Consistently half a second faster than the C solution.  Without the -fllvm flag I'm still getting 8.182 seconds, so the NCG backend is doing well in this case too.
Conclusion: Haskell is awesome.
Resulting Code
factorCount number = factorCount' number isquare 1 0 - (fromEnum $ square == fromIntegral isquare)
    where square = sqrt $ fromIntegral number
          isquare = floor square

factorCount' :: Int -> Int -> Int -> Int -> Int
factorCount' number sqrt candidate0 count0 = go candidate0 count0
  where
  go candidate count
    | candidate > sqrt = count
    | number `rem` candidate == 0 = go (candidate + 1) (count + 2)
    | otherwise = go (candidate + 1) count

nextTriangle index triangle
    | factorCount triangle > 1000 = triangle
    | otherwise = nextTriangle (index + 1) (triangle + index + 1)

main = print $ nextTriangle 1 1

EDIT: So now that we've explored that, lets address the questions

Question 1: Do erlang, python and haskell lose speed due to using
  arbitrary length integers or don't they as long as the values are less
  than MAXINT?

In Haskell, using Integer is slower than Int but how much slower depends on the computations performed.  Luckily (for 64 bit machines) Int is sufficient.  For portability sake you should probably rewrite my code to use Int64 or Word64 (C isn't the only language with a long).

Question 2: Why is haskell so slow? Is there a compiler flag that
  turns off the brakes or is it my implementation? (The latter is quite
  probable as haskell is a book with seven seals to me.)
Question 3: Can you offer me some hints how to optimize these
  implementations without changing the way I determine the factors?
  Optimization in any way: nicer, faster, more "native" to the language.

That was what I answered above.   The answer was

0) Use optimization via -O2
1) Use fast  (notably: unbox-able) types when possible
2) rem not mod (a frequently forgotten optimization) and
3) worker/wrapper transformation (perhaps the most common optimization).

Question 4: Do my functional implementations permit LCO and hence
  avoid adding unnecessary frames onto the call stack?

Yes, that wasn't the issue.  Good work and glad you considered this.


Input:
'has_key()' or 'in'? I wonder what is better to do:
d = {'a': 1, 'b': 2}
'a' in d
True

or:
d = {'a': 1, 'b': 2}
d.has_key('a')
True

Output:
in is definitely more pythonic.
In fact has_key() was removed in Python 3.x.


Input:
Why is [] faster than list()? I recently compared the processing speeds of [] and list() and was surprised to discover that [] runs more than three times faster than list(). I ran the same test with {} and dict() and the results were practically identical: [] and {} both took around 0.128sec / million cycles, while list() and dict() took roughly 0.428sec / million cycles each.
Why is this? Do [] and {} (probably () and '', too) immediately pass back a copies of some empty stock literal while their explicitly-named counterparts (list(), dict(), tuple(), str()) fully go about creating an object, whether or not they actually have elements?
I have no idea how these two methods differ but I'd love to find out.
I couldn't find an answer in the docs or on SO, and searching for empty brackets turned out to be more complicated than I'd expected.
I got my timing results by calling timeit.timeit("[]") and timeit.timeit("list()"), and timeit.timeit("{}") and timeit.timeit("dict()"), to compare lists and dictionaries, respectively. I'm running Python 2.7.9.
I recently discovered "Why is if True slower than if 1?" that compares the performance of if True to if 1 and seems to touch on a similar literal-versus-global scenario; perhaps it's worth considering as well.

Output:
Because [] and {} are literal syntax. Python can create bytecode just to create the list or dictionary objects:
>>> import dis
>>> dis.dis(compile('[]', '', 'eval'))
  1           0 BUILD_LIST               0
              3 RETURN_VALUE
>>> dis.dis(compile('{}', '', 'eval'))
  1           0 BUILD_MAP                0
              3 RETURN_VALUE

list() and dict() are separate objects. Their names need to be resolved, the stack has to be involved to push the arguments, the frame has to be stored to retrieve later, and a call has to be made. That all takes more time.
For the empty case, that means you have at the very least a LOAD_NAME (which has to search through the global namespace as well as the __builtin__ module) followed by a CALL_FUNCTION, which has to preserve the current frame:
>>> dis.dis(compile('list()', '', 'eval'))
  1           0 LOAD_NAME                0 (list)
              3 CALL_FUNCTION            0
              6 RETURN_VALUE
>>> dis.dis(compile('dict()', '', 'eval'))
  1           0 LOAD_NAME                0 (dict)
              3 CALL_FUNCTION            0
              6 RETURN_VALUE

You can time the name lookup separately with timeit:
>>> import timeit
>>> timeit.timeit('list', number=10**7)
0.30749011039733887
>>> timeit.timeit('dict', number=10**7)
0.4215109348297119

The time discrepancy there is probably a dictionary hash collision. Subtract those times from the times for calling those objects, and compare the result against the times for using literals:
>>> timeit.timeit('[]', number=10**7)
0.30478692054748535
>>> timeit.timeit('{}', number=10**7)
0.31482696533203125
>>> timeit.timeit('list()', number=10**7)
0.9991960525512695
>>> timeit.timeit('dict()', number=10**7)
1.0200958251953125

So having to call the object takes an additional 1.00 - 0.31 - 0.30 == 0.39 seconds per 10 million calls.
You can avoid the global lookup cost by aliasing the global names as locals (using a timeit setup, everything you bind to a name is a local):
>>> timeit.timeit('_list', '_list = list', number=10**7)
0.1866450309753418
>>> timeit.timeit('_dict', '_dict = dict', number=10**7)
0.19016098976135254
>>> timeit.timeit('_list()', '_list = list', number=10**7)
0.841480016708374
>>> timeit.timeit('_dict()', '_dict = dict', number=10**7)
0.7233691215515137

but you never can overcome that CALL_FUNCTION cost.


Input:
How to get line count cheaply in Python? I need to get a line count of a large file (hundreds of thousands of lines) in python. What is the most efficient way both memory- and time-wise?
At the moment I do:
def file_len(fname):
    with open(fname) as f:
        for i, l in enumerate(f):
            pass
    return i + 1

is it possible to do any better?

Output:
One line, probably pretty fast:
num_lines = sum(1 for line in open('myfile.txt'))


Input: null object in Python? How do I refer to the null object in Python?

Output:
In Python, the 'null' object is the singleton None.
The best way to check things for "Noneness" is to use the identity operator, is:
if foo is None:
    ...


Input:
Terminating a Python script I am aware of the die() command in PHP which stops a script early.
How can I do this in Python?

Output:
import sys
sys.exit()

details from the sys module documentation:

sys.exit([arg])

Exit from Python. This is implemented by raising the
    SystemExit exception, so cleanup actions specified by finally clauses
    of try statements are honored, and it is possible to intercept the
    exit attempt at an outer level.
The optional argument arg can be an integer giving the exit status
    (defaulting to zero), or another type of object. If it is an integer,
    zero is considered âsuccessful terminationâ and any nonzero value is
    considered âabnormal terminationâ by shells and the like. Most systems
    require it to be in the range 0-127, and produce undefined results
    otherwise. Some systems have a convention for assigning specific
    meanings to specific exit codes, but these are generally
    underdeveloped; Unix programs generally use 2 for command line syntax
    errors and 1 for all other kind of errors. If another type of object
    is passed, None is equivalent to passing zero, and any other object is
    printed to stderr and results in an exit code of 1. In particular,
    sys.exit("some error message") is a quick way to exit a program when
    an error occurs.
Since exit() ultimately âonlyâ raises an exception, it will only exit
    the process when called from the main thread, and the exception is not
    intercepted.

Note that this is the 'nice' way to exit.  @glyphtwistedmatrix below points out that if you want a 'hard exit', you can use os._exit(errorcode), though it's likely os-specific to some extent (it might not take an errorcode under windows, for example), and it definitely is less friendly since it doesn't let the interpreter do any cleanup before the process dies.


Input:
Differences between distribute, distutils, setuptools and distutils2? The Situation
Iâm trying to port an open-source library to Python 3.  (SymPy, if anyone is wondering.)
So, I need to run 2to3 automatically when building for Python 3. To do that, I need to use distribute. Therefore, I need to port the current system, which (according to the doctest) is distutils.

The Problem
Unfortunately, Iâm not sure whatâs the difference between these modulesâdistutils, distribute, setuptools. The documentation is sketchy as best, as they all seem to be a fork of one another, intended to be compatible in most circumstances (but actually, not all)â¦and so on, and so forth.

The Question
Could someone explain the differences? What am I supposed to use?  What is the most modern solution? (As an aside, Iâd also appreciate some guide on porting to Distribute, but thatâs a tad beyond the scope of the questionâ¦)

Output:
As of September 2014, all of the other answers to this question are a year out-of-date. When you come across advice on Python packaging issues, remember to look at the date of publication, and don't trust out-of-date information.
The Python Packaging User Guide hosted on Readthedocs is worth a read. Every page has a "last reviewed" date displayed, so you can check the recency of the manual, and it's quite comprehensive. The official Python 3.4 docs now link to this resource, just to add credence to it.
Summary of tools:
Here's a summary of the Python packaging landscape in September 2014:

Distutils is still the standard tool for packaging in Python. It is included in the standard library (Python 2 and Python 3.0 to 3.4). It is useful for simple Python distributions, but lacks features. It introduces the distutils Python package that can be imported in your setup.py script.
Setuptools was developed to overcome Distutils' limitations, and is not included in the standard library. It introduced a command-line utility called easy_install. It also introduced the setuptools Python package that can be imported in your setup.py script, and the pkg_resources Python package that can be imported in your code to locate data files installed with a distribution. One of its gotchas is that it monkey-patches the distutils Python package. It should work well with pip. It sees regular releases.
Distribute was a fork of Setuptools. It shared the same namespace, so if you had Distribute installed, import setuptools would actually import the package distributed with Distribute. Distribute was merged back into Setuptools 0.7, so you don't need to use Distribute any more. In fact, the version on Pypi is just a compatibility layer that installs Setuptools.
Distutils2 was an attempt to take the best of Distutils, Setuptools and Distribute and become the standard tool included in Python's standard library. The idea was that Distutils2 would be distributed for old Python versions, and that Distutils2 would be renamed to packaging for Python 3.3, which would include it in its standard library. These plans did not go as intended, however, and currently, Distutils2 is an abandoned project. The latest release was in March 2012, and its Pypi home page has finally been updated to reflect its death.
Distlib is a tool that aims to implement a subset of the previous tools' functionality, but only functionality that is very well-defined in accepted PEPs. It should hopefully be included eventually in the Python standard library. It is still being developed and is not recommended for end-users yet.
Bento is a packaging solution designed to replace Distutils, Setuptools, Distribute and Distutils2, written from the ground up. Its primary developer is also a core developer of numpy/scipy, so he's familiar with non-simple use-cases for packaging systems. Its first commit was in October 2009, and the latest commit as of writing was in August 2014, although the authors are not updating its Pypi page correspondingly. It's in active development but it is not mature yet, and it is not as widely known as Setuptools yet.

Recommendation:
So in conclusion, out of all these options, I would recommend Setuptools, unless your requirements are very basic and you only need Distutils. Setuptools works very well with Virtualenv and Pip, tools that I highly recommend.
As a side-note, I recommend using Virtualenv 1.10 or higher, as it is the first release that recognises the Setuptools/Distribute merger, for both Python 2 and 3.


Input:
How to find if directory exists in Python In the os module in Python, is there a way to find if a directory exists, something like:
>>> os.direxists(os.path.join(os.getcwd()), 'new_folder')) # in pseudocode
True/False

Output:
You're looking for os.path.isdir, or os.path.exists if you don't care whether it's a file or a directory.
Example:
import os
print(os.path.isdir("/home/el"))
print(os.path.exists("/home/el/myfile.txt"))


Input:
pip install mysql-python fails with EnvironmentError: mysql_config not found This is the error I get
(mysite)zjm1126@zjm1126-G41MT-S2:~/zjm_test/mysite$ pip install mysql-python
Downloading/unpacking mysql-python
  Downloading MySQL-python-1.2.3.tar.gz (70Kb): 70Kb downloaded
  Running setup.py egg_info for package mysql-python
    sh: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/zjm1126/zjm_test/mysite/build/mysql-python/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: not found

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/home/zjm1126/zjm_test/mysite/build/mysql-python/setup.py", line 15, in <module>

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/zjm1126/.pip/pip.log
(mysite)zjm1126@zjm1126-G41MT-S2:~/zjm_test/mysite$ pip install mysql-python
Downloading/unpacking mysql-python
  Running setup.py egg_info for package mysql-python
    sh: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/zjm1126/zjm_test/mysite/build/mysql-python/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: not found

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/home/zjm1126/zjm_test/mysite/build/mysql-python/setup.py", line 15, in <module>

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/zjm1126/.pip/pip.log

What can I do to resolve this?

Output:
It seems mysql_config is missing on your system or the installer could not find it.
Be sure mysql_config is really installed.
For example on Debian/Ubuntu you must install the package:
sudo apt-get install libmysqlclient-dev

Maybe the mysql_config is not in your path, it will be the case when you compile by yourself
the mysql suite.


Input:
How can I force division to be floating point in Python? Division keeps rounding down to 0 I have two integer values a and b, but I need their ratio in floating point.  I know that a<b and I want to calculate a/b, so if I use integer division I'll always get 0 with a remainder of a.
How can I force c to be a floating point number in Python in the following?
c = a / b

Output:
You can cast to float by doing c = a / float(b). If the numerator or denominator is a float, then the result will be also.

A caveat: as commenters have pointed out, this won't work if b might be something other than an integer or floating-point number (or a string representing one). If you might be dealing with other types (such as complex numbers) you'll need to either check for those or use a different method.


Input:
Renaming columns in pandas I have a data table using pandas and column labels that I need to edit to replace the original column labels.
I'd like to change the column names in a data table A where the original column names are:
['$a', '$b', '$c', '$d', '$e']

to
['a', 'b', 'c', 'd', 'e'].

I have the edited column names stored it in a list, but I don't know how to replace the column names.

Output:
df = df.rename(columns={'$a': 'a', '$b': 'b'})
# OR
df.rename(columns={'$a': 'a', '$b': 'b'}, inplace=True)

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rename.html


Input:
Remove items from a list while iterating in Python I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria.
for tup in somelist:
    if determine(tup):
         code_to_remove_tup

What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.

Output:
The answers suggesting list comprehensions are ALMOST correct -- except that they build a completely new list and then give it the same name the old list as, they do NOT modify the old list in place. That's different from what you'd be doing by selective removal, as in @Lennart's suggestion -- it's faster, but if your list is accessed via multiple references the fact that you're just reseating one of the references and NOT altering the list object itself can lead to subtle, disastrous bugs.
Fortunately, it's extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration -- just code:
somelist[:] = [tup for tup in somelist if determine(tup)]

Note the subtle difference with other answers: this one is NOT assigning to a barename - it's assigning to a list slice that just happens to be the entire list, thereby replacing the list contents within the same Python list object, rather than just reseating one reference (from previous list object to new list object) like the other answers.


Input: Is there a function in Python to print all the current properties and values of an object? So what I'm looking for here is something like PHP's print_r function. This is so I can debug my scripts by seeing what's the state of the object in question.

Output:
You want vars() mixed with pprint:
from pprint import pprint
pprint (vars(your_object))


Input: mkdir -p functionality in Python Is there a way to get functionality similar to mkdir -p on the shell from within Python.  I am looking for a solution other than a system call.  I am sure the code is less than 20 lines, and I am wondering if someone has already written it?

Output:
mkdir -p functionality as follows:
import errno
import os

def mkdir_p(path):
    try:
        os.makedirs(path)
    except OSError as exc:  # Python >2.5
        if exc.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else:
            raise

Update
For Python â¥ 3.2, os.makedirs has an optional third argument exist_ok that, when true, enables the mkdir -p functionality âunless mode is provided and the existing directory has different permissions than the intended ones; in that case, OSError is raised as previously.


Input:
Map two lists into a dictionary in Python Imagine that you have:
keys = ('name', 'age', 'food')
values = ('Monty', 42, 'spam')

What is the simplest way to produce the following dictionary ?
dict = {'name' : 'Monty', 'age' : 42, 'food' : 'spam'}

This code works, but I'm not really proud of it :
dict = {}
junk = map(lambda k, v: dict.update({k: v}), keys, values)

Output:
Like this:
>>> keys = ['a', 'b', 'c']
>>> values = [1, 2, 3]
>>> dictionary = dict(zip(keys, values))
>>> print dictionary
{'a': 1, 'b': 2, 'c': 3}

Voila :-)  The pairwise dict constructor and zip function are awesomely useful: https://docs.python.org/2/library/functions.html#func-dict


Input:
How do you return multiple values in Python? The canonical way to return multiple values in languages that support it is often tupling.
Option: Using a tuple
Consider this trivial example:
def f(x):
  y0 = x + 1
  y1 = x * 3
  y2 = y0 ** y3
  return (y0,y1,y2)

However, this quickly gets problematic as the number of values returned increases. What if you want to return four or five values? Sure, you could keep tupling them, but it gets easy to forget which value is where. It's also rather ugly to unpack them wherever you want to receive them.
Option: Using a dictionary
The next logical step seems to be to introduce some sort of 'record notation'. In python, the obvious way to do this is by means of a dict.
Consider the following:
def g(x):
  y0 = x + 1
  y1 = x * 3
  y2 = y0 ** y3
  return {'y0':y0, 'y1':y1 ,'y2':y2 }

(edit- Just to be clear, y0, y1 and y2 are just meant as abstract identifiers. As pointed out, in practice you'd use meaningful identifiers)
Now, we have a mechanism whereby we can project out a particular member of the returned object. For example,
result['y0']

Option: Using a class
However, there is another option. We could instead return a specialized structure. I've framed this in the context of Python, but I'm sure it applies to other languages as well. Indeed, if you were working in C this might very well be your only option. Here goes:
class ReturnValue(object):
  def __init__(self, y0, y1, y2):
     self.y0 = y0
     self.y1 = y1
     self.y2 = y2

def g(x):
  y0 = x + 1
  y1 = x * 3
  y2 = y0 ** y3
  return ReturnValue(y0, y1, y2)

In python the previous two are perhaps very similar in terms of plumbing- After all { y0, y1, y2 } just end up being entries in the internal __dict__ of the ReturnValue.
There is one additional feature provided by Python though for tiny objects, the __slots__ attribute. The class could be expressed as:
class ReturnValue(object):
  __slots__ = ["y0", "y1", "y2"]
  def __init__(self, y0, y1, y2):
     self.y0 = y0
     self.y1 = y1
     self.y2 = y2

From the Python Reference Manual:

The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance.

Option: Using a list
Another suggestion which I'd overlooked comes from Bill the Lizard:
def h(x):
  result = [x + 1]
  result.append(x * 3)
  result.append(y0 ** y3)
  return result

This is my least favorite method though. I suppose I'm tainted by exposure to Haskell, but the idea of mixed-type lists has always felt uncomfortable to me. In this particular example the list is -not- mixed type, but it conceivably could be. A list used in this way really doesn't gain anything with respect to the tuple as far as I can tell. The only real difference between lists and tuples in Python is that lists are mutable, wheras tuples are not. I personally tend to carry over the conventions from functional programming: use lists for any number of elements of the same type, and tuples for a fixed number of elements of predetermined types.
Question
After the lengthy preamble, comes the inevitable question. Which method (do you think) is best?
I've typically found myself going the dictionary route because it involves less set-up work. From a types perspective however, you might be better off going the class route, since that may help you avoid confusing what a dictionary represents. On the other hand, there are some in the Python community that feel implied interfaces should be preferred to explicit interfaces, at which point the type of the object really isn't relevant, since you're basically relying on the convention that the same attribute will always have the same meaning.
So, how do -you- return multiple values in Python?

Output:
Named tuples were added in 2.6 for this purpose.  Also see os.stat for a similar builtin example.
>>> import collections
>>> point = collections.namedtuple('Point', ['x', 'y'])
>>> p = point(1, y=2)
>>> p.x, p.y
1 2
>>> p[0], p[1]
1 2


Input:
Count occurrence of a character in a string What's the simplest way to count the number of occurrences of a character in a string?
e.g. count the number of times 'a' appears in 'Mary had a little lamb'

Output:
str.count(sub[, start[, end]])
Return the number of non-overlapping occurrences of substring sub in the range [start, end]. Optional arguments start and end are interpreted as in slice notation.

>>> sentence = 'Mary had a little lamb'
>>> sentence.count('a')
4

Or:
from collections import Counter
sentence = 'Mary had a little lamb'
print(Counter(list(sentence))['a'])


Input:
Delete an element from a dictionary Is there a way to delete an element from a dictionary in Python?
I know I can just call .pop on the dictionary, but that returns the element that was removed. What I'm looking for is something returns the dictionary minus the element in question.
At present I have a helper function that accepts the dictionary in question as parameter, and then returns a dictionary with the element removed, Is there a more elegant solution?

Output:
The del statement removes an element:
del d[key]

However, this mutates the existing dictionary so the contents of the dictionary changes for anybody else who has a reference to the same instance. To return a new dictionary, make a copy of the dictionary:
def removekey(d, key):
    r = dict(d)
    del r[key]
    return r

The dict() constructor makes a shallow copy. To make a deep copy, see the copy module.


Input:
Generate random integers between 0 and 9 How can I generate random integer numbers between 0 and 9 (inclusive) in Python?
i.e. 0 1 2 3 4 5 6 7 8 9

Output:
Try:
from random import randint
print(randint(0,9))

More info: https://docs.python.org/3.1/library/random.html#random.randint


Input:
How to create a constant in Python Is there a way to declare a constant in Python. In Java we can create constant in this manner:
public static final String CONST_NAME = "Name";

What is the equivalent of the above Java constant declaration in Python?

Output:
No there is not. You cannot declare a variable or value as constant in Python. Just don't change it.
If you are in a class, the equivalent would be:
class Foo(object):
    CONST_NAME = "Name"

if not, it is just
CONST_NAME = "Name"

But you might want to have a look at the code snippet Constants in Python by Alex Martelli.


Input:
Why does "not(True) in [False, True]" return False? If I do this:
>>> False in [False, True]
True

That returns True. Simply because False is in the list.
But if I do:
>>> not(True) in [False, True]
False

That returns False. Whereas not(True) is equal to False:
>>> not(True)
False

Why?

Output:
Operator precedence 2.x, 3.x. The precedence of not is lower than that of in. So it is equivalent to:
>>> not (True in [False, True])
False

This is what you want:
>>> (not True) in [False, True]
True

As @Ben points out: It's recommended to never write not(True), prefer not True. The former makes it look like a function call, while not is an operator, not a function.


Input:
"Large data" work flows using pandas I have tried to puzzle out an answer to this question for many months while learning pandas.  I use SAS for my day-to-day work and it is great for it's out-of-core support.  However, SAS is horrible as a piece of software for numerous other reasons.
One day I hope to replace my use of SAS with python and pandas, but I currently lack an out-of-core workflow for large datasets.  I'm not talking about "big data" that requires a distributed network, but rather files too large to fit in memory but small enough to fit on a hard-drive.
My first thought is to use HDFStore to hold large datasets on disk and pull only the pieces I need into dataframes for analysis.  Others have mentioned MongoDB as an easier to use alternative.  My question is this:
What are some best-practice workflows for accomplishing the following:

Loading flat files into a permanent, on-disk database structure
Querying that database to retrieve data to feed into a pandas data structure
Updating the database after manipulating pieces in pandas

Real-world examples would be much appreciated, especially from anyone who uses pandas on "large data".
Edit -- an example of how I would like this to work:

Iteratively import a large flat-file and store it in a permanent, on-disk database structure.  These files are typically too large to fit in memory.
In order to use Pandas, I would like to read subsets of this data (usually just a few columns at a time) that can fit in memory.
I would create new columns by performing various operations on the selected columns.
I would then have to append these new columns into the database structure.

I am trying to find a best-practice way of performing these steps. Reading links about pandas and pytables it seems that appending a new column could be a problem.
Edit -- Responding to Jeff's questions specifically:

I am building consumer credit risk models. The kinds of data include phone, SSN and address characteristics; property values; derogatory information like criminal records, bankruptcies, etc... The datasets I use every day have nearly 1,000 to 2,000 fields on average of mixed data types: continuous, nominal and ordinal variables of both numeric and character data.  I rarely append rows, but I do perform many operations that create new columns.
Typical operations involve combining several columns using conditional logic into a new, compound column. For example, if var1 > 2 then newvar = 'A' elif var2 = 4 then newvar = 'B'.  The result of these operations is a new column for every record in my dataset.
Finally, I would like to append these new columns into the on-disk data structure.  I would repeat step 2, exploring the data with crosstabs and descriptive statistics trying to find interesting, intuitive relationships to model.
A typical project file is usually about 1GB.  Files are organized into such a manner where a row consists of a record of consumer data.  Each row has the same number of columns for every record.  This will always be the case.
It's pretty rare that I would subset by rows when creating a new column.  However, it's pretty common for me to subset on rows when creating reports or generating descriptive statistics.  For example, I might want to create a simple frequency for a specific line of business, say Retail credit cards.  To do this, I would select only those records where the line of business = retail in addition to whichever columns I want to report on.  When creating new columns, however, I would pull all rows of data and only the columns I need for the operations.
The modeling process requires that I analyze every column, look for interesting relationships with some outcome variable, and create new compound columns that describe those relationships.  The columns that I explore are usually done in small sets.  For example, I will focus on a set of say 20 columns just dealing with property values and observe how they relate to defaulting on a loan.  Once those are explored and new columns are created, I then move on to another group of columns, say college education, and repeat the process.  What I'm doing is creating candidate variables that explain the relationship between my data and some outcome.  At the very end of this process, I apply some learning techniques that create an equation out of those compound columns.

It is rare that I would ever add rows to the dataset.  I will nearly always be creating new columns (variables or features in statistics/machine learning parlance).

Output:
I routinely use tens of gigabytes of data in just this fashion
e.g. I have tables on disk that I read via queries, create data and append back.
It's worth reading the docs and late in this thread for several suggestions for how to store your data.
Details which will affect how you store your data, like:
Give as much detail as you can; and I can help you develop a structure.

Size of data, # of rows, columns, types of columns; are you appending
rows, or just columns?
What will typical operations look like. E.g. do a query on columns to select a bunch of rows and specific columns, then do an operation (in-memory), create new columns, save these.
(Giving a toy example could enable us to offer more specific recommendations.)
After that processing, then what do you do? Is step 2 ad hoc, or repeatable?
Input flat files: how many, rough total size in Gb. How are these organized e.g. by records? Does each one contains different fields, or do they have some records per file with all of the fields in each file?
Do you ever select subsets of rows (records) based on criteria (e.g. select the rows with field A > 5)? and then do something, or do you just select fields A, B, C with all of the records (and then do something)?
Do you 'work on' all of your columns (in groups), or are there a good proportion that you may only use for reports (e.g. you want to keep the data around, but don't need to pull in that column explicity until final results time)?

Solution
Ensure you have pandas at least 0.10.1 installed.
Read iterating files chunk-by-chunk and multiple table queries.
Since pytables is optimized to operate on row-wise (which is what you query on), we will create a table for each group of fields. This way it's easy to select a small group of fields (which will work with a big table, but it's more efficient to do it this way... I think I may be able to fix this limitation in the future... this is more intuitive anyhow):
(The following is pseudocode.)
import numpy as np
import pandas as pd

# create a store
store = pd.HDFStore('mystore.h5')

# this is the key to your storage:
#    this maps your fields to a specific group, and defines
#    what you want to have as data_columns.
#    you might want to create a nice class wrapping this
#    (as you will want to have this map and its inversion)
group_map = dict(
    A = dict(fields = ['field_1','field_2',.....], dc = ['field_1',....,'field_5']),
    B = dict(fields = ['field_10',......        ], dc = ['field_10']),
    .....
    REPORTING_ONLY = dict(fields = ['field_1000','field_1001',...], dc = []),

)

group_map_inverted = dict()
for g, v in group_map.items():
    group_map_inverted.update(dict([ (f,g) for f in v['fields'] ]))

Reading in the files and creating the storage (essentially doing what append_to_multiple does):
for f in files:
   # read in the file, additional options hmay be necessary here
   # the chunksize is not strictly necessary, you may be able to slurp each
   # file into memory in which case just eliminate this part of the loop
   # (you can also change chunksize if necessary)
   for chunk in pd.read_table(f, chunksize=50000):
       # we are going to append to each table by group
       # we are not going to create indexes at this time
       # but we *ARE* going to create (some) data_columns

       # figure out the field groupings
       for g, v in group_map.items():
             # create the frame for this group
             frame = chunk.reindex(columns = v['fields'], copy = False)

             # append it
             store.append(g, frame, index=False, data_columns = v['dc'])

Now you have all of the tables in the file (actually you could store them in separate files if you wish, you would prob have to add the filename to the group_map, but probably this isn't necessary).
This is how you get columns and create new ones:
frame = store.select(group_that_I_want)
# you can optionally specify:
# columns = a list of the columns IN THAT GROUP (if you wanted to
#     select only say 3 out of the 20 columns in this sub-table)
# and a where clause if you want a subset of the rows

# do calculations on this frame
new_frame = cool_function_on_frame(frame)

# to 'add columns', create a new group (you probably want to
# limit the columns in this new_group to be only NEW ones
# (e.g. so you don't overlap from the other tables)
# add this info to the group_map
store.append(new_group, new_frame.reindex(columns = new_columns_created, copy = False), data_columns = new_columns_created)

When you are ready for post_processing:
# This may be a bit tricky; and depends what you are actually doing.
# I may need to modify this function to be a bit more general:
report_data = store.select_as_multiple([groups_1,groups_2,.....], where =['field_1>0', 'field_1000=foo'], selector = group_1)

About data_columns, you don't actually need to define ANY data_columns; they allow you to sub-select rows based on the column. E.g. something like:
store.select(group, where = ['field_1000=foo', 'field_1001>0'])

They may be most interesting to you in the final report generation stage (essentially a data column is segregated from other columns, which might impact efficiency somewhat if you define a lot).
You also might want to:

create a function which takes a list of fields, looks up the groups in the groups_map, then selects these and concatenates the results so you get the resulting frame (this is essentially what select_as_multiple does). This way the structure would be pretty transparent to you.
indexes on certain data columns (makes row-subsetting much faster).
enable compression.

Let me know when you have questions!


Input:
Convert hex string to int in Python How do I convert a hex string to an int in Python?
I may have it as "0xffff" or just "ffff".

Output:
Without the 0x prefix, you need to specify the base explicitly, otherwise there's no way to tell:
x = int("deadbeef", 16)

With the 0x prefix, Python can distinguish hex and decimal automatically.
>>> print int("0xdeadbeef", 0)
3735928559
>>> print int("10", 0)
10

(You must specify 0 as the base in order to invoke this prefix-guessing behavior; omitting the second parameter means to assume base-10. See the comments for more details.)


Input:
How do you remove duplicates from a list in whilst preserving order? Is there a built-in that removes duplicates from list in Python, whilst preserving order? I know that I can use a set to remove duplicates, but that destroys the original order. I also know that I can roll my own like this:
def uniq(input):
  output = []
  for x in input:
    if x not in output:
      output.append(x)
  return output

(Thanks to unwind for that code sample.)
But I'd like to avail myself of a built-in or a more Pythonic idiom if possible.
Related question: In Python, what is the fastest algorithm for removing duplicates from a list so that all elements are unique while preserving order?

Output:
Here you have some alternatives: http://www.peterbe.com/plog/uniqifiers-benchmark
Fastest one:
def f7(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq if not (x in seen or seen_add(x))]

Why assign seen.add to seen_add instead of just calling seen.add? Python is a dynamic language, and resolving seen.add each iteration is more costly than resolving a local variable. seen.add could have changed between iterations, and the runtime isn't smart enough to rule that out. To play it safe, it has to check the object each time.
If you plan on using this function a lot on the same dataset, perhaps you would be better off with an ordered set: http://code.activestate.com/recipes/528878/
O(1) insertion, deletion and member-check per operation.


Input: Directory listing in Python How do I get a list of all files (and directories) in a given directory in Python?

Output:
This is a way to traverse every file and directory in a directory tree:
import os

for dirname, dirnames, filenames in os.walk('.'):
    # print path to all subdirectories first.
    for subdirname in dirnames:
        print(os.path.join(dirname, subdirname))

    # print path to all filenames.
    for filename in filenames:
        print(os.path.join(dirname, filename))

    # Advanced usage:
    # editing the 'dirnames' list will stop os.walk() from recursing into there.
    if '.git' in dirnames:
        # don't go into any .git directories.
        dirnames.remove('.git')


Input: Checking whether a variable is an integer or not How do I check whether a variable is an integer?

Output:
If you need to do this, do
isinstance( <var>, int )

unless you are in Python 2.x in which case you want
isinstance( <var>, ( int, long ) )

Do not use type. It is almost never the right answer in Python, since it blocks all the flexibility of polymorphism. For instance, if you subclass int, your new class should register as an int, which type will not do:
class Spam( int ): pass
x = Spam( 0 )
type( x ) == int # False
isinstance( x, int ) # True

This adheres to Python's strong polymorphism: you should allow any object that behaves like an int, instead of mandating that it be one.
BUT
The classical Python mentality, though, is that it's easier to ask forgiveness than permission. In other words, don't check whether x is an integer; assume that it is and catch the exception results if it isn't:
try:
    x += 1
except TypeError:
    ...

This mentality is slowly being overtaken by the use of abstract base classes, which let you register exactly what properties your object should have (adding? multiplying? doubling?) by making it inherit from a specially-constructed class. That would be the best solution, since it will permit exactly those objects with the necessary and sufficient attributes, but you will have to read the docs on how to use it.


Input:
How do I unload (reload) a Python module? I have a long-running Python server and would like to be able to upgrade a service without restarting the server. What's the best way do do this?
if foo.py has changed:
    unimport foo  <-- How do I do this?
    import foo
    myfoo = foo.Foo()

Output:
You can reload a module when it has already been imported by using the reload builtin function in Python 2:
import foo

while True:
    # Do some things.
    if is_changed(foo):
        foo = reload(foo)

In Python 3, reload was moved to the imp module. In 3.4, imp was deprecated in favor of importlib, and reload was added to the latter. When targeting 3 or later, either reference the appropriate module when calling reload or import it.
I think that this is what you want. Web servers like Django's development server use this so that you can see the effects of your code changes without restarting the server process itself.
To quote from the docs:

Python modulesâ code is recompiled and
  the module-level code reexecuted,
  defining a new set of objects which
  are bound to names in the moduleâs
  dictionary. The init function of
  extension modules is not called a
  second time. As with all other objects
  in Python the old objects are only
  reclaimed after their reference counts
  drop to zero. The names in the module
  namespace are updated to point to any
  new or changed objects. Other
  references to the old objects (such as
  names external to the module) are not
  rebound to refer to the new objects
  and must be updated in each namespace
  where they occur if that is desired.

As you noted in your question, you'll have to reconstruct Foo objects if the Foo class resides in the foo module.


Input: Can someone explain __all__ in Python? I have been using Python more and more, and I keep seeing the variable __all__ set in different __init__.py files.  Can someone explain what this does?

Output:
Linked to, but not explicitly mentioned here, is exactly when __all__ is used. It is a list of strings defining what symbols in a module will be exported when from <module> import * is used on the module.
For example, the following code in a foo.py explicitly exports the symbols bar and baz:
__all__ = ['bar', 'baz']

waz = 5
bar = 10
def baz(): return 'baz'

These symbols can then be imported like so:
from foo import *

print bar
print baz

# The following will trigger an exception, as "waz" is not exported by the module
print waz

If the __all__ above is commented out, this code will then execute to completion, as the default behaviour of import * is to import all symbols that do not begin with an underscore, from the given namespace.
Reference: https://docs.python.org/3.5/tutorial/modules.html#importing-from-a-package
NOTE: __all__ affects the from <module> import * behavior only. Members that are not mentioned in __all__ are still accessible from outside the module and can be imported with from <module> import <member>.


Input:
How do I protect Python code? I am developing a piece of software in Python that will be distributed to my employer's customers. My employer wants to limit the usage of the software with a time restricted license file.
If we distribute the .py files or even .pyc files it will be easy to (decompile and) remove the code that checks the license file.
Another aspect is that my employer does not want the code to be read by our customers, fearing that the code may be stolen or at least the "novel ideas".
Is there a good way to handle this problem? Preferably with an off-the-shelf solution.
The software will run on Linux systems (so I don't think py2exe will do the trick).

Output:
"Is there a good way to handle this problem?"  No.  Nothing can be protected against reverse engineering.  Even the firmware on DVD machines has been reverse engineered and AACS Encryption key exposed.  And that's in spite of the DMCA making that a criminal offense.
Since no technical method can stop your customers from reading your code, you have to apply ordinary commercial methods.

Licenses.  Contracts.  Terms and Conditions.  This still works even when people can read the code.  Note that some of your Python-based components may require that you pay fees before you sell software using those components.  Also, some open-source licenses prohibit you from concealing the source or origins of that component.
Offer significant value.  If your stuff is so good -- at a price that is hard to refuse -- there's no incentive to waste time and money reverse engineering anything.  Reverse engineering is expensive.  Make your product slightly less expensive.
Offer upgrades and enhancements that make any reverse engineering a bad idea.  When the next release breaks their reverse engineering, there's no point.  This can be carried to absurd extremes, but you should offer new features that make the next release more valuable than reverse engineering.
Offer customization at rates so attractive that they'd rather pay you do build and support the enhancements.
Use a license key which expires.  This is cruel, and will give you a bad reputation, but it certainly makes your software stop working.
Offer it as a web service.  SaaS involves no downloads to customers.


Input:
What is the best project structure for a Python application? Imagine that you want to develop a non-trivial end-user desktop (not web) application in Python. What is the best way to structure the project's folder hierarchy?
Desirable features are ease of maintenance, IDE-friendliness, suitability for source control branching/merging, and easy generation of install packages.
In particular:

Where do you put the source?
Where do you put application startup scripts?
Where do you put the IDE project cruft?
Where do you put the unit/acceptance tests?
Where do you put non-Python data such as config files?
Where do you put non-Python sources such as C++ for pyd/so binary extension modules?

Output:
Doesn't too much matter.  Whatever makes you happy will work.  There aren't a lot of silly rules because Python projects can be simple.

/scripts or /bin for that kind of command-line interface stuff
/tests for your tests
/lib for your C-language libraries
/doc for most documentation
/apidoc for the Epydoc-generated API docs.

And the top-level directory can contain README's, Config's and whatnot.
The hard choice is whether or not to use a /src tree.  Python doesn't have a distinction between /src, /lib, and /bin like Java or C has.
Since a top-level /src directory is seen by some as meaningless, your top-level directory can be the top-level architecture of your application.

/foo
/bar
/baz

I recommend putting all of this under the "name-of-my-product" directory.  So, if you're writing an application named quux, the directory that contains all this stuff is named  /quux.
Another project's PYTHONPATH, then, can include /path/to/quux/foo to reuse the QUUX.foo module.
In my case, since I use Komodo Edit, my IDE cuft is a single .KPF file.  I actually put that in the top-level /quux directory, and omit adding it to SVN.


Input:
How do I check if a variable exists? I want to check if a variable exists. Now I'm doing something like this:
try:
   myVar
except NameError:
   # Do something.

Are there other ways without exceptions?

Output:
To check the existence of a local variable:
if 'myVar' in locals():
  # myVar exists.

To check the existence of a global variable:
if 'myVar' in globals():
  # myVar exists.

To check if an object has an attribute:
if hasattr(obj, 'attr_name'):
  # obj.attr_name exists.


Input:
Getting the length of an array in Python In Python, is the following the only way to get the number of elements?
arr.__len__()

If so, why the strange syntax?

Output:
my_list = [1,2,3,4,5]
len(my_list)

The same works for tuples:
my_tuple = (1,2,3,4,5)
len(my_tuple)

And strings, which are really just arrays of characters:
my_string = 'hello world'
len(my_string)

It was intentionally done this way so that lists, tuples and other container types didn't all need to explicitly implement a public .length() method, instead you can just check the len() of anything that implements the 'magic' __len__() method.
Sure, this may seem redundant, but length checking implementations can vary considerably, even within the same language. It's not uncommon to see one collection type use a .length() method while another type uses a .length property, while yet another uses .count(). Having a language-level keyword unifies the entry point for all these types. So even objects you may not consider to be lists of elements could still be length-checked.  This includes strings, queues, trees, etc.


Input: Is there any way to kill a Thread in Python? Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

Output:
It is generally a bad pattern to kill a thread abruptly, in Python and in any language. Think of the following cases:

the thread is holding a critical resource that must be closed properly
the thread has created several other threads that must be killed as well.

The nice way of handling this if you can afford it (if you are managing your own threads) is to have an exit_request flag that each threads checks on regular interval to see if it is time for him to exit.
For example:
import threading

class StoppableThread(threading.Thread):
    """Thread class with a stop() method. The thread itself has to check
    regularly for the stopped() condition."""

    def __init__(self):
        super(StoppableThread, self).__init__()
        self._stop = threading.Event()

    def stop(self):
        self._stop.set()

    def stopped(self):
        return self._stop.isSet()

In this code, you should call stop() on the thread when you want it to exit, and wait for the thread to exit properly using join(). The thread should check the stop flag at regular intervals.
There are cases however when you really need to kill a thread. An example is when you are wrapping an external library that is busy for long calls and you want to interrupt it.
The following code allows (with some restrictions) to raise an Exception in a Python thread:
def _async_raise(tid, exctype):
    '''Raises an exception in the threads with id tid'''
    if not inspect.isclass(exctype):
        raise TypeError("Only types can be raised (not instances)")
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid,
                                                  ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        # "if it returns a number greater than one, you're in trouble,
        # and you should call it again with exc=NULL to revert the effect"
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, 0)
        raise SystemError("PyThreadState_SetAsyncExc failed")

class ThreadWithExc(threading.Thread):
    '''A thread class that supports raising exception in the thread from
       another thread.
    '''
    def _get_my_tid(self):
        """determines this (self's) thread id

        CAREFUL : this function is executed in the context of the caller
        thread, to get the identity of the thread represented by this
        instance.
        """
        if not self.isAlive():
            raise threading.ThreadError("the thread is not active")

        # do we have it cached?
        if hasattr(self, "_thread_id"):
            return self._thread_id

        # no, look for it in the _active dict
        for tid, tobj in threading._active.items():
            if tobj is self:
                self._thread_id = tid
                return tid

        # TODO: in python 2.6, there's a simpler way to do : self.ident

        raise AssertionError("could not determine the thread's id")

    def raiseExc(self, exctype):
        """Raises the given exception type in the context of this thread.

        If the thread is busy in a system call (time.sleep(),
        socket.accept(), ...), the exception is simply ignored.

        If you are sure that your exception should terminate the thread,
        one way to ensure that it works is:

            t = ThreadWithExc( ... )
            ...
            t.raiseExc( SomeException )
            while t.isAlive():
                time.sleep( 0.1 )
                t.raiseExc( SomeException )

        If the exception is to be caught by the thread, you need a way to
        check that your thread has caught it.

        CAREFUL : this function is executed in the context of the
        caller thread, to raise an excpetion in the context of the
        thread represented by this instance.
        """
        _async_raise( self._get_my_tid(), exctype )

As noted in the documentation, this is not a magic bullet because if the thread is busy outside the Python interpreter, it will not catch the interruption.
A good usage pattern of this code is to have the thread catch a specific exception and perform the cleanup. That way, you can interrupt a task and still have proper cleanup.