Snow Leopard’s Python 2.6 site-packages (and a psycopg2 issue)

THE ULTIMATE EDIT: I’ve got new instructions on how to conquer Symbol not found: \_PQbackendPID! Check them out here.

Maybe I’m a newb, maybe I’m not a newb. But I had a hard time installing Django (from scratch) on my new install of Mac OS X 10.6 Snow Leopard. It appears my issue was with the site-packages location. It’s got python26 installed by default, installed in

 /System/Library/Frameworks/Python.framework

So I ran my handy site-packages finding command:

 python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"

It gave me this path.

 /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

So I linked Django to that directory. But when I tested my setup to see if it would load the Django modules, it wouldn’t. So I printed my pythonpath (going into the python shell and running import sys; print sys.path) and I found the only default site-packages directory in pythonpath.

 /Library/Python/2.6/site-packages

Once I linked Django there, it worked like a charm. Hope this saves you a 5-minute headache or something.

Update: I also ran into a bit of a tiff installing psycopg2 from source. I ran the following commands, which installed psycopg2 into the previously-listed System site-packages path:

 python setup.py build
 sudo python setup.py install

Rather, once I ran the easy install command (the setuptools for this seem to be included), my python setup found all the modules it needed for PostgreSQL integration.

 easy_install .

Again, hope this helps another 5-minute headache!

Another edit: there’s been some discussion in the comments about a problem with PostgreSQL/Python integration throwing errors in Snow Leopard. Namely, it’s the following error message:

 Symbol not found: _PQbackendPID

The general, scientific Internet consensus is that it’s due to a weird mixture of 32-bit plugins with a 64-bit process. Until more and more things become available as 64-bit, you’ll probably want to run Apple’s build of Python 2.6 as 32-bit for the time being. Here’s the terminal command to do just that.

 defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

Not sure if that needs to be run with sudo, but there you have it. Thanks to @Mark and @John Simons for laying this out and explaining it.

Django newbness: ImageField and the missing request.FILES data

I’ve created a new category for stuff like this that I discover while writing Django stuff, and this would be the first post in that category (I’ll go back through and assign all my old applicable posts to this Django category, too).

Anyway, I was having trouble with an ImageField in a model. I created a ModelForm that used this model to map out the associated form fields, including one with a file input widget for the ImageField. When it came time for me to use this with a template, I had all the fields in place, the proper directories had the proper permissions and all the settings were set, and I tried to upload the image. The form validated and it acted like nothing happened, like no picture was ever attached. This had me frustrated.

So I tried inserting an assert False to see what my locals() were. There was nothing in request.FILES, where there should have been. After some Googling and trying to find the right way to phrase my Google search, I found my solution here.

Apparently my form tag didn’t have one critical attribute:

 enctype="multipart/form-data"

After I had that jewel, all was golden.

Django and the admin date/time picker

Author’s note: I’m using the Django trunk with all this, but the following directions at least work with the 1.1 release.

The problem

So here’s my dilemma. In developing a few various forms for Etcetera (namely a manual equipment checkout form) I’ve run into what may be the hairiest thing yet for end-users of my system: Django’s forms.DateTimeField. It’s great for validating form data to make sure it’s in a compatible YYYY-MM-DD HH:MM format to store in a date field in the database. Unfortunately for those using the system, this restrains them to a very strict format to follow, and perhaps a difficult one to remember (we do have one user who’s in ROTC and is quite comfortable with 24-hour time). So why should we be stuck with a single field (or even a split field, for that matter) where a user is confined to typing in this date/time by hand? I wanted to utilize the Django admin site’s date picker widget.

And you might ask this: why not use jQuery UI’s slick calendar? Or some other DHTML/JS-blended über-calendar to add a date picker? Simple: reduce dependencies. I don’t want to have to rely on installing any more frameworks or hacks, no matter how simple they are to implement. I want this to be as self-contained within the Django framework as possible.

The hint

Enter django.contrib.admin.widgets.AdminSplitDateTime.

It’s simple, clean, effective, easy-to-use. So I can just import the widgets package, specify widgets.AdminSplitDateTime as the my forms.DateTimeForm’s widget, and roll, right? It doesn’t seem to work like that. The widget depends on some extra JavaScript files (and CSS styles) in your django/contrib/admin/media directory, as well as the admin’s i18n script. There’s a couple of additional lines of code you’ll have to add that won’t be apparent at first sight.

The solution

I found this in a handy tip in a Google Groups discussion that implements it just the way it’s supposed to work. I’ll reiterate it here. Basically you set your form field’s widget to widgets.AdminSplitDateTime in your form class, and link in the JavaScript files from the admin’s media site into your Django templates. You’ll also have to set up a URL reference in your URLconf to the Django admin’s i18n script. Step by step, here it goes!

Step 1: Setting up your form

I’m assuming you already have your forms.py file in your app right now. For my code, I’ve got a project called etcetera and an app called checkout. To show you where it all starts, I’ve got an abbreviated version of my model, which is under checkout/models.py.

I’ve got my form, CheckoutModelForm, which uses forms.ModelForm, and overrides the two date fields. Notice how I’ve imported django.contrib.admin.widgets as adminwidgets, and that I’ve specified custom widgets for my two DateTimeFields.

On a related note, I developed the checkout app its own git branch. This gets weird switching to work on it from the master: running git checkout checkout sometimes throws me for a loop!

Step 2: A minor URLconf change (and a settings check)

Before we do anything to our templates, we’re going to add a minor change to our root URLconf. As my project is etcetera, the file I’m talking about is etcetera.urls. It’s possible you already have the admin site installed. in that case, you’ll only need to add the line for admin docs, and the line for admin i18n.

We’re also going to check your project’s settings.py file. Make sure you have your admin’s media directory set (the setting is ADMIN_MEDIA_PREFIX). Mine’s the default, /admin/. So if you don’t have it, add the following quick line to your settings.py file.

 ADMIN_MEDIA_PREFIX = '/media/'

Now we’re ready to modify our templates to accept the changes brought about by this new form field.

Step 3: Tweaking the templates

You’ll probably want to take a look at my base template, which has a few blocks specified where I’m going to inject code with my specific template (the one that’ll extend this one). My base template is called root.html. For the sake of brevity, I’ve removed all the stuff that’s not really applicable to the task at hand (header/footers, a few additional stylesheets and scripts, blocks, includes, etc).

Here’s the template that my edit view renders with CheckoutModelForm. It’s simply called edit.html, and I’ve removed some stuff here for the sake of brevity as well.

You’ll notice some extra scripts specified here. Those’ll end up between the

While it’s not exactly part of my templates — it’s definitely used by them — I should go ahead and talk some of the CSS used by this date/time picker. I’ve taken some of this code right from the admin media’s widgets.css file and tweaked it. It resides at the bottom of my screen.css file. Note that while all the admin media is accessible from /media/, all my personal media for the site is from /_media/. Hope that’s not too confusing. There’s a whole section in the widgets.css file that deal with the calendar and time picker, and it’s denoted by comments so you won’t miss it. It’s pretty much the classes .calendar, .clockbox, and a couple of others. As always, with CSS, you should modify it to your liking and to get it to jive with the rest of your content. Here’s my finished product.

Sound off in the comments if you have any questions. I hope I’ve covered everything!

Conquering Symbol not found: _PQbackendPID

Author’s note: this is in reference to a previous post, which didn’t quite involve this error, but evolved to a discussion on it anyway. It’s become a problem for me as well, and I didn’t want to run Python as 32-bit, so I’ve figured something out. Here it is.

I’ve found what may be the best solution for this, and it avoids MacPorts’ mumbo-jumbo (of which I’ve been trying to avoid as much as I can). It actually involves Homebrew, a new Mac package manager that bests MacPorts in a number of ways. It involves rolling up your sleeves and doing a little more work, but it adheres much more strictly to the Mac-like idea of not having to sudo for stuff as much as possible.

Anyway, install Homebrew, which I’ve done, to /usr/local, and use it to install python and postgresql (the directions for all this are at the github page for Homebrew). Then, download the source for psycopg2 — either recent build will work, I’ve used .11 and .13 — and unzip it somewhere, and then use the easy_install . command like before; this should use the new Python inside Homebrew’s setup (inside /usr/local/bin) instead. This worked for me like a charm, all without MacPorts (thank goodness)! Note that to get your shell to recognize the new Python over the old one, you’ll have to prioritize /usr/local/bin on your $PATH. You should probably already know how to do that.

Why does this work? From poking around the Intarwebs, as far as I can tell, it lays in how PostgreSQL is installed under Snow Leopard. I’ve tried so many things, and the only ones that seem to work are MacPorts’ build, and Homebrew’s build. I’ve tried building from source, using the installer from EnterpriseDB, etc. — and since Homebrew and MacPorts have specialized build strings (for architecture and what-not) for Leopard and (more specifically) Snow Leopard, this builds it in the right way for psycopg2 to access it — without the nasty _PQbackendPID error. Hope this helps!

I’m not sure how well this works in virtualenv — I haven’t gotten into using them yet, although I hear daily that I should. Anybody have a good learning resource on how to use virtualenv? If somebody lets me know, that’ll help me greatly.

Now I’m off to fix this, which I’ve done too many times to count:

Update on Etcetera, screenshots too!

I’m giving a brief presentation on the Django-based system I’m building for Missouri State University’s Educational Technology Center tomorrow, and thus I’ve done a few obligatory slides on what it can do. Here’s a SlideShare viewer for the PDF I’ve saved from my slides.

I’m developing it using Django’s subversion trunk for now, but once 1.2 is released (before too long, I imagine), I’ll lock in the feature set there and use the 1.2 line. All development is done on my local machine, which is setup with virtualenv and pip. I’m currently using TextMate and occasionally MacVim as my text editors. I then use my local machine as an Apache test environment as well, running through a mod_wsgi daemon. All the development code is currently hosted on github, but that may change if the university doesn’t want the code listed publicly. It’s there for convenience, as I can post updates to the application there, and whoever manages the server in the future can just pull the changes. It’s also great for collaboration, as others can view the code and make suggestions (which are surely needed; I’m still in the Django-learning phase).

The production server running it is a Mac OS X Leopard Server-running quad-core Intel Xserve. It uses PostgreSQL 8 in both development and production. Here’s some numbers (my favorite part).

• The system is serving roughly 16 (soon to be about 25) users
• It houses nearly 6000 rows of equipment — everything in ETC’s inventory
• It holds close to 1000 past and present work order tickets for equipment problems across campus
• It will track about 1200 equipment checkout orders per fiscal year from 91 buildings/locations and 324 organizational/business units on campus
• I really need to install Google Analytics code to get usage statistics… once I do, I’ll have those numbers as well

It runs quite nicely in tandem with Nathan Borror’s slick redesign of our department’s website, also written in Django (it uses docutils to parse reStructuredText files into HTML).

Extending Django’s manage.py

I was sitting at work one day faced with a dilemma. This particular brain puzzle was translating a certain business need into code; something that, usually, isn’t too hard for me to do. This problem involved Etcetera’s equipment checkout module and the manner in which it interacts with the equipment inventory module.

The problem

When a patron submits a checkout request for equipment online, they specify the date the loan begins and the date the loan ends, usually in the ballpark of a week’s time. Our folks behind the scenes interacting with the system, say, when they’re speaking with a patron over the phone, need to see what equipment we have available and at what time. So when the equipment delivery guys attach a specific equipment item to a checkout, that equipment’s status within the system gets marked as reserved. The business need (now that you have the background story) is this: any equipment that’s reserved needs to show that it’s checked out during the time window that it’s actually checked out.

Here’s such a piece of equipment.

My options

SQL queries

My first thought was to use what I had been taught in college: SQL update queries. Mmm! I’d devise a clever little script that would send a command-line update bit to psql every few minutes to switch over the equipment attached to checkouts that had a start date in the past and a return date in the future, and that weren’t already marked as completed/archived. I racked my poor feeble brain on this method for 6 hours before falling to my knees and begging to be rendered unconscious. I’m okay with SQL, but this was beyond my expertise and I shouldn’t have to waste this much time on something that seems fairly simple.

Using Django’s manage.py

Enter solution #2 (also known as why the task at hand already seemed simple). What I really wanted to do was use Django’s API, specifically its model and QuerySet APIs, to accomplish the database update. But this was supposed to be a temporal event: something that fired at certain points in time. I had already planned to run the SQL script using launchd (we’re hosting the Django app on a Mac OS X Server), but I had no idea how I’d get a Python script to run and properly find the Django module (I’m running my code inside a Python virtualenv — try it sometime). My missing piece in this puzzle was merely a shell script a better plist. So here’s what I’ve got.

Digging in

Creating the command class

I’ve got a management command built into my checkout app. Doing so requires that, within the app’s folder, you have another folder, management. Inside this folder, create a blank __init__.py file, and another folder, named commands. This folder must also contain a blank __init__.py file (remember, this blank file tells the Python interpreter that the folder contains runnable code; basically, it’s saying the folder is a module) as well as a Python script file that goes by the name of the command you want to make. My command is named equipment_status, so my file is equipment_status.py. Here’s my code for that file.

Note that the class name is Command and it’s a subclass of django.core.management.base.NoArgsCommand. That’s because my function doesn’t take any arguments. If yours will, subclass it off of …base.BaseCommand — and your code will be a little different than mine. Pretty much, mine uses the handle_noargs method, and acts like a regular Django/Python function, and uses the Django model API — a million-and-a-half times better than writing raw SQL. Trust me on that one.

Triggering the shell script plist with launchd

But now we need this to run every 120 seconds, and that’s where launchd and the /Library/LaunchAgents folder comes in. Here’s my plist file, named edu.missouristate.etc.equipment-status.plist, which is based on the reverse DNS hostname of the machine it’s made by, or run on, or something… it’s the custom, whatever. And it’s running on that website, so that’s the name.

These options for this plist file are ones I’ve found on Apple’s Manual Page for launchd. The page doesn’t do the best job in the world explaining what options you have to include and which you don’t (it’s a little daunting to look at), but I’ve made my way through it and developed the script above, which pretty much has the bare minimum.

• StartInterval defines and integer of the number of seconds between each launch of the program/script. Here, I’ve set it to 120 seconds (2 minutes).
• Label must be defined, and here it’s the filename without the plist extension. Best to be consistent.
• WorkingDirectory is the cd command of the plist, if you will; it specifies the directory we’ll be working in, which in this case is the project folder within my virtualenv.
• ProgramArguments is a string array that specifies the absolute path to the python binary I want to use (it’s in my virtualenv directory), the manage.py command, and the equipment_status method argument.
• RunAtLoad is set only because I was trying to figure out if the script would run properly or not, and I didn’t want to have to wait 2 minutes after I loaded it into launchctl to see if it’d work.

So that’s it for the code. I had to make sure the plist had the appropriate permissions (it must be owned by root:wheel to run, otherwise you’ll get a ‘dubious ownership’ error when you attempt to load it — you may have to sudo chown it). There’s one more command you’ll have to run to enable this script.

sudo launchctl load -w /Library/LaunchAgents/edu.missouristate.etc.equipment-status.plist

After that it should run every 2 minutes. Check Console.app if you don’t think it’s working right. If you search for the name of the plist file, you’ll get a nice query of error messages (or stdout messages) if there are any. That’s it!

Also, props to Ted Kaemming for helping me figure some of the Django stuff out.

Edit: I’ve updated my post and the GitHub gist for the plist with Brooks Travis’s awesome tip: cutting out the shell script and specifying the virtualenv’s python as the one to run.

Etcetera: nearing 1.0 release

As I’ve been punching away at building Etcetera, I seem to have lost sight of versioning and building anticipation and what-not… and it really doesn’t matter that much to me (nor does it matter to those that are using the system), but in thinking about it, I’ve realized that I probably should have some sort of milestone that I’m building towards, as it gives me a good outline of where I should be headed in terms of features and capabilities. I’ve decided that milestone is the 1.0 release. Once it’s in place, only minor feature revisions will be added to the code base (as well as fixes and things) unless something totally drastic happens (like a mandate from the heavens).

I’m getting closer and closer to the feature goal, and this is a brief overview of what I’ve gotten done so far.

• Actions menu: while this used to be along the ‘headline’ of the page, I’ve moved it to a more expandable box on the right-hand side of the page with a fixed width. I’d like to get it looking better (perhaps at the same height of the whole heading section), but for now, it’s there and it’s easier to add commands to without the concern of running out of room.
• Expanded model fields: in the Checkout model, I’ve added an activation feature (in the form of a datetime form field) that drives the organization of tickets between various sections of the Checkout application. Now the equipment technicians can toggle when they’ve delivered a ticket and it gets timestamped. Better for record-keeping, too. The WorkOrder model in the Service app has a modified department field — instead of a basic text charfield, it’s now a foreignkey mapped to our organizational unit table. There’s an additional department_text field that assumes the old functionality for the public’s benefit.
• Better date/time input: per this post, I’ve implemented something that easily tops the built-in admin date/time picker widget functionality of Django, and it’s a set of form subclasses that use python-dateutil for date/time string validation, and it’s damn good at handling a regular user’s interpretation of the date and time. For most of our public form users, the time picker led to the most confusion, so I used a touch of CSS to hide it temporarily. This serves as the fix in shining armor for that problem. Besides… who in the States likes to use (let alone understand) 24-hour time?
• Per building/department/equipment quick reports: the feature I’m currently working on implements a display of the open checkout and workorder tickets per department or per building on campus. I’ve also added this per equipment; instead of open tickets, it shows everything prior. And I think it’s a paginated list. Can’t remember.
• Google Charts API integration: a cool thing that I was working on but kinda put on the backburner. Google’s got a great easy-to-use charts API that I’ve learned how to use. There’s even Python bindings available for it. Problem is there’s not a whole lot of good documentation on using it, so I’ve had to do a lot of tinkering with it. I’m hoping to have it ready with the 1.0 release, but no promises there.

You can track my progress on Etcetera’s GitHub page, if that’s your thing. If not, I should be posting more updates soon about things as I gear up for that 1.0 release!