Final Stretch

So the past two days have been madness trying to get the recently christened WebBotWar (or just WebBot) to actually work on the web.

We had it working locally some time last week (I think... the days are really starting to mush together), but OpenShift was a whole other thing. This boiled down to two basic problems we had:

• pybotwar depended on pyBox2D, which needs to compile, which doesn't work well even when you have control of the machine
• We were relying on memcached to provide cheap communication between the pybotwar process and the frontend. As near as I can tell, memcached is not actually supported on OpenShift Express, though that might not actually be true.

The first was surprisingly easy to fix, though it took me some time to actually think of the solution. In the final setup, there are essentially three repositories: our modified pybotwar, the TG2 webbot frontend, and a meta-repository containing both of the previous two in the proper places. This third repo is not meant to be actually used to develop, its only purpose is to be pushed to OpenShift and act as a quick pull for someone looking to run WebBotWar themselves. The practical upshot of this is that if we commit pyBox2D inside the pybotwar directory of our meta-repo, pybotwar can find its dependencies, and no one else needs to have to bother with it.

The second problem was more tricky, and eventually resulted in a rather simple patch that just happened to take me around 12 hours to get right. A quick Google of OpenShift Express Python and NoSQL led me to MongoDB, which has some benefits and drawbacks compared to just shoving bits into memory, but seems to work very well in practice and is probably the right way to go regardless. To be perfectly fair, memcached is a type of NoSQL, but MongoDB is actually supported by OpenShift in an easily-installable manner, and despite its more finicky syntax, it works, which is something I failed to get with memcached.

Meanwhile, the rest of my team was hard at work making massive progress on other fronts. Facebook authentication works, as does uploading custom robot definitions, though I don't think the two are plugged into each other yet. As well, there are brand new pretty images for the robots and the turrets.

There's a few outstanding problems left, but (as long as I don't push anything broken) you can have a look at webbotwar in action here.

TurboGears2 on OpenShift, just like it should be

After much work and many trials, I finally have an app pushed to OpenShift with no manual tweaking necessary. As often happens with these things, the solution was much simpler than expected.

Note: I still don't have a foolproof 'follow this' solution ready, as the one I built works exactly as I want it to, but:

• It needs a lot of love and cleanup
• It requires an external git script that isn't well documented

The first isn't much of a problem, and can be worked out over the next few days. I'm more worried about the second one. For the curious, the script is git-subtree, which acts like a submodule except it is more transparent to the repository which is a plus given OpenShift's odd structure.

Back on topic, when we last left off this topic, I had finally gotten OpenShift to acknowledge a project in a directory other than tg2app. This is useful because, at least for me, most of my projects are not named tg2app. That turned out to be stupid problem I had made for myself, but unfortunately, the next problem to tackle was not.

You see, when setting up an app on OpenShift, you have very little control over the actual environment the app is running in (this isn't entirely true, but is a useful fiction, especially as the service is likely to become more 'plug-and-go'). One of the few ways you can retain control is through a series of post-commit hooks, one of which was starting off the problematic section of code. When you first push your code to OpenShift, it needs to set up your database so it is ready to store information and do other databasey things.

Naturally, this wasn't happening.

First up was a problem with OpenShift. Python's default egg cache (not too important, it's a place python can use to extract files from installed packages temporarily) is not writable in OpenShift, so that needs to be set before anything else will work. Next, the proper MySQL library is not installed by TurboGears by default (the default is to use sqlite), so that had to be added to the requires list.

And then I hit yet another wall. Despite everything being set up properly, I could not connect to the MySQL database on OpenShift. It wasn't a problem with MySQL, because I could connect fine with the MySQL client. It wasn't even a problem with SQLAlchemy, because I was able to connect from a short example script. Finally, in a fit of insanity, I tried running the build script directly. I'm not even sure why, I was just at the point I would do anything just to see if it would work.

And, strangely enough, it did.

This had some pretty profound implications. It meant something was different during the build hook than in normal execution. Armed with this new knowledge, I headed over to OpenShift's IRC channel to get some answers (I had actually been in there for some time prior, just not with enough information for themore ruby-oriented users to help).

They told me that yes, indeed there was a difference. During the build step, the database is stopped, hence why I could not connect to it. There were, however, hooks for deploy and post_deploy, during both of which the database would be running. I moved the calls needing database access to deploy, and suddenly everything worked! I made a few more changes, cleaned up my tree, and tested it on a new app I wanted to get on OpenShift, and it (mostly) worked. There were a few problems left, but they seemed to be mostly my fault (and problems with the application, not OpenShift), so it looked like I had finally fixed deploying a standard TurboGears app. I've no doubt that there's something I've left out, but I'm pretty amazed at the progress I've made so far, and learned a lot about both OpenShift and TurboGears.

Openshift Troubles Continued

I figured out the problem I was having with OpenShift.

To put it simply, I didn't pay enough attention.

For reference, when moving an existing TurboGears app to OpenShift, make sure you add the changes in config/app_cfg.py

As soon as I saw that, I felt really silly for missing it. I was so sure that I had gotten all the relevant changes, but apparently I somehow missed this file.

More detailed directions coming soon.

OpenShift Troubles

Recently I've been playing with OpenShift, a new(ish) service from Red Hat as a sort of 'push to cloud' deployment strategy. It's interesting for people like me who can whip up a site quick, but don't necessarily have the framework in place to host it.
Due to my work on CIVX, I've gotten pretty familiar with TurboGears, and the idea of being able to take a site like that and run it without having to set up apache or fiddle with paster sounded real nice. Unfortunately, it was not so simple as it seems.
First up was to get something, anything, running. For a while, OpenShift was throwing a 500 error when you tried to get a new application registered. A brief poke into their IRC told had them aware of the problem and it was fixed shortly. Next came the other part that should have been easy, running TurboGears.
Luke our favorite wizard around these parts, wrote a bit of code to get most of the available applications up and running without too much effort, including the currently unsupported TurboGears. Running this went off without a problem, but the resulting site gave me another happy 500 error. After a consult with Remy, we determined there were some missing version requirements that kept the site from running. After pulling those edits out of his repo and moving them upstream to Luke's, I had a working default TurboGears site.
Until I tried to log in. Then I got another 500 error.
I was beginning to get used to this, but it was still annoying to make a small change, then push it to the server and wait for the server to update the settings before I could test it. Even more fun was the occasional 503 error when OpenShift couldn't keep up with my rapidly building and tearing down sites.
Feeling that that was going to be a project by itself, I set about moving all my non-db-interfacing files to this new repository. The prebuilt version assumes that the site internally is named tg2app, and I was having trouble convincing it to go by anything else. Eventually I just decided to move files across one by one; first the templates that don't care what they're named, then the root controller, than the new model and widget. A lot of frustration, many 403 and 404 errors later, I had something that pretended to work as long as I didn't use the database. But since the database is kind of the point of the site I was building, this was not exactly acceptable.
So back to the drawing board then. I had a hunch something was wrong when I saw SQLAlchemy errors scroll by every time I reloaded the site. My best guess is that SQLAlchemy is failing to create the tables needed to run the site and continuing on blindly. Once I realized that, I dumped a test db from my local copy to the mysql db, and suddenly everything was working. Or almost everything, anyway.
I could read form the db fine, but any time I tried to modify it, I got another dreaded 500 error. I poked into everything I could find to try to figure out where it was failing, and finally determined it couldn't be on my end, as my local copy worked just as expected.
Finally I stumbled across the anwser, almost accidentally. When I moved the db from local sqlite to mysql, I failed to set the auto increment setting on the id of my new databases, so when I neglected to provide an id for the new entries I was making, mysql quite rightly complained at me. Unfortunately, since I can't find how to re-enable debug mode (nor should I really try), I wasn't getting any good error messages.
So what is the site that has been giving me all these troubles? It's a little site I set up to publicly shame Remy into stopping smoking: remysmoke-qalthos.rhcloud.com

FLOSS Seminar Update

So it's week 5, what has been going on?

The class has split into three groups, working on different games. My group's idea is a social update to the venerable likes of robocode, though how exactly that is going to pan out is still debatable.

Ideally, this would comprise of a wrapper around the basic game, not requring much tweaking of the internals, though a new interface would be required for use on the web. All in all the class is shaping up to be pretty interesting.

Last night I finished up the second homework for the FLOSS Seminar. This one was to find an open-source project and fix some bug in their code. Not too hard, something I actually do a lot these days. But still not something that's easy to do on command.

(tl;dr: I forked [this] GNOME-Shell extension and submitted [this] pull request)

But let's get back to that in a minute. Depending on how you count it, I've been a part of this open-source community here at RIT for about two and a half years. I've been using Linux consistently since 2007, and was a casual user of open-source software for years before that, even when I didn't know it.

The notion of giving back, of being able to help make software better for everyone... back when I started, I never thought I'd be pushing changes to fix bugs in someone else's project. And now here I was, trying to find some new project to help because I'd submitted patches for all the problems I could think of last week. It's become so natural that being asked to do it seems weird in a way.

The hardest problem with this assignment wasn't actually finding the right bug, or even fixing it. The hardest problem was convincing my brain that this was work, and that it really was due soon. Many times I caught myself scouring bug trackers, looking through code, and telling myself, 'stop messing around and get to work!' So I'd dutifully search around, open my task list, look at the next item in my queue: "Bugfix HW for FLOSS". Oh, right...

Anyway, in the end, the bug I fixed was actually right in front of me the whole time. Prior to GNOME 3.2 coming out a few months ago, I had an extension I'd found on the Internet, one of the first for the new GNOME-Shell. It was a simple extension, all it did was stop the fancy overview mode from triggering when you hovered your mouse in the top left corner of the screen. I have a passionate dislike of hot corners in general, and after having swapped the buttons in the titlebars on my windows after playing in Ubuntu so much, I found I would usually activate it by accident at least ten times a day. I must have done at least thirty times before I remembered that somewhere on GitHub was a repository with a broken extension I was dying to fix.

The work itself wasn't too bad once I got down to it. I spent some time fiddling with JavaScript and looking through meager GNOME documentation, none of it really giving a sense of what was going on. Eventually I had to give up and look through some other extensions I had on hand that did similar things, but eventually I stumbled on a solution that seems to work and shoudn't conflict overmuch with other extensions (I think).

It seems the code I cloned from GitHub has been abandoned. The user who uploaded it hasn't done anything visible on GutHub since August, when he uploaded the files, and this is the only repository he has on the site. But I submitted a pull request anyway. And to my surprise, this morning I check GitHub, and my code has been merged. Who knows what's going to happen to it now, since the owner of the repository has indicated he has left GNOME-Shell. I, however, am planning on keeping it running for the forseeable future, especially now that I understand GNOME-Shell's particular brand of JavaScript now.

Post-Break Tasks

Today I got quite bit done. I reduced the whitespace between items in the grid view on the beta site, I wrote and exposed an edit form for the category headers, and I figured out but have not yet implemented the fancy calendar at people.rit.edu/fls5835. The main problem with that last one is that the calendar is built and pulled in by Javascript, and changing it to fit in a different space will be some trouble.