Apr 25 2010

This has always bugged me (scroll right to see the whole thing):

[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first):, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] , referer: http://mahara-test/admin/extensions/test.php?

I like using log files to display messages instead of the screen, but somewhere along the way somebody decided that the 'referer' (sic) was needed in the logs too, and that makes them totally unreadable.

Here's my fix, when tailing a log file:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/, referer:.*//'

And, instantly:

[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first):
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]

Much better! But we can do better again:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/\[error\]\s+\[client.*?\]\s+//; s/, referer:.*//'

And we end up with:

[Tue Apr 20 01:06:47 2010] [WAR] 7d (lib/web.php:916) Undefined variable: message
[Tue Apr 20 01:06:47 2010] Call stack (most recent first):
[Tue Apr 20 01:06:47 2010]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446
[Tue Apr 20 01:06:47 2010]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916
[Tue Apr 20 01:06:47 2010]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22
[Tue Apr 20 01:06:47 2010]

Hooray! Readable PHP log files.

You don't want to have to remember to type all of that, so for best results, put it in a script, chmod +x and you have:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | phptail

You could even combine it with Monkeytail to make viewing the logs even easier ;).

I've trawled through the PHP codebase looking for where it adds the referer in a vain attempt to change it, or provide a patch to make it a configuration option. Sadly, I never found it. If anyone finds out, please let me know!

Thanks to Martyn for the original idea, and assisting with the arcane perl stuff.

Like this post? Subscribe to my RSS feed and follow me on twitter to hear about new posts early.

Want to share this post?

Aug 23 2009

Here's a snapshot from my point of view about what's going on across all aspects of Mahara.

Hackfests

Yesterday, Penny, Jordi (both from Mahara partner Liip) and I had another weekend hackfest. Penny has been adding a framework for unit testing, and managed to get the first test working. Jordi was working on swapping Smarty for Dwoo, while I continued hacking on View Themes support. Pics: Switzerland end, NZ end.

Penny has been working on unit testing, as it's something we developers have realised has been lacking in Mahara for some time now. What unit tests do is give us more confidence that when we make a change (whether it's a bug fix or new feature), that the rest of Mahara continues working exactly the way it used to - i.e., we didn't inadvertently add any more bugs. This has the result that we can continue development safe in the knowledge that we didn't break anything that worked before, which will result in better quality. It will also make it easier for enthusiastic hackers to make changes to Mahara and know they haven't broken anything important.

Unit testing isn't the only way we've been trying to integrate testing - Gerald Quimpo from Catalyst has been resurrecting the Selenium tests that come with Mahara, and has done an excellent job in getting them working again, while also setting up a Continuous Integration server (Hudson) so the tests will be run regularly. Soon, we'll have that available publicly so everyone can see which tests are currently passing and failing.

Backtracking a tiny bit - Jordi has been working on swapping Smarty for Dwoo, a change we would like to do for a couple of reasons. Firstly, Smarty is bigger and slower, perhaps largely due to the fact that it needs to support PHP4 (which Mahara doesn't support). Secondly, Smarty doesn't make it easy for us to output variables that are HTML escaped by default - which is currently the leading cause of security vulnerabilities in Mahara. We haven't had many security vulns actually, Mahara is reaping the benefits of a security-oriented development ethic. But of the few we've had, almost all of them have been because someone forgot to put |escape on the end of a smarty variable. So switching to Dwoo will cut down that entire avenue of attack.

Mahara 1.2

We released the first beta of Mahara 1.2 last week. That means we're looking for bug reports now, and encourage people to download it and try out the new features. Here's a short list of the coolest new features in 1.2:

  • Import and Export - Mahara supports the LEAP2A standard for import/export, and in addition, allows you to export an HTML version of your portfolio (including a nice HTML represntation of an individual View).
  • "My Files" and file handling improvements - now the My Files section makes it easier to quickly upload files, and degrades to work without javascript. You can now upload files directly into Views, and use a standard directory browser to choose files in Views. For developers, we created a pieform element for uploading/attaching files, which will make it easier than ever to do this in custom plugins.
  • Theming - we're shipping with 6 themes, to give people an idea of how they can customise their Mahara. We've done a lot of rework of how themes are created, so from 1.2 it'll be even easier to create a custom theme. We even documented the process (finally!).
  • Usability/Comprehension tweaks - Now groups get a default forum, and blogs get a default blog. If there is only one forum or blog, users are taken straight to it. This should really help in preventing the problem where people create a blog as if it's a blog post, and should make the forums an even more useful tool.
  • Improved Moodle/Mahara integration - with a custom patch to Moodle 1.9, you'll be able to submit a View to Moodle for assessment as an assignment answer. How useful is that? Now people can keep their data in their portfolio, and mash it together into a View for assessment over on Moodle, where all the useful grading functionality is :)

There's s still a bit of work to go before Mahara 1.2 will be ready though. I need to finish off LEAP2A import - currently resume import is only half finished, and view import is just begun. We'll also need a fancy UI for it, and of course there are some inevitible bugs that will need squashing.

Also, I'd really like to get view themes to a point where it can be merged also. That'll really help people personalise their environment. If I get around to it, maybe we can add a theme chooser for users to change how they see the entire site, not just individual Views.

Other than that though, I think we're looking pretty good!

Code Contribution Policy

We are currently preparing a policy for contributors to the codebase. We aren't going to be wanting copyright assignment, but we are going to make sure all contributions are as legal as we can make them ;). Unlike many open source projects, Mahara is actually in a pretty good position legally, as most members of the team are well aware of the legal issues around open source and licensing, and we take legal issues seriously. Our policy won't be onerous, but will ensure we stay legit now and into the future.

Mahara 1.3

Though 1.2 is not out yet, talk is already shifting in some departments to the plans we have for Mahara 1.3. The big focus there will be usability, and we're going to target that in a couple of main areas - the navigation, and View creation.

For the navigation, our current plan is to adopt a dropdown menu structure, organised more to how users think when they're looking for something. The main categories will be roughly along the lines of "My Stuff", "Friends & Groups" and "Sharing". We'll build a demo of this at some point after 1.2 so the community can see it in action and give feedback, though I've seen it in action elsewhere and it's easy to grasp. Furthermore, the addition of "horizontal navigation" - where when you're at one page you're given suggestions about what else you might be looking for - should help tremendously as well.

We're also going to have a good look at the View creation process. The Drag and Drop and layout concepts won't be changing, but the layout of the editing screens themselves will be improved a lot. We're hoping we can even ditch a step to make things simpler. I think the biggest gains will be seen on the View Access screen, which currently can be a bit of a minefield to navigate. We'll get mockups out some time after 1.2 also.

Other stuff we might do could include Group Views, a User Dashboard (as a View of course!), and overhauling the notification system.

Mahara Growth

And last but not least, here are some interesting statistics about Mahara and its community:

We now have over 3000 members of the mahara.org community, with 17 partners worldwide. The community growth has been steady since we launched the new mahara.org last November. The site receives nearly 25,000 visits a month (for the geeks among you: 57% firefox and 27% IE). Of these visitors, we get the most from the US, followed by the UK, Germany, Spain, Australia and then New Zealand.

Since we added the ability for sites to register on mahara.org in May, we now have over 50 sites doing this every week. These sites together add to some 12,000 users, but this doesn't include mahara.org, or the New Zealand MyPortfolio service - if these numbers were included, the total would rise to nearly 25,000. Again for the geeks - we are seeing a 75/25 split of MySQL to PostgreSQL. Hopefully things will skew more to the PostgreSQL side of that equation as time goes on :). Of the registered sites, the largest seem to be run by Blackpool University, Uni Krems (Austria) and Birmingham City University, all of which have thousands of users.

Like this post? Subscribe to my RSS feed and follow me on twitter to hear about new posts early.

Want to share this post?

Jun 18 2009

OK so my last post was just all text, which is a slog to read through. So here, I made a video about Mahara 1.2's import/export:

I don't think I got the sound levels right and I'd have loved to have added a background track, but video editing on linux is a pain. It's only my second video so be nice. The next one should be better again ;)

Like this post? Subscribe to my RSS feed and follow me on twitter to hear about new posts early.

Want to share this post?