Padre blogs

Why is PHP so much easier for newbies?

Why does Java have the best IDE tools?

Why is Ruby prettier than Perl?

Why does Perl have the best package repository?

As I've played through Mass Effect 2 over the last few weeks, I see some interesting parallels.

In the Mass Effect universe, human technology is bootstrapped by the discovery of an ancient abandoned alien observation outpost on Mars, and the further discovery that the dwarf planet Charon is really an abandoned but active interstellar jump gate covered in ice.

Other similar species have done the same, resulting in a galactic community of around a dozen civilisations all based around the same basic technological underpinnings.

Despite these civilisations believing a recently (50,000 years) extinct civilisation built the gates, it turns out the technology is perhaps millions of years old.

Every 50,000 years, the synthetic AI race that built them returns from hiding in intergalactic space to wipe out all of the existing advanced species based on "their" technology, and reset the galaxy for the next set of civilisations to rise.

In a conversation between the game's protagonist and one of these old AIs, we are lambasted by the AI for taking the shortcut on their technology. The jump gates and other technology is left in place intentionally, so that each new generation of civilisations take a controlled and predictable development path, making it easier to destroy them.

The AI posits that it is the overcoming of adversity on your own that drives true technological advancement, and that easy routes make you (technologically) weak.

I think you can see something similar in the development of the different programming languages.

Java is long and wordy, taking a long time to type. The need to work around this limitation resulted in the proliferation of powerful IDEs, resulting in the annual 20 million line of code Eclipse release train.

PHP as a web language would have been stillborn if it didn't deal competently and quickly with the need to easily deploy code, the result of which is that you can effortlessly just change .html to .php, add a hello world tag, and upload via FTP as normal (something Perl still can't do well).

Python's need to gain mindshare against an entrenched Perl led to a huge focus on being easy to learn, to a simplification of the language, and to hugely popular things such as the PyGame library and game competitions.

Faced with the lack of truly great package repository, and with a web-heavy community, Ruby became the "prettiest" language. Creating an elegant website is both expected and required if you are going to gain mindshare for an idea.

And Perl's messy syntax and difficulties in the area of maintaining large codebases, combined with a pragmatic sysadmin-heavy community, resulted in an unmatched packaging system that allowed code to be maintained in small pieces, with enormous volumes of support infrastructure around it.

The ease of publishing and trend to smaller package that the CPAN allowed conversely means that the Perl community has never really had the need for pretty and elaborate websites, and the smaller package size means that we lack the giant headline libraries that make the payoff from website work better.

Our bias towards a pragmatic tech-savvy sysadmin userbase means we haven't really provided anything like the focus on learnability that has driven Python's gradual dominance in the mindshare of the young. It takes a certain rigour in your prioritisation to intentionally remove and dumb down existing powerful features so that the language is easier to learn.

Even for Strawberry, which focuses on the userbase with the lowest traditional knowledge, we intentionally have the smallest and most maintainable website possible and we don't even have the kind of introductory screencasts that we really really need (which should be easy but which I never seem to find the time to do).

If you throw a bunch of Perl coders against some PHP coders in a website competition, it is not unexpected that when both sides play to their strengths you will see something like http://geo2gov.com.au/html?location=e.g.+1+Oxford+Street from the Perl coders and something like http://www.hackdays.com/knowwhereyoulive/postcodes/view/2000 from the PHP coders.

The former required a massive amount of data extraction, transformation, aggregation, a gigabyte-sized PostGIS database, and deployment via a Linux virtual appliance to Amazon EC2 to allow for strategic load-shedding.

The latter required the ability to turn data into presentable, understand, information for real humans, and to make it pretty enough that they WANT to look at it.

Driving true technological progress, then, may often be about identifying weaknesses that are hard to solve but aren't completely impossible (and don't have any crippling long-term conceptual flaws at an economic or project-management level).

The three best projects I have driven - PPI, Strawberry, and (in part) Padre - all share this property. All three of these represent hard but not impossible problems, and require an awareness about which issues are intractable and which issues merely exist because there's been no need to solve them any better.

Padre in particular has suffered greatly from issues with Wx quality and threading. But given the low takeup of both threading and Wx it was reasonable to move forward on the basis that these would be fixed once there was something depending on them, and driving a need to fix them.

All of our early problems are gone now, and there is continued pressure to find ways to improve our use of (and the efficiency of) Perl's native ithreads.

Similarly, the creation of Strawberry required a lengthy year-long process of fixing Win32 bugs in all kinds of toolchain and low level modules, because we'd never had a proper working developer feedback loop before.

Similarly, Perl's current push for marketing and blogging and websites is directly resulting from Python's success in mindshare capture.

So my question for you to ponder this week is the following:

What can you see that Perl as a whole struggles to do well, for which a good solution is not impossible, and is only being held back by smaller problems which would go away if there was a working candidate solution put in place that needed those small problem solved.

In the spirit of trying to jam as many speed hacks into Padre as possible, I've finally taken it upon myself to take the awesome work demonstrated in ORLite::Array (which uses ARRAY based objects instead of HASH based objects) and moved it into the ORLite core.

The removal of the need for a hash slice in DBI doubles the speed of SELECT statements, reduces the memory cost of objects, and makes accessors quicker.

I've also integrated support for Class::XSAccessor, so now not only are they faster ARRAY objects (if you want them) the accessors are all XS-accelerated as well.

As a bonus to this I've altered the ORLite::Mirror sub-class to always use ARRAY-based objects by default, which means that all of the ORDB family of modules just instantly got doubled in speed without me needing to do any new releases.

So all up, Padre 0.56 is looking awesome.

To quote one person in the #padre channel, "padre is so fast to start now that the splashscreen is an irritation"

Now that the new resource locking system has been landed for a couple of weeks and some of our worst performance bugs have been resolved, we've been able to uncover and fix a ton of routine performance problems.

Padre 0.55 landed the first pass of these, but the upcoming Padre 0.56 is looking to be incredibly fast and has ditched much of the weight that you expect from an IDE. It's actually starting to feel light like an editor again, instead of heavy like an IDE.

Amoungst the improvements, we have a new tricksy startup mechanism that can apply startup preferences without having to load the user's config file at all (or any of the accompanying weight).

The startup process has become so fast that we're seriously considering ditching the splash screen (because it slows down the startup too much) and open files in an existing Padre has become nearly instantaneous.

Opening files is at least twice as fast thanks to an encoding detection shortcut for simple ascii files. Changing tabs is 5+ times faster due to removal of some filesystem operations and refresh locking. Closing a file is at least twice as fast, and 3-4 times faster if you choose to use the new feature preference to disable Padre's remembering of cursor locations in files.

Closing groups of files "Close This Project" and "Close Other Projects" etc has gained locking and now works incredibly quickly. With the cursor memory off the time between clicking close and the Padre window vanishing is almost instantaneous now.

Finally, we've also added a number of new refresh shortcutting to different tools and widgets, which makes Padre in general much snappier when moving around inside the same project or just doing operations that trigger off refresh cascades within the same file.

The sum total of all these improvements is that Padre feels almost like a whole new editor. It's fresh, snappy, and generally a joy to use.

And even more fancy improvements are in the pipeline. Mattias and Steffen between them have spawned a special new "slave master" threading branch which is already able to save 4-5meg of RAM per background thread by spawning of a master thread very early in the Wx startup process, while our list of loaded modules is very small. As a bonus, because it doesn't have to fork the entire Wx application tree, the slave master branch also fixes the long-time "Leaked Scalar" bug.

0.56 is definitely shaping up as something special and when it comes out I highly commend it to everyone to take a look, especially if you haven't had a look at Padre in the last 10 or so releases.

Ricardo Signes (RJBS) has just announced the release of perl 5.11.4, the latest development version on the way to perl 5.12. Of much less significane but the Padre team has also released version 0.55 of Padre, you know, the Perl IDE. You can read about it in the announcement of Peter Lavender.

In order to make it easier for people to try those I built a new experimental version of the Stand Alone Padre for Linux.

It contains

• perl 5.11.4
• Padre 0.55
• Padre::Plugin::PerlTidy
• Padre::Plugin::PerlCritic
• Padre::Plugin::Plack

During the build I encountered only one issue. A line in Pod::POM generates a warning about defined %hash being deprecated. As Parse::ErrorString::Perl, one of the dependencies of Padre is checking for errors using Test::NoWarnings, this module now fails to build. I had to manually patch Pod::POM in order to allow the installation of Parse::ErrorString::Perl.

In additon two major plugins of Padre failed to install:

Padre::Plugin::Perl6 has a deep dependency on YAML::LibYAML which fails and Padre::Plugin::Catalyst depends on Devel::Caller which currently fails on perl 5.11.4.

If you'd like to try it download perl-5.11.4-xl-0.03.tar.bz2 (29,804,273 bytes). Unzip it using

  tar xjf perl-5.11.4-xl-0.03.tar.bz2

and run the padre.sh:

  ./perl-5.11.4-xl-0.03/perl/bin/padre.sh

Enjoy and report any issues!

Yesterday someone called me for whom I did some work 7 years ago that he is now at a new company and need some quick Perl script written. A 2 days long work. As I was just heading home from a conference and I was close to their office we decided I go over so we can talk about it.

They have a system that provides its results in a CSV file but they have a client that needed the result in an XML file with some extra processing.

So the project involved processing a CSV file and based on a small XML configuration file the script was supposed to generate an XML output file.

As the system of the client runs on Windows this had to work there.

After looking at the actual project we decided I'll start to work on it immediately at their offices.

They put me in front of a Windows machine. I downloaded and installed the Stand Alone Padre editor and had almost everything I needed. As it has a version of Strawberry Perl for Windows underneath it already had XML::LibXML for XML processing.

I tried to use it in order to process the configuration file but I have never used that module and I felt it would be faster to install and use XML::Simple for reading the small configuration file. The already configured cpan client allowed me to type

   c:> cpan XML::Simple

and after a few seconds I already had the module. When I got to the point of processing the CSV file I probably could get away with a simple split() statement on every row but I did not want to take the risk of having a field with quotation marks and a comma inside a value breaking the code. So I opted for [dits://Text::CSV_XS]. I had to install it but it was easy:

  c:> cpan Text::CSV_XS

After about 5 hours total work including the initial discussion and finding a computer for me that took about 2 hours I was done with the script.

Then comes the hard part.

The actual production system does not have access to the Internet so one cannot simply use the cpan client. They could use a VPN to copy files to the machine just could not let the cpan client fetch the modules.

Luckily the solution was not difficult. I told them to copy Strawberry Perl to the system and the source code version (.tag.gz files) of the two modules to the production machine. After installing Strawberry Perl they could use

  c:> pip Text-CSV_XS-0.70.tgz

to install the module.

Of course I was lucky as there were only two extra modules I needed and neither of them had dependencies that Strawberry did not have. In a more complex case I might need several other modules and as the person who will install this on the target system does not know any Perl s/he will only feel it as an unnecessary complex installation.

Solutions

A

It would be nice if there was a simple way to build a package from this. That could be the planned Strawberry Perl Professional that will include many other CPAN modules reducing the chance that I'll need extra modules.

B

It could be a way to build the MSI installer based on a released version of Strawberry Perl and several modules from CPAN. This should be very easy that does not need any extra things beyond Strawberry Perl and the specific extra modules.

C

It could be taking the Strawberry Perl MSI package and the source versions of the required extra modules and zipping them together. On the target system the sysadmin could

1. unzip the file
2. install Strawberry using the .MSI package
3. Run a script that will install the extra modules that were supplied with the package

The last item needs some thinking so the script will either make sure that the dependencies are installed first or will let the user configure the local directory holding the extra files as the cpan server where pip needs to look for dependencies.

Thinking about this, this might be built by using CPAN::Mini we just need to configure the cpan client to look there first.

FOSDEM will take place in less than a month and the Perl stand is still waiting to be filled with interesting projects. I'am going to be at FOSDEM and I am going to give two talks. One of is 15 minute long lightning talk about Padre, the Perl IDE under the title Building an open source team, getting the project to users against the odds.

The other one is going to be in the distribution mini-conference about CPAN, packaging on CPAN and how CPAN authors and downstream packagers of the distributions (e.g. Debian, Ubuntu, Mandriva, Fedora, SuSE, etc...) can interact better. For this second talk I have 45 minutes.

So when people will come to the Perl stand Padre will be represented quite well. What about your project? Will you come to FOSDEM to show it? Is there anyone in your project who might be coming and could show it?

Even if none of the project members will come, the people who are going to be at the Perl stand will be able to show your project if you help us prepare the material. You don't have much time though as I am going to be off-line the last week of January so basically you have 10 days now to get the initial material ready.

What we will need is a tool we can use to demonstrate your project on our notebooks. We will need and easy way to install it - maybe its own live CD, maybe adding it to the live DVD Thomas Fahle created.

Remember there are going to be some 3-4000 people attending FOSDEM, but thet are not Perl programmers. I am not sure what exactly to expect. I am sure people will ask questions about Perl 6 and they will want to know if Perl can be used to create web applications. So the material should be ineteresting enough for them to look at your project or to look at Perl and try to get them start to use it.

So, if you'd like some publicity to you project. Start talking to me now about it!

Yesterday while I was not watching the IRC channel the issue of moving to Git came up again. You can see the log of the conversation. I know many Perl projects moved to Git, including the Perl 5 Porters. Maybe it is time for Padre too, to move?

I have to tell you people I really am getting fed up with the Git fanboyism. This time it wasn't so bad - probably because I did not see it live happening - but for a long time I wanted to write about this already. Git might have a huge technical advantage over Subversion and as we are told it also has an enormous social advantage but if I am pushed (no pun intended) to use Git then I feel very uneasy. To say the least.

I don't know what is your reaction to sales people that are trying to push you into buying things. Usually I kick them out even if I need that thing and will try to buy somewhere else. That means more time invested and sometimes even higher price. I know it is not rational but rather emotional. I am ok with that. I know most of the people reading this will make only rational decisions after carfully weighting all the technical aspects (e.g. of Git and Subversion) but I let emotions get involve as well.

I allow myself doing this as I learned in my marketing studies that most people actually make their decisions based on emotions and then rationalize them. As we were told, it is not surprising that a Porsche comes with a huge manual. After all you don't buy it to pick up chiks. You buy it because of its technical superiority. Or so you tell your friends and yourself!

Well, maybe it was something else we learned. I don't know. I almost failed that class.

Anyway back to

Padre and Git

I have been using Subversion for many many years now, since before it was release as 1.0 and I have been using Git for some time. For a long time I did not get anything about Git. The first thing I liked about it was actually Github. Specifically the ease I could create one-off patches. Even more so to create several small patches to other projects.

If a project is using Subversion I can pull it repository make a small change and send the patch. That's not bad but it requires an off-the-band communication via e-mail or nopaste or putting it in the bug tracking system or some other way to send the patch. The author also has to deal with pulling the patch from that place - and the fact that these places vary already create a head-ache - in short it is more complex than I'd want.

It is much worse in case I want to make several small patches that might depend on each other. I can't easily do it with subversion. I could have done it with SVK or Git, creating a local branch of the remote SVN repository, making several commits locally and then sending the patches one by one.

The other way would be to ask the reposioty owner to give me a commit bit. Even if s/he gives out commit bits easily as happened with Pugs and as happens with Padre it still takes some time to find the person who can give that commit bit. If I just want to make a few small changes the overhead might be too high.

Github

That's where Git+Github actually has an advantage. I - as someone outside of the project does not need to wait for anyone. I can just fork, make changes and push it back to Github. The maintainers of the project can pull my changes later on. This can work very nicely for small patches though for bigger changes people should still talk to each other first.

I don't think it would make much sense for Padre to keep the core developers work on their own fork as with the speed of our development we could easily create integration problems but we are told this only the fear of the unenlightened.

I can see an advantage of moving to Git and Github but I have my fears from the switch. Besides, the biggest disadvantage I see is that many of our developers don't not know Git - including myself. Actually I know several of them did not know even SVN before they joined the project. The main issue Adam Kennedy mentioned regarding Git was the lack of good Git client on Windows. I don't know the state of the Git GUI client market on Windows so it might be just his totally unaccepted emotional reaction (see above) to the pushing but I can't talk in his name.

Anyway I'd be ok moving to Git and Github and keeping a central repository on Github for the core developers and enjoying the benefits of Github for one-off changes but I have a precondition. In order to make that happen we have to make sure our developers can easily use Git so we need a superb Git plugin for Padre with Github integration that works on all operating systems we support.

So take it as a challenge. Improve the Git plugin to the point that it is super easy to use it even without knowing much about Git and you removed the biggest obstacles from our way to Git.

In 2008 the Padre project was about building a large development team, with the actual writing of the editor being almost a secondary task (albeit an important one). We knew the actual coding job was utterly immense and too hard for just the few people initially involved to deal with.

A lot of the work on heaviest work on the editor in 2008 was on deep subsystem stuff so that we could more easily recruit more people, like the Config/DB API, the Plugin API, the Locale API, and the basic distribution structure and code layout. And in the process we made enough of the actual editor that people could Understand what we were building.

In 2009 the Padre project took that team building success and built the editor into something we can now honestly call an IDE without our inner pedants cringing.

There are still various problems to solve, lots of rewriting of shitty first implementations to do, and "essential" features to add. An IDE is an Open Problem, and so that is never going to change and there will always be More Things To Do.

But I think it's reasonable to say that we've now built an IDE that the Perl community can call its own, and an IDE that you won't want to leave once you get used to some of the ways in which it excels at Perl development.

In another few months, we should be at something we can reasonable use as a part of the default Strawberry install for the upcoming all-in-one Strawberry Professional distribution.

However, for the most part Padre still lives in a mythical fantasy world where there is only one developer, only one host, and only one desktop. It is still, primarily, a tool for the individual developer. And now we're finally in a position to do something about it.

In 2010 the Padre project will evolve from an IDE into a Collaborative IDE. This isn't something that will happen by fiat. Nothing in Padre happens by fiat, and there is no roadmap.

But if you use Padre on a regular basis, you can see that the place where the most development time is being "wasted" now has now clearly moved from working with Perl code to working with projects, teams, and the internet.

In 2010 you will start to see some of the following:

1. The rise of Policy

One of the biggest themes in Padre is the concept of "Intuition", that your editor should recognise (reliably) what you are doing and adapt to it without having to be told, and in a way that doesn't remove freedom in the process.

Padre already can recognise projects without being told where they are, identify the files and assets in a project without being told where they are, and save files without you saying where. It knows about tabs vs spaces warfare, it knows what language you speak, and it knows how to distinguish things you personally prefer from things you happen to be doing on the current machine.

In 2010 Padre will learn about when it is that the needs of the team outweigh the needs of the individual, and how to selectively apply preferences and intuition differently in different open files, based on the specific preferences of different projects.

You will be able to define, at a project level, rules for your project that Padre will automatically follow, such as spaces vs tabs, perltidy rules, perlcritic preferences, and so on.

This is especially important for making Padre more relevant in corporate environments, where adherence to coding policies and standards are essential.

2. The demise of the host

Even though it hasn't been used to any great effect as yet, Padre has always understood that some configuration is about the user, and some is about the system.

In 2010 Padre will finally start to use that information. A 4th year student team from the University of Western Ontario is currently implementing a synchronisation client/server for configuration data, similar to Mozilla Weave or Google's older bookmark sync.

This will allow all your personal preferences to migrate across multiple installations automatically, without getting mixed up with data that only refers to the machine you are currently working on.

3. The push towards Portable

Another area that I hope to see movement on this year are configuration and GUI improvements to make Padre much more tolerant to changes in the underlying operating system.

While many applications do not support this, I'd like to see Padre automatically adapt to changes in screen position and resolution, mounting and dismounting of drives, and a variety of other crazyness that developers often do in the process of creating applications.

This tolerance and adaptation to screen geometry errors should also allow better configuration of windows and dialog positioning, so that you can more easily tailor the layout of Padre to your particular multi-screen developer setup.

4. The swarming of Collaboration

Finally, the experimental Swarm plugin is (after a long gestation) starting to get closer to producing a usable collaborative communications subsystem that can be used as the basis for chat, file sharing, remote peer review, and other collaborative goodies.

In 2010 Padre should start to take advantage of some of these new capabilities to add new and interesting real-time "Intuitive Collaboration" abilities (which should be quite interesting at conferences and hack-fests).

In summary, it's looking like a great year for Padre, and by the end of the year Padre should hopefully be starting to become the corporate Perl editor of choice.

Adam Kennedy have outlined his plans for Padre for 2010. I don't have such long term plans. Being inpatient in this regard I am usually very bad at projects that take a lot of time to show any signs of life. In a way the Release soon, release often mantra of the open source world very much fits me.

Anyway, there are some Padre related news here:

Padre on ActivePerl on Windows

First of all thanks to Mark Dootson Padre 0.53 is now packaged for ActivePerl on Windows. In order to install it check the instructions on his ppm repository.

As he told me the padre.exe does not work there but the padre.bat does. I am not sure if this is related to the bug that Padre Stand Alone does not start first time after installation. or if it is some other issue but it is better to be awre of it.

If you are using ActivePerl on Windows now is your time to try it and let us know of any issues.

Padre on FOSDEM

FOSDEM is just a month away. My lightning talk proposal about Padre was accepted. That means I'll have 15 minutes to talk about it.

As you might have read we are also going to have a Perl stand there for which I am looking for people to help. We need to make sure there are always going to be at least 1 but better 2 people at the stand to talk to visitors. Explain about projects. Explain where Perl stands now. What is Perl 6 and when will it be ready. What is Moose. What is Catalyst. You know, there are going to be lots of questions.

On FOSDEM we will have to be prepared on how to talk to people who are very intrested in Open Source and programming but has little - in the good case - or wrong information - in the bad case - about Perl.

Padre on CeBIT

This is not really any Padre issue though Padre will also be represented. Our applicaton for a free Perl booth was accepted by the organizers of CeBIT. That means, a month after FOSDEM we have another event - no talks here just a booth - where we can show Perl and Perl related projects to visitors.

This is going to be entirely different from a Perl conference or from FOSDEM as CeBIT and its visitors are really far from the open source world. If on FOSDEM we will have to talk to people who understand our environemnt in general just not our language than on CeBIT we will talk to people many of whom don't know or care about open source or programming at all. They really just want the business value in our tools and products.

• i finally took the time to investigate moose, and decided to port some of my code to use it.
• i discovered dist::zilla, and definitely plan to use it in all my dists.
• i discovered some other cool new perl modules (thanks guys).
  
• i uploaded some new dists on cpan, including games::pandemic (which i'm quite proud of, even if i need to continue hacking on it).
• i updated & released some of my existing dists.
  
• i contributed to other projects such as padre, dist-zilla, etc. sending patches is a good way to thank the author for their work.
• it's even easier to contribute with github, which i'm using more and more for my projects (and it's no more slow by now, woohoo!). of course, git is the master piece allowing this easy sharing.
  
• on mandriva's front, i resurrected parrot rpm, and finally shipped rakudo.
  
• 2009 was also the year of the great migration to %perl_convert_version
• ... with lots of new perl modules available as rpm (i am managing 450 of them).
  
• ... and finally i took ownership of perl rpm. (thank you perl5 porters btw for 5.10.1 and the push for next stable version!)

http://padre.perlide.org/trac/ticket/1

Our Christmas release of Padre 0.53 is done now, and it's looking really really awesome.

My contribution to this release is a shiny new resource locking system, which has significantly improved the speed of most startup, shutdown and file operations.

A couple of people have asked me to post about how it works, so here's the short and simple version.

First, some background on resources in Padre and Wx.

When managing performance and blocking issues in Wx, the main players are the visibility state (changed via ->Show and ->Hide), the update state (changed via ->Freeze and ->Thaw) and the busy state (changed via various mechanisms).

Visibility is most important during startup and shutdown.

When starting up you want to delay the appearance of the window until the event loop is bootstrapped and you are able to take user events on the window, but you can't wait for things like opening files because this may take quite a long time and cause the editor to appear slow and "bloaty". It's a trade off between performance, and PERCEIVED performance (which is almost as important).

When shutting down, things are a bit clearer. As soon as you are sure that the editor will no longer need to interact with the user in any way, you can proactively ->Hide the window and finish shutting down while invisible (letting the user get on with their next task).

The update state is applicable across the entire lifetime of the application.

If updates are enabled, every action results in a paint event (or will be captured by the next paint event). This creates the appearance of things happening, but changes to the application take longer because of the cost of repainting.

Also, often you don't want to be faster in this way. If you are working with a list box, during the process of deleting all the values and generating a new (slightly different) set of values you don't WANT the user to see the box empty and incrementally refill. It acts a form of "flicker" (one of the great enemies of GUI applications). What you want is for the list box to instantly transition from one filled state to the next filled state creating the impression of incremental change even where the underlying code isn't incremental.

The way you solve this problem in Wx is to explicitly disable repainting via the ->Freeze method as late as possible, quickly make your changes to the GUI structure, and then as quickly as possible re-enable painting via ->Thaw.

This must be fast, because if the users are doing anything they will notice after a quarter to half of a second. Also, after 2-10 seconds in some cases the operating system will start to get concerned about your application. Windows for one will sometimes spontaneously ask Wx to repaint your menu after 5-10 seconds. If painting is disabled, the result is a blank white bar where your menu used to be.

Finally, for Padre specifically, we also need to be concerned about expensive refresh operations. If you open a file, we need to regenerate the directory tree, the function list, outline, menu, toolbar, title, recent files list, and directory list and some of those things (like the directory tree) are expensive.

We need to be sure that we avoid updating elements that don't need to be updated (opening a new file in the currently project context shouldn't result in a project file list refresh, for example) and that we avoid or delay any pointless operations (like updating the GUI during a multi-file open) until after the last file is done.

Unfortunately, the update and refresh issues are problematic when most of your application is event-driven, but all your refresh operations are imperative.

When you first start writing Wx code, you usually do something like this.

    sub open_file {
            my $self = shift;
            my $file = shift;

            $self->Freeze;

            my $panel = $self->create_editor;
            $panel->load_file($file);
            # 10 more lines of setup code here...

            $self->refresh_all;

            $self->Thaw;
    }

This kind of thing works, but it will only work if called directly from the user event and only if it doesn't throw an exception.

What if $panel->load_file errors due to file permissions, and we don't get a chance to call ->Thaw?

Or what if we want to open several files?

sub open_files {
    my $self = shift;
    foreach my $file ( @_ ) {
            $self->open_file($file);
    }
}

In this case, open_files will freeze and thaw repeatedly, slowing down the loading process, and will pointless refresh all the GUI resources after each file.

Or what if we want to open a named group of files (a "Session").

sub open_session {
    my $self = shift;
    my $name = shift;
    my @files = $self->get_session($name);
    $self->close_all;
    $self->open_files(@files);
}

Now we are at two levels of indirection.

Or we want to have a ->next_session method (three), or to support arbitrary scripted "macro" actions contributed by the user (four)?

The real problem with imperative management of these resources is that they don't encapsulate cleanly. You can't nest them inside each other ad infinitum and have code calling your not have to care how you are implemented.

Wx takes care of part of this problem for you, and hints are the right general solution.

    my $guard_object = Wx::WindowUpdateLocker->new($window);


This built in "locker" class generates objects that fire ->Freeze on the parameter window at constructor time, and ->Thaw at destructor time (i.e. when they fall out of scope).

Importantly, these objects also nest properly so that the ->Thaw only occurs on the outermost destructor. This provides the encapsulation we need so badly.

Unfortunately, the ORLite-based SQLite database classes in Padre::DB we rely on quite heavily don't have this ability (and Wx native asynchronous SQLite support isn't available yet).

Worse, our resource refresh logic doesn't follow this pattern at all. Often you can't just "lock" the refreshing of the whole window or the directory list, because you have very specific times that you want the refresh to fire.

For example, if the user has two files open and is switching between one and the other, we want to initially lock updates while we change the editor panel to the new file but we DON'T want to refresh the other tools. Once we release the update lock to show the new file, we know it will take around a second for the user to notice and adjust to what they are seeing. So we want to use THAT time to refresh the other tools, applying another update lock during the changes to prevent screen flicker and avoid distracting the user with unexpected secondary movement.

To resolve all these problems, I've added a new locking API based on a similar guard object principle to the internal Wx one, except that it is able to control several locking states at the same time. More importantly it's also able to understand the interplay between the different lock types.

The canonical usage looks something like this.

    my $lock = Padre::Current->main->lock('DB', 'UPDATE', 'refresh_menu', 'refresh_title');

This indicates that we should "lock" changes to the SQLite database (causing all database operations to take place in a transaction), lock painting updates, and that some time in the future we'll need to do a refresh of the menu structure and the window title.

To simplify and add extensibility to the implementation, the lowercase refresh locks all match directly to methods on the main window class. So anyone can add a new "lock" type just by adding a refresh_something method.

When the lock handle expires, the lock manager follows a specific release plan. First, release the update lock. Second, fire any refresh events that have accumulated. Thirdly, commit any pending database statements and release our database connection (again, taking advantage of the user needing a small amount of time between screen updates and their next action).

Where the value of a custom lock manager comes in is the Padre-specific semantics.

For example, locks on the database and update state don't interact with each other, but refresh method locks persist beyond the scope of the lock if the release occurs within a higher parent lock.

Take the following example.

SCOPE: {
        my $lock1 = $main->lock('UPDATE', 'refresh_menu');
        SCOPE: {
                my $lock2 = $main->lock('refresh_directory');
        }
        SCOPE: {
                my $lock3 = $main->lock('DB');
        }
}

In this example, the database lock will clear at the end of the nested scope, and a commit will occur. However, the 'refresh_directory' method will NOT fire in the nested scope because there are other active related locks in effect in a parent scope. Instead, the 'refresh_directory' will be transfered up to the higher lock.

When $lock1 releases, it will fire BOTH of the refresh events.

This is a basic implementation of the concept (less than 100 lines) but it does work quite nicely. However, it does have some problems remaining to be solved in the next version.

The syntax is still a bit clunky. To fire a refresh event immediately (if you are at the top level) you still need to do the following.

$main->lock('refresh');


This will do an immediate ->refresh, but remains lock-aware so that if you are in a higher level lock it will add refresh to the list of things to fire later. It would be nice to have a cleaner syntax for this case, or do be able to have the ->refresh method itself inherently know if it is in a lock and just shortcut return true.

Another problem is that the refresh methods come in a natural heirachy that the lock manager doesn't know about.

For example, the top level ->refresh will itself call ->refresh_menu, ->refresh_title and so on. But the lock manager isn't aware of that.

If two nested scopes ask for 'refresh' and 'refresh_menu' locks it will always fire both refresh methods, even those doing the first one means we don't need the second one.

And it won't fire the locks in a reliable order either. Even if ->refresh was smart enough to automatically clear the 'refresh_menu' lock, we can't be sure that ->refresh_menu won't fire first, before the higher one.

This problem discourages the creation of finer-grained locks, because it would increase the number of collisions and pointless double/triple/etc refreshing.

By discouraging finer locks we end up using bigger locks more often, resulting in more waste due to pointless refreshing of GUI elements.

Ideally, the locker would contain information on the relationships between refresh events, so that the refresh events can be fired in the most "correct" order, with pointless methods filtered out.

As Peter Lavender has announced Padre 0.53 was released a few hours ago. Among many other things it contains the first version of a built-in debugger. You can download and install Padre from CPAN but if your are using Linux there is also a new binary package that you can download and use.

It is a very experimental package, not only because of Padre but because it is using perl 5.11.3 the recently released monthly build of perl 5.12 to be.

While building it I encountered a few problems related to changes made to perl. These changes might affect your code as well. You might want to try your modules to see how they behave on 5.11.3.

One of the modules was generating warnings like this:

   defined(%hash) is deprecated at .../SomeFile.pm line 82.
   

Another one was generating warnings like this:

  UNIVERSAL->import is deprecated and will be removed in a future perl at .../SomeFile.pm line 7
  

They passed their tests but Padre, during its own tests checks if there are not unexpected warnings using Test::NoWarnings and fails if there are. So I had to patch those modules in order to allow the installation to proceed. ( Reports were sent to the authors so I hope these will be fixed soon ).

Test::Exception failed one of its test. I force installed the module and reported to the author and to p5p where I quickly got a response from David Golden pointing out that there is already a fixed developer version on CPAN.

I also had to patch one of the tests of Padre due to a change in a warning text perl 5.11.3 gives. The same change was then committed to the version control of Padre.

It was a bit annoying to hunt down the issues as my build system does not report correctly yet but in the end I got it working. When I tried to install the Perl 6 and the Catalyst plugins I encountered further issues so I decided to postpone them.

If you want to try it now, look at the instructions:

Download

Download perl-5.11.3-xl-0.03.tar.gz (File size: 29,002,849)

  wget http://perlide.org/download/binary/perl-5.11.3-xl-0.03.tar.gz

Unzip it

  tar xzf  perl-5.11.3-xl-0.03.tar.gz
  

and You can run the padre.sh file:

  ./perl-5.11.3-xl-0.03/perl/bin/padre.sh 

Enjoy and report any issues.

Devel::Declare vexes me greatly.

On one hand, it's interesting to see the kinds of things you can do when you have the ability to temporarily take over parsing of the language.

On the other hand, this ability to just parse whatever the hell you want means the gradual decline in utility for the entire Perl development toolset. It's like someone has taken source filters (which everyone at least KNEW were "evil") and made them popular and "safe".

The real problem here is not that these problems are occurring, but they are occurring for what often seems like (or actually is) a very good reason. And as anyone who has seen my "Nothing Can Possibly Go Wrong" talk should know, the most dangerous failures are the ones we are seduced into because it seems like such a good idea.

For example, take this cool implementation of URLs via Devel::Declare.

Mixed that with LWP::Simple and you can do things like this, that actually run.

my $content = GET http://ali.as/;

Or how about inline SQL, as featured in a couple of other languages. I'm fairly certain that by adding hooks for, say, SELECT, INSERT, UPDATE and DELETE I could easily make the following work.

my $arrayref_of_hashes = SELECT * FROM table_name;


This stuff is supremely shiny, and if you're anything like me as soon as you look at it you want it badly.

But these features aren't free.

In the general case (looking at ARBITRARY modules rather than specific modules) you lose the following.

1. No/degraded syntax highlighting.
2. No/degraded PPI parsing.
3. No/degraded shiny Padre refactoring tools.

You also lose support for things like Perl::Critic almost entirely. PPI's statement classifier drives a lot of critic functionality. It only exists because PPI can reliably find the ends of statements.

If you break the ability to find the end of a statement (as many D:D modules do to define their own block keywords like "class { }") then the parse tree starts to get really really whacky and critic will inevitably start to spit out false positives.

Once you start spewing false positives in enough volume, you undermine confidence in the tool and we might as well stop using it (or at least turn off entirely rule subsets).

There are proposals for ways to extend PPI so somehow support these new keywords (I've even agreed with Matt on how we would go about it) but I continue to suspect they don't work in the general case because you have to have installed the PPI extensions in advance, before you attempt to parse a document containing those D:D modules (or you allow arbitrary execution of user code, at which point Acme::BadExample will complete pwn your machine).

This process gradually grinds onwards. D:D gets more mature and more used, Perl 5.12 contains core enhancements that help make it work better, further legitimising the concept.

Each step along the way is completely rational, and completely defensible. And it all looks great in blog code snippits. To paraphrase one prominent Perl core dev "Should we stop making the language better because it breaks PPI?"

And yet, at the risk of bringing down Godwin's Law on myself, I can help but see the kind of effect described in the famous poem First They Came.

First they came for the communists, and I did not speak out—because I was not a communist;
Then they came for the trade unionists, and I did not speak out—because I was not a trade unionist;
Then they came for the Jews, and I did not speak out—because I was not a Jew;
Then they came for me—and there was no one left to speak out for me.

The difficulty level to modify the parser and break the development tools is already starting to move from "a large effort by experts" to "check out this neat URL hack".

At some point, the use of D:D-style parser hacks becomes routine and normal, and people start implementing new extensions just for their one specific project.

And at some point the primary interface to Moose or Catalyst is using D:D or something like it, because it's SO much more elegant. And so any code written with Moose isn't compatible with Perl Critic, and so on and so forth.

Some time soon, before Perl turns into Smalltalk (for which there are basically NO tools at all that don't also involve running the code), I think we need to have this discussion.

How many useful development tools are we willing to sacrifice (making development harder) in order to make development "easier".

I am sorry for the delay in the announcement but I am having a busy time wrestling with both clients and Padre as I would like to make sure the debugger already works in the next version. Anyway the next meeting of Rehovot Perl Mongers will take place on 22 December, between 18:00-22:00 in the Weizmann Institute. For directions take a look at the web site of the Rehovot Perl Mongers.

On this meeting Hadar Levi will give another talk, this time about PDL::Indexing explaining the basics of indexing and threading in PDL, The Perl Data Language

I think I'll use some time to give you a deeper introduction to Padre. I'll plan to look at

1. How the debugger was added (and by that time it should be past tense)
2. How to add a simple plugin.

If there is interest - as I'll ask on the spot - I'll give a short introduction to Perl. Either to everyone or to the smaller group who are new(ish) to Perl.

After that we will have exercises bot to beginners and more advanced users.

Please RSVP to my e-mail address if you are planning to come.

Please also tell other parties about this meeting and also urge them to sign up to either of our mailing lists that are relevant to them Rehovot.pm mailing list or Israel.pm mailing list