As previously mentioned in this blog, our team is busy rewriting a complex application from Cobol to Perl.
While studying the old Cobol code, I often discover some areas where the initial design was well-thought and well-organized, but later became blurred and confused over 20 years of maintenance. So part of our analysis task is to do some archeology, trying to understand the historical layers, and to sort out what is still relevant to the current needs of our users.
Unfortunately, the same phenomenon also starts appearing in the Perl code ! Some parts were written two years ago, and then had to evolve for various reasons ... and sometimes the initial design becomes blurred in this process.
One may think that when this happens, it is because the initial design did not have the proper abstraction / parameterization hooks to make it easy to extend. But sometimes when doing the initial design of a component, you don't have a complete picture yet of what is going to surround that component ... or the requirements may have changed because this is a long-term project, and life doesn't stop while we are working on this application. So what is really needed to keep it clean is constant refactoring.
The problem is that maintenance operations do not have the same metrics : maintainers are evaluated by how many tickets they solved and how long it took; so there is a natural tendency to just "get it to work". Spending additional time on refactoring operations brings no immediate reward : users won't see a change, managers won't understand why you need to revisit code that was already written, and there is an additional risk of introducing regressions.
It's easy to understand that on a collective level there would be some rewards on the long-term (better maintainability, cleaner architecture, etc.) ... but on the long-term the maintainer will probably have gone to another project !
Thursday, October 29, 2009
Saturday, October 17, 2009
how to reconcile audits and agile development ?
According to recommendations recently emitted by the Swiss working group for IT government audit (Swiss chapter of ISACA, international Information Systems Audit and Control Association), every important IT project in Swiss government should have at least 10 documents ready for the auditor (among about a hundred kinds of documents defined by the Swiss project management method Hermes ) :
1. Feasibility study
2. Specifications
3. Cost effectiveness analysis
4. Integration into the IT environment
5. Requirements
6. Concept for an internal control system (ICS)
7. System architecture
8. Tests (test plan and documentation)
9. Acceptance by the user
10. Final assessment
1. Feasibility study
2. Specifications
3. Cost effectiveness analysis
4. Integration into the IT environment
5. Requirements
6. Concept for an internal control system (ICS)
7. System architecture
8. Tests (test plan and documentation)
9. Acceptance by the user
10. Final assessment
The recommendations explicitly insist that this list also applies to "new so-called 'agile' development methods".
For our Perl project at Geneva courts of law, this means that we must produce such documents to be ready for occasional auditors. The problem is, that the Hermes method was mainly inspired by good old waterfall development methods on mainframe computers, and some of the documents listed above just do not make much sense in our context; so instead of helping to better structure and organize the project, they just represent an additional burden.
For example, some parts of the application start in an exploratory way, without formal specifications, and are progressively shaped into working functionalities; tests are not planned in a document, but written in a galaxy of test files; etc.
I guess that the pressure for formal deliverables in project management is probably stronger in government than in private companies, but nevertheless people doing big Perl projects in any context probably also have at least some of such constraints. Any testimonies on that ?
Tuesday, October 6, 2009
I don't hate the stash ... but I loathe the session.
A couple of days ago, John Napiorkowski asked : Does Anyone Else Hate the Stash?
True enough, the stash in Perl Catalyst is a big bag in global memory. Any method along the chain can read or write into that bag. Then you pass the whole thing to the Template Toolkit (TT), that integrates the Catalyst stash into its own stash, and again any template fragment can read or write into the TT stash. So when studying one particular component along the chain, it is indeed sometimes hard to track what is in the stash at that point, and where the data came from. I frequently need to resort to the perl debugger to sort out such situations.
Nevertheless, I don't hate the stash, because it is sooo convenient to let various software components collaborate at little cost. Setting up a more controlled way of passing information between components would be quite tedious and would imply more maintenance. It is like when having several humans in a team : if collaboration is harmonious, it leverages some multiplicative power; if not, people start treading on each other's toes, and the global result is unsatisfactory. A simple step for ensuring harmonious collaboration is to partition the stash namespace through additional levels of hashrefs (same principle we use on CPAN for avoiding collisions between module names).
Furthermore, global memory in stash is not too risky because it is very limited in time : at the end of the request the whole stash is cleaned up. Unfortunately, there is something much worse than the stash : the session !
Some colleagues tend to like putting stuff into session storage, because it's easy to program sequences of requests without having to propagate state through URL parameters or JSON data. I try to avoid it as much as possible because :
True enough, the stash in Perl Catalyst is a big bag in global memory. Any method along the chain can read or write into that bag. Then you pass the whole thing to the Template Toolkit (TT), that integrates the Catalyst stash into its own stash, and again any template fragment can read or write into the TT stash. So when studying one particular component along the chain, it is indeed sometimes hard to track what is in the stash at that point, and where the data came from. I frequently need to resort to the perl debugger to sort out such situations.
Nevertheless, I don't hate the stash, because it is sooo convenient to let various software components collaborate at little cost. Setting up a more controlled way of passing information between components would be quite tedious and would imply more maintenance. It is like when having several humans in a team : if collaboration is harmonious, it leverages some multiplicative power; if not, people start treading on each other's toes, and the global result is unsatisfactory. A simple step for ensuring harmonious collaboration is to partition the stash namespace through additional levels of hashrefs (same principle we use on CPAN for avoiding collisions between module names).
Furthermore, global memory in stash is not too risky because it is very limited in time : at the end of the request the whole stash is cleaned up. Unfortunately, there is something much worse than the stash : the session !
Some colleagues tend to like putting stuff into session storage, because it's easy to program sequences of requests without having to propagate state through URL parameters or JSON data. I try to avoid it as much as possible because :
- data in session storage is likely to produce unwanted "action at distance"
- the URL API is no longer RESTful (calling the same URL with same parameters might yield different results)
- there is a cost in serializing / deserializing the session data at each request
- session data is limited in size
- so session data is not appropriate for storing recursive datastructures of unknown sizes
Programmers a tempted by the easy aspect of session data, but are not always conscious of the limitations above.
Saturday, October 3, 2009
YAPC presentation styles
One personal comment I got from the YAPC::EU::09 Survey results was : "Split slides so they contain less text". Well, while attending several talks, I felt exactly the reverse : I wished the speaker had condensed slides so they contain more text !
Finding the right balance is really a difficult question. It is true that I have a tendency to fill slides with a lot of material, in order to exploit complementary channels : while my voice gives the general idea, or emphasizes a particular point, the slide can convey more detailed information, and people in the audience can grab more content if they are especially interested in one particular aspect.
Probably I like this style because it corresponds to my own way of learning. When I was at school, at a time where beamers were rare and expensive, most teachers used physical transparencies. Some of them had the habit of putting the transparency and immediately hiding it with a piece of paper; then they would progressively uncover the slide, one point at a time. I hated that habit, because I was forced to think at the same speed as the teacher. If I see the global picture at once, I can immediately choose which points seem more important to me, and focus on them, maybe already preparing a question, or think back at what was said before, or anticipate what is probably going to be said next. But if the teacher dictates the rhythm, and decides to pause for 5 minutes on a point which is important to him, but not to me, or decides to quickly skip over a detail which I need to elaborate in my head, then I'm in trouble.
The modern way of uncovering slides one point at a time is the Takahashi style (lots of slides, very few words, huge font), which seems much praised in the Perl community. I must admit that I was quite impressed the first time I saw a presentation in this style : it is quite efficient for a lightning talk, or to create some suspense at a particular point in a presentation. However, if many speakers adopt this style just out of fashion, without deliberate thinking about which effect they want to achieve, then at the end of the day it becomes quite boring, and I feel like having watched several hours of videoclips. After such a day I don't really remember what were the highlights.
Thursday, September 24, 2009
Design patterns, or why Java needs external crutches
In my last blog about architecture and design I promised to come back on the topic of design patterns. This term was coined in a famous book that proposed a catalogue of "Elements of Reusable Object-Oriented Software", and became a best-seller in the Java world (usually referred to as the "GoF" book, for "Gang of Four" authors). Feeling the golden ore, all editors quickly produced a whole line of similar books, where patterns were adapted to various domains and languages : for C##, for Ruby, ..., etc. I almost expected to see patterns books for assembly code and for Befunge, but these never came out !
So what about patterns for Perl ? To my knowledge, no major editor published any book on that topic, which is kind of surprising because one would think that there is some money to make. However, Phil Crow privately published Perlish patterns (cheeply available as e-book), which is really worth reading; Phil proposes the following explanation in his introduction :Eventually, I came to understand that there were several reasons why patterns were never as enthusiastically embraced in our community as they were in others. Some of the patterns just apply new names to common techniques. Some are represented in Perl's core, so we don't think much about using them, at least in their normal forms. Some apply better to languages which focus on object orientation.
I very much agree with this analysis. The original GoF book was refreshing to read, but when programming in Perl I never think in terms of those patterns, because the standard language features plus some common CPAN modules answer most of my needs for structuring my programs, even when assembling large bodies of functionality.
The fact that Java is so verbose, and that everything has to be an object, results in code that is often spread among lots and lots of classes. So to condense that information, it is no surprise that Java programmers need other abstractions like "patterns", so that they can think in terms of larger units. For the same reason, they also need sophisticated tools like Eclipse for navigating through the class hierarchy, and they need costly tools like Rational Rose to see and design the big picture, and generate code skeletons. I'm always surprised to hear such tools presented as strenghts of the Java world, while to me they are just necessary consequences of the way Java code is layed out.
In standard Perl, we have hashrefs and arrayrefs, we have closures, multiple inheritance, namespace manipulation primitives, dynamic classes and dynamic methods, functional grep, map and other List::MoreUtils goodness; we can assemble those into dispatch tables, delegation structures, function and method templates and factories ... enough patterns to fill a whole life !
Monday, September 21, 2009
hit by operator precedence and right associativity
While studying a bug, I wrote the following test program :
use strict;
use warnings;
use Data::Dumper;
my $bool = 1;
my %h;
$bool ? $h{true} = 't' : $h{false} = 'f';
print Dumper(\%h);
The ternary expression starting with $bool was supposed to be a concise way to write a conditional, but the result was a disaster. Can you guess the output ?
Here it is : $VAR1 = { 'true' => 'f' };
This really seems totally insane : something is assigned to the 'true' slot of the hash, but the value comes from what was supposed to be in the 'false' slot !
OK, the ternary expression above is wrong, because the '?:' operator has higher precedence than '=', so one should really write
$bool ? ($h{true} = 't') : ($h{false} = 'f');
But how comes that perl issues no error, no warning, and happily produces a very strange result ? It seems that both sides of the conditional are executed simultaneously, and collapse in a mysterious way.
I tried running the script through B::Deparse to understand how it was parsed, but the output was exactly like the original source, so it really seems to be legal Perl !
It really took me a while until the 'aha' moment that made me realize that because of right-associativity, and because conditional expressions can be lvalues, and because "Unlike in C, the scalar assignment operator produces a valid lvalue" (perlop dixit), this was parsed as
($bool ? ($h{true} = 't') : $h{false}) = 'f';
So the $bool test chooses an lvalue between $h{true} and $h{false}, and it doesn't matter that this lvalue is first assigned a 't', because later the main assignment puts a 'f' into it.
Obvious, isn't it ?
use strict;
use warnings;
use Data::Dumper;
my $bool = 1;
my %h;
$bool ? $h{true} = 't' : $h{false} = 'f';
print Dumper(\%h);
The ternary expression starting with $bool was supposed to be a concise way to write a conditional, but the result was a disaster. Can you guess the output ?
Here it is : $VAR1 = { 'true' => 'f' };
This really seems totally insane : something is assigned to the 'true' slot of the hash, but the value comes from what was supposed to be in the 'false' slot !
OK, the ternary expression above is wrong, because the '?:' operator has higher precedence than '=', so one should really write
$bool ? ($h{true} = 't') : ($h{false} = 'f');
But how comes that perl issues no error, no warning, and happily produces a very strange result ? It seems that both sides of the conditional are executed simultaneously, and collapse in a mysterious way.
I tried running the script through B::Deparse to understand how it was parsed, but the output was exactly like the original source, so it really seems to be legal Perl !
It really took me a while until the 'aha' moment that made me realize that because of right-associativity, and because conditional expressions can be lvalues, and because "Unlike in C, the scalar assignment operator produces a valid lvalue" (perlop dixit), this was parsed as
($bool ? ($h{true} = 't') : $h{false}) = 'f';
So the $bool test chooses an lvalue between $h{true} and $h{false}, and it doesn't matter that this lvalue is first assigned a 't', because later the main assignment puts a 'f' into it.
Obvious, isn't it ?
Saturday, September 19, 2009
Learning architecture and design
Matt Trout asks about how people learned what they know about architecture and design.
As far as I am concerned, I've always been more or less interested in that subject, but only started to study it more seriously about ten years ago, when I left the academic research world and started writing real software instead of writing papers about software.
So where to go when one is interested in design ? One source of information is books. My sources are quite similar to the ones cited in Matt's article. Currently I'm reading Beautiful Architecture(actually I bought it at YAPC::EU::09), which I enjoy very much because the articles are of high quality and cover a vast territory. The previous book in that series Beautiful Code is also worth reading, although a bit less interesting in my taste. Of course I also read a couple of books about design patterns ... but I'll blog another time about those.
Despite the fact that they seem to sell well, books on design are are not numerous ... probably because they are so hard to write ! I mean, writing a regular technical book is hard; producing good designs is hard; so writing a good book that highlights the design process is necessarily even harder. Actually, books mentioned above are never an organized discourse starting at A and ending at Z; what they do is supply a catalog in which the reader can grab interesting ideas, and that's probably the best any book on design can do.
Which brings us to the point : books on design are nice, they open your mind, but that's seldom the place where you really learn. Design is acquired by practice, using or reading other people's designs, and then doing your own through trial and error. It reminds me of my counterpoint courses : although there are a few recipes, it's only after having studied a dozen Bach fugues and having painfully written two or three in the same style that one really understands what it means to design a fugue.
So I enjoy browsing through technical manuals and APIs of many components, even if I'll never use them. For example, it's quite instructive to study the difference between the object models of Microsoft Word and Microsoft Excel, two products in the same family, but with huge differences in design. Word is an infamous amalgamation of fuzzy concepts (anybody ever understood how automatic numbering works ?), while Excel is a beautiful join of many powerful features into a single orderly framework.
Here is a list of some designs that I considered particularly inspiring:
Thanks Matt for having asked this question about design, that was an opportunity to put some thoughts into shape, and I'm eager to read what other people will write on that topic.
As far as I am concerned, I've always been more or less interested in that subject, but only started to study it more seriously about ten years ago, when I left the academic research world and started writing real software instead of writing papers about software.
So where to go when one is interested in design ? One source of information is books. My sources are quite similar to the ones cited in Matt's article. Currently I'm reading Beautiful Architecture
Despite the fact that they seem to sell well, books on design are are not numerous ... probably because they are so hard to write ! I mean, writing a regular technical book is hard; producing good designs is hard; so writing a good book that highlights the design process is necessarily even harder. Actually, books mentioned above are never an organized discourse starting at A and ending at Z; what they do is supply a catalog in which the reader can grab interesting ideas, and that's probably the best any book on design can do.
Which brings us to the point : books on design are nice, they open your mind, but that's seldom the place where you really learn. Design is acquired by practice, using or reading other people's designs, and then doing your own through trial and error. It reminds me of my counterpoint courses : although there are a few recipes, it's only after having studied a dozen Bach fugues and having painfully written two or three in the same style that one really understands what it means to design a fugue.
So I enjoy browsing through technical manuals and APIs of many components, even if I'll never use them. For example, it's quite instructive to study the difference between the object models of Microsoft Word and Microsoft Excel, two products in the same family, but with huge differences in design. Word is an infamous amalgamation of fuzzy concepts (anybody ever understood how automatic numbering works ?), while Excel is a beautiful join of many powerful features into a single orderly framework.
Here is a list of some designs that I considered particularly inspiring:
- the NeXT operating system and programming environment had a nice language (Objective-C) and a beautiful OO architecture, with a generic notion of object inspector for editing object properties ... really a joy to use and program. Too bad all of this disappeared for lack of market penetration.
- when Netscape first proposed to integrate Javascript into Web pages so that they could become dynamic, I had a "wow" moment : this opened so many possibilities ! Besides, the documentation was extremely well-written (many years after, it's still a useful reference, especially the chapters on how to exploit prototype-based inheritance).
- in the same vein, I had another "wow" moment one or two years later when Internet Explorer first came with the notion of manipulating the DOM through scripting (initial versions of Javascript in Netscape could not do that). Again, this opened a whole new world, and the API was quite clean and very well documented (not respecting standards is another story). By the way, at that time the MSDN library site was really cool, with support for keyboard arrow command while navigating through the documentation tree --- later on they moved to .NET technology, and were no longer able to support keyboard navigation !
- the Apache architecture is amazingly well-thought for extensibility, with its clean separation of each phase in the request lifecycle, and the possibility to to insert hooks in each of those phases. Actually I didn't study Apache directly, but only through mod_perl, which exposes almost everything of the Apache API to Perl programming, and is another piece of amazing design. I must say, however, that mod_perl has a peculiar way of doing OO, through a kind of home-made mixing of packages into common namespaces, which for the time was quite clever but took me some time to understand. I guess this would all be written with "roles" if redesigned in modern Perl.
- several important CPAN modules would be worth discussing here, but that would bring us too far ... maybe later in another blog entry. Let me just state that every time I came to discover another module of Andy Wardley, I felt a sense of beauty : to me, Template Toolkit, AppConfig, Pod::POM
Thanks Matt for having asked this question about design, that was an opportunity to put some thoughts into shape, and I'm eager to read what other people will write on that topic.
Sunday, September 13, 2009
Centralized bug reporting is precious
I recently came across a module author who refuses bug reports submitted through rt.cpan.org, claiming that the system has many deficiencies, and increases the workload on module authors. The refusal is even automatic: you get an email explaining why the author dislikes RT, and the ticket is automaticallly canceled. For this person, bug reports are to be sent through regular email.
So let's think about this position.
True enough, RT has no opt-out possibility -- as soon as you have published something on CPAN, you are registered in the system, and can't decide to get out : the only options for an author are a) play the game; b) ignore tickets; c) explicitly refuse tickets as just described above. True enough, prolific authors are likely to receive many reports from RT. So for an author, it can indeed be felt as a burden.
However, for module users, it is extremely useful to have all bug reporting on one centralized platform, because this leverages collective knowledge. Even when an author is not responding at all -- and there might be many good reasons for not responding --, the system is nevertheless interesting, because you can find out if a problem was already met by somebody else, if some workarounds have already been identified, etc. When the author is responding, you can additionally know if the problem is currently taken into account or not, and sometimes there is already some patch or preliminary release to be tested.
Communicating about bugs through point-to-point plain email will just satisfy the bug reporter and the module author. Communication through a centralized bug tracking system is useful to the whole community. So, authors, please don't dismiss RT ... and if you think RT has some problems, you can always open a ticket !
So let's think about this position.
True enough, RT has no opt-out possibility -- as soon as you have published something on CPAN, you are registered in the system, and can't decide to get out : the only options for an author are a) play the game; b) ignore tickets; c) explicitly refuse tickets as just described above. True enough, prolific authors are likely to receive many reports from RT. So for an author, it can indeed be felt as a burden.
However, for module users, it is extremely useful to have all bug reporting on one centralized platform, because this leverages collective knowledge. Even when an author is not responding at all -- and there might be many good reasons for not responding --, the system is nevertheless interesting, because you can find out if a problem was already met by somebody else, if some workarounds have already been identified, etc. When the author is responding, you can additionally know if the problem is currently taken into account or not, and sometimes there is already some patch or preliminary release to be tested.
Communicating about bugs through point-to-point plain email will just satisfy the bug reporter and the module author. Communication through a centralized bug tracking system is useful to the whole community. So, authors, please don't dismiss RT ... and if you think RT has some problems, you can always open a ticket !
Sunday, September 6, 2009
Birth of DBIx::DataModel (or : Perl ORMs in 2005)
DBIx::DataModel is the object-relational mapping (ORM) layer we use for our projects at Geneva's law courts. It is much less known that DBIx::Class, but nevertheless a few people have shown interest in it, so I thought it's worth blogging about some of its design aspects. Today's blog is about history : a couple of words about why and how DBIx::DataModel was born.
Back in 2005, I had already written a number of Perl/DBI programs for Geneva's law courts, for data reporting or statistics purposes. Each of those programs was a separate entity; but the corpus kept growing, and some repetitive patterns started to appear, especially about expressing joins between the many tables that sit in our database. Then came the idea to also choose Perl, no longer just for reporting programs, but for rewriting our old Cobol business application. The application is mission-critical, and programming it in Perl seemed crazy to some people in house, so an initial design and proof of concept were of order to demonstrate feasibility. At this point, it was quite clear that some kind of ORM would be needed on top of DBI, to deal with SQL in a more abstract way.
After a CPAN survey, I identified Class::DBI as being the most likely ORM candidate module for our needs (other contenders at that time were Alzabo, Tangram, DBIx::Recordset, among many others). However, there were a couple of annoying limitations :
In other words, if the goal was to produce SELECT [columns] FROM [tables] WHERE [conditions], there was not enough flexibility for the the columns part, nor for the tables part, nor for the conditions part !
Adapting Class::DBI to work around these limitations seemed quite difficult, and I didn't find any substitute in the other modules mentioned above. So I started designing my own ORM, trying to optimize for SQL flexibility. In particular, adopting the wonderful SQL::Abstract from Nate Wiger was a good solution for expressing any kind of conditions in the WHERE clause as nested Perl datastructures. I also tried to apply the DRY (Don't Repeat Yourself) principle to relationships : if there is a 1-to-n relationship between two tables, better say it in one single place, rather than saying "has_one" on one side and "has_many" on the other. Finally, on the delicate problem of matching database joins to some kind of object-oriented entity, I came to the conclusion that the closest concept was the one of multiple inheritance : a record coming from a join between Artist and CD is an object that belongs to both the Artist and CD classes. This initial design evolved internally for a couple of months, until an initial CPAN release in September 2005.
Meanwhile the outside world had been evolving. In May 2005 came an extension to Class::DBI called Class::DBI::Sweet that also relied on SQL::Abstract. One month later, Class::DBI::Sweet was extended with some features for doing joins (prefetch feature and join argument to search method). Then in August came the first release of DBIx::Class, bringing a fresh design to work around Class::DBI limitations, but with an eye to backwards compatibility. That initial release had no support for database joins, but it was later extended in end September with prefetch, in the same vein as Class::DBI::Sweet. DBIx::Class very quickly showed to have an active and growing community.
At this point, the question was whether or not to adopt DBIx::Class. The advantage would have been to join a community and benefit from shared progress on common concerns. However, DBIx::DataModel had grown to something sufficiently mature and sufficiently different, so switching ORM implied some cost and mental shift. Furthermore, some original ideas of DBIx::DataModel seemed worth pursuing, and actually proved very useful in our projects. So finally the module is still alive, had a major refactoring in 2008 ... and anybody who is interested in helping to improve it is cordially welcomed!
Next blog entries will highlight some specific design features.
Back in 2005, I had already written a number of Perl/DBI programs for Geneva's law courts, for data reporting or statistics purposes. Each of those programs was a separate entity; but the corpus kept growing, and some repetitive patterns started to appear, especially about expressing joins between the many tables that sit in our database. Then came the idea to also choose Perl, no longer just for reporting programs, but for rewriting our old Cobol business application. The application is mission-critical, and programming it in Perl seemed crazy to some people in house, so an initial design and proof of concept were of order to demonstrate feasibility. At this point, it was quite clear that some kind of ORM would be needed on top of DBI, to deal with SQL in a more abstract way.
After a CPAN survey, I identified Class::DBI as being the most likely ORM candidate module for our needs (other contenders at that time were Alzabo, Tangram, DBIx::Recordset, among many others). However, there were a couple of annoying limitations :
- columns retrieved from any given table were fixed at table declaration time; no possibility to later modulate the column list at the query level, resulting in unnecessary data transfers;
- joins declared in Class::DBI could be used to issue successive queries for navigating between related records, but not for doing a database join in a single query;
- conditions as specified in search or search_like were too limited in expressivity (no boolean combinators, no comparison operators, etc.).
In other words, if the goal was to produce SELECT [columns] FROM [tables] WHERE [conditions], there was not enough flexibility for the the columns part, nor for the tables part, nor for the conditions part !
Adapting Class::DBI to work around these limitations seemed quite difficult, and I didn't find any substitute in the other modules mentioned above. So I started designing my own ORM, trying to optimize for SQL flexibility. In particular, adopting the wonderful SQL::Abstract from Nate Wiger was a good solution for expressing any kind of conditions in the WHERE clause as nested Perl datastructures. I also tried to apply the DRY (Don't Repeat Yourself) principle to relationships : if there is a 1-to-n relationship between two tables, better say it in one single place, rather than saying "has_one" on one side and "has_many" on the other. Finally, on the delicate problem of matching database joins to some kind of object-oriented entity, I came to the conclusion that the closest concept was the one of multiple inheritance : a record coming from a join between Artist and CD is an object that belongs to both the Artist and CD classes. This initial design evolved internally for a couple of months, until an initial CPAN release in September 2005.
Meanwhile the outside world had been evolving. In May 2005 came an extension to Class::DBI called Class::DBI::Sweet that also relied on SQL::Abstract. One month later, Class::DBI::Sweet was extended with some features for doing joins (prefetch feature and join argument to search method). Then in August came the first release of DBIx::Class, bringing a fresh design to work around Class::DBI limitations, but with an eye to backwards compatibility. That initial release had no support for database joins, but it was later extended in end September with prefetch, in the same vein as Class::DBI::Sweet. DBIx::Class very quickly showed to have an active and growing community.
At this point, the question was whether or not to adopt DBIx::Class. The advantage would have been to join a community and benefit from shared progress on common concerns. However, DBIx::DataModel had grown to something sufficiently mature and sufficiently different, so switching ORM implied some cost and mental shift. Furthermore, some original ideas of DBIx::DataModel seemed worth pursuing, and actually proved very useful in our projects. So finally the module is still alive, had a major refactoring in 2008 ... and anybody who is interested in helping to improve it is cordially welcomed!
Next blog entries will highlight some specific design features.
Thursday, August 27, 2009
Emacs/Perl integration
In answer to YAPC::EU::2009 's call for volunteers for the beginner's track, I offered a presentation on Perl development under Emacs. Part of the motivation for doing that talk was to force myself to have a look at various Emacs extensions (so far I'd been using just the standard cperl-mode coming with Emacs, which is already very rich in features). So here are the conclusions.
[Disclaimer: this was a quick, informal survey; maybe I misunderstood some points, or maybe some of the difficulties encountered were bound to my specific environment (Emacs on Win32); so I hope I don't do any injustice to these modules]
[Disclaimer: this was a quick, informal survey; maybe I misunderstood some points, or maybe some of the difficulties encountered were bound to my specific environment (Emacs on Win32); so I hope I don't do any injustice to these modules]
- Emacs::EPL looks like it was a very nice integration work. For example it could redirect things like STDIN/STDOUT or %ENV variables for Perl scripts running under Emacs, so that user interaction would go through Emacs mechanisms (e.g. the minibuffer). It also allowed people to write Emacs extensions in Perl instead of e-lisp ! This sounds really interesting, but unfortunately it hasn't been updated since 2001 and I couldn't get it to work. I really wish that somebody would resurrect that module.
- Devel::PerlySense is an ambitious project with lots of features; it runs a Perl process in the background to run/debug code, perform static analysis, etc. However it was so CPU-intensive that I found it hardly usable for daily work.
- Sepia is an active project (latest release July 09), also running an inferior Perl process, also quite rich in features (it goes as far as analyzing Perl opcodes to deduce information about packages/methods/etc.). Here I had another problem : it would only recognize core modules and functions. I didn't find how to make it understand that there was also some code under perl/site/lib !
- Emacs::PDE installed smoothly and provides a number of nice features, like perldoc/perltidy/perlcritic integration, tree views of sources/documentation, interactive perl shell, etc.
So in the coming weeks I plan to learn more about Emacs::PDE and maybe add it to my collection of tools for daily work.
Sunday, August 23, 2009
Be fair to ActiveState
Recently I heard several people emitting strong comments against ActiveState,
the company who first published a Perl distribution for Windows. The tone of those comments was about the same as people crusading against Microsoft.
I feel this is not fair.
True enough, Windows users now have an alternative solution with Strawberry Perl, which is more similar to Perl environments on Unix boxes. Strawberry Perl is progressing very well, which is rejoicing, and is likely to have a positive impact on the number of Perl installations in Windows world. Personally I haven't switched to Strawberry yet, but it's likely to happen in a near future, especially becausee I'm interested in the combination with Perl::Dist. Together these look like it will be possible to package a customized distribution for our organization, that would install everything we need in a single click ... but I still need a little bit of time to experiment with all that.
However, this is not a reason for despising ActivePerl, which continues to work well, and for quite a number of years has already been able to collaborate with the mingw gcc compiler for installing CPAN modules with C extensions. Personally I owe quite a lot to ActivePerl, which
almost 10 years ago provided me with an initial Perl environment to get some work done on our win32 workstations; without that, our projects at Geneva's law courts would probably never have started.
ActivePerl also did quite a lot to integrate Perl with various Windows components (PerlScript, OLE, etc.). This provides a very interesting environment for Win32 automation and has been tremendously useful for some migration tasks in our organization. Unfortunately, this usage of Perl is not widespread enough; it seems that only few people realized the power of Perl for Win32 administration, probably because many Windows sysadmins are so used to point-and-click that they just don't have the culture of automating common repetitive tasks.
Finally, ActivePerl's presentation of documentation in HTML with tree navigation was a real blessing for getting acquainted with the vast body of Perl and CPAN literature. I liked it so much that it became a major source of inspiration for developing the Pod::POM::Web HTTP documentation server.
As a whole, the Perl community greatly benefited from ActiveState's work, so it seems to me that people disregarding that company are just caught in vrittis (patterns of mind) that let them disregard any company whatsoever.
Thanks, ActiveState, and all the best for your future business plans.
the company who first published a Perl distribution for Windows. The tone of those comments was about the same as people crusading against Microsoft.
I feel this is not fair.
True enough, Windows users now have an alternative solution with Strawberry Perl, which is more similar to Perl environments on Unix boxes. Strawberry Perl is progressing very well, which is rejoicing, and is likely to have a positive impact on the number of Perl installations in Windows world. Personally I haven't switched to Strawberry yet, but it's likely to happen in a near future, especially becausee I'm interested in the combination with Perl::Dist. Together these look like it will be possible to package a customized distribution for our organization, that would install everything we need in a single click ... but I still need a little bit of time to experiment with all that.
However, this is not a reason for despising ActivePerl, which continues to work well, and for quite a number of years has already been able to collaborate with the mingw gcc compiler for installing CPAN modules with C extensions. Personally I owe quite a lot to ActivePerl, which
almost 10 years ago provided me with an initial Perl environment to get some work done on our win32 workstations; without that, our projects at Geneva's law courts would probably never have started.
ActivePerl also did quite a lot to integrate Perl with various Windows components (PerlScript, OLE, etc.). This provides a very interesting environment for Win32 automation and has been tremendously useful for some migration tasks in our organization. Unfortunately, this usage of Perl is not widespread enough; it seems that only few people realized the power of Perl for Win32 administration, probably because many Windows sysadmins are so used to point-and-click that they just don't have the culture of automating common repetitive tasks.
Finally, ActivePerl's presentation of documentation in HTML with tree navigation was a real blessing for getting acquainted with the vast body of Perl and CPAN literature. I liked it so much that it became a major source of inspiration for developing the Pod::POM::Web HTTP documentation server.
As a whole, the Perl community greatly benefited from ActiveState's work, so it seems to me that people disregarding that company are just caught in vrittis (patterns of mind) that let them disregard any company whatsoever.
Thanks, ActiveState, and all the best for your future business plans.
Friday, August 14, 2009
Object-oriented accessors considered (sometimes) harmful
One of the big principles of object-oriented (OO) programming is encapsulation, which says that data inside an object should be hidden from the outside world, and all accesses to the data should go through methods. That's where OO accessor/mutator methods come from.
Encapsulation brings a number of advantages, like being able to change the internal representation of an object, validating any attempt to change state, blurring the difference between stored values and computed values, avoiding using object storage for action at distance, etc. All of this has been discussed for many years in numerous books and papers, and is wonderfully implemented in Moose (see its Attributes manpage).
However, encapsulation also brings a number of disadvantages, especially in Perl which, contrary to Java or Eiffel, is a multi-paradigm language. Below I'm discussing those disadvantages, not with the intention of throwing away encapsulation, but rather to issue a warning : don't blindly adopt encapsulation everywhere; think and weight where it is a gain and where it is a hindrance.
The canonical example in OO literature is the "point object" (even in the Moose synopsis). Now, in contrary to objects like a device driver or database handle, that would have lots of internal data to hide, a point object contains nothing but public information (its x and y coordinates). If we have direct access to the stored values, we can use a number of convenient Perl idioms, while doing the same using getter/setter methods is much more cumbersome. Consider the following examples:
Translating such idioms to getter/setter methods is left as an exercise to the reader...
If on the contrary, that arrayref is considered private, and clients must use getter/setter methods to access it, then clients have no choice but copying the entire array back and forth between the remote object and the local handling code. Needless to say, this has an impact on performance. In order to avoid it, the object could also implement additional methods to insert an item into the array, remove the last, etc. ... but this is reinventing a list API, and it will never be as rich and flexible as the collection of builtin Perl constructs for dealing with lists!
The same reasoning goes of course for hashref attributes.
It may be the case that meta-information as managed by Moose could perhaps help in traversal algorithms (ask the metaclass about which methods link to "subobjects" of the current one), but I'm not aware if somebody has already worked on that.
An interesting design is the one of DBI : objects (handles) are totally encapsulated, yet they exploit the power of tie to expose their attributes through a conventional hashref API, instead of OO getter and setter methods. This is a very clever compromise.
As far as I am concerned, I purposedly designed DBIx::DataModel to fully exploit the dual nature of Perl objects, having both the OO API for executing methods, and the hashref API for accessing column values. I wouldn't necessarily do that everywhere, but for row objects in a database, which are very open in nature, this just seemed an appropriate solution.
Encapsulation brings a number of advantages, like being able to change the internal representation of an object, validating any attempt to change state, blurring the difference between stored values and computed values, avoiding using object storage for action at distance, etc. All of this has been discussed for many years in numerous books and papers, and is wonderfully implemented in Moose (see its Attributes manpage).
However, encapsulation also brings a number of disadvantages, especially in Perl which, contrary to Java or Eiffel, is a multi-paradigm language. Below I'm discussing those disadvantages, not with the intention of throwing away encapsulation, but rather to issue a warning : don't blindly adopt encapsulation everywhere; think and weight where it is a gain and where it is a hindrance.
Good Perl idioms are no longer available
The canonical example in OO literature is the "point object" (even in the Moose synopsis). Now, in contrary to objects like a device driver or database handle, that would have lots of internal data to hide, a point object contains nothing but public information (its x and y coordinates). If we have direct access to the stored values, we can use a number of convenient Perl idioms, while doing the same using getter/setter methods is much more cumbersome. Consider the following examples:
# string interpolation
print "point is at coordinates $point->{x} / $point->{y}\n";
# symmetry transform
($point->{x}, $point->{y}) = ($point->{y}, $point->{x});
# zoom
$point->{$_} *= $zoom_factor for qw/x y/;
# temporary push aside
{ local $point->{x} += $far_away;
do_something_without_that_point_in_the_way();
} # point automatically comes back to its previous location
# nested datastructures and possibly auto-vivification
push @{$point->{styles}}, qw/square big/;
$point->{menu}{options}{color} = 'green';
Translating such idioms to getter/setter methods is left as an exercise to the reader...
No obvious distinction between "setter" methods and other methods
When I read some unknown code and find something like $obj->{foo} = $x;
$obj->bar($y);
I immediately know that the state of the object changes at line 1, while I have no clue at line 2 : I must go to the doc of method bar (or even worse, at the source code), to know if this is printing something out, modifying a database record, or just setting a bar attribute in memory inside object $obj.Hard to debug
Sometimes the best way to find a bug is to go step by step in the debugger and examine which values sit in memory. However, if we use chained accessors to traverse a datastructure, the number of steps to go through becomes a nightmare. For example consider the following hypothetical line in a Catalyst controller : if ($c->request->body->length < $self->min_body) {If I come across that line while debugging, I know that there are chances that the bug might sit in the min_body method, so I want to step into that method; but before getting there I'll have to step through all accessor methods for request, body and length, which is totally uninteresting for my debugging purpose.Non-scalar attributes must either copy or reinvent an API
If an object contains an arrayref attribute, directly accessible and publicly documented as such, then every client can happily push, shift, reverse, grep, etc. into the arrayref using common Perl syntax.If on the contrary, that arrayref is considered private, and clients must use getter/setter methods to access it, then clients have no choice but copying the entire array back and forth between the remote object and the local handling code. Needless to say, this has an impact on performance. In order to avoid it, the object could also implement additional methods to insert an item into the array, remove the last, etc. ... but this is reinventing a list API, and it will never be as rich and flexible as the collection of builtin Perl constructs for dealing with lists!
The same reasoning goes of course for hashref attributes.
No generic data traversal modules
Data traversal modules like Data::Visitor, or import/export modules like JSON or YAML, cannot take any decisions when they come to an opaque object that one is not supposed to peek into. This means that generic traversal tools are just inapplicable, or that every object has to implement some support for traversal (like a visit_value method); but writing such methods for every new class is kind of tedious.It may be the case that meta-information as managed by Moose could perhaps help in traversal algorithms (ask the metaclass about which methods link to "subobjects" of the current one), but I'm not aware if somebody has already worked on that.
Methods are slower than direct access to attributes
No much to expand here: obviously method calls have a cost, especially in Perl which is more dynamic than other OO languages, so the compiler has less information to possibly optimize the method calls. So when traversing deep datastructures through nested accessor calls, there is a definite impact on performances.Conclusion
In Perl, fully encapsulated objects are sometimes the best solution, sometimes not; weight these considerations before taking strong design decisions.An interesting design is the one of DBI : objects (handles) are totally encapsulated, yet they exploit the power of tie to expose their attributes through a conventional hashref API, instead of OO getter and setter methods. This is a very clever compromise.
As far as I am concerned, I purposedly designed DBIx::DataModel to fully exploit the dual nature of Perl objects, having both the OO API for executing methods, and the hashref API for accessing column values. I wouldn't necessarily do that everywhere, but for row objects in a database, which are very open in nature, this just seemed an appropriate solution.
Thursday, August 13, 2009
extending DBD::SQLite functionalities
About one year ago I started my first project involving XS programming : this was because I needed specific collation sequences in SQLite databases. The basic SQLite implementation in C had support for user-defined collations, but at that time the Perl API in DBD::SQLite did not give access to it ... so I had to write the glue myself.
Learning XS was kind of fun : the C structures, the dozens of macros, the stack manipulation functions and the "mortalization" concept, all of this is pretty well documented, so once you have a real project to get motivated, you can enter that field without too much trouble.
Once done, I sent the patch to the DBD::Sqlite author, and got no answer, which was less fun. Fortunately; Audrey Tang incorporated the patch in her DBD::SQLite::Amalgamation version of the driver.
Nowadays the situation is very different, because Adam Kennedy has taken over maintenance of DBD::SQLite, and there are both an open source repository and a mailing list, and a reactive Kenichi Ishigaki who cares about release management, so it's much more pleasant to participate.
What I did recently was to study the SQLite API and add support for all other native functions that so far were not exposed to the Perl DBD::Sqlite driver --- not that I really need that at the moment, but just for completeness's sake. Such functions are, for example, hooks to be run whenever a
Another interesting functionality is another hook called
If anybody is interested, that stuff is already available on CPAN in the current developer release of
Learning XS was kind of fun : the C structures, the dozens of macros, the stack manipulation functions and the "mortalization" concept, all of this is pretty well documented, so once you have a real project to get motivated, you can enter that field without too much trouble.
Once done, I sent the patch to the DBD::Sqlite author, and got no answer, which was less fun. Fortunately; Audrey Tang incorporated the patch in her DBD::SQLite::Amalgamation version of the driver.
Nowadays the situation is very different, because Adam Kennedy has taken over maintenance of DBD::SQLite, and there are both an open source repository and a mailing list, and a reactive Kenichi Ishigaki who cares about release management, so it's much more pleasant to participate.
What I did recently was to study the SQLite API and add support for all other native functions that so far were not exposed to the Perl DBD::Sqlite driver --- not that I really need that at the moment, but just for completeness's sake. Such functions are, for example, hooks to be run whenever a
commit, rollback or update occurs; this means for example that if your transaction does some stuff outside of the database -- like creating files or directories --, you can program a rollback hook that will 'undo' these changes in case of failure.Another interesting functionality is another hook called
set_authorizer, that can decide whether a given SQL statement is authorized or not on that database handle. This could be used for example to specify that only administrators are allowed to update or insert data, while any user can read.If anybody is interested, that stuff is already available on CPAN in the current developer release of
DBD::SQLite ... enjoy!
Wednesday, August 12, 2009
YAPC::EU::09 paper on managing Geneva's law courts
This year's YAPC::EU::2009 conference had printed proceedings.
For this occasion I wrote a paper about my talk Managing Geneva's law courts, from Cobol to Perl. An html copy of the paper is now online. The slides are available at slideshare.
For this occasion I wrote a paper about my talk Managing Geneva's law courts, from Cobol to Perl. An html copy of the paper is now online. The slides are available at slideshare.
Tuesday, August 11, 2009
The Perl community is exceptional
This year I went to the YAPC::EU::2009 conference in Lisbon. I missed the one from last year, so being a little less accustomed to those gatherings, I recovered the same feeling that I got at
my first YAPC conference in Birmingham in 2006 : this community is like nothing else I've seen before.
I had been to quite a lot of computer science conferences in my previous life, where the climate was very different. In the academic world, people mostly think about the papers: if your paper is accepted, you go to the conference to add one more line on your publication list; if it is not accepted, maybe you go nevertheless to see what are the ideas in fashion, and to do a bit of networking that may help you getting accepted on the following year, or build an international circle that will give more credibility to your next research proposal. In that world, it's nice to see the community again, but meanwhile there is always a feeling of competition, because the person sitting next to you might well be that anonymous expert that turned down your last paper submission. Question sessions may be quite harsh, too, because people in the audience sometimes are more interested in showing off their own knowledge, rather than genuinely trying to clarify what the speaker was talking about.
YAPC conferences are very different. People come because they love it, and some of them even come on their own time and money. They love the fun, of course, the special dresses, the jokes, the culture; but they also love to join for shaping new ideas, new tools, new constructs that will be helpful to the whole community. They don't come for marking their territory; they come for passion, and they know that by working together, everybody can help Perl to improve and expand.
I think I've never heard any aggressive question at a YAPC: rather, the audience sometimes helpfully points to some resources that the speaker might have overlooked.
All of this makes YAPC very much enjoyable, so I'll definitely try to attend next year in Pisa.
my first YAPC conference in Birmingham in 2006 : this community is like nothing else I've seen before.
I had been to quite a lot of computer science conferences in my previous life, where the climate was very different. In the academic world, people mostly think about the papers: if your paper is accepted, you go to the conference to add one more line on your publication list; if it is not accepted, maybe you go nevertheless to see what are the ideas in fashion, and to do a bit of networking that may help you getting accepted on the following year, or build an international circle that will give more credibility to your next research proposal. In that world, it's nice to see the community again, but meanwhile there is always a feeling of competition, because the person sitting next to you might well be that anonymous expert that turned down your last paper submission. Question sessions may be quite harsh, too, because people in the audience sometimes are more interested in showing off their own knowledge, rather than genuinely trying to clarify what the speaker was talking about.
YAPC conferences are very different. People come because they love it, and some of them even come on their own time and money. They love the fun, of course, the special dresses, the jokes, the culture; but they also love to join for shaping new ideas, new tools, new constructs that will be helpful to the whole community. They don't come for marking their territory; they come for passion, and they know that by working together, everybody can help Perl to improve and expand.
I think I've never heard any aggressive question at a YAPC: rather, the audience sometimes helpfully points to some resources that the speaker might have overlooked.
All of this makes YAPC very much enjoyable, so I'll definitely try to attend next year in Pisa.
Saturday, August 8, 2009
Hello, world
OK, time to go blogging.
For a couple of months, it had been one of these things I knew I wanted to do sooner or later ... but preferred to leave until later. Somehow there was always something more urgent, a bug to fix, a release to prepare, a doc to write, an analysis to rediscuss, a new musical piece to rehearse, so I never got into really doing it.
Matt Trout already tickled me a couple of weeks ago in his commentary about DBIx::DataModel, but his vehement exhortations at YAPC::EU::2009 finally convinced me that there is no excuse to delay it any further : I have to join that Ironman competition, a grand idea for getting people to write and propagate thoughts and facts about Perl development.
Perl blogging had been around for a while, in particular in use.perljournals, but the Ironman competion really gave a new thrust to it; and the information that circulates through that new means is really valuable, because people share thoughts, considerations, design issues as soon as they emerge, even partially shaped, and that's good enough to get readers to also start thinking about those issues. Blogging is less intrusive than mailing lists: you don't throw something out saying "hey, folks, this is for you", you rather leave some dangling thought somewhere, and whoever comes across it can grab it if interested; so even if your idea is vague or controversial, it doesn't matter.
Once the decision to blog was taken, I had to decide about the how and where. I already had a L, no very much used so far, but good enough. Many people complain about the use.perl site, and true enough, it has a number of inconveniences, and didn't evolve since it was launched a couple of years ago; but it has the merit to exist and still is a valuable resource for the Perl community.
However, I felt that blogging in use.perl would somehow miss the point of Ironman, which is to get the world to hear about perl outside of the Perl echo chamber.
The other solution would be to go like real geeks, who have their own internet domains, running software they have chosen or written themselves, caring about configuration and maintenance issues ... however I haven't found the time to do it for the last 6 months. So finally there I am, hosted by blogger.com, and I know that everything written here will definitely be indexed, so the goal of propagating Perl culture should be attained...
Hello, world, please come back and stay in tune !
For a couple of months, it had been one of these things I knew I wanted to do sooner or later ... but preferred to leave until later. Somehow there was always something more urgent, a bug to fix, a release to prepare, a doc to write, an analysis to rediscuss, a new musical piece to rehearse, so I never got into really doing it.
Matt Trout already tickled me a couple of weeks ago in his commentary about DBIx::DataModel
Perl blogging had been around for a while, in particular in use.perl
Once the decision to blog was taken, I had to decide about the how and where. I already had a L
However, I felt that blogging in use.perl would somehow miss the point of Ironman, which is to get the world to hear about perl outside of the Perl echo chamber.
The other solution would be to go like real geeks, who have their own internet domains, running software they have chosen or written themselves, caring about configuration and maintenance issues ... however I haven't found the time to do it for the last 6 months. So finally there I am, hosted by blogger.com, and I know that everything written here will definitely be indexed, so the goal of propagating Perl culture should be attained...
Hello, world, please come back and stay in tune !
Subscribe to:
Posts (Atom)
