Web application implementation step 3: Framework vs Methodology
RT @thibauld Web application implementation step 3: Framework vs MethodologyNow you must be wondering why I'm so skeptical about frameworks... actually, I have nothing against frameworks in themselves (see my post on the subject), some are better than others but generally speaking, they are all useful in some way. No, the problem lies more in what people tend to expect from frameworks. Indeed, a lot of people expect that, because they're using a good framework, they'll be able to code efficiently and deliver a great web app quickly. Unfortunately, this is a wrong reasoning... your coding efficiency depends more on your methodology than on the framework you're using. Just as an illustration, coding version 1.0 of freelance business club required only 3 months, starting from scratch and without using any framework.
The ultimate rule for coding efficiently is the following: first think, then code. I will now explain in more details the methodology I personally use when building web applications. This methodology is summed up in the following schema:
Process listing
This is, by far, the most important step, the one that can save you an incredible amount of time! When you start a project, everybody has a gazillion ideas on the application features: "it must do this, that" "how about this ?" "It would be great if we could do that" "You know what would be great ? this!"... Don't get me wrong, having people brainstorming around the application potential features is great, even essential, and you should write down all these great ideas. But obviously, you won't be able to have all these features for version 1.0.
First, you should translate these features into processes. A process puts a feature into perspective. The question to ask is the following: "To which business process does this feature belong to ?". You'll be amazed eventually at how many proposed features don't belong to any real business process and are, thus, irrelevant to your application. In particular, developers (me included) often think about adding "cool" features from a technical point of view... but each "cool" feature, even the smallest ones, take time to code so it is very important to think about its real usefulness from the end user point of view. Thinking in terms of business processes rather than in term of features is a good method to separate relevant features from irrelevant ones.
At this point, don't forget the "backend" processes, that is, the ones required to run your web application. These backend processes are too often overlooked, which result in people finally spending a lot of energy to administrate their web app. And "no", PhpMyAdmin cannot be your only backend application.
Once you you've listed all your processes, the next step is to break them down in milestones. The most crucial processes will go in the first milestone, the less important ones in the next milestone.... ect... the least important ones going in the "later" pseudo-milestone. At the end of this step, you've got all your processes listed by milestone.
Now, before going any further, I encourage you to perform some research to see, for each process, if there's not an available application/api/webservice out there that you can leverage to help you implement this process faster and/or better. This can be a huge booster and might prevent you from reinventing the wheel (and probably a poorer one).
Maquetting / Wireframing / Prototyping
Now that you know which processes to implement for the next milestone, you need to "wireframe" it (in french we say "maquetter", I'm not sure of the english word for it). What is important in this step is to synchronize everybody on each screen which are to be develop. What's in a screen:
- General layout of the page
- Information displayed
- Links available
To build wireframes, I personally tend to use Impress (the PowerPoint like module of OpenOffice.org) but you can use anything you like, pen and paper are ok too... If you're a Windows user, a friend of mine uses Axure, a commercial tool dedicated to wireframing and prototyping. Once you've got all your screens ready, you may want to turn them into a real prototype to have your users validate how you will implement the business processes. On linux, I found that such a task can be easily realized with kimagemapeditor: just import screenshots of your screens and define the links between them using html image maps. You'll end up with a static website in which you can navigate easily.
Once this is done, you can have your graphical designer start working to turn these ugly wireframes into art!
URL listing
At this point, once you have all your screens and their relationships setup, it is time to decide the URL of each resource. It is very important to think about your URL strategy beforehand for several reasons. First, as you're building a web application, you cannot ignore Google and need to implement a google-friendly URL scheme. Then, thinking about your URLs help you switch mentally between your user requirements and how you will code it.
For example, in the code architecture I implemented, this step is key as I cannot code anything until I translated my URL scheme in a .htaccess (using the urlrewriting Apache module). This .htaccess determines how I organize my presentation handlers.
Database design
During the previous step, you probably began thinking about the 'objects' you are going to manipulate. It is now time to write it down by coming up with a complete and coherent database design. I cannot stress enough how much database design is key. If you fail here, you'll have to code a lot more than needed. You will not only have to paliate the inefficiencies of your database design by coding more, but you will also have to deal with poor application performance sooner than expected.
Does "referential integrity" sound familiar to you ? it should... Too few people realize how powerful a database like Mysql or Postgresql is. SQL being a declarative language, a lot of developers don't like it much but it's a pity... it can be really powerful and you can delegate _a lot_ of work to your database. Letting your database handle work saves you coding time, and will probably boost your application performance. Database design is a vast subject... If I have time, I'll probably write a post dedicated to it but meanwhile, I encourage you to read this great book : The Art of SQL.
Coding
Now you can begin coding 
Of course, I also have a methodology for coding. This is good practice to implement process by process (you can parallelize work depending on your team size). The most important thing being what the end user sees in the end, you should start for each process with the presentation layer, then continue with the business layer to finally end with the data layer. To ensure a good level of quality, once you finished coding a layer, validate it using dummy values from the n-1 layer (because the n-1 layer has not yet been implemented). If you follow this methodology, you'll be amazed at how fast you'll be able to develop your web application.
That's all for today, see you next post!
Web application implementation step 1: choose your weapons!
RT @thibauld Web application implementation step 1: choose your weapons!During the next series of posts, I will describe the different steps involved in building a web application. These posts will only deal with the technical aspects of implementing a web application that has already been designed. Web application design is another subject which I'll develop in another series of posts.
I'll take the new FBC website as an example. It is a good example as we almost had to start from scratch to build the new version. Indeed, the architecture in place was extremely inflexible, the worst being :
- Webpage layout relied on field values in SQL tables
- 1 url <=> 1 php page containing everything from HTML to SQL
Functionally speaking, their website was working well but it had been at the expense of countless hours spent testing and testing and retesting that every process was functional. But now they were stuck... Only changing the graphical chart would have needed to go change *every* php file and restest the whole thing entirely... When I saw that, it was obvious that it would take us less time to rebuild the entire architecture from scratch and then add the new features than trying to work on this extremely rigid and non evolutive code organization. Of course, we would need to reutilize pieces from the old code in order not to re-implement the business process, but it was pretty much everything we would be able to reutilize.
So the first step when building a web application (and not a website) is choosing your weapons, that is the technology stack you're going to build your application upon :
- Which OS are you going to use ?
- Which webserver ?
- Which database ?
- Which programming language and framework ?
Choosing the OS
I'll not elaborate on this one, use Linux. The only reason I can see for not using Linux are :
- you know nothing about Linux and don't want to learn
- you were asked explicitely not to use Linux
- there are obvious reasons which require your app run on another system
If none of the above matches, choose Linux. And among all Linuxes, if you have no preference, I'd personally go for a Ubuntu Server LTS (Long Term Support) which is free and provides security updates during 5 years. Just check that the LTS does not end next year The current Ubuntu Server LTS is version 8.04 (code name Hardy Heron).
Choosing the webserver
Again, I'll not elaborate much: Apache 2 as it has proven capabilities to run high traffic websites, great documentation (official and non official), lots of modules availables... etc... There are other webservers but, unless you want to restrain it to very specific tasks, use Apache to serve your dynamic pages.
Choosing the database
I assume you want an Open Source database here. It's hard to choose here between MySQL and PostgreSQL . PostgreSQL is probably more powerful but it is also probably a little harder to get into. For example, the fulltext search in MySQL in a lot easier to use than in PostgreSQL but PostgreSQL's fulltext search allows you to perform more complex queries. In the same spirit, MySQL provides handy but limited functions (ex: GROUP_CONCAT) where PostgreSQL offers you less handy but more powerful equivalents (ex: array).
Both database has a good official documentation but I think MySQL benefits from a lot more unofficial documentation compared to PostgreSQL. In terms of proven capabilities, both have examples of usage in very demanding contexts...
Finally, the choice is up to you BUT if you use MySQL, do not forget to use InnoDB tables instead of MyISAM tables. By default, tables are created using MyISAM. Using MyISAM table can be a good choice but it must be a known choice. Indeed, MyISAM table does not support FOREIGN KEYS nor does it support sql transactions so it is very limited and will inevitably result in added programming overhead if you build a complex web application upon MyISAM tables.
Choosing a framework
Rule 1: be skeptical
First, you should not use a framework for the sake of using a framework. Instead, you should really think at what you expect from your framework. I'm personally a bit skeptical about a lot of frameworks because, in my opinion, a lot of them end up being bloated, trying to reinvent the wheel and with finally a limited number of really useful functionalities. Keep in mind that using a framework is a binding investment: First it needs time to learn how to use it efficiently and, second, it generally deeply affects your code. So you'd better choose your framework well if you don't want to loose a lot of time !
Rule 2: know why you need a framework
Beware of frameworks that try to provide a solution for every aspect of your application as they are likely to belong to the "bloated" category! I personnally prefer a framework which focuses on doing 1 thing right than a framework that tries to simplify everything. For each functionality, ask yourself : "Do I really need my framework to do this ? Will it really help me be faster in my work ?
Rule 3: Beware of marketing crap
Here are examples of marketing functionalities I always consider with skepticism :
- i18n support (internationalization): In which way is it better than gettext ?
- RESTful URLs: You mean Apache URL Rewriting (and optionally a routing index.php) ?
- Easy Ajax: I did not know Ajax was hard to implement on the server side! Are you looking for a Javascript framework ?
- Full database abstraction: This one must be considered really really carefully. Database design is one of the most crucial part when you build a web application. In the same manner, how you query your database has an enormous impact on your application performance. This is why I am really really skeptical when it comes to delegating the database management to the framework. Some frameworks even create the database for you! Unless you're talking of a very simple web app, I think it just does not make sense... The only explanation I can think of is that a lot of web developers have a very little knowledge of SQL and, for them, it's easier/better to have the framework handle the database. On the other hand, it is also true that that database abstraction (especially ORM) are useful to handle all the simple queries. As a conclusion, concerning database abstraction, I would only use one if I'm sure that it's flexible enough to :
- Work on top of the database schema I designed
- Let me perform raw SQL queries when needed
I see a framework as a toolbox to help you concentrate your efforts on implementing real business code. So what can you expect from a framework ? A templating system, a caching system, a testing / debugging framework... All of these are very useful functionalities that a framework can offer because coding them would sure be a waste of time!
Choosing a programming language
There are a lot of very good languages for web programming (PHP, Python, Ruby...) and I can see a few differences:
- Framework compatibility: Once you've chosen your framework, you probably already know the programming language you'll be using.
- Quality and quantity of documentation: I think PHP has the best and most comprehensive documentation, followed by Python and Ruby.
- Library exhaustivity: Again, in my opinion, PHP has the deepest library available, followed by Python and Ruby. You should check beforehand that a language has all the required library for the app you want to implement. For example, if you plan to use SOAP based webservices, you probably want to avoid Python as (last time I checked) it has one of the most limited SOAP library. PHP, on the contrary, has a good gsoap compatible SOAP implementation.
- Familiarity with the language: One of the most important criteria of course! How productive are you with each language ? Even if all of these languages are relatively easy to get into, mastering a language takes time... so if you're a Ruby guru and have never used PHP, then you're probably better off with Ruby...
- Language beauty: Sorry guys but this is unfortunately NOT a good criteria. Had it been, then I would have rated Ruby first, then Python and then.... far behind, PHP
To conclude this long post, I think the most important rule is :
Use production ready technologies which have proven to be performant in similar contexts
Each time you have to choose a technology stack, no matter what it is and what bloggers and/or developers tell you about it, find relevant examples of web applications using this very stack to be sure, provided you do things well, the chosen stack will be able to fit your needs (performance, scalability...).
Stay tuned for the next post where I'll talk about the technology stack I've chosen for FBC!