Category Archives: technology

Minify Gzip

Minify vs gzip

While discussing some common optimisation techniques for the web, a discussion ensued on why are assets both minified and gzip’ed? Is it good for performance? Is there a benefit in minifying and then gzipping? Is it okay to gzip for HTTPS? What the heck, does HTTPS have to do with compression?

The average size of top 1,000 web pages is 1961kb today, compared to 1720kb in Aug 2014. This is a 14% growth, in just a year!  ~ HTTP Archive stats

Amount of data downloaded, by browsers to render a page has been steadily increasing. On the other hand, the performance requirements for end-users aren’t getting any better. Talking about web performance entails talking about 4 things

  1. Network transit time
  2. Response preparation (at server)
  3. Response parse (at client)
  4. Execution time (time taken to apply response on the page)

Given that, we can’t necessarily reduce data that gets generated, the only other way is to reduce its size for network transit (wire weight).

Minification is the practice of removing unnecessary characters (ex: comments, whitespace) from code

.. to reduce its size thereby improving load times. The browser can read and use it just like the original file. But since now, the wire weight is reduced, the page will load relatively faster. It also bundles various files into one, thus reducing the amount of requests to the web server.

Gzipping looks for repeating strings and replaces them with a pointer to the first occurrence

..The browser must decompress any compressed files before using them and this adds to the response parse time. We might have shaved off on some network transit time, but added to the response parse times.

Statistics to prove that gzip and minify is important

Normal Minified Gzipped Minified and gzipped
 JQUERY 2.1.3  241.59 KB 82.34 KB 71.68 KB 28.87 KB
 bootstrap-3.1.1.css  118 KB 98 KB  80 KB 17 KB

*gzip – gzip -6 file.js — the default compression level is 6 on a scale from 1 (fastest, but less compression) to 9 (slowest, but best compression).

A quick calculation gives us this.. On a 100 Mbps link,

Time to download Size
18.4319 ms 241.59 KB.
2.20261 ms 28.87 KB.

Needless to say, that this clearly tells us, to ensure static assets should always be minified and gzipped both.


Where did the name Dynamic Programming come from?

… the 1950’s were not good years for mathematical research. We had a very interesting gentleman in Washington … Secretary of Defense, and  .. had a pathological fear and hatred of the word, research. not using the term lightly; … using it precisely. His face would suffuse,  he would turn red, and .. get violent if people used  the term, research, in his presence. 

What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons.

I decided therefore to use the word, ‘programming.’

I wanted to get across the idea that this was dynamic, this was multistage,…I thought, let’s … take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that is it’s impossible to use the word, dynamic, in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning.

It’s impossible.

It was something not even a Congressman could object to.

Richard Bellman, “Eye of the Hurricane: an autobiography”, 1984

Setting up Nginx with Unicorn for Rails

I have moved away from using Thin to using Unicorn in production for a number of reasons

Some articles to convince you of the same thing.

  1. Unicorn is Unix
  2. Unicorn Power
  3. Most important for me was the ability of Unicorn master and worker processes to respond to Unix signals. This means, zero downtime deployments.

You need more convincing? Let me know and we can talk!

So with that out of the way, let’s see how you can configure nginx to work with unicorn.

  1. Install nginx via the package manager. (For AMI Ec2:sudo yum install nginx, For Ubuntu : sudo apt-get install nginx)
  2. Use this template for nginx unicorn configuration and then fill in only the required details.
  3. Note that the way it differs from thin configuration is by giving the server value as a location to the socket.
worker_processes 1;
user nobody nogroup;

pid /tmp/;
error_log /tmp/nginx.error.log;

events {
  worker_connections 1024; # increase if you have lots of clients
  # Set this to on if you have more than 1 working processes
  # This will allow only one child to watch the pollset and accept
  # a connection to a socket
  accept_mutex off; # "on" if nginx worker_processes > 1

http {
  include mime.types;
  default_type application/octet-stream;
  access_log /tmp/nginx.access.log combined;

  # This tells Nginx to ignore the contents of a file it is sending
  # and uses the kernel sendfile instead
  sendfile on;

  # Set this to on if you have sendfile on
  # It will prepend the HTTP response headers before
  # calling sendfile()
  tcp_nopush on;

  # This disables the "Nagle buffering algorithm" (Nginx Docs)
  # Good for websites that send a lot of small requests that
  # don't need a response
  tcp_nodelay off;

  gzip on;
  gzip_http_version 1.0;
  gzip_proxied any;
  gzip_min_length 500;
  gzip_disable "MSIE [1-6]\.";
  gzip_types text/plain text/html text/xml text/css
             text/javascript application/x-javascript

  upstream unicorn_server {
   # This is the socket we configured in unicorn.rb
   server unix:/var/rails/testapp/tmp/sockets/unicorn.sock »

  server {
    listen 80;
    client_max_body_size 4G;
    server_name _;
    keepalive_timeout 5;

    # Location of our static files
    root /home/ec2-user/testapp/public;

    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;

      # If you don't find the filename in the static files
      # Then request it from the unicorn server
      if (!-f $request_filename) {
        proxy_pass http://unicorn_server;
    }    error_page 500 502 503 504 /500.html;
    location = /500.html {
      root /home/ec2-user/testapp/public;

If you have unicorn correctly configured then start unicorn, restart nginx and you should be set.

Image upload to the PageDown (markdown) editor via AJAX in Rails

I have been using the updated version of the WMD markdown editor called as Pagedown for a blog engine, and wanted to have an Image Upload functionality.

If you take a look at Pagedown wiki, you will realize it has a very limited set of API’s – but has the one we need , a hook for ‘insert Image button’ called as insertImageDialog.

I had the following requirements for achieving Image Upload

  1. Markdown Editor (Pagedown)
  2. Bootstrap
  3. Rails + AJAX Uploads of Images.
  4. Preview of the image + ability to select size, before inserting into post.

Markdown Hook for Image Button.

The example from the Pagedown wiki, shows how the insertImageDialog.hook can be used.

editor.hooks.set("insertImageDialog", function (callback) {
      alert("Please click okay to start scanning your brain...");
      setTimeout(function () {
          var prompt = "We have detected that you like cats. Do you want to insert an image of a cat?";
          if (confirm(prompt))
      }, 2000);
     return true; // tell the editor that we'll take care of getting the image url

This tells us to register a plugin on the insertImageDialog hook that returns true (which tells the editor not to create its own dialog). so the callback method is the one which returns the actual URL of the uploaded images.

I recommend you try the above code, so that you know how the hooks are working within your editor.

Twitter Boostrap Markdown

This project is a pagedown (markdown) editor, styled for bootstrap. Take a look at the demo. I have made use of these pagedown code.

I would advice you to take a look at the source of this demo, to understand how all components tie-up.

RAILS + AJAX Uploads of Images + Image Preview and Re-sizing.

For doing AJAX uploads using a jQuery plugin and carrier-wave turned out to be easier because I went ahead with ‘jquery-fileupload-rails’ . Though there can be some challenges if you are using nested_forms. (This Rails-casts should help you figuring out the integration

With all the components laid above, I can walk you through the approach I took…

1. First Create a Modal for the dialog to appear on click of the insert Image..

<!-- Modal -->
<div id="fileModal" class="modal hide fade" tabindex="-1" role="dialog" aria-labelledby="fileModalLabel" aria-hidden="true">
 <div class="modal-header">
 <button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button>
 <h3 id="fileModalLabel">Upload Images</h3>
 <div class="modal-form">
 <div class="tabbable tabs-left"> <!-- Only required for left/right tabs -->
 <ul class="nav nav-tabs">
 <li class="active"><a href="#tab1" data-toggle="tab">Upload</a></li>
 <li><a href="#tab2" data-toggle="tab">Enter URL</a></li>
 <div class="tab-content">
 <div class="tab-pane active" id="tab1">
 <%= f.file_field :photo %>
 <div class="preview" id="preview" >
 <div class="tab-pane" id="tab2">
 <p><input type="text" class="img-url" placeholder="Ex:" id="img-url" /></p>
 </div> <!-- End of Modal-Body -->
 <div class="modal-footer">
 <button class="btn" data-dismiss="modal" aria-hidden="true">Close</button>
 <button class="btn" id="insert_image_post">Insert Image</button>
<% end %>

The above modal code would render like this. Image Upload Markdown Bootstrap Rails

2. Bind the Pagedown hook to show the above modal.

This is the easy part, as you saw above how the insertImageDialog hook behaves, we would link these together as.

<script type="text/javascript">
 (function () {
 var converter = new Markdown.Converter();
 var help = function () {''); }
 var editor = new Markdown.Editor(converter);
 editor.hooks.set("insertImageDialog", function (callback) {
 setTimeout(function () 
       $("#insert_image_post").click(function(e) {
            var images;
            images = "" + ($(this).filter("[src]").attr("src"));
       else {
          var image=$("#img-url").val();
 }, 0);
 return true; // tell the editor that we'll take care of getting the image url

To actually see how Ajax Jquery Uploads and carrierwave would interact you can look at the Railscasts .

For the entire working code, look at the following repository.

Highly Scalable Blog

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. This aspect of NoSQL is well-studied both in practice and theory because specific non-functional properties are often the main justification for NoSQL usage and fundamental results on distributed systems like the CAP theorem apply well to NoSQL systems.  At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques.

I would like to thank Daniel Kirkdorffer who reviewed the article and cleaned up the grammar.

To  explore data modeling techniques, we have to start with a more or less systematic view of NoSQL data models that preferably reveals trends and interconnections. The following figure depicts imaginary “evolution” of the major NoSQL…

View original post 3,581 more words

Setting up Rails with Nginx + Thin on Amazon EC2 with Capistrano

1. Setting up Rails 3.2.12 on EC2 with Amazon Linux AMI

Amazon Linux AMI has Ruby 1.8.7 as default. To install ruby 1.9.3 and Rails 3.2.12 do the following

#Install ruby 1.9 by running 
sudo yum install ruby19

Note that /usr/bin might contain  ‘gem’ which actually refers to 1.8 version. So, in order to install gem1.9 do the following

sudo yum install ruby19-devel

If you do ruby -v you might still see 1.8, in order to change that,

cd /usr/bin
sudo rm ruby
sudo ln -s /usr/bin/ruby1.9 /usr/bin/ruby
sudo ln -s /usr/bin/gem1.9 /usr/bin/gem
ruby -v 
>> ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-linux] 

Now that you have ruby 1.9.3 correctly installed, we can install rails.

sudo yum groupinstall "development tools"     #(Download size: 120M !)
gem install rails --no-ri --no-rdoc
rails -v
>> Rails 3.2.12

#Also install appropriate DB that you would be using. Look here if you want to install postgresql on ec2 
#Example, if using sqlite3
sudo yum install sqlite-devel

Congratulations, you have installed Rails 3+ on your Linux AMI.

2. Installing Nginx

sudo yum install nginx

3. Install Thin

gem install thin

4. Configure Git Repository

Let’s say we want to deploy a Rails App called as  rails-demo , for that lets initialize a git-repo as

 mkdir -p ~/git/rails_demo.git
 cd ~/git/rails_demo.git
 git --bare init

This would have initialized a git repository at ~/git/rails_demo.git/ .

5. Add server keys to authorized_keys [Deployment + Git Server === EC2 Linux AMI]

We would need to do this, since Capistrano would be accessing our code-repository as if it were remote, and hence the server-keys would have to be added to its own authorized keys.

test -e ~/.ssh/ || ssh-keygen -t dsa
cat ~/.ssh/ >> ~/.ssh/authorized_keys

6. Rails Deployment using Capistrano [Local Machine from where you want to push a rails application]

Lets say, we have a bare bones rails app created using only rails new <appname>.

rails new rails_demo
cd rails_demo
# Add the following to the Gemfile
gem 'capistrano'
gem 'thin'
gem 'bigdecimal'
# now run bundle install to install these 2 gems.
bundle install

A) Capify Deploy.rb

#In your rails application directory, run

capify . 

#[add] writing './Capfile'
#[add] writing './config/deploy.rb'
#[done] capified!

Go to deploy.rb and edit it as below. Only change the fields in bold

set :user, 'ec2-user'
set :domain, 'ec2-11-222-33-444.<region>'
set :application, "rails_demo"
set :repository, "#{user}@#{domain}:git/#{application}.git"
set :deploy_to, "/home/#{user}/#{application}"
set :normalize_asset_timestamps, false

role :web, domain # Your HTTP server, Apache/etc
role :app, domain # This may be the same as your `Web` serverrole :db, domain, :primary => true # This is where Rails migrations will run
default_run_options[:shell] = false
default_run_options[:pty] = true
# miscellaneous options
set :deploy_via, :remote_cache
set :scm, 'git'
set :branch, 'master'
set :scm_verbose, true
set :use_sudo, false
# Define all the tasks that need to be running manually after Capistrano is finish
namespace :deploy do
task :bundle_install, :roles => :app do
 run "cd #{release_path} && bundle install"
after "deploy:update_code", :bundle_install
 desc "install the necessary prerequisites"
 task :bundle_install, :roles => :app do
 run "cd #{release_path} && bundle install"
 run "cd #{release_path} && rake assets:precompile"end

*Note to change the fields in bold.

B) Thin Configuration

# Generate this thin configuration on local, which default the options to 3 thin 
# servers and production environment
thin config -C config/thin-config.yml -c ~/rails_app --servers 3 -e production

This would generate the config file with details as below. Change the chdir field to /home/ec2-user/rails_demo/current/  . (which is where the app would be deployed on ec2)

 chdir: /home/ec2-user/rails_demo/current/
 environment: production
 port: 3000
 timeout: 30
 log: log/thin.log
 pid: tmp/pids/
 max_conns: 1024
 max_persistent_conns: 100
 require: []
 wait: 30
 servers: 3
 daemonize: true

C) Git Usage

Add all these files to git by doing the following

git init .
git add .
git commit -m "rails demo to ec2"
git remote add aws ssh://ec2-user@<ec2-ami-name>/~/git/rails_demo.git/
ssh-add <local-key>
git push aws master

D. Capistrano Deployment

cap deploy:setup
cap deploy:check
cap deploy

If everything goes well, take a break – you deserve it since the deployment has been done and we only need to start / configure nginx now. If you see on the deployment server there would be current directory here

/home/ec2-user/rails_demo/current  – which contains the latest deployment, we did.

7. Configuring + Starting Nginx and Thin

Now, that the application is on the server, let’s start thin and nginx.

cd /home/ec2-user/rails_demo/current 
thin start -C thin-config.yml

(This would start 3 thin instance on ports 3000 , 3001 and 3002 .)

Now, all we need to do is redirect nginx to these ports.

cd /etc/nginx 
vi nginx.conf 

and add the below content at appropriate places. I have only shown the  part to be change in bold of the http section. Only this section needs to be changed for nginx.

http {
 include /etc/nginx/mime.types;
 default_type application/octet-stream;
 server_names_hash_bucket_size 128;
 log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
 keepalive_timeout 65;
#gzip on;
upstream thin {
server {
 listen 80;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
location / {
 root /home/ec2-user/rails_demo/current/public/;
 #root /usr/share/nginx/html;
 #index index.html index.htm;
# First attempt to serve request as file, then
 # as directory, then fall back to index.html
 try_files $uri $uri/ /index.html;
 # Uncomment to enable naxsi on this location
 # include /etc/nginx/naxsi.rules
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;
if (-f $request_filename/index.html) {
 rewrite (.*) $1/index.html break;
if (-f $request_filename.html) {
 rewrite (.*) $1.html break;
if (!-f $request_filename) {
 proxy_pass http://thin;
 } }
# redirect server error pages to the static page /40x.html
 error_page 404 public/404.html;
 location = /40x.html {
 root /usr/share/nginx/html;
# redirect server error pages to the static page /50x.html
 error_page 500 502 503 504 /50x.html;
 location = /50x.html {
 root /usr/share/nginx/html;
include /etc/nginx/conf.d/*.conf;

Save this file, and

sudo service nginx start


Software Licensing Basics Part 2

On September 27, 1983, computer programmers logging on to the Usenet newsgroup net.unix-wizards encountered an unusual message. Posted in the small hours of the morning, 12:30 a.m. to be exact, and signed by rms@mit-oz, the message’s subject line was terse but attention-grabbing. “New UNIX implementation,” it read. Instead of introducing a newly released version of Unix, however, the message’s opening paragraph issued a call to arms:

Starting this Thanksgiving I am going to write a complete Unix-compatible software system called GNU (for Gnu’s Not Unix), and give it away free to everyone who can use it. Contributions of time, money, programs and equipment are greatly needed.

Free as in Freedom, Sam Williams

(When looking at various aspects of open source – its important to understand the backdrop which led to the creation of such licenses. ‘Free as in Freedom‘ (available online) is a wonderful read.)

… Continuing from Part 1..

1. What are the most widely used licenses?

The first version of the GPL appeared in 1988, it didn’t get too much attention until authors, including Linux creator Linus Torvalds, started using it. At this point, the realization of its implications started sinking in. For corporations, that were used to doing business on closed software – this was a truly dangerous idea.

Several derivatives of that license have evolved over this period…

Public Domain: Not really a license. But equivalent to relinquishing any rights to your code under copyright law.

BSD / MIT: Oldest and most liberal licenses available. They basically put no restrictions on how the software can be used. Ex: FreeBSD

GPL: Allow people to freely use your software as long as they don’t charge for it and use the same license for parts of the program that they wrote themselves.

LGPL: A derivative of GPL, LGPL includes an exception that is intended to allow code that is released under other licenses to co-exist with and call the LGPL code. It was meant to be a compromise between the strong copy-left GPL and permissive licenses such as the BSD licenses and the MIT License.

MPL/CDDL: MPL is used by Mozilla, Firefox, and many other projects. CDDL is based on MPL with a few minor tweaks to make it more applicable outside of Mozilla. CDDL is used by many Sun products such as Solaris. The MPL allows source code to be mixed with other files under a different license (also proprietary). However, code files licensed under the MPL must remain under the MPL and freely available in source form.

CPL/EPL: CPL was derived from the old IBM Public License and was also influenced by MPL. CPL was originally used by Eclipse, but that project switched to EPL. EPL is a newer version of CPL with some improvements in its patent language to make it more acceptable to businesses.

Dual License: This isn’t really a license, it just means making your software available under two or more different licenses.

It’s important to note a point as mentioned in slashdot comment

The GPL license is conducive to liberating software.
The BSD license is conducive to liberating people.
With the GPL license, the software maintains more of the freedom than the programmers who work on it.
With the BSD licenses, the programmers maintain more of the freedom with what they are allowed to do with derivative code.

Public Domain BSD / MIT ASLv2 (Apache) GPL (v2) LGPL MPL / CDDL CPL /EPL
Protected by copyright No Yes Yes Yes Yes Yes Yes
Can be used in closed source projects Yes Yes Yes No* Yes Yes Yes
Code using this can be commercially used Yes Yes Yes No* Yes Yes Yes
Bug Fixes, modification to be released No No No Yes Yes Yes Yes
Explicit Patent License No No Yes No (but v3 address this) No Yes Yes

2. Can I use code with those licenses – the way I want?
3. Can I derive / extend that code?
4. Would I have to make the source code available to other if I  do (3)?Comparison of License

5. As a developer how should I distribute my code?

It’s important to reason out – why are you releasing the code. Deciding that – makes the decision just a bit easier.

How Do I Choose A License? (pdf)

How Do I Choose A OSS License?

Here’s a ‘License Chooser‘ – I stumbled upon while trying to solve this license mystery myself. Hope that’s useful enough!

Recommended Further Reading

1. On the Decline of GPL

2. Choosing a license for FOSS

3. OSS License . How to pick and OSS License Part 1 Part 2

4. Comparison of open source licenses

5. Choosing a License and Applying It (eBook)