Autoscaling in OpenStack Using Heat and Ceilometer, part 1

Posted: September 13, 2014 in Cloud Computing, Cool Projects, Heat, OpenStack, Software
Tags: , ,

Recently, I have been spending a fair amount of time tinkering with Red Hat Enterprise Linux OpenStack Platform 5 (RHEL-OSP 5) which is Red Hat’s Icehouse based offering of OpenStack.  My goal was to learn how to get OpenStack to scale workloads up and down as needed.  Elasticity like this is one of the essential characteristics of cloud computing as defined by the National Institute of Science and Technology (NIST), and is one of the capabilities that OpenStack has that traditional data center virtualization systems typically don’t possess. I was able to use the packstack utility to install OpenStack onto three virtual machines running on a Red Hat Enterprise Linux 7 based laptop.  With only 16GB of RAM in the laptop, the virtual machines are a bit limited, but it is still a good platform for learning and experimenting.  The three VMs were configured as follows:

Controller: 2GB RAM, 30GB HDD + 2x 5GB for Swift, 4vCPU
Neutron Network: 1GB RAM, 10GB HDD, 2vCPU
Compute: 8GB RAM, 110GB HDD, 8vCPU

The below graphic is from the OpenStack Installation Guide’s Example Architecture section and shows how the services and networks should be configured in an Icehouse deployment.

Notice that the Neutron Network node required three network interfaces.  In previous versions of OpenStack I was able to get everything working with just two networks interfaces, but with Icehouse I ended up using three networks and it seems to work pretty decently configured like that.

Once I had everything configured, it was time to turn my attention to Heat and Ceilometer and see if I could get a simple two-node (web server and database) WordPress website to scale the web server component if the CPU becomes busy.

Finding Heat Template and the Supporting Infrastructure

I thought it would be a good idea to go hunting for a Heat template that I could use as a starting point for my experimentation.  I found a decent looking auto-scaling template on GitHub and decided to start looking it over.

The template is pretty close to what I want OpenStack to do, and after looking it over, I knew that there were a few steps that I would have to do to finish setting up my infrastructure to support his stack deployment.  First, I had to locate the lb_server.yaml file that is referenced in the web_server_group section of the template which is also locate on GitHub.  Secondly, I would have to set up a local repo somewhere to host the RPM packages that are needed to do the actual installation of WordPress.  I determined that configuring a web server on the host (my laptop) and hosting both the repo and the lb_server.yaml file there would make the most sense.  Since this post is really about the Heat template, I will not go into how I set up the repo or the web server to host it, but know that it needs to be done for everything else to work.  You can set up your infrastructure however you think it will work best for you.

The Template

There were a few things that I wanted to change about the template which requires a few tweaks.  Below is the entire template broken into different sections.  I will point out the changes to that I made to the template that is available from GitHub and what each section does.  The entire template file is listed below.

heat_template_version: 2013-08-22

description: >
  HOT template to deploy two servers (database and WordPress web server) into an
  existing neutron tenant network and create a load balancer.  The load balancer
  will be assigned a floating IP addresses and balance traffic to the available
  web servers.  Depending on the CPU utilization, the number of web servers online
  will be scaled up or donw. 

parameter_groups:
  label: configuration_data
  description: These items pertain the the configuration of the instances.
  parameters:
    key_name:
      type: string
      description: Name of keypair to assign to servers
    image:
      type: string
      description: Name of image to use for servers
      default: rhel6.5-x86_64
      constraints:
      - allowed_values: [ rhel6.5-x86_64,rhel7-x86_64 ]
        description: Image ID must be either rhel6.5-x86_64 or rhel7-x86_64
    flavor:
      type: string
      description: Flavor to use for servers
      default: m1.small
      constraints:
      - allowed_values: [scale, m1.small, m1.medium]
  label: database_parameters
  description: These values are used to configure the database
  parameters:
    db_name:
      type: string
      description: WordPress database name
      default: wordpress
      constraints:
      - length: { min: 1, max: 64 }
        description: db_name must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z][a-zA-Z0-9]*'
        description: >
          db_name must begin with a letter and contain only alphanumeric
          characters
    db_username:
      type: string
      description: The WordPress database admin account username
      default: admin
      hidden: true
      constraints:
      - length: { min: 1, max: 16 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z][a-zA-Z0-9]*'
        description: >
          db_username must begin with a letter and contain only alphanumeric
          characters
    db_password:
      type: string
      description: The WordPress database admin account password
      default: admin
      hidden: true
      constraints:
      - length: { min: 1, max: 41 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_password must contain only alphanumeric characters
    db_root_password:
      type: string
      description: Root password for MySQL
      default: admin
      hidden: true
      constraints:
      - length: { min: 1, max: 41 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_password must contain only alphanumeric characters
  label: network_parameters
  description: These values are used to configure the load balancer and the instances.
  parameters:
    public_net_id:
      type: string
      description: ID of public network for which floating IP addresses will be allocated
      default: 57177826-f330-4aae-b5ad-822356d3a906
    private_net_id:
      type: string
      description: ID of private network into which servers get deployed
      default: 967938a8-78ae-49b3-bef6-e293e9a6751d
    private_subnet_id:
      type: string
      description: ID of private sub network into which servers get deployed
      default: b7cb50e6-b650-46ce-a891-66b858761a2d

resources:
  wp_dbserver:
    type: OS::Nova::Server
    properties:
      name: wp_dbserver
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - port: { get_resource: wp_dbserver_port }
      user_data:
        str_replace:
          template: |
            #!/bin/bash -v

            lokkit --port=3306:tcp
            
            cat << EOF > /etc/yum.repos.d/local.repo
            [wordpress]
            name=WordPress for Enterprise Linux 6 
            baseurl=http://192.168.0.1/repos/wordpress_rhel6
            enabled=1
            gpgcheck=0
            EOF

            yum -y install mysql mysql-server 
            chkconfig mysqld on
            service mysqld start
            
            # Setup MySQL root password and create a user
            mysqladmin -u root password db_rootpassword
            cat << EOF | mysql -u root --password=db_rootpassword
            CREATE DATABASE db_name;
            GRANT ALL PRIVILEGES ON db_name.* TO "db_user"@"%" IDENTIFIED BY "db_password";
            FLUSH PRIVILEGES;
            EXIT
            EOF
          params:
            db_rootpassword: { get_param: db_root_password }
            db_name: { get_param: db_name }
            db_user: { get_param: db_username }
            db_password: { get_param: db_password }

  web_server_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      min_size: 1
      max_size: 3
      resource:
        type: http://192.168.0.1/lb_server.yaml
        properties:
          flavor: {get_param: flavor}
          image: {get_param: image}
          key_name: {get_param: key_name}
          pool_id: {get_resource: pool}
          metadata: {"metering.stack": {get_param: "OS::stack_id"}}
          user_data:
            str_replace:
              template: |
                #!/bin/bash -v

                lokkit --service=http                

                cat << EOF > /etc/yum.repos.d/local.repo
                [wordpress]
                name=WordPress for Enterprise Linux 6 
                baseurl=http://192.168.0.1/repos/wordpress_rhel6
                enabled=1
                gpgcheck=0
                EOF

                yum -y install httpd wordpress
                setsebool -P httpd_can_network_connect=1
                chkconfig httpd on
                service httpd start

                sed -i "/Deny from All/d" /etc/httpd/conf.d/wordpress.conf
                sed -i "s/Require local/Require all granted/" /etc/httpd/conf.d/wordpress.conf
                sed -i s/database_name_here/db_name/ /etc/wordpress/wp-config.php
                sed -i s/username_here/db_user/ /etc/wordpress/wp-config.php
                sed -i s/password_here/db_password/ /etc/wordpress/wp-config.php
                sed -i s/localhost/db_address/ /etc/wordpress/wp-config.php

                service httpd restart
              params:
                db_name: { get_param: db_name }
                db_user: { get_param: db_username }
                db_password: { get_param: db_password }
                db_address: { get_attr: [ wp_dbserver, first_address ] }

  wp_dbserver_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_param: private_net_id }
      fixed_ips:
        - subnet_id: { get_param: private_subnet_id }

  web_server_scaleup_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: web_server_group}
      cooldown: 60
      scaling_adjustment: 1

  web_server_scaledown_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: web_server_group}
      cooldown: 60
      scaling_adjustment: -1

  cpu_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average CPU > 50% for 1 minute
      meter_name: cpu_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 50
      alarm_actions:
        - {get_attr: [web_server_scaleup_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: gt

  cpu_alarm_low:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-down if the average CPU < 15% for 10 minutes
      meter_name: cpu_util
      statistic: avg
      period: 600
      evaluation_periods: 1
      threshold: 15
      alarm_actions:
        - {get_attr: [web_server_scaledown_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: lt
  
  lb_vip_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_param: private_net_id }
      fixed_ips:
        - subnet_id: { get_param: private_subnet_id }

  lb_vip_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network_id: { get_param: public_net_id }  
      port_id: { get_resource: lb_vip_port }

  lb_pool_vip:
    type: OS::Neutron::FloatingIPAssociation
    properties:
      floatingip_id: { get_resource: lb_vip_floating_ip }
      port_id: { 'Fn::Select': ['port_id', {get_attr: [pool, vip]}]}

  monitor:
    type: OS::Neutron::HealthMonitor
    properties:
      type: TCP
      delay: 3
      max_retries: 5
      timeout: 5

  pool:
    type: OS::Neutron::Pool
    properties:
      protocol: HTTP
      monitors: [{get_resource: monitor}]
      subnet_id: {get_param: private_subnet_id}
      lb_method: ROUND_ROBIN
      vip:
        protocol_port: 80
        ## session_persistence:
        ##   type: SOURCE_IP

  lb:
    type: OS::Neutron::LoadBalancer
    properties:
      protocol_port: 80
      pool_id: {get_resource: pool}

outputs:
  WebsiteURL:
    description: URL for WordPress wiki
    value:
      str_replace:
        template: http://host/wordpress
        params:
          host: { get_attr: [lb_vip_floating_ip, floating_ip_address] }

Lets start looking at the code that makes up this template.  The first sections are mandatory and determine the version of Heat the template will be using.  Any dates after 2014-05-23 indicate it is a HOT (Heat Orchestration Template) file.  The second key is a description of what the template deploys.

heat_template_version: 2013-05-23

description: >
  HOT template to deploy two servers (database and web server) into an existing
  neutron tenant network and create a load balancer.  The load balancer will be 
  assigned a floating IP addresses to route traffic from the internet to the web
  servers.  Alarms ar create to scale the web server up and down based on CPU
  utilization rates.

The next section of the template is the parameters section.  This section defines all of the parameters that are used to deploy the instances that the Heat template has been designed to deploy.  The description of the data will appear when the user hovers over the parameter field in the OpenStack Horizon interface.  The parameters can include validation constraints as well as default values.  When more than one allowed value is given, the parameter will appear as a drop-down list in Horizon.  Since there are so many parameters listed, I provided some defaults for all of the ones I could such as image name and network parameters.

parameters:
  key_name:
    type: string
    description: Name of keypair to assign to servers
  image:
    type: string
    description: Name of image to use for servers
    default: rhel6.5-x86_64
    constraints:
      - allowed_values: [ rhel6.5-x86_64]
        description: Image ID must be rhel6.5-x86_64
  flavor:
    type: string
    description: Flavor to use for servers
    default: m1.small
    constraints:
      - allowed_values: [m1.small, m1.medium]
  db_name:
    type: string
    description: WordPress database name
    default: wordpress
    constraints:
      - length: { min: 1, max: 64 }
        description: db_name must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z][a-zA-Z0-9]*'
        description: >
          db_name must begin with a letter and contain only alphanumeric
          characters
  db_username:
    type: string
    description: The WordPress database admin account username
    default: admin
    hidden: true
    constraints:
      - length: { min: 1, max: 16 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z][a-zA-Z0-9]*'
        description: >
          db_username must begin with a letter and contain only alphanumeric
          characters
  db_password:
    type: string
    description: The WordPress database admin account password
    default: admin
    hidden: true
    constraints:
      - length: { min: 1, max: 41 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_password must contain only alphanumeric characters
  db_root_password:
    type: string
    description: Root password for MySQL
    default: admin
    hidden: true
    constraints:
      - length: { min: 1, max: 41 }
        description: db_username must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_password must contain only alphanumeric characters
  public_net_id:
    type: string
    description: ID of public network for which floating IP addresses will be allocated
    default: 57177826-f330-4aae-b5ad-822356d3a906
  private_net_id:
    type: string
    description: ID of private network into which servers get deployed
    default: 967938a8-78ae-49b3-bef6-e293e9a6751d
  private_subnet_id:
    type: string
    description: ID of private sub network into which servers get deployed
    default: b7cb50e6-b650-46ce-a891-66b858761a2d

The next section is called resources and this is where things really start getting interesting.  Since this section is pretty lengthy, it will be discussed in several parts.  In the first part that is defined is the database server.  This is not going to be scaled, so it is appropriate to define it as an OS::Nova::Server.  There are a few other parameters defined  on lines 86-92.  This includes the name of the server, and information that OpenStack Nova needs to deploy the instance such as the image, flavor, key and network that the instance will reside on.

Perhaps the most interesting part of this segment of the file is the user data section.  In this section a  bash script is included that does the installation and configuration of the database server.  This script is delivered to the instance as it boots up.  At the end of this section is a list of parameters that are used as variables withing the bash script.

resources:
  wp_dbserver:
    type: OS::Nova::Server
    properties:
      name: wp_dbserver
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - port: { get_resource: wp_dbserver_port }
      user_data:
        str_replace:
          template: |
            #!/bin/bash -v

            lokkit --port=3306:tcp
            
            cat << EOF > /etc/yum.repos.d/local.repo
            [wordpress]
            name=WordPress for Enterprise Linux 6 
            baseurl=http://192.168.0.1/repos/wordpress_rhel6
            enabled=1
            gpgcheck=0
            EOF

            yum -y install mysql mysql-server 
            chkconfig mysqld on
            service mysqld start
            
            # Setup MySQL root password and create a user
            mysqladmin -u root password db_rootpassword
            cat << EOF | mysql -u root --password=db_rootpassword
            CREATE DATABASE db_name;
            GRANT ALL PRIVILEGES ON db_name.* TO "db_user"@"%" IDENTIFIED BY "db_password";
            FLUSH PRIVILEGES;
            EXIT
            EOF
          params:
            db_rootpassword: { get_param: db_root_password }
            db_name: { get_param: db_name }
            db_user: { get_param: db_username }
            db_password: { get_param: db_password }

Notice on line 92, there is the code “get_resource: wp_dbserver_port”. This tells Heat to retrieve the value for the network port from another snippet of code called wp_dbserver_port.  That snippet of code starts at line 182 and assigns a network port and IP address to the database server.

  wp_dbserver_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_param: private_net_id }
      fixed_ips:
        - subnet_id: { get_param: private_subnet_id }

The next section is a bit more complicated and defines the web server group that will be used to scale the application.  The type that is assigned to this group is OS::Heat::AutoScalingGroup.  There are also properties defined for the minimum and maximum number of instances that OpenStack Nova should deploy in support of this group.

The user_data section, just like the same section for the database server, is responsible for building and configuring the web server instance.

  web_server_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      min_size: 1
      max_size: 3
      resource:
        type: http://192.168.0.1/lb_server.yaml
        properties:
          flavor: {get_param: flavor}
          image: {get_param: image}
          key_name: {get_param: key_name}
          pool_id: {get_resource: pool}
          metadata: {"metering.stack": {get_param: "OS::stack_id"}}
          user_data:
            str_replace:
              template: |
                #!/bin/bash -v

                lokkit --service=http                

                cat << EOF > /etc/yum.repos.d/local.repo
                [wordpress]
                name=WordPress for Enterprise Linux 6 
                baseurl=http://192.168.0.1/repos/wordpress_rhel6
                enabled=1
                gpgcheck=0
                EOF

                yum -y install httpd wordpress
                setsebool -P httpd_can_network_connect=1
                chkconfig httpd on
                service httpd start

                sed -i "/Deny from All/d" /etc/httpd/conf.d/wordpress.conf
                sed -i "s/Require local/Require all granted/" /etc/httpd/conf.d/wordpress.conf
                sed -i s/database_name_here/db_name/ /etc/wordpress/wp-config.php
                sed -i s/username_here/db_user/ /etc/wordpress/wp-config.php
                sed -i s/password_here/db_password/ /etc/wordpress/wp-config.php
                sed -i s/localhost/db_address/ /etc/wordpress/wp-config.php

                service httpd restart
              params:
                db_name: { get_param: db_name }
                db_user: { get_param: db_username }
                db_password: { get_param: db_password }
                db_address: { get_attr: [ wp_dbserver, first_address ] }

Another point of interest appears on lines 131 and 132 above.  On these lines, Heat calls for another configuration file, lb_server.yaml.  The code for this file is shown below.

heat_template_version: 2013-05-23
description: A load-balancer server
parameters:
  image:
    type: string
    description: Image used for servers
  key_name:
    type: string
    description: SSH key to connect to the servers
  flavor:
    type: string
    description: flavor used by the servers
  pool_id:
    type: string
    description: Pool to contact
  user_data:
    type: string
    description: Server user_data
  metadata:
    type: json
resources:
  server:
    type: OS::Nova::Server
    properties:
      flavor: {get_param: flavor}
      image: {get_param: image}
      key_name: {get_param: key_name}
      metadata: {get_param: metadata}
      user_data: {get_param: user_data}
      user_data_format: RAW
  member:
    type: OS::Neutron::PoolMember
    properties:
      pool_id: {get_param: pool_id}
      address: {get_attr: [server, first_address]}
      protocol_port: 80

Essentially what it does is use the parameters from the main configuration file, creates the instance and assigns it as a member of the load balancer pool.  We will be discussing the load balancer configuration later in this document.

The next section of the template creates other resources that are needed for the deployment to function properly.  The resources include network ports, virtual IP addresses (vip), scaling policies and alarm settings.  The first block in this section is snippet of code that creates the port and IP address of for the database server as was discussed earlier.

The next section is where the Heat and Ceilometer magic really take place.  These sections create the policies that Heat will use to determine how to scale the web server using the web_server_group that was defined earlier by either adding an instance or removing an instance of the web server.

  web_server_scaleup_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: web_server_group}
      cooldown: 60
      scaling_adjustment: 1
 
  web_server_scaledown_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: web_server_group}
      cooldown: 60
      scaling_adjustment: -1

After that, the Ceilometer alarms are defined. The alarms determine when Heat should scale the application by setting a threshold on a metric that Ceilometer monitors.  A  complete list of the measurements that Ceilometer monitors can be found at http://docs.openstack.org/developer/ceilometer/measurements.html.  In the template, the cpu_util meter is monitored.  If cpu_util goes over 50% for 1 minute, the scale_up policy is run and another web server instance is created.  If the cpu_util goes below 15% for 10 minutes, then a web server is removed.  These policies ensure that the application runs efficiently and that no resources are be consumed that are not needed.

  cpu_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average CPU > 50% for 1 minute
      meter_name: cpu_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 50
      alarm_actions:
        - {get_attr: [web_server_scaleup_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: gt
 
  cpu_alarm_low:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-down if the average CPU < 15% for 10 minutes
      meter_name: cpu_util
      statistic: avg
      period: 600
      evaluation_periods: 1
      threshold: 15
      alarm_actions:
        - {get_attr: [web_server_scaledown_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: lt

The next several blocks of code are responsible for the creation of the load balancer.  Specifically, these sections create the following resources for the HAProxy based load balancer:

  • Virtual IP port (lb_vip_port)
  • Floating IP Adress so external users can access the application (lb_vip_floating_ip)
  • The vip for the pool of web servers behind the load balancer (lb_pool_vip)
  • A health monitor for the web servers (monitor)
  • The pool of web server resources (pool).  The pool can be configured to support different forms of session persistence.  In the template I am using I have it configured for source IP persistence, meaning that the same source IP will be routed to the same web server instance.  I commented this out though to show that each webserver in the pool does respond to HTTP requests.
  • The actual Load balancer (lb)

The code behind all of this is listed below.

  lb_vip_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_param: private_net_id }
      fixed_ips:
        - subnet_id: { get_param: private_subnet_id }
 
  lb_vip_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network_id: { get_param: public_net_id }  
      port_id: { get_resource: lb_vip_port }
 
  lb_pool_vip:
    type: OS::Neutron::FloatingIPAssociation
    properties:
      floatingip_id: { get_resource: lb_vip_floating_ip }
      port_id: { 'Fn::Select': ['port_id', {get_attr: [pool, vip]}]}
 
  monitor:
    type: OS::Neutron::HealthMonitor
    properties:
      type: TCP
      delay: 3
      max_retries: 5
      timeout: 5
 
  pool:
    type: OS::Neutron::Pool
    properties:
      protocol: HTTP
      monitors: [{get_resource: monitor}]
      subnet_id: {get_param: private_subnet_id}
      lb_method: ROUND_ROBIN
      vip:
        protocol_port: 80
        ## session_persistence:
        ##   type: SOURCE_IP
 
  lb:
    type: OS::Neutron::LoadBalancer
    properties:
      protocol_port: 80
      pool_id: {get_resource: pool}

The last section simply returns (outputs) the url for the WordPress application and can be seen on the overview page for the stack that the Heat template creates.

A complete guide for Heat Templates can be found at http://docs.openstack.org/developer/heat/template_guide/

In the next part of this tutorial, we will deploy the template and run a few test to learn how it works and learn more about Heat and Ceilometer.  See you then!

Comments
  1. patzaks says:

    HOT made easy to understand. Waiting for part 2.
    Thank you very much.

  2. Pawan kumar says:

    Its very easy and clear explanation about auto scaling

  3. […] Autoscaling in OpenStack Using Heat and Ceilometer, part 1 […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s