bcwaldon.cc

OpenStack Image Service - Grizzly Planning

bcwaldon@gmail.com (Brian Waldon) — Tue, 06 Nov 2012 00:00:00 -0800

So what is Glance?

As there has been some discussion about what Glance's place is in the OpenStack lineup, the first thing that should be made clear is what Glance intends to provide to its consumers. The primary objective of Glance is to publish a catalog of virtual machine images. Rather than own users' image data, Glance simply tracks where that data resides. Glance owns the metadata provided by users.

An important use case of Glance up until now has been the ability to stream image data through the Glance HTTP service. Glance will continue to support this to keep compatibility with the v1 and v2 API specifications, but the feature will not be supported in any future major versions of the API. Glance should not stand in the way of a user booting an image on a cloud - if a user prefers to stash an image in Swift and boot directly from that URI, it should be possible!

Grizzly Planning

The Glance team has a lot to tackle during the next release cycle. The major pieces have been outlined below...

Multiple Image Locations

In the interest of being more flexible with the storage locations of image data, Glance will allow users to provide several sources of image data for a single image record. An example use case would be for a cloud operator to dump a high-traffic image on an ephemeral volume for quick cloning at boot time, but keep a copy in a more persistent data store such as Swift in the case that the volume is lost.

Image Workers

Asynchronous image processing can be quite taxing to the API nodes, so Glance will provide a framework for offloading these processes to some type of worker. Glance already allows asynchronous image upload through the use of a 'copy_from' parameter, which will be the first process to be offloaded into this new framework. Two major features that have been discussed at previous design summits are image data conversion and inremental image coalescing - both of which can also be tackled once we develop this framework.

https://etherpad.openstack.org/GrizzlyImageWorkers

Incremental Images

In order to minimize data duplication and unnecessary network traffic, Glance will support the upload of 'incremental images.' An incremental image is a representation of the changed blocks of a disk with a link back to original image from which it was created.

Image Property Protection

Nova cares about several 'magic' properties of images in order to properly boot them - some of the major ones being block_device_mapping, kernel_id, ramdisk_id, instance_uuid and image_type. The problem with this is that the owner of an image can modify those properties at will, possibly rendering it un-bootable. Glance will allow trusted privileged clients such as Nova or the cloud Admin to set specific properties on images that are write- or read-protected from the image owner.

https://blueprints.launchpad.net/glance/+spec/api-v2-property-protection

OpenStack Images API v2.1

The v2.0 OpenStack Images API was released with Folsom, yet it lacks a few major features:

Image Sharing

It was decided that the image sharing capabilities (also known as 'image membership') of the v1 Images API needed to be revisited for the v2 API, but that design work was not completed by the time Folsom, and therefore the v2.0 API, was released. This will be tackled in the Grizzly timeframe.

Create From Location & Copy-From

Images can not yet be created from an explicit backend store location - this will be enabled based on the work for Multiple Image Locations. Additionally, the asynchronous image upload process will be added to the v2.1 API.

Domain Object Model

Glance has suffered a bit of a mis-balance between new code and refactoring the existing code. We plan to completely overhaul the internals of Glance with a domain object model that should help re-align the codebase.

https://blueprints.launchpad.net/glance/+spec/glance-domain-logic-layer

glance-registry vs glance-api

The v1 and v2 Images APIs were implemented with seperate paths to the Glance database. The first of which proxies queries through a subsequent HTTP service (glance-registry) while the second talks directly to the database. As these two APIs should be talking to an equivalent system, we will be realigning their internal paths to talk through the service layer (created with the domain object model) directly to the database, effectively deprecating the glance-registry service.

Upgrading OpenStack Glance - Essex to Folsom

bcwaldon@gmail.com (Brian Waldon) — Wed, 05 Sep 2012 00:00:00 -0700

As Glance consists of only two services, the upgrade process is pretty straightforward. This post walks through what I did to upgrade an Essex Glance deployment to Folsom.

Full disclosure - everything on my system has been installed from source.

Backup

This isn't a required step, but it's always good to keep a backup of config files. I moved all of my config and paste files from /etc/glance/ to /etc/glance.essex/:

etc/
    glance.essex/
        glance-api.conf
        glance-api-paste.ini
        glance-registry.conf
        glance-registry-paste.ini

Don't worry about backing up your actual image data - Folsom will be able to pick up right where Essex left off.

Configuration

The biggest pain of the upgrade is dealing with the paste configuration files. Thankfully, this should be the last release that you have to go through this process!

Copy the essex config files (not paste files) into a newly-created /etc/glance directory:

etc/
    glance/
        glance-api.conf
        glance-registry.conf
    glance.essex/
        glance-api.conf
        glance-api-paste.ini
        glance-registry.conf
        glance-registry-paste.ini

Copy the paste files provided by the glance source into the new config directory - not your old essex paste files!

etc/
    glance/
        glance-api.conf
        glance-api-paste.ini
        glance-registry.conf
        glance-registry-paste.ini
    glance.essex/
        glance-api.conf
        glance-api-paste.ini
        glance-registry.conf
        glance-registry-paste.ini

If you have any custom paste pipelines, now would be the time to copy them into your new paste files. I'm not going to cover this process since I assume you know what you're doing if you wrote your own pipelines.

Auth Paste Configuration

NOTE: Skip this step if you aren't using Keystone.

Move all of the auth_* and admin_* attributes in the [filter:authtoken] sections out of your paste files into new [keystone_authtoken] sections of the corresponding configuration files.

For example, I took the auth\_* and admin_* values out of /etc/glance.essex/glance-api-paste.ini, which looked something like this:

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = 127.0.0.1
service_port = 5000
auth_host = 10.0.2.15
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = secrete

And put them at the bottom of /etc/glance/glance-api.conf in a new [keystone_authtoken] section, which then looked like this:

[keystone_authtoken]
auth_host = 10.0.2.15
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = secrete

I did the same thing for /etc/glance.essex/glance-registry-paste.ini and /etc/glance/glance-registry.conf.

v2 API Configuration

With the Folsom release comes a new HTTP API, the OpenStack Images API v2.0. You must decide to either leave it on and configure it, or turn it off completely.

To enable the v2 API, simply copy your sql-related glance-registry configuration into your glance-api configuration file. The relevant bits of my /etc/glance/glance-api.conf looked like this:

[DEFAULT]
enable_v2_api = true
sql_connection = mysql://root:secrete@localhost/glance?charset=utf8
sql_idle_timeout = 1800
sql_max_retries = 10
sql_retry_interval = 1
db_auto_create = false

To turn it off, simply set the enable_v2_api flag to 'false'.

[DEFAULT]
enable_v2_api = false

Update Keystone

NOTE: Skip this step if you aren't using Keystone.

Since Glance depends on importing a piece of middleware owned by Keystone, you must update your local Keystone code to Folsom.

Since this walkthrough is aimed at Glance, I won't dive into what it takes to upgrade a full Keystone deployment. Here's what I did to make the latest Keystone code available to Glance:

$ pip install keystone --upgrade

Update Other Dependencies

Use the pip-requires file that ships with the Glance source code to determine what you should install or update. I ended up asking pip to install everything for me so I didn't miss any updated version requirements:

$ pip install -r tools/pip-requires --upgrade

I do want to highlight that there are two completely new dependencies that may or may not be packaged on your platform, jsonschema and python-swiftclient.

Stop Services

Now stop glance-api and glance-registry.

$ glance-control api stop
Stopping glance-api  pid: 29451  signal: 15

$ glance-control registry stop
Stopping glance-registry  pid: 29447  signal: 15

Upgrade Database

Run glance-manage db_sync to upgrade your database - it should end up at version 15:

$ glance-manage db_sync
2012-09-06 00:14:20     INFO [migrate.versioning.api] 13 -> 14... 
2012-09-06 00:14:20     INFO [glance.db.sqlalchemy.migrate_repo.schema] creating
table image_tags
2012-09-06 00:14:20     INFO [migrate.versioning.api] done
2012-09-06 00:14:20     INFO [migrate.versioning.api] 14 -> 15... 
2012-09-06 00:14:20     INFO [migrate.versioning.api] done

$ glance-manage db_version
15

Start Services

Now start glance-api and glance-registry

$ glance-control registry start
Starting glance-registry

$ glance-control api start
Starting glance-api

OpenStack Folsom & Glance

bcwaldon@gmail.com (Brian Waldon) — Mon, 20 Aug 2012 00:00:00 -0700

As we are coming up on the Folsom OpenStack release, I thought it would be a good idea to cover what landed in Glance over the last several months. It is becoming increasingly difficult to keep up with all of our projects, so I hope this overview helps!

Update 2012-09-12: Updated High-Profile Bugs to reflect bugs fixed in folsom-rc1

Python Client Rewrite

A new client library, python-glanceclient, was created during the Folsom release cycle to replace the client that currently lives in the Glance repository. This new client has already been integrated into Nova, Cinder, Horizon and Devstack.

As a library project, python-glanceclient does not follow the same release cycle as core projects such as Glance. It uses a three-part versioning mechanism and is distributed through OpenStack-supporting distros, pypi, and GitHub.

The importance bit is that the (now considered 'legacy') client code in Glance is officially deprecated in favor of this new client. It will still ship with Folsom, but it prints warnings to stderr when used from the command-line and raises UserWarning when imported directly in python. Expect the legacy client code to be removed from the Glance repository sometime early in the Grizzly release cycle.

The new client does maintain 100% CLI compatibility for interactions with the v1 Images API, but that legacy usage will never be updated to work with the v2 API. The python interface does not maintain compatibility with the legacy client, however, the python module does not conflict with that installed by the Glance source. Expect to see more about python-glanceclient in a subsequent post.

OpenStack Images API v2.0

Version 2.0 of the OpenStack Images API is largely defined based on the work that was finished in the Folsom release of Glance. Expect a separate post covering the v2.0 API spec itself.

Two new configuration options were added to control the availability of specific API versions: enable_v1_api and enable_v2_api. These are boolean flags that control whether an API should be deployed with glance-api. If you want to prevent users from accessing the new v2 API, just set enable_v2_api to False (it defaults to True).

I want to specifically thank Mark Washenberger, Alex Meade, Iccha Sethi and Nikhil Komawar of Rackspace for their outstanding work on the v2.0 API implementation!

See the super-blueprint on launchpad.

High-Profile Bugs

Features are always awesome, but the biggest request of the OpenStack community has been to work on stabilization and bug-fixing. The following is a list of major bugs that were fixed in the Folsom release cycle, many more lower-priority bugs were fixed and can be seen on launchpad. Bugs that affect stable/essex have been backported accordingly.

Keystone Integration

1010519: Use a case-insensitive check when determining if a user has admin privileges

Swift Integration

979745: Actually delete images from swift when using Keystone authentication.
1002791: Replace usage of swift.client with swiftclient - swift dependency replaced with python-swiftclient.
994296: Allow swift account name to contain the '@' character.
997658: S3 store backend now works with Swift's S3 compatibility middleware (Swift3) - use s3_store_bucket_url_format=path.

SSL

1032451: Allow server to validate client certificate using CA cert specified in ca_file config option.
1007093: Images less than 4 KB in size are now uploaded only once; data is no longer duplicated.

Database

1012823: Prevent database auto-creation - configurable using db_auto_create config option.
975651: Update all image id references in db migration 12; image properties were being ignored.

Policy

1036193: New image policy added to cover downloading image data: 'download_image'.
1043482: Missing policy files don't cause server to crash - a default set of rules is now used instead.

Miscellaneous

1038994: Maximum image size is now configurable using image_size_cap (default 1 TB) and chunked transfer-encoding HTTP requests are now properly validated.
978130, 1042823: Multiprocess-enabled servers now respect SIGINT and SIGHUP
1036193: Individual workers now get a personal database connection rather than sharing one.
1019421: Glance will clean up partially stored images if filesystem fills up - only applies to the filesystem store driver.
1031842: Partially-cached images are removed when a client connection is prematurely closed.
1021054, 1021740: Admins can now share images regardless of ownership or existing sharing permissions.

Show me the Features!

Tenant-Specific Storage Locations in Swift

The ability to store image data in the image owner-specific locations has always been an interesting feature on our backlog as storing all users' images in a single account is a no-go for some deployers. We ended up with a solution that is only implemented for Swift, but done in such a way that we can expand it easily during Grizzly.

The implementation adds a couple of configuration options:

swift_store_multi_tenant: this must be set to 'True' to enable this feature (it defaults to 'False').
swift_store_admin_tenants: this is a list of tenants, referenced by id, that should be granted read and write access to all swift containers created by Glance.

Assuming you configured 'swift' as your default_store and you enable this feature as described above, images will be stored in a swift endpoint pulled from the authenticated user's service_catalog. The created image data will only be accessible through Glance by the tenant that owns it and any tenants defined in swift_store_admin_tenants that are identified as having admin-level accounts.

Special thanks to Dan Prince of RedHat for tackling this feature!

See the blueprint on launchpad.

Image Replication Tool

At the Folsom OpenStack Design Summit, we decided that the best first-run at image replication would be to write a simple python script that could talk to multiple Glance endpoints and sync images between them. During this release cycle, we developed the glance-replicator tool.

It is currently only available through GitHub, but we hope to develop a better distribution method in Grizzly.

Once you have it installed, you should run the help command to determine specific usage: glance-replicator help. The tool already has a great list of features:

Replicate binary image data along with metadata
- Local replication will download image data to a local directory
- Live replication can circumvent local disk access
Compare Glance endpoints to see what images need to be replicated
Determine the effective size of a Glance endpoint
Prevent replication of specific metadata attributes
Provide separate authentication tokens for master and slave endpoints

Thanks a ton to Michael Still of Canonical for his fantastic work on this tool!

See the blueprint on launchpad.

Database Driver Pluggability

The internal pythonic interface used to interact with sqlalchemy models, a.k.a. the database driver, has limited deployers and developers to using only the backends that sqlalchemy itself supported (MySQL, PostgreSQL, SQLite, etc.).

The database driver has been made pluggable using the data_api configuration option. A deployer of Glance can use the default sqlalchemy API, or one could specify any python module path that implements the same interface.

The in-memory database driver, referred to as the 'simple' driver, has been promoted from a test helper to the first sqlalchemy alternative. It should only be used for testing and in extremely rare deployment configurations. It doesn't write anything to the local filesystem, so expect your data to be lost if you restart your services. It cannot share between services, so you are limited to a single glance-registry and glance-api process (the workers config must be 0). Additionally, data will not be shared between the v1 and v2 APIs.

If you are interested in writing a driver for your favorite backend, you can use the test suite defined in glance/tests/functional/db/__init__.py. See the test modules test_simple and test_sqlalchemy in that same directory for an example of how to integrate with the tests. Unfortunately, the testing suite is incomplete and cannot yet guarantee complete interface compliance. The sqlalchemy api is the de facto standard as of now, but that responsibility will transition to the test suite in the Grizzly timeframe.

See the blueprint on launchpad.

Store Driver Pluggability

Similar to how the database driver was made pluggable, the list of available image store drivers has been pushed into a configuration option known_stores:

known_stores = 
    glance.store.filesystem.Store,
    glance.store.http.Store,
    glance.store.rbd.Store,
    glance.store.s3.Store,
    glance.store.swift.Store,

If you want to write an additional store driver, you can implement the interface defined by the base class in glance.store.base.Store and append the pythonic path of your store class to the known_store option. Feel free to use that base class as your super class and define the methods that raise NotImplementedError, or write your own from scratch.

Testing is not as straightforward at the moment, so please refer to the files here for reference.

Thanks to Josh Harlow of Yahoo for making this happen!

See the blueprint on launchpad.

What about Grizzly?

While plans are far from being solidified for the next release cycle, I can throw a few teasers out there:

Protected Properties: Deployers will be able to set read/write permissions on individual image properties.
Privileged Services: Glance endpoints should be able to identify trusted clients (like Nova, Cinder or Horizon) and respond with additional information.
Domain Objects: The internal APIs of Glance have never been well-defined. We hope to create a proper domain object layer to help minimize code repetition in the API controllers and database drivers.
Images API v2.1+: Image sharing will make its way into the v2 API, along with several other yet-to-be finalized features.

Using Warlock & JSON Schemas

bcwaldon@gmail.com (Brian Waldon) — Sun, 19 Aug 2012 00:00:00 -0700

The warlock python library was created out of a need to interact with JSON schemas in a pythonic way. It is designed to generate self-validating python classes that conform to a JSON schema definition.

Installing warlock

Install it on Ubuntu 12.10:

$ apt-get install python-warlock

Install it using pip:

$ pip install python-warlock

Pull it from GitHub:

$ git clone http://github.com/bcwaldon/warlock
$ cd warlock/
$ python setup.py install

Using warlock v0.4.0

The first thing you'll need is a JSON schema. The following is a simple example, but warlock can support any valid JSON schema you throw at it:

>>> schema = {
    'name': 'Country',
    'properties': {
        'name': {'type': 'string'},
        'population': {'type': 'integer'},
    },
}

Creating a new warlock model is easy:

>>> import warlock
>>> Country = warlock.model_factory(schema)

The model_factory call returns your self-validating warlock model. Create a new instance of your model by passing in the necessary data:

>>> sweden = Country(name='Sweden', population=9453000)

Get and set attributes using dot-syntax or dict-syntax:

>>> sweden['name']
'Sweden'
>>> sweden.population = 9782000
>>> sweden['population']
9782000

The power of warlock is in the automatic validation. Trying to change an attribute of sweden to something the schema won't allow results in raising a warlock.core.InvalidOperation exception:

>>> sweden.population = 'N/A'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "warlock/core.py", line 63, in __setattr__
    self.__setitem__(key, value)
  File "warlock/core.py", line 58, in __setitem__
    raise InvalidOperation()
warlock.core.InvalidOperation

Attempting to create an instance of Country with a bad value for name raises a ValueError:

>>> Country(name=None, population=10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "warlock/core.py", line 37, in __init__
    raise ValueError()
ValueError

Accessing an attribute that doesn't exist raises an AttributeError:

>>> sweden.abbreviation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "warlock/core.py", line 47, in __getattr__
    raise AttributeError(key)
AttributeError: abbreviation

The changes attribute of your model instance returns a dictionary containing only the modified attributes:

>>> norway = Country(name='Norway', population=4952000)
>>> norway.changes
{}
>>>norway.population = 5013000
>>> norway.changes
{'population': 5013000}

Client-side magic

The use case that drove the creation of warlock was to be able to automagically verify client-side modifications to an entity before sending it back to a REST-like API. The python client we provide for v2 of the OpenStack Images API downloads a JSON schema and createa a warlock model representing a virtual image. The following shows how a user can download and modify an image using that python library:

>>>import glanceclient
>>>client = glanceclient.Client(…)

# An instance of a warlock model is returned below after retrieving
# the JSON schema and specific image data over HTTP
>>>image = client.images.get('12')

# Sample changes a client might make
>>> image.tags.append('ubuntu')
>>> image['min_ram'] = 512

>>>client.images.update(image)

The actual glanceclient code to fetch and update the image using warlock is as simple as this:

import warlock
import http

schema = http.get_schema('image')
model = warlock.model_factory(schema)

def get(image_id):
    image = http.get_image(image_id)
    return model(image)

def update(image):
    http.update_image(image.changes)

Future of warlock

There are a couple of major things on my wishlist at the moment:

Deleted Attributes: the 'changes' feature doesn't allow a user to remove attributes from an image. This is absolutely necessary when modifying entities.
Cleaner Exceptions: the mix of warlock-specific and standard python exceptions isn't a great experience. The code should use standard exceptions as much as possible and ensure it is returning the proper information.