Scratch Paper

Azure: Fabric Controller

2009-10-15T09:51:00.000-07:00

In Azure, the Fabric Controller's view of a deployed service is a bit like this:

There is a logical service, which consists of several logical roles (e.g., for our example application, the service was the overall application and the roles included a Web and two Worker roles). There may be multiple instances of any single Logical Role, each of which is a Logical Role Instance (LRI). In the above picture, there are three instances of the Logical Role R_2. Each LRI is mapped to a single Logical Node, which is a hosting environment (e.g., VM). There may be multiple Logical Nodes (VMs) running on any given Physical Node or computer.

The purple ovals correspond to information provided by the application developer. That is, in the configuration (service model) for this app, the developer provides the Service Description, a Role Description for each role (which provides attributes of that role, such as, what hosting environment is required, how many resources are needed, and so on), and a Role Instance Description. The Fabric Controller maps each Role Instance Descriptor to its corresponding Logical Role Instance. Similarly, each Role Descriptor is mapped to the appropriate Logical Role. (Note that not all of these connections are shown in the above picture.)

In this way, deploying a service consists of mapping the graph describing an application's topology (as provided in that app's service model) TO the graph describing the inventory managed by the FC.

Driving the logical nodes

When an LRI is bound to a logical node (VM), that logical node's goal state is set to be whatever the LRI's goal state is. For example, Logical Node LN_22 in the above has a goal state which corresponds to LRI_22's goal state. The logical node also has a current state, which is obtained from the physical node on which this VM is running. The current state is kept up-to-date by communicating with the underlying physical hardware.

They don't say how it's constructed but evidently — in addition to knowing the current state and the goal state for each VM (OK) — the state machine for the corresponding LRI is also known. A state machine identifies all the possible states an instance can be in as well as what events cause transitions between what states. There is a stream of events — obtained via polling or interrupts — which provide updates on the physical hardware. These events are used — along with the state machine — to determine the current state. Periodically, the FC figures out what the appropriate next steps are given: the current state, the state machine, and the goal state. Then those actions are performed.

Identifying when there is a problem and handling it (automatically)

The FC maintains a cache of what it believes is the current state of each node. The Fabric Agent (which lives on each node) communicates with the FC to help the FC keep this cache updated. The FC detects when a Role dies. Multiple sources participate in monitoring Role health (and notifying the FC when a Role is not healthy): the Load Balancer issues probes to machines (i.e., pings them), a Role can also notify the FC that it (the Role) is unhealthy. When the FC learns (through whatever channel) that some Role is not healthy, the FC updates the current state for that node. Then the appropriate next steps to take — to get that node closer to its goal state — are determined and undertaken.

When a node goes offline, an attempt is made to recover that node. If the reason was a hardware failure (or, generally, something which cannot be remediated automatically) the role is migrated to another node.

From: http://channel9.msdn.com/pdc2008/ES19/

(Note that the above exposition would have benefited greatly from a few concrete examples — as far as what a typical goal state is, what a typical current state is, how the state machine (which identifies what's needed to transition between states) for a logical role instance is created, and so on.)

More detailed notes on Azure architecture

2009-10-14T15:42:00.000-07:00

Below we have some notes from: "Introducing the Windows Azure Platform" (by David Chappell)
http://go.microsoft.com/fwlink/?LinkId=158011 as well as from the video presentation (from PDC 2008) by Erick Smith and Chuck Lenzmeier, "Under the Hood: Inside the Windows Azure Hosting Environment " http://channel9.msdn.com/pdc2008/ES19/. The PPNT slides associated with that video lecture can be found here: http://mschnlnine.vo.llnwd.net/d1/pdc08/PPTX/ES19.pptx.

On:

General comments
A motivating app
Azure OS Compute — Roles, Agents, What an app consists of
Azure OS Storage — Blobs, Tables, and Queues. How one can store, access, and update data using the Azure OS.
The Fabric Controller: manages application provisioning (resource allocation), deployment, monitoring, and maintenance (including identifying when an app instance has gone down and needs to be restarted, when other hardware needed for the app has failed, when an app needs to scaled up or down in response to demand).
The service model: a contract between the application creator and the Azure platform. A way for the app creator to define how he wants his app to be managed. The FC enforces the Azure-side of this contract — ensuring the app stays "healthy," according to the definition of health provided by the app's creator.
The service lifecycle — and its constituent parts.
Miscellaneous Other — including security

General comments
At a high level of abstraction, think of the Azure Platform as a gigantic computer — though in actuality it consists of many physical computers, switches, routers, load balancers, and so on. A single logical OS runs over all of these different pieces of hardware and manages them. So in the same way that a user adds a physical device (such as a NIC) to his PC — by installing the driver for that device — the Azure OS adds switches, computers, and so on via installing the driver for the device. The driver in this case may not necessarily be the software that ships with the hardware (as would be the case when a user installs some hardware on his PC) but instead some Azure-specific code which lets the Azure OS monitor and fully manage the given machine, whatever it may be.

As noted, a user can run an app on Azure OS or store data using Azure OS or both. If he runs an app on Azure OS, he can perform horizontal scaling — that is, run more than one instance of the same app on different machines. In effect the app is replicated across different resources so that the overall app bandwidth (e.g., ability to perform a distributed computation or serve clients) is increased.

An app itself can consist of more than one "role," where there are front-end roles (those which receive HTTP/S connections) and back-end roles (those which do not accept incoming network connections; they are more processing or computation focused). So there can be replication of a particular role (e.g., create a new instance of the front-end "Web" role for every 1500 new client connections I receive) or of the entire app itself (e.g., create another instance of this entire web application, where that instance will include some number of instances of front-end and back-end roles). (CONFIRM THIS: it is certainly the case that one can replicate instances of particular roles; unclear whether replication can also be done at the level of an entire app, i.e., at app-granularity.)

A motivating app — for concreteness
I think our exploration of Azure would benefit greatly from considering some particular example app. So let's imagine the following app (that I just made up right this second): a credit-checking app used by landlords to determine whether or not to rent to some would-be tenant. The front-end of the app accepts a connection from a landlord who does two main things (via filling out a Web form): (1) provides all of the information about the would-be tenant (name, age, birthdate, SSN, previous addresses, previous landlords, references, ...); (2) provides order-fulfillment information (identifying where the finished report should be sent, how the landlord is going to pay for use of the service, and so on).

Let's assume we can use a single front-end role to accept this information. Depending upon the design of the app, the landlord may need to submit more than one form (e.g., one form for info about the would-be tenant and a second form for order fulfillment info). For simplicity, let's assume there's a single form and a single front-end role. Naturally, the number of instances of that front-end role should depend upon the level of demand for the service at any given instant of time. (Note that an app can increase the number of front-end instances, back-end instances, or both. Similarly, an app can "shed load" by killing instances of front- or back-end roles.)

Then there are at least two back-end roles: one for payment processing and a second for generating and supplying the report. The back-end roles can make outbound network connections, as necessary, to process the credit card payment or to retrieve info on the would-be tenant (even to invoke other Web services to perform some part of the due diligence involved in report generation), and so on.

Clearly there must be some information sharing across these roles; the info gathered by the front-end role needs to be made available to the back-end roles. Further, if there are multiple instances of each back-end role, those instances need to cooperate so that they don't duplicate one another's work. There are a number of different ways to structure this workflow; an obvious approach is for the front-end role to populate a record or document with the info it has gathered then to add that record/document to a queue — where the back-end roles take from this queue. So there's the familiar producer/consumer relationship there. Clearly, we want some kind of notification to be sent to idle Workers whenever a new job is available.

I'm not sure whether it's even possible for the Web role to create a network connection directly to the Worker role. We know that the outside world cannot connect to the Worker role; not sure whether that prohibition also extends to internal network connections. Depending upon the support for real-time notification of Workers that work is available, it might be appealing to be able to directly poke a worker. Some tradeoffs in polling versus interrupt-based notification but no obvious new twists on those tradeoffs presented here.

Azure OS Compute
True of both a Front-End Role and a Back-End Role:

Runs in a VM.
Can interact with Azure Storage:

Read/write messages from/to a queue.
Read/write data from/to a table.
...

Interacts with an Agent, which runs in the same VM.
Can write to / read from the local filesystem (within the VM); such changes will not persist across reboot though.

A Front-End Role (referred to in MSFT literature as a "Web Role")

Receives HTTP/S connections.
Runs on IIS7 (in a VM).
Is stateless with respect to clients. What does that mean? Each separate "request" from a client (e.g., HTTP GET or POST message) could be routed to a different front-end role. Hence, all important client state must either be returned to the client (so that he can present that info with each request he makes) or put in the database (where all front-end roles can access the info). Clients can retain and re-present state via maintaining it in a URL parameter or cookie.

A Back-End Role (referred to in MSFT literature as a "Worker Role")

Does NOT receive any incoming network connections.
Can make outbound network connections.

A Windows Azure Agent

Runs in the same VM as the front- or back-end role.
Exposes an API which the Roles can invoke in order to:

Write to the log
Send an alert to the application owner
...

Is the way that an app can interact with the Fabric Controller.

An app consists of:

N instances of Front-end Roles (can be zero)
M instances of Back-end Roles (can be zero)
A "hardware" load balancer (only necessary if have >1 Web or front-end role)
A config file

How many Web role (front-end) instances
How many Worker role (back-end) instances

A model: which defines what it means for this app to be healthy, the ways in which the app can be unhealthy (i.e., constraints which, when violated, indicate unhealthiness), remedial actions to take when an app is unhealthy (e.g., healthy means 1500 or fewer client connections per Web role; if all Web roles are currently managing 1500 client connections and the app receives another client connection, create a new instance of the Web role and send the new client to that instance).
A unified log which contains messages received both from Web and Worker instances.

Azure Storage

"All info held in Azure Storage is replicated three times." Provides strong consistency.

Every blob, table, and queue is named using URIs and accessed via standard HTTP operations. So these objects can be accessed locally (by Worker and Web Roles running in the same data center, for example) or they can be accessed across the WWW. Can use REST services to identify and access each Azure Storage data object. There are libraries which encapsulate the REST protocol operations (e.g., ADO.NET Data Services or Language Integrated Query (LINQ)) or one can make "raw HTTP calls" to achieve the same effect. If query a table stored in Azure Storage and that query has lots of results, can obtain a continuation object which lets you process some results then get the next set of results, and so on. So a continuation object is like a list iterator.

Blobs: a blob consists of binary data. Can be upto 50GB each. Can have associated metadata.

A storage account can have multiple containers; a container cannot contain another container (i.e., containers can't be nested). A blob name can contain slashes though — so can create the illusion of hierarchical blob storage.
Each container (akin to a file folder) holds blobs.
Can subdivide a blob into blocks then if there is a network failure while that blob is being downloaded, don't have to re-download the entire blob — just the blocks that weren't successfully delivered the first time.
Each container can be marked public or private.

Public blobs: can be read by anyone; writing requires a signed token.
Private blobs: both read and write operations require a signed token.

A blob can be reached by:
http://<StorageAcct>.blob.core.windows.net/<Container>/<BlobName>

Table: actually a set of entities, each of which has a set of properties (and values for those properties).

Access table via conventions defined in ADO.NET Data Services. Cannot query a table in Azure Storage using SQL. And a table cannot be required to conform to a particular schema.
The reason for this non-relational representation is that it makes it easier to "scale out storage," i.e., to spread the data in the table across machines. Harder to do this if the data is stored in a relational table (what's hard is probably performing aggregate functions over the entire table, which is not allowed for tables Azure OS Storage).
An Azure table can contain billions of entities holding terabytes of data.
A table consists of some set of entities.

Each entity consists of some set of properties.

Each property consists of a Name, Type, and Value.

So in a way, we can think of a table as a nested structure: every property conceptually consists of a table which holds three tuples, in particular those which give the property's name (<Name, NameVal>), value (<Value, ValueVal>.), and the type of that value (<Type, TypeVal>). Supported types include Binary, Bool, DateTime, Double, GUID, Int, Int64, String. A property's type can change over that property's lifetime, depending upon what value is currently stored for the property.

For example, some property p_12 might be represented by tuples: <Name, "CreationTime">, <Type, "DateTime">, and <Value, "12-01-2009 05:43:22 GMT">. So this property (p_12) says that the entity that it describes was created on December 1st, 2009, early in the morning (which is interesting since that date has not yet arrived but no mind!).

The next layer of nesting is that an entity contains a table consisting of all of that entity's properties. The final layer of nesting is that an actual Azure Storage Table consists of some set of entities and each entity's associated properties table.

Each entity can be upto 1MB in size. Accessing any part of an entity results in accessing the entire entity.

Queue: the primary purpose of a queue is to provide a way for Web roles and Worker roles to communicate with one another. A queue contains messages, the format of which can be application-defined. Each message can be upto 8KB. A Worker role reads a message from the queue — which does not result in the message being deleted. Rather, after a message is read, that message is not shown to anyone else for 30 seconds. Within that 30 seconds, the Worker who read the message could have handled the task and deleted the message. Or that Worker might have died before completing the task, in which case, the message would be shown to other Workers after 30 seconds have elapsed. This 30-second period is referred to as the visibility timeout.

Note that a message might be delivered multiple times. Also, there is no guarantee on the order in which messages are delivered. Presumably this means that a particular message on the queue might get "starved," i.e., never delivered to any Worker (in contrast to traditional FIFO queue semantics).

Azure Storage — Access control: it sounds like they use a token-based system in order to control who can access what data. In particular, when a user creates a storage account, she's given a secret key. Every request made to the storage account (to modify account settings, to access some blob stored as part of this account, to edit a table, and so on) carries a signature which is signed by this secret key (like a token). Having such a token means that you can access all the data stored in that account (as well as the account settings themselves?); i.e., the granularity at which access control decisions are made is account-level. Presumably, a user Alice who owns an account can provide a signed token to other users whom Alice wants to allow to read/modify stored data (CONFIRM.) Regrettably, it appears that Alice has to let these other users have all types of access (create, modify, delete) to everything (all data as well as account settings?). That said, part of the Windows .NET Cloud Services is access control so perhaps this could be layered on top of storage in order to achieve a more fine-grained access control system? (CHECK.)

Update: there is evidently now a way to provide container-level or blob-level access (rather than only being able to give account-wide access). See: http://blog.smarx.com/posts/new-storage-feature-signed-access-signatures

Windows Azure storage operates with “shared key authentication.” In other words, there’s a password for your entire storage account, and anyone who has that password essentially owns the account. It’s an all-or-nothing permission model. That means you can never give out your shared key. The only exception to the all-or-nothing rule is that blob containers can be marked public, in which case anyone may read what’s inside the container (but not write to it).

Signed access signatures give us a new mechanism for giving permission while retaining security. With signed access signatures, my application can produce a URL with built-in permissions, including a time window in which the signature is valid. It also allows for the creation of container-level access policies which can be referenced in signatures and then modified or revoked later. This new functionality enables a number of interesting scenarios...

Azure Storage — Concurrency: The basic model appears to be that a user or app would download a blob or entity then modify that blob/entity locally. After that, the user/app would attempt to commit those local changes — by writing the updated version back to storage. So, what happens if more than one app attempts to modify the same storage object (e.g., blob or table) at the same time? The first writer wins. Subsequent writes (of the same version) will fail. This is a form of optimistic concurrency and is achieved via maintaining version numbers. Note that an app can force his changes to stick by unconditionally updating an entity or blob.

The Fabric Controller (FC)
One per datacenter. Is replicated across several machines (maybe 5—7). Owns all of the resources in the fabric (including computers, switches, load balancers). Communicates with each machine's Fabric Agent. Every physical machine has a Fabric Agent running on it which keeps track of how many VMs are running on this machine, how each Azure app in each VM is doing, and so on. The FA reports this info back to the FC which uses it (along with the app's configuration file or model) to manage the app so that the app remains healthy (as defined in the app's configuration file).

The FC monitors all running apps, manages its own infrastructure (e.g., keeps VMs patched, keeps OSs patched, ...), and performs resource allocation using its global view of the entire fabric along with the configuration file for the app that needs to be deployed. All of this is done automatically. The FC presently (i.e., in the CTP) allocates VMs to processor cores in a one-to-one manner (that is, no core will have more than one VM running on it).

The FC maintains a graph which describes its hardware inventory. A node in the graph is a computer, load balancer, switch, router, or network management device. An edge is a networking cable (e.g., connecting a computer to a switch), power cable, or serial cable. When an app needs to be deployed, the FC may allocate an entire physical machine or, more commonly, may subpartition that machine into VMs and allocate one or more VMs to the given role. The size of the allocation depends upon the configuration settings for that role.

The service model
Is "declarative," which means rule-based. So the model or specification consists of a set of rules. Those rules are used by the Fabric Controller in initially allocating/provisioning resources for your service and to manage the service over its lifetime (i.e., maintain its health). The model contains info that you might otherwise communicate to the IT department that was going to deploy and run/manage your service in-house. This technique/setup is referred to as model-driven service management.

What it looks like:

Contains the topology of your service (which comprises a graph):

Which roles? How many instances of each?
How are the roles linked (i.e., which roles communicate with which other roles)?
What interfaces exposed to the Internet?
What interfaces exposed to other services?

For each role, defines attributes of that role, e.g.,

What hosting environment does that role require? (E.g., IIS7, ASP.NET, CLR, ...)
How many resources does that role need? (e.g., CPU, disk, network latency, network bandwidth, ...)

Configuration settings: all are accessible at run-time. Can register to be notified when the value of a particular setting changes.

Application configuration settings: defined by the developer. Can think of these as being akin to command-line arguments that one would pass to a program. The app will behave differently depending upon the values of these arguments. Can use these to create different versions of your app where the versions differ on the basis of their values for these configuration settings.
System-defined configuration settings: predefined by the system.

How many fault domains?
How many update domains?
How many instances of each role?

In the CTP, don't get to use entire service-model language. Instead, will be provided with various templates; can choose one and customize it for your particular app. Eventually, more flexibility (a higher level of control over service model configuration) will be exposed.

The service lifecycle

[Developer] Develop code. Build/construct service model.
[Developer/Deployer] Specify desired config.
[FC] Deploy app; FC maps app to hardware in the way specified in the config/model.

Resource allocation: choose hardware to allocate to the app based on the app's service model. Is basically a ginormous constraint satisfaction problem. Performed as a transaction: all resources allocated for an app or none are. Example constraints include:

Question: Are there both system constraints and application-defined constraints? Then both sets are combined? For example, some of the sample constraints seem like something that the system would specify, such as "Node must have enough resources to be considered" or "Nod must have a compatible hosting environment" whereas others seem more app-specific, as in "Each node should have at most a single instance of any given role assigned to it."

Provisioning: Each item of allocated hardware is assigned a new "goal state." The FC will drive each piece of hardware to its goal state.
Upgrades:

[FC] Monitor app state and maintain app health, according to the service model and the SLA.

Handle software faults, hardware failures, need to scale up/down.
Logging infrastructure gathers the data needed to diagnose an app as unhealthy (where "health" is application-defined and provided in the app's service model).

Miscellaneous Other
Development environment allows desktop simulation of the entire Azure cloud experience. Can run your app locally and debug that way.
Can specify which data centers your app runs in and stores its data in.
Two-stage deployment to the cloud (which can also be used to achieve zero-downtime upgrades):
Upload application, access via .cloudapp.net, where cloudapp.net points to a load balancer (i.e., the DNS entry contains the load balancer's virtual IP (VIP)). Then connections to the app will be routed through this load balancer.
Presumably anyone can connect to the app (i.e., it's world reachable) but not everyone knows the particular globally unique ID associated with this app?
To make the app run live, DNS entry created which ties the application's domain name (e.g., myazureservice.net) to the load balancer's VIP.
Can layer any desired HTTP-based authentication mechanism on this. Or can use Windows Cloud Services (the Live Service in particular) which provides authentication.
Machines are grouped into fault domains, where a fault domain is a collection of systems with some common dependency, which means that this entire set of machines could go down at the same time. For example, all machines that sit behind a single switch are in the same fault domain as are all machines that rely on the same power supply.
In configuring the settings for your Azure app and storage, you might specify that two instances of the same thing (e.g., two identical Web roles or two instances of Azure Storage) should not be located within the same fault domain.
And actually, for the case of Azure Storage, the Storage app takes care of making sure that it has more than one version of itself and is spread across different fault domains (rather than the Fabric Controller doing this).
Easy to determine — given the FC's graph of all inventory (i.e., hardware) — what the fault domains are. Can determine statistically what the likelihood of failure is for each such domain.
Update domains: if you're doing a rolling upgrade of your service, a certain percentage of your service will be taken offline at any time (and upgraded). That percentage is the size of your update domain.
Security/Isolation
Use VMs to isolate one role from others
For PDC, app can only be managed code, which can be easily sandboxed
Run app code at a reduced privilege level
Network-based isolation
IP filtering
Firewalls: each machine configured to only allow traffic that machine expects
Automatically apply Windows security patches (heh).
MIX 2009 introduced the ability to run Web or worker roles with more privileges than previously was allowed. In particular, there are Code Access Security (CAS) levels, such as Partial Trust and Full Trust. In order to be able to run non-.NET (a.k.a., non-managed or native) code, the Web or Worker role needs to be able to spawn a process or to issue the Platform Invoke command. Apparently, one can only do those things with Full Trust. Hence, this is how MSFT will support non-managed code: you will configure your Web or Worker role to execute your non-managed code. So the Web or Worker role bootstraps your code.
http://blogs.msdn.com/windowsazure/archive/2009/03/18/hosting-roles-under-net-full-trust.aspx

Thus, in order for this to work, the code you execute has to be copy-deploy, which means that you don't have to install the code (since even Full Trust does not include administrative or install privileges) — instead, you can merely copy the code to the VM and run it.
http://blog.smarx.com/posts/does-windows-azure-support-java

The MSFT VMM is based on their Hyper-V Hypervisor, which was released in June 2008.
TODO:

Read up on REST.
Get more details on fabric controller and data-center architecture etc.

Can see more about the FC — as far as redundancy goes — here (starting at about 30:00).

Get more details on the "service model," which is how the app developer communicates with the Azure OS Compute platform about how to manage his app.
More details on how the FC drives nodes into the goal state (i.e., monitors and maintains app health).
How do these virtual IPs and dedicated IPs (where the latter make up a pool that "back" the virtual IPs) — how does all of this work? Presumably there is some address translation at some point (mapping a connection to a virtual IP to instead be to a real IP) — like NAT — but at which point? At the load balancer?

Maybe just means an internal IP address, such as 192.168.1.1?

More on Azure

2009-10-14T13:04:00.000-07:00

Below is some basic data on Windows Azure as well as brief exploration into relevant background topics (such as managed code and dynamically generated Web page content). The below contains no details on the various Azure Services (such as Live Services and .NET Services) and few details about the Azure OS run-time. It's all pretty high-level and gauzy.

Future posts may fill in more details such as:

What does execution look like on the cloud (what kind of isolation between web apps and so on)? Each machine in the fabric runs one instance of Azure OS? And each instance of Azure OS runs an instance of IIS7? And is there a one-to-one mapping between apps and instances of Azure OS (surely not)? Etc.
How exactly can one specify the health constraints of his app (where the Fabric Controller uses these conditions to automatically manage the app/service — including scaling the app up or down (creating/deleting instances) or restarting the app)?
What do the various Service APIs look like? (E.g., for SQL Data Services, Live Services, .NET Services, and so on) What operations are available?
How will MSFT support web apps written in non-MSFT languages (e.g., python), i.e., how will incorporate support for non-managed code?
...

Notes from Manuvir Das, "A Lap Around Azure"

Windows Azure == OS for the cloud.
It's the lowest layer. Services are layered on top of this foundation, including:

Live services
.NET services
SQL services (a.k.a. "Data Services")
SharePoint services
MSFT Dynamics CRM services

These services are implemented using REST, HTTP, and XML.
What's a cloud? A set of connected servers.
What can you do on the cloud (i.e., on those servers)? Install and run services; store and retrieve data.
A client can access Azure services via calling into a "managed class library" (presumably some code that contains RPC-like stubs which — at run-time — invoke the appropriate code).

In the desktop computing world, an OS provides:

An environment in which to run an app (abstracts away the particular underlying hardware config)
Access to a shared file system, which provides isolation via access control
Resource allocation from a shared pool
Support for different programming environments

In a cloud computing OS, you want all of the above plus 24/7 operation, pay-as-you-consume, transparent administration (i.e., which hides the complexity of remote mgmt as much as possible).

What features does Azure provide?

Automated service management.

The developer defines rules about which code should be executed under which conditions (where a condition might be, "URL x was visited") as well as the code itself. The platform follows these rules in deploying, monitoring, and managing the developer's service.

A powerful hosting environment. All of the hardware for actually running and serving your code (servers, load balancers, firewalls?). Two possible execution modes: direct and virtualized (where the latter is via a hypervisor).
Scalable and available storage in the cloud. Provides abstractions such as: blobs, tables, and queues.
A rich familiar developer experience. Includes a local testing/debugging environment which provides a complete simulation of the cloud. So that developers can test their app in an environment as close to the real thing as possible. The simulated enviro lets the application do everything it would be able to do if it were actually running on the Azure OS in the cloud.

More on Automated Service Management (Is this the so-called Fabric Controller? Yes, I think so.)
The developer writes his code as before. But he also now creates a model, which specifies:

What should the service topology look like? How big? Use cloud storage? How many front-end roles? How many back-end roles? Should the front- and back-end roles be able to talk to one another? If so, how? (Note that these various roles can connect to the outside world, which enables the developer to communicate with a particular role directly.) A common idiom is for the different roles (e.g., Web Role, Worker Role) communicate via shared storage — i.e., one adds to a queue (is a producer) and the other takes from that queue (is a consumer).
How to define health of my service (i.e., conditions under which I should be alerted because such conditions indicate that the service needs attention of some sort).

MS detects failures (of hardware, software, bugs, ...)
MS detects violation of health constraints (i.e. detects when your app's execution is outside of the definition of healthy)
MS needs a way to transparently fix stuff (ideally w/o a human in the loop). Maybe by rebooting your service, moving it to another server.
Achieve this via abstraction. Application code refers to logical resources (rather than to particular IP addresses or underlying CPUs). If need to obtain the actual hardware addresses at run-time, can invoke APIs which provide physical values (e.g., addresses) corresponding to specified logical resources.

Configuration settings: what particular values or parameters do I want to be able to change at run-time without having to reploy the entire service?

Note that you don't *have to use* their automated, abstracted thing. They provide a so-called "escape hatch" or Raw mode, which lets the developer build a VM from the ground up and run his service within that VM (where the developer would be responsible for managing that service as well). So this offering much more closely resembles Amazon's EC2 service except even with this the developer doesn't actually supply his own VM (as he would with EC2) but rather configures one of their VMs.

More on Azure Storage: massive scale, availability, durability. Geo-distribution. Geo-replication. This is NOT to be confused with SQL Services because Azure Storage does not expose the full database-management interface (which would include querying, insertion, schema creation, and so on). Only can upload data (and presumably delete it?). And available ways to structure data are very simple.

Large items of unstructured data: Blobs, file streams
Structured data (referred to as "service state"): Tables, caches
Service communication: queues, locks

Cloud Storage is accessible from anywhere on the Internet. Has REST APIs on it.

More on Developer Experience

Support for a variety of programming languages: ASP.net, .NET, native code, PHP

Bunch of tools and support, including for logging, alerts, tracing, ...

Including the much touted "desktop SDK for full simulation of the cloud"

My own look at things — questions and comments

Can I only run a web application on Azure? E.g., if I wanted to run a mail server on Azure, could I do that or not? If not, what mechanisms are actually used to prevent this? Restrict traffic that's not on a standard HTTP/HTTPS port?

Yes — according to this PDC 2008 session, an input endpoint (which is an app's externally-reachable interface to the world) must use either port 80 or port 443 (note that an app doesn't have to have an input endpoint). Hence, you couldn't define an app to run on Azure which had an input endpoint of port 25 (for SMTP), for example.

For the non-MSFT languages which can run on Azure (e.g., PHP, Python, Ruby), what run-time environment does a program in such a language run? Does the user choose the particular run-time environment?
And then what about sandboxing? What kind of isolation among various apps running on the Azure Fabric?

Since someone will ask, what are the benefits of moving an app to the cloud?

If you have a customer-facing web application and you have customers scattered across the globe then can use Azure to run an instance of your app at various geographic locations. This will have the effect of reducing latency for customers accessing your application from India, for example.
Availability: Relieves the user from having to maintain redundant servers (and infrastructure for fault tolerance); let MSFT handle that.
Scalability: Let MS also handle scaling of your application automatically.

Frees app-provider up from responsibility of maintaining a number of machines for the service that corresponds to the expected peak load for that service (despite the fact that normal use might be well below peak).

Zero-downtime upgrades.
If you need huge amounts of storage or the ability to do batch processing or the ability to run an application on a very large data set (for example, as with MapReduce), you can achieve this by running your app / doing your processing in the cloud. Frees you up from having to purchase and maintain physical resources, especially since the job may only be a temporary thing (and hence the physical resources would be idle most of the time).

At some point, will look in more detail at each Azure Service being offered — to understand what can do with that service, who uses it, and so on. But for starters here they are:

Compute on Azure OS.
Store using Azure OS.

Cheap, efficient, not necessarily very expressive.
That is, this is NOT a full-service relational database interface that enables SELECT, INSERT, and so on. This is a very simple interface that only exposes a couple different storage formats (blob, queue, simple [non-relational] table) and presumably only a couple ways to manage (operate on) the stored data.

Access Live services for...
Access .NET services for...
Access SQL services for...
Access SharePoint services for...
Access MSFT Dynamics CRM services for...

CRM: Customer Relationship Management; software and/or processes/strategies. Includes all methods by which a company responds to (or reaches out to) its customers. So call center, sales force, marketing, tech support, field services. Players in this area: Oracle (Siebel, PeopleSoft), SAP, salesforce.com, Amdocs, MSFT, Epiphany, and others.

The so-called "Fabric Controller"

The Windows Azure Fabric is a "scalable hosting environment" that is "built on distributed MS data centers." The Fabric Controller manages resources, performs load balancing, observes developer-provided constraints/requirements for an app as well as the real-time conditions for that app/service and responds accordingly (by, for example, automatically provisioning additional resources, restarting the service, taking away some unneeded resources, and so on). The FC scales app resources automatically as demand rises and falls. Used to deploy service and manage upgrades.

The Azure OS performs advanced tracing and logging on apps that run on it — so that developers can monitor the status of their apps as far as compute, storage, and bandwidth. Presumably, the constraints that developers specify are in terms of the type of things that can be observed using this tracing. Note that since MS is allowing apps written in non-MS languages, the types of "signals" that can be used to monitor the health of an app are likely OS-level (rather than language-level) things. For example, easy to identify the # of open sockets and heap size. More difficult for an OS to have visibility into language-level resource usage such as the number of locks created/acquired and so on. Hence, the types of things that one can specify in the application model must be rather generic and observable from the OS-level. I wonder whether — for apps written in MS languages and which will run in an MS environment (such as .NET framework) — one can specify a different (richer) set of constraints in the application model.

Notes from Steve Marx, Developing and Deploying Your First Cloud Service

Managed code == code written using an MSFT language and which executes on MSFT run-time; e.g., .NET, IIS7, WCF
Can only run a web application on Azure OS, not an arbitrary network app.

Enforced via only allowing an app to have an input endpoint whose port is 80 or 443; this means that the app can only receive traffic on the port for HTTP or the port associated with HTTPS.

The cloud on your desktop: complete offline cloud simulation. Actual cloud == set of connected servers or machines, also referred to as the fabric; it's what your app will run on. The "cloud on your desktop" is a set of processes (all of which run locally), where each process simulates a server. So the set of local processes are your "cloud on the desktop"; also referred to as "development fabric."
There's a UI for playing with / seeing this development fabric; can see each service deployment. For each service deployment, can see all the roles that this service has defined and, for each role, all of the instances of that role.
When create a new project, two associated files are created: the service definition file (ends in *.csdef) and the service configuration file (ends in *.cscfg). These are XML files which contain metadata about your service. The service definition file defines all of the roles and, for each role, defines the input endpoints for that role (where an endpoint consists of a name, protocol (e.g., HTTP or HTTPS), and port (80 or 443, respectively). If there are any configuration-specific settings or parameters, those parameters are declared here (but given values elsewhere). The service configuration file identifies for each role the number of instances of that role (that should be created) and — for any configuration settings declared in the service definition file — provides values for those settings/parameters.

Horizontal scaling: have code running on a single server; to scale == add more servers

But what about state? How to share state across various instances? (Why need to share state? Because a user could interact with a different instance each time and the fact that there are multiple instances needs to be transparent to that user — i.e., the user can begin a transaction on one server and continue that transaction on another) Solution: Use a single centralized database (a "durable store") to store all state; no server stores any state locally. All servers access state from this store == all servers have same view of state.

Have this available in Azure as the Azure OS storage: blobs, simple tables, queues. Access this storage via REST and ADO.NET Data Services.

A brief detour — some background on Microsoft languages and run-time environments
Evidently Microsoft has developed an infrastructure which enables code to be written in any one of several different languages then run on a variety of platforms. Parts of their infrastructure are reminiscent of Java with its bytecode and platform-specific JVMs. In particular, MSFT created various languages (e.g., C#, J#, VB.NET). Each of these languages has its own compiler which takes a program in the language and produces code in an intermediate language. This code is platform-agnostic (much the same as Java bytecode); the intermediate language is CIL — Common Intermediate Language. Then for each different hardware/OS platform, there is a Common Language Runtime (much the same as there are different JVMs for each hw/OS platform). CIL code is executed on the CLR. The CLR compiles and caches the CIL code just-in-time to the appropriate hardware instructions given the underlying CPU architecture (e.g., x86).

So the Microsoft languages for which a compiler exists (which converts a program in that language into a program in the intermediate language, CIL) are referred to as managed; so a program written in C#, J#, or VB.NET is managed code. Such code (after being compiled into CIL) executes on the CLR, which is a managed environment. Unmanaged code by contrast is that which is not compiled into CIL and does not run within the CLR. So when we say that something is a CLI Language, we mean that there is a compiler that takes a program in that language and produces the corresponding CIL code. (Fyi, CIL was "previously known as MSIL — Microsoft Intermediate Language.") Actually, "managed code" is a general term which covers any code that runs within a VM (rather than executing directly on the underlying hardware). A C++ program written with Microsoft Visual C++ could be compiled into managed code (to run in the .NET CLR) or unmanaged code (using the old MFC framework). But in general all code written in a particular language will be compiled into managed or unmanaged code (rather than being able to be compiled at will into one form or the other).

The CLR evidently exports an API that offers a lot of functionality that is usually provided by an OS. In particular, the CLR provides functions for: memory management, thread management, exception handling, garbage collection, and "security." This functionality is provided in the Class Libraries. The class libraries implement common functions such as: read/write, render graphics, interact w/DB, manipulate XML documents. The .NET Framework Class library consists of two libraries:

Base Class Library (BCL): small subset of entire class library. Core set of classes that serve as the basic API of the CLR. Many of the functions provided by MSCORLIB.DLL and some of those provided in System.DLL and System.core.DLL. Akin to the standard libraries that come with Java.
Framework Class Library (FCL): superset of BCL classes. Entire class library that ships with .NET Framework. Includes expanded set of libraries: WinForms, ADO.NET, ASP.NET, and others.

So, as portrayed above, the Common Language Infrastructure (CLI) consists of the intermediate language (CIL) and the environment within which that intermediate language executes (the CLR). Together the CIL and the CLR comprise the CLI.

What kind of benefits do you get by using a CLI?

Can have a program which combines components written in different high-level languages.
Can compile once (into CIL) and run anywhere (for which there exists a CLR).
There are also some benefits of managed code generally: the ability to provide stronger type safety (since can do run-time type checking), garbage collection

Another brief foray away from the main topic here: Web Page Content
A web page can have only static content (the web page's content doesn't change) or it can have dynamic content. In the latter case, where does the dynamism come from? Two possibilities.

There might be client-side scripting which changes the way the page is presented depending upon mouse movements, keyboard input, or timing events. So the dynamism is in how the content is presented. Languages used to achieve this include JavaScript (part of Dynamic HTML) and Action Script (part of Flash). These scripts might inject sound, animation or change the text. One can also perform remote scripting using these languages; remote scripting entails a Web page asking for more information from the server without having to reload the page. We see this in XMLHttpRequest, for example.
Secondly, server-side scripts might dynamically generate different web page content depending upon the data provided by the user in an HTML form (e.g., the user enters his name (John) and the generated page says, "Hi John! Welcome to..."), URL parameters, the browser type, or DB/server state. The server-side languages used for this type of dynamic content generation include: PHP, Perl, ASP, ASP.NET, JSP, ColdFusion, and so on. These languages use the Common Gateway Interface (CGI) to produce dynamic web pages.

Suffice it to say, we could explore these topics much more carefully and thoroughly. But the above suffices for our purposes, which are to understand what ASP or ASP.NET are (answer: server-side scripting languages, like PHP).

References

On the topic of Model-View-Control

MSDN's Model-View-Controller: http://msdn.microsoft.com/en-us/library/ms978748.aspx
Especially what Dean Helman wrote here: http://ootips.org/mvc-pattern.html
http://en.wikipedia.org/wiki/Model–view–controller

Ad networks, exchanges, and auctions

2009-10-09T13:00:00.000-07:00

Definitions

§ Display advertising: ad formats – videos, images and interactive ads

§ Reach: of all unique Internet users, the percent that an ad network encounters. Pay attention to whether this is reported in terms of US or globally when comparing one network's reach to another's.

§ Maximum yield: from the publisher’s perspective, the highest price he can get for each impression.

§ Direct sales: the publisher sells his ad space directly to the advertiser, in contrast to an ad network, which serves as an intermediary between publishers and advertisers. Note that direct sales has been the prevailing model for selling premium publisher space – e.g., the front page the NYT etc.

§ Ad slot: defines the size of an ad and the specific location where the ad will appear on a particular page or on a group of similar pages. The name of an ad slot might incorporate: the site name (ESPN), the section within that site (Football), the subsection within that section (e.g., NCAA College Football), page position (e.g., Top, Middle, Center combined with Left, Right, Center), and dimensions (728x90).

§ Placement: a set of ad slots. It lets you group together slots that a single advertiser (or ad network) might want to simultaneously target.

§ An insertion order (IO): an agreement between an ad seller and a buyer that specifies the details of the campaign. Contains one or more line items. Serves as the purchase order and contract between the parties. Often includes info such as pricing, impression goals, delivery options, and targeting details.

§ Line item: an advertiser’s commitment to purchase a specified # of impressions, clicks, or time (CPD) on certain dates at the specified price.

§ Ad trafficking: done by an ad server; entails providing the code – which downloads and tracks an ad – to the publisher to embed in his site (so that when a user executes that code, the ad will be served to him). I.e., delivery of the ad.

§ Forecasting: a challenge for publishers. The act of predicting what inventory – in which ads can be placed – you will have in the future.

§ Ad operations: handles fulfillment of online ad sales. Entails trafficking (the day-to-day execution of campaigns; including ad delivery), inventory management (inventory forecasting), sales, web development (produce and develop the advertising activity).

See also: Google’s “Introduction to Ad Operations”
http://services.google.com/training/gamtutorials/Intro_to_Ad_Operations/

What’s an ad server?

Ad serving refers to the tasks involved in storing an ad and delivering it on-demand to the specified user. It also entails keeping track of how many times each ad was served, where an ad was served to, how many times the ad was clicked on, and so on. These are all reporting-related activities. An ad server may also perform targeting – wherein the server decides which ad to serve based on the user who will view the ad. In this case, the ad server might store a cookie on every user’s computer (for users who interact with this ad server at some point) and use that cookie to figure out stuff about the user – age, sex, marital status, education level, household income, and so on. That info – along with more mundane facts, such as what operating system a user is running, which browser he’s using and so on – is used as input into the decision about what ad to serve the user.

What are some more specific features involved at the various stages of ad serving?

§ To make the size of a bid depend on an ad’s past performance.

§ To cap the frequency with which an ad is served, the pace at which it’s served, the targeting (to whom it’s served).

§ Dynamic selection: target only the impressions you want

§ A centralized clearing system – don’t have to interface directly with each publisher.

§ Provide targeting, which in this context means: advise the advertiser on which sites he might like to display his graphical ad on. So the ad server doesn’t necessarily facilitate this transaction but he provides guidance into identifying placements that make sense, given the advertiser’s target audience segment, geography, time of day, website content, browser, OS, keywords, …

There are a number of ways for a publisher to access this functionality:

1. Run his own ad server (machine) on his own site premises; e.g., OpenX is an open-source ad server implementation (in the same way that Apache is an implementation of a web server) that a user can download, install, run, and manage in-house. The publisher controls every aspect of ad sales, storage, delivery, and reporting – as well as manages the physical hardware used to perform these tasks.

2. Run an instance of OpenX in the cloud. In this case, the publisher retains total control over all aspects of ad serving (sales, storage, delivery, reporting) but doesn’t have to run and maintain the actual physical ad server in his own equipment room.

3. Contract with an ad server who provides everything but sales for a fee. That is, the ad server does storage, delivery, and reporting; the publisher takes care of sales. The ad server may provide info about the type of users that a publisher attracts. This info can be used by the publisher in identifying advertisers that are a good fit for his visitors.

4. Finally, a publisher can contract with an ad network. In this case, the ad network runs its own ad server (which does storage, delivery, and reporting) as well as handles ad sales. Ad networks were developed to enable an advertiser to simultaneously have his ads shown on various Web sites without having to contract individually with each such site. Google, Yahoo, Microsoft, AOL, Fox Interactive Media, AdBrite, ValueClick and so on are all examples of ad networks. Note that when people talk about ad networks, they generally mean display-ad networks. That is, an ad network is an interface between people who want to show display ads to users and sites who have space available for rent for just that reason.

This is probably the most common model – in terms of what the highest number of publishers do. It may not be the most common model in terms of revenues; that is, high profile publishers together command the lion’s share of display advertising revenues. And these high profile (or premium) publishers have historically done direct sales of their advertising space – rather than relying on an ad network.

References:

http://en.wikipedia.org/wiki/Ad_serving

http://www.openx.org/ad-server

http://en.wikipedia.org/wiki/Advertising_network

http://www.clickz.com/3626556

Background on Ad Networks

§ Definition: an intermediary between publishers and advertisers.

§ Primarily associated with the graphical or display advertising space, rather than the search advertising market.

§ Ad Networks provide a way for an advertiser’s content to run across the Web – i.e., on lots of websites that are not necessarily under the control of the same administrative domain.

§ Provide a way to mitigate the effects of evermore audience fragmentation.

§ An ad network may also serve or host the actual ads that run on these publisher pages; that is, visiting the publisher page causes a request to be sent to the ad network and the ad content to be downloaded from (i.e., served by) that ad network.

§ Tracks ads.

§ Reports on the distribution of ads.

§ Handles the transaction between advertiser and publisher.

§ Frees publisher up from having to maintain a sales organization (in order to monetize his inventory).

§ Gets paid on a revenue-sharing basis.

How do ad networks vary?

§ What audience they reach: size, demographics, …

§ The ways in which they can target: vertical, contextual, behavioral, demographic, re-targeting, geographic, site-specific

§ Type of compensation they accept: CPM, CPC, CPL (Lead), CPA

§ Ad formats they support: display, text, in-text, video, mobile, in-game, blog, RSS, email, audio/podcast, widgets

§ Their business model: revenue share, arbitrage, rep firm

§ The extent to which they provide related services to publishers and advertisers.

§ The amount and quality of user information they possess.
To the extent that behavioral marketing takes off, someone like Google has reams of information that they can deploy to better perform such targeted marketing. This is in contrast to an entity such as Microsoft that doesn’t have as much of this kind of info.

Background on Ad Exchanges

§ What’s an ad exchange? A trading exchange for display advertising in which website owners and advertisers can reach deals on prices and placement of ads.

§ A stock exchange for online display ads; i.e., a real-time marketplace with an auction-based system and open bidding process – sells impressions in real-time or on-the-spot (auction platforms are associated with impression-by-impression purchasing).

§ They can also be used as a futures market – to acquire reserved inventory.

§ A web site puts up ad space for auction and ad agencies bid for those spots.

§ What problem does it solve? Managing co-ordinated, large-scale display advertising campaigns across the net is a logistical nightmare. Given that an effective campaign has to span a multitude of display ad formats and thousands of sites, it takes ages to plan and manage campaigns.

§ Who are the players in the exchange?

§ The large online publishers participate in the exchange to sell their inventory.

§ Ad networks (and agencies) are the buyers. So an advertiser contracts with an ad network or ad agency which maintains a network of properties (placements) where the advertiser’s ads can be placed.

o The buyer will likely have his own ad serving technology.

§ You can also buy space on the AdSense publisher network via the exchange.

§ Similarly, if you’re an AdWords customer (i.e., advertiser), you can buy ad space on the exchange – via your AdWords interface (you accomplish this by electing to have your ads shown on the Google Content Network).

§ And if you’re an AdSense customer (i.e., publisher), you can sell display space on your site on the exchange – via the AdSense interface (check this out).

§ “On an exchange, publishers can choose to sell their inventory blind, private, or branded. Which they choose will differ from publisher to publisher, but obviously branded inventory will still be able to command a premium, taking it out of the commodity area. Publishers can also disqualify particular advertisers from whom they do not want to accept advertising (for competitive or other reasons), or make positive qualifications regarding the type of advertisers they will accept (e.g., women's interest only). On the buy side, marketers or their agencies are offered similar control.”

Benefits of Ad Exchanges

§ Transparent, dynamic pricing; open bidding.

§ Simplified, standardized business processes (ad sale/purchase) makes things easier.

§ Better liquidity for ad inventory.

§ Smaller advertisers have equal access to exchange inventory as the Big Guys – with bigger ad budgets and better relationships – do. A leveling of the playing field.

§ Can use technology to automate things such as: bid on any ad space that has properties X, Y, Z and is available for less than $N.

§ Eliminates intermediaries and their margins.

§ A publisher gets access to many more advertisers while still retaining control over who can advertise on the publisher’s site.

§ Buyers can use technology that lets them bid in real-time (based on various criteria and via an automated agent?).

How do ad exchanges vary?

http://www.clickz.com/3630872

§ Their inventory

§ Their targeting methods

§ Their placement options

§ Their pricing models

Does a publisher usually have an exclusive relationship with a particular ad exchange – so that the publisher cannot offer his space on multiple exchanges simultaneously? Also, are the ad exchanges themselves connected in any way – so that a buyer on one ad exchange can purchase through it property that is listed on another exchange?

I think a goal is certainly to interconnect the ad exchanges (if this isn’t the case already). Also note that a publisher may have generic deals with advertisers – not for any specific impression but for impressions which match certain criteria. Those generic deals can be compared with real-time bids (obtained in the ad exchange auction) to see which is best and make the sell decision then. In this way, a publisher reconciles an advertiser’s desire to make a futures buy (reserve inventory) with the benefits of a per-impression sale.

Incorporating AdWords bids into the DoubleClick Ad Exchange

OK, we’ll walk through an example to hopefully illuminate at least the auction process itself, if not the entire context. Let’s say that there is some slot (i.e., impression) being offered on the DoubleClick Ad Exchange. And that slot has been explicitly targeted by an AdWords advertiser (via a site placement for his display ad) or the slot contextually matches the advertiser’s graphical ad (or industry, whatever). And actually let’s say that the slot matches several AdWords advertisers’ bids: ad_1, ad_2, ad_3, and ad_4.

Advertiser	CTR	# clicks/M	CPC	eCPM	QS	Ad Rank
ad_1	1%	10	$3	$30	7	$210
ad_2	2%	20	$3	$60	8	$480
ad_3	5%	50	$2	$100	6	$600
ad_4	3%	30	$3	$90	7	$630

eCPM is the effective CPM rate and is calculated by identifying the # of clicks per 1000 impressions that an ad is expected to obtain (# clicks/M) then multiplying that value by the cost per click bid. Quality Score is multiplied by eCPM in this case in order to get AdRank (rather than by the CPC). Then we can see that ad_4 wins and ad_3 comes in second place; only these two bids will be considered by the Ad Exchange.

Advertiser	Ad Exchange bid	>= minCPM?	Publisher’s cut	Ad Exchange cut
ad_3	$580	Yes	$493	$87.00
ad_4	$552	Yes	$469.20	$82.80
minCPM	$520	Yes	$442	$78.00
adExch_1	$375	No
adExch_2	$500	No
adExch_3	$750	Yes	$637.50	$112.50

First we have to take out the AdWords cut so that the bid represents: what the publisher will make plus what the Ad Exchange gets for the trouble (transaction). Let’s say (HYPOTHETICALLY) that the AdWords revenue share is 8%. Then we have: ad_4 == $630 * 0.92 == $579.60 (let’s call it $580) and ad_3 == $600 * 0.92 == $552. Naturally, taking out the AdWords revenue share doesn’t change the relative ordering between these two ads – because we assume that revenue share is the same (rate) for everyone. Now let’s combine those bids with other Ad Exchange bids and with the publisher’s min CPM.

So now we’re considering the most lucrative AdWords ads (ad_3 and ad_4) alongside of the publisher’s minimum CPM and any Ad Exchange bids for that slot (impression). The next thing we do is subtract the publisher’s cut from each bid (which we’re assuming to be 85% across the board; rumor has it that 50% is a more likely estimate) – so that we’re left with what the DoubleClick Ad Exchange stands to make on the deal. THEN we take into account that the publisher may have other bids for this space – that were obtained by his own ad-sales team. You can think of these as commitments that the publisher has made to various advertisers – but these commitments can be fulfilled within a time window (rather than being a real-time thing –specific to a single impression). So the publisher could always use one of those bids. And, if one of those bids is better than all of the bids from the Ad Exchange (including the AdWords bids) – which are all represented as what the Exchange stands to make on the deal – then the publisher should place serve this in-house ad in this impression.

So the DoubleClick Ad Exchange interfaces with the publisher’s own ad server to obtain that ad server’s bids for these slots. Those bids are specified in terms of what the Ad Exchange would make on the transaction and hence can be compared (apples-to-apples) to the values in the final column of the above table. So then we’re looking at:

Advertiser	DoubleClick Exchange cut
ad_3	$87.00
ad_4	$82.80
minCPM	$78.00
adExch_3	$112.50
inHouse_1	$96.00
inHouse_2	$73.25
inHouse_3	$101.50

So in this case, the winner of the slot was one of the bidders from the Ad Exchange. If the publisher has “dynamic allocation,” then this transaction will complete automatically (without requiring any approval) and the ad from the adExch_3 advertiser will be shown to the Web site visitor whose impression triggered this whole execution cycle.

Note that a publisher doesn’t necessarily have any in-house bids (i.e., bids from “the publisher’s own ad server) – i.e., its direct sales channel is empty for this slot. Note also that it appears then that the final decision about which bid wins is made in order to optimize the DoubleClick Exchange’s cut. But – so long as the publisher’s cut is the same across all advertisers – this is the same as getting the maximum cut possible for the publisher. The rate is the same across all advertisers and hence the winning bid will be the one with the largest principle – of which the cuts are being taken.

See: http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=146606

I also really like these companies that create new ad-hoc employment marketplaces

2009-10-08T13:19:00.000-07:00

For example, JustAnswer is a company that lets random people post questions on a variety of topics (from auto repair to programming to relationship advice). The questioner specifies how much the answer is worth to him. Then the question is posed to the community. Whomever answers the question in a way that satisfies the questioner gets paid.

Similar to lots of other efforts to "crowd source" resolution of various questions. Major difference (with JA) is that the answerer gets remunerated (i.e., paid for his trouble). Another major difference with JustAnswer in particular is the amount of due diligence they do in order to "certify" various folks (potential answerers) as being Expert-level in some category. The idea being that if I ask a question about an area I don't know much about, I can rely on JA's accreditation of folks as a gauge of who is trustworthy. I took their test to become certified as an expert in programming and found it reasonable: 10 multiple choice questions, not cake, reasonably generic (rather than w.r.t. specific idioms/languages/paradigms).

But basically I am always really pumped about products that create opportunities for small-scale entrepreneurs to run their own businesses — whether that business consists of answering questions (using JA), writing apps for the iPhone, writing apps/games for Facebook (or some other social network), etc.

We also saw this with Coghead awhile back which let individual coders (or small teams thereof) create applications on the Coghead platform and then sell those apps as a service to whomever. So an individual programmer becomes in effect a SaaS provider by leveraging the Coghead foundation. Of course we see something similar with AppEngine — where Google worries about hosting, server administration, and so on. All you have to do is create your Web service (in Java or Python) via providing code to execute when particular URLs are visited and you're off to the races.

The whole thing results in a lot more liquid, task-specific employment market — and gives workers a lot more options. In a way it's not different than the advances we've seen in resource management in computer science. Fueling the resurgence of virtual machines the past decade was the fact that companies had servers that were not being used wisely. Might have a dedicated server for each different service (e.g., mail, web, news, ftp, ...) which resulted in poor resource allocation because some resources sat idle much of the time while others were (at the same time) oversubscribed. With virtualization, resource allocation became more fluid. A high-demand service could be spread across machines whereas quieter (lower demand) services could all safely share the same hardware without adversely affecting one another.

In any case, I'm way bullish on any company or product that encourages this same "efficient resource allocation" of human capital. Develop a more just-in-time marketplace where matching humans to jobs occurs dynamically and in response to real-time demand. Taking the real-time per-impression ad-auction paradigm and applying it to humans and jobs/work to be done.

Some interesting companies (just the gist), 10/06/2009

2009-10-07T12:43:00.000-07:00

Rapleaf: http://www.rapleaf.com/

Figure out what your customers are doing on the Web as far as social networking etc. goes; for individuals: figure out what is being said about you on the Web.
TrialPay: http://www.trialpay.com/

Real-time, dynamic generation of "deals" for consumers; so if you're at the WinZip site but decide not to buy the software then later are about to make a purchase at Wine.com, you might be offered a deal to let you get that WinZip software for cheaper — along with your wine. Rather than deals being a one-size-fits-all static kind of thing, a deal is generated for a specific user based on that user's past behavior AND based on what it might take to convert that user.
JustAnswer: http://www.justanswer.com/

Everyday people can pose questions about everything from car repair to computer repair to relationships. A person provides his question along with what it's worth to him to get the question answered. Then a swarm of experts will jump in and try to resolve the matter — earning the fee in the process.

Not so different from aardvark.com and fixya.com except that the answerer gets reimbursed in this case. Also, JA takes steps to make sure that the answerer is qualified to answer the question. In order to sign up to be a Programming Languages expert, I had to successfully complete a straightforward multiple-choice quiz of programming questions.
Adify: http://www.adify.com/

These are guys responsible for building the various vertical ad networks; e.g., the Martha Stewart Living ad network whose anchor is the Martha Stewart site but which also includes related sites/publishers.
meebo: http://www.meebo.com/

Overcomes the historical limitation of chat programs which was that a chat client was tied to a particular chat network and so you could only chat with someone on your same network (sounds almost insane to type this; what a dumb way to do things!). With meebo, you chat via a Web interface and you can chat using an AIM, Yahoo, MSN, MySpace, Google Talk, etc. client and you can chat with anyone using any other chat client. You can also easily create chat rooms.
Yelp: http://www.yelp.com/

Standard review site, like CitySearch. Provides reviews of dentists, restaurants, dating services, clothing stores, and so on. A user can vote on reviews and a business can respond to reviews about it. Reviewers are encouraged to generate original content such as lists of the top happy hour spots, the best place for a cheap mani/pedi, and so on. Reviewers are also encouraged to share more info about themselves — to personalize their profiles.

When you search for something on Yelp, you might also be provided with sponsored search results. That is to say that Yelp probably derives a substantial portion of its revenues from search marketing (i.e., selling text ads based on keywords). A business can also pay Yelp to put one of the reviews of this business at the top of the business's page. So that when a user visits that business's page, the first review she sees is this selected one. The business still has no control over its other reviews — including over the order in which those reviews appear.

TODO: Look into: whether they deliver display ads yet, how they sell the ads that appear in search results — based on keywords? via an auction? on an impression-by-impression basis? and so on. Basically, do they roll their own ad sales/placement or are they a publisher that belongs to some network, such as AdSense — and hence they outsource all ad-sales-related activities. Presumably since it's Yelp's primary source of income, they manage ad sales directly.
Cooliris: http://www.cooliris.com/

Their products are Cooliris and CoolPreviews. Cooliris is a browser extension which lets you explore the contents of your disk drive — in particular to look at the pictures in any folder and navigate through those pictures in some cool way. So that is the heart of the Cooliris proposition — the way that you navigate through your photos, the way that they present those photos to you, and so on. So in a way, what Cooliris competes with is your computer's traditional way of displaying your photos to you. For Windows, if a folder contains pictures, I can see small thumbnails of each picture (maybe each is 1.5 inches by 1.5 inches) or I can view the photos in a slideshow. With the slideshow presentation, I see a much larger version of each photo but I can only see a single photo at a time.

By contrast, with Cooliris, you can simultaneously be looking at a large collection of photos, each of which is a much larger thumbnail. So for starters, you might be looking at two rows of photos, each of which consists of around 4 photos. And the thumbnails for each photo are much larger than traditional thumbnails — maybe 4 inches by 2 inches (but the sizes are variable). Then the new thing is that you can rotate those rows so that — instead of looking at the two rows (one covering the top half of the screen and the second covering the bottom half) straight-ahead — you are looking at the rows from a sidelong glance. This lets you simultaneously look at more than 8 photos. Maybe more like 12. The effect is that your screen becomes 3-dimensional and the plane containing the photos pivots on its leftmost vertical edge or its rightmost vertical edge, like this:

By presenting the photos to you using three dimensions, they can present more photos for you to consider at once. Moreover, as you scroll through the rows of photos by using the sidelong view, the photos cruise by you. So it's a very dynamic, bouncy-even presentation.

Now they take this same technology for viewing photos on your hard drive and apply it to viewing photos of products at the marketplace. So there is a way to "go shopping" from your Cooliris plug-in. Similarly, you can visit various channels, each of which contains a collection of videos related to some topic such as Sports, News, Sci-Tech, Entertainment News, TV, and so on. See also: http://www.pcmag.com/article2/0,2817,2353223,00.asp
Jigsaw: http://www.jigsaw.com/

This is a modern version of the Dun and Bradstreet databases that I remember using in my investment analyst job — in order to gain information on particular companies. Jigsaw maintains a database of both companies/organizations and individuals. The idea is that, for any individual, you should be able to view that individual's current position and contact info (basically his business card). Also, one can go from an organization to identifying key individuals within that organization (or can just learn more about the organization generally). So if you know you need to talk to the VP of Sales, you can figure out who that is.

A user can "get points" by adding information about contacts and companies. The information you enter is associated with you — so that there's the basis for reputation scores, that is, for discounting information in the Jigsaw database depending upon who provided that information. Viewing certain info about companies/individuals requires redeeming points (i.e., so you must have previously contributed to the Jigsaw database) OR a paid subscription.

Primary users are expected to be folks in sales, marketing, or recruiting. See also: http://www.jigsaw.com/corp/jigsaw_corporate_overview.pdf

TODO: Play with their data; how good is it? How accurately does it capture individuals/businesses of which I am aware?
box.net: http://www.box.net/

Is a way for a company to host documents in the cloud so that employees in arbitrary locations can collaborate on these documents. The documents can be internal (not publicly visible) or external (publicly visible). Enables creation of online file systems. Users can collaborate on documents. Can use their interface to share (via configuring permissions), access, and manage files in the cloud. All communication (including downloading, uploading, editing files?) takes place over an encrypted pipe (SSL). Your files are stored on multiple servers so that, if one hard disk drive fails, your data is not lost.

An alternative to Microsoft SharePoint, which is derided as "too complicated." An improvement over emailing docs back and forth between team members (collaborators). Better than FTP where there is no notion of "merging changes;" if one user changes the doc and replaces the old version on the FTP site, that old version is gone. So changes occur at document-granularity. Hence, their primary value proposition is the ease with which one can use their system.

Sharing: can share individual files or a whole folder of files. Share with someone via providing their email address. Each document can have an associated thread for comments. Can also create "discussions," which are presumably comment threads that are updated in real-time (i.e., not at all different from a chat room on meebo). Can create tasks, to organize work flow. All of your documents, tasks, comments, and discussions can be searched and modified by the people you elect to provide these capabilities to. (Actually, a document's contents cannot apparently be searched without a box business subscription.)

With a box business subscription, you can view previous versions of a document, customize the box interface, search the content of your documents, view reports, manage users through an admin console.

TODO: register and play with their interface, including observing how intuitive and easy-to-use their sharing interface is — as well as how scalable (how easy and intuitive to maintain many sharing policies, to create sharing policies which apply to data not-yet-created, and so on). Etc.
meraki: http://meraki.com/

Traditionally, if you wanted to create a wireless LAN for your enterprise (), you would buy a collection of access points (APs), each of which was a hardware device that provided wireless Internet access to users within some radius of the device. To manage your wireless network then you had to log into each separate AP and configure it then monitor its status. That is, you couldn't log into a centralized server in order to get a network-wide view or in order to configure a network-wide policy. You had to interact individually with each separate AP.

Recognizing the need for centralized monitoring and configuration, vendors introduced a number of controller-based systems with thin APs (also called dependent APs). Unlike standalone APs, most thin APs cannot operate on their own. Rather, they rely on one or more WLAN hardware controllers that need to be installed in wiring closets.
In this scenario, the hardware controller is a centralized management interface. It directs traffic between wireless and wired networks. It also lets clients roam from one AP to another. The AP connects to the controller over an Ethernet cable; through this cable, the AP obtains both Internet connectivity and power. This set-up is referred to as a: controller-based deployment with tethered APs. Controllers are expensive, they have installation costs. Also, when a WLAN controller fails, all APs connected to that controller lose connectivity (i.e., fail). So the controller is a single point of failure. ("Dual-redundant controllers, while technically possible, are often prohibitively expensive.")

The next phase of evolution was to not require that APs physically connect to a controller but rather have a logical tunnel back to the controller. Presumably in this case, the AP derived its power by plugging into a wall socket. And there was a wireless connection from the AP to the controller. Sending data through an AP means the data would also go through that AP's associated controller. With this architecture, still need expensive, redundant controllers. This doesn't work so well if have multiple sites that all need to be on the same wireless LAN.

With that backdrop, then, meraki's solution entails a single piece of hardware: the access point (actually a set of access points depending upon the number of clients and size of the region for which wireless service is to be provided). Then these access points connect to meraki's data center which consists of a bunch of servers which are used to configure, manage, and monitor a wireless LAN. So the hardware controllers are effectively centralized and moved to the cloud. Then multiple different wireless LANs can be run/managed using the same hardware controllers (so, some virtualization going on).

The Meraki Cloud Controller is out of band, which means that client traffic never flows through it... Control traffic flows between the APs and the Cloud Controller via a persistent tunnel. All sensitive data, such as configuration details, user names, and passwords, are encrypted... Multiple geographically distributed data centers are used to ensure that networks continue to function even in the event of a catastrophic failure. All management is done remotely through a Web browser... The administrator can also remotely diagnose the APs, using standard tools like ping, from the Meraki remote management interface.
TODO: Spend more time understanding the various architectural elements and how the meraki system differs from previous solutions etc.
Rocket Fuel: http://rocketfuelinc.com/index.html

It's an ad network. Their customers are advertisers (who want to run a campaign) and other ad networks (who want to add intelligence to their targeting and optimization but lack the in-house expertise to do so). RF also has partnerships with publishers (on whose sites the ads are run). From their site:

Rocket Fuel Inc. is building the first intelligent ad serving technology platform that combines the best of social, behavioral, contextual, geographical, search and other data sources to understand consumer interest and intent... We’re experts at predictive modeling and customer segmentation. Our core expertise is in developing and using technology to process and scale huge amounts of data to predict the likelihood of responses from individual users – we find audiences designed for your needs.
There should be no contextual ad networks or behavioral ad networks. It's like debating whether you should have eyes or ears. If you can have both, it's a win and there's no point debating. There should just be smart ad networks that use as much relevant information as they can to pick the best impressions for a given campaign. An ad server should just use whatever data is found to be valuable in selecting the best ad for an impression, or the best impression for an ad.

So I recently added Google Analytics code to this blog...

2009-09-21T10:36:00.001-07:00

Mostly to play with the functionality and so on. But a funny thing happened on the way...
You are given a code snippet to add to your page's HTML; a portion of which is copied here:

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
...

So I copied and pasted the above — as is — to the window of the Blogger editor (for editing blog posts). The Blogger editor has two editing modes: Edit HTML — which is supposed to let you directly edit the page's HTML — and Compose — which is more like an MS Word interface to editing a blog post; one can easily bold, italicize, change font, create bullets, and so on in this mode (using buttons rather than specifying HTML markup tags). Then that editor transforms your entered text into HTML.

Because I was given HTML code though (for analytics tracking), I used HTML mode and copied the given JavaScript scripts (as above) directly into the bottom of the page then saved and published. However, some *additional* code was added — by the Blogger editor — to the scripts that I inserted. In particular, XHTML line breaks were inserted within the JavaScript script. So, the above became:

<script type="text/javascript"> <br/>
var gaJsHost = (("https:" == document.location.protocol)? "https://ssl." : "http://www."); <br/>
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>

So now I have XHTML line breaks (<br/>) within the analytics tracking scripts, which caused any JavaScript interpreter (parsing the above script) to choke and stop processing the script — and hence not to perform the analytics-related tasks. This explains why I wasn't seeing tracking information for clicks that I was sure had landed on my pages. The fix was simple of course: remove all line breaks in the analytics scripts. Interesting all the same.

(How did I verify that a JavaScript interpreter chokes when it sees <br/> mid-script? Using this sandbox.) (Note also that as far as Google Analytics was concerned, my tracking code was installed properly, which means their checker needs to be fixed.)

Another curiosity is whether I should create separate Analytics "profiles" for each different page of my blog. And whether visiting my blog homepage will cause the analytics tracking scripts (which are inserted within every individual blog post) to *all be executed*. That is, I added analytics tracking scripts to each individual blog post. And when you visit my blog, you automatically see all posts. So presumably, every time I get a visitor to my homepage, all N analytics scripts will execute (where N is the number of blog posts) and so it will seem as if I've had N * x visitors (where x is the number of people who visit lizstinson.blogspot.com).

(some) Google Technology, briefly: PageRank, MapReduce, Bigtable

2009-09-15T13:01:00.000-07:00

Below are some brief notes, meant only to capture the Big Picture. This simplification may come at the cost of precision and even accuracy. My apologies to the creators. By way of summarization, Google's philosophy is "in-house development designed to run on commodity hardware."

PageRank:

One signal of many. Examine entire link structure of the Web. Identify which pages are most important. Note that this is not the same thing as identifying which pages are most important relative to some query or within a given vertical. I think it's interesting that they use this overall importance measure rather than identifying importance within a given domain. Then if it's important to have an overall metric, assign relative importance to each domain then calculate each page's resulting overall importance. In any case, from my incredibly cursory reading, it appears this is not what they do. They have an overall importance metric. Separately, they perform hypertext matching to compare Web documents to the query string.

The rationale behind my computing "overall importance" within a domain is that what seems unimportant one day can be quite important the next, e.g., consider pages dealing with "anthrax vaccines" pre-anthrax-letter-mailing and post-. The relative importance of such pages increases dramatically. And so by tweaking relative importance of various domains, we can automatically adjust the relative importance of all documents within those domains. But certainly this approach would have all sorts of challenges of its own, such as what the hell are the domains? How to handle documents that cross domains? Can domain identification be automated? And so on.

In any case, your search results are obtained by considering the relative importance of all pages on the Web and combining that information with which pages best match your particular search query. So it seems that the hypertext analysis is used to determine which documents match and the "overall importance" metric is used to rank the matching documents. So is the idea that all documents match equally? Probably not; probably they use buckets. So all documents in this bucket match approximately equally; hence, use "overall importance" to rank the documents within this bucket. Do the same for the next bucket and so on. Ranking of results across buckets could be done by using all results from the first bucket (which perhaps maps the most closely) followed by all results from the second bucket (which maps the second most closely) and so on. Or something more sophisticated where the results from various buckets are interleaved.

Note that the parsing they do of Web pages (in order to build inverted indices which will be used to perform hypertext-matching) does not just scan the page's content. It's more sophisticated, taking into consideration fonts, the way a page is organized or divided, where a word occurs within a page (early on or later). This is meant, in part, to frustrate those who would manipulate results by inserting metatags which suggest that a page contains data that it does not actually visibly present.

References:

MapReduce:

The problem that MapReduce was designed to solve was that Google (and others) had large sets of input data that they needed to perform computation over. If they ran that computation on a single computer, it would take a very long time. So they needed a way to break up the computation into smaller chunks (i.e., parallelize it), distribute those chunks to multiple computers, then merge the results. The raw data in this case might be, for example, crawled documents or web request logs. And the computation or processing might be to: (1) identify the most frequent queries, (2) represent the crawled documents graphically, (3) summarize the number of pages crawled by each host, or other tasks. The processing or computation is straight-forward but the input data set is HUGE and so it has to be split across multiple machines; otherwise it would take forever.

There are a number of questions: How to parallelize this computation? How to distribute the (input) data (to the various machines)? How to handle failures?

So MapReduce is an abstraction that was introduced to allow solving common problems (such as how to handle failure) one time — in the MapReduce implementation — rather than having this machinery be reinvented for each different problem instance. MR "hides the messy details of parallelization, fault-tolerance, data distribution and load balancing in a library."

The various tasks that they were needing to solve in a distributed manner involved a couple common components. That is, the meat of those tasks could be broken up into two phases: (1) MAP: apply a function to each input item in order to get a set of intermediate key/value pairs; this function is provided by the user and essentially decomposes the input data in a way that makes sense given the target computation; (2) REDUCE: given several sets of intermediate key-value pairs (i.e., one set is generated by each machine participating in the computation and acting on some input data), merge the key-value pairs that have the same key.

As a practical example, consider the problem of identifying the most frequently queries given a set of web request logs. Maybe we'd give one web request log to each participating machine. Then each machine would parse that log using the MAP function, which would result in the machine having a set of key-value pairs where the key is the query string and the value is the frequency with which that string appears in this web-request log. In this case, REDUCTION is trivial; in particular, reduction involves merging all machines' results, which means that — for each machine which has a key in common — we add those machines' values for that key to obtain the key's ultimate value (i.e., total frequency). Remember that the "value" in this example is the frequency with which a query string (the key) appeared in the given machine's input data (web request log). Hence, REDUCE here is arithmetic addition but you can imagine a case where the combination of intermediate computation results is more complicated.

Note then that the steps are as follows:

Invoke MAP on an input pair; results in MAP returning a set of intermediate key-value pairs.
MR library obtains these key-value pairs from all participating machines. Then MR library looks at this aggregate set of intermediate key-value pairs (those contributed by each machine) and identifies common intermediate keys.
For each common intermediate key identified in the previous step, invokes REDUCE — supplying the common intermediate key I and all (intermediate) values for I. This function combines those values to obtain a single output value (corresponding to the given input key). Note that in principle, REDUCE could result in merely a different set of values (rather than being required to produce a single aggregate output value); that said, "typically just zero or one output value is produced per Reduce invocation."

Note that the way that the values (for the given intermediate key) are supplied to Reduce is via an iterator. This makes it OK for an intermediate key to have a set of associated values that is too large to fit into memory.

How to handle failure? Re-execute. In particular, failure is presumably a single machine dropping dead (or some small number of machines). And since the computation on each such machine is independent of that on other machines, we can restart or re-execute the computation that occurred on the failed machine without having to restart the entire job.

MR is a widely applicable model:

Source: http://labs.google.com/papers/mapreduce-osdi04-slides/index-auto-0005.html

MapReduce "is often used for generating and modifying data stored in BigTable[2], Google Reader,[3] Google Maps,[4] Google Book Search, "My Search History", Google Earth, Blogger.com, Google Code hosting, Orkut[4], and YouTube[5]."

Source: http://en.wikipedia.org/wiki/BigTable

References:

http://labs.google.com/papers/mapreduce.html (2004)
http://labs.google.com/papers/mapreduce-osdi04.pdf (paper)
http://labs.google.com/papers/mapreduce-osdi04-slides/index.html
http://en.wikipedia.org/wiki/BigTable
Hadoop provides an open-source implementation of Map-Reduce; see: http://en.wikipedia.org/wiki/Hadoop.

Bigtable:

It's a database and its implementation is built on top of Google's File System (GFS), Scheduler, Lock Service, and MapReduce. Non-Googlers can use Google's Bigtable implementation by developing programs for Google's AppEngine which use the engine's datastore (which sits on top of Bigtable). "Bigtable is used by more than sixty Google products and projects, including Google Analytics, Google Finance, Orkut, Personalized Search, Writely, and Google Earth."

Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving).

Motivating example: Webtable, "a copy of a large collection of web pages and related information that could be used by many different projects; let us call this particular table the Webtable. In Webtable, we would use URLs as row keys, various aspects of web pages as column names, and store the contents of the web pages in the contents: column under the timestamps when they were fetched..." This retention of time-modified information enables versioning.

A table does NOT have a fixed number of columns. Each table has multiple dimensions; it might have a row for each URL and columns corresponding to: the content stored at that URL at some time, the language of that content, ... Each cell would have an associated time as well, making every two-dimensional table at least a three-dimensional table.

Each table is split — at row boundaries — into multiple tablets. That is, a range of rows is dynamically partitioned into a tablet. So a tablet might contain some number of rows; all of a row's contents can be found within a single tablet. Tables are divided into tablets in order to get tablets of size approximately 100 — 200MB. Each machine stores around 100 tablets (so 100 * 200MB == 20GB) [according to notes from a talk in October 2005, so that may have changed substantially as the cost of disk space continues to drop]. This sizing is done to optimize Bigtable for the underlying file system implementation and:

This setup allows fine grain load balancing (if one tablet is receiving lots of queries, it can shed other tablets or move the busy tablet to another machine) and fast rebuilding (when a machine goes down, other machines take one tablet from the downed machine, so 100 machines get new tablet, but the load on each machine to pick up the new tablet is fairly small).

Special tablets keep track of the file system locations of data tables (akin to how a directory structure keeps track of where particular file blocks are stored on disk). These "meta-tablets" keep a mapping between tablet ID and location within the GFS. If a tablet or segment exceeds the 200MB limit, they use compression to reduce the segment's size.

Bigtable Data Model

A table is really a multi-dimensional map which is indexed by row key, column key, and timestamp. Each value in the map (i.e., cell contents) is an uninterpreted array of bytes. The map is distributed and persistent, which means that different parts of it may be stored permanently on different machines in different locations. It's also sparse (though this seems to me more a function of the table's contents than of the table itself), which means that lots of cells have no values; for example, if each row corresponds to a URL then there are many more columns (each corresponding to a feature) than most URLs exhibit. As a contrived example, imagine that there was a column for each possible language. Then a row (URL) will only have an X in the column corresponding to the language in which this URL's content is written. Hence, all other language columns would be empty for this row.

The Bigtable paper represents a table as having three dimensions; hence, one can access a cell value by providing the string identifying the row, the string identifying the column, and the 64-bit integer representing the timestamp. Alternatively, by just providing the row and column, one obtains an array of values contained in that cell at different times (value v0 at time t0, v1 at t1, and so on).

Dynamic Control over Data Layout and Format: There is a claim that clients can specify the locality of data (namely, Bigtable "provides clients with a simple data model that supports dynamic control over data layout and format, and allows clients to reason about the locality properties of the data represented in the underlying storage."). In particular, Bigtable stores data in lexicographic order by row key. Then the row keys that a client uses determine which data will be stored together and hence enable the client to group data that should be accessed together in adjacent rows. I'm not sure how frequently the "dynamic partitioning" of a table into tablets occurs (whether it's something that's done once up front and retained thereafter OR IF INSTEAD a table might be repartitioned as the contents of various rows grow/shrink).

For example, in Webtable, pages in the same domain are grouped together into contiguous rows by reversing the hostname components of the URLs. For example, we store data for maps.google.com/index.html under the key com.google.maps/index.html. Storing pages from the same domain near each other makes some host and domain analyses more efficient.

Sample Application — Google Analytics, which:

provides aggregate statistics, such as the number of unique visitors per day and the page views per URL per day, as well as site-tracking reports, such as the percentage of users that made a purchase, given that they earlier viewed a specific page.

To enable the service, webmasters embed a small JavaScript program in their web pages. This program is invoked whenever a page is visited. It records various information about the request in Google Analytics, such as a user identifier and information about the page being fetched. Google Analytics summarizes this data and makes it available to webmasters.

The Bigtable for this application consists of one row per end-user session, where the row is named by a tuple consisting of the website name followed by the time the session was created. Then sessions that visit the same site are contiguous in the Bigtable (which is kept sorted lexicographically by row name, recall) and are sorted chronologically. A summary table (approximately 20TB) is generated periodically for each such "raw click table" by running various MapReduce jobs which extract info from the Bigtable (approximately 200TB).

Other applications: Google Earth and Personalized Search.

A lesson they learned in building Bigtable was not to add new features until there's a use case for them.

References:

http://labs.google.com/papers/bigtable.html (2006)
http://en.wikipedia.org/wiki/BigTable
http://andrewhitchcock.org/?post=214
There is also an open-source implementation of Bigtable as part of Hadoop; it's called HBase: http://hadoop.apache.org/hbase/.

Tangential items that came up in writing the above:

Inverted indices: a non-inverted (or forward) index contains a list of documents and, for each document, identifies the words that appear in that document. An inverted index then lists the words (i.e., is indexed by) and identifies which documents each word appears in. So an inverted index maps from content to location. There are two types of inverted indices: (a) a record level inverted index (or inverted file) provides, for each word, references to all documents containing that word. The second type is a word-level inverted index (a.k.a. inverted list or full inverted index) which lists, for each word, the documents in which that word appears and, for each document, the position(s) within that document at which the word appears.
http://en.wikipedia.org/wiki/Index_(search_engine)
http://en.wikipedia.org/wiki/Inverted_index
http://en.wikipedia.org/wiki/Full_text_search
Functional (programming) languages:

Notes on The Long Tail by Chris Anderson

2009-08-15T08:33:00.001-07:00

The Long Tail, by Chris Anderson

Introduction

Our culture has been fixated on best seller lists for about a century. The reason is that the distribution costs for movies, television shows, music groups, even books were such that only a relatively few could be distributed. So the goal was to maximize the audience for the relatively few TV shows, movies, and music groups that are distributed so as to recoup costs. As a practical example, consider your local bookstore. It only has so much shelf space. So naturally it wants to choose the books to populate that shelf space that have the greatest chance of selling. And so the focus on best sellers. Anderson refers to the condition wherein only a small fraction of possible products is provided to an audience as economic scarcity.

For some people, a blockbuster movie really is the best movie they can imagine; whereas for other people (who also see that blockbuster movie), the movie is merely good enough. So there is a sheepherding effect of broadcast distribution where audiences are cajoled into seeing a movie, for example, because it's the best available.

With broadband, every audience member can access different content (as opposed to broadcast which entails sending a single signal to all audience members). Consequently, the old paradigm was a single mass market (a state that is optimized for broadcast economics) and the new paradigm is millions of niche markets. When users are given choice, they exercise it. Furthermore, the best sellers are selling comparatively fewer instance (than they sold before); so these niche markets are taking from the single mass market.

These niche markets always existed; the difference is that the cost of reaching their natural customers has fallen. Anderson presents an analogy for these falling distribution costs; in particular, he likens them to a receding tide. As the tide falls, an underlying landscape is revealed that had been there all along, just underwater. "Think of the waterline as being the economic threshold... the amount of sales necessary to satisfy the distribution channels."

These niches are a great uncharted expanse of products that were previously uneconomic to offer. Many of these kinds of products have always been there, just not visible or easy to find.

Now the niche products are available via Netflix, iTunes, Amazon, ...

Quiz: given a digital jukebox which contains 10,000 albums. What percent of albums sell at least one per quarter? The answer: 98%. "What if non-hits — from healthy niche products to outright misses — all together added up to a market as big as, if not bigger than, the hits themselves?" This 98-percent rule held for Amazon, Apple, Netflix. In looking at the distribution of sales across this vast expanse of inventory, still have lots of sales for a relatively few hits. Then it fell off steeply but never to zero. This is a long-tail curve because the tail is very long relative to the head. Anderson reflects: "I realized that, for the first time, I was looking at the true shape of demand in our culture, unfiltered by the economics of scarcity."

His claims: the tail is longer than we realize (i.e., there's more variety), people can economically access products at the tail (in contrast to pre-broadband times), the aggregation of all those niche markets is itself a significant market. New efficiencies in distribution, manufacturing, and marketing — changing definition of what was commercially viable. The Long Tail "is really about the economics of abundance — what happens when the bottlenecks that stand between supply and demand in our culture start to disappear and everything becomes available to everyone."

Chapter 1: The Long Tail

The tyranny of locality: need to find a sufficiently large local audience for a product in order for it to be economically viable to carry that product. E.g., a movie theatre will not show a film unless it can attract an audience of at least 1500 people over two weeks. The same applies to book or music stores, movie rental joints, and so on. So lots of movies might have a great national audience but can't crack into the various local markets because of this hurdle.

Another limitation of the physical world is that there are limited distribution channels for many goods. A book store shelf can only hold so many books; a radio spectrum can only be divided into a finite number of bands; a coaxial cable can only carry so many TV channels; and so on.

These conditions have meant that one needs to aggregate large audiences in one geographic area. And this has led to the focus on releasing hits (as a way to ensure that the audience-size hurdle is cleared and a profit can be generated). He refers to this as hit-driven economics.

The long tail is long. Consider Rhapsody (an online music retailer) which has (at print) 4M tracks. Every one of Rhapsody's top 1M tracks is streamed at least once a month. Also, if you add up all the non-hits, you get a market that rivals the hit market. For Amazon, 25% of its book sales (by unit? or by aggregate value or dollar amount?) come from outside of the top 100,000 titles. And this percent is the fastest growing part of their businesses. "The act of vastly increasing choice seemed to unlock demand for that choice." "More than 99% of music albums on the market today are not available in Wal-Mart."

Also, these fringe sales are incredibly cost efficient since it costs very little for Amazon to carry another title (much less than it would cost a traditional brick-and-mortar retailer which would need to allocate shelf space for the title and so on). And for purely digital services (where don't even need a warehouse to carry the physical product because there isn't one) such as iTunes, the cost efficiency is even greater.

When you can dramatically lower the costs of connecting supply and demand, it changes not just the numbers, but the entire nature of the market. This is not just a quantitative change, but a qualitative one too. Bringing niches within reach reveals latent demand for non-commercial content. Then, as demand shifts toward the niches, the economics of providing them improve further, and so on, creating a positive feedback loop that will transform entire industries — and the culture — for decades to come.

Chapter 2: The Rise and Fall of the Hit

American was a rural society where people were distributed across the land — creating a very fragmented culture. And so developed regional accents and traditions. Then the industrial revolution brought people to the cities. Then media were developed which created a common culture: commercial printing, photography, the phonograph. Pop culture was transmitted using these technologies, which linked people together and synchronized them. All were reading the same newspapers.

Then broadcast mediums were developed: radio and TV. But initially these were just able to generate signals strong enough to reach local and regional (rather than national) audiences. Then AT&T developed long distance networks, over which voice and audio could be transmitted nationally. The first broadcasts were carried nationally by going over long distance phone lines. 1935 - 1950s was the Golden Age of Radio: Edward Murrow, Bing Crosby. Then TV took over as the principle inculcator of shared culture.

But for the music industry, it peaked in 2000 and by 2005, music sales had dwindled more than 25% from their peak. And the number of hit albums dropped more than 60%. "In other words, although the music industry is hurting, the hit-making side of it is hurting more. Customers have shifted to less mainstream fare..."

Where did the customers go? One hypothesis is that people continued to acquire new music but did so from free providers such as Napster. So it isn't that demand dropped, it's just that this demand could be satisfied without a commercial transaction or sale. Anderson points out that services such as Napster not only provided music free of cost but they also substantially increased the available variety of music. BigChampagne tracks all files shared on the major P2P networks. "What it's seeing in the data is nothing less than a culture shift from hits to niche artists." He also mentions the existence of mashups which are combinations of tracks: one track from one artist played (or layered) over another. Or similar transformations of original content that results in new content. A micro-hit is a hit within a narrow music niche, of which there might be hundreds of thousands.

The traditional model of marketing, selling, and distributing music has gone out of favor. The major label and retail distribution system that grew to titanic size on the back of radio's hit making machine found itself with a business model dependent on huge, platinum hits — and today there are not nearly enough of those. We're witnessing the end of an era.

But current media industries are still oriented to the hit-making model; it's still all about finding and creating blockbusters. "Setting out to make a hit is not exactly the same thing as setting out to make a good movie." And "We are turning from a mass market back into a niche nation, defined now not by our geography but by our interests."

Chapter 3: A Short History of the Long Tail

He makes the point here that the Long Tail phenomenon is not different in kind than previous innovations, only in scale and digital particulars. For example, back in the late 1800s was when Sears Roebuck got his start, offering customers a much wider variety of things and at much better prices than those folks could get in their local "general stores." Sears was able to make this work by creating a mail-order catalog and buying goods in volume. He also listed in his catalog stuff that he sourced from other proprietors (a network of suppliers), who would then ship stuff directly to the customer — which sounds an awful lot like Amazon's Marketplace.

Sears and Roebuck also developed innovative supply chain efficiencies. In particular, they studied and improved the assembly line of preparing goods within the warehouses for shipping to the customer. Finally, they used "viral marketing" by having customers share catalogs with non-customers and giving a referral bonus (discount on goods) to the sharer.

With the development of the Internet, a lot of the constraints of bricks-and-mortar establishments (amount of shelf space, locations, staff, working hours, weather) fall away — leaving unlimited selection.

Chapter 4: The Three Forces of the Long Tail

The theory of the Long Tail can be boiled down to this: Our culture and economy are increasingly shifting way from a focus on a relatively small number of hits (mainstream products and markets) at the head of the demand curve, and moving toward a huge number of niches in the tail. In an era without the constraints of physical shelf space and other bottlenecks of distribution, narrowly targeted goods and services can be as economically attractive as mainstream fare.

"The true shape of demand is revealed only when customers are offered infinite choice."

Lots more non-hits than hits.
The cost of reaching people who would be interested in the non-hits is falling.
Need ways for such people to find or locate such goods. These ways (he refers to them as filters) "can drive demand down the Tail."
Once more niche goods generated and people learn about them, "the demand curve flattens. There are still hits and niches, but the hits are relatively less popular and the niches relatively more so."
There are so many niche products that if you add up demand for all of them, its comparable to demand for the hit products.
Only when infinite choice has been made available to consumers in a way that they can grok it can we determine "the natural shape of demand" — "undistorted by distribution bottlenecks, scarcity of information, and limited choice of shelf space."

A Long Tail occurs when it becomes cheaper to reach niches (as with Amazon, which can offer random books to customers that would not be economical for local booksellers to offer — since the local booksellers wouldn't get enough sales of the random books to justify keeping them in stock). There are a few specific factors that together encourage TLT:

1. Democratize the tools of production.

Basically, make it so that just about anyone can generate stuff. The PC did this; now, webcams and software for blogging and for recording/editing music and video are ubiquitous. "The result is that the available universe of content is now growing faster than ever... the number of new albums released grew a phenomenol 36% in 2005, to 60,000 titles... largely due to the ease with which artists can now record and release their own music."

2. Cut the costs of consumption by democratizing distribution.

For content that consists of bits (e.g., MP3s, videos, ...), everyone who creates can also be a distributor — for example via uploading their video production to YouTube.

Over decades and billions of dollars, Wal-Mart set up the world's most sophisticated supply chain to offer massive variety at low prices to tens of millions of customers around the world. Today anybody can reach a market every bit as big with a listing on eBay.

3. Connect supply and demand.

This consists of the way by which customers become aware of and are able to efficiently search the variety of niche goods. For example, iTunes recommendations, Netflix recommendations, Amazon recommendations, blogs, customer reviews, and so on. All of these things drive demand down the Tail. "Because it's now so easy to tap this grassroots information when you're looking for something new, you're more likely to find what you want faster than ever. That has the economic effect of encouraging you to search farther outside the world you already know..."

Toolmakers, producers: how we make stuff

Populates the Tail.

Aggregators: how we distribute stuff (e.g., eBay, Amazon, ...)

Makes everything in the Tail available.

Filters: how we find stuff (e.g., Google, blogs, recommendations, best-seller lists)

Helps people find what they want.

Chapter 5: The New Producers

The open-source movement has been a software phenomenon where lots of people voluntarily pitch in. But this same phenomenon is occurring outside of software with things like spotting stars since astronomical sitings requires a viewer have his telescope focused on a particular part of the sky, it's easily parallelized with different folks training their telescopes on different parts of the sky. No expertise required.

The pushing down of the tools of production from the few elite to the masses has meant that there are now many more creators. "Don't be surprised if some of the most creative and influential work in the new few decades comes from this Pro-Am class of inspired hobbyists, not frmo the traditional sources in the commercial world. The effect of this shift means that the Long Tail will be populated at a pace never before seen." (Pro-Am is a term which describes Pros working alongside of Amateurs.)

He also talks about the shift from elites generating authoritative content within a controlled framework to the current model where systems are self-governing, as with Wikipedia: "now we're depending more and more on systems where nobody's in charge; the intelligence is simply 'emergent,' which is to say that it appears to arise spontaneously from the number-crunching." The more people who participate, the more the law of large number is at play, which means the better the quality overall. In such probability-based systems, "order arises from what appears to be chaos, seemingly reversing entropy's arrow." Such systems also improve over time.

With these statistical systems, there is only a statistical level of quality: some things will be great, some will be mediocre, some will be very bad. Thus, the quality range is broader in these systems than it would be in traditional governed systems, in which all output will exceed at least a minimum quality threshold. With traditional systems, the range of material covered is much less broad as is the depth at which each is covered. This is for "bandwidth" issues: only so much space can be used and only a finite number of creators. These constraints are not in play with a Wikipedia-like system.

This is the world of "peer production," the extraordinary Internet-enabled phenomenon of mass volunteerism and amateurism. We are at the dawn of an age where most producers in any domain are unpaid, and the main difference between them and their professional counterparts is simply the (shrinking) gap in the resources available to them to extend the ambition of their work. When the tools of production are available to everyone, everyone becomes a producer.

Why do people create content for free? Expression, Fun, Experimentation, Reputation. Some artists are using free content as a way to build an audience that will eventually (hopefully) become paying customers for the wares on display (or for future, related wares). See Lulu.com, which is a do-it-yourself publisher... just about anyone can get published now. Even traditional booksellers are using print-on-demand, which entails printing batches of a book title when sufficient demand warrants it (rather than at the outset printing a humongous batch). With print-on-demand, there is a smaller gamble taken. Print-on-demand converts a physical market into a digital one, freeing booksellers from occupying warehouse shelves with a lot of physical goods.

He also talks about the "increasingly frictionless mobility in the Long Tail," which lets content at the bottom of the Tail move to the top of the Tail quickly and easily. Then he talks about Andy Samberg and his crew and how they broke into SNL. "Think about how many of those potential talents now have a chance to find a real audience, thanks to the democratized distribution of the Internet."

Chapter 6: The New Markets

Consider the used books market. Previously, it was a very difficult market to search the full breadth of. One could go to his local used bookstore which had, more or less, a random collection of used books. One couldn't find a particular used book very easily, though surely somewhere that book was available for resale (or would be if the owner knew there was a buyer). Richard Weatherford developed Alibris to make this market more efficient. Each used-book seller entered his inventory into the database, enabling a person to easily search across the inventory of thousands of used-book stores.

By bringing millions of customers to the used-bok market, this gave used-book stores even more incentive to computerize their inventories, which, in turn, gave Alibris... even more inventory to sell. It was a classic virtuous circle, and the effect supercharged used-book sales...

Alibris is a Long Tail 'aggregator' — a company or service that collects a huge variety of goods and makes them available and easy to find, typically in a single place. What it did by connecting the distributed inventories of thousands of used-book stores was to use information to create a liquid market where there was an illiquid market before... it tapped the latent value in the used-book market. And it did it at a tiny fraction of the cost that it would have required to assemble that much inventory from scratch, by outsourcing most of the work of assembling the catalog to the individual booksellers, who type in and submit the product listings themselves.

Other aggregators (or entities that provide access to something that the entity itself does not own): Google, iTunes, Netflix, eBay, ...

There are a couple categories of online aggregators: those that sell physical goods online and those that sell digital goods online. The aggregators that sell physical goods can offer incredible variety but at some point they hit a space limitation (for storing the goods). (But what if you outsource storage to individual micro-proprietors like Amazon's Marketplace effectively does? Then you don't run into the physical storage limitations, right?) By contrast, the aggregators that sell digital goods can sell everything all the way down the Tail. Distribution costs in the physical goods case are cost-to-ship and in the purely digital model it's merely broadband megabytes. Pure digital retailers can choose between selling goods as standalone products or as a service. So in the digital case, we have "the holy grail of retail — near-zero marginal costs of manufacturing and distribution."

As this program [Amazon Marketplace] continues to grow, Amazon gets closer and closer to breaking the tyranny of the shelf entirely. It doesn't have to guess ahead of time where the demand is going to be, and it doesn't have to guess at how big that demand will be. All the risk within the Marketplace program is outsourced to a network of small merchants who make their own decisions, based on economics, on what to carry.

On Amazon getting closer to economic nirvana: Amazon is using print-on-demand, as described above, which stores books as digial files until they're purchased at which point they're printed on laser printers. "Now Amazon can retain an inventory that takes up no space and has no cost at all: These books and movies remain files in a database somwhere until they're ordered." The biggest cost to publishers is the cost of returns from booksellers. With print-on-demand, this goes away — "potentially reducing returns radically."

"The ultimate cost reduction is eliminating atoms entirely and dealing only in bits. Pure digital aggregators store their inventory on hard drives and deliver it via broadband pipes... Because the goods are digital, they can be cloned and delivered as many times as needed, from zero to billions. A best-seller and a never-seller are just two entries in a database; equal in the eyes of technology and the economics of storage."

Chapter 7: The New Tastemakers

A few stories of how tech is being used in music and how this is changing the music industry.

First, Bonnie McKee was a new singer. And there were questions about how to market her, what audience (demographic) to target. The studio thought she was a natural fit for adult contemporary, women ages 25 - 35. To validate this hypothesis, they did a limited release of her songs using LAUNCHcast, which is an online radio station. A LAUNCHcast user has a playlist created specifically for him (or her) based on his/her ratings of previously listened to songs — i.e., an adaptive recommendation system. The label paid for placement and promotion (on LAUNCHcast) — essentially buying an initial audience but also buying that audience's feedback on McKee's songs. Also, LAUNCHcast is part of Yahoo!, which has lots of demographic info about its users, which could be provided back to the label along with the ratings. Through this experiment, the studios learned that in fact the audience which responded best to McKee's songs were girls ages 13 - 17.

The second story is about a band called My Chemical Romance whose label established a presence for the band on two music-heavy social networking sites. Released a couple singles to limited audiences then used file-trading data (which tracks were the most searched? downloaded?) to figure out which singles to release in what order (release the hottest one now). The band also leveraged its online audience to real-world effects, namely by encouraging audience members to contact radio stations and ask them to play the band's songs. The band's fans were given access to exclusive tracks, access that they repaid via heavy word-of-mouth on behalf of the band.

The third story is about a band that was able to displace the role of the music label entirely by using resources on-line (and because of technological advances which drove down the cost of recording studio time). Birdmonster used the Internet to identify possible gigs and to build a base of support (via MySpace). This base was leveraged by the band in its attempts to secure gigs — to demonstrate that the band would be able to draw a crowd. Birdmonster recorded three tracks at a local independent studio then self published this mini-album via CDBaby. CDBaby takes albums on consignment and sells them online. CDBaby transferred the tracks to iTunes to sell. The band drummed up free marketing by sending tracks to MP3 blogs. As a consequence of this attention, buzz, and sales, Birdmonster started getting offers from established recording labels — offers which Birdmonster turned down.

The traditional role of a music label is to scout talent and provide financing, distribution, and marketing. Because of advances in digital recording technology, the price of studio time was lower than ever — about $15k for an entire album. So financing wasn't a problem. And the band was able to get distribution via CDBaby and Cinderblock which released the tracks via iTunes, Rhapsody, etc. Finally, the band got free high-credibility marketing from various music blogs. Technology has shifted the balance of power from the label to the band.

Yahoo! music ratings, Google PageRank, MySpace friends, Netflix user reviews — these are all manifestations of the wisdom of the crowd. Millions of regular people are the new tastemakers. Some of them act as individuals, others are parts of groups organized around shared interests, and still others are simply herds of consumers automatically tracked by software watching their every behavior.

As the culture fragments from a single mass market to millions of niche markets, we see the rise of microcelebrities — that is, celebrities for each such niche; people whose opinions are respected when it comes to fashion, music, and so on. The slice of the world over which such people have authority can be as narrow as a niche is.

The resources that help a person locate stuff that person might like are filters, which "sift through a vast array of choices to present you with the ones that are most right for you. That's what Google does when it ranks results..." This is not dissimilar from the role that the Yellow Pages played in years of old. Filters have the effect of driving demand down the tail as with Netflix's movie recommendation system which results in most Netflix rentals being "back catalog" (rather than new releases). As Reed Hastings says by way of explanation: "it's not because we have a different subscriber. It's because we create demand for content and we help you find great movies that you'll really like."

As an example of the types of filters that people use in perusing an online music store. You can choose a category (niche) and will be presented with a best-seller list for that category. If you click on an artist or song within that category, you might be presented with related artists and music. You can also see an artist's followers who have perhaps written reviews and created playlists. Finally, you can try a custom radio station tailored around a particular artist or micro-genre. Tehse are all ways of weeding through the vast database of music in the Tail.

There are some limitations to current music filtering. In particular, the same sample format is presented for all genres of music (the 30-second portion of a track) even though this sample may not be equally helpful in evaluating music from all genres. Also, even just the listing format for a song: artist, album. This is specialized for the pop-music domain and may be less appropriate for classical music which has a composer and separately a performer. Ideally, the way things are presented for each niche is specialized to that niche. Bestseller lists across genres are not helpful in the way that rankings within a genre are.

Since the Long Tail contains so much stuff (the vast majority of stuff), the filters to navigate it must be better than those needed to navigate the head, which just contains a comparatively few items. Noise is a big problem in the Tail. Because hit products are tailored to have the greatest audience, they are lowest common denominator. What this means is that the products in the head won't necessarily be the best products for you. Because the products in the head have specifically been designed to be the least offensive, the most vanilla. Hence, the best products for you (the ones that most closely match your specific tastes) are likely to be in the Tail somewhere. And there is a wider range of quality in the Tail generally than there is in the head.

"As the Tail gets longer, the signal-to-noise ratio gets worse. Thus, the only way a consumer can maintain a consistently good enough signal to find what he or she wants is if the filters get increasingly powerful."

There are pre-filters, which determine what gets made and how (i.e., shape the development of a product), and post-filters, which determine how well-liked the made things are. A pre-filter attempts to predict what will best appeal to audiences whereas a post-filter merely amplifies how audiences feel about a particular thing; "in Long Tail markets, the role of filter shiftsw from gatekeeper to advisor. Rather than predicting taste, post-filters such as Google measure it."

Chapter 8: Long Tail Economics

80-20 rule: demonstrated variously; first witnessed in wealth distribution (20% of the people had 80% of the wealth). Zipf noted that the frequency with which a word appeared was proportional to one divided by that word's rank (2nd item occurred half as much as the first; 3rd item occurred one third as much as the first; and so on). These are both examples of power law distributions. The conditions that lead to power laws in consumer markets are: (1) variety of products, (2) inequal quality of products, and (3) network effects that amplify the differences in quality. These result in a "predictable imbalance" of markets and culture... Success breeds success.

If you plot a power-law distribution on a log-log scale (both the x and y axes are logarithmic) then you get a straight line with negative slope (the steepness of which depends on the particular distribution). When we look at box office receipts, however, we note that instead of seeing a straight line across, we see a straight line that steeply drops off. So the line is in effect truncated. What happened? Do box office receipts NOT follow a power-law distribution?

They do. What we're seeing is the effects of the constrained carrying capacity of the US theatrical industry. That is, the movie screens across the country can only carry so many movies; in particular, around 100. There might be some variation in which movies are carried by which theatres (e.g. contrast a megaplex in a Midwestern city to an art house theatre in the East Village) so we see movies ranked 100 - 500 getting box office receipts that obey the power laws. But movies ranked 500 to 13,000 have no theatrical distribution. Hence the steep drop off in box office receipts. "The lesson is that what we thought was a naturally sharp drop-off in demand for movies after a certain point [rank] was actuallly just an artifact of the traditional costs of offering them." What we're starting to see is non-theatrical distribution channels (e.g., the Internet and direct-to-DVD) are becoming major markets.

The 80-20 rule is an example of a Zipf distribution and has classically been applied to products and revenues (20% of products account for 80% of revenues). Since carrying costs for hits and misses were the same (and substantial), this meant that a substantially greater share of profits came from hits (whereas the sales of misses merely covered their costs but yielded no margin). With lower carrying costs of inventory, the 80% of products become more profitable and so proprietors would do well to offer them. A Long Tail retailer, by contrast, carries much more inventory and at cheaper costs. The 20% of traditional retailers (e.g., 1000 albums) represents just 2% of LT retailers (since LT retailers carry much more variety — many more different products). This 2% accounts for 50% of sales, the next 8% accounts for 25% of sales, the bottom 90% accounts for 25%.

Consider new-release DVDs which big-box retailers (e.g., Best Buy) sell at a loss when they first come out. Why? Because they are a loss leader, a product sold at or below cost to stimulate other profitable sales (i.e., they bring people into the store where those people buy more/other stuff as well). Also DVD manufacturers allow unsold new releases to be returned, which lowers the risk to retailers. An effect of this market is that Best Buy and company set the prices for these new releases, even for other businesses which have different models — e.g., Blockbuster which doesn't sell many other products (and hence for which the loss leader advantage is lost).

Blockbuster and Co. want to create a market that isn't so depeendent on new releases then BECAUSE new releases are not profitable for them. The way to do this is to drive demand down the curve. Long Tail products are cheaper to acquire (DVD manufacturers charge more for new releases and less for older titles) and hence can be very profitable — as long as inventory costs are kept close to zero. So, LT retailers: (1) offer many more products, (2) sales are spread more evenly between hits and niches, and (3) there are profits at all levels of popularity.

Does a LT mean that the hits at the head sell less? More? The same?

What shifts demand down the Tail? (1) Variety, (2) Lower search costs — easier to navigate this increased variety in order to find what you want, and (3) sampling — able to "test drive" a purchase (read a chapter of the book, listen to 30 seconds of the song) — which reduces the purchase risk.

Quantifying the effect of lowered search costs for the same inventory: compare online to catalog sales. Both have same inventory; easier to navigate the online inventory. The effect was that customers tended to buy further down the Tail online. In the catalog, 20% of the products accounted for 84% of sales. Online, 20% accounted for 71% of sales.

Measuring the effect of having more selection available: compare a retailer with unlimited shelf space with one which has limited shelf space. Used industry-wide data fror bricks-and-mortar entertainment purveyors. Compared to Rhapsody and Netflix. The results were that the online demand curve is much flatter: the average niche music album titles sold twice as many copies online as offline; the average niche DVD sold three times as many copies online. The top 1000 albums accounted for 80% of the offline market and less than 1/3 of the online market.

Does TLT grow the pie ior just slice it differently?

Some forms of entertainment are non-rivalrous, i.e., you can do something else while you consume them. You're more likely to consume more if it doesn't cost you more to do so. This means that subscription services may be more likely to drive demand down the tail as consumers can explore niche markets without paying more. It may be that users will be more satisfied with their niche purchases than they were with hit products (because the niche product more closely matches the consumer's preferences than one-size-fits-all blockbusters) and furthermore that this increased satisfaction will yield more sales (i.e., grow the pie).

The effects of TLT on pricing? See reading.

Microstructure in TLT

Power laws are fractal — no matter how far you zoomin, they still look like power laws; also described as "self-similarity at multiple scales." The music market is made up of thousands of niche micromarkets or miniature ecosystems, each of which has its own head and tail.

How come the powerful feedback loops that yield 80-20 distributions do NOT have the same effect in TLT — i.e., do not make hits ever more popular and niches ever less so?

Because we've noted that what we see in TLT is a flattened power-law (rather than exaggerated), that is, there is less difference between hits and niches. The answer is that the recommendation networks have more muted effects between or across genres than they have within a genre. "Popularity exists at multiple scales and ruling a clique doesn't necessarily make you the homecoming queen."

TLT of time: new things sell better than old things.

Since TLT is about abundance and economics is based on the premise of scarcity, does that mean that everything we know about econ no longer applies? That is, two of the main scarcity functions of traditional economics are the marginal costs of manufacturing and distribution; these functions are trending to zero in LT markets of digital goods. Anderson claims that since the overall system is still constrained by scarcity, the traditional laws of economics are still in play. Gilder: "In every industrial revolution, some key factor is drastically reduced in cost. Relative to te previosu cost to achieve that function... physical force in the industrial revolution."

That suggests a way to put this in an economic context. If the abundant resources are just one factor in a system otherwise constrained by scarcity, they may not challenge the economic orthodoxy.

Chapter 9: The Short Head

Online shopping still accounts for less than 10% of American retail. There are tactile and instant-gratification advantages of bricks-and-mortar establishments. Also, there will always be blockbusters; "For each way that we differ from one another, there are more ways that we are alike." Hits still have unmatched impact and are a source of common culture. LT aggregators need to provide the full range of variety, "from the broadest appeal to the narrowest, to be able to make the connections that can illuminate a path down the Long Tail that makes sense for everyone" (i.e., familiar head products give customers a toe-hold into the market which enables then exploring products on down the tail). And because "Customers love one-stop shopping." iTunes succeeded in part because it provided a critical mass of mainstream music.

How come MySpace — which focuses mostly on independent music — still works, despite its non-provision of mainstream musicians? Because of its combination of social networking with music, which has the effect of keeping both fresh (don't have burnout of pure social networking where folks are connecting just for connection's sake).

Population clusters also exhibit power laws.

138 million Americans shop at Wal-Mart every week.

The Short Head: Wal-Mart carries only 4500 unique CD titles (Amazon carries 800k). Wal-Mart accounts for 20% of all music sales in America (presumably this includes both their online and physical sites). Wal-Mart has something in about every category of product but not much (broad and thin as opposed to deep and narrow); they provide "a veneer of variety."

Challenge of physical goods: they force us into crude categorization. A can of tuna cannot simultaneously be located in all different categories that it might be in; e.g., health food, canned goods, fish, protein, low-fat, casserole components, and so on. Have to guess at what category most people would use for tuna fish (and for every other product). "With the evolution of onlione retail, however, has come the revelation that being able to recategorize and rearrange products on the fly unlocks their real value." "The efficiency and success of online retail hyave illuminated teh hcost of traditional retail's inflexibility and taxonomical oversimplifications." The tricky question of where to put things: the ontology problem.

On the other hand, think about a world of ad-hoc organization, determined by whatever makes sense at the time. That's more like a big pile of stuff on a desk instead of rows of items stringently arranged on shelves. Sure it may seem messy, but that's just because it's a different kind of organization: spontaneous, contextual order, easily reorderedinto a different context as need be... a world of infinite variety and little predetermined order; a world of dynamic structure, shaped differently for each observer.

The economics of broadcast are such that one can reach a million people as easily as one (for the same cost); and revenues (e.g. from advertising) are variable. There are constraints: only 24 hours in a day, only so many channels per cable. So the way to make money is to get a big enough audience to make the most from that slot. "The general principle of the last hundred years of entertainment economics was that content and distribution were scarce and consumer attention was abundant... It was a sellers' market and they could afford to waste attention." This led to ad clutter; in the mid-1980s, ad time per hour went from 6 minutes 48 seconds to 12 minutes 4 seconds.

More recently, people have started to watch less television. Or rather, people brought up on the Internet watch less television than their predecessors did at their age. "The audience is migrating away from broadcast to the Internet, where niche economics rule."

Chapter 10: The Paradise of Choice

Huge explosion in variety available. Why? Globalization and hyperefficient supply chains. The variety of goods imported increased by a factor of 3 from 1972 to 2001. Another answer: demographics. We are more diverse and people want to be special rather than wanting to achieve some singular common ideal (e.g., being like the Joneses).

But growing chorus that too much choice is no good; it's oppressive because we can't navigate it and become stressed by the fear of making the wrong choice (something that increases in likelihood as more options are made available). The solution is to help users navigate the array of choices so that they can be confident in their particular selection. The navigational support can come in the form of different orderings of stuff: by price, ratings, date, genre. Can determine what "people like you" have preferred (bought). The limitation of the supermarket shelf is the absence of this extra ordering information, which is simply physically more difficult to provide to the customer in this environment vs. in the online environment. This is the resolution of the paradox of choice, which held that consumers loved choice but that having more of it didn't actually seem to make them happier: we not only need choice, we also need information about the various options, so as to increase the likelihood that we'll be happy with our ultimate selection.

The more choice we have, the more we have to decide what we really want. The more we reflect on what we really want, the more involved we get in the creation of the goods we buy and use [via customization]. The more we participate in the creation of products and services, the more choices we end up creating for ourselves.

Does more choice make people buy more, though? Results are inconclusive.

Chapter 11: Niche Culture

House music (the Rave scene) was a reaction against the bankruptcy of blockbuster culture. The spread of affordable technology — mixing decks and multi-track recorders — let DJs produce house music. Clubs and warehouses served as democratized distribution channels. The market fragmented into hyperspecialized genres: deep house, funky house, dub house. Needed a mechanism to navigate this new landscape (i.e., serve as a filter) — the label was used to fill this need. House music producers opened up their goods to be remixed and tweaked: "A house reord that does well often attracts remixes from other producers; it becomes a kind of platform... As the number of complements increases, the value of the platform track snowballs."

Seeing a shift from mass culture to massively parallel culture, where lots of little tribes or microcultures and each of us belongs to many different ones. Almost all of us, though we may be pretty mainstream, "goes super-niche in some part of our lives." That is, Postrel: "Most of us cluster somewhere in the middle of most statistical distributions. But there are lots of bell curves, and pretty much everyone is on a tail of at least one of them." Anderson: "This has always been true, but it's only now something we can act on. The resulting rise of niche culture will reshape the social landscape. People are re-forming into thousands of cultural tribes of interest, connected less by geographic proximity and workplace chatter than by shared interests."

We've seen this in the news industry; "In effect, blogs pick off the mainstream media's customers one by one by being niche where their old-media precursors are mass." A concern about whether this will fracture our relationships to one another — since we are no longer consuming the same mass culture. Anderson claims that "What we've lost in common culture, we've made up in our increased exposure to other people [who populate the niches we explore]... Rather than being loosely connected with people thanks to superficial mass-cultural overlaps, we have the ability to be more strongly tied to just as many if not more people with a shared affinity for niche culture."

Chapter 12: The Infinite Screen

Chapter 13: Beyond Entertainment

eBay: the Long Tail of merchants and of products. Has sales volume equivalent to Wal-Mart's. Distributed inventory: it's just a web site where buyers and sellers meet and agree upon a price. Inventory costs: $0. Self-service model: merchants create own product listings, handle own packaging and mailing. So eBay has very few employees relative to its revenues; it has around $5M in revenues per employee (Wal-Mart has around $170,000 in revenues per employee). Also, eBay provides filters (search, multi-level category stucture) to help buyers navigate.

eBay is the largest used-car dealer and largest seller of automotive parts. It extends from the head to the tail. But eBay is not perfect. It doesn't have a standardized product-description framework and so it doesn't have knowledge of what's being sold on its site — knowledge that could be navigated by a computer program. For this reason, eBay isn't able to build product recommendation systems and so on, which would presumably drive demand even further.
KitchenAid: mixers come in various colors. In making all of them available to consumers, a Long Tail emerged.
LEGO: has a mail-order business that caters to enthusiasts. At least 90% of its products are not available in traditional retail outlets, which accounts for about 10 - 15% of LEGO's sales. These sales have higher margins, though (no retailer to share profits with).
Salesforce.com: rather than ship software that was installed in the usual way and used to manage sales contacts, Salesforce offers software as a service. Customers navigate their browsers to the site and effectively run the software on Salesforce's servers. Meant that businesses could save on tech expenses (relating to administering and maintaining software installations). Benioff responded to the threat of Oracle and Co me-too'ing him to death by opening up his platform to other independent developers to develop their own software as a service and make it available to customers.

The head of the software business is Microsoft. The costs of writing software plummeted, the cost of delivering software decreased as move from CD-ROM to download model. Now can also navigate the world of software products more easily too.
Google: the traditional advertising market — classic, hit-centric industry. The ad-business historically entailed a lot of expensive schmoozing because: "a lack of trusted performance metrics makes salesmanship and personal relationships key to winning business" (I quote this because it precisely describes the type of field I wanted completely to avoid in my own career.) Ad sales people focus on the largest and most lucrative of potential advertisers: "In other words, the system is biased toward the head of the advertising curve."

"As with every other makret we've looked at, that head is just a tiny fraction of the potential market. But because it's so expensive to sell advertising the traditional way, the smaller potential advertisers have been left to their own devices, mostly picking up a phone and placing a classified ad or sending some homemade display copy to the local newspaper. That's pretty much how advertising has worked for most of the past century."

"What Google realized is that if it could take most of the cost out of both selling and buying advertising, it could dramatically increase the pool of potential ad buyers and sellers. Software could do almost all of the work, therby lowering the economic barrier to entry and reaching a much larger market."

Google's ad model: (1) is based on keywords, which is a Long Tail; (2) dramatically lowered the cost of reaching the market — use a simple and cheap self-service model; (3) provided the same benefits to publishers — allowing them to be a platform on which ads could be served the same way that traditional big newspapers were in the past. The pages on which ads are served (i.e., the publishers' sites) are cost-free inventory for Google. So the opportunity costs are born by the publisher, not Google.

Google expanded its search options from keyword and text to other vertical categories: Maps, Scholar, Products, News, Books, Video, and so on. These are like different niches. Search results are customized in a way that makes sense for the category; e.g., video search results would include a clip, for books — a snippet, for images — a thumbnail, and so on.

Quotes from: Closing the Innovation Gap (Estrin)

2009-08-10T14:22:00.000-07:00

Closing the Innovation Gap, by Judy Estrin

Introduction: Innovation Is Not Optional

Chapter 1: The Capacity for Change

Curtis Carlson, CEO of SRI: "When you start a project, you don't know enough about the competition or the customer needs. You haven't developed the best ideas or the best technology. So it's the nature of the game that in the beginning, most of what you're going to do is going to be a failure."

"Best Buy launched an internal initiative to listen much more closely to the employees who were talking to customers every day."

"The willingness to give it a shot and allow people a second, third, or fourth act in their careers has differentiated not only Silicon Valley's but also America's culture from that of other countries. Nokia researcher Henry Tirri says, 'If I pose a queestion to a class of 100 students at a university in Finland, I'll get only one hand up, but I'll be totally convinced that the answer will be correct. If I do the same in the United States, I get 99 hands raised, out of which 90 are probably wrong. But they're willing to try.'"

"Openly sharing information also creates pathways for valuable feedback. Some of the most costly failures in Silicon Valley were the result of projects that had been kept under wraps for too long."

"Increasingly, leaders are recognizing the value of what IDEO, a leading product design company, calls T-shaped people — those who have a depth of knowledge in a particular area, but also the breadth to communicate well with people in other disciplines."

Chapter 2: The Innovation Ecosystem

Chapter 3: Inspirational Innovation

"Paul Baran was a young researcher at Rand when the U.S. Defense Department realized that its strategic command-and-control systems were vulnerable because they were over-centralized. By studying the ways in which the neural networks in the human brain recover from a catastrophic injury like a stroke, Baran and his team conceptualized models for a new kind of telecommunications network that would resemble a fishnet, with redundant links that enabled the network to automatically route communications around damaged areas. They discovered that when you have three times as many links as you need, there would always be a viable path through the network."

Bell Labs: between 1960 and 1983

* transistor, cell phone, laser, light-emitting diode, Unix, digital camera chips

* 6 Nobel prizes

* parent company was a government-sponsored monopoly

* financing came from a tax on phone service

Chapter 4: Narrowing Horizons

"The Vietnam War diverted money from basic research and resulted in increasing resistance to the military among university scientists."

"The founding of Nasdaq opened up another avenue of financial return, which in turn spurred more entrepreneurial activity. Unlike the NYSE's, Nasdaq's "trading floor" existed only in cyberspace. Nasdaq had lower listing requirements and fees, creating a vehicle that enabled smaller, more volatile companies to access public markets and fund their future growth."

"Legislation allowing pension funds to invest in venture capital firms [in the early 1970s] significantly increased the dollars available for startups."

In the 1980s (Reagan administration): "People who had been looking forward five to ten years were now looking only two to three years out, because they had to show commercial justification for their work. 'I saw this huge retraction in the research horizon,' recalls Van Jacobson... 'In Washington, you had to demonstrate two things: you had to use commercial gear whenever possible, and whatever you were doing had to be commercially relevant.'"

"In 1980, Congress passed the Bayh-Dole Act, giving universities ownership of intellectual property resulting from federally funded research... Bayh-Dole created a conflict between scientific motivations and financial ones — between the information sharing that is the coin of the realm in academia AND figuring out how to capitalize on intellectual property." (I reworded this.)

"This shift [toward more scientists becoming entrepreneurial] was heralded by some as progress: academic researchers were being brought down from their ivory towers to face real-world problems. But the effect was to narrow the scope of research to work that could be ported to the marketplace as quickly as possible."

Danny Hillis, co-founder of Applied Minds: "The very process of a company becoming good at its business is often a process of driving out the unknowns and the uncertainties. Lots of people have gotten much better at getting the slop out of the system. Unfortunately, that slop was where a lot of the innovation was happening. It provided the openness to look for new opportunities."

"Start-ups became the research engines of the pharmaceutical industry."

"The discovery of DNA in the 1950s occurred at about the same time as the discovery of the semiconductor. But there was no way that scientists could deal with the vast number of variables and complexity of genetics without the aid of computers... 'It was almost like there was a holdup in biology until the semiconductor led to the type of computing power thta allowed you to be able to look at a system that had 20,000 independent variables, or genes,' says Genomic Health CEO Randy Scott. 'But once the technology for experimentation started to come into play and the computer technology was there to be able to analyze it, that's when the game really started.'"

Around 1989: "The technology industry was so strong that no one noticed that we had stopped planting the seeds of future growth. The rest of the world was not standing still. The number of articles in science and engineering published by U.S. authors stayed flat through the 1990s, while people in other nations were growing more prolific."

"The pharmaceutical industry focused on making drugs for which millions of prescriptions are written every year. 'We drove the world to a blockbuster model,' says Genomic Health CEO Randy Scott. 'It has become so expensive to develop a drug that the only drugs you can justify developing have to be worth half a billion to a billion dollars or more.' To sell those drugs, Big Pharma pushed for the ability to market its products directly to consumers... [in 1997 the FDA changed its policy to allow this] Dollars spent on pharmaceutical advertising have increased 28 percent a year since then." (emphasis mine — wow that's huge)

"By then, the role of start-ups in driving innovation was becoming so clear that many larger companies were beginning to depend on them for their own growth. Partnerships and acquisitions became a key strategy for 'outsourcing' innovation..."

Heidi Roizen on the late 90s: "In that crazy period, you saw the emergence of the entrepreneurial equivalent of the day trader. They saw an opportunity to get rich by building something that somebody wanted to buy and flipping it."

Chapter 5: Losing Our Balance

"Less than a year after the first hint of corporate scandal, in July 2002, Congress passed the Sarbanes-Oxley Act (SOX). The 66-page legislation, passed with very few 'no' votes, established new standards for U.S. public companies, boards of directors, management, and public accounting firms... it added millions of dollars a year in expenses for compliance and legal fees that could have gone toward R&D and innovation."

"The administrative and financial burden of SOX compliance was especially tough on small to medium-sized businesses."

"Shareholderes became more active and vocal, demanding to know, 'Where were the board members at Enron or WorldCom?' With each new investigation, pressure was put on directors to show their muscle, eroding the critical trust relationship between shareholders and boards, and between boards and management."

"There were also new regulations in investment banking that attempted to address real and potential conflicts of interest that arose during the bubble. A side effect of these new laws was that a lot of talent moved from investment banking into firms that were not subject to regulation, such as venture capital and hedge funds."

"The excesses of the dot-com era exposed weaknesses in judgment, controls, and regulation. But the overreaction of the country's leadership had the unintended consequence of creating an atmosphere in which boards and executives were less willing to take meaningful risks on products and business strategies. The real punishment was to long-term shareholder returns through a loss of focus on innovation."

Chapter 6: Green-Thumb Leadership

"To create room for risk taking, you have to actively protect those who have been involved in a project that doesn't succeed beyond saying that they didn't get fired — this time. Too often, other employees want to punish innovators who fail, becqause they were competing for the same resources."

"if you talk about user-contributed content, few people think about Disney. But it was the first to come up with the concept in 1989, recognizing the trend of people using portable video cameras to record family events that were not only valuable keepsakes for posterity, but often hilarious. The result was America's Funniest Home Videos, which debuted nearly two decades ago. 'Why is it that five years ago we didn't see that you could collect videos on the Web and create a site that mirrored the show, with a bit of a twist?' ... It took a start-up like YouTube to think orthogonally and apply the home-video concept to the Web."

"Being open to new ideas also means being willing to cannibalize your own product or business model before someone else does. The music industry could have benefitted greatly by opening its eyes and changing its distribution model before Napster did it for it. The newspaper industry is currently ripe for such a quantum shift..."

"Requiring that a proposed project or service have demonstrated demand can also kill off a company's future growth, he [FedEx CEO Fred Smith] adds. 'It's like saying there's not much demand for people crossing the Hudson River at 96th Street. You have to build the bridge before there's a market.'"

"One force that insidiously undermines the health of baby tigers is the tendency of small business units in large companies to feel that they are undervalued and of marginal importance. Any large and profitable venture tends to become obsessed with the business that led to its first success, which steals energy from the risk-taking, exploratory ventures that could turn into the next one."

"'The genomics of the future is going to be able the science becoming more personalized for smaller markets,' says former FDA commissioner David Kessler... This will require adapting test methodology and developing new ways to identify the efficacy of a drug earlier in the testing process, such as measurement techniques that rely on molecular genetics to track the progress of specific diseases."

"'Most of us have gotten trapped by our own business models,' says Autodesk's Carol Bartz. 'We're expected to make a lot of money so we can take our 90 percent gross margins. But it's a shame we couldn't take 10 of those points and do more research, because we've allowed ourselves to be bound by expectations and become victims of our own success.'"

"In the mid-1980s, Intel made a bold move that ended up saving the company — exiting from the declining memory business and focusing on microprocessors. But that move was possible only because a small team at Intel was already developing microprocessors."

"The corporate research organizations of today can no longer afford to run like the ivory-tower labs of the past, centralized and isolated from the mainstream business. Instead, they shold be porous, providing tight connectivity with the rest of the company."

"Some leaders today have an aversion to the concept of separate groups committed to basic research, seeing them as an antiquated organizational structure. 'The research stuff on the hill is death,' says Eric Schmidt of Google. 'It may be a good model for someone else — though I doubt it — but it's clearly a bad model for us.' ... Google Research is tightly integrated with the core business, which keeps it tethered to real-world needs."

"At Applied Minds, the first test of a new idea is a gut-level one: is the proposed project something that people would feel good about working on? The second test is economic, a classic risk-reward analysis. The third test is whether or not the employees believe they would do a better job of executing the project than anyone else."

"Best Buy has adopted the strategy of creating small, temporary networks to tackle specific projects, drawing people from diverse areas of the company."

"People who naturally play the role of knowledge connectors are critical when building relatinoships across communities, disciplines, or divisions, facilitating communication between disparate groups. The best connectors can quickly synthesize information across a broad range of topics, communicate well, and bring the right people together..."

"Opportunities for acquisitions or partnerships that could bring benefit to multiple businesses within a company often fall through the cracks because no one is willing to champion or pay for something that will be shared across the company. Levy [managing director of Silicon Valley Connect] calls this the 'integrator's dilemma.'"

"'If you overimpose metrics, you sacrifice creativity and open thinking,' says Joost CEO Mike Volpi... Metrics-driven bonuses are easy to implement and understand, and are effective when focusing on yearly financials and incremental improvements. But they need to be complemented with programs that reward behaviors that are not measured in a spreadsheet, recognizing and rewarding value creation in addition to today's revenue and profit."

About offshoring: "For products that are well defined, the assignment of labor to far-flung groups can be relatively straightforward. Forcing this discipline too early can limit the kind of broad-based exploration that thrives in face-to-face interactions and in fruitful collaborations among engineering, marketing, and management. For companies with advanced technology groups, it's best to create networks of complete teams, as opposed to just offshoring a piece of the development."

"Because of SOX regulation, companies now have to be larger to go public, so more companies are opting for getting acquired instead."

"To make a difference, some of these [green] technologies will require infrastructural changes that only very large companies or the government can provide."

"Private equity buyouts of public companies release those companies from the pressure of meeting the Street's quarterly expectations. But the companies end up under mountains of debt, and suddenly every staff meeting is focused on how to pay the interest on the loan..."

Chapter 7: Reviving the National Ecosystem

Chapter 8: Next-Generation Innovators

"But one-third of our eighth graders never receive a high school diploma, and of those who do, 40 percent do not pursue a college education. And half of those who do enroll in college never earn a bachelor's degree." (Shocking if true)

"Some elementary school teachers may have taken only one introductory course in math or science while earning their teaching credentials. 'Many teachers are very fearfl of science and math,' says Sally Ride, 'so they actually teach in a way that discourages kids from asking questions.' But learning how to ask questions and test hypotheses makes these subjects interesting and is crucial in learning how to think like a scientist."

"Teaching math and science skills presents a special challenge because students need to understand each level before moving on to the next. In an increasingly mobile society without consistent educational standards, it's too easy for students to miss a step along the way... Agreeing on a shared set of federal milestones would not be interfereing with states' power to select a curriculum — it would just be ensuring that kids throughout the country are getting the same basic level of education."

"Now the country [Singapore] is implementing what's called the full inquiry method, with teachers emphasizing questioning among small groups of students. 'I've watched groups of seventh graders doing Newtonian mechanics, all trying to get a gut level understanding of what Newton's laws actually mean. Then they have to explain their understanding to other kids in the class. It's just awesome,' he [scientist John Seely Brown] says."

"Most of the chemicals that were once staples of home chemistry sets are now illegal to sell to children. Even in science classrooms, fears about safety and liability are limiting hands-on experimentation."

"we can now build virtual words that would have cost millions a few years ago. 'I got wrapped up into a protein in a 3-D game a few months ago,' says John Seely Brown. 'Now I know why protein folding is so complicated. Before that, I had no visceral sense of what a protein was.' Computation and visualization can bring dry science lessons alive, and even restore a sense of awe about the ways the universe works."

Jamshed Bharucha, Chronicle of Higher Education: "We learn more effectively when we generate the material to be learned than when we passively receive it."

Thoughts from: Talent is Overrated by Geoff Colvin

2009-08-04T14:19:00.000-07:00

Below are some quotes from this book that summarize the major findings/arguments. In all cases, emphasis (italics) are mine.

What genetic brain-science research is telling us

"the research is revealing many ways in which DNA and the environment interact from the moment of conception all the way through life, and suggesting increasingly that the concept of strict nature-versus-nurture conflict is unhelpful in understanding how people actually develop." (pg 81)

Why care about "deliberate practice"?

"the differences between expert performers and normal adults reflect a life-long period of deliberate effort to improve performance in a specific domain" (pg 63)

Deliberate practice "can extend one's ability to perform at high levels far longer than most people believe." (pg 79)

What is deliberate practice?
"Deliberate practice is characterized by several elements..." (pg 66)

It is activity designed (often with a teacher's help) specifically to improve performance.
It can be repeated a lot.
Feedback on results is continuously available.
It is highly demanding mentally.
It isn't much fun.

"deliberate practice requires that one identify certain sharply defined elements of performance that need to be improved, and then work intently on them... The great performers isolate remarkably specific aspects of what they do and focus on just those things until they are improved; then it's on to the next aspect." (pg 68)

How much do I have to do it?

"A finding that is remarkably consistent across disciplines [violin, chess, sports, ...] is that four or five hours a day seems to be the upper limit of deliberate practice, and this is frequently accomplished in sessions lasting no more than an hour to ninety minutes." (pg 71)

How exactly do I do it?

Deliberate practice demands that "we insistently seek out what we're not good at." (pg 71) "He [Tichy] labels the inner circle 'comfort zone,' the middle circle 'learning zone,' and the outer one 'panic zone.' Only by choosing activities in the learning zone can one make progress. That's the location of skills and abilities that are just out of reach." (pg 68)

"Top performers repeat their practice activities to stultifying extent." (pg 69) "More generally, the most effective deliberate practice activities are those that can be repeated at high volume." (pg 70)

"Regardless of how well it [deliberate practice] is designed, another important variable is how much effort a person puts into it... Measuring the intensity of practice may be difficult, but it's clearly significant." (pg 80)

What is the effect of deliberate practice? I.e., how does it make us better?

"Frequently when we see great performers doing what they do, it strikes us that they've practiced for so long, and done it so many times, they can just do it automatically. But in fact, what they have achieved is the ability to avoid doing it automatically. When we learn to do anything new—how to drive, for example—we go through three stages. The first stage demands a lot of attention as we try out the controls, learn the rules of driving, and so on. In the second stage, we begin to coordinate our knowledge, linking movements together and more fluidly combining... In the third stage, we drive the car with barely a thought. It's automatic. And with that our improvement at driving slows dramatically, eventually stopping completely." (pg 82)

"By contrast, great performers never allow themselves to reach the automatic, arrested-development stage in their chosen field. That is the effect of continual deliberate practice—avoiding automaticity... Avoiding automaticity through continual practice is another way of saying that great performers are always getting better." (pg 83)

The above was conceptually how deliberate practice works;

now for a few more details.

Perceive more (where perception can be visual, auditory, tactile, ...).

For a fast serve in tennis, rather than watch (or try to watch) the ball, exceptional players watch the server's body to predict where the serve will land. This is an example of working around a limitation (in reflex speed as far as 150mph serves go). "the accomplished performers, facing the familiar limits on response time, didn't react any faster than the novices, but they understood what they were seeing much more quickly" (pg 87). Similar examples for typists, jugglers, drivers. Other themes: understand the significance of indicators that average performers don't even notice, look further ahead, know more from seeing less, see differences that others don't see.

What if response time is not the perceptual bottleneck? E.g., interpreting X-rays. No points for doing so quickly; rather, the challenge is to determine the correct diagnosis. All were looking at the same X-rays, the top performers were able to pick up on more subtle cues—which enabled discriminating among multiple possible diagnoses so as to obtain the correct one.
Know more. Domain-specific knowledge makes a difference. Doesn't suffice to merely apply superior reasoning methods to an arbitrary problem in a problem-independent manner. Stunning example of this was the quest to build a computer that could beat a man at chess. The computer could consider 100 million possible moves a second and still was beaten by Kasparov. Top performers have more knowledge AND their knowledge is better organized and consolidated.
Remember more. Finding: expert players could glance at a chess board in some position and recall it (all 25 pieces) perfectly whereas novices looking at the same board could only recall the positions of around 5 pieces. But, when instead put pieces randomly on the board, the difference in recall between the two groups was non-existent. "The conclusion was that top-ranked chess players did not possess incredible general memories but did possess an amazing ability to remember real chess positions." (pg 98)

How? The above experiment tested short-term memory; "virtually everyone's short-term memory falls in the range of five to nine items." The chess researchers found that "the masters had only average short-term memories in that they recalled only five to nine 'items,' just like the novices. The difference had to be in what those 'items' were." (pg 99) From whence sprung chunk theory: everyone remembers the same number of chunks of information. "For the novices, a particular piece on a particular square was a chunk. But for the masters, who had studied real positions for years, a chunk was much larger, consisting of a whole group of pieces in a specific arrangement."

Chunk theory explains short-term memory but not longer term memory differences, such as the chess players who would play — blindfolded — ten games simultaneously. The claim is that the memory used in this case is neither short-term (which does not have a long enough time span) nor long term (which doesn't produce fast enough recall) where short-term is like RAM and long-term is like disk. Supposedly, the explanation is something referred to as expert working memory, which involves the expert having some structure in which to incorporate the new information where that structure enabled recall at some later time. As an example, consider a programmer who has written a very long computer program. Because the programmer knows what the various routines he has written accomplish, he can remember all sorts of information about the program — information that would otherwise seem like random letters and symbols to someone who didn't understand the program's functionality.

"the superior memory of great performers doesn't just happen. Since it is built on deep understanding of the field, it can be achieved only through years of intensive study. It further requires consistently relating new information to higher-level concepts..." (pg 102)

Even larger claim: deliberate practice causes the brain to change in helpful ways. Then there is a discussion of myelination. Colvin doesn't claim that DP causes myelination but rather that the way in which DP changes the brain is akin to the way in which practice (in general) causes myelination. Myelination occurs when myelin builds up around neurons and nerve fibers; a neuron or nerve fiber with myelin around it "works better." (pg 103) How does it occur? When send a signal through a fiber over and over, that eventually causes myelin to build up over that fiber. By way of example, stroking a particular piano key in a particular way corresponds to sending a signal through a particular fiber. (But this makes it sounds as if doing something rotely can have a great effect when all previous discussion of deliberate practice makes it seem as though one must concentrate while doing it. The answer to this riddle is that Colvin isn't tying myelination to deliberate practice directly (yet).)

OK, how do I actually apply these findings to become better?

There are a couple different ways to practice, depending upon what skill (or activity) we're trying to improve.

The music model. Use for playing a musical piece, giving a speech/presentation, writing, even preparing for an interview.

In this case "you know what you want to convey, and the challenge is to convey it effectively. The message can be broken down into pieces and each piece analyzed for intent, then practiced repeatedly with immediate feedback from a coach or..." (pg 111)
The chess model. Since there are thousands of books with positions from real games between top players, "study a particular position and choose the move you would make, then compare it with the move chosen by the master; if they're different, figure out why and which is better." (pg 112) Also used in case studies in business school.
The sports model. Training involves two components: (a) conditioning which builds strength and capacities (e.g., endurance) and (b) work on specific critical skills. The tricky part about working on such skills — e.g., hitting a baseball, throwing a football to a receiver — is that they have to be performed differently every time "because the situations in which they're encountered are never the same." (pg 114)

"Conditioning [in business] means getting stronger with the underlying cognitive skills... basic math and accounting in financial jobs, basic science in engineering jobs, basic language skills in editorial jobs." The second type of practice (specific skill development) "is based on focused simulation." Unpredictable opponents, fast responses, dynamic situations. Practicing these can be difficult because they involve other people.

Practicing in the work. Basically, whatever job you have to do that day, make that deliberate practice of some sort. Self-regulation.

* Before the work: set specific goals about the process to use.

* During the work: Observe yourself while doing the work; use metacognition ("knowledge about your own knowledge, thinking about your own thinking"; pg 108).
* After the work: Debrief. Evaluate your performance. Figure out how to do better. Adapt.

Another thing you can work on day-to-day is deepening your domain knowledge. "your objective is not just to amass information. You are building a mental model — a picture of how your domain functions as a system." What does a mental model buy you?

It's a framework in which to easily incorporate new information.
It helps you distinguish relevant from irrelevant information.
It lets you project what happens next.

"A mental model not only enables remarkable recall, it also helps top performers learn and understand new information better than average performers, since they see it not as an isolated bit of data but as part of a large and comprehensible picture." (pg 123)

Skipped Chapter 8: Applying the Principles in Our Organizations

Chapter 9: Performing Great at Innovation

"Time was when you could turn the crank on a good business model for thirty or forty years, and sometimes much longer; the regulated-utility model of AT&T and electricity companies worked for close to a hundred years." (pg 147)

"Creativity and innovation may even be the key to the future economic prosperity of America and other developed countries, at least according to one line of thinking. ... It [this idea that America's future depends on innovation]'s radical because for three hundred years the source of economic dominance has clearly been leadership in science and technology; the countries or regions that were the most advanced technologically have also been the most prosperous." (pg 147)

"how does Target thrive as a discount retailer against the massive power of Wal-Mart, a company more than five times its size that commands by far the world's most advanced retail computer systems? In part it does so by arranging for some of the world's top designers... to design some of the home's most pedestrian products... and then selling them in massive volume at discount prices. Following that strategy, Target can never be commoditized." (pg 148)

"The other thing we think we know about creativity is that it can be inhibited by too much knowledge. We often say that someone is 'too close to the problem' to see a solution. The broader principle is that if you know too much about a situation... then you can't have the flash of insight that is available only to someone unburdened by a lifetime of immersion in the domain." (pg 150)

But... "The greatest innovators in a wide range of fields... all have at least one characteristic in common: They spent many years in intensive preparation before making any kind of creative breakthrough. Creative achievement never came suddenly." (pg 151)

"These findings remind us strongly of the ten-year rule that researchers have found when they study outstanding performers in any domain." (pg 152)

"Until recently, researchers have often thought of creativity in two categories: Big-C creativity, which yields famous, influential products like the integrated circuit or Huckleberry Finn; and little-c creativity, which produces everyday creations like a TV commercial or a florist's arrangement of flowers. But ... have suggested that both types of innovation exist 'on the same developmental continuum,' and that the continuum extends even further back than little-c creativity, to what they call mini-c creativity. In this framework, 'all levels of creative performance follow a trajectory...' This perspective is highly significant because it ties together evidence showing that creative achievement is attained in the same way as other kinds of achievement." (pg 159)

How can an organization foster (encourage) innovation?

Create "innovation networks" within the organization. "'Since new ideas seem to spur more new ideas, networks generate a cycle of innovation.'" (pg 162)
"In a survey of six hundred executives, those at the top thought the main reason why their company wasn't more innovative was that it didn't have enough of the right people. Lower-level management held a markedly different view — that the company had the right people but the culture kept them from innovating as they should." (pg 163)

Organizations should: tell people what's needed and give them freedom to innovate. "What's important is that people understand the organization's priorities and thus know where innovation will do the most good." (pg 164)

"Practice in childhood causes myelin to build up more than does practice in adulthood." (pg 171)

What type of environment is best for encouraging innovation?

Findings from a study of the parents (home environments) of 120 young high achievers (in various fields — art, science, athletics): "Despite wide variations in the parents' backgrounds, professions, and incomes, their homes tended to be child-oriented. Kids were important, and the parents were willing to do a lot — almost anything — to help them. The parents also believed in and modeled a strong work ethic. Work came before play, obligations had to be met, goals were to be pursued... 'To excel, to do one's best, to work hard, and to spend one's time constructively were emphasized over and over again.'" (pg 172)

"In an organization this progression [to a master teacher from an initial convenient teacher] is analogous to choosing developmental assignments taht continually stretch an employee's abilities. Employees aren't children, but many of them, like children, will not voluntarily keep seeking new work experiences taht stress their weakest professional muscles; the temptation to continue doing what you do comfortably is too great." (pg 173)

"A stimulating environment was one with lots of opportunities to learn and high academic expectations. A supportive environment was one with well-defined rules and jobs, without much arguing over who had to do what, and in which family members could rely upon one another... But in the fourth combination, the environment that was both stimulating and suppportive, students were much more engaged, attentive, and alert in their studying... Most organizations are not intellectually stimulating, even when the field itself might seem fascinating..." (pg 174)

Deliberate practice == the fountain of youth?

"Studies in a very broad range of domains — management, aircraft piloting, music, bridge, and others — show consistently that excellent performers suffer the same age-related declines in speed and general cognitive abilities as everyone else — except in their field of expertise... But while excellent pianists slowed down like everyone else in how fast they could respond to a choice on a screen, which is not a skill that makes much difference to a pianist, they didn't slow down at all when it came to piano-related skills like finger tapping or finger coordination. They could do those things as if they hadn't aged at all... When it comes to tasks that are part of their domain of expertise, great performers can keep performing at a high level even after their skills outside their domain have deteriorated." (pg 180)

Why bother?

"[Shizuka] Arakawa's road to the gold medal [in figure skating] involved at least twenty thousand derrier impacts on an unforgiving surface [the ice skating rink floor]... Arakawa's story is not just impressive in itself but also valuable as a metaphor. Landing on your butt twenty thousand times is where great performance comes from." (pg 187)

"A study of figure skaters found that sub-elite skaters spent lots of time working on the jumps they could already do, whlie skaters at the highest levels spent more time on the jumps they couldn't do, the kind that ultimately win Olympic medals and that involve lots of falling down before they're mastered." (pg 187)

Who bothers?

"Creative people are focused on the task (How can I solve this problem?) and not on themselves (What will solving this problem do for me?)." (pg 188-189)

"His [Csikszentmihalyi's] famous work on 'flow' describes a state in which a person is so totally involved in a task that time slows down, enjoyment is heightened, and the task seems almost effortless. This 'high' is achieved when the challenge just matches the person's skills..." (pg 189)

"The concept of flow might even help explain one of the particular puzzles of motivation to practice. It's a considerable 'might' because the research has not been done... at a deeper level one has to suspect that practice is somehow meeting an inner need for anyone who can maintain it at an intense level for years. It seems plausible that the role of practice in producing the highly enjoyable flow state could be part of it." (pg 189-190)

What not to do to a creative person

"'The intrinsically motivated state is conducive to creativity, whereas the extrinsically motivated state is detrimental.' Other studies showed that virtually any external attempt to constrain or control the work results in less creativity. Just being watched is detrimental." (pg 191-192)

Cloud Service providers

2009-07-10T12:51:00.000-07:00

So this is all about determining what kind of app I can run on the Amazon cloud vs. the MSFT cloud vs. the Google cloud. Are these guys all considered "cloud service providers"? Who else should I care about? (surely, Yahoo! has something?) What do their offerings look like? How do they differ from one another? What kind of app do they let me build (i.e., what do I upload to the cloud)? What type of functionality does their API expose (what can my app do that it couldn't do before)? Another question: how nice is the interface for managing one's stored data or web service at each of these cloud service providers? Answer: the GUI web interface for managing cloud resources seems to be a selling point for all.

The biggies of which I am aware:

* Amazon Web Services (AWS)

* MSFT Azure

* Google AppEngine

Also look into:
* Coghead (on second thought...)

Overall Observations

All of the databases provided by Amazon, MSFT, and Google are schema-less. That is, they collectively represent a big shift away from SQL and relational databases and toward something more unstructured, like a hashtable or RDF triple store. Even Microsoft's SQL Service (which is a service that runs in the cloud and can be accessed by applications) doesn't actually run a SQL server!
Each vendor provides a way to upload blobs or large binary objects of data (Amazon S3, MSFT Azure Storage, and MSFT SkyDrive for Windows Live users).

Amazon Web Services (AWS)

We'll talk in the below about:
* Amazon SimpleDB
* Amazon Simple Storage Service (S3)
* Amazon Elastic Compute Capacity (EC2)

Amazon SimpleDB

An alternative to building and maintaining a DB (e.g., SQL) in-house. Used to store, process, and query data sets. Can be used in conjunction with EC2 and S3. Automatically indexes uploaded data; provides query interface for accessing that data.

The actual format (in which data is stored) sounds a lot like a triple (RDF) store. Create a "domain" which contains <attr, value> pairs. Can only query within a single domain (not across domains). There can be multiple entries for a single attribute (attr) mapping it to various different values. No pre-defined or set schema. Amazon indexes the data (for fast query access). Can query via the SELECT or the QUERY API, which are discussed in more detail here. Designed for fast access to a (relatively) small amount of data. Pricing is based on amount of storage, CPU time consumed by queries, and amount of data transferred to/from storage; note that transferring data from storage to EC2 is free of charge (i.e., to perform a calculation over that data). Not storing raw data the way that S3 does but instead storing hash table entries (i.e., attr, value pairs) which they index variously.

Amazon Simple Storage Service (S3)

Simple interface (web service) to store and retrieve any data. How is S3 used? People store at S3: static web data, backups of files, files to share, and data to be processed using an EC2 application. That data is stored on the same infrastructure that Amazon's marketplace runs on; thus, quick access, high reliability, sophisticated, fault tolerant.

Can write, read, or delete an object; each object is anywhere from 1 byte to 5 GB in size. When upload an object, provide a key for it, which is subsequently used to retrieve the object. Each object is stored in a bucket in the same location that the bucket is stored (either US or Europe). Can grant access to the object to the public, to specific users, or to no one at all. Pricing is based on amount of storage occupied, amount of data transferred in or out, and the number of requests on data objects. The operations one can perform on data include: PUT, COPY, POST, LIST; it seems as though a data object is treated a bit like an opaque blob (in contrast to SimpleDB which enables performing SQL-like queries over data objects, which in that case are attribute-value pairs).

Amazon Elastic Compute Capacity (EC2)

The first two Amazon services that we looked at were ways to upload, store, and then access that stored data in the cloud. EC2 provides a way to effectively upload an application to the cloud; i.e., to convert an application that you run on a server that you maintain in an equipment room under your control to an application that runs on Amazon's infrastructure. A small outfit might create a VM (Amazon Machine Image (AMI)) and install Apache web server on that VM, along with all of its site-specific web data. Then they would remap the DNS hostname to point to this VM, which is part of EC2. This means that the small outfit probably doesn't have to worry about denial-of-service (DoS) attacks anymore; plus, they don't have to maintain backup/redundant network connections etc. Many of the IT department is pretty much outsourced to Amazon. The small outfit just produces the web site data and uploads it to its VM, as appropriate.

The way I think of EC2 is as a replacement for a server that is used to host a customer-facing app; the most obvious and common type of these is a web server. So instead of running Apache locally, I'll simply move it to the Amazon cloud. I'm still running the same app on top of the same hardware configuration; the location of the app (and hence who is responsible for ensuring connectivity to the app etc.) has changed. You create an instance on EC2 using the same parameters that you would use in selecting a server: HW config (memory, CPU speed, amount of storage), which libraries should be loaded on the server, and which applications should be loaded on the server. These are not web-apps or applications built to some Amazon API but rather the same Linux-based or Windows-based applications; only difference is that the apps are not running on an Amazon VM.

By contrast, other cloud service providers actually provide an API. Then you write an application using that API. Your application is most likely a web-service. For example, below (with Google App Engine), instead of uploading a virtual machine (and specifying hardware configuration etc.), one merely uploads an application, written using a Google-provided API. So it's much more lightweight but you might need to rewrite your application so that it can be hosted on Google whereas with Amazon EC2 your app runs in an identical environment. So no changes are needed to the application itself. (Note that Amazon does have an API for managing your EC2 instances, such as instantiating and running them as well as configuring access control among them.)

Microsoft Azure

We'll talk in the below about:

* Azure OS

* .NET services

* SQL services

* Live services

Their whitepaper (pdf) — I don't recommend this; it's sort of typically sucky whitepaper writing (where they say that certain things are "harder than most people think" without providing any insight at all as to how or why); that said, there is some information in there. The basic offerings:

Azure Execution: Run a Windows application on Microsoft's cloud computing operating system (OS). Not for running arbitrary applications...
Azure Storage: Store data using Microsoft's cloud computing OS.
.NET cloud service: Provides "distributed infrastructure services," such as a directory service for providing a mapping between a cloud service URI and an actual physical location.
SQL cloud service: Presumably provides data storage/access.
Live cloud service: Synchronizes data across desktops and devices.

An app can do any single one of the above or any combination of the above; e.g., might have an app which runs at the customer site (so-called "on-premises applications") but stores data on the cloud using Windows Azure Storage or stores data using MSFT's SQL cloud service (or both). An app can also run on the cloud, store data locally (on-premises), and access cloud services. Where specifically might an app be running? An app might run on the Azure OS in the cloud, on Windows Server in some company's equipment room, or on a user's desktop (using Windows XP or Vista) or mobile device (using Windows Mobile). This app might access cloud services. So the Azure Services Platform consists of a cloud OS (Azure) on which applications can run as well as a set of "developer services," including .NET services, SQL services, and Live Services.

Here is video of Ray Ozzie (chief software architect at MSFT) introducing Azure. In this, he positions Azure as part of the "Web tier" of MS offerings (where the experience tier contains Windows Vista and Mobile and the enterprise tier contains Windows Server). The goals for Azure were two-fold: (1) Backwards compatible or *respect the installed base*; in Ray Ozzie's words: let current Windows developers apply their existing skills and knowledge as well as deploy existing code. So provide ability to leverage knowledge of Windows developer tools (such as Visual Studio) and environments (e.g., .NET framework). (2) Provide an "open environment" in which programmers can use arbitrary languages, tools, run-times (i.e., not relegated to only being able to use Microsoft tools).

Windows Azure OS: Compute and Storage

There seems to be a bit of redundancy in offerings between (2) and (4) above but there are some key differences. Azure Storage can be used to store blobs and does not use SQL or any other relational DB. Data is accessed via the REST protocol and using "a straight-forward query language" (that is not SQL), whatever that means. The unstructured-ness of the data storage makes Azure Storage sound a lot like Amazon's S3 service which enables uploading blobs (of widely varying sizes) to the cloud. However, one can also use Azure Storage to put data in tables which can be queried; this sounds much more like Amazon's SimpleDB service. So maybe Azure Storage is effectively a hybrid of both Amazon offerings, in which case look for it to have worse performance since it's not optimized for either (quite different) case.

Initially, could only run apps built using .NET framework on Azure OS. These are apps written in C# and "other .NET languages." It's expected that will be able to run non-.NET apps on Azure but presumably these must still be Windows apps (and there are no details here). Managing apps that run on Azure: is via the app's configuration file which identifies the # of instances and other parameteres. To use the web-based management interface, need a hosting account if are going to actually run an app on Azure and a storage account if going to store data on Azure.

.NET Services: to address common infrastructure challenges in creating distributed apps.

Access Control: I think the challenge is that one organization provides a user with an authentication token while another organization hosts the app. The solution is to supply a user with a token which contains a set of claims. That token is provided to the app which determines what type of access the user has, given his claims. This requires a way to translate claims created in one scope (the user's affiliation) into equivalent claims in a second scope (the application provider); this is referred to as claims transformation. Alternatively, need to have identity federation, which lets claims created in one scope be accepted in another.

As a concrete (if toy) example, say that ACM has a policy which lets university students read its articles for free. So we need one entity (the university) to provide a token to each student which the student would provide to the second entity (ACM) demonstrating that he should be provided access to the requested ACM article. How does ACM know whether the token is valid (actually generated by the university) or not? What about token revocation? What language should the token use to encode the "right to read any article for free" such that the ACM article service automatically understands what that right means in this context?

We hear the term federated in tech a lot and it refers to the existence of autonomous domains that cede some power to a central authority which provides some service in exchange (that no single domain itself could provide). When we talk of federated identity management, we mean that one entity vouches for the identify of some user to another entity which consequently provides a service to that user. We might also think of it as providing a unifying description of a single person across multiple domains, which might use different authentication technologies etc. So ACM and the University might know Joe Student in different ways and might store information about his identity differently but they both agree on a particular representation so that each can refer to Joe S. in a way that the other understands. Some additional reading on federated identity mgmt. What's the big deal? Why would you care? FIM lets a company easily outsource various business functions without requiring that the company divulge all of its customer information.
Service Bus: It sounds like this is a directory or mapping service; an on-premises application provides a web service endpoint to SB. SB generates a URI for this endpoint; clients can locate and access the service via this URI (and with help from the SB's directory service which lists all such URIs). Presumably, the URI resolves to some network location in the cloud and all traffic to/from that location is forwarded on to the on-premises application. So the SB acts as a network proxy for traffic to/from the web services endpoint. The actual mechanics is that the web service endpoint opens a persistent connection to the SB; this connection is not torn down. Since the connection originates at the on-premises location, there is no need to punch any holes in the firewall. Traffic is sent over this connection. The client only ever sees the SB's IP address, not the IP of the on-premises web service.

Maybe the Service Bus can also do authentication so then instead of merely passing all received (client) traffic to the target application (endpoint), the SB will check the client's credentials and only pass traffic through to the target if those credentials are sufficient.
Workflow: Have some logic in the cloud that coordinates various applications, integrating them to achieve some overall effect. Maybe each individual application communicates with the Workflow service to get its inputs and provide its outputs; the Workflow service in turn passes the first app's output to the second app as input and so on. (Again, all conjecture!)

SQL Services: A Database in the Cloud

Exposes both SOAP and RESTful interfaces. Built on MSFT's SQL Server. It sounds like data is stored in a RDF-style format (unstructured); a single data item consists of a property which has: a name, type, and value, which sounds a lot like an RDF statement which consists of a subject, property, and value (and is also reminiscent of Amazon's SimpleDB service described above). No pre-defined schema. So why didn't they just offer actual SQL Server in the cloud? Issues with relational DBMS: scalability, availability, and reliability. The unstructured organization of data makes replication and load balancing easier and faster.

Live Services: MSFT's API for web applications

Windows Live is a family of web applications provided by MSFT. It includes an email app, an instant message app, an app for managing photos, and an app for managing contacts. There is also an app for storing data (SkyDrive) as well as a calendar, something like Evite, and something called Office Live (for creating/editing documents). These applications use Live services to store and manage their data. What the Live Framework is all about is making that data available to other applications, i.e., to applications that are not part of the Windows Live suite. This would let users (you!) create apps and leverage that data; e.g., an app which used the map or directions info provided by the Live Framework. The MSFT Live suite is very reminiscent of Google's set of web services (such as Gmail, Google Docs, Google Calendar, Google Maps, Picasa, and so on).

What's sort of interesting about this is that MSFT talks about making map, email, contact, etc. data available through Live Services rather than talking about exposing a web services endpoint on each individual application (e.g., creating a web service endpoint as part of Hotmail which responds to queries for its data; doing the same for Live Office, Sky Drive, Messenger, and so on). Rather than creating all of those endpoints, since these applications already manage their data through Live Services, just let arbitrary web applications also access that data through Live Services (via Live Framework). So they're creating a single interface on Live Services rather than an interface on each MSFT Live application.

The below figure is my conceptualization of Live Services. There is some suite of Windows apps that interact with and rely upon Live Services in order to store their data; this includes those mentioned above: Windows Live email, contacts, SkyDrive, Maps, Photos, ... The basic idea is to allow other applications to also access that data (maps, directions, photos, calendar, contacts, email, chat, ...). The way this is achieved is that Live Services lives in the cloud and is accessed (using HTTP or RSS) via the Live Operating Environment (LOE), which also lives in the cloud. Thus, any application that can request data via HTTP can use Live Services data (doesn't need to be a .NET framework app or even a Windows app generally).

The most exciting thing about Live services is that you can use it to synchronize data across systems; e.g., you could create a mesh which consisted of your desktop computer, laptop, and mobile phone. Each such device would run an instance of Live Operating Environment (LOE), which would be responsible for synchronizing data across all systems in the mesh (think of it as a network of communicating LOEs). The user specifies which data on these different systems should be kept in sync. Note that the different devices may be running very different operating systems, even OSs not in the MSFT family. LOE can be run on: Vista, XP, Mac OS X, and Windows Mobile 6. Every mesh also implicitly includes the user's Live services account and hence includes his data stored on the cloud. Thus, if a user has contacts stored on his hotmail account, these could be synchronized with those on his mobile device and so on. A user can also provide access to data from his mesh with other users.

An app (which runs on a user's device or in the cloud) accesses that user's mesh data through the local LOE or from the cloud LOE. If the app wants to access the mesh data belonging to a user's friend, the app presumably interacts with the cloud LOE, requesting the friend's data (presumably the friend has already provided the necessary permissions). A user can also run an application on his mesh (i.e., on any machine within that mesh); data modified by that app will be kept synchronized. There is a directory of mesh-enabled web apps. A mesh-enabled web app can also be shared with friends (this is their social-networking-apps angle).

Another option involves a user who has a Windows Live account (with lots of associated data — from various Windows Live applications) but he has an iPhone, Linux computer, and so on (no device which runs a Windows OS). And the data he wants to keep synchronized lives on these systems for which there is no LOE. The solution is for the user to have Silverlight installed as a browser plug-in on each device; .NET apps can run on Silverlight; it's like creating a Windows execution environment inside the browser on top of any arbitrary OS. Then this app (which is built for Silverlight) will be able to access the cloud LOE. This is MSFT's multi-platform support, I think, and it uses Silverlight as a bootstrap. The application in this context is a web app.

(How does this work for data which is represented using different formats on different systems? E.g., office documents on Mac? contacts on Mac? and so on. Maybe practically the system works much better when all devices are running a Windows OS. Also: how about Live Services data? Which of it is available to everyone? Only the generic info? Is it possible for a user to make his data available to everyone? Or temporarily? I'm thinking about my Personal Marketplace idea here. And how a user might provide temporary access to (some portion of) his Live services data for a fee.)

Questions
* What is the isolation between multiple apps all running on the cloud OS (Azure)?

* Accessing the services described is via a new API? Or existing API with new arguments?

* What about non-MS apps? Can they run on Azure? Store data on Azure? Take advantage of cloud services?

Google AppEngine

More on AppEngine; note to self: read following and update below http://www.stanford.edu/class/ee380/Abstracts/081105-slides.pdf

AppEngine provides a way for folks to run their web applications on Google's infrastructure; it is not a general-purpose computing platform for any arbitrary code. They run and serve code for you; your code can use some of the functionality used by Google's own web apps (e.g., Google Docs, Gmail, ...), such as Google Accounts, Google File System (GFS), and BigTable. Can write an AppEngine app using the following languages: Java, JavaScript, Ruby, Python. Your app can run either in response to receiving a web request or to a cron job. You can also limit who can execute your app. AppEngine exposes a bunch of APIs including: datastore, memcache, URL_fetch, mail, images, and Google Accounts.

AppEngine consists of:

A "scalable serving architecture" — i.e., a way for a request to your web service to reach a running instance of your code. When you submit code to Google, it will be pushed to a bunch of fault tolerant servers. Google automatically scales resources for your code on-demand (you don't neeed to specify a priori the # of machines, CPUs, etc.).

Your code runs in a sandbox which provides limited access to the underlying OS. The sandbox disallows writing to the file system; an app can only read files that are uploaded with the app's code. Google provides some APIs for persistent storage. So for example if your app's Java bytecode tries to open a socket or write to a file, a run-time exception will be generated.
Python run-time (and now a Java run-time too). The Python run-time includes a fast interpreter. There is a uniform infrastructure, consisting of the API and developer tools, which apply regardless of the language you're programming in.
SDK: is written in python; there's a release of it for Linux, Mac, and Windows. Comes with a AppEngine simulator that lets you test, run, and debug your apps locally (don't need to deploy code to their servers in order to see what the code does).
Web-based Administrative Console: see app status, control who can administer the app, control which version of the app receives the most hits; also contains a bunch of tools, including those which: let you look at request logs, app logs, your data objects, let you hook up a domain to your app, let you see which errors are being received on which URLs, see traffic reports, etc.
Scalable data store, i.e., their persistence layer: this is based on BigTable instead of SQL.

The Application Environment

* Serves web pages

* Automatically scales resources and load balances on-demand

* Provides API for authenticating users

* Provides API for sending email (using Google Accounts)

* Provides persistent storage which can be queried and sorted.

* Provides local development environment which simulates AppEngine on a user's computer

* Provides scheduled tasks

The Sandbox: limits access to the underlying OS

Why? So that your app is portable... i.e., can be migrated from any system to any other system (since doesn't have any environmental dependencies). How does sandbox constrain app execution?

Network: Outbound: your app can only connect to other hosts via the URL fetch and email services. Inbound: your app can only receive connections via HTTP or HTTPS on the standard ports.
File system: Writing: App cannot write to the file system. Reading: App can only read the files that were uploaded with the app initially. Instead, the app should use the datastore, memcache, or other services in order to maintain state across requests.
Execution: app is only executed when receives a web request or a cron job. An AppEngine program consists of a map between URLs (on which requests might be received) and the applicable program to run for each. An app must return a response within 30 seconds of being executed. After returning that response, the request handler (i.e., app) cannot spawn a subprocess or execute code.

The Java Runtime Environment

The JVM implements the restrictions on the sandbox environment; hence, bytecode that tries to write a file or open a socket will throw a run-time exception. The AppEngine implementation for Java also includes an implementation of some standard Java API functions so that such functions use AppEngine services. For example, if your program invokes the java.net HTTP APIs, your program will be using the AppEngine URL fetch services. This might make porting existing apps easier since don't need to program to an entirely new API; rather, the changes are under-the-hood. Similarly, the APIs for JavaMail, Java Data Objects, and Java Persistence are all reimplemented by AppEngine. That said, AppEngine does have some new APIs which can be used to access its datastore, memcache, Google Accounts, URL fetch, mail, and images services. (So in some cases it appears there is >1 way to skin — or URL fetch — a cat.)

The Python Runtime Environment

See: http://code.google.com/appengine/docs/python/overview.html

The Datastore

So this sounds like an RDF store which contains triples, each of which consists of a subject, predicate (or property), and object (or property value). Queries can be executed over this store, and the store is actually spread across multiple machines (so it's distributed). A property's value can be String, Boolean, byte string, ..., and so on. This is another example of a non-relational database; it's schemaless. That said, your application code can create and enforce some structure on your datastore (would this be akin to requiring that data be part of some ontology?). Use the Java Data Object (JDO) and Java Persistence API (JPA) to access the datastore or can access the datastore directly using its own API.

The datastore is strongly consistent and uses optimistic concurrency control.

What is app execution cycle?

* Receive request on URL X

* Google AppEngine identifies handler (i.e. program) for X

* AppEngine invokes that handler (executes that program)

* Program must return a response within 30 seconds

An AppEngine application

An app consists of a *.yaml file which identifies the application's name and version, as well as which run-time and which version of the AppEngine API the app uses. The YAML file also provides a mapping from URLs to code which should be invoked when such URLs are visited. You can start up the development web server (on local machine) via: $dev_appserver <appName> which will print out a URL which you then visit in your browser to test the app. E.g., the following encodes that visiting any URL should result in invoking the main.py python script.

-url: *.
script: main.py

Below is from Campfire One, which was held in April 2008.

Geambasu — Menagerie (WWW, April 2008)

2009-06-29T09:30:00.000-07:00

Notes on: Organizing and Sharing Distributed Personal Web Service Data (pdf)

Authors: Geambasu, Cheung, Moshchuk, Gribble, Levy (Univ of Washington)

Venue: WWW, April 2008

My initial comments: Allow me to warn you that in all likelihood you will have an easier time digesting this material if you go straight to the source. In the below, I wrestle with what the hell the authors are talking about in certain places which is sure to confuse you more if the same questions didn't arise in your own mind as you were trying to understand how the system works. I.e., caveat emptor. On the other hand, if you want to understand how to become a better writer by seeing firsthand how many different ways something can be interpreted then, by all means, eat your heart out.

The problem: A user's data is scattered across the Web and lives under the control of different domains (Flickr, Google Docs, Google Calendar, Picasa, YouTube, ...) which expose data differently (in some cases at a stable, predictable URL and, in other cases, not) and provide different ways to share that data. How can a user: (1) organize, search, and archive that data; (2) create a collection of data that consists of objects owned by different players AND share that collection "in a protected way"; (3) manipulate these data objects using standard applications or scripts (similar to the way that a user can manipulate each file on his desktop system using hundreds of utilities, such as cat, ls, grep).

With desktop computers, a file can be operated on by multiple different applications regardless of which application created the file. Whereas in the web service world, data created as part of one web service (e.g., docs authored at Google docs) cannot be operated on by other web services or using standard utilities (such as ls and grep). Also, on the desktop it's easy to compose applications or utilities in order to get new, more interesting functionalities; e.g., think of all of the command-line shell utilities (such as grep, find, ls, cat, ..., which can be combined using various operators (e.g., pipe, input or output redirection, and so on)).

Their solution: Have each Web service implement a standard interface that enables accessing and modifying objects that live on that Web service. Then a composite application can aggregate objects from these various services in order to provide a user with a single logical view of his scattered Web data. Note that in their conceptualization, the composite app isn't actually downloading and managing the various objects but rather embedding pointers to those objects and presenting thumbnails of them. But the solution principally is a standard API, implemented by each Web service that houses user data. The API deals with how one refers to objects (naming), how Web services expose objects via stable URLs, how access to objects is mediated, and how data objects are shared.

Thus, the solution requires the various Web services to do something in order to support it; as an intermediate deployment strategy, one can create a proxy for each target Web service where the proxy effectively implements the API on behalf of the service. Naturally, each proxy would be very Web-service-specific.

Goals: want a user to be able to create a collection of objects and specify policy for that collection as a whole despite the fact that the objects live under the control of different web services which export different access control policy frameworks. Another goal is to enable a user U to share her data with user V even if some of that data lives on web services where V does not have an account (and does not want to get one); i.e., want the ability to share data that is part of some web service with a user who does not have a relationship with that web service.

Challenges

Not all Web services expose the objects they house in a way that enables referring to such objects externally.
Every Web service has its own access control, authentication, authorization framework. No single uniform way to configure access control policy. A Web service may only support coarse-grained (all-or-nothing) access control policies.
To access a person's data on Web service S, you may be required to have your own account with S.
The access control frameworks of Web services might not support write privileges and might not support revocation of rights.
In addition to possibly not providing a stable URL to access a particular object, Web services might not provide a way to programmatically manipulate such objects.

What will web services need to do to support this solution?

Each origin Web services must export an interface to enable other web services to access data that lives on the origin Web service. Each origin Web service must provide metadata for each data object O (that lives on the service) such that O could be rendered (and operated on) within an arbitrary web page.

What does the solution require?

Need a single global namespace. Every object and object collection must have a unique and permanent name in this namespace.
The access control interface (to a user's heterogeneous data/object collection) should enable that user to share some portion of that data (rather than requiring the user provide all-or-nothing access). This interface should also enable an entity to access an object on a Web service even if that entity has no account with that service.
Each object should support some set of functions which will be invoked on the object by other Web services or utilities or directly by the end user. One such function is that each object must be able to be embedded and rendered within an arbitrary web page.

So they built a system that realizes this vision...

Menagerie

Consists of Menagerie Service Interface (MSI) and Menagerie File System (MFS).

Menagerie Service Interface (MSI)

At first blush this is sort of confusing in terms of who implements the interface? Who uses this interface? These are incredibly simple questions with only a limited range of answers, given the context, and yet the answers are not explicitly stated. So we'll be forced to reason them out; so let's look at the described API functions and the players, which include the origin web services (e.g., Flickr, Picasa, KG) — which own users' data objects — and non-origin web services (also referred to as composite web services) — which access objects owned by other web services. A few types of functions are mentioned:

An origin web service calls export in order to add the name of one of its objects to the global namespace. Presumably, this function is implemented by a special Web service, namely the global namespace Web service which maintains a directory of all objects exported by any Web service. This function returns the stable URL for the object?

Questions: So is this like a directory service where I can obtain a URL given some object description? Where does the URL point? To the origin web service? I.e., who actually hosts the various data objects? Presumably, the origin web service hosts its objects and must provide an externalized link to such objects (i.e., must make a stable URL available which can be used to refer to / name the given object).

Comments: I think I was taking the term "export" too literally; they just mean that if you query a Web service (using some MSI API function), it will provide a unique name for each object which can be combined with the service's domain name to obtain a globally unique object ID.
A capability is provided by an origin web service (e.g., Flickr) to a non-affiliated user (e.g., my brother who doesn't have an account on Flickr) to access a particular object (e.g., my album of 4th of July photos) for some period of time.

Questions: but who actually asks for the capability? The object owner?? (E.g., my brother asks me and I interact with Flickr to obtain the capability then give it to my brother?) Otherwise, how does Flickr authenticate the person asking since Flickr doesn't have a relationship with that person?
The third type of API function mentioned as part of MSI are object-independent access functions, such as, to read, write, or render a particular object or to obtain an object's metadata. These are perhaps the most straightforward to imagine who implements them and who invokes them. Presumably, an origin web service provides an implementation of these access functions which are invoked by a non-origin web service in order to interact with an object owned by the origin Web service.

For example, let's say we have a photo aggregator app (i.e., a non-origin web service) which creates a virtual file system which contains thumbnails of all of my photos (which are stored, say, on various photo sites, such as, Flickr, Shutterfly, Picasa, and KodakGallery). So this photo aggregator app (PAA) would be invoking these functions on individual photo objects. For example, let's say p1, p2, and p3 all live on Flickr, the PAA will call p1.getThumbnail(), p2.getThumbnail(), p3.getThumbnail() to obtain the thumbnails of photos p1, p2, and p3, respectively.

Let's recap; below is our conjecture:

Function	Does what?	Who implements?	Who invokes?
export	Exports a name for an object into the global namespace.	The web service which acts as the global namespace.	The origin web service.
generateCapability	Obtains a token to present to an origin web service in order to gain access to some object stored on that web service.	The origin web service.	Some non-origin web service. Or perhaps the user who owns the object.
readObject	Returns an object's contents.	The origin web service.	Some non-origin web service.

Menagerie File System (MFS)

Allows an app to mount a remote MSI object hierarchy into a local file system namespace. For example, I could mount Ann's entire collection of photos (which lives on KodakGallery) and operate on those photos locally. (Would the effects of my edit or tagging operations be pushed back to Ann's version on Kodak Gallery?) So thereafter my app can operate on Ann's photos as if they were files stored on my computer.

Implementation particulars

After all of that rife speculation, let's get down to the details. They talk about three types of operations: "object naming" (list, getattr, mkdir), protection (create_capa, revoke_capa), and access (read, write, get_summary, get_URL).

Object Naming

Each thing (object) has two names:

One provided by the user; this name is meaningful to the entity who owns the data. For example, a photo on Flickr might be referred to using the path to the photo (i.e., the name of the album that contains it) along with the actual photo name.
One provided to composite web services for them to use in invoking operations on that object; this is referred to as the service-local ObjectId or ObjectId for short. Menagerie generates globally unique object IDs by combining an object's service-local ObjectId with the domain name of the service hosting that object. So if Flickr contains objects with service-local ObjectIds n1, n2, ... then the Menagerie names for these objects will be flickr.com:n1, flickr.com:n2, ...

Presumably, the origin web service generates this service-local ObjectId (they don't say). I assume that the origin web service creates the ObjectIds because the origin web service implements the MSI interface and most functions in that interface take this ObjectId argument. Hence, that value must be meaningful for the service.

Calling list returns the mapping between the two names. A user's object name hierarchy is the collection of his names for all of the objects he has access to; note that a user's hierarchy may contain the names of objects that belong to someone else. The service-local ObjectIds can be independent of where the object lives within its owner's hierarchy.

Why need two different names for a single object? Because we don't necessarily want to expose a user's organization of her photo albums to the people with whom that user shares a photo? So that the name for an object is stable, even across operations such as the user renaming/moving that object somewhere else in the hierarchy (this is probably why). Also, by using a single ObjectId to refer to a single object, they can identify all accesses to that object and cache it as appropriate. By contrast, if an object is reachable by many different users and the object lives somewhere different in each user's hierarchy and we used those hierarchical paths to identify accesses of the object, we might miscount the number of accesses to an object since that object has different names according to different people.

They describe operations which can be performed on a name hierarchy; presumably, these operations are implemented by each origin web service.

list: given a directory's ObjectId, return the name and ObjectId for every object within that directory.
getattr: given an object's ObjectId, return metadata about that object (e.g., its type, size, last modified date, etc.) as well as a capability for that object.

(So this function always gives out a generic capability to whomever asks? Or it performs a privilege check? Or the access control is done earlier — for example, a user is only given the ObjectId for objects that this user is allowed to access. So knowing the ObjectId implies that one has sufficient privileges to read the object?

ANSWER: They assume that capabilities are generic (rather than specific to the holder of the capability) and the distribution of them is only to people who should have them. A web service can however require that a user authenticate himself to the service in addition to requiring that the user possess the necessary capabilities.)
mkdir: add a collection of objects to the hierarchy (to the user-specific hierarchy?).

This is a little strange. Presumably if a user wants to organize his objects on the target Web service, he can do so using that Web service's available controls. Not sure where the created directory lives. (Or is this for augmenting some other directory hierarchy structure, for example one maintained by the service itself for its own purposes?)

Can definitely imagine a composite web service maintaining a user-created directory structure; perhaps that's the intended context for this API call (rather than thinking of mkdir as creating a directory on the origin Web service). Not sure.

Again, it's at least modestly unclear who is invoking these functions. Does the data owner invoke these functions in order to manage his data? Maybe each user U who has an account with a web service S invokes these functions; in this scenario, the owner U of an object obtains the capability needed by a friend F who wants to see that object. U presumably can obtain the root directory's ObjectId and then calls list as appropriate to obtain the ObjectId of a photo P that U wants to share with F. Then U calls getattr on P's ObjectId and returns the provided capability to F. At least in this (conjectured) case, it makes sense how authentication works since we have a user U who has an account with S invoking functions implemented by S.

Protection

They have a hybrid capability-based protection system. What's the "hybrid" part? With normal (non-hybrid) capabilities, possessing a capability alone suffices to gain the described access to the specified object. But in Menagerie, an origin web service can require that a user provide a capability as well as authenticate himself in order to get access. A Menagerie capability is an unforgeable token that contains the globally unique ID for some object as well as a set of access rights. The semantics are that a user who holds a capability is allowed to access the specified object in the specified way(s). Note that a capability does not contain any identifying information as to who is allowed to access the specified thing in the specified way; the right applies to any user who holds the capability.

As mentioned, a web service can also exert some control over who exercises a capability. In particular, a web service issues two types of object rights: (1) open-access, which gives the capability holder direct access to the specified object without further authentication, and (2) closed-access, wherein the web service can also require that a user authenticate himself.

Implementing capabilities

If a user possesses a capability for some object O, he actually possesses access rights for all objects rooted at O. So the idea is that a capability might be given for a directory D, enabling the holder to access any object within D or within a subdirectory of D, and so on. Each capability consists of an 8-byte "root node global ID" (which is presumably the object's service-local ObjectId along with the domain name of the service where that object lives) and a 16-byte password. These two values are also stored at the web service (that issues the capability) along with the applicable open- and closed-access rights for that object. The password field is to ensure that an attacker who knows the global ID for an object cannot successfully forge a capability for that object.

Most MSI functions require a capability as well as the ObjectId:

* list( capa, objectId )

* getattr( capa, objectId )

* create_capa( capa, objectId, rights )

* read( capa, objectId ): to read an object

* write( capa, objectId, name, content ): to write an object

* get_summary( capa, objectId ): to get an HTML thumbnail of an object
* get_URL( capa, objectId ): to get the URL of an object on its home Web service

The exceptions are:
* mkdir( capa, parentId, name )

* revoke_capa( object_capa, revoke_capa )

Revoking a capability for an object O requires a valid capability on O which has the revocation rights enabled. Revocation entails zero'ing the rights field in that capability's CapTable entry.

Then they give some real-world examples of web services using capabilities.

Accessing Objects

MSI provides a few ways to access objects:

Can embed an object from a remote service within a page. The service that owns the object is responsible for its presentation and interaction controls.
Can get object metadata via get_summary, which returns an HTML code snippet that displays a thumbnail of the object, and get_URL, which returns the object's URL within its parent service.
Can invoke an object-independent access function (e.g., read, write, delete) on an object.

Implementation

The MSI is an XML RPC layer on top of HTTP which means that you can invoke an MSI function by making an XML RPC call, which can be done in various programming languages, including JavaScript.

To build a composite Menagerie app, need web services that support MSI. As an incremental deployment strategy, they created MSI proxies for non-MSI services (including Gmail, Yahoo! mail, Flickr, YouTube, Google Spreadsheets), each of which implements the MSI functions and Menagerie protection model on behalf of a service. So each proxy is service-specific. For services which provide a developer API, creating the proxy was straight-forward. For other services, had to do screen scraping etc. which probably wouldn't scale and is kind of a mess generally.

The Menagerie File System lets an application mount a user's MSI name hierarchy then operate on objects within that hierarchy as if they were local (e.g., invoking system calls or utilities (such as cp and tar) on such objects). A system call (such as fopen, getattr, readdir, read, write) executed on a mounted object gets passed via VFS to MFS which translates that system call to the corresponding MSI call on the Web service which owns this object. They use a couple caches to speed up access to the mounted FS.

Menagerie Applications

The Menagerie Web Object Manager (WOM): use this to create a virtual file system where can drag and drop various objects stored at various web services (without actually affecting how those objects are stored at their home services). Can also share a view of this file system (i.e., export some portion of this file system) to other users -- to give them access to some part of your virtual hierarchy. So in this way, the composite application is both a client (for each web service that owns some object contained in the virtual hierarchy) and a server (for other applications that request to see part of this virtual file system). The kinds of objects one can store in a folder include videos, emails, pictures, spreadsheets, and so on. Moreover, if a user clicks on an object within the folder, the user will be directed to the object's home location (on its parent web service). So if a folder contains a Google docs spreadsheet, the user can click on the particular spreadsheet icon which will open up the spreadsheet (possibly for editing) in Google docs. As mentioned, WOM exports MSI which means a user can get a capability for a WOM virtual folder and give that capability to others.

The Menagerie Group Sharing Service (MGS): instead of a single user creating a collection of Web objects then optionally sharing that collection (read-only) with others, MGS is all about a group collectively owning a collection into which each group member adds various objects from different Web servioces. Lets users share a single virtual desktop.

Some questions

Are capabilities the right model for this? Will a user understand that a capability (i.e., this thing which looks like a URL) should only be forwarded to or shared with someone to whom you want to grant access? When we repurpose things that users know in one context (e.g., URLs) to have security relevance, this can sometimes cause problems.
Once we have some Web service that uses closed-access rights for its objects, what's the benefit of a capability in that scenario? Is the idea that we can partition the various rights that a user might have for an object so that only a small slice of them require authenticating with an object's home Web service? For example, broadly allow users to read an object but writing an object requires authentication.
Somewhat out of the scope of this document is what the typical life cycle of a capability will be. E.g., some user wants to share her 4th of July photos on Flickr with family and friends so she sends an email with a capability which identifies both how to access the photos and provides the privileges necessary to do so. Then her sister Sarah runs an application (like WOM) which invokes MSI functions in order to download thumbnails of the given photos; Sarah adds these photos to her own WOM virtual file system. Naturally, exercising a capability can only be done by a program that invokes MSI functions.
So a Menagerie capability provides access to all objects rooted at some node (objectId). This is hard-coded into their system and seems not very general. Or at the least, this is a topic of discussion: are there common cases where a user might like to share access to an object without providing access to all of that object's children (and grandchildren and great-...).

What's a gadget?

2009-06-17T16:06:00.000-07:00

A gadget is an application that is specified via an XML file, which can contain HTML and hence JavaScript and even Flash code. There are two types of gadgets: those whose content is stored in the specification (HTML gadgets) and those whose content is stored at some network-reachable location specified by a URL (URL gadgets). We confine our discussion here to HTML gadgets (the default type). A gadget can store data persistently, run JavaScript code, and download content from remote websites.

Gadgets differ in terms of their configurability (i.e. the extent to which a user can specify his preferences for gadget operation and/or look-and-feel) as well as in whether the user inputs information into the gadget or instead merely receives info from the gadget (that is, whether communication flow is bi-directional in an ongoing basis). For example, many gadgets for news sources (e.g., the Wall Street Journal, Scientific American) at present provide few options for user configurability; also, the flow of information is mostly uni-directional — from the content provider to the user. By contrast, some applications have initial configuration opportunities but thereafter the flow of information is uni-directional (from the gadget to the user); e.g., a Clock application is initially configured with time zone but thereafter merely informs the user of the current time. There's no notion of the user interacting with that time in some way. Finally, some gadgets may or may not have user-configured settings but do have ongoing user input. For example, the to-do list gadgets involve the user providing information (i.e., items to be done); hence, the flow of information is bi-directional: from the gadget to the user ("this is your current list") and from the user to the gadget ("add item X" or "delete item Y").

Is there such a thing as a gadget mashup? I.e., a gadget which gathers information from more than one webpage? E.g., could I write a gadget which grabbed data from boxofficemojo (box office receipts), rotten tomatoes (movie reviews), and fandango (or some other movie-time site)? YES. I.e., a gadget is not restricted to only interacting with a single target location, right? YES. And a gadget can read/write from the network, correct? YES.

One future thought: Presently, we think of each OpenSocial app being deployed to a single container (i.e., social network such as Orkut or Linked-In); the next logical step is for an app to work across containers, i.e., to be a mashup between my Orkut profile/network AND my Linked-In profile/network. Not sure what the OpenSocial API offers w.r.t. this inter-social-network interaction.

Making Gadgets Accessible and Installing Them
Examples of gadgets can be found here. Enabling the public to use a gadget requires making the gadget world-reachable. There are a couple options for achieving this; a person can host a gadget as part of a website, the same way that he would host an HTML page. Also, Google provides two free alternatives for gadget hosting: (1) the Google Gadget Editor (GGE) and (2) Google Code Project Hosting. Once you've built a world-reachable gadget, you can list it in a directory (which people peruse to find worthwhile gadgets). Then an individual can add your gadget to some webpage (i.e., install the gadget) by inserting a "script" tag on the given webpage. The script's "src" attribute will point to the given gadget's XML file (i.e., contain that XML file's URL). That URL may also encode display (or other) preferences for the gadget.

Note that with this approach, one doesn't need to worry that his local version of a gadget (XML file) is out-of-sync with the remote version of the gadget because the user doesn't store the XML file locally but rather points to the remote version. (The user's browser may actually cache a local copy of the gadget when gadget execution is first triggered. But presumably this local copy would be refreshed the next time that the gadget's execution was triggered — which could be mere seconds later.)

Gadget components

A gadget specification (or <Module> element in XML) — such as this one — contains a few high-level elements: <ModulePrefs> (encodes gadget properties and preferences, such as the gadget's title, author, desired size), <UserPref>'s (used for persistently storing data and configuring or personalizing a gadget) and <Content> (controls the gadget's operation). <Content> contains HTML which itself can contain JavaScript etc.

<UserPref> can be used to store application state persistently; e.g., a user's grocery list or preferred background color. The values for user prefs are updated automatically when the user types into the "user preferences edit box" as below with the "Preferences for Rowan" box. When the user interacts with the gadget (entering a grocery item or a URL of a photo), the <UserPref> portion of the gadget's XML file is automatically updated accordingly. That's why, in a gadget's JavaScript code, we won't necessarily see any calls to set(...) preference values.

The code in the gadget's <Content> section can obtain and use the user's (stored) preferences via the gadgets.prefs API. This API can also be used to programmatically set user preference values (so long as setprefs is included). E.g., in the below, we see that three user preference variables (myname, myphoto, mychoice) are defined, each as a <UserPref>. The values for those variables are set when the user types into the user preferences edit box ("Preferences for Rowan" in this case) as discussed above. In the <Content> section below, we see JavaScript code that obtains the values of those variables using prefs.getString() and prefs.getBool(); this retrieves the stored value for the named variable. Note that the Prefs API also supports prefs.getInt().

User Prefs section contains:

<UserPref name="myname"

display_name="Name" default_value="Rowan"/>

<UserPref name="myphoto"

display_name="Photo"

default_value="http://blah.com/rowan-headshot.jpg"/>

<UserPref name="mychoice"

display_name="Show Photo?"

datatype="bool" default_value="true"/>

Content section contains:

var prefs = new gadgets.Prefs();

var html = "hello " + prefs.getString("myname") + "!";

if (prefs.getBool("mychoice") == true) {

html += '<img src="' + prefs.getString("myphoto") + '">';

}

For purely programmatic manipulation of user preferences, a gadget might define a method that is invoked onload; this method would obtain the values of various of the user's preferences and modify the gadget's display accordingly (e.g., setting the background color according to the user's preference). The gadget might also contain a method which stores user preference values; the trigger for this might be onmouseout (when the mouse moves outside the boundaries of the object) or onblur (when the object loses focus).

For XML or HTML code (i.e., within the <ModulePref> or <UserPref> sections OR within the HTML part of the <Content> section), you can obtain a particular user preference value via: __UP_userpref__, where userpref is the name of the desired user preference (e.g. myname, myphoto).

If you do not want to let a user modify some value and you want to store that value persistently (e.g., the user's high score for a game), you create a UserPref and give it type "hidden." You can modify the value of that variable programmatically, but the user will not be able to modify that variable's value.

Sharing a Gadget and Its Data

Moreover, one can also have shareable user preferences; e.g., a single list variable that is shared by multiple users (of the same gadget). Then, instead of just the one person modifying the shopping list, multiple users are modifying the same list. It appears that every gadget can by default be shared. That is, every gadget (that I've seen!) has the menu option "Share this gadget." Configuring a gadget to support shareable prefs entails adding <optional feature="shareable-prefs"/>. Depending upon a gadget's configurability and whether the user provides input to the gadget in an ongoing basis, sharing takes on different meanings.

For a gadget which has minimal configurability and no ongoing user input, there is little real sharing going on since users don't modify or interact with the gadget in any interesting way (there isn't any user-supplied data and hence nothing to share). E.g. the Wall Street Journal gadget as of time of writing.
For a gadget which has some configurability but minimal ongoing user interaction, there might be the option for a user to "share his configuration settings" with sharees. E.g. the Simple Clock gadget as of time of writing.
For a gadget which has ongoing user interaction, we can not only allow a sharee to "View my content," but we can also allow sharees to "View and edit my content." An example of this are the to-do list and post-it note gadgets where each sharer can contribute to a single (shared) list.

Gadget layout on iGoogle

One can add a gadget to his iGoogle page or to some other gadget container. On the iGoogle page, by default, the JS for all gadgets appears to run in the same execution context. Each gadget is defined as part of some section via the HTML <div> tag as illustrated below. A division can define its own function to be invoked on certain input events, including: onclick, ondblclick, onmousedown, onkeypress, and so on. Hence, if one clicks his mouse while that mouse is within the division for gadget X then gadget X's function for onclick will be invoked (if it exists). Presumably divisions are disjoint portions of the screen real estate, so there is no ambiguity about which division should receive an event. I have 20+ gadgets installed on my iGoogle page and the following HTML code exists for each:

<a id="lnat_110"

href="#"

title="To-Do List"

onclick="_IG_PushHistory('max110', 'selectMod','');return false;"

>To-Do List

</a>

</div>

The To-Do list gadget to which I'm referring lives here. Like all gadgets, it is defined by an XML file. Once the gadget has been installed onto a container page (e.g., iGoogle), it generates this JavaScript. Everywhere we see __MODULE_ID__ in the XML file is replaced with a "110" in the installed gadget's JS. This number identifies this particular gadget on my iGoogle page; the naming convention is presumably to ensure that there is no naming conflict with some other gadget's JavaScript function. This is necessary because all of the JavaScript for all of the gadgets on a person's iGoogle page appears to run in the same execution context.

Gadget execution

All of the JavaScript for all of the gadgets on a person's iGoogle page appears to run in the same execution context. That is, a gadget is not contained within its own iframe (for which a separate execution context would be created) but rather all of the JS of all of the gadgets is in the same HTML page. I actually have the ToDo list gadget installed three different times on my iGoogle page; each version contains a different type of list, one is for personal To-Dos, another for research-related To-Dos, and so on. So that means that my iGoogle page contains three copies of the same functions, each having slightly different names; e.g., loadToDos110, loadToDos117, and loadToDos109, corresponding to gadgets 110, 117, and 109, respectively.

As noted above, part of what spurred this investigation into gadgets was me wondering where a social networking app executes. And we know that the OpenSocial API (which is implemented by individual social networking sites or containers, such as Orkut) contains the gadgets API. So the following is, at present, my best conjecture as to the answer to that question. Let's imagine that Suzy is an Orkut user and installs an app on the Orkut site and gets her five closest Orkut friends to do likewise. When Suzy's browser visits the Orkut site and she logs into her account, presuably the application's JavaScript code is delivered to and executes in Suzy's browser (rather than executing on the Orkut web servers, for example, and just sending Suzy's browser some display code). That application may invoke an OpenSocial API function which itself calls out to the Orkut servers to obtain Suzy's friend list or similar. Think of this as the API function containing an RPC client stub which calls the RPC server, which runs at Orkut. This answers the question about how the OpenSocial API function interacts with the data stored on the Orkut site (i.e., the data that does not live in Suzy's browser).

Gadget security

Evidently there is a library that you can include as a "Required feature" of your gadget which"isolates" your gadget from other gadgets running on the same page: the locked-domain feature. No details on how this is implemented.

Note that if your gadget requires cookies, you may run into some trouble with IE and Safari. The reason is that a gadget is typically considered a "third party site" because the party hosting the site (e.g., iGoogle) is not the same as the party hosting the gadget (e.g., gadgetsrus.com). So in this scenario, the gadget code is running in an iframe on the iGoogle page. Evidently cookies can only be set for the containing webpage (e.g., iGoogle.com) and it would probably be a security violation to allow gadgetsrus.com to save a cookie for iGoogle.com (could open the door for spoofing attacks etc.). The best solution here would be to enable a cookie to be set for an iframe.

Remote Content

Can only download from a URL; can't use general sockets API to connect to arbitrary host/port. And even more specifically, the documentation suggests that only website URLs are legitimate (vs. FTP URLs).

Gadget privileges
Some basic issues in creating a social networking application include identifying the privileges needed by the application in order to serve its purpose and creating an installation and then execution environment (for the app) that confines the app to only access the data it needs and is authorized to access.

The Legacy Gadgets API

When I look at the code for several gadgets (NYT, Wikipedia, ToDo list, WSJ, ...), I see several calls to the legacy gadgets API, as identified by prefix _IG_ and as listed here. For example, there are calls to:

_IG_FetchContent, _IG_FetchXmlContent, _IG_FetchFeedAsJSON : Fetches content at the specified URL then invokes the given callback function upon completion of download.
_IG_GetCachedUrl : The passed URL identifies a file that is currently being cached; this function returns the proxy URL for that cached version of the file.
_IG_Analytics : Record a page view to a Google Analytics account. There are two arguments: the ID for the Analytics account and the path for the virtual page (whose hit counter should be incremented).
_IG_Prefs : For getting/setting UserPref's.
_IG_Tabs : Creates new instance of a Tabs object, which itself represents a set of tabs.
_IG_AdjustIFrameHeight : Requires the dynamic-height feature.
_IG_RegisterOnloadHandler : Pass a callback function that should be invoked when this page has completed loading.
_IG_RegisterMaximizeHandler : Presumably this enables setting a callback function for if the given window is maximized.
_IG_DD_Create :

So the above were the calls within the XML files that correspond to the different gadgets. When I look at my iGoogle page (the one which references each of these gadgets), I see other invocations of gadgets API functions, including:

_IG_AddEventHandler, _IG_AddModuleEventHandler, _IG_AddCustomEventHandler, _IG_AddDOMEventHandler
_IG_RemoveDOMEventHandler, _IG_RemoveModuleEventHandler
_IG_TriggerCustomEvent
_IG_callPostLoad
_IG_NavigateToGadget
_IG_LN_init
_IG_FormatLeftNavTitles
_IG_Callback
_IG_PushHistory
_IG_SetTitle
_IG_DD_init, _IG_DD_open, _IG_DD_create
Object types: _IG_inlinedTransitionStrategy, _IG_MiniMessage, _IG_iFrameTransitionStrategy, _IG_noMaxSupportTransitionStrategy
...

References

* Gadgets API overview

* Gadgets API reference

* Gadget FAQ