Distributed LEGO image archive/gallery

P2P protocol

• The protocol is used to exchange notifications of changes to the distributed archive.
• Each node sends notifications to a semi-permanent subset of its peers.
• The notifications are sent as un-normalised Unicode UTF-8 encoded text messages. (UCS-4 appears to takes more space, even when gzipped. TCP has more overhead than UDP and requires a more complicated server implementation.)
• Notifications of changes the node didn't know of are retransmitted to all other connected nodes than the one the notification originated from.
• Notifications are terminated by the LF character.
• Fields in a notification are delimited by the HT character.
• Notification types:

  Has-copy

  image-key, host, publisher, tag-list

  No-copy

  image-key, host

  Off-line

  host

  Sending-to

  from-host, to-host-list

  Policy-violation

  image-key, reviewer, policy-id, text

  Current-load

  host, load-fraction

  Please-copy

  image-key

  New-host

  (no fields)

• Images are indexed with an image-key, which is a cryptographic hash of the image content. (The command anytopnm | pnmnoraw | sha1sum has the required properties.)
• Clear text fields (i.e. tag and text) are composed of an ISO language code field and the actual text field.
• The load-fraction is a decimal number describing how large a fraction of the load limit, the node is running at at the moment. If the node has several imposed non-storage limits, it should send the largest load-fraction.

Node database

Tables:

Published by:

image-key, publisher

Mirrors:

image-key, host

Tags:

image-key, tag-language, tag-text

Policy violations:

image-key, reviewer, policy-id, language, text

Blocked images:

image-key

Load:

host, load-fraction

Connections:

from-host, to-host

Storage:

image-key, file-name

Notification actions

On has-copy:

• If image-key is in Blocked images then stop here.
• If image-key isn't in Published by, then add (image-key, publisher) to the table, and mark the notification for retransmission.
• If (image-key, host) isn't in Mirrors, then add (image-key, host) to the table, and mark the notification for retransmission.
• For each tag; if (image-key, tag-language, tag-text) isn't in Tags, then add (image-key, tag-language, tag-text) to the table, and mark the notification for retransmission.
• Remove entries for image-key in Tags which aren't in the tag-list. If any entries are removed, then mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.

On no-copy:

• If image-key is in Blocked images then stop here.
• If (image-key, host) is in Mirrors, then remove (image-key, host) from the table, and mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.
• If image-key isn't in Mirrors, then remove all image-key entries from Published by, Tags and Policy violations.

On off-line:

• If host is in Mirrors, then remove all host-entries from the table, mark the removed image-keys for clean-up, and mark the notification for retransmission.
• If host is in Load, then remove all host-entries from the table, and mark the notification for retransmission.
• If host is in Connections, then remove all host-entries from the table, and mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.
• For each image-key marked for clean-up; if it isn't in Mirrors, then remove all its entries from Published by, Tags and Policy violations.

On sending-to:

• For each to-host; if (from-host, to-host) isn't in Connections, then add (from-host, to-host) to the table, and mark the notification for retransmission.
• Remove entries for (from-host, to-host) in Connections which aren't in the to-host-list. If any entries are removed, then mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.

On policy-violation:

• If image-key is in Blocked images then stop here.
• If (image-key, reviewer, policy-id, language, text) isn't in Policy violations, then insert/replace the entry in the table, and mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.

On current-load:

• If (host, load-fraction) isn't in Load, then insert/replace the entry in the table, and mark the notification for retransmission.
• Retransmit the notification, if it is marked for retransmission.

On please-copy:

• If image-key is in Blocked images then stop here.
• If there already is a copy of the image image-key in local storage, then stop here.
• Chose one of these three options:
  • Do nothing.
  • Retransmit the notification.
  • Download a copy of the image image-key to local storage, check that the image content corresponds to the key.
  (It is up to the individual node to decide how to chose among the options.)

On new-host:

• Send the whole node database to the originating host as has-copy, sending-to, policy-violation and currenct-load notifications.
• Do not retransmit the notification.

Notification use

Has-copy:

• Is used by a host joining the network, to announce which images it has in local storage.
• Is used when a host has mirrored an image.
• Is used when a new image has been published on a host.

No-copy:

• Is used when a node decides to remove an image from its cache.
• Is used when a node no longer can find an image file, where it should be on disk.

Off-line:

• Is sent if a host doesn't respond to HTTP requests as announced.
• Is sent by a host as a part of a planned shut-down.

Sending-to:

• Is used when a node changes the set of peers it sends notifications to.
• Is intended to be used to plan reorganisation of the P2P network, if it lacks cohesion.

Policy-violation:

• Is intended to be used, when a reader considers an image to be in breach of one of the network policies.

Current-load:

• Is used to inform the system of how close a node is to its load limits.
• This notification should not be sent at fixed periods.

Please-copy:

• Is intended to be used to request other nodes to mirror un-mirrored images.
• Is used to increase the number of copies of much requested images.

On new-host:

• Is intended to be the first notification a node sends, when it joins a network.

Known limitations

• Since we store a complete copy of the index on all the nodes, quite a bit of disk space will be used just for the index. But 1 Gb disk should still be enough to store an index for 2-3 million images (assuming 3 copies/image and 5 tags/image).

Missing parts

We still need:

• A protocol for uploading new images to the system.
• A notification for reporting mismatches between image content and image keys.