int d1, d2, d3, d4;
// runtime-assign those variables
int vla[d1][d2][d3][d3];

Now the question is, how to properly cast a pointer to this type when passing it to a function that accepts such a type in a way that makes the compiler happy? The syntax is somewhat unusual:

void func(int d1, int d2, int d3, int d4, int vla[d1][d2][d3][d4]){}
// ...
int * x = malloc();
func(d1, d2, d3, d4, (int (*)[(int)(d2)][(int)(d3)][(int)(d4)])x);

Tested with gcc 4.6.1 and the –std=c99 compiler option. I could not find information about this anywhere on the web so I hope this will help others who wonder how it should be done.

Up until now I hated to use Thunderbird because it did not allow me to send e-mails like that. Since gmail looks ugly as hell now I came back to Thunderbird and set out to solve this issue once and for all. I searched for “thunderbird plain text wrap” and whatnot. A smart fellow on a forum had the answer:

There are two hidden preference settings which one can access through Edit > Preference > Advanced > General > Config Editor button:

• mailnews.display.disable_format_flowed_support must be set to true to avoid rewarpping of messages received for which the sender allows flowed formatting
• mailnews.send_plaintext_flowed must be set to “false” to avoid sending flowed format e-mails

The problem seems to be something called “flowed format” but I’m too ignorant to go into the details of what it does, what it means and why I would want it. I’m just happy the e-mails my Thunderbird sends look the way I want them to look. Finally. Thank you rsx11m!

The information age spawned databases and networks capable of extracting and storing large amounts of private data. Those databases are often unknown to us and if we know of their existence we can not control them. They store personal information, communication and financial transactions. This gathering of private data happens against our will if we believe surveys that show that we actually do care about privacy. Skeptics and experts caution us but the majority of web users is forced to give in to the subtle but grave disintegration of privacy, pushed forward by industry and government. They are growing their databases steadily, expanding the records they keep on all of us.

Good question, image by Garrett Coakley

We can see the consequences of these uncontrollable, central databases today. In what is believed to be one of the largest data security breaches in history, attackers stole personally identifiable information of 77 million PlayStation Network users earlier this year.

Accidental exposure of personal data is another problem. It is very difficult to control who has access to which piece of information. People get fired for how they behave online because they confuse personal with public communication. The web does not forget. And ever since the uprisings in the Arab world it should be clear to everybody that what one posts online can have severe consequences, including imprisonment and torture.

There are a variety of interesting judicial and ethical approaches to cope with these issues. And there is cryptography – a technological means of preserving privacy. Cryptography enables anonymity, the concept of ‘publishing information while ones identity is publicly unknown’ as well as privacy, the ability to to ‘seclude oneself or information about oneself and reveal oneself selectively’.

But almost nobody uses cryptography. Asked if he encrypts his e-mail, Bruce Schneier, cryptographer and highly regarded computer security specialist answers “I do not, except for special circumstances”. He further argues that for more people to encrypt their communication, services like Gmail would have to do it by default. This will of course never happen, since those services draw their revenue from reading our messages.

It has to work out of the box

But the more important point Schneier makes is this: what has to happen to spread the use of cryptology? It has to work out of the box. No additional application should be required, no plug-in, no add-on and certainly no driver installation. There exists a concept that could potentially offer a transparent solution for everyone: browser based cryptography.

Browsers have evolved from being a mere presentation and navigation tool for the world wide web to a platform for collaboration and information sharing web applications. If browsers were able to do cryptography, every web users could potentially benefit from it. JavaScript engines have evolved rapidly to a point where they are efficient enough to handle the complex algorithms that cryptography entails. It is ironic that the same web applications that threaten our privacy are the main reason such powerful engines were developed in the first place.

The idea of browser based cryptography is simple: before users upload their personal data to application hosts they encrypt the data in the browser. The host only receives encrypted blobs of data and since users don’t share their key with the host the data is secure. If they decide to share their data with someone else they can provide them with means of decrypting the blobs. Users are in control at all times.

But JavaScript cryptography has many critics and there has been some discussion whether or not it is a viable solution. But the potential is vast and the issue of retaking privacy is too important to dismiss the technology right away. The discussion should not stop here. Solutions can be found to the given objections.

JavaScript Cryptography Criticism

The trust model certainly is a problem and seems inconsistent. The general assumption is that users don’t trust application providers with their data, thus the need for encryption. Modern web applications however download their code from the very same provider and consequently also download the code required for decryption and encryption. A contradiction: users don’t entrust providers with their data but they trust the provider to deliver the application and most importantly the correct cryptography code. Critics argue that users can decide to either trust or not trust whoever hosts an application. If they trust the host there is no need for encryption. If they don’t trust the host they should not share their data in the first place. If someone suspects a host has malicious intentions JavaScript cryptology is worthless.

The situation changes if the ‘honest-but-curious’ adversary model is taken as a basis. It assumes that the company providing a web services carries out the stated instructions and is not lying to the user. It is further assumed that it might do more than it promised such as storing the data for an unreasonably long time or even sharing data with third parties. In such a case JavaScript cryptography might be viable. Furthermore if the company is attacked and database dumps are stolen, the data is worthless. For an attacker to gain access to the data the web application source code has to be modified and users have to use the malicious application. To defeat such attacks, browsers would have to be able to validate web applications to make sure that they were not modified.

Dystopia 2, image by Hervé Girod

Today the browser is an environment not very well suited for cryptography. There is the threat of cross-site attacks: a web page loads content from many sources and all of those can potentially modify the cryptography code. Due to the dynamic nature of the JavaScript language, an attacker can replace correct code with a malicious version. Such an attack can only be discovered through tedious code analysis of all sources.

Critics also argue that browsers lack some crucial primitives important for cryptography such as a random number generation. Fortunately browser vendors see the need for such functions and are moving toward implementing them. Browsers further lack a secure key store, a crucial component in every crypto-system. It is also important to have the ability to securely erase secrets from memory once they are no longer needed. Since JavaScript engines employ garbage collection there is no way to control when objects are deleted and secrets are forgotten.

These issues are significant but they can all be addressed with care. Web developers, cryptographers and browser vendors will have to work together to find solutions to these shortcomings. Once this happens I expect to see secure JavaScript cryptography applications that satisfy even the skeptics. There is still a lot to be done but the potential is tremendous. So we should get to work.

Existing Implementations

Despite the criticism there are already some implementations that utilize JavaScript cryptography out there. They try to make do with the current state of browser support for de- and encryption. Some supplement browser capabilities with custom add-ons. This of course defeats the purpose of JavaScript cryptography but is a necessity at the moment since browser support is still in its infancy.

Aldo Cortesi’s cryp.sr is the most interesting project. It is an online list-manager that encrypts user data in the browser and sends only an encrypted blob to the host. It inspired people to think about what JavaScript cryptology can do and what it means to encrypt data in the browser. Encrypted user data is completely exposed as the application intentionally lacks authentication (except to prevent overwrites). Only the passphrase protects the data. Cortesi uses the expression ‘host-proof’ in his documentation of the project which is a dubious term, especially in the context of JavaScript cryptography. It was coined by Richard Schwartz and emphasizes that the host does not have to be trusted. This is a difficult claim. Users still have to trust the host because it grants the privilege of encrypting the data and it can revoke that privilege at any time. The ‘honest-but-curious’ adversary model again makes more sense.

More questionable applications are Lockify, Zero-Knowledge Box and Clipperz because they explicitly advertise the security of their products that is solely based on JavaScript cryptography. Critics argue that appearance of security is worse than no security at all. Their claim of increased security should indeed be taken with a grain of salt: data that otherwise would not be uploaded due to security considerations should not be stored with those hosts. Even so, the cryptology is a welcome addition to the security measures these companies take. It accomplishes the goal of preserving privacy.

Vintage Mail Boxes, image by Nathen Jantzen, all rights reserved

When asked to choose between a regular web application and a JavaScript cryptography enabled one, the latter is preferable due to privacy considerations since the user does not lose control of his data. Claiming that JavaScript cryptography enabled applications, with the current state of research and browser support, are more secure than others is debatable. We are not there yet.

Alternatives

There are a number of alternatives and especially the concept of storing encrypted data with a curious or even untrusted host is not new. Traditionally, host applications have been used to handle cryptographic operations. These tools must be installed and have to be properly set up by the user. Mobile platforms might be an ideal environment for these alternatives. Installing applications is hassle-free and very common on mobile devices. Due to the well defined platform, developers can keep user effort to configure these applications to a minimum.

Another promising alternative is a browser add-on called Cipherbox. Its developers recognize that it is unlikely that application providers will enable cryptography as well as the shortcomings of current JavaScript solutions. In their architecture they make a point of separating the interaction with the web content from the cryptology functionality. This could proof to be a significant security advantage over other solutions that give web content access to cryptology functions.

A colleague and I devised the idea of a cryptography enabled http proxy that is similar to the Cipherbox. The proxy is a trusted instance possibly hosted locally or connected via a VPN. All http traffic is sent through the proxy. It analyzes the traffic and encrypts and encrypts relevant parts like messages or images depending on its configuration. We implemented a prototype that is capable of transparently encrypting and decrypting Facebook messages using gpg. A proxy like this could run on a user’s FreedomBox and can in theory be extended to provide crypto-functionality for various platforms including for example Gmail.

A special form of cryptography called homomorphic encryption could enable users to take advantage of both cryptography and computing as a service at the same time. If encrypted data is sent to hosts, they usually can not process the data. If instead a homomorphic encryption scheme is in place, for certain algebraic functions on the plaintext, equivalent functions exist that can be applied to the ciphertext. Proponents of this technology argue that it could enable widespread use of cloud computing by ensuring the confidentiality of private data.

Conclusion

Controlling ones personal data is more difficult with every new database and network based innovation. At the same time privacy is more important than ever in a world that prepares to conglomerate health records, gathers and centralizes consumer behavior data and merges individual financial records into powerful profiles. Cryptography is an effective safeguard we must implement to prevent exploitation and discrimination based on our personal information. Every user must be enabled to use cryptology to control the data he wishes to share.

Browsers vendors must implement the building blocks required for cryptography including a secure key store that can be managed by the user. They should also include means of validating a running application against a checksum. Cryptographers and web developers must work together to implement correct and easy to use de- and encryption functionality for browser based applications.

More people must start thinking about this problem, more ideas are needed and should be carefully vetted by cryptographers and security experts. User interface specialists should work on making cryptography a transparent process. We need to get everyone involved and try to revert the damage that has already been done.

This is not an introduction to an dystopian fantasy world but an excerpt from a promotion video for Opera Unite, a framework that allows users to host information from their home computer. It was a bold attempt to change the centralized architecture of the Internet. A number of smart people have been pondering this idea even before Opera’s experiment failed miserably.

Communication breakdown, image by miuenski

And the concept of a decentralized web is gaining traction: more and more people realize something has to change. The cause for this trend is obvious: the number of data security and privacy disasters that were made public has spiked in recent times . In April ’11 for example an update to the security terms of service of the widely used Dropbox tool revealed that contrary to previous claims, Dropbox Inc. has full access to user data.

An analysis of the changes to the Facebook privacy policy over time paints a gloomy picture of how the world’s largest social network changed “from a private communication space to a platform that shares user information with advertising and business partners while limiting the users’ options to control their own information”.

With more and more of our personal data moving to centralized servers or “cloud services” – a term that should be used as an euphemism – we’re no longer in control. But there is hope in sight: there are dozens of projects out there that try to stop the trend of centralization and data consolidation.

Decentralized Applications

The most popular of the lot is probably Diaspora. The project got a lot of attention in April 2010 when they managed to raise about $200.000 from almost 6500 supporters. The software looks and feels very much like Facebook or Google+. The innovation is that users are allowed and even encouraged to set up their own Diaspora node. This essentially means allowing users to set up their own Facebook server at home (or wherever they want). The Diaspora nodes are able to interact with each other to form one distributed social network. Furthermore, instead of users having to log in to one central server, they may choose one of many servers administered by different entities. In the end they can decide whom to trust with their data and there is no one entity that has access to all the data.

A social network project that is also worth mentioning follows the same principle. Its name is Buddycloud. The main difference between Buddycloud and Diaspora can be found in their implementation details: Buddycloud builds upon XMPP (Extensible Messaging and Presence Protocol), a more than 10 year old and often implemented specification for “near-real-time, extensible instant messaging, presence information, and contact list maintenance”. There are many unknowns in this area so building on such proven protocols instead of defining new standards might proof to be an advantage. But there are many more social networking projects out there. Wikipedia has a nice list.

The Unhosted project implements another concept. Instead of providing a specific decentralized service it aims to be a meta-service. And after talking to Michiel de Jong I have the impression his plan is even more crucial. He aims to create something fundamental, a protocol, an architecture, a new way of writing web applications. The idea is the following: the traditional architecture of a hosted website provides both processing and storage. An unhosted website only hosts the application, not the data. Unhosted wants to separate the application from the data. By storing the data in another location and combining both application and data only in the browser, the application provider can never access the data. An ingenious and very ambitious idea. I hope they succeed!

Decentralized Storage

A project that aims to replace Dropbox is ownCloud, an open personal cloud which runs on your personal server. It enables accessing your data from all of your devices. Sharing with other people is also possible. It supports automatic backups, versioning and encryption.

The Locker Project has similar goals. They allow self-hosting (installing their software on your own server) and offer a hosted service similar to what Dropbox provides. The service pulls in and archives all kinds of data that the user has permission to access and stores this data into the user’s personal Locker: Tweets, photos, videos, click-streams, check-ins, data from real-world sensors like heart monitors, health records and financial records like transaction histories (source).

Shimmering, image by Jason A. Samfield

A third project worth mentioning is sparkleshare. It is similar to the other projects in this category but allows pluggable backends. That means you can choose to use for example Github as backend for your data or of course your personal server. Awesome!

Freedom to the Networks

Projects such as netless carry the idea even further because after the data is liberated, the connection itself is a soft spot. Network connections should be liberated from corporate and government control by circumventing the big centralized data hubs and instead installing a decentralized wireless mesh network where everyone can participate and communicate.

The adventurous netless project plans to use the city transportation grid as its data backbone. Nodes of the network are attached to city vehicles – trams, buses, taxis and possibly – pedestrians. Information exchange between the nodes happens only when the carriers pass by each other in the city traffic. Digital data switches its routes just the same way you’d switch from tram number 2 to bus number 5. Very inspiring.

Another idea is to utilize networks of mobile phones to create a mesh network. The serval project is working on this. And they have a prototype for the Android platform ready.

The German Freifunk community pursues a similar goal. It is a non-commercial open initiative to support free radio networks in the German region. It is part of the international movement for free and wireless radio networks (source).

A purely software based project is Tor. It is free software and an open network that helps its users to defend against a form of network surveillance that threatens personal freedom and privacy as well as confidential business activities and relationships.

Peer to Peer Currency

One integral thing this article did not talk about yet is money. Bitcoin, a peer to peer currency, might be the missing puzzle piece. The Bitcoin system has no central authority that issues new money or tracks transactions – it is managed collectively by the network.

A major problem of digital currency has been preventing double-spending. Digital money can be copied multiple times so a mechanism is necessary to forbid spending money twice. Bitcoin refrains from having actual digital coins. The system is merely one large transaction log that tracks what money was transferred where.

Each participant has a pair of public and private keys to sign transactions and to allow others to verify transactions. The transactions are entered into a global ever running log that is signed in regular intervals. The signing of the log is designed to require extensive computation time. The entire network of participating users is required to sign the log.

This protects the entire system from false signatures and from anyone tempering with the log and modifying past transactions. An attacker would have to have more computational power at his disposal than the entire Bitcoin network to forge transactions.

Users that give their computing time to the network are rewarded with Bitcoins for their troubles. This is also how the money is generated in the first place. In addition, participants that transfer money are free to include a transaction fee in their order. This extra money is given to the particular user signing the transaction.

A considerable number of sites have emerged that accept Bitcoins in exchange for services or goods. You can buy for example socks online or even pay for your lunch at a burger restaurant in Berlin.

Conclusion

In closing, I find it encouraging, that so many people feel that things have to change and are developing ideas and projects to make it happen. We will see many exciting things in the future and despite the overwhelming might of well-established products, I am hopeful.

Launch of Shuttle Discovery, image by Scott Kublin

The repository can be found here: https://github.com/sschaetz/mgpu and documentation is available from here: http://sschaetz.github.com/mgpu/

There are also archives available: mgpu_0_1.tar.bz2 and mgpu_0_1.zip.

cat /proc/driver/nvidia/version

On my system this returns:

NVRM version: NVIDIA UNIX x86_64 Kernel Module  270.41.19
GCC version:  gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)

GNUtralized, image by Bruce Bottomley

He speaks about privacy: the freedom to choose who to share what information with. He speaks about freedom of speech: the liberty to speak and communicate freely without censorship. He speaks about how we give up those freedoms, often without noticing or worse, without caring. By signing up to facebook. By using Gmail.

It’s insane if you think about it. We communicate with our friends, send them private messages, through a database in Northern California. We store our personal information including images in that same database. And while we put more and more information, more and more of our lives in that database, its owners are busy thinking about ways of how to make money with our data. By now I’m sure they have come to at least one conclusion: to make money they will share our data with entities we never intended to share it with.

It gets worse, and Eben Moglen elaborates this very well in his keynote speech at Fosdem 2011. All our data is routed through and even stored in a company’s database. We can choose to trust that particular company to not do anything bad with our data. But we can never be sure that the company will act in our interest if it is offered incentives (“We won’t shut you down if you comply”) to share certain data with certain government agencies. Since our data is stored in a centralized manner, agencies the likes of NSA would have a field day.

So indeed, Stallman was right when he called cloud computing a trap. He advises us “to stay local and stick with our own computers”.

free, image by Gisela Giardino

Eben Moglen sees the urgency of these problems so he is taking action: informing the public with persuasive talks and gathering smart, ingenious and influential people to find solutions under the newly founded FreedomBox Foundation. The ingredients to the solution, in accordance to Stallman’s suggestion have been identified:

1. cheap, compact plug computers that can be used as personal servers
2. a free (as in freedom) software stack that is designed to respect and preserve the user’s privacy
3. mesh networking technology to keep the network alive if centralized internet connections fail

The idea is simple: replace all cloud services (social network, e-mail, photo storage etc) with distributed systems. The FreedomBox project aims to provide everyone with a compact, cheap, low-power plug computer. Such a device will come preloaded with free software that “knows how to securely contact your friends and business associates, is able to store and securely backup your personal data and generally maintaining your presence in the various networks you have come to rely on for communicating with people in society”.

There are a number of other projects out there that are worth mentioning here. A very interesting assessment of the current state of the social web and existing standards is the W3C Social Web Incubator Group report. The Diaspora project develops distributed and secure social networking software and is worth keeping an eye on. The FreedomBox community maintains a comprehensive list of existing projects.

I expect the foundation to develop a software package that provides stable system software and that encompasses a number of existing projects that allow users to move from cloud services to their own server. I will help were I can and so should you!

Bethany Cooper, image by Joe Futrelle

The basic idea is to have a generic class that either creates the required objects for you or returns an existing object, if you already created it. Inefficient code like this:

algorithm(T1 input, T2 output, parameters)
{
  buffer mybuff(buffer_parameters);
  calculation(input, output, buffer);
  // buffer goes out of scope and is destroyed
}

for() algorithm(input, output, parameters);

creates a buffer object in each invocation of algorithm. This is oftentimes unnecessary. The buffer object could be reused in subsequent calls which would result in code such as:

algorithm(T1 input, T2 output, parameters, workspace & ws)
{
  buffer & mybuff = ws.get<buffer>(name, buffer_parameters);
  calculation(input, output, buffer);
  // buffer is not destroyed
}

{
  workspace ws;
  for() algorithm(input, output, parameters, ws);
} // workspace goes out of scope; all objects are destoryed

The name in the above code acts as an object identifier. If the same object should be used a second time it has to be accessed using the same object identifier.

More Features

Specifying only the name of an object as its identifier might not be enough. The following code could be ambiguous and might result in an error:

buffer & mybuff = ws.get<buffer>("mybuff", size(1024));
// later call
buffer & mybuff = ws.get<buffer>("mybuff", size(2048));

When using mybuff after the later call, it might be expected that the buffer has a size of 2048. But the workspace object returns the object created before which is smaller. A simple workaround would be to always resize the buffer before it is used.

As an alternative the workspace object can be instructed to not only consider the name as an object identifier when checking if the object already exists but also some parameter of the constructor. Wrapping parameters with the “arg” method marks them to be considered when resolving existing objects.

buffer & mybuff = ws.get<buffer>("mybuff", workspace::arg(size(1024)));
// later call
buffer & mybuff = ws.get<buffer>("mybuff", workspace::arg(size(2048)));

This code will create two different buffer objects.

Implementation

The workspace holds a map of pointers to the objects it manages. The map’s key is an arbitrary name for the resource. In addition to the pointer, a functor that properly destroys the object referenced by the pointer is stored.

A call to get<T> will check if an object with ‘name’ already exists. If not, an object is created using the parameters (as well as a function object to destroy it). If it exists a reference is returned.

If the workspace goes out of scope, the destructor iterates over all object the workspace holds and destroys them in reverse order of creation.

I used some of the nifty new C++0x language features (variadic templates, std::function and the lambda functions) to implement the workspace class. I thought this was a good idea since the Final Draft International Standard was approved recently. I found the features to be very powerful as well as incredibly useful.

The argument dependent lookup is implemented in a crude way: the arguments are appended to the name string as characters. This should be sufficient for basic use and works well for all built-in types.

Arguments that are supposed to be added to the object identifier are detected as they are wrapped in a special_arg class. Partial template specialization and the unpack operator help appending the arguments to the object identifier.

The current implementation can be found in my workspace git repository.

Discussion

It might very well be that something like this already exists in a more elegant form and that I have simply not been aware of or failed to identify as a solution to this problem. I’m for example aware of pool allocators but I believe that I propose a more abstract concept. Buffers that are part of a workspace still could use a pool allocator.

Also the code in the repository is written rather quickly and I’m quite sure there are some rough edges where the code is not as efficient as it could be (passing by reference comes to mind).

You can never go too far, image by Baboon

Unfortunately (or fortunately) I could not find a production ready lock-free queue in C++ for my platform (Linux). There is the concurrent_queue class available in Visual Studio 2010, there’s the unofficial Boost.Lockfree library (code is here) but those options were not satisfying. I needed something simple that worked right out of the box.

Herb Sutter wrote an excellent article a while back about designing a lock-free queue. It’s straight forward and functional. The only flaw it has from my point of view is the use of C++0x atomics which I wanted to refrain from including in my project for now. There’s of course the unofficial Boost.Atomic library, but again I wanted something simple.

So I reread Herbs article, especially the passage that explains the requirements for a “lock-free-safe” variable: atomicity, order and the availability of a compare and swap operation. Digging into the Boost.Atomic code I found that it is rather simple to get atomicity and order when using gcc. I’m using version 4.4.3.

Atomicity can be ensured using the atomic functions of gcc and order can be ensured using a memory barrier trick:

type __sync_lock_test_and_set (type *ptr, type value)
asm volatile("" ::: "memory")

The first line is an atomic test and set operation and the second line is equivalent to a full memory barrier. With this functionality I was all set. Here is the modified code for the method the producer calls:

void produce(const T& t)
{
  last->next = new Node(t);                      // add the new item
  asm volatile("" ::: "memory");                   // memory barrier
  (void)__sync_lock_test_and_set(&last, last->next);
  while(first != divider)                       // trim unused nodes
  {
    Node* tmp = first;
    first = first->next;
    delete tmp;
  }
}

I replaced the atomic members with regular pointers but exchanged the assignments that require atomicity and order by calls to __sync_lock_test_and_set preceded by the full memory barrier trick. The function call must be cast to void to prevent a “value computed is not used” warning.

bool consume(T& result)
{
  if(divider != last)                        // if queue is nonempty
  {
    result = divider->next->value; 	              // C: copy it back
    asm volatile("" ::: "memory");                 // memory barrier
    (void)__sync_lock_test_and_set(&divider, divider->next);
    return true;          	                       // report success
  }
  return false;           	                    // else report empty
}

The full class can be downloaded here. Please note that the class lacks some important functionality if your use-case is non-trivial (a proper copy-constructor comes to mind).

What it is all about

There is a collection of definitions of lock-free and non-blocking and wait-free algorithms and data structures that can be interesting to read.

The idea of lock-free data structures is that they are written in a way that allows simultaneous access by multiple threads without the use of critical sections. They must be defined in a way that at any point in time any thread that has access to the data sees a well defined state of the structure. A modification was either done or not done, there is no in-between, no inconsistency.

This is why all critical modifications to the structure are implemented as atomic operations. It also explains why order is important: before critical stuff can be done, preparations have to be finished and must be seen by all participants.