InSTEDD Technology Blogs

1st PHI Workshop at Mahidol: Collaborations in Public Health Informatics among MBDS and SEAMEO Countries

2008-11-06T12:11:12-06:00

Nico and I were invited to present at the 1st Workshop on Public Health Informatics organized by Center for Excellence for Biomedical and Public Health Informatics (BIOPHICS), Faculty of Tropical Medicine at Mahidol University in Bangkok, Thailand (Sep 29-Oct 10, 2008). The workshop was co-organized by University of Washington and the Rockefeller Foundation. This two-week workshop introduced the discipline of public health informatics to an elite group of public health and information technology experts from the MBDS and SEAMEO countries (Vietnam, Yunnan, Cambodia, Lao PDR, Myanmar, Indonesia, Philippines, Thailand, etc.) The course included topics on the application of latest advancements in information science and technology in supporting public health practice, education and research.

On the morning of Thursday October 9, 2008, Nico and I gave 3 lectures on the application of machine learning for early detection and response, Open Source movement and licensing, and discussed various Open Source applications for public health. We introduced biosurveillance, its current challenges and limitations, and proposed a practical “collaborative approach” for effective early disease detection and response from local, national, and global perspectives.

We then had a lunch break with Assoc. Prof. Pratap Singhasivanon, Dean of Faculty of Tropical Medicine who also supervises BIOPHICS. We discussed with Dr. Pratap InSTEDD’s role in the region and the MBDS, the various projects and technology tools currently underway, the innovation lab model, and our gratitude for extending his invitation to us to present at the 1st PHI course.

We spent the majority of the afternoon in the computer lab where we offered hands-on training on various Open Source tools and projects, their implication to public health, and their role in the MBDS region. We also used this opportunity to introduce InSTEDD’s tools including: Mesh4X, Geochat (Overview, Details and source), Riff and RNA.

At the end of day, we had a quick tour of BIOPHICS and its various state-of-the art projects. We were briefed of BIOPHICS multi-clinical trials center (several million patients), observational studies and disease registry, and had a tour of the BIOPHICS data center. The clinical trials ranged from PK study, Phase I to Phase III studies, and Phase IV monitoring adverse drug reaction (ADR). We were also offered a demonstration of a versatile and mobile patient tracking system. Other services include computerized system development for disease surveillance and medical/laboratory data. BIOPHICS has experts in basic science, health science, public health, medical and information sciences and they offer various consulting services related to applications of genetics statistics, bioinformatics and clinical informatics.

On Friday October 10, 2008, each country had a representative speaker from their group who presented on their country's current challenges in disease surveillance and response, what they learned from the workshop, and how best to apply that to address the current system's limitations and challenges.

We were pleased to see how InSTEDD's tools and “collaborative” approach was mentioned in various discussions and considered to be part of the solution (Picture: The Cambodian team presented on their future solution and suggested using various InSTEDD tools and technologies as part of this solution.)

Afterwards, Nico and I sat on a final panel discussion that addressed participants’ questions with regards to the material presented during the workshop. We received various questions on the role of Geochat and SMS in enhancing and augmenting current surveillance and response activities in the MBDS and SEAMEO countries. Given the fact that text-messaging solutions can be cost-prohibitive for many localities, we discussed how Geochat’s “Gateway” offers an affordable alternative—requiring coordination between mobile providers and government agencies—and more features than regular text-messaging provided by local mobile companies.

We then concluded the session with graduation ceremony where participants received their certificate in PHI, the first for the region! Congrats!

After graduation, Dr. Moe Ko Oo, (Regional Coordinator for the MBDS project), Dr. Yin Myo Aye (Data Analyst & Manager for the MBDS project) and I had a meeting to further discuss the project and InSTEDD's contribution to the region. During the workshop we introduced various open source efforts projects; in particular we discussed two efforts which aggregate news media and ProMed reports, HealthMap and BioCaster. Dr. Moe and I discussed the importance of providing situation awareness capacity for the MBDS region by extending these efforts from detection into response. We discussed in details; as I mentioned in a previous blog, how the majority of current systems have been geared towards specific data sources and detection algorithms but much less effort has been focused on how these systems will "interact" with humans. Dr. Moe was very keen towards our “collaborative” approach, and I quote: “HealthMap, the data they are presenting in their web is good for us to know, but we would like to see more on it” and that efforts as such should offer a “…better analytical way for future planning and advocacy purpose.” We also discussed how our approach could further refine and enhance classification of health events especially at the earliest stages of an infectious disease outbreak. I had a follow-up discussion with Dr. John Browntein and his team is working with ProMed (recent grant from Google (healthMap/ProMed) through the Google Predict and Prevent initiative) on extending the current HealthMap platform to incorporate collaborative tools.

Mesh4x goes mobile with JavaROSA, allows you to sync data on your handset with no Internet

2008-10-29T12:11:22-05:00

The latest batch of advances in mesh4x and JavaROSA allows you to do forms-based data input and editing on any java-enabled phone and synchronize with other phones or a server.

You can define a generic form, load the form definition to any java-enabled cell phone loaded with the JavaROSA forms clients extended with a mesh4x transport component, do data entry in your phone and synchronize the data 2-way with a server or directly peer-to-peer with another phone handset.

The form definitions are saved and exchanged as XForms, and the data as XForm models. The data can be exchanged over http (if the phone users can afford GPRS and have a data connection) or over compressed SMS messages. This can even happen between phones directly - you enter the phone number of another handset running the app and press "sync". Tondat describes this in detail in his latest blog post. The clients depicted here look awful on the emulators as they use J2ME Polish ( http://www.j2mepolish.org), which then makes the app look great on specific handset models and adapts the UI to the capabilities of each phone.

This extends the scenarios of JavaROSA- from data-entry bringing it closer to a collaboration tool, where the information being entered can be edited by multiple users and shared from a central database back to the phone in the field.

This contrasts with "data collection" pattern of data entry solutions...if you believe information is power, data collection creates a vast vacuum cleaner shifting the balance of power: away from those in the field who understand the data the best and can act on it the soonest, towards the center. But does it need to be this way?

At InSTEDD we look at information flows such as those and ask ourselves: What information should be flowing back to the field? How can the person at A work better with the person at B beyond just sending data? How do we shift from 'sending' data to sharing realtime and enriched information two ways? The mesh4x + JavaROSA effort is addresses some of these questions.

This was made possible through a collaboration and set of code contributions we had with the JavaROSA team. JavaROSA is an implementation of OpenROSA which could become a strong player in the mobile data gathering and sharing space in the near future. Kudos to Clayton Sims, Jonathan Jackson, Andreas Kollegger, and everyone else from the javaROSA team for your work and friendly attitude!

The XForm definitions are stored in an http service behind a REST API (http://sync.instedd.org/ which is a strawman of a mesh4x cloud-based service. If you played with our map-sync technology you have already used this service).

Next Steps

Our strategy with mesh4x is to contribute code to existing projects being deployed in the field that need 2-way synchronization, data exchange over SMS, or multi-master storage based on standards. Episurveyor, Gather, Pendragon etc come to mind.

Within this directive, our roadmap on mesh4x will involve effort in four areas:

Cloud Services
Data Standards
Client Applications
Transformers and Adapters

1. Cloud Services: A scalable server implementation supporting security standards . We have a skeletal solution built in C# that we grew from the 'sse' open source project in Codeplex, (which has moved to http://mesh4x.org as well). We host an instance at http://sync.instedd.org/. But it uses a relational database, so we would have to change the storage layer if we wanted to grow it for real. So what are our options? Java on EC2/S3 seems to be the shortest path given the code we already have in the project, but Python on Google App Engine sounds enticingly simple to maintain and scale, at the expense of initial effort to port the sync libraries to Python. Which seems a unnecessary until you consider that Inveneo, the African Access Point and other platforms prefer Python or Ruby for a bunch of good reasons. We'd like your input - .NET+MySQL, Java, or Python + GAE?

2. Default Data Exchange Standards: Using XForms is simple and works for easy scenarios. We'll advocate use of RDF when XForms will fall short, but by all means we wanted to avoid a custom/ad hoc way of defining a typed dictionary 'schema', versioning of that schema, and of encoding entities following the schema.

Following a clear set of standards for data formats will allow easier mapping of information from one system to another, and the creation of tools that allow end-users to define how their systems integrate.

Isn't this obvious? Defining which standards to support early in the process is critical because it is easy to reinvent the wheel in this space. Even accidentally. Anyone who can code their way out of a paper bag can define a custom way of serializing dictionaries (a collection of names and values such as name:Ed, country:Cambodia) and define a schema model for it in an hour or so. But inventing one just tends to lead to incompatibilities in the long run, and lack of interoperability in humanitarian systems is an obstacle that anyone with experience has seen get in the way of collaboration and data sharing. It is much smarter to support a well documented subset of a standard such as RDF or XForms -and define extensions as needed (both standards allow schemas/ontologies to be extended). If we can we pledge to play along, applications from multiple organizations will add up to be 'more than the sum of the parts'

3. Client Applications: We desperately would like to implement or contribute to a stand-alone fat (aka rich) client that you can use on your desktop to synchronize two data endpoints. Ideally this client would allow you to set the endpoints for synchronization, mapping of data schemas, filters, and managing conflicts - in a secure and easy-to-use interface. Any pointers?

4. Transformers and Adapters: There are many existing applications out there that do their work very well. Sometimes two applications serve similar purposes for different audiences or contexts. Sometimes new applications have to coexist with politically entrenched older systems. While we are building common-purpose adapters to mesh4x (such as Hibernate, Java RMS, and KML which we already have but also CSV and google spreadsheets for example), we already hear demand for specific adapters that take into account particular needs of real-world applications that are already deployed in the field. Which systems should we start with? We have been approached with questions about mesh4x and OpenMRS (http://openmrs.org/) or Sahana (http://www.sahana.lk/).

We'd love your input!

A roadmap toward a European healthgrid

2008-05-20T17:47:37-05:00

This is an exciting work towards building an environment of sharing of resources across heterogeneous and dispersed health data:

molecular data (e.g. genomics, proteomics)
cellular data (e.g. pathways)
tissue data (e.g. cancer types, wound healing)
personal data (e.g. PHR, EHR)
population data (e.g. epidemiology)

Also, an application which can be accessed by all users as a tailored information system according to their level of authorization and without loss of information. Collaboration is the heart of this especially across multiple disciplines. In addition, many standards have not yet been realized (e.g., HL7 (Health Level Seven), the US based Standards Development Organization that currently offers asynchronous messaging), this makes a grid approach far more powerful where organizations join the grid and make their information available for querying, processing, analysis, etc.

Few challenges remain, such as:

How do we secure and maintain high performance of such distributed structure of data integration and computing?
How do we close the gap between grid standards and health-related standards [some nice work's been done here by Power, et al]?
How do we go about next-generation open source ontologies for medical informatics?
How do we close the gap between hospital policies, public health policies, etc. and the grid approach?
How do we go about consumerism and patient ownership of her or his data?

If anything, I hope this raises more awareness of the grid application in health and public health - much still remains to be answered.

Successes are already underway in the health community, for example:

in the European health community (HealthGrid©)
US CDC [presentation on the Public Health Grid by Ken Hall (BearingPoint, Inc) and Dr. Tom Savel (US CDC) at the recent HimSS February 2008 meeting]

But also there are lots of lessons which we can learn from the innovative thinking of efforts like the Google Cloud (click here to learn more about cloud computing).

I'd be very interested to hear your thoughts on this...

Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response

2008-09-07T17:47:37-05:00

A system for early detection, situational awareness and coordinated response is essential to effectively mitigate the threat (morbidity and mortality) of a health-related event and to improve health. The progress made to-date in biosurveillance worldwide is significant and should be evolved to meet existing and emerging needs. There are existing processes, relationships, technologies, policies, infrastructures, and advances in science and technology that provide a solid foundation to a truly integrated biosurveillance solution. Of paramount importance is the need to strengthen the capacity and enable data-driven decision-making of public health services from the local to the district, national and global levels—both strategically and tactically.

In looking at the current landscape; however, we found the majority of the designs, analyses and evaluations of early detection (or biosurveillance) systems have been geared towards specific data sources and detection algorithms. Much less effort has been focused on how these systems will "interact" with humans. For example, consider multiple domain experts working at different levels across different organizations in an environment where numerous biosurveillance algorithms may provide contradictory interpretations of ongoing events.

Nico and I have been working on project codenamed RNA (or Event Evolution) to provide the public health, disaster and humanitarian communities with 1) a "collaborative” virtual environment supporting the entire life cycle of an event, and 2) a ubiquitous biosurveillance capability. Our objective is to connect early event indications to a coordinated and timely response, therefore reducing the response cycle. Through a hybrid (event-based and indicator-based) surveillance approach, we aim to provide processes, methods, and technology tools for streamlining collaboration between domain experts and machine learning algorithms. By synthesizing a wide variety of health-related event indications into a consolidated picture, RNA is anticipated to help the user community to:

Rapidly identify, characterize, localize, and track health-related events
Maintain a global awareness of the situation
Integrate and analyze data relating to human health, animal, plant, food, and environment
Disseminate alerts

The humans are essential part at every step of the life cycle of an event. Humans understand the meanings of information, languages, images, etc. better than machines alone and can make contextually relevant collections of information. Each collection can build the power of the knowledge network in order to corroborate or refute different hypotheses especially at the early and sketchy stages of an event.

RNA will consist of several high-level modules, including:

Data processing
Automatic feature extraction, data classification and tagging
Human input, hypotheses generation and review
Predictions and alerts output
Field confirmation and feedback

The data processing module allows users to assimilate, broker, and/or collect information from several sources (SMS messages (e.g., Geo-Chat microformat), RSS feeds, email list (e.g., ProMED or ProMED MBDS), documents, web pages, scholarly publications, electronic medical records, animal disease data, environmental feed, remote sensing, VoIP, alerts, etc.). A number of ontologies (geographical and disease) will be supported in multiple languages, including: English, Spanish, French, Japanese, Vietnamese, Thai, Korean, Chinese, Arabic, Russian, and Khmer. Disease ontologies will include: Medical Subject Headings (or MeSH), Systematized Nomenclature of Medicine--Clinical Terms (SNOMED-CT), Logical Observation Identifiers Names and Codes (LOINC®), and The International Classification of Diseases (ICD9/10.)

The automatic feature extraction, data classification and tagging module is extensible allowing the introduction of machine learning algorithms (e.g., Bayesian). The components of this module can extract and augment features (or metadata) from multiple data streams; such as: 1) at the earliest stages of a disease outbreak it extracts source and target geo-location, time, route of transmission (e.g., person-to-person, waterborne), 2) at the later stages of a disease outbreak it provides detection and suggestion of tags for new sources based on the evidence on other sources using Support Vector Machines (or SVM) (as will be discussed below). Additional feature extraction include "data decorators"; for example, extracting features, such as: soil moisture, temperature, and mosquitoes density from a reference NASA remote sensing database during a suspected Dengue fever outbreak investigation following heavy rainfall and flooding associated with the landfalls of a hurricane in the Lower Rio Grande Delta. In addition, these components help detect relationships between extracted features within a collaborative space or across different collaborative spaces (e.g., Riff, the ProMED MBDS network.) We plan on using a smaller but more comprehensive set of event classification as follows:

Large aerosol release
Building/vessel contamination
Small release or contamination
Continuous or intermittent release of an agent
Contagious person-to-person
Commercially distributed products
Waterborne
Vector/host –borne
Sexual or parenteral transmission

This high-dimensionality of threat space classification can be further reduced to:

Single or focus event
Continuous/sustained event
Distributing/disseminating event

With human input, this module can help suggest possible classes (or a combination of classes) depending on time, space, and where we are in the life cycle of an event. Possible classifications include the following (Note: All tags/classifications can follow a hierarchical construct):

Syndromes (e.g., dermatological, gastrointestinal, musculoskeletal, neurological, respiratory)
Symptoms (e.g., fever, cough, sore throat, diarrhea)
Routes of transmission (e.g., person-to-person, waterborne, foodborne, aerosol)
Diseases (e.g., TB, HIV, Influenza, Influenza/Avian Influenza)
Hosts (e.g., human, animal, plant, multiple)
Time and geo-location (spatio-temporal features)

For example, at the earliest stages of an infectious disease outbreak, classification of an event could indicate that “there is an unknown respiratory event, transmitted person-to-person, detected in location X, and is spreading with a Y spatio-temporal pattern, across regions Z1, Z2, and Zn.”

The human input and review module is exposed as a set of features that allow users to comment, tag, and rank the elements (positive, neutral, or negative). Additionally, users (or groups, such as communities of interests) can generate and test multiple hypotheses in parallel, further collect and rank sets of related items (evidence), and model against baseline information (for cyclical or known events).

With that information together, the field confirmation and feedback module helps maintain and record history of a list of ongoing possible threats. Components in this module allow domain experts to focus their field investigation and information gathering in order to confirm or refute hypotheses under consideration. Feedback information is then fed into the virtual collaborative network to update (increase or decrease) the reliability of the sources and credibility of the users in light of their inferences or decisions. By analyzing the factors contributing to the identification of various events, it will be possible to help future epidemiologic investigation through play-back or retrospective analysis. Possible information we anticipate recording include:

Which automated systems generated the most reliable alerts, and for what types of conditions?
Which human users where the most effective in identifying conditions?
Which indications are the most effective in identifying a health event?
What factors help to minimize or aggravate a health event?
Which elements of the biosurveillance life cycle require the most time and/or collaboration?

The network history can provide a common point of evaluation (for the overall situation awareness and the individual processes of the biosurveillance network) for a variety of surveillance and response techniques.

Over the summer, the Humanitarian FOSS (HFOSS) Project Summer Institute 2008 (May' 08 - July' 08) carried out an internship project mentored by InSTEDD and a number of HFOSS faculty. During this internship, Juan Pablo Mendoza and Qianqian Lin developed ALPACA Light Parsing And Classifying Application (ALPACA) to:

Transform raw unstructured documents (e.g., news reports, ProMED mail, etc.) into machine readable and analyzable data using a text parsing module
Categorize documents using a SVM classifier using libSVM for: a) Classification into a predetermined (user-defined) list of categories as described above (syndromes, symptoms, routes of transmission, diseases, etc.), and b) Suggesting additional tags and/or topics using a Naive Bayes classifier given existing topics and monitoring human input and review. This is especially helpful with new (emerging) threats or those threats that we know about but we experience them at a much bigger scale than usual (e.g., far more virulent flu virus than we’ve experienced over the past few years)

We tested ALPACA against two widely accepted early sources of information in the public health community; Reuters news and ProMED mail. Results are shown here:

ALPACA is extensible through a plug-in functionality that provides a simple way to add additional parsers and classifiers to the application. We are continuously adding and testing additional algorithms and we welcome your contribution to help us better calibrate existing classifiers and parsers as well as introduce additional ones (you can visit our collaborative space here.)

With RNA we hope to provide the user community with a ubiquitous capability that enables detection, prediction and response to health-related events through a collaborative environment that combines data exploration, integration, search and inference—providing more complex analysis and deeper insight. We've demonstrated RNA's initial capabilities (feature extraction, classification, and tagging and item clustering with a spatio-temporal context) as part of Riff and leveraging mesh4x during a demonstration for the MBDS in SE Asia last week. In the future we also plan to offer RNA as a service that can be integrated with other platforms and networks.

The analytical and collaborative support does not end at the early detection of an event; we envision RNA to provide a rich and flexible functionality during and after an event for maintaining situational awareness, effective response planning, and evaluation. We plan on providing RNA's libraries, tools and applications in the Google Code soon. In the meantime, we look forward to your feedback and contribution.

Some Definitions

Public health: "is the study and practice of managing threats to the health of a community. The field pays special attention to the social context of disease and health, and focuses on improving health through society-wide measures like vaccinations, the fluoridation of drinking water, or through policies like seatbelt and non-smoking laws. The goal of public health is to improve lives through the prevention and treatment of disease. The United Nations' World Health Organization defines health as "a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity." In 1920, C.E.A. Winslow defined public health as "the science and art of preventing disease, prolonging life and promoting health through the organized efforts and informed choices of society, organizations, public and private, communities and individuals." The public-health approach can be applied to a population of just a handful of people or to the whole human population. Public health is typically divided into epidemiology, biostatistics and health services. Environmental, social, behavioral, and occupational health are also important subfields." [Source: http://en.wikipedia.org/wiki/Public_health]
Epidemiology: "is the study of factors affecting the health and illness of populations, and serves as the foundation and logic of interventions made in the interest of public health and preventive medicine. It is considered a cornerstone methodology of public health research, and is highly regarded in evidence-based medicine for identifying risk factors for disease and determining optimal treatment approaches to clinical practice. In the work of communicable and non-communicable diseases, the work of epidemiologists range from outbreak investigation to study design, data collection and analysis including the development of statistical models to test hypotheses and the documentation of results for submission to peer-reviewed journals. Epidemiologists may draw on a number of other scientific disciplines such as biology in understanding disease processes and social science disciplines including sociology and philosophy in order to better understand proximate and distal risk factors." [Source: http://en.wikipedia.org/wiki/Epidemiology]
Outbreak: "is a classification used in epidemiology to describe a small, localized group of people or organisms infected with a disease. Such groups are often confined to a village or a small area. Two linked cases of an infectious disease are usually sufficient to constitute an outbreak. Outbreaks may also refer to epidemics, which affect a region in a country or a group of countries, or pandemics, which describe global disease outbreaks." [Source: http://en.wikipedia.org/wiki/Outbreak]
EID: "Emerging infectious diseases (EIDs) are caused by pathogens that have increased in incidence, geographic or host range, have changed pathogenesis, or are newly-evolved or newly-recognized. Over three-quarters of emerging infectious diseases are a result of zoonotic pathogens. Evidence suggests that emerging diseases are driven largely by anthropogenic environmental changes and/or changes in human demographics and behavior. In certain areas, these factors act on a background of high pathogen biodiversity and will alter host-parasite dynamics driving the emergence of known and unknown pathogens." [Source:http://www.conservationmedicine.org/eid_overview.htm]
Biosurveillance

Keeping our infrastructure 'in the cloud' and our costs close to the ground

2008-03-25T14:26:00-05:00

At InSTEDD -like at any other non-profit..or a well-run business- there is a constant evaluation of how we are using our donors' money, looking for ways we can reduce overhead and anything that doesn't translate directly into mission-related impact.

Given how much of our focus is on technology, it is natural that this concern affects how we design the infrastructure that supports our work. In this post I share the toolset we use to support the lifecycle of our technology which is effective as well as lean. Perhaps others can take advantage of the evaluation work we did or can suggest useful alternatives..

Our key requirements are:

The tools work with our Scrum+XP (eXtreme Programming) processes
The tools work for an internationally distributed team even when in the field
They are efficient cost-wise as well as adequate for the task and reliable

These requirements led us to evaluate many approaches. Ultimately, we opted for an infrastructure that requires no intranet, and no on-premise servers. That means no extra staff of acolytes & operators simply to keep the things going, and associated savings on power/heat/rackspace management...expenses decidedly not core to our mission.

I'm exaggerating. We do have an intranet. It has a printer.

By using software-as-a-service or software+services we have the advantages of lesser operations and increased reliability. We must also take a hard look at three contentious areas:

Security: Will the hosted service provide us with the level of confidentiality, transport security, and the management of user privileges that we need?
Data Portability: Will the hosted service allow us to import - and even important - EXPORT data to another service? We didn't want to fall into a lock-in scenario with 'trapped data'. Both on and off-premise backups are a must.
Accessibility: Will the service be accessible in the field? How Can it cope with low bandwidth connections? Is it possible to work offline?

In the end, we have arrived at the following list of tools that as a set fare well with our way of working and our needs. You can see the rough cost structure for the services that aren't free (When I say 'Free' I mean gratis/no cost/Free as in Beer.)

Engineering Tools:

Google Code -http://code.google.com/- We use Google Code for the source code control (SCC) of components and tools we release as FOSS . It provides issue tracking, a wiki, and downloads in addition to the source code control features. Free.

CVSDude - http://cvsdude.com/ - We host in CVSdude the source code for projects in their early stages when they aren't open source yet. Monthly fee.

Tortoise SVN - http://tortoisesvn.tigris.org/ - Tortoise is the most popular SCC client in the team. CVS & Subversion allow working offline - and managing multiple copies of the source trees on the client, and Tortoise allows you to manage these with ease. Another great feature is that we can keep in the same source tree a mix of projects hosted on Google Code and CVSDude, allowing developers to just do single update and commit operations. This helps us work on our ~~embarrassing~~ early code while keeping the open source projects up to date with no extra hassle. Monthly fee.

Fogbugz - http://www.fogbugz.com/ - We use Fogbugz for our work-item, task, and bug management. Batch updates are easy with its AJAX-based list management. It allows you to create private and shared views, exports data in multiple formats, provides email notification and reports (eg burndown charts) that are useful in agile processes. And - great for testers - it even has a client that allows you to take screenshots, annotate them, and attach them to new bugs. Finally, it has a killer feature that allows me to create new tasks by just typing and pressing "enter" ("killer" because its abuse guarantees my death at the hands of the engineering team). Monthly fee.

Virtualization - Although it is not "hosted infrastructure", virtualization saves on hardware costs. We run most of our work in virtualized environments which include including dev boxes, boxes running demos, boxes for testing or building. Some devs run these from XP, Linux, or Mac OS. We tend to use VMWare, which does a good job of allowing VPCs unfettered access to USB ports and such when working with special devices. Free or not depending on the virtualization product you use and the OSs you are hosting.

The collective annual fee of all the services listed above roughly equal the cost of one moderate-size server with no OS, no software and no support staff.

Communications & Sharing

Skype - http://www.skype.com/ - For voice, video and chat. At crunch times, our team keeps a Skype channel open -- sometimes for hours at a time. It provides a sense of literally being in the same room. We are also looking at ooVoo, vSee and N-way for video. Basic Skype is free. Additional features, such as forwarding calls to a cell phone, are very cheap.

Conference calling- http://www.intercall.com/- We chose a conference call provider that has access numbers in over 50% of the world's countries. Although costs are based on the number and location of call participants, overseas tolls are avoided, which is a significant savings on international calls.

Twitter (http://www.twitter.com) "Micro-messages" are great for ad-hoc communications, especially by SMS users spread across several countries. Within the team we send twitter direct messages by prefixing messages with d as in "d some-name I just uploaded the new version check it out". I use Twhirl as a Twitter desktop client. Free.

Sharedview - (Link) We use this new Microsoft tool for quick-and-easy screen sharing. Drawback: it only runs on Windows. But most of us have some flavor of Windows running - even if it is in a virtual machine on Linux or MacOS. Free.

Google Docs - (http://docs.google.com/) We use Google docs for taking notes and brainstorming during conference calls. It allows multiple users to collaboratively edit a document in real time. Once completed, though, we copy the document into MS Word or OneNote and save in Groove, which makes it possible to access the information offline. Free.

Groove - (Link) Microsoft Groove is useful for team coordination, managing "knowledge bases" of technologies and, most important of all, for tracking user requirements in the field. Since Groove is inherently an offline tool, it shines when Internet connectivity is an issue, but local connectivity is possible. It is not a traditional "cloud" product, but is based on a secure mesh architecture that allows pure peer-to-peer interaction. Unfortunately (hint!), it only works on the Windows OS, it speaks non-standard protocols over the wire, and has no "Web Access"/"Live" component to it. It is a part of Microsoft Office Ultimate. License Fees.

Still needing improvements...

These are some of the shortcomings with these hosted services which we hope will be addressed in the upcoming years:

Offline access: Many web-based tools would be more useful and valuable if they also offered a thoughtfully-designed, well-architected, reliable client for offline usage (and it takes more than just sprinkling Google gears around your javascript to achieve this, but that's a topic for another post).
Unified authentication - The growth in number of sites using single sign on technologies such as OpenID is encouraging but more would be better. In addition, services such as access control and other crosscutting features could be added into the mix (a trend I encouraged at an AOP panel long ago).
Support for Integration - I'd like to see more sites view themselves as 'building blocks' -- as part of a larger solution instead of trying to be the 'one stop shop'. Data, process, and UI integration APIs are always welcome.

So far this mix of products has been working well. The increase in bandwidth, along with tools and standards has allowed us to have core engineering-mission-critical tools online, and "software as/plus services" a cost-effective strategy we use everyday.

Mesh4x: New Open Source Project for Data Meshes

2008-04-24T01:26:00-05:00

Today we created a new open source project to host InSTEDD's efforts on data meshes.

The goal for the project is to provide libraries, tools and applications that simplify using standards-based data meshes. Our contributions will be based on the requirements observed in global health, community development and humanitarian aid.

You can find the project here: http://code.google.com/p/mesh4x/

Our first contribution to the project consists of some libraries that implement the FeedSync specification, an open standard that describes version vectors, and processes for conflict detection and conflict preservation. FeedSync also happens to be one of the underpinnings of Microsoft's consumer-targeted Live Mesh, but could be used happily on any platform as it's based on extensions to RSS and ATOM - an obvious idea is to build a Feedsync javascript adapter for Google Gears).

Because of the project's emphasis on standards, we structured the source tree so it would host implementations in more than one platform and language. 'Mesh4x' has 2 starter source code folders - Mesh4j (Java), Mesh4n (.NET - a large C# contribution done by Clarius Labs and the Microsoft XML MVPs who already had an open source version, unit tests and all). We hope to eventually see Mesh4php, Mesh4r (Ruby) and so on.

It's a start. InSTEDD's work in SE Asia in addition to the input of humanitarian aid agencies and other providers of technology for social good will be the drivers behind our contributions. We expect work in these areas:

Adapters to different stores (e.g. MySQL, or application-specific formats, such as KML), for servers & clients, and the ETL (extract, transform, load) that goes at heterogeneous endpoints. I heard a great idea today for building open-source VMs that run on Amazon's cloud hosting.
Implementations that work on mobile devices (for example, we are currently refactoring the Java library to run in J2ME)
Support for different transports (plain XML over files, or HTTP is a start, but there are optimizations that can be done for low bandwidth, no-Internet scenarios, or integrating with a transport mesh like WASTE).
Integrating implementations with standards-based authentication and data signing approaches.
..your contributions!

Applicability for Humanitarian & Health Scenarios

So why are we at InSTEDD interested in this? Data meshes have some interesting properties:

Symmetrical: They allow data to exist in a concurrent multi-master environment where updates can be applied at any node in the mesh.
Asynchronous: They allow offline updates to information and synchronization with other nodes without requiring data locks, essential for occasionally connected applications.
Dynamic: The synchronization can happen even in constantly changing connectivity topologies. I can sync to a server and later the sync can be done between my client and another client, who could then sync with another server if the first one is there, and so on.

These properties make them very suitable for humanitarian, crisis, and health care environments, where information sharing, data system integration, and technologies that assist politically neutral solutions are beneficial. For example:

Symmetry allows you to have two audiences work on the same data through different applications, with no application being the 'master'. You can also have data sharing of sensitive information between countries or organizations with no country hosting more or less data than the other. See Mary Jane's post about the NGOs in Cambodia, to understand how important this symmetry and neutrality can be. It also allows data to move around a user independently of the device it's been created on.
Asynchronicity allows work to happen in environments where data connections are unavailable, bandwidth is low, or the only 'transport' is a USB stick.
Dynamism allows the field teams to share data amongst themselves and servers as early as possible. Unlike email, there is no need to wait for connectivity to a specific server to let the information free.

One idea could be to add data mesh capabilities to Sahana, allowing any instance running on a server or laptop to edit the information and 'sync' both ways with any other server or laptop. We have also heard scenarios where users of FrontlineSMS could synchronize information amongst themselves. If anyone is interested we'd happily work with you to see how to approach this..

Come and participate - lets share our scenarios, ideas, and code here: http://code.google.com/p/mesh4x/

BarCamp in Phnom Penh, Cambodia..and sustainable technology practices

2008-03-18T18:37:00-05:00

BarCamp is an ad-hoc gathering born from the desire for people to share and learn in an open environment. It is an intense event with discussions, demos and interaction from participants.

(From barcamp.org)

A group of enthusiasts is now organizing a BarCamp in Phnom Penh, Capital of Cambodia. Check out the website and Google groups discussions. If you are using Twitter we have a channel #BarCampPhnomPenh which you can follow here (hashtags allow you to see at a glance all 'tweets' that contain that keyword).

http://barcampphnompenh.org/

Thanks to Beth Kanter and Tharum for hookup and helping us contribute, respectively!

InSTEDD's support right now is around facilitating the conversation with technology firms who might want to sponsor the event, and finding attendees for the event. If you want to chip in contact us or jump straight into the discussion list above! Even if you can't attend, useful computer materials and sponsorship are always welcome. And of course - Can you spell "swag"?

Cambodia has a quite energetic ICT community. And as InSTEDD ramps up its work in the region we hope to become a part of it.

InSTEDD will be building a small engineering team in Phnom Penh late this year (lots of details TBD, like how we'll work with local partners in setting this up), so maybe we'll meet some candidates in the process leading up to BarCamp, too. I think that will help us build technology with sustainability in mind.

.."Sustainable technology"?

The notion of sustainability - as in sustainable agriculture, sustainable manufacturing, sustainable architecture, etc - applies to technology as well.

It is possible to throw money at a problem with the best of intentions and have very little impact in the long term, or leave things even worse than at the beginning. But one can build a structure of skills, knowledge and capital that folks can use to grow initial efforts into greater, unexpected things. Sustainable technologies can continue to exist for a longer period of time beyond an initial flurry of activity without drawing more from its environment than it gives back.

Technology ventures have many ways of becoming sustainable. From the economic sustainability perspective, for example, one way is to become a commercial product which attracts enough revenue to maintain a team that keeps the product alive and relevant to its users for enough time. Another way is to 'release' the products into open source and allow an open community influence or take over the direction. At InSTEDD we are publishing our work as open source and free services because it makes sense as a long-term strategy to serve the regions we work in. I think there's a knowledge base to be shared amongst non-profits and their beneficiaries about building and deploying technology with sustainability in mind, a library of patterns and case studies about what works and what doesn't in the long run.

I believe that being a 'good neighbor' and participating in the local IT community helps create products that work better, accelerate discovery of related local work, and provide opportunities for folks to get involved with the systems that are being used to improve their own countrywide health.

We'll be posting updates, and discussing topics that we expect will come up as BarCamp takes shape.

Hope to see you in Phnom Penh!

SMS Applications and Microformats - lots of work to do!

2008-04-21T16:26:00-05:00

I got a comment at this weekend's Alt.Net conference - which was echoed in mikel's blog - about us not using a location microformat in the Friends Nearby and GeoChat proof of concept applications on the InSTEDD site.

In retrospect, it would have been nice to have the support for the microformat, and we should have. But it would have not -I believe- been used much, if at all.

Our bad on the omission (easy to add), but I think it would be good to explain why we did what we did, why we'd do it again (with the addition of the microformat support), and why I think a lot of usability testing is still required to make the the conversation about microformats from SMS phones more realistic.

Some background- microformats, like anything that increases interoperability and has the long-term potential of reducing user training, are pure goodness. Microformats specify ways to represent common pieces of information, such as the following for position:

l:lat,lon

...where l is a small L for location, and lat, lon are latitude and longitude. You can also add a location name like so:

l:cityname=lat,long

However in our proof of concepts we accept the following formats, and had to do some extra tricks to work with the input of the non-tech-savvy users. Here are some examples:

latlongmessage	Basic - Lat and long in decimal format with a point, a comma or any (with poor eyesight, when stressed, or in sunlight, it is easy to mix a . with a ,)
32.121 32. 121 32, 121	Weird variations of numeric input for decimal lat/longs. Spaces, commas and points all appear in unexpected spots
32.55.55 32, 55, 55	Weird variations of numeric input for Degrees Minutes Second lat/longs. Many folks had GPSs and copied what the screen showed, which by default is DMS for most devices.
Context-specific funneling e.g. 121.234 to -121.234	We call 'funneling' the act of correcting location using some encoded common sense based on context. If you are reporting fires in California, USA from a truck and suddenly your marker moves to the Yellow Sea right off China, it is quite probable you forgot the 'minus' in your longitude report. For Golden Shadow (which expected activity of people to stay in the area) we just did a blanket rule - we make everyone's location move to the USA. For Friends Nearby and future global apps, we make a hit-test for political boundaries inferred from shapefiles, and feedback to the user the inferred position 'Hope you had a nice trip to Laos' or 'Looks like you are in a plane or a boat'. This gives the user a chance to correct.
Choice of separators	We chose * instead of , ; or # as it was more accessible on all phones we tested, without needing a trip to the symbols menu in most cases.
Geocoding cities and addresses Palo Alto, CASunny! Phnom PenhKh'mim Pain-haa	In an urban setting, lat/longs are a nuisance unless you are in a flood or vectoring a helicopter. We accept addresses or town names and geo-code them using Google's gocoding APIs.
Reduce the need to report location	Assume the sender hasn't moved if no new location is submitted. Reduce the amount of times the user has to go through this complex procedure as much as you can!
Errors and omissions on the above, attempt to resolve automatically AND give a fallback for people to resolve and correct.	Loosing data and bothering the user with 'try again' is more unacceptable than trying to infer the intent of the message, so we did our best. On case of failure, add the last known good position, and log the issue so that a human could correct manually or call up and ask 'where are you?'

Where the rubber meets the road

We did a usability test with CERT volunteers and folks from the local search and rescue team. It was a diverse audience - we had in the same room a range of cell phones (from baby Nokias to Blackberrys) and the people had a range of expertise (some use their phone for email and calendar, others never used SMS before). We assumed we just had one chance to train them so we explained it once, and asked them to start sending messages. The log we got was invaluable as raw input - and informed the parsing algorithms to make them more robust for the actual exercise.

A quickie 1st hand test you can try at home: I have a collection of diverse phones to try these things out on - even phones with Khmer Script input support!. So let me see...it takes approximate 70 keystrokes to enter L:123.45,67.89 in a small Nokia, with no errors. To enter 123.45*67.89 it took me approximately 50 keystrokes. The microformat takes ~40% more key presses (and this particular phone uses the * key for the symbol pad, so that even plays against my point).

I know folks in diverse settings have tried the microformat and it works to communcate position (of course), but I'm not sure it was a representative audience of a broad set of non-tech-savvy users. The breakdown is not in the data itself, but the usability of the format.

Lessons learnt and realizing it's an ongoing effort

The key lesson for me is to make sure we accept the location microformat in addition to more user-friendly formats. If folks know the microformat beforehand (an exception rather than the norm) they can expect 'it just works'.

Alternative paths include applying machine learning feature-extraction efforts to the information, or building a smart client for rich phones that formats message for you (from the GPS?), so the user never sees the location 'wire' format at all. All approaches have pros and cons.

I am also wondering what activity is ongoing in the area of nanoformats, microformats which are slightly friendlier for numeric pad input. It's trivial to invent tiny ways of representing bits of information. The problem is that unless done right they can shift power away from the end user towards the engineers who consume that data. I don't see that as a positive power shift.

To make the effort real these nanoformats would have to get usability testing and feedback from real users in real situations to grow them into something intuitive and easy to enter in different phones.

To do that we (the 'big we') have to continue to experiment with easy to remember schemes which can be trained in one shot, can be context-specific, can be easier to discover, recall and communicate, and works even for a health volunteer who cares a lot about the content and doesn't care about the format at all.

In the end, I believe the best formats, like the best technologies, will be invisible to the end users.

If you are a geo-geek hope to see you at Where 2.0 (InSTEDD is presenting there) in a couple of weeks so we can continue the dialogue!

Mesh4x adds generic database support

2008-05-07T00:43:00-05:00

First of all - a very heartfelt support to the Myanmar population in this times of crisis. Many friends are either already there or on their way to help as part of UNDAC teams. It's a tough situation in a tough context, and all my hopes reach out to the communities there so they can recover soon. Unfortunately, it won't go back to "normal" for a long time, if ever. I was in Peru last week and the August '07 earthquake still defines how people live in Pisco. The press and much of the aid has left and the town is still...leveled. Throw in a major disaster in a non-resilient environment, with a bunch of foreign aid with varied commitments to the region, and the long term outcomes are very hard to predict.

This week we made significant updates in mesh4x. One of them is a Hibernate adapter, which allows you to plug into the mesh almost any relational database available in the market

Hibernate Adapter

In our first scenario, let's say you have, or you are quickly hacking together, an application to help enter, analyze and report information. You have a database schema, and you'd like to integrate it with an excel database that field folks are using for data entry. You need to make sure updates and deletes somehow make it out to the spreadsheets, and that folks' updates make it back in. Furthermore, you'd like folks in the field to synchronize spreadsheets with each other directly - thus making it a classic mesh scenario. With the Hibernate adapter, our goal is to allow you to mesh-enable your database by just mapping your entity fields to your database fields.

Hibernate, as most developers know, is an Object-Relational Mapper library for Java. With this adapter you can now integrate into a data mesh any database engine that Hibernate supports, which is an impressive list. By supporting Hibernate as an adapter we allow every user to customize the mapping of the mesh data to their database schema using familiar tools, and get support for a lot of databases. There is still some work to do - for example, as of today the adapter still requires the database schema to revolve around the fact that the rows are being synchronized in a mesh. We expect in the upcoming weeks to remove this restriction and use two separate 'repositories', one for the synchronization information (which you shouldn't care about) and another one for your data.

This will allow you to point to almost any existing database schema and mesh it up without messing it up. (Apologies, couldn't resist).

You can always reach the project through http://mesh4x.org.

Here you can see a list of adapters and suggest your own.

Build maps collaboratively with new Mesh4x KML adapter

2008-05-08T12:55:00-05:00

A handful of months ago I met Kersten Jauer, UN Information Officer for the Central African Republic (CAR). CAR is a large country in Central Africa, surrounded by Sudan, Chad, Cameroon, Congo, and DRC; 67% of its population lives with under $1 a day and is scoured by constant internal rebellions and gender-based violence.

Kersten spends a lot of the time in the field in CAR, and put together an amazing map of the whole country to support logistics and NGO programs. Roads, provinces, bridges, fuel pumps, it all got captured by hand in Google Earth and saved as KML files. By the time I got it, Kersten's KML had grown to be 11 MB, an amazing amount of information patiently collected and edited, and periodically shared online with all those working to improve the region.

Contrast the map above showing the CAR KML with the map on the right showing the same region as seen by default in Google Earth. What got my attention was a little note in the KML:

If you would like to comment on this file or have suggestions please email to MapsAndGoogleEarth+car@hcpt.jot.com

To add a placemark just email it with a short description to the same address or kersten.jauer@undp.org

Please also check out the maps section on http://hcpt.jot.com

The map was built collaboratively, but imagine the workload Kersten must have had getting little snips, integrating them on the larger map, and then letting folks know of updates. And how would the map be maintained whenever Kersten was attending to some emergency?

Mesh4x KML Adapter

We started building a simple instance of a KML adapter for Mesh4x this week. This adapter would allow a team of people edit a KML file and then 'synchronize' it with all the others. For example, I could add a pushpin saying a bridge is down, and you could be editing another pushpin or moving it around to represent that a logistics truck has moved. When we synchronize, the truck moves around in my KML and the broken bridge appears in yours.

This could be synchronized peer-to-peer (a KML on your disk to a KML on a USB drive or someone else's box) as well as via a 'cloud' web service. Note this is changing the data inside the KML, it is not just 'file sharing'. The adapter knows about KML and keeps track of versions of fine-grained elements (pushpins, placemarks, polygons) inside the same file. It is an example of how a data mesh could be used to synchronize fine-grained data between applications.

We chose KML for this adapter as it is a standard ("OGC KML") that is widely used and supported by Google Earth (of course), Microsoft Virtual Earth, as well as nice tools that work offline and can be used in the field such as GeoPDF.

We have a sample UI (shown here) to let you play around with the basics. The effort is still on the libraries and we don't have a neat UI to let you choose endpoints or resolve conflicts, but all will come in due time. Other restrictions include having to put your placemarks in a "Shared Items" folder in your KML, and styles don't get replicated. We foresee no problems working out these constraints over the coming weeks.

To try it out, make sure you have Java installed and:

Get the sample application from http://code.google.com/p/mesh4x/downloads/list
Double click on mesh4j-KML-DemoApp.jar
Point to a KML or open the sample ones
Edit the location of Sample Pushpin 1 in File 1
Add a new pushpin in File 2
Press synchronize, and after both files should have the updated Sample pushpin 1 AND the new pushpin!

Another advantage of a data mesh is that endpoints can be heterogeneous, as long as you do the appropriate mapping. Eventually you will be able to sync a spreadsheet with columns such as Title/Description/Lat/Long into KML pushpins and back quite easily.

We hope to be showing this at Where 2.0. A lot of the team has been focusing on supporting the Myanmar disaster relief, so progress this week has been a bit random, but we still want your feedback!

Learn more about Kersten's Work in CAR at www.hdptcar.net, or get Kersten's latest epic KML.

See the Mesh4x project at http://mesh4x.org