Hello to everybody!
I’m very glad I’ve been given the opportunity to write on this blog, a special thank to Samuel Phung for this!
In my first post on this blog, I’d like to give you a technical description of KitchenPal, 2nd prize winner in the Microsoft hobbyist competition embeddedSPARK 2011.
What's for dinner? It's the question of the hour. Too many home managers look for answers in the supermarket at 5 P.M. Harried, harassed by hungry children, they rack their brains for an answer to the dinner-hour question. Three meals a day. Seven dinners a week. From supermarket to pantry, refrigerator to table, sink to cupboard, the kitchen routine is always the same. We all know the plain and simple fact: into each day, one dinner must fall. What's a possible answer? Menu planning.
Are you looking for inspiration? You enjoy cooking for your friends and family, so how many times you've been searching throughout your cookbooks and notepads, looking for a recipe? How many times a friend of yours asked for a recipe you have in handwritten copy only? Ever desired for a way to instantly browse, search and explore your recipes collection or to look for inspiration in a friend’s cookbook?
Now, let’s suppose you've decided on a recipe to cook for dinner. Do you have all the ingredients? If you had a working and organized pantry, a planned reserve of food and stuff used in the home, you would know the answer immediately. But usually, it is not: people compile approximate shopping lists, shop the grocery stores for what they want, always forget something they needed, then bring it all home and stuff the purchases in the freezer, refrigerator, cabinets, pantry and then hope for the best when it’s time to cook something.
We all live in a digital age, the age when computers control our music, photos, videos, messages and entertainment. Why not let them control our meal planning, manage our recipes and track our food inventory? What about managing the foods we need to create the recipes? What about creating recipes out of the food we have?
Nowadays lots of alternatives and tools exist on the Internet and can be used on PCs and smart phones. But they have one, main, big problem in common: they need a personal computer, and people who know how to use it. Is it possible to make their life easier?
KitchenPal is the complete tool for everyday housekeeping, cooking and shopping. It is a three-in-one device with touch-screen display, suitable for housewives, housekeepers and people who love cooking. It can be placed wherever is most suitable for one’s needs, but it will usually stay in the kitchen. Its main purposes are:
In addition, it can be integrated with a mobile device application which runs on both Windows Mobile 6.x and Windows Phone 7.
Here’s the device prototype video, submitted to the contest:
The prototype has been implemented using the Microsoft-provided EmbeddedSPARK kit (iCop VDX-6318-MSJK, 256MB RAM, Flash-IDE 512MB disk) and a 7’’ LCD Touch-screen display, compatible with Windows Embedded CE 6.0 and Windows Embedded Compact 7. It has been integrated with 4 USB ports and Ethernet connection. The board and the external ports have been installed in a home-made chassis, based on an old car-radio device.
The companion application has been implemented on two different mobile devices: a Samsung GT-i8000 Omnia 2 (Windows Mobile 6.5) and an LG E-900 Optimus 7 (Windows Phone 7).
The system runs a Windows Embedded Compact 7 OS Image, which has been rebuilt from scratch following the provided step-by-step instructions. It has been stripped down to the minimum requirements, in order to have a small footprint and a fast boot. Just essential BSP drivers have been integrated (chipset, USB, LAN), minimum USB support (HID and removable disks only), FTP/Telnet servers for development purposes, .NETCF 3.5, some multimedia support (Alphablending API, DirectDraw, DirectShow, image/audio codecs) and Silverlight for Windows Embedded support for a better on-screen keyboard experience. In addition, a third-party touch-screen display driver and some helper applications (CoreCon, AutoLaunch, RegFlush) have been added. At last, the standard system shell has been replaced with KitchenPal main application.
When I started to plan the user interface for the device, I was reading some books about the new Windows Phone 7 platform and its simple, cool, clean, modern Metro UI: immediately I wanted to use a similar interface for the challenge device and its mobile companion applications. I think that Metro is the perfect touch UI model for non-technologic people: easy to learn once and use everywhere!
My initial idea was to develop a Metro-like interface using Silverlight for Windows Embedded but, looking around on the Internet, I found the Fleux open-source project (by Jose Gallardo Salazar) which mimicked a Windows Phone 7 UI on Windows Mobile 6.x, using .NET Compact Framework. That was great!
Some doubts existed about performance issues using .NETCF instead of Silverlight/C++, but after some initial testing both on the Mobile phone and the device, I decided for Fleux: KitchenPal has been developed using Windows Forms and .NETCF 3.5, in C#.
I used Fleux library as a starting point, learned its internals and programming model, and extended/fixed it to work both on Windows Mobile 6.x devices and Windows CE devices as well. Also I added a number of custom controls (ApplicationBar, TextBox+SIP integration, WrapPanel, Button, ImageButton, Tile) which are not available within the open-source project and planned some refactoring work in order to be able to use it at design-time in the future.
User interface has been organized in multiple Views (totaling 33) and UI Controls (32, in total), adopting some Silverlight/MVVM patterns in order to have a common project organization and an easier, parallel development cycle on multiple platforms.
Until now, everything has been developed in code, without design-time support, which required a lot of time for trial and error testing and UI refinement (more than a month, in free time). Different was for the Windows Phone 7 UI implementation: thanks to its native support for Metro interface and to the powerful Microsoft Blend 4 designer, the same interface required less than one week to be fully implemented.
Most of the Views (in Silverlight naming-convention they correspond to Pages) are made of Pivot controls, inertia-driven scrolling lists (StackPanel + ScrollView), TextBlocks, ImageElements, ImageButtons. Some pages, where item details are shown, use Grid containers and TextBlock/TextBox controls. Each page has an ApplicationBar on the bottom, which contains up to 5 ImageButtons used for navigation purposes or custom KitchenPal tasks (i.e. barcode scanning, cloud sync manual request).
Also localization support has been considered, using String resources in external assemblies and the possibility to use a different version of the database for each supported language (Recipe names/descriptions/instructions, Products info, etc.); now, only English language is implemented.
KitchenPal can be used both in connected and disconnected scenarios. If an Internet connection is available, synchronization between devices can take place.
All data is stored locally in a SQLite database. This DB engine has been chosen because of its performances and for its full managed code availability: this was a main requirement in order to implement a companion application on Windows Phone 7 platform. On Windows Mobile and Windows Embedded Compact the open-source System.Data.SQLite ADO.NET provider has been used: in order to have it working both on the Mobile phone (ARM-based) and on the embedded device (x86-based), the native part of the provider had to be rebuilt for the x86 platform. The latest official build (for ARM) was almost one-year-old, so, while recompiling, I decided to update SQLite to the latest available version (3.7.4), with some headaches…
On Windows Phone 7, instead, I used the C#-SQLite for WP7 open-source library as a starting point, and made some tricks and fixes to make it work with the same DB schema and file used on Windows Mobile 6.x and embedded devices. Thanks to a layered architecture, it will be easy to eventually change the data provider in the future (maybe when there will be an official supported DB on Windows Phone).
Here’s a summarizing E-R Diagram. The system supports almost all the features listed in Round 1 document: Shopping List management, Products catalog, Pantry management, Stores and Aisles, Recipes collections, Meal planning.
To implement the device-to-device synchronization feature, a cloud-based storage has been implemented using the Windows Azure platform: data is stored in Azure Tables and exposed via a dedicated WCF service hosted in a small, single-instance WebRole. Windows Azure has been chosen because of both work and personal interests in learning and using this cloud technology, after an introductory training course I followed in December 2010. This project represented a good subject to put in the field the things I learnt.
A dedicated module of the system, Syncronization Manager, is responsible for all sync operations and conflicts management. Synchronization is performed with a custom algorithm inspired by SyncML standard, where each device is at the same time a SyncML client/server: client role, when using the local data store, server role when using data from Windows Azure. Some additional metadata is stored in both data-stores (record IDs mapping, sync anchors, change logs), to efficiently support sync operations and reducing data exchange to the minimum. Two SyncML standard modes are supported: slow-sync (first-time synchronization, all data checked and exchanged) and two-way (only differences from the last good synchronization are exchanged).
KitchenPal has been designed with modularity in mind, in order to better separate functionalities and facilitate code-reuse, testing, maintainability and future expandability, even though it is just a prototype.
The decision to use .NETCF 3.5 and C# also made it possible to adopt a single codebase architecture: almost the same managed code runs both on the device and on the Windows Mobile companion application (minor differences are in UI, due to different screen size/resolution). In addition, thanks to .NET and designing a N-layer architecture made it possible to use everything (UI layer excluded) also for Windows Phone 7 devices. Potentially, everything can be used as-is for desktop and/or Silverlight for Web applications, too.
Here’s a diagram summarizing the whole software architecture:
The user interface modules, one for each supported device. Embedded and Mobile UI shares the same codebase, implemented with Windows Forms and based on Fleux library. Slightly differences are present, due to different resolution/screen size constraints. Instead, Windows Phone 7 has a complete different implementation, with Silverlight technology and a little of MVVM design pattern. Ideas from MVVM have been used also for the other platforms, in order to have a common project organization and an easier, parallel development cycle. For each platform, a different UI project has been created, linking common project’s source files, resources and database between them to avoid duplication and management issues.
This module handles global application settings, using standard .NET App.config configuration file and a custom KitchenPal section. For platforms which don’t support App.config (.NETCF and Windows Phone 7), a number of helper classes have been implemented, to uniform settings management to full .NET Framework behavior.
This layer provides all the logic and high-level classes in order to work with data. Data includes Recipes, Ingredients, Shopping lists, Products, Categories, Pantry, Stores, Store Aisles, Meals, Menu, Coupons, etc. It completely ignores which data storage is used or how data is retrieved from the cloud: via App.Config it is possible to specify which Data Layer Provider and Service Agent have to be used for each single Entity and DLLs are dynamically loaded at run-time and instances created when needed. Right now possible choices are SQLite / SQLiteWP7 / WCF for all entities but, if required, it would be possible to add additional providers for a single specific entity – i.e. an XML provider, a CSV provider, etc.
For each supported entity exists a relative Business Layer class, implemented using singleton pattern, which exposes static methods to perform actions on entities (CRUD operations and KitchenPal specific tasks). Presentation layers or other modules use these classes to work with data.
Main purpose of this module is to retrieve/store data from either the local data store or the cloud. It routes Business Layer requests to the low level Data Access layers or to the Service agents (if data has to be retrieved from the Internet), and provides methods to map low level data entities to business entities (and vice versa). It is implemented as a singleton class which higher levels call and internally maintains Data Layer / Service Agent instances for each supported entity.
This layer implements low level data-store related functions. It is divided in two layers, interface and implementation, which let the Data Provider being abstracted from a specific data-store technology. For the project, two DAL implementations have been developed: one, DAL.SQLite, to be used with SQLite ADO.NET Provider on Windows CE/Windows Mobile and another one, DAL.SQLiteWP7, for C#-SQLite on Windows Phone 7. Depending on a specific data-store, this layer has a number of low level DAL entity classes, different from BLL ones, which are used to abstract from data-store physical implementation (i.e. file entry / ADO.NET Reader / table row, etc.). Dedicated mapping methods have been implemented in order to convert between DAL Entities and physical entities. For each supported entity exists a relative Data Access Layer class, which exposes methods to perform data-store related actions on entities.
This layer implements low level remote-service related functions. It is divided in two layers, interface and implementation, which let the Data Provider being abstracted from a specific remote-service technology. For the project, only an implementation for a WCF service has been provided, ServiceAgents.WCF. Thanks to the layered architecture, it is easy to eventually add alternative service providers (i.e. REST, TCP, Web Services, etc.). Depending on a specific service-provider, this layer has a number of low level service entity classes, different from BLL ones, which are used to abstract from remote-service physical implementation (i.e. JSON / XML / raw strings, etc.). Dedicated conversion methods have been implemented in order to convert between each type. For each supported entity exists a relative Service Agent class, which exposes methods to perform actions on entities over the Internet.
Main purpose of this module is to synchronize the local data storage with the Azure-based remote storage and handle sync-conflicts. As already stated, it implements a custom algorithm inspired by SyncML standard and uses Business Layer classes and entities in order to work with data. Synchronization can happen either on request (some UI windows have commands to let the user manually sync device with the cloud) or automatically, when changes occur in the data – i.e. adding/removing elements from shopping lists or pantries).
This module contains all the Computer Vision logic to perform barcode detection and decoding in a photograph. It has been fully implemented in managed code, without any dependencies on third party native libraries. This way it is possible to use the same library (almost without modifications) on Windows Phone 7.
To implement the synchronization feature, as already stated, a Windows Azure-based storage has been implemented: data is stored in Azure Tables and exposed via a dedicated WCF service hosted in a WebRole small instance. This solution avoided the implementation of a specific Azure related module on Mobile devices, Phone 7 and on the Embedded device.
In addition, the same WCF service gives KitchenPal the ability to interact with Facebook: users can share their favorite recipes with friends. The open-source Facebook C# SDK has been used: it exists for several platform already, but to avoid the implementation of a specific version for .NETCF, I preferred to wrap it in a WCF service which runs on the cloud and has a common client implementation on all the platforms: it’s the service that uses Facebook API to retrieve friends data and to post Recipes on friend’s walls. Also, the service adds a security barrier which avoids storing private authentication ID/keys both for Azure Storage and Facebook on each device. A dedicated KitchenPal application has been created on Facebook, in order to be able to use public Facebook API from custom applications.
KitchenPal Mobile Companion applications have a built-in barcode scanner from photographs. Its interface is really simple, it exposes just a single Scan method which, taken a photograph (using the built-in phone camera, 640x480 image), returns a list of detected barcodes. It has been implemented in fully managed code from scratch, with the help of my friend Antony Vitillo, who’s a Computer Vision expert, in order to avoid any third party CV components and to make the scanner compatible also with Windows Phone 7. Using managed code introduced some performance issues on Mobile devices, but on Windows Phone 7 the scanner is fast enough.
At the moment, the library supports EAN-13/UPC-A barcodes only, but it has been designed to add new barcode standards easily. It works quite well in various lightning conditions, also with softly blurred images, but not with too much “curved” barcodes (i.e. those on bottles), due to the difficult to identify bars and spaces widths with precision. EAN-13 format has been chosen because it’s one the most used but, at the same time, it’s quite difficult to be decoded: it uses bars and spaces with four slightly different widths, very easy to be confused when blur exists.
The algorithm works like this: the original image is divided in blocks; blocks are then analyzed and classified by number of edges and their orientation: those with similar edge conditions are grouped together and supposed to contain a barcode. For each group of candidate blocks, a decoding process take place: bars and spaces widths are retrieved with scanline interpolation and function analysis, and then compared with barcode-specific heuristics: for each comparison a confidence score is given. Higher the score, higher the decoded value can be considered valid. At last, the highest score decoded value is returned. If more than one barcode area is detected, the decoding is performed on each, returning a barcode for each area. If a decoded barcode has a too low confidence score (minimum threshold can be configured, among many other parameters), it is considered invalid and not returned.
It required more than one month to be implemented and it hasn’t been easy to make it work both on Windows Mobile 6.x and Silverlight for Phone 7, due to different image handling, performance optimization issues and other platform differences but, thanks to C#/.NET, a single codebase is used: without modifications, the library works well also on desktop/Silverlight for Web applications.
Developing KitchenPal and its Mobile companion applications has been fun and represented a great challenge and an interesting opportunity to study the details of Windows Embedded Compact 7, Windows Azure, Windows Mobile and Windows Phone 7 technologies and tools. It took a lot of free time, patience, study, experimentation and work to make the proposed idea a real, working and (possibly) useful technological and innovative kitchen tool for everyday housekeeping, food management and cooking. I hope that everybody, from the teenager to the old grandmother, may take benefit of its features.
Are you going to release any of the source code? or specifically the changes you made to the Fleux codebase?
@flo: Source code for the Fleux Library has been released. It is available on my other blog (http://embeddedlounge.blogspot.com).
Re: Experimenting with WES8 and Kinect
any chance you could save the configuration and share it? I see the pics but would like to make a bare...
Re: embeddedSPARK 2011 Finalist – KitchenPal project
@flo: Source code for the Fleux Library has been released. It is available on my other blog (http://embeddedlounge...
Are you going to release any of the source code? or specifically the changes you made to the Fleux codebase...