Sunday, March 27, 2005

Ted's Excellent Adventure (2 of 2)

Just because someone shows up in jeans and sneakers doesn't mean that they're not a great programmer, and I was willing to give Ted the benefit of the doubt. Usually I'm not usually one to judge someone based on their look or clothes. Well, on second thought, I take that back. I usually do form an opinion, then remind myself that I shouldn't judge a book by its cover.

Ted's book was tall and in its forties. He had blond, wavy hair and an easygoing smile. His voice was pleasant and accentless, and though his thick frame wasn't as brawny as it had once been I could tell by looking at him that he hadn't stared at computer monitors his whole life. And he definitely liked the "casual" part of business casual.

After the cursory tour of the company--oh look, here are more offices, and just wait till you see our bathrooms!--I gave him the executive overview of what we did and what the project was about. We set up a work area and held a small meeting about the project.

In a previous CE project that was similar to ours, Ted said, the team he worked on used a 3-tier architecture approach. He recommended it for our project as well. We agreed. After all, the application needed some architecture. Right now it was just a collection of loosely related dialogs and a lot of code written in here's-my-first-crack-at-it-MFC (by me).

Meanwhile, I was working on some nagging CE issues. We still didn't have a way to share memory between the UI app and the real-time app. Our attempt at using memory-mapped files hadn't worked. (That turned out to be a blessing in disguise.) I had a devilishly hard time getting Platform Builder to configure and export a custom SDK, and when I finally did it wouldn't work. So I let Ted port the application into the 3-tier framework. I had some questions about it but I waited until he was finished.

While we worked we got to know more about Ted. He was a former Air Force Search and Rescue parajumper, but the pay rate didn't cut it so he switched to programming. (Some switch!) As an independent contractor he had worked on projects for Intel and Alpine. He did software projects for a living, but his meraki aspiration was in board game design. He had invented a multi-level tic-tac-toe game and was hoping that a big retailer would pick it up. He gave a few of us his game as a present.

The framework that Ted implemented was a sound one, but it had some bureaucratic parts that I didn't like. And while Ted had been on the team that implemented it, he wasn't the designer and wasn't privy to the details that I wanted to know about. It had been previously implemented on a "push" system, where the process control (or object) layer worked off a timer to constantly push data out to the (dumb) dialogs. Our design was different: the dialogs had to be intelligent and needed to request data from the control layer when they recognized the need for new data. Ted didn't like the idea.

Ted had a few idiosyncracies, as we all do. One was that he was fairly certain of his opinions without being able to say why. (On the other hand, I'm also rather opinionated but can usually make up a pretty good reason.) So it was frustrating to try to elicit a reason for why he was sometimes so firmly attached to a particular idea.

He also had a minor disease I started calling "weekenditis." On Monday mornings he would arrive in the depths of despair, bemoaning the sad state of our project and pledging to incorporate a time-consuming, application-wide fix that wasn't in the timeline. Usually the problem wasn't nearly as bad as he made it out to be. I learned to defer these fixes and steer him back to the more important tasks at hand.

He had helped us implement SourceSafe, which replaced our highly efficient but less-than-traceable verbal system. (Don't laugh: I know of at least one Fortune 500 company that still works on major projects by emailing unencrypted .zip files back and forth.) He used it to check out files and make tiny changes to irrelevant things, like code formatting. I would rather have had him spend his time on more important things.

But the small drawbacks were easily outweighed by how big of an asset he became to the project. He fit right into the team, and that's not an easy thing to do. He found a nearby place to stay ( and quickly made friends with almost everyone in the company except Vince Pawlowicz, who didn't get along with anyone anyway. (It helped that he had a cool game to give out.) Our "personal Moses" he was not, but maybe our hopes had been unrealistic anyway. I refactored the timeline1 and we moved on.

For better or for worse, Ted was now a permanent part of the team.

[1] Rutherford, Alec and I had a meeting about how best to use Ted in the light of us still needing a "CE expert." We made some adjustments to the timeline (i.e., shuffling our already-overworked resources around but not adjusting the deadline). Alec remarked that he didn't see how we could possibly finish the project on time, which apparently wasn't what Rutherford wanted to hear because he said nothing but smiled falsely and closed the meeting.

Wednesday, March 23, 2005

Ted's Excellent Adventure (1 of 2)

Q: How comfortable would you be hiring a guy based on a 10-minute phone interview?
A: Not very

Nevertheless, that was the position that we found ourselves in after scouring the country for WinCE developers. I suppose it meant that either the CE developer market was very hot and no one was available to work on new projects, or that the market was very cold and no one was available to work on new projects.

To be fair, we were asking a lot. We needed an experienced, on-site developer who was free within 3 weeks and available for about 3 months. We were looking for our own personal Moses, come to free us from the shackles of our DOS world and lead us into the promised land of Windows CE. There were other people (and companies) available, but none that met those criteria.

After searching every way we knew how, including using a national developer contracting service, we ended up with exactly three prospects. One was Daron, the WinCE training instructor. He was a Microsoft MVP in whom I had every confidence. The other two were prospects from the independent contractor agency.

On the telephone, Sudeep Swarnapurishwara sounded every last bit like his name foretold. Although he lived in the area and was very experienced, he had a few strikes against him. His English was terrible (despite having lived in the U.S. for a dozen years), he sounded too confident about anything that we asked him (including topics where he misunderstood what we were asking him), and he was taking a month-long trip to India during the month the project was scheduled to end.

Our next call was more promising. Ted Nulder lived in California but was willing to relocate to Chicago for the duration of the project. He had experience with two previous CE projects and he interviewed very well. I think the call with him gave some people in the conference room that warm fuzzy feeling.

So between Daron and Ted it came down to a matter of price. Daron didn't come cheap: between $160 - 200/hr, depending on whether he worked off- or on-site. I tried to beat up his company on the price--I've learned a thing or two from our killer purchasing manager--but they weren't willing to budge. By comparison, Ted cost about half that. In our selection discussions I pointed out that productivity between different programmers varies by an order of magnitude, but at that price difference I could feel the hollowness of my own argument.

Our own engineers bill out at $200/hr, so for us to balk at that kind of price conjures up images of pots and kettles. But balk we did. Ted made arrangements to start on Jan. 4, the first working day of 2005. For $90/hr, we were the proud new owner of a CE expert. Our Moses was on his way to lead us through the desert of development into a future flowing with milk and honey products and profits.

Or so we thought.

I was interested to meet the kind of guy that would move to a different city and live there while working on a long-term project. (I'm married, with a 2-year-old son and another on the way, so it wasn't something that I would've done.) So Tuesday morning, when the receptionist rang and said that Ted had arrived, I walked to the foyer to greet him.

An old quote says that you never get a second chance to make a first impression, and I believe it. What kind of impression I made on him, I'll probably never know. But the impression that he made on me, I'll never forget.

Even to this day when I think about it, I can still feel the way my heart sank. I think it was the jeans and sneakers that said it best.

Saturday, March 19, 2005

Writing Kernel-Level Code

If you need access to kernel-level routines, Windows CE provides the OEM Adaptation Layer (OAL) to do just that via custom kernel IOCTLs. A custom IOCTL lets you write code with kernel-mode functionality and provides your application with a means to call it.

First, a quick timeout for some pretty graphics. The OAL looks something like this:

OAL Architecture
(Now why did they ever replace good ol' ascii art with those newfangled graphics?)

While we're in timeout I'll take a second to point out that IOCTL stands for I/O control (code) and can be pronounced eye-okk'-tull if you're feeling brave.

Okay, back to the tutorial. Creating a new IOCTL isn't very hard. All you need to do is define a custom code for kernel handling that can be called in an application with KernelIoControl. Here's how to do it.

1. Open oemioctl.c, which should be in
The basic structure of the OemIoControl function is a large switch statement. Adding new functionality is as easy as defining a new case and handling it.
Note: You should be able to find a list of currently defined codes in ..\OAK\INC\pkfuncs.h.

2. Define a new control code. Valid control codes for OEMs are between 2048 - 4095. (MS reserves 0 - 2047 for itself.) Hopefully 2,000 custom codes are enough. (For comparison, we're using 17 codes right now.)

A control code might look like this (excuse the width problems):

The third parameter, METHOD_BUFFERED, is ignored in CE. CTL_CODE expands to
#define CTL_CODE( DeviceType, Function, Method, Access )
( ((DeviceType) << 16) | ((Access) << 14)
| ((Function) << 2) | (Method) )

3. Add a new case under the main switch statement in OemIoControl:

retval = TRUE;
Note: This is a good time to put defensive programming to work. Any problems here and you'll hang or crash the entire system. For starters, make sure that you check the actual size of the incoming buffer against the reported size.

(From the MSDN 2535 lab notes: Just for testing purposes, you might want to try incrementing a counter every time you pass your IOCTL and printing a RETAILMSG that displays the value of the counter. If you run into some problems and you're wondering if anything is actually happening this is a nice sanity check.)

4. Build your platform.

5a. If you're using Platform Builder, open your project and add the following where you need it:
#include <kfuncs.h>

KernelIoControl(IOCTL_ZOOM_TAXI, // I/O ctl code
NULL, // input buffer pointer
0, // input buffer size
&dwTaxi, // output buffer pointer
sizeof(dwTaxi), // bytes that can be returned
// in output buffer
&dwNumTaxis); // size of data returned

Make sure that you #define IOCTL_ZOOM_TAXI in your project as well.

5b. If you're using eVC, you'll need to configure and export your custom SDK from PB first, then install it. The standard emulator won't know about your custom control codes. (I'll write more about custom SDKs later.) Then add the above code.

6. Make sure to add the WINCEOEM preprocessor symbol, which includes OEM headers (see kfuncs.h). On the Project menu, click Settings. On the C++ tab, select the Preprocessor category and add
at the end of the Preprocessor definitions list.

In the Additional include directories field, add these two directories:

As you can see, it's not very hard to do. For small things like hardware info and hardware access, it's a lot easier to do than writing a driver. But be careful how many times you call into the kernel. KernelIoControl is handled by generating an exception, which the kernel routs to the OAL and to your IOCTL. The cost of a kernel call in a real-time environment needs to be evaluated in your app.

By the way, if you're wondering what the heck we're doing with a taxi call in the kernel, uh, we're not. But someone else once did.

Below is the PDF of the MSDN lab:
PDFMSDN 2535B_Lab5.pdf (con permiso)

Microsoft MVP Steve Maillet writes: In Windows CE, "kernel mode" simply means that a thread has access to the virtual address space 0x80000000 - 0xFFFFFFFF without triggering an access violation. That is only difference between user mode and kernel mode in Windows CE. Any thread can call SetKMode() to switch between the modes as long as the process it is running in is trusted (or the trust system is disabled).

Tuesday, March 15, 2005

The New Old Thing

Apart from Microsoft Word and Excel, I can't think of any applications that made smooth transitions from their previous, powerhouse DOS selves into Windows. In fact, every application that I can think of apart from the above two was either more awkward to use or had less features than its DOS counterpart. This was due to at least a few things:

  • Writing Windows apps (at the time) was harder
  • The Windows event model was more restrictive than a comparable (i.e., complete control) DOS app
  • The companies who produced these apps didn't fire their entire development team and hire Win16-proficient programmers. There was a natural learning curve as the programmers started learning how to program for Windows.
  • The shell, which had previously been subservient to the all-powerful application, was now the master. The applications were learning to accept their new, lesser role.
The net effect was usually that Application X for Windows, v1, was a buggy, clumsy product. It was followed a few months later by a less-buggy, clumsy product. These apps were usually written pre-MFC 1.0. They didn't understand the MDI interface. They didn't use standard Windows controls. Their dialogs tried to control the application instead of vice versa. And the print system was usually ported by someone who could print in ANSI codes in his sleep but was a novice when it came to device contexts and the Windows driver model. I could go on.

But I digress. That was just an illustration of the choices that we faced (granted, 10 years later) in moving our embedded app to CE. We decided to go with a two-pronged approach. The real-time part would run alongside the kernel and the user interface part would run as a stand-alone application. We would use some kind of shared memory (we didn't know what yet) to communicate between them.

The smallest timer resolution in CE is 1ms, but we needed something much more accurate. We included the profiler and overloaded the profiler interrupt to poll a little faster, like oh about 20,000 times per second. (Just a few snips here and there in timer.c.) I know that it's not the Windows Way, and now we won't be able to use the profiler. But hey, this is still DOS2Win, v1. We're allowed a few of these.

The CE OAL makes it easy to write kernel-level code via kernel IO control codes. (Course 2535, Lab 5) Writing code for custom control codes (called IOCTLs) in oemioctl.c compiles the code into the kernel, where you can call it in the application via KernelIOControl later.

Figuring out how to run code faster than 1ms and how to run in kernel mode were two big steps to porting our trusty old code to CE. But we still needed to get the apps to talk to each other, figure out how to control the real-time app from the UI app, and test the whole thing on some hardware to see if it ran fast enough. All of these were milestones that loomed large on our ever-diminishing horizon.

Monday, March 14, 2005

Fits and Starts

It was becoming quickly apparent, even to lummoxes such as myself, that the timeline was far too agressive. I put together a quick spreadsheet to show that even with another developer (which we didn't have) we wouldn't be able to finish the project in time. I held a meeting with Alec and Rutherford. Rutherford, in classic style, took my two pages of notes and reserved a conference room for the three of us to talk about them.

   "So what are you saying?" he asked, finally giving me free reign without asking a leading question.
   "I'm recommending that if the date is more important than the product, we should do this in Zinc, then back-port it to Windows CE. If the product is more important, then let's do it in CE and move the date."
   The mention of Zinc was like offering to kill his dog, but the proposition to move the deadline made him pale in comparison. He started shaking his head as I was talking, as if to let me know that no matter what I said he wouldn't approve.
   "So what you're saying," he attempted to summarize as he walked towards a whiteboard, "is that for a greater chance of success we could do it in Zinc-DOS. But if we added a programmer," he emphasized, using my project-won't-be-complete-even-with-additional-help spreadsheet in a way I hadn't intended, "we might be able to finish it on time."
   Trying to put it into terms we were all familiar with, he continued. "So if we do it in Zinc, then move it later, it's like getting a...double. But if we hire another programmer and do it in CE Microsoft it's like hitting a home run."

Oh no! The dreaded appearance of a climactic sports term in an Important Meeting! I've seen too many important decisions swayed people who casually threw out phrases like "home run," "slam dunk" and "touchdown." They're great. They get your blood pumping. Yeah! A slam dunk! Let's see the replay! Awesome! Let's go do this, uh, thing! Uh, what were we talking about?

   "Yeah," I allowed, cringing. "We might hit a home run."

Alec sided with Rutherford, and so it was decided--as much as it was ever really a decision--to go ahead with CE and hire a contract worker for additional development. We needed an expert to help show us the ropes. I recommended the instructor from the course I had just taken. He was available. I posted an ad in a CE newsgroup. We contacted Doug Boling, who was busy on another project. You would think these guys would be perfect--after all, they have the guy who wrote the definitive MFC book--but they couldn't get a quote together.

For my part, it was time to sit down and learn all about app development on CE with MFC. MFC isn't Microsoft's darling anymore. That prize goes to C# and the .Net Framework these days. (Don't be surprised if you see a billboard that reads "C# - The official programming language of Microsoft Corp." on that link, although I don't know of any major MS applications written in C#.) As far as I know, no one has written much about MFC on CE. Maybe it's a dying breed. More likely, it's just not cool anymore and it won't sell books.

Despite the lack of printed matter, the web proved to be an excellent resource. I saved some of the links that helped me along the way:

CE Architecture


CE Graphics

It was really slow going at first, but I learned in fits and starts. While I'm usually the kind of guy who thrives in a completely new, sink-or-swim environment, it's always tough learning under the gun. The newsgroup is an excellent resource. I have a theory that if you want to learn something fast, get yourself to the point where you can answer (some) other people's questions. Then you'll know that you at least know something.

Then one week, after almost a month of futzing around with example code, reading books and tutorials, and tinkering with prototypes, it all clicked. I had just gotten back from vacation, it was shortly before Christmas, and I decided to implement what I knew. I figured I could always learn more, but I might as well start somewhere. That was a great week. I wasn't even working very much overtime. I was in the zone. (It must've been all those home runs and slam dunks.) I wrote over 2,500 lines of code that week. And the application started to come alive.

Sunday, March 13, 2005

MSDN 2540

Warning: boring reading ahead!

Here are the contents of the MSDN 2540 course:

Update: Embedded Fusion now offers MSDN 2540N on DVD. It gives you the option of spending $500 for a dry course on DVD rather than $3,000 + expenses for a dry course in person. I don't know if you can place a value on having a good instructor help you through the labs, but $500 is pretty cheap in comparison.

Update to the update: (from a comment on an earlier post) 2540, 2530 and 2535 are the Windows CE .NET V4.x versions of the MS training for Windows CE. MSDDN 2540N is the V5.0 content that was completely re-written for V5.0 [Ed. note: "completely" might be a bit of a stretch, as in how the Platform Builder 5.0 docs were "completely" rewritten from 4.2.]

  1. Module 1: Advanced Building of a Windows CE Image
    • Directory Structure of Platform Builder
    • The Build Process
    • Building in the Integrated Development Environment
    • Modifying Configuration Files
    • Creating a Component
    • Modifying CEC Files With the CEC Editor
    • Adding a BSP
    • Export Wizard
    • Lab 1: Advanced Image Customization and Build

  2. Module 2: Advanced Debugging
    • Using Breakpoints
    • The Real World: Debugging Process
    • Using Remote Tools
    • IDE Debug Commands
    • What are Debug Zones?
    • Other Debugging Techniques
    • eXDI
    • Lab 2: Exploring Debugging Features

  3. Module 3: Kernel Features
    • Definition of Real Time
    • Windows CE Kernel Features
    • Handling Processes, Threads, and Fibers
    • Protecting Applications
    • Synchronization Objects
    • Memory Model
    • Structured Exception Handling
    • Optimizing a Windows CE Device
    • Lab 3: Kernel Tracking Thread Priorities

  4. Module 4: Understanding Device Drivers
    • Windows CE Architecture
    • Driver Source
    • Driver Resource Management
    • Resource Manager
    • Device Driver Interrupt Handling and the IST Model
    • Device Driver Memory Management
    • New DMA Routines
    • Driver Loading Mechanism
    • Service Manager (Service.Exe)
    • Device Notification Mechanism for Applications
    • Device Power Management
    • Common Driver Architectures
    • Native Device Drivers
    • Stream Interface Drivers
    • USB
    • NDIS
    • FSD
    • Emulation Support
    • Lab 4: Implementing a Stream Interface Driver

  5. Module 5: Networking and Communications
    • Web Technologies
    • Lab 5.1: Enabling a Web Server
    • Networking Options
    • Lab 5.2: Implementing Microsoft Message Queuing
    • Component Services
    • Networking Security
    • Real-Time Communications
    • Multimedia Communications

  6. Module 6: Implementing a Shell
    • What Is a Shell?
    • Shell Options
    • Components of a Custom Shell
    • Required Shell Functionality
    • Installing a Custom Shell
    • Handling Device UI Rotations
    • Customizing the UI
    • Lab 6.1: Implementing and Handling Screen Rotation
    • Lab 6.2: Windows CE Skin Implementation

  7. Module 7: Preparing an Application Development Environment
    • Windows CE .NET Application Development
    • Native Application Development
    • Win32 Programming Primer
    • Lab 7A: Exploring Embedded Visual C++ 4.0
    • Managed Application Development
    • Building, Deploying, and Launching the Application
    • Overview of the .NET Compact Framework
    • Adding .NET Support to a Windows CE Device
    • Lab 7B: Exploring Visual Studio .NET

MSDN 2535

Warning: boring reading ahead!

Here are the contents of the MSDN 2535 course:

  1. Module 1: Advanced Building of a Windows CE Image
    • Directory Structure of Platform Builder
    • The Build Process
    • Building in the Integrated Development Environment
    • Building in the Integrated Development Environment (continued)
    • Modifying Configuration Files
    • Creating a Component
    • Modifying CEC Files With the CEC Editor
    • Adding a BSP
    • Export Wizard
    • Lab 1: Advanced Image Customization and Build

  2. Module 2: Advanced Debugging
    • Using Breakpoints
    • The Real World: Debugging Process
    • Using Remote Tools
    • IDE Debug Commands
    • What are Debug Zones?
    • Other Debugging Techniques
    • eXDI
    • Lab 2: Exploring Debugging Features

  3. Module 3: Kernel Features
    • Definition of Real Time
    • Windows CE Kernel Features
    • Handling Processes, Threads, and Fibers
    • Protecting Applications
    • Synchronization Objects
    • Memory Model
    • Structured Exception Handling
    • Optimizing a Windows CE Device
    • Lab 3: Kernel Tracking Thread Priorities

  4. Module 4: Implementing a Boot Loader
    • Role of the Boot Loader
    • Boot Options
    • Implementing a Boot Loader
    • Sample Boot Loader Requirements
    • Building a Boot Loader
    • Debugging a Boot Loader
    • X86 Boot Options
    • Boot Loader in Manufacturing

  5. Module 5: OEM Adaptation Layer
    • Operating System Boot Sequence
    • Developing OAL
    • Required OAL Functions
    • Optional OAL Functions
    • Debugging an OAL
    • Building the Windows CE Kernel
    • Implementing OAL Registry Functions
    • Power Management
    • Implementing Certification Model
    • Lab 5: Exploring the OAL

  6. Module 6: Device Driver Architecture
    • Built-In Vs. Installable Drivers
    • Device Manager
    • ActivateDeviceEx
    • Registry Enumerator
    • Services
    • Bus Drivers
    • DMA
    • Resource Manager
    • Interrupt Model
    • Device Driver Power Management
    • CETK
    • Lab 6: Implementing a Stream Interface Driver

  7. Module 7: Device Drivers Examples
    • Driver Source
    • Arrangement of the Driver Libraries
    • Battery and Notification LED Drivers
    • PC Card Socket
    • Keyboard Device Drivers
    • Display Drivers
    • DirectDraw
    • USB
    • 1394
    • NDIS
    • Audio Device Drivers
    • File System Driver
    • Lab 7: Implementing a File System

MSDN 2530

Warning: boring reading ahead!

Here are the contents of the MSDN 2530 course:

  1. Module 1: Overview of Windows CE .NET
    • Overview of Windows CE .NET
    • Core Operating System Architecture
    • Advanced Features of Windows CE .NET
    • Networking and Communications
    • Real-Time Communication
    • Multimedia Support and Architecture
    • Security Features and Architecture
    • Internationalization

  2. Module 2: Overview of System Development
    • Selecting a Windows Embedded Operating System
    • The Windows CE Platform Development Cycle
    • The Application Development Options

  3. Module 3: Configuring and Building with Platform Builder
    • Introduction to Platform Builder
    • Platform Development Cycle
    • Configuring the Platform
    • Building the Platform Image
    • Downloading the Image
    • Demonstration: Configuring, Building and Downloading

  4. Module 4: Overview of the Windows CE .NET Debugging Process
    • Overview of the Debug Process
    • Release Builds and Debug Builds
    • Introduction to the Kernel Debugger
    • Demonstration: Kernel Debugger Windows
    • Understanding Windows CE Remote Tools
    • Other Debugging Techniques

Friday, March 11, 2005

The Beginning, A Very Good Place To Start

Everything must start at the beginning. (I think I read this pearl of wisdom somewhere.) Our beginning involved a lot of speculation about what we could and couldn't do. The Filipinos have a term for it: pauli uli, or as we say it in English, running around like a chicken with your head cut off. Jack Ganssle says it better: furious activity is no substitute for understanding.

Unfortunately, understanding was at a premium, so we made the requisite substitution with a big helping of furious activity. I filled out the online form for a Windows Embedded Introductory Kit. (In hindsight, I strongly recommend ordering the kit on CD or DVD. The downloads can take literally hours, even with a high-speed Internet connection. And what if you have to reinstall... I did. Twice.)

The kit comes with Platform Builder 5.0 and Embedded Visual C++ 4.0 (eVC). (Visual Studio 2003 will only compile apps for WinCE if you want to use the .Net Framework.) Whidbey, the upcoming Visual Studio 2005, will have integrated support for compiling native apps for CE, and eVC will say good-bye. (eVC is really just a tempermental, stripped-down Visual Studio with Platform Manager support, anyway.)

We bought a book by Doug Boling, Programming Microsoft Windows CE .Net. It's a good book, but it was ironic that we had decided to write the UI app in CE with MFC. The book proudly states in the introduction: "Since the goal of this book is to teach you how to write programs for Windows CE, the examples avoid using a class library such as MFC [...]" (!) It's a good book, and I recommend it despite the pages and pages of cut-and-paste source listings.

Eventually it was decided that I would attend a CE training session. There are a few companies that do such training, and only two were offering a session in November: VenturCom (now Ardence) in Boston and BSQUARE in Seattle. We chose the one on the right coast and off I went for a week of training. (George didn't go, which was a shame, but he works part time.)

On the flight to Boston I happened to sit next to a Director of Engineering for General Dynamics. We talked and I showed her the software spec and our timeline. A former engineer herself, she was quick to tell me how nuts we were. In return, I scoffed at their policy that a productive programmer wrote just 300 lines of code per week, but I started to have my doubts whether our timeline was doable.

The training was excellent, thanks to the instructor. The MS docs were out of date (and sometimes plain wrong) and the computers we were working on were less than stellar, but the quality of the teacher more than made up for it. Remember that favorite teacher of yours back in high school? The one who really loved what he did, was good at it too, and showed a geniune interest in you? They say that the instructor can make or break the course, and this guy made it.

Windows CE training comes in three courses: MSDN 2530, 2535 and 2540. The meat of the course is in 2540 but the other two are good as well. I'll post the TOC for all three courses in some upcoming posts.

Update: Embedded Fusion now offers MSDN 2540N on DVD. It gives you the option of spending $500 for a dry course on DVD rather than $3,000 + expenses for a dry course in person. I don't know if you can place a value on having a good instructor help you through the labs, but $500 is pretty cheap in comparison.

With my week-long CE training firmly under my belt I was ready to tackle the world. I wrote a summary of my trip in an email to family and friends and flew home.


Monday, March 07, 2005

Why Windows CE?

It looks like I've gone live, thanks to Mikehall's "look at Windows CE, Windows XP Embedded (and anything else that's cool or interesting)." Since this is in the CE category, maybe that means it's not cool or interesting. But hey, I'll take one out of three. I should also mention that I've never written a weblog before. Any tips some of you more experienced posters have would always be appreciated. (Heck, I'll even take grammer tipses.)

So why did we choose Windows CE? There were a lot of reasons; most of them were good. As I mentioned before, we were using an unsupported platform for development. But we'd used it for ten years, so we knew exactly what we could and couldn't do. But now (this was early August 2004) we were at a crossroads. A major OEM had contacted us and arranged a demo of an as-of-yet uncompleted system for March 2005. Like any company in our shoes, we agreed to the date first and asked the engineers second.

George and I had come up with a rough DOS/Zinc timeline that was already aggressive. Perhaps "timeline" is the wrong word. It was a deadline with tasks stuffed into it to make them fit, AKA a "Stalin deadline." ("There's no line like the deadline," Stalin used to say. OK that sucked I just made it up.) Alec, our new Engineering Manager, was eager to make a good impression with the Director. Alec was a wizard at chip design. He was in his early forties, but his young looks and great hair were betrayed by his considerable paunch. We held a meeting to see how we could cram the timeline into the deadline.
"Come on, a week to work on fonts?" Alec asked with mock incredularity.
"Okay, maybe I can do it in 3 days," I allowed.
We hadn't factored an OS change into the mix.

Looking back, I think our options were fivefold:

  1. Continue using Zinc
    This would've required that we reinstate our licenses for two developers, upgrade to the current shipping version, and buy a high-speed graphics library.
  2. Zinc's former big brother: VxWorks
  3. Embedded Windows: CE or XP
    We never really considered XP because of its size.
  4. Linux
    As Donald Rumsfeld was (in)famously quoted, this was a "known unknown." I have some experience with Linux and SunOS, but not in the embedded world. And Linux isn't a real-time kernel without real-time extensions. (Not including projects from sites that end in .edu, which we weren't seriously considering.)
  5. Something else or roll our own: ???
This was a good starting point, and we used it to start looking at prices. WinCE got a boost from the fact that our potential SBC manufacturer provided a BSP for its PC/104 SBC.

Our potential costs looked something like this:

Zinc~$6,000/developer* + royalties**
VxWorks~$10,000/developer + annual maintenance + royalties
WinCE$995/developer for Platform Builder + $3/device***
LinuxIf the point of using Linux is that you get a free, open-source kernel, paying big bucks for real-time extensions doesn't seem to make sense. So Linux didn't make the cut.

* For starters we needed licenses for two developers
** Our current Zinc run-time license didn't include royalties
*** $3 is for the core OS. The Pro version, which includes little niceties like Help, costs $16/device. We thought that we would only need the core OS. Also, Platform Builder came with a 120-day evaluation period. (Tip: definitely get the CD or DVD; the download-over-the-web install can take up to 4 hours, even on a T1.)

By comparison, our current costs were $0/developer and about $25/MS-DOS license. (In case you're wondering, yes, 24 years later Microsoft still sells MS-DOS.)

If you read through that table just now like I read through a similar list of options back in October, I think you'll agree that it wasn't much of a choice. There were still some critical questions to answer, though. Could the bulk of a modern OS run as fast as practically running on bare hardware? How hard would it be to port 100,000 lines of code over to CE? Would our I/O handling be fast enough?

One of the biggest problems, which I may post about later, is the bewildering way that Microsoft markets CE. What is Windows CE? Eight years after its release, I'd wager that a lot of people in the embedded industry couldn't give me the right answer. MS has done a great job of confusing the market about CE and Pocket PC. (As a bit of proof I submit: the first MS-owned link from a "Windows CE" Google search.) And putting the ".Net" moniker in CE 4.2 was about the worst thing they could've done in the emdedded market. MS already has a reputation for bloatware; what self-respecting engineer wants to put .Net--which for all intents and purposes to most people means the .Net Compact Framework--in an embedded controller?

I found some good articles that helped reassure us about CE:
The last article, about running .Net managed code and still getting mostly deterministic behavior, reassured us that Win32 code could at least run faster than 50-odd µsecs. Still, without taking the steps to port a good portion of the codebase to run a test, we couldn't measure how fast it would run.

But something else that was less than reassuring was a high level of confidence in CE by people who had no idea why they were so confident in it. Maybe it's the classic difference between engineers and people who manage engineers. Rutherford, our Director of Engineering, unnerved me with his confidence about switching to CE. Fiftiesh with a graying pompadour and a self-described bureaucrat, Rutherford had a background in steel manufacturing. Maybe he held Microsoft stock, maybe using a Microsoft product gave him a warm and fuzzy feeling, or maybe it was the fact that we'd recently lost a sale to a competitor whose system ran Windows. (Even though our controller ran theirs into the ground during a week of independent testing, they taunted us with our "old" DOS system. It was wrong, but it stuck, both with our non-customer and back at the office.)

For watever reason, Rutherford was set on CE. He couldn't remember its name--to this day he calls it "CE Microsoft"--but it got his vote. I remember preparing for a checkpoint meeting early on in the design phase about the control flow of the new UI. Rutherford and I were talking about the switch:
"That's if we move to CE," I reminded him, expressing my doubts.
"We're going to CE Microsoft," he said pointedly.

CE 5.0 had just been released, but our vendor was only supplying a BSP for 4.2. Version 5.0 has some great features for developers but there's little incentive from a customer standpoint to want to upgrade. Someone in Korea took the time to put together a really nice PDF that shows a comparison of CE 5.0 vs. 4.2. (Update: Found the original.)

In case you're wondering if I think we made the right decision, I absolutely do. There's no question in my mind that we needed to move away from DOS and start using a more modern RTOS and better toolchain. Whether this project was the best project to jump into a new OS is what I'm less sure about.

In Detroit and at other car-makers, a common engineering practice is to change one major system on a new model, but not more than one. For example, BMW might change the engine on their 3-Series, or put it on a new frame, but they don't typically do both in the same year. We were changing everything: new hardware, a new OS, new software and a new toolchain. And we were doing it on a timeline that was so tight it didn't allow for losing even a half-day. It was the classic case of biting off more than we could chew.

With the decision to use CE locked in place, we were all set to start porting our real-time app and writing the new UI app. The only problem was that none of us had written for CE before, and we didn't know where to start!

Sunday, March 06, 2005

Good Ole Days

This is all leading up to why we chose Windows CE. It's taken me longer than I thought I would, and I think I should (quickly) explain our previous and current systems before I discuss our Big Decision.

No engineering company is complete without the folklore of its past, and ours is no exception. We began with the first controller, as most companies do. It practically used vacuum tubes and mechanical relays to accomplish the ridiculously easy task of checking the glue level and sounding an alarm if it got too low. That grew into controllers that were dedicated to looking for some particular pattern of events. They were hardware-only. We even sent our customers manuals with instructions for how to set a row of 8 DIP switches to represent binary distances! We quickly left that behind in favor of the enormous, 8-bit power of the Boss Bear, which featured a 2-line, 40-character display.

Unfortunately, the Boss Bear BASIC programs were limited to 32K, which meant that as the program matured we had to strip out features to include others. Sometimes 3 or 4 lines of code were crammed into a single line of text to save space. Variables names were shortened to the minimum, comments were left completely out. That suited George just fine, who would rather not comment anything anyway. George was and is the longest-tenured programmer here. He has a slight build, wears his hair long, and works part time. He lives away from the office but commutes in once or twice a month. He's quirky to work with and at the same time he's one of the most likeable guys in the world. In some ways, he reminds me a lot of Mel; the guy could probably write a compiler in 8 lines of code using only pointer arithmetic, preprocessor directives and macro expansion. Uncommented, of course. One day I'll have to write about George, but that's a post for another day.

Well, the world of Boss Bears came and went, and we pushed into the age of embedded boards running on Intel chips with our next system. It was written in a mixture of C/C++ and ran on a 486 DX2/66 on the STD bus. It used a 10.4" EL display and had a full keypad and dedicated buttons on the front. It was a marvel of its day. The display used to burn in within weeks, but that was a small price to pay compared to the potential of this 486 powerhouse!

The program ran on MS-DOS 5, which was overclocked to give us a 50µsec polling loop. It used the Zinc Framework. If you've never heard of Zinc you're not alone. In fact, the nine of us who have heard of Zinc are the oddballs. Zinc employed a write-once, compile-for-anything idea. It wasn't actually "anything," but they started with compile keys for 31/2 major OSes: DOS, Windows, Unix and OS/2. Zinc was too far ahead of its time. If the tides of history had flowed a little differently, it might have been what Java is today. Unfortunately, the company fell on hard times and was bought by WindRiver. But WindRiver couldn't figure out what to do with it and later sold it to PSA, a group of former Zinc consultants.

Eventually, though, our customers demanded more and more powerful machines, so the next version was born. We stuck with the STD bus but upgraded the processor to a Pentium, sans bunny people and blue people, which ran as hot as that SBC form factor would allow. The display was upped to a 12" TFT running at 640x480 but the effective area stayed the same. The right side and bottom were covered with a two-column keypad for data entry and a button strip for commonly-used functions.

Astute readers have probably noticed a pattern emerging from generation to generation. In each new system, software grew more powerful and took over what had been dedicated hardware before. Previously, for example, for an important toggle button we had a physical pushbutton that was mounted into the face of the controller, and wired into I/O. Now it was done with a screen overlay.

And the next controller, the Oswald Project, called for zero hardware buttons. Instead, a single 15" display running at 1024x768x16bpp would take care of all controller functions. But we had a problem: Zinc couldn't drive anything higher than 640x480 without a new graphics library. And that library didn't work on the version of Zinc that we had. And the license for the version of Zinc that we had was expired.

The time was ripe to pick a new development system and toolchain. Up next: our choice of OS (for real this time).

Saturday, March 05, 2005

The Company

If I was a box maker and I wanted to find a bad box that was moving at more than 20 MPH and pull it out of my production line, how would I do it? It turns out that it's a very difficult and unique problem. That question, and others related to carton gluing and detection, are worth about $50 million annually to our tiny niche of the packaging market.

But our niche solves an important problem. (Every business exists because it solves some problem. And businesses thrive when they solve the right problems for the right people.) We solve a problem that is unique, we solve it very well, and there are big companies that are willing to pay us a lot of money to provide them with our solution.

In the world of boxes there are two major types: corrugated (the kind of boxes we used to move into our house last summer ) and cartons (the kind of box that my Cinnamon Life comes in). The corrugated market is a down-and-dirty market. Boxes are cheap, and so is the equipment that runs them. But the folding carton market is a gold vein. Anything that comes in a box--from a tiny tube of Blistex to a big box of Cheerios to a smaller tissue box to a complicated bottle carrier--is produced in the folding carton market. Consumer-oriented companies have one way to get you to buy their product when you're at the store: the look of the packaging that it comes in. So it's in their best interests to produce the best looking cartons that will catch you or your kids' eyes first.

It's a huge business. The next time you finish your last can of Coke, rip open that Coca-Cola Fridge Pack™ (Pepsi countered with its Fridge Mate™--how original) and you'll notice three or four patent numbers printed on one of the flaps. Most people, myself included, would never think that someone would patent something as simple as a box. But if you look at them closer, they're not so simple. A lot of engineering goes into the designs and several Fortune 500 companies are willing to spend a lot of research dollars to preserve their competitive edge.

Cartons are usually printed on 4- or 6-color presses, then run through a cutter, then loaded onto a folder-gluer. The folder-gluer is a bit of a misnomer: it folds, but it doesn't usually glue. That part is left up to systems like ours, which glue the boxes, and do quality control on them. Any spoilage on the gluer costs the most. The most expensive part of the life cycle of a carton is on the gluer. Even worse, a single misfed or skewed carton can jam up the entire line and ruin hundreds of other cartons. The Georgia pine used in the substrate of some cartons, like soft drink 24-packs for example, is so strong that when it jams up it can bend the steel of the machine.

But it's not just the strength of the cartons that bends the machine, it's the speed that they move at. Some gluers can run upwards of 200,000 cartons per hour (CPH). That's really, really fast. Even 60,000 CPH is so fast that when I look at it it looks like a solid sheet of board moving past.

Even at those speeds, we produce a system that can lay up to 16 lines of glue on a single carton, monitor that glue (or any other pattern), and pull out a single bad carton from the jet stream of good ones. It's not cheap; a complete system can cost around $80,000. But if you're the guy who just bought a $500,000 folder-gluer, we're in business to make sure that you pay a little more to get a great gluing/detection system to go along with it.

It's taken us a few tries to get it right. Our current system is the fifth generation of controllers that we make. This generation is as perfect as it can get, but it's also a maxed out system, and our customers are always clamoring for more features. So our next generation of controllers will be expandable to 24 patterns, big enough to run even those tricky beverage carriers. And it will be cheaper. It'll also run faster. It will be easier to use, too, thanks to a completely reinvented GUI. Oh, it'll also solve world hunger. Enter the Oswald Project.

Friday, March 04, 2005

Hello, WWWorld

Partly at the suggestion of Mike Hall, I've decided to start a weblog about a very, very small corner of the universe known as Windows CE. Specifically, this is a blog about a real-world project using CE.

There are other blogs about WinCE written by people who know far more about it than I may ever know. (I'll link to them as I find them.) And there are some good online tutorials for getting started with CE. But as far as I know there are no weblogs about the trials and triumphs of actually working on a corporate CE project. So I'm starting one today.

Today is March 4, and this marks the seventh month of the Oswald Project, although in all fairness we've only been doing serious development since Jan. 4 of this year. The finish date for this project was supposed to be March 25, which is a scant 3 weeks away from today. But despite our three month death march, we are still far behind schedule and will badly miss our target date. That day was supposed to be a live demo at one of the most important (potential) customers of my company's 76-year life.

Speaking of my company, I will try to post everything that I can about this project within the confines of a competitive marketplace. In the ideal world of academics and research I wouldn't have any qualms about posting information relevant to this project. But in the dog-eat-dog marketplace of free enterprise and international espionage, there is a possibility that our work does not go unnoticed by our competition. And our competition has been nipping at our heels as of late...

So to protect the innocent--and the guilty, of which there are a few--the names and specifics have been changed. I will call this project the Oswald Project. Oswald is an obscure painter.

Painting aside, this blog will mostly cover the technical details of what it takes to create a brand new Windows CE project and ship it to a customer. But if I start to wax philosophical or wander off into stories of my personal life, please forgive me. I’ll soon get back on course.

I also have a lot of background to write to get you up to speed. (We'll see how that fits with my 70-hour work weeks.) But eventually, I'll catch up to real life and start posting about what's currently happening. Until then these posts will have a historical bent. Perhaps those are better: with the passing of time, feelings and opinions fall by the wayside and only the facts remain.

Up next: some background on the company and our reasons for choosing WinCE.

This blog Copyright © 2005 D. Philippe